Juno splits transformer inference across JVM processes connected by gRPC. Two distribution
strategies are available, selected with --pType at startup. This page is the authoritative
description; the feature overview and README link back here instead of repeating it.
Pipeline parallel (--pType pipeline, default)¶
Transformer layers are split into contiguous blocks and assigned to nodes. The activation
tensor flows serially: node-1 -> node-2 -> node-3. Each node holds a contiguous depth
slice. Adding nodes increases total VRAM, enabling larger models. Cost: N-1 sequential gRPC
hops per decode step.
Every node also wires a NodeKVCacheAdapter into its handler, and, if a LoRA adapter is
attached, a read-only LoraAdapterSet.
Tensor parallel (--pType tensor)¶
Every node holds all transformer layers but only a horizontal slice of the weight matrices:
attention heads [headStart, headEnd) and a proportional FFN width slice. The coordinator
broadcasts the input token embedding to all nodes simultaneously, collects partial logit
vectors, and reduces them via element-wise sum (star AllReduce). Adding nodes increases
throughput and reduces per-node memory pressure. Cost: one broadcast + N parallel gRPC calls
per decode step.
Constraint: numHeads % nodeCount == 0.
Star AllReduce requires no InfiniBand and no inter-node communication. The coordinator collects and sums in O(N x vocabSize).
See also¶
<- 2.1 Overview | Table of Contents | 2.3 Handler Routing ->