JUno

Why Juno?

• Your organization accepts only on-prem or air-gapped deployments, with strict disclosure policies, or runs clusters of commodity or heterogeneous hardware.
• Your organization has mature Java teams and infrastructure, or enforces strict security policies against Python processes running in production.
• You are building Edge or IoT clusters with small heterogeneous machines and need LLM inference without a Python runtime dependency.
• You are building Java-native products - especially in fintech, enterprise tooling, or any domain where the JVM is the standard deployment target.

Quick Start

We may use jUno in several ways. All share the same GGUF model file and the same CLI launcher - only the invocation differs. Full flag reference and examples: docs/howto.md · README.md.

But let's check out a model first:

# example - TinyLlama 1.1B, ~637 MB
wget https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/\
resolve/main/TinyLlama-1.1B-Chat-v1.0.Q4_K_M.gguf -P models/

Console modes

local — Offline or stand-alone mode, no JVM cluster no gRPC wire. The fastest way to try a model locally or run LoRA training.

./juno local --model-path models/TinyLlama-1.1B-Chat-v1.0.Q4_K_M.gguf

cluster — Offline and stand-alone mode, with JVM cluster on localhost with gRPC wire. The fastest way to test distributed behavior of model locally.

./juno cluster --model-path models/TinyLlama-1.1B-Chat-v1.0.Q4_K_M.gguf

lora — Offline or stand-alone mode, no JVM cluster no gRPC wire. The fastest way to train a model locally using interactive LoRA console.

./juno lora --model-path models/TinyLlama-1.1B-Chat-v1.0.Q4_K_M.gguf

Stand-alone HTTP API server

Option --api-port N to any local or cluster jUno invocation will enable OpenAI-compatible HTTP server. And expose POST /v1/chat/completions and GET /v1/models. Any client that speaks the OpenAI wire format - LangChain, LlamaIndex, the OpenAI SDK - works with only a base-URL change.

./juno local --model-path model.gguf --api-port 8080

Programmatic integration

Import one POM so every cab.ml module shares the same version:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>cab.ml</groupId>
      <artifactId>juno-bom</artifactId>
      <version>0.1.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>
<dependencies>
  <dependency>
    <groupId>cab.ml</groupId>
    <artifactId>juno-player</artifactId>
  </dependency>
</dependencies>

Then play or train your favorite model right from your code.

Distributed deployment

Deploy juno-master and juno-node within your network to obtain distributed inference. Choose parallelism type, byte order and other options to meet your hardware/software in the best condition. (See full examples in scripts/aws/juno-deploy.sh.)

# coordinator — one active instance per cluster
/usr/bin/java \
  --enable-preview --enable-native-access=ALL-UNNAMED \  # JDK 25 required
  --add-opens java.base/java.lang=ALL-UNNAMED \          # reflection access
  --add-opens java.base/java.nio=ALL-UNNAMED \           # NIO access
  -XX:+UseG1GC -XX:+AlwaysPreTouch -Xmx4g \             # GC + heap
  ${JFR_OPT:+$JFR_OPT} \                                # optional: --jfr 60s
  -DJUNO_HEALTH=true -DJUNO_HEALTH_PORT=8081 \           # health dashboard
  -jar juno-master.jar \
  --model-path /models/model.gguf \
  --pType pipeline \                                     # pipeline | tensor
  --dtype FP16                                           # FP16 | FP32 | INT8

# worker node — repeat on every inference host
/usr/bin/java \
  --enable-preview --enable-native-access=ALL-UNNAMED \
  --add-opens java.base/java.lang=ALL-UNNAMED \
  --add-opens java.base/java.nio=ALL-UNNAMED \
  -XX:+UseG1GC -XX:+AlwaysPreTouch -Xmx12g \            # larger heap for shards
  ${JFR_OPT:+$JFR_OPT} \                                # optional JFR profiling
  -DJUNO_USE_GPU=true \                                  # false → CPU quantized
  -Djuno.byteOrder=BE \                                  # BE (default) | LE
  -Dnode.id=1 \                                          # unique per node
  -Dnode.port=50051 \                                    # gRPC listen port
  -Dmodel.path=/models/model.gguf \
  -Djuno.lora.play.path=/adapters/model.lora \           # optional LoRA adapter
  -Djuno.health.url=http://master:8081/health \          # optional health report-back
  -jar juno-node.jar cab.ml.juno.node.NodeMain

AWS Cluster Deployment

scripts/aws/juno-deploy.sh is the unified cluster lifecycle script. Hardware is auto-detected during bootstrap: GPU instances set JUNO_USE_GPU=true (CUDA is pre-installed in the golden AMI). The OpenAI-compatible API is available on port 8080 after setup. Full options and LoRA deploy flow: docs/howto.md - AWS.

One-time setup

cd scripts/aws
nano launcher.sh  # set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION

Commands

# GPU cluster (3 x g4dn.xlarge, T4 16 GB VRAM)
./launcher.sh juno-deploy.sh setup

# CPU cluster (3 x m7i-flex.large)
./launcher.sh juno-deploy.sh setup \
  --instance-type m7i-flex.large \
  --model-url https://huggingface.co/.../model.gguf

# Deploy with LoRA adapter
./launcher.sh juno-deploy.sh setup \
  --lora-play /absolute/path/to/model.lora

# Teardown
./launcher.sh juno-deploy.sh teardown

Architecture

Two distribution strategies, selected with --pType. Pipeline (default): contiguous layer blocks, serial activation flow, LAN-friendly, no InfiniBand required. Tensor: all nodes hold all layers but a horizontal weight slice; coordinator broadcasts tokens in parallel and sums partial logit vectors (AllReduce).

On-prem and cluster topology

Run juno-master as the coordinator and juno-node on each worker with gRPC between them. Any node count is supported - use pipeline parallelism to scale total VRAM, tensor parallelism to scale throughput. The same roles run locally (forked JVMs), on bare metal, or on cloud instances. Commodity hardware works well; so does modern datacenter hardware.

Sequence: generation loop

On startup, ClusterHarness forks node processes. Each calls ForwardPassHandlerLoader.load(), routing to LlamaTransformerHandler or Phi3TransformerHandler based on general.architecture in the GGUF. EmbeddedNodeServer wires a NodeKVCacheAdapter after loadShard(). On inference: coordinator tokenizes, runs prefill+decode. Session KV blocks are reused across turns - only new tokens are prefilled on turn N+1.

Modules

Multi-module Maven project. All cab.ml artifacts at version 0.1.0 are available on Maven Central. Import juno-bom to align versions. Details: docs/arch.md.

Supported Models

Any GGUF file with a supported architecture. Chat template is auto-detected from the model filename.

Architectures

Quantization types

F32 · F16 · BF16 · Q8_0 · Q4_0 · Q2_K · Q3_K · Q4_K · Q5_K · Q6_K

All quantization types stay compressed in memory; dequantization runs block-by-block inside the matmul loop. On GPU, Llama and Phi-3 dequantize once at load time and keep weights as FP16 on device (DeviceHalfMatrix).

CLI Reference

./juno is the unified launcher at the project root. Requires JDK 25+ and pre-built jars. Full flag table and examples: docs/howto.md.

Modes

For the full list of flags (--model-path, --dtype, --pType, --heap, --jfr, --lora-play, LoRA-specific flags, merge flags, environment variable overrides) and usage examples, see docs/howto.md.

GPU Acceleration

GPU inference via CUDA 12.x and cuBLAS. All projection weights are uploaded once at load time as IEEE FP16 (DeviceHalfMatrix); forward-pass matmuls use cublasHSSgemvStridedBatched. Per-call H2D transfer is limited to the small input/output activation vectors - CPU cores are idle during generation.

• FP16 resident weights. Both Llama and Phi-3 handlers dequantize once on load and keep weights on device, roughly halving VRAM versus FP32 resident.
• CUDA streams. Per-thread non-blocking streams with cudaMemcpyAsync; cuBLAS calls are serialized with a per-context lock.
• Multi-device. One shared GpuContext per CUDA device. Pin a node to a specific GPU with -Djuno.cuda.device=N.
• OOM fallback. On cudaMalloc failure, partial device buffers are closed and inference falls back to CPU quantized matmul for those projections - no crash, no restart.
• Explicit lifecycle. releaseGpuResources() frees VRAM on shard unload, reload, or handler swap.

LoRA Fine-Tuning

Parameter-efficient fine-tuning via low-rank adapter matrices. The base GGUF is never modified. Adapters persist to a .lora checkpoint; the same handler serves training and inference. Full guide, rank selection, and checkpoint format: docs/LoRA.md.

• ./juno lora - fine-tuning REPL. Use /train for free-text, /train-qa for Q&A pairs (auto-generates 4 phrasings with the model's chat template).
• --lora-play PATH - apply a pre-trained adapter read-only at inference, in both local and cluster modes. In cluster mode the adapter is injected into every forked node JVM.
• ./juno merge - write a new GGUF with the 44 LoRA-patched projection tensors stored as F32 (the LoRA delta is smaller than Q4_K noise; re-quantizing would erase training). All other tensors are copied verbatim.
• JVM API. LoraTrainer (in juno-player) exposes trainRawText(), trainQaPair(), and save() for programmatic use.

Health Dashboard

The health module exposes an auto-refreshing dashboard served at GET / on the health port. Each node runs a HealthReporter that probes local metrics every 5 s. Per-node cards show VRAM, CPU load, circuit state, and a role-conditional metric: Latency P99 on the coordinator, Throughput MB/s on worker nodes.

Circuit-open nodes are highlighted in red; half-open in amber. The coordinator's port 8080 also embeds a simplified health view inline.

JFR Profiling

All hot paths are instrumented with custom Java Flight Recorder events. Pass --jfr DURATION to any command; a juno-<modelStem>-<timestamp>.jfr file is written on exit. Open in JDK Mission Control → Event Browser. Full guide and extraction examples: docs/howto.md → Metrics.

Key Design Decisions

Performance

Measured on tinyllama-1.1b-chat-v1.0-q4_k_m.gguf via AWS JFR cluster runs. CPU: m7i-flex.large. GPU: g4dn.2xlarge. TPS = coordinator-side TokenProduced.tps.

• GPU dominates at all concurrency levels. 17–37× faster than CPU; 21× faster on multi-turn where prefill cost compounds harder on CPU.
• CPU pipeline sweet spot: 3 nodes. 1.59 tps peak. Adding nodes beyond 3 degrades TPS - network sync per forward pass outweighs compute savings.
• Tensor-parallel is a net loss on CPU at this model size. 3-node tensor: 0.44 tps versus 1.59 tps pipeline. No benefit on GPU single-node either.
• CPU dtype: FP16 wins. FP16 (1.54) > INT8 (1.13) ≈ FP32 (1.06). No AVX-512 INT8 path - scalar fallback. LE byte order adds ~19% overhead vs BE.
• GPU dtype: flat. FP32 on GPU is only 7% slower than FP16 - driver upcasts internally.

Changelog

Last 5 development sessions. Scroll horizontally to see all. Full history on GitHub.