Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

11.2. Changelog

Status

Session 51: Documentation update: docs/ folder restructured as juno-documentation MyST Jupyter Book.

juno-documentation

Status

Session 50: /train-file-qa: multi-fact Q&A from a JSON file in one training loop; HTTP API.

/train-file-qa

Status

Session 49: LoRA Tier 11 (complete): --lora-microbatch CLI/env + VRAM OOM auto-fallback.

LoRA microbatch CLI and VRAM ladder (Tier 11)

Status

Session 48: LoRA Tier 9 (complete): microbatch GEMM + published GPU speed gates.

LoRA GPU microbatch and product gates (Tier 9)

Status

Session 47: LoRA Tier 10 (complete): multi-arch GPU residency + production gates.

LoRA multi-arch GPU residency (Tier 10)

Status

Session 46: LoRA Tier 9 (start → completed in Session 48): --lora-train-device productization.

LoRA GPU train-device (Tier 9)

Status

Session 45: LoRA Tier 8: train-file scheduling and corpus caps.

LoRA train-file scheduling (Tier 8)

Status

Session 44: LoRA training progress bar (loss → target).

Status

Session 43: LoRA Tier 6: multi-architecture training (CPU oracle).

LoRA multi-architecture (Tier 6)

Status

Session 42: LoRA REPL UX + WebUI model dropdown.

Status

Session 41: LoRA Tier 7 (complete): JFR metrics for all adapter modes and operations.

LoRA JFR metrics (Tier 7)

Status

Session 40: LoRA Tier 5 (complete implementation): QA-LoRA + merge policies.

LoRA QA-LoRA and quantized merge (Tier 5)

Status

Session 39: LoRA Tier 5 (Gate A start): shared GGUF K-quant codec layer.

LoRA QA-LoRA / quantized merge foundations (Tier 5 Gate A)

Status

Session 38: LoRA Tier 4 (start): resident transpose primitives and baseline instrumentation.

LoRA GPU training foundations (Tier 4)

Status

Session 37: LoRA Tier 3 (phase 1–2): rsLoRA, Kaiming, checkpoint v2, DoRA.

LoRA advanced adapters (Tier 3)

Status

Session 36: LoRA Tier 2: schedules, AdamW, dropout, validation, and LoRA+.

LoRA training quality (Tier 2)

Status

Session 35: LoRA Tier 1: projection coverage, token-weighted accumulation, and clipping.

LoRA correctness foundation (Tier 1)

Status

Session 34: Windows launcher fixed: run.bat and juno.bat fully functional on Windows.

Windows launcher (scripts/run.bat, juno.bat)

All subcommands (cluster, local, lora, merge, test) and flags are now working on Windows.

Root cause fixes:

Documentation updated:

Status

Session 33: Model support documentation: Phi-3 supported; Gemma, Qwen 2 / Qwen3 / Qwen3.5 under development.

Supported model status (docs)

User-facing docs now state a single, consistent model-support policy:

Familygeneral.architectureStatus
LLaMA, Mistral, TinyLlama, …llama, mistral, …Supported via LlamaTransformerHandler
Phi-3 / Phi-3.5phi3Supported via Phi3TransformerHandler
GemmagemmaUnder development (LlamaTransformerHandler + gemma template)
Qwen 2 / 2.5qwen2Under development (Llama handler + QKV bias groundwork)
Qwen3 denseqwen3Under development (Qwen3TransformerHandler in progress)
Qwen3-MoEqwen3moeUnder development (Qwen3MoeTransformerHandler in progress)
Qwen3.5qwen35Under development (hybrid DeltaNet; separate handler)

Updated files:

Policy: Phi-3 is production-ready in docs and validation (local + cluster). Gemma and all Qwen families remain under development until dedicated validation lands.

Status

Session 32: ROCm/HIP backend for AMD GPU inference via Panama FFI.

AMD GPU support (ROCm/HIP + rocBLAS)

Full first-class AMD GPU support alongside the existing NVIDIA CUDA backend. The GPU abstraction layer auto-selects CUDA > ROCm > CPU at startup with no configuration required. Tested on AMD Radeon RX 7900 XT (gfx1100, ROCm 7.2.x).

New production classes (node module):

Modified production classes:

New tests (55 total, 0 failures on RX 7900 XT):

Run ROCm-tagged tests:

mvn test -pl node -Dgroups=rocm

Performance (RX 7900 XT, ROCm 7.2.x):

ShapePathTime (5 runs)
32000×2048rocblas_sgemv host FP32408 ms

All existing 194 unit tests pass unchanged.

Status

Session 31: Panama FFI for Juno math: JavaCPP / bytedeco removed, CUDA bindings rewritten with java.lang.foreign.

Panama FFI GPU bindings (node module)

The entire CUDA bridge has been rewritten using the Java 25 Panama Foreign Function & Memory API (java.lang.foreign.Linker, SymbolLookup, MemorySegment, Arena). The org.bytedeco:cuda-platform dependency has been removed from node/pom.xml.

New production class:

Modified production classes:

node/pom.xml: org.bytedeco:cuda-platform dependency removed. maven-surefire-plugin argLine updated: --enable-native-access=ALL-UNNAMED, --add-opens java.base/java.lang=ALL-UNNAMED, --add-opens java.base/java.nio=ALL-UNNAMED.

New test: CudaBindingsTest: two scenarios:

Run GPU-tagged tests: mvn test -Dgroups=gpu -pl node

All existing tests pass unchanged.

Status

Session 30: Maven Central publish configuration.

Maven Central publish (pom.xml, all module POMs)

All modules configured for publishing to central.sonatype.org via the Central Portal publisher. Version set to 0.1.0-RC across root POM and juno-bom.

Changes:

Status

Session 29: OpenAI-compatible REST API (POST /v1/chat/completions, GET /v1/models).

OpenAI-compatible API

Any client that speaks the OpenAI Chat Completions wire format: LangChain, LlamaIndex, LiteLLM, the OpenAI Python/Node SDKs, or any internal tool built against openai.*: works against Juno with a single base-URL change. No prompt reformatting, no adapter library, no glue code.

New classes (coordinator module):

Modified: InferenceApiServer: constructs OpenAiChatHandler in the constructor (passing the latency callback so HealthReporter still records P99). Routes POST /v1/chat/completions and GET /v1/models[/{modelId}] to the handler. The existing POST /v1/inference and POST /v1/inference/stream endpoints are untouched.

Modified: ConsoleMain (juno-player module): --api-port N flag starts a RequestScheduler + InferenceApiServer alongside the existing REPL in both local and cluster modes. A virtual-thread shutdown hook calls apiServer.stop() on JVM exit. buildLocalModelRegistry() populates a ModelRegistry from the in-process LlamaConfig so GET /v1/models returns the loaded model immediately.

Modified: scripts/run.sh: --api-port N flag wired into both cmd_local() and cmd_cluster(). Environment override: API_PORT.

New file: api/src/main/resources/juno-api.yaml: OpenAPI 3.0.3 spec for the public client-facing API. Documents all request fields with their Juno internal mappings, the SSE chunk event sequence, Juno extension fields (x_juno_priority, x_juno_session_id, x_juno_top_k, x_juno_latency_ms, x_juno_retry_after_ms, x_juno_queue_depth), and all error codes.

New test: OpenAiAdapterTest: unit tests for all four mapping helpers.

Field mapping summary (request):

OpenAI fieldJuno internalNotes
modelmodelIdFirst loaded model if omitted
messages[].role / .contentChatMessageText only; images not supported
temperatureSamplingParams.temperature0.0–2.0; default 0.7
top_pSamplingParams.topP0.0–1.0; default 0.9
max_completion_tokensSamplingParams.maxTokens1–32768; default 200
max_tokensSamplingParams.maxTokensDeprecated alias
frequency_penaltySamplingParams.repetitionPenalty1 + max(0, fp/2)
streamroute selectionfalse → blocking JSON; true → SSE
n:Only 1 is accepted; other values → 400
stop, presence_penalty, logit_bias, user, seed:Silently ignored
x_juno_priorityRequestPriorityHIGH / NORMAL / LOW
x_juno_session_idInferenceRequest.sessionIdEnables KV-cache reuse across turns
x_juno_top_kSamplingParams.topK0 = disabled; default 50

All modules compile. All existing tests pass. OpenAiAdapterTest (4 assertions) passes.

Status

Session 28: Health dashboard: CPU load metric, role-conditional secondary metric, node throughput.

Health dashboard fixes

Fix 1: temperatureCelsius → cpuLoad. /sys/class/thermal is unavailable on EC2 VMs; the Temperature row always showed a dash placeholder. Replaced with process CPU utilisation read from OperatingSystemMXBean.getCpuLoad() (0.0-1.0, available on all JVM platforms, no sysfs). Changes:

Fix 2: Role-conditional secondary metric: coordinator shows Latency P99, nodes show Throughput. Latency P99 was populated by HealthReporter.recordLatency(), which is only called from InferenceApiServer on the coordinator JVM. Worker nodes always showed a dash placeholder. Added a nodeRole field ("coordinator" | "node") to NodeHealth and NodeHealthDto so the dashboard can branch:

Wiring:

Investigation 3: Why 1 of 10 concurrent sessions produced no tokens (no code change). Root cause: gRPC ServerBuilder.forPort(port) with no custom executor defaults to a thread pool bounded by ~2 × CPU count (4 threads on m7i-flex.large). With 9 sessions concurrently running prefill (26 steps × 9 = up to 234 in-flight blocking stubs), all 4 gRPC threads on each node were saturated. The 10th session’s first pipeline.forward() call queued behind them for ~8.5 minutes until prefill of the other 9 finished. The fix is ServerBuilder.forPort(port).executor(Executors.newVirtualThreadPerTaskExecutor()): virtual threads don’t block OS threads on gRPC I/O. JFR evidence: juno.ForwardPass.decode.p95_ms = 3095 ms on node-1 (coordinator node running layers 0–8 plus the REST server) vs 914 ms on node-2; coordinator log confirms 10 tokenizer encodes but only 9 near-simultaneous prefills.

All modules compile. All existing tests pass (NodeHealth, HealthEvaluator, HealthReactor constructors updated to 9-arg signature).

Session 27: GPU lifecycle, multi-device shared contexts, CUDA streams, Llama VRAM fallback, docs.

All modules build and all tests pass. Verified end-to-end with:

Session 26: Phi-3 GPU matmul, FP16 resident weights, CLI and local GPU wiring.

Phi3TransformerHandler GPU path uploads dequantized fused QKV / FFN slices and output projection as DeviceHalfMatrix (IEEE FP16 on device, roughly half the VRAM of DeviceFloatMatrix). Forward uses CudaMatVec.sgemv(DeviceHalfMatrix, x), implemented with cublasHSSgemvStridedBatched: same (CUBLAS_OP_T, m=cols, n=rows, lda=cols) layout contract as the proven cublasSgemv_v2 path for row-major A. Host float[] activations are converted to FP16 for the per-call device x buffer; accumulation stays FP32. Earlier cublasSgemmEx / cublasGemmEx mixed-dtype attempts returned NOT_SUPPORTED / INVALID_VALUE on common stacks; the HSS strided-batched GEMV avoids that.

Session 26: Native LoRA merge (juno merge).

LoraMerge (new, node module) writes a new GGUF file from a base model and a .lora checkpoint without re-quantising the patched tensors. The 44 LoRA-adapted projection weights (wq/wv on every layer) are stored as F32; all other tensors are copied verbatim in their original quantised encoding. F32 is required because the LoRA delta (~6×10⁻⁴) is smaller than Q4_K quantisation noise (~3×10⁻³): re-quantising would silently erase the training. Verified: merged TinyLlama recalls /train-qa facts (name “Dima”) correctly under ./juno local with no .lora sidecar.

GgufReader gains five new public methods needed by the GGUF writer: ggufFileOffset(), metadataSectionEnd(), tensorOrder(), tensorNelems(name), and keeps the existing tensorAbsoluteOffset / tensorType / tensorDims. Internal storage changed from HashMap to LinkedHashMap so tensorOrder() is stable. A List<String> tensorOrder field is added to preserve insertion order.

LoraMergeMain (juno-player module): CLI entry point for juno merge. Reads --model-path, --lora-path, --output, --heap. Derives <model>.lora and <model>-merged.gguf as defaults.

run.sh gains cmd_merge() and the merge) dispatch case.

ConsoleMain /merge-hint REPL command updated: now prints the actual ./juno merge invocation instead of the old “contributions welcome” message.

Three bugs fixed during development of LoraMerge:

Session 25: Code quality: dead code removed, test helpers moved to test scope, docs fully updated.

CyclicForwardPassHandler moved from node/src/main to node/src/test. It is a deterministic stub with no business value without a model; it belongs exclusively in the test compilation unit. EmbeddedNodeServer no longer imports it: the three call sites (pre-load placeholder, model-load-failure fallback, no-model stub mode) are now served by a new private StubForwardPassHandler inner class that returns zero-filled arrays of the correct shape with no test machinery. node/pom.xml gains a maven-jar-plugin test-jar execution so other modules can still import CyclicForwardPassHandler; coordinator/pom.xml and juno-master/pom.xml declare the node:tests classifier dependency.

VRAM / OOM: GPU buffer allocation is wrapped; on failure (including cudaMalloc OOM), partial device buffers are closed and the handler falls back to CPU quantised LlamaTransformerHandler.matVec-style matmul for those projections.

ConsoleMain: missing break after --cpu fixed: parsing no longer fell through into --lora, which incorrectly set loraMode when forcing CPU inference.

ConsoleMain.runLocalRepl: one shared GpuContext + CudaMatVec instance for every in-process shard load (avoids redundant cuBLAS contexts and matches production “one GPU per JVM” usage).

Tests: CudaMatVecBackendTest.device_half_matrix_sgemv_matches_host_path (512×512) anchors FP16 resident correctness vs LlamaTransformerHandler.matVec.

JFR: MatVecEvent.backend cuda-resident-fp16 labels the Phi FP16 device path. (As of session 27, Llama GPU resident weights also use cuda-resident-fp16; cuda-resident remains for DeviceFloatMatrix / tests.)

Session 26: LoRA inference overlay (--lora-play), Q&A training mode (/train-qa), diagnostic tracing, and AWS deploy hardening.

--lora-play PATH: apply trained adapters at inference in any mode

Pre-trained .lora checkpoint files can now be applied read-only at inference time without entering the lora REPL. Three modes are supported:

local mode:

./juno local --model-path model.gguf --lora-play /path/to/model.lora

ConsoleMain.runLocalRepl() calls LoraAdapterSet.load(Path.of(loraPlayPath)) before building the shard handlers and passes the result into ForwardPassHandlerLoader.load(..., playAdapters).

cluster mode (forked JVMs):

./juno --model-path model.gguf --lora-play /path/to/model.lora

ClusterHarness.withLoraPlay(path) injects -Djuno.lora.play.path=PATH into every forked node JVM command. EmbeddedNodeServer.NodeServiceImpl reads this property at construction and loads adapters inside loadShard() before the ForwardPassHandlerLoader call.

AWS deployed cluster:

./launcher.sh juno-deploy.sh setup --lora-play /absolute/path/to/model.lora

See AWS section below.

ForwardPassHandlerLoader: new LoRA overload

// New canonical overload: all others delegate to this
public static ForwardPassHandler load(
    Path modelPath, ShardContext context, MatVec backend,
    LoraAdapterSet adapters) throws IOException

When adapters != null, the loader routes to LoraTrainableHandler (inference-only, no optimizer attached) instead of the architecture-specific handler. When adapters == null the existing phi3 / llama dispatch is unchanged. selectBackend() promoted from package-private to public so juno-player-module callers can reuse it.

ClusterHarness: withLoraPlay() fluent method

harness.withLoraPlay("/path/to/model.lora");

Stores the path and injects -Djuno.lora.play.path=PATH into the launchNode() JVM command, after the JFR flags. Without this, forked node JVMs start with loraPlayPath=null and run the base model regardless of what the coordinator is told.

/train-qa: conversational Q&A training

New REPL command in lora mode for training single-fact associations:

you > /train-qa What is my name? A: Dima
  Question: What is my name?
  Answer  : Dima

  Formatted as 4 Q&A pairs  ·  model type: tinyllama
  Training  rank=8 · lr=1.0E-4 · 40 steps ...
  ✔ done  loss=▼ 1.53 (−0.83)

The command auto-generates 4 phrasings of the question (exact, Can you tell me: ..., Please answer: ..., plus one repeat) to improve generalization. The chat template appropriate for the model type (detected from the model path) is applied to each pair. Flags --lora-steps-qa N and --lora-early-stop F control training depth.

Separator syntax: Q: <question> A: <answer> or <question> A: <answer>.

Diagnostic tracing (--verbose)

All tracing is prefixed [TRACE] for easy grep. Added to:

LocationWhat is shown
LoRA REPL startupModel type (chat template key), model path, all LoRA hyperparameters
/train-qaExact formatted training text with ↵ for newlines, token count, token IDs (verbose only)
Per training step (verbose)step=N loss=F chunk=M/T ms=D
Cluster inference (verbose)Chat template key used for each inference request
juno-deploy.sh bootstrapPer-node params baked into user-data script
juno-deploy.sh SCPLocal source, remote target, per-node node.env patch
juno-deploy.sh coordinator envFull cluster-nodes.env contents echoed after write

AWS deploy hardening (juno-deploy.sh)

Multiple bugs fixed during end-to-end AWS validation:

Double base64 encoding (cloud-init rejected user-data). --user-data was passed as a pre-base64-encoded string. AWS CLI base64-encodes it again; cloud-init received double-encoded garbage and logged Unhandled non-multipart (text/x-not-multipart) userdata. Fix: write user-data to a temp file and pass file:///tmp/juno-userdata-*.sh: the CLI reads it raw and does single encoding. The [TRACE] size line now also prints first-line: #!/bin/bash so shebang presence is visible in the setup log.

TRACE logs contaminating user-data. _build_node_userdata is called as USER_DATA=$(_build_node_userdata ...) which captures all stdout. The four log / [TRACE] calls inside the function were writing to stdout, prepending ANSI escape codes before #!/bin/bash. Cloud-init saw no shebang on line 1 and skipped execution. Fix: all log calls inside _build_node_userdata now redirect to stderr with >&2.

Relative --lora-play path not resolved. When called from scripts/aws/, a path like ../models/model.lora resolves to scripts/models/model.lora (which doesn’t exist). _scp_lora_to_nodes hit the [[ ! -f ]] guard and returned silently, leaving node.env with empty JUNO_LORA_PLAY_PATH. Fix: --lora-play is resolved to absolute path at parse time via realpath. setup() also validates the file exists before any AWS spend.

Race condition: coordinator started before node restart completed. _scp_lora_to_nodes previously used systemctl restart --no-block and polled systemctl is-active to detect readiness. The old instance remained active during shutdown so the poll returned immediately, _write_cluster_env_and_start_coordinator ran, and the coordinator sent loadShard to the old (no-LoRA) instance. The restarted instance came up 19 minutes later, too late. Fix: synchronous stop → patch → start per node: systemctl stop juno-node (synchronous, waits for JVM exit), sed patch of node.env, systemctl start juno-node (synchronous, returns once gRPC port is bound, ~2s). Coordinator only starts after all three nodes have confirmed active status with correct env.

Local relative path baked verbatim into cluster-nodes.env. Even when SCP succeeded, the coordinator received JUNO_LORA_PLAY_PATH=../models/... (the pre-realpath value), causing model load failed: ../models/... on the nodes. Fix: _scp_lora_to_nodes updates the global LORA_PLAY_PATH to the remote absolute path (/opt/juno/models/<basename>) before returning, so _write_cluster_env_and_start_coordinator writes the correct value.

_write_cluster_env_and_start_coordinator missing closing brace. The } was accidentally elided, causing scan_regions() to be parsed as part of the function body.

End-to-end verification:

you> what is my name?
bot> Dima

Confirmed working on 3 × m7i-flex.large AWS cluster (eu-north-1) with TinyLlama-1.1B-Chat-v1.0.Q4_K_M and a .lora adapter trained locally, SCPed and deployed via juno-deploy.sh setup --lora-play.

Session 34: Windows launcher fixed: run.bat/juno.bat fully functional; docs updated with Windows examples. (this session)

Session 33: Model support documentation: Phi-3 supported; Gemma, Qwen 2 / Qwen3 / Qwen3.5 under development. (unchanged)

Session 24: Configurable activation byte order (--byteOrder BE|LE). (unchanged)

Session 22: Q2_K and Q3_K quantization support. (unchanged)

Session 21: Two new deployment fat-jar modules and a unified AWS script. (unchanged)

Session 20: GPU inference actually wired end-to-end. (unchanged)

Session 19: metrics module, Meta-Llama 3 tokenizer fix, AWS infrastructure scripts. (unchanged)

Session 18: GPT-2 BPE tokenizer, JFR instrumentation fixes. (unchanged)

Session 17: AWS infrastructure scripts. (unchanged)

Session 14: LoRA fine-tuning + JFR profiling. (unchanged)

<- 11.1 Release Notes  |  Table of Contents

11. Releases
11.1. Release Notes