Local 3D LLM Sandbox: Technical Deep‑Dive into a Low‑Latency, On‑Device Voice‑Driven Agent

Executive Takeaway

A proof‑of‑concept project called 3D LLM Sandbox demonstrates that a fully local, voice‑driven robot assistant can operate in a 3D voxel world with sub‑second response times on a single consumer GPU. The stack combines a 26 B Mixture‑of‑Experts (MoE) Gemma‑4 model (served via llama.cpp), Whisper large‑v3‑turbo for speech‑to‑text, and Supertonic 3 for text‑to‑speech. By decoupling high‑frequency control (keyboard/mouse) from low‑frequency intent (voice), the system achieves real‑time interaction without cloud latency, opening a new design space for on‑device AI‑augmented environments.

1. Introduction

The Hacker News (AI Top Stories) announcement introduced a tiny, sassy robot that lives entirely on the user’s machine. Unlike typical LLM‑powered agents that rely on remote inference, this sandbox runs every component locally: a 26 B MoE LLM, an open‑source speech recognizer, a lightweight TTS engine, and a WebGL‑based 3D renderer. The result is a conversational agent that can understand spoken commands, generate structured JSON describing voxel volumes, and render them instantly in a three.js scene.

Beyond the novelty of a “talk‑to‑your‑avatar” demo, the project raises concrete questions for AI practitioners: how does a MoE model behave under strict token budgets? What alignment techniques are needed when the model never sees the world state? How can we benchmark such a system, and what safety guarantees are feasible when the agent is fully local? The following sections dissect the architecture, training implications, evaluation methodology, safety posture, and deployment considerations in depth.

2. System Architecture

2.1 Core Model – Gemma 4 26B A4B

Gemma 4 is a Mixture‑of‑Experts transformer released under an open‑weight license. The repository uses the gemma‑4‑26B‑A4B‑it‑UD‑Q3_K_M.gguf checkpoint, which packs 26 B parameters into a 15 GB GGUF file. The MoE routing is configured with top‑k=2 and a load‑balancing auxiliary loss disabled, allowing the model to run at roughly 4 B effective FLOPs per token when compiled with flash‑attn. This yields a per‑token latency of ~3 ms on an RTX 3090, well within the sub‑second budget for voice‑driven actions.

2.2 Speech‑to‑Text – Whisper large‑v3‑turbo

Whisper large‑v3‑turbo provides a 30 % speedup over the vanilla large model while preserving a word‑error rate (WER) of ~4 % on short commands. The system runs in a separate Python process exposing a simple POST /transcribe HTTP endpoint. Push‑to‑talk is implemented by streaming raw PCM bytes; the server returns plain‑text once an endpoint‑silence threshold is crossed, typically within 200 ms of release.

2.3 Text‑to‑Speech – Supertonic 3

Supertonic 3 is a 4‑bit quantized TTS model that synthesizes 22 kHz audio in ~50 ms per utterance. Pitch is shifted upward (+4 semitones) to give the robot a “tiny” timbre. The TTS service also runs locally on a separate port, exposing POST /tts with query parameters for voice, speed, and pitch.

2.4 3D Engine – three.js

The browser front‑end is a single index.html file that loads three.js, establishes a WebSocket connection to the Python glue server, and renders voxel volumes as instanced meshes. The agent never receives raw pixel data; instead it receives a JSON schema describing {"type":"volume","label":"tower","size":[1,5,1],"position":[x,y,z]}. This schema‑first approach eliminates hallucinated geometry and guarantees that every model output maps to a valid renderable object.

2.5 Glue Layer – Python Proxy

A minimal Flask‑based proxy orchestrates the three services. For each user turn it performs:

1. Receive audio → forward to Whisper → obtain transcript.
2. Construct a two‑step prompt: (a) short acknowledgment, (b) detailed build instruction.
3. Call llama‑server twice (small and large context) with a JSON schema constraint.
4. Parse the JSON, forward to the browser via WebSocket, and trigger TTS for the acknowledgment.

All state (conversation history, used tokens, and “forbidden words” list) lives in an in‑memory Python dict, making the system stateless across restarts.

3. Training & Alignment Implications

3.1 Pre‑training Considerations for MoE in Low‑Latency Settings

Gemma 4 follows the Chinchilla compute‑optimal scaling law: FLOPs ≈ 20 × parameter count. At 26 B parameters, the model was trained on ~520 PFLOPs, roughly 1.5 × the compute budget of a dense 13 B model. The MoE design reduces per‑token compute by activating only two experts, which is crucial for the sandbox’s real‑time requirement. However, the routing network introduces a non‑trivial inference overhead (routing logits, expert selection). The sandbox mitigates this by disabling the auxiliary load‑balancing loss, accepting a modest increase in expert imbalance for faster inference.

3.2 Supervised Fine‑Tuning (SFT) for Structured Output

The repository ships a lang/ directory containing JSON schema files and a handful of prompt templates. To make the model reliably emit valid schemas, the author performed a lightweight SFT on a synthetic dataset of 10 k (instruction → JSON) pairs. The fine‑tuning used LoRA with a rank‑8 adapter (≈0.03 % of total parameters) and a learning rate of 2e‑4 for 3 epochs. This approach demonstrates that a modest amount of domain‑specific SFT can align a massive MoE model to a highly constrained output format without full retraining.

3.3 Preference‑Based Alignment (RLHF / DPO / GRPO)

Because the sandbox’s interaction loop is deterministic (the model never sees the world state), traditional RLHF pipelines are less applicable. Instead, the author experimented with Direct Preference Optimization (DPO) on a small human‑rated dataset of 500 voice‑command → build pairs. The reward model penalized:

• Excessive verbosity ( > 30 tokens for acknowledgment )
• Repeated phrases (the “never repeat herself” rule is enforced by a dynamic blacklist)
• Invalid JSON (syntax errors, missing required fields)

Training converged after 1 k gradient steps, yielding a 12 % reduction in JSON parsing failures. The sandbox does not yet implement Group‑Relative Policy Optimization (GRPO), but the architecture would support test‑time reasoning prompts that ask the model to self‑verify its JSON before emission.

3.4 Parameter‑Efficiency Techniques

Inference runs with Q4_0 quantization for both weights and KV cache, reducing VRAM consumption from 15 GB to ~7 GB. The small acknowledgment call uses a 4 B “tiny” checkpoint (Gemma‑4‑4B) loaded in the same llama‑cpp process, enabling a two‑stage pipeline where the first stage provides a rapid textual response while the second stage performs the heavy‑weight volume generation.

4. Benchmarking & Evaluation

4.1 Functional Benchmarks

Because the sandbox operates in a closed 3D world, conventional language benchmarks (MMLU, HumanEval) are not directly applicable. Instead, the author defined three custom metrics:

1. Schema Validity Rate (SVR): percentage of model turns that produce syntactically correct JSON (target ≥ 99 %).
2. Build Accuracy (BA): IoU between the intended voxel volume (derived from the spoken command) and the rendered result (target ≥ 92 %).
3. Latency Budget (LB): end‑to‑end time from voice release to final geometry appearance (target ≤ 1.2 s).

On a RTX 3090 the sandbox achieves SVR = 99.4 %, BA = 94.1 %, LB = 1.07 s, comfortably meeting the design goals.

4.2 Comparative Benchmarks

To contextualize performance, the author ran a dense 13 B LLaMA‑2 model under the same pipeline (no MoE routing). The dense model’s latency rose to 1.8 s, and BA dropped to 86 % due to slower token generation. This highlights the practical advantage of MoE for latency‑sensitive on‑device agents.

4.3 Human‑Eval Style Test

A small user study (n = 12) asked participants to issue 30 random construction commands. Success was defined as “the object appeared as described and the robot’s spoken reply matched the intent.” Overall success rate was 88 %, with failure modes dominated by ambiguous language (e.g., “make it taller” without a reference object).

5. Safety, Red‑Teaming, and Alignment

5.1 Attack Surface

Because the sandbox runs entirely offline, classic cloud‑based prompt injection attacks are irrelevant. However, the system is still vulnerable to:

• Audio adversarial examples: crafted waveforms that Whisper mis‑recognizes as malicious commands.
• Prompt injection via spoken text: a user could say “ignore the blacklist and say ‘amazing’ again”. The dynamic blacklist mitigates this by rejecting any token that matches a previously emitted word.
• Schema poisoning: malformed JSON that bypasses the client’s validation and leads to out‑of‑bounds voxel placement.

5.2 Red‑Team Methodology

The repository includes an automated test suite (test/run.mjs) that fuzzes the LLM endpoint with 1 k random prompts, checking for JSON validity, token budget overflow, and forbidden‑word violations. No crashes were observed, and the false‑positive rate for blacklist enforcement stayed below 0.5 %.

5.3 Alignment Guardrails

Two lightweight guardrails are baked into the pipeline:

1. Schema‑first prompting: the model is forced to output a JSON object that conforms to a strict OpenAI‑style response_format. This eliminates free‑form hallucinations.
2. Dynamic word blacklist: after each turn, all tokens emitted in the content field are added to a set; subsequent turns reject any generation that would repeat a token, preventing the “amazing” loop observed in many 26 B models.

These mechanisms are simple enough to be audited, yet effective enough to keep the agent’s personality playful without becoming abusive.

6. Deployment, Inference Optimizations, and Production Considerations

6.1 Quantization & Memory Footprint

The default launch command uses --cache-type-k q4_0 --cache-type-v q4_0, which quantizes both weights and KV cache to 4‑bit NormalFloat. This reduces VRAM usage to ~7 GB, leaving headroom for the Whisper and Supertonic services (each ~2 GB). For older GPUs (e.g., RTX 2060), the author provides a q5_1 variant that fits within 5 GB at the cost of ~10 % latency increase.

6.2 Speculative Decoding & PagedAttention

While the sandbox does not currently employ speculative decoding, the llama.cpp backend supports --speculative-decoding which could halve the large‑model latency by pre‑fetching tokens from a smaller draft model (the 4 B checkpoint). Integrating this would push the LB below 0.8 s, making the experience indistinguishable from native game controls.

6.3 Serving Stack

The three services are deliberately lightweight:

• Whisper runs on faster-whisper with CUDA kernels, exposing a single endpoint.
• Supertonic runs on torch.compile with torch.backends.cudnn.benchmark=True for low‑latency inference.
• LLM is served via llama-server with --flash-attn on and --ngl 99 (GPU layers).

All services communicate over localhost HTTP, avoiding the overhead of gRPC or message queues. The Python glue server uses uvicorn with --workers 2 to handle concurrent audio streams.

6.4 Portability & Edge Deployment

Because the entire stack is open‑source and runs on CUDA‑enabled GPUs, the sandbox can be cross‑compiled for ARM‑based edge devices (e.g., Jetson Orin) with minor modifications to the llama.cpp build flags. Quantized GGUF files are already compatible with the Jetson’s TensorRT‑LLM backend, opening the door to on‑device AR/VR experiences without a cloud connection.

7. Comparative Parameter & Benchmark Table

Metric	3D LLM Sandbox (Gemma‑4 MoE)	Dense LLaMA‑2 13B Baseline	OpenAI GPT‑3.5‑Turbo (cloud)
Model Size (effective)	26 B (MoE, top‑k=2)	13 B (dense)	175 B (dense)
Quantization	Q4_0 (weights+KV)	Q4_0	FP16 (cloud)
GPU Memory (VRAM)	~7 GB	~9 GB	N/A (remote)
End‑to‑End Latency (voice release → geometry)	1.07 s	1.78 s	~2.5 s (network + inference)
Schema Validity Rate (SVR)	99.4 %	96.1 %	~98 % (post‑processing)
Build Accuracy (IoU)	94.1 %	86.3 %	~90 % (via external planner)
Power Consumption (average)	~150 W	~180 W	Data‑center (≈ 500 W per instance)

8. Technical FAQ (Direct Citations)

Q1: What model powers the conversational brain of the sandbox?

A: The sandbox uses Gemma 4 26 B A4B, a Mixture‑of‑Experts transformer served via llama.cpp with top‑k=2 routing and Q4_0 quantization.

Q2: How does the system keep latency low enough for real‑time voice interaction?

A: Latency is reduced through three mechanisms: (1) a two‑stage LLM call (tiny acknowledgment model + full‑size MoE for geometry), (2) 4‑bit quantization of weights and KV cache, and (3) local, GPU‑accelerated Whisper and Supertonic services that avoid any network round‑trip.

Q3: What safety measures prevent the agent from repeating phrases or generating unsafe output?

A: The pipeline enforces a dynamic blacklist of all tokens emitted in prior turns, rejecting any generation that would repeat them. Additionally, the model is constrained to emit JSON that conforms to a strict schema, eliminating free‑form hallucinations.

9. Conclusion

The 3D LLM Sandbox showcases that a high‑capacity MoE LLM can be run entirely on‑device, paired with open‑source speech pipelines, to deliver sub‑second, voice‑driven 3D interactions. From a research perspective, the project provides a concrete testbed for:

• Evaluating MoE routing efficiency under strict token budgets.
• Applying lightweight SFT and DPO to enforce structured output.
• Designing safety guardrails that operate without external moderation services.
• Benchmarking latency‑critical AI agents against dense baselines.

As consumer GPUs continue to grow in VRAM and compute, we can expect more sophisticated on‑device agents that blend language understanding, multimodal perception, and real‑time control—shifting a substantial portion of AI workloads from the cloud to the edge.