Nanite for Experts

Abstract

Mixture-of-experts (MoE) language models activate only a fraction of their parameters for each token, but conventional local inference engines still treat most expert weights as if they must remain continuously resident in GPU or system memory. This creates a mismatch between the model's sparse computation graph and the computer's physical memory hierarchy. It is especially costly on consumer PCs, where a capable model may fit across RAM and storage but not in VRAM.

We present Nanite for Experts, an experimental inference architecture developed for Lemonade and implemented in a private llama.cpp fork. The system treats NVMe storage, pageable RAM, pinned host memory, and GPU memory as a coordinated hierarchy. It observes authoritative native router decisions, learns token-aligned expert transitions, predicts a bounded future working set, unions expert demand across multi-token prediction (MTP) batches, and stages virtual 256 KiB expert tiles through triple-buffered CUDA streams. A persistent device page table lets the runtime execute GPU-resident expert fragments while retaining the original mapped tensors as the correctness source and CPU fallback.

The central result is not that aggressive expert caching automatically makes local MoE inference fast. It does not. On an RTX 3080 test system, several high-hit-rate CUDA experiments were slower than a tuned layer-placement baseline because PCIe traffic, synchronization, kernel launch overhead, and fragmented matrix work dominated the saved computation. The most reliable improvements came from correct layer placement, native MTP, exact RAM prefetch, expert-major batch union, conservative admission, and a stable 16-expert working set. Exact RAM prefetch improved a paired generation test from 12.75 to 14.66 tokens/s, while MTP depth four improved an all-CPU-expert test from 26.17 to 35.60 tokens/s. The best tested layer split reached 52.03 tokens/s. By contrast, indiscriminate CUDA expert staging produced transfer cliffs and visible application lag.

These results establish a practical architecture, a set of negative findings, and a narrower path toward useful virtualized expert execution: optimize the exact routed path first, bound transfer demand, preserve a stable working set, amortize launch overhead in a persistent expert-major operator, and introduce lossy expert level-of-detail formats only behind measurable quality gates and an exact fallback.

1. The local MoE residency problem

Sparse MoE models separate total model capacity from the amount of computation used for one token. The test model, Qwen3.6-35B-A3B, has approximately 35 billion total parameters but activates about 3 billion parameters per token. Its official model card describes 40 layers, 256 routed experts, eight routed experts plus one shared expert per token, native MTP, and a 262,144-token context window.[1]

That sparsity does not make the inactive weights disappear. A conventional runtime must still choose where the full tensor set lives:

• VRAM is fast but scarce.
• System RAM has greater capacity but requires CPU execution or PCIe transfer.
• NVMe has much greater capacity but much higher access latency.
• The operating-system page cache can hide some storage latency, but it is not a deterministic GPU cache.

The problem is therefore not merely to load a large model. It is to make the weights required by the next routed computation available before the computation stalls, without moving so much speculative data that transfer work overwhelms useful inference.

The official llama.cpp CLI exposes important controls such as GPU layer placement, CPU placement of MoE tensors, and draft MTP models.[2] Nanite for Experts builds on those native controls. It does not replace the router or invent an approximate routing decision. Instead, it adds a speculative data plane around the authoritative execution path.

2. Relationship to prior systems

This work was initially motivated by Colibri's core observation: VRAM, RAM, and NVMe can be managed as a single expert-weight hierarchy instead of as unrelated storage tiers. Colibri describes learned hot-expert caching, expert union across a batch, asynchronous I/O, and one-layer router lookahead, and reports 71.6% one-layer expert predictability in its target workload.[3] Nanite for Experts is an independent implementation in a Lemonade-specific llama.cpp fork; it does not incorporate Colibri source code.

The broader research literature supports the same direction while exposing different tradeoffs:

• MoE-Infinity uses activation-aware expert caching and prefetching for batch-one inference, reporting substantial per-token latency improvements over several offloading systems.[4]
• SiDA-MoE uses a lightweight predictor and hash-based expert loading across CPU and GPU memory, reporting up to 3.93x throughput, 72% latency reduction, and 80% GPU-memory reduction with a reported quality tradeoff as low as 1% in its evaluated settings.[5]
• MoE-Lightning models offloaded MoE execution as an overlapped CPU, GPU, and I/O pipeline and demonstrates the importance of paged weights, pinned memory, and communication-compute overlap.[6]
• FloE combines expert prediction with weight compression to reduce PCIe traffic, accepting measured model-quality degradation in exchange for much higher effective capacity and throughput.[7]

Headline speedups from these systems are not directly comparable to our results. Hardware, models, quantization, request batching, accuracy criteria, and baselines differ. Our current evaluation focuses on a single interactive request on one consumer PC and places a stronger emphasis on retaining the native model as the source of truth.

The name Nanite for Experts is an analogy to Unreal Engine's Nanite virtualized geometry, which streams fine-grained data and automatically manages working detail rather than requiring every source asset to be fully resident.[8] No Epic technology or code is used. The analogous idea is to virtualize expert weights at a finer granularity and materialize only the working set needed near the current execution frontier.

3. Design goals

The architecture follows six rules.

3.1 Authoritative routing

Native MUL_MAT_ID routing decisions determine which experts execute. Predictions affect only placement and prefetch. A wrong prediction may lose time, but it must not select a different expert.

3.2 Fail-open latency, fail-closed correctness

The original memory-mapped model tensor remains available throughout inference. If a prefetched page, GPU slot, generation identifier, deadline, or checksum is invalid, the runtime falls back to the native tensor path.

3.3 Bounded speculation

Every speculative action consumes explicit budgets for bytes, queue entries, GPU slots, and time. Exact demand outranks predicted demand. Newly predicted experts are probationary until native routing validates their usefulness.

3.4 Overlap rather than blocking

Storage reads, RAM staging, host-to-device transfers, CPU expert work, and GPU expert work should overlap where dependencies allow. CUDA asynchronous transfers require pinned host memory, and useful copy-compute overlap requires separate non-default streams on capable devices.[9]

3.5 Stable working sets

Maximizing instantaneous cache hits is not necessarily optimal. On a PCIe-constrained system, a smaller persistent set can outperform a larger rolling cache if it avoids constant replacement.

3.6 Measure fidelity before compression

An expert level-of-detail representation may reduce bandwidth, but it changes model arithmetic. Compression belongs behind perplexity, task-quality, and long-generation gates, with an exact residual or original-weight fallback.

4. System architecture

Keeping these planes separate was a major correctness improvement. Earlier prototypes accidentally allowed speculative route learning to trigger unrelated transfers. Separating prediction from admission reduced one test's unnecessary prompt transfer from approximately 1.42 GB to 16.3 MB.

5. Implementation

5.1 Correct topology and layer placement

The first optimization was measurement, not caching. Lemonade's original display implied 41 routed layers and 92% expert placement. Native traces established that the evaluated model has 40 routed layers. On the test machine, the best measured configuration offloaded 13 routed layers to the GPU and kept 27 on the CPU, leaving one layer of VRAM reserve. Attempting to place more layers produced a severe throughput cliff.

This result matters because any adaptive cache is downstream of base placement. A cache cannot compensate for an incorrect memory budget.

5.2 Exact RAM and page-cache prefetch

On Windows, the prototype uses PrefetchVirtualMemory to issue a performance hint for discontiguous mapped expert ranges. Microsoft documents this API as a way to bring address ranges into physical memory efficiently, while warning that it is a strong hint rather than a guarantee and can cause memory pressure if overused.[10]

The mapped tensors remain the execution source; prefetch only changes their expected residency. This path produced byte-identical output in the paired test and is the safest adaptive optimization currently enabled.

5.3 Native MTP and expert-major batch union

MTP predicts multiple future token candidates. DeepSeek-V3 identifies MTP as useful not only for training signal but also for speculative decoding acceleration.[11] In an MoE runtime, a multi-token candidate window also exposes a larger expert graph before the next execution step.

Rather than transfer an expert separately for every token position, the prototype unions expert identifiers across the MTP window and schedules each unique expert once. In a measured five-position window corresponding to MTP depth four, 5,600 assignments collapsed to 3,670 unique expert uses, eliminating 1,930 redundant movements, or 34.5% of assignment-level transfer demand.

5.4 Virtual expert tiles

Large expert tensors were divided into 256 KiB virtual tiles. Three CUDA streams rotate through copy, compute, and reuse stages so that one tile can transfer while another executes. Tiles are addressed through persistent GPU page-table entries with sentinel values for absent or stale generations.

The tile size is a scheduling unit, not a lossy representation. Reassembling exact tiles produced the same output hash in the validation test. The tiled transfer experiment improved its paired throughput from 11.06 to 12.66 tokens/s, but moved 1.61 GB across 7,504 tiles. That traffic volume showed why tile-level correctness alone does not guarantee system-level speed.

5.5 Simultaneous CPU-miss and GPU-hit execution

The hybrid executor partitions exact routed work into resident GPU hits and CPU misses. Both branches use persistent workspaces, then merge results into their original output positions. A cost gate requires enough GPU hits and sufficient routed coverage before hybrid execution is admitted; otherwise the entire operation follows the mature native path.

This implementation exposed a crucial limitation. Small fragmented matrix operations do not automatically become efficient merely because their weights reside in VRAM. Host launch and synchronization costs become first-order. CUDA Graphs can amortize repeated host launch overhead, particularly when individual kernels are short.[12] A persistent primary graph raised the second-request scheduler result from 0.55 to 27.52 tokens/s, but the strict long run still trailed the layer-placement baseline.

5.6 Static and rolling caches

An unbounded or rapidly rolling expert cache generated high hit counts but also high transfer volume. The production-facing default therefore uses a learned, static 16-expert working set per relevant layer window. Rolling replacement remains an explicit experimental mode.

The scheduler admits high-confidence early candidates to GPU slots and a wider set to RAM, while enforcing a default 384 MiB speculative byte cap. Candidates are ordered by exactness, deadline, tier, and confidence. The application exposes the mode as CUDA Nanite cache + CPU fallback when the experimental runtime is active.

6. Experimental method

The current results come from iterative engineering benchmarks on a Windows workstation with an NVIDIA RTX 3080 (SM 8.6) and a quantized Qwen3.6-35B-A3B model of roughly 23 GB. Unless noted otherwise, paired runs used deterministic sampling settings, fixed seeds, identical prompts, and output hashes to detect arithmetic or routing differences.

7. Results

7.1 Reliable improvements

MTP depth four was the best of the tested depths: depth five reached 34.48 tokens/s and depth six reached 30.24. For the MTP-four test, eight CPU threads reached 40.23 tokens/s, compared with 38.45 at 12 threads and 34.29 at 16 threads. More host threads were not automatically faster.

7.2 Placement cliffs

The discontinuity is more important than the absolute values: nominally placing more work on the GPU can reduce throughput sharply when it removes memory reserve or triggers a less favorable execution path.

7.3 Adaptive CUDA scheduler results

The fused prototype reduced logical jobs from 108 to 54, yet throughput collapsed. Reducing a scheduler's job count is not useful when the resulting kernels have poor shapes, lose optimized quantized paths, or add synchronization.

7.4 Working-set policy

The static policy cut transfer volume by approximately 98.2% relative to rolling width 16 and more than doubled adaptive throughput, despite abandoning continuous replacement. A width sweep reinforced the existence of a working-set optimum:

Width 16 was selected as the conservative application default because it retained exactness in the longer paired test and avoided the transfer cliff. Width 20 remains a tuning candidate, not a proven universal optimum.

8. The governing bottleneck

For a routed layer with overlapped CPU misses and GPU hits, a useful lower-bound model is:

The equation explains several initially counterintuitive results:

• A high cache-hit count can coexist with poor throughput if filling that cache moves too many bytes.
• Fine tiles reduce wasted bytes per miss, but increase scheduling, launch, and metadata pressure.
• GPU residency helps only when the GPU kernel is large and efficient enough to repay transfer and launch costs.
• CPU and GPU concurrency helps only when neither branch waits at frequent synchronization barriers.
• Predicting more experts improves coverage but can worsen the PCIe term.

The objective is therefore not maximum prediction accuracy or maximum hit rate. It is minimum critical-path time under a transfer budget.

One practical admission rule used by the application is:

The terms must be calibrated on the user's actual hardware. A desktop with a faster PCIe link, more VRAM, or higher CPU memory bandwidth should make different decisions from the current RTX 3080 test system.

9. Correctness and fidelity

The prototype distinguishes exactness of routing, exactness of weights, and bitwise equality of arithmetic:

• Routing exactness: native route IDs are authoritative in all supported modes.
• Weight exactness: RAM prefetch and virtual tiling use the original model bytes.
• Arithmetic exactness: CPU and CUDA quantized kernels may accumulate in different orders. Even when both consume identical weights, mixed execution can change logits and eventually token selection.

This distinction surfaced in testing. The rolling width-16 path recorded many correct GPU hits but diverged in a longer generation because mixed CPU/CUDA arithmetic altered the sequence. A short matching hash is therefore a regression gate, not proof of universal semantic equivalence.

Request features and traces remain local. The application stores hashed prompt features rather than prompt text and caps learned traces at 192 per model. This is a privacy boundary for the adaptive scheduler; it does not imply that every optional Lemonade feature is offline. Connectors, licensing, updates, mail, MCP tools, and web research have their own explicit network behavior and are described separately in the product's network disclosure.

10. Why the first expert LOD failed

The first level-of-detail experiment represented an expert tensor as a shared mean and low-rank basis:

For a sample block-zero down projection at rank 32, the representation suggested a 3.18x storage reduction. It retained only 62.5% of the measured energy and produced 60.7% relative reconstruction error. That is not acceptable evidence for deployment, so the format was rejected.

This negative result narrows the next design:

• fit per tile or per structurally related expert group rather than one broad tensor basis;
• quantize coefficients and residuals using the target kernel's actual format;
• retain an exact residual or full-weight escape path;
• evaluate perplexity, routed-token logits, long generation stability, and application tasks;
• admit approximate tiles only when their saved transfer time exceeds decode and correction cost.

FloE demonstrates that compressed expert offloading can be fast, but its reported gains include a measured quality tradeoff.[7] Lemonade's preferred product mode remains fidelity-first, so an LOD format cannot be considered complete based on compression ratio alone.

11. What we learned

11.1 Placement is the first optimization

Correcting routed-layer counts and preserving VRAM headroom delivered a stronger result than the early adaptive schedulers. Hardware-aware placement should precede fine-grained paging.

11.2 Lookahead is useful only with admission control

Router prediction, request graphs, and MTP expose future demand, but every prediction has a byte cost. Useful lookahead answers which already-affordable transfer should happen first, not simply which expert may be used.

11.3 Expert union is a real algorithmic win

The MTP union removed 34.5% of assignment-level redundancy before touching CUDA. This is analogous to vectorization: reorganize the computation around common data, then optimize the kernel.

11.4 Stable residency can beat reactive caching

The rolling cache achieved more hits but paid for them repeatedly. A learned static set captured durable locality and left the remaining work to an optimized fallback.

11.5 Fine granularity shifts the bottleneck

Virtual tiles reduce overfetch, but enough tiny operations turn a bandwidth problem into a launch and synchronization problem. The ultimate execution unit should be expert-major and persistent, not thousands of independently orchestrated fragments.

11.6 Fidelity has multiple levels

Exact expert IDs and exact weight bytes are necessary but not sufficient for identical generation. Mixed-kernel arithmetic needs explicit long-horizon validation.

12. Next research milestones

The evidence supports the following order of work:

1. Build a persistent expert-major CUDA operator. Consume routed positions, page-table entries, and multiple resident experts in one long-lived graph or kernel family. Preserve optimized quantized operations and remove per-tile host orchestration.
2. Calibrate the user's actual hierarchy. Measure NVMe read latency, pageable-to-pinned copy rate, pinned PCIe bandwidth, CPU memory bandwidth, kernel launch cost, and concurrent copy/compute efficiency at startup.
3. Optimize utility, not hits. Schedule experts by expected critical-path time saved per transferred byte, with hard bandwidth and latency budgets.
4. Use MTP as a scheduling horizon. Continue expert-major union and explore whether accepted draft paths can provide enough lead time for the next routed layer without transferring rejected branches prematurely.
5. Co-design cache geometry with kernels. Choose slot count and tile geometry from actual matrix shapes, quantization blocks, and GPU occupancy rather than a storage-only heuristic.
6. Broaden evaluation. Test multiple consumer GPUs, PCIe generations, CPU memory configurations, models, contexts, prompts, and output lengths. Report cold, warm, and steady-state behavior separately.
7. Revisit expert LOD last. Develop per-tile compressed residual formats only after the exact path is bandwidth-measured and launch-efficient. Require task and perplexity gates plus an exact fallback.

The immediate goal is not a speculative 4x or 10x claim. It is to make the persistent exact path consistently beat the tuned native baseline. Larger gains can then be pursued through model-runtime co-design and measured compression rather than through uncontrolled caching.

13. Lemonade integration

Lemonade now discovers the local tiered runtime, passes model and hardware policy into launch configuration, and can identify when the experimental CUDA cache and CPU fallback are active. The Models view estimates a speed range from active parameters, context, RAM fit, CPU resources, GPU placement, and known measured paths, while labeling the actual runtime that will be used.

These values are intentionally estimates, not synthetic precision. Once a user benchmarks a model, measured device-specific results should supersede catalog estimates. The same calibration data can feed the scheduler's transfer economics, closing the loop between product guidance and runtime policy.

14. Reproducibility

The implementation and engineering notes live in the private llama.cpp-lemonade fork and the Lemonade application repository. The most relevant local documents are:

• llama.cpp-lemonade/LEMONADE-EXPERT-CACHE.md
• llama.cpp-lemonade/EXPERT-LOD.md
• Lemonade/ADAPTIVE-EXPERT-PROTOCOL.md

The fork includes focused benchmark and validation scripts for layer placement, MTP depths, route traces, expert unions, tiled transfers, scheduler modes, static and rolling widths, strict hash checks, and the live application launch path. A publishable follow-up should freeze the exact commit, model checksum, compiler flags, CUDA version, full hardware inventory, prompt suite, and raw result files.

15. Conclusion

Nanite for Experts turns sparse local MoE inference into a memory-scheduling problem with an exact native escape path. The implemented system can observe real routes, learn bounded transitions, preload RAM, union future expert demand, stage fine-grained tiles, maintain device page tables, and divide exact work between CPU and GPU. It is integrated far enough for Lemonade to launch and identify the runtime on user hardware.

The experiments also show where the attractive metaphor breaks down. Language-model experts are not geometry clusters: they participate in latency-sensitive dense linear algebra, and moving or launching them at excessively fine granularity can cost more than it saves. The best current design is consequently conservative: correct placement, native MTP, exact prefetch, stable residency, aggressive fallback, and transparent measurement.

That is a useful outcome. The project now has both a functioning virtual expert architecture and evidence about what must be true for it to become faster: predictable demand, low transfer volume, persistent expert-major execution, sufficient arithmetic intensity, and fidelity gates that extend beyond short output hashes. Those constraints transform “run a much larger model locally” from a caching slogan into a testable systems research program.

References

1. Qwen Team. Qwen3.6-35B-A3B model card.
2. ggml-org. llama.cpp command-line documentation.
3. JustVugg. Colibri repository and architecture notes.
4. Xue et al. MoE-Infinity: Activation-Aware Expert Offloading for Efficient MoE Serving.
5. Du et al. SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models. MLSys 2024.
6. Xue et al. MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs.
7. FloE: On-the-Fly MoE Inference on Memory-constrained GPU Hardware.
8. Epic Games. Nanite Virtualized Geometry.
9. NVIDIA. CUDA C++ Best Practices Guide.
10. Microsoft. PrefetchVirtualMemory documentation.
11. DeepSeek-AI. DeepSeek-V3 Technical Report.
12. NVIDIA. CUDA Graphs.

Cite this report

This report describes a prototype evaluated on a single system. Reuse the numbers with the hardware and method attached to them, and treat every figure as a paired result within its own experiment rather than a normalized benchmark. Corrections and replication attempts are welcome at support@aionx.aionapp.org.