FP4 QAT (W4A4)
3.6x weight bytes; 1.5-3x end to end
- Quality cost
- ~1% with QAT/native; PTQ worse when overtrained
- Overlaps with
- Hardware generation, MoE memory
- Evidence
- Vendor research, production adoption
The levers
Built fromresearch/01-report-path-to-1000x.md (edit)
A roughly 1000x cut in cost and energy per solved task, at fixed quality, is credible against a 2024 baseline (a dense ~405B transformer in BF16 on H100s) but not, on current evidence, against the late-2026 frontier. Against 2024, the market has already delivered somewhere between ~40x and ~470x at constant capability (Epoch measures ~13x per year), and a deliberate full-stack program built only from techniques that already work in production reaches an overlap-adjusted central estimate of ~700x in dollars and ~2,000x in energy, so 1000x against 2024 is mostly a 2027 to 2029 market outcome rather than a research breakthrough. Against the harder baseline, a late-2026 DeepSeek-V4-class sparse MoE with FP4 experts, compressed attention and multi-token prediction on GB300 racks, the same program adds a central ~35x in dollars and ~80x in energy (range ~4x to above 1,000x, with the top end requiring every lever to land at once). That is worth about 1.5 years of the market trend: a well-run program puts a team ahead of the curve rather than multiplying it. The gap between these estimates and the naive product of each lever's best published figure (~106 against 2024, ~400x against 2026) is the central lesson, because most levers shrink the same bytes moved per token, the binding bottleneck rotates from KV cache to weights to arithmetic to communication, and speculation, diffusion and batching all spend the same idle compute. The most credible stack combines a fine-grained MoE at 1 to 2 percent activation trained natively in FP4, hybrid linear plus compressed-sparse attention with 4-bit KV, trained-in multi-token drafting, RL that prices tokens per task, and a routing cascade whose distilled student runs on hardwired or SRAM silicon while the flagship runs on Rubin-class GPUs with disaggregated serving. No absolute solution exists: the largest single risk is that tokens per task are rising faster than model-side levers cut them, and nobody has published a controlled measurement of how these levers compound.
Source: research/01-report-path-to-1000x.md, line 3Edit on GitHub
Cost per solved task factors into four terms: tokens per task, bytes and FLOPs per token (weight bytes, KV bytes and arithmetic), cost or energy per byte and per FLOP (the hardware), and utilization (batching, scheduling and idle time). The physics explains why overlap is the rule. On H100-class silicon, reading one 8-bit weight from HBM costs about 90x the two FLOPs it feeds, and the batch size at which amortized weight reads equal arithmetic energy is about 90 for 8-bit weights and about 45 for 4-bit (derived from arXiv 2607.26571). Above that batch a GPU becomes arithmetic-bound, except for KV reads, which grow with context and are never shared across sequences. Landauer's limit sits 108 to 109 below today's per-token energy, so physics is far from binding; the practical limits are picojoules per bit of wire and memory traffic.
Source: research/01-report-path-to-1000x.md, line 29Edit on GitHub
Batching hides weight reads only until arithmetic binds
Per-token energy to read a Dense 70B model's weights from HBM, shared across a batch, against the 0.073 J of arithmetic each token needs. Above the crossover batch, fewer bits stop helping on a GPU.
| Weights | Read energy at batch 1 | Crossover batch |
|---|---|---|
| 16-bit weights | 13.1 J | 180 |
| 8-bit weights | 6.54 J | 89.8 |
| 4-bit weights | 3.27 J | 44.9 |
| 1.58-bit weights | 1.29 J | 17.7 |
Agentic workloads push hard on the KV term. A September 2026 study of SWE-bench Pro, Terminal-Bench 2.0, GAIA and similar suites found that decode is 91.0 to 98.6% of LLM time, mean contexts run 69 to 80K tokens (maxima 146 to 166K), and prefix-cache hit rates run 84.6 to 99.5%. With Qwen3.6-27B, raising concurrency from 15 to 20 collapsed the hit rate from 98.85% to 16.78% (alphaXiv 2605.26297). KV capacity therefore sets batch size, and batch size sets cost.
Bottleneck rotation explains why naive multiplication fails. In B2024 at 64K context, HBM capacity binds: per-token bytes come out to roughly the per-sequence KV times HBM/(HBM minus weights) (derived), so cutting KV is the first lever that matters. Compressing KV 10x to 50x lets batch grow until weight reads amortize. Arithmetic then binds, and MoE active-parameter cuts and FP4 tensor math take over. After that, communication and scheduling bind, which expert-parallel MoE serving shows clearly: DeepEP dispatch costs ~0.17 ms per layer, expert load balancing alone was worth 1.49x on prefill and 2.54x on decode, and two-batch overlap hurt decode by 27% at 32 tokens per device while helping by 25.5% at 256 (LMSYS). A lever multiplies the total only while its term binds, and the overall gain is capped by the next floor.
Six overlap families follow. In the bytes-per-token family, low-bit weights, MoE at small batch, on-chip or hardwired weights, and batching all compete for the same weight-movement term. In the idle-compute family, speculation, MTP, diffusion decoding and large batches all convert unused FLOPs in a memory-bound decode into tokens, so they should be counted once. In the active-parameter family, MoE efficiency leverage, distillation and capability density, early exit and activation sparsity all shrink the parameters touched per token at fixed quality. In the KV family, MLA, hybrid linear layers, sparse or compressed attention and KV quantization compete for the same term. Gains already embedded in hardware, such as FP4 and LUT weight formats, are part of Blackwell and Rubin generational numbers, so counting them again against a Rubin FP4 baseline double counts (SemiAnalysis). Finally, the tokens-per-task family (reasoning efficiency, effort control and routing) is the only one that multiplies cleanly with all per-token levers, because it acts on a different factor of cost per task.
Source: research/01-report-path-to-1000x.md, lines 31 to 35Edit on GitHub
3.6x weight bytes; 1.5-3x end to end
1.0-1.3x over 2-bit kernels on CPU/GPU
FP8 2x; NVFP4 ~2x more
1.1-1.3x alone
FP4 is now a first-class inference and training format. NVIDIA's 12B hybrid Mamba-Transformer trained on 10T tokens in NVFP4, the longest publicly documented 4-bit pretraining run, held relative loss within 1% of FP8 in the stable phase and ~1.5% after decay at about 830 tokens per parameter, with ~16% of linear layers kept in BF16. Downstream scores were close but mixed (MMLU-Pro 62.62 vs 62.58, GSM8K 89.08 vs 92.27), with code showing the largest gap at 59.52 vs 56.67 (arXiv 2509.25149). Production adoption is broad: DeepSeek-V4 applies FP4 QAT to routed experts, GPT-OSS uses MXFP4 QAT for more than 90% of its parameters, and NVIDIA reports that NVFP4 PTQ of DeepSeek-R1 lands within ~1% of FP8 (AIME 2024 89 vs 91, a vendor measurement) (NVIDIA Research). The ceiling is clear but realization falls short of it. BF16 to NVFP4 (~4.5 effective bits) cuts weight bytes ~3.6x, Blackwell FP4 math is 4x (GB200) to 6x (GB300) over BF16, and NVIDIA cites ~3x FP8 peak inference throughput, yet published end-to-end gains are mostly 1.5x to 3x.
Ternary and 2-bit formats are where vendor claims and independent measurement diverge most. Microsoft's largest official native ternary model is still the 2.4B BitNet b1.58 2B4T, and its "100B on a CPU" figure is a runtime capability claim with no released checkpoint (microsoft/BitNet). Prism ML's Ternary Bonsai 2 27B claims 98.2% retention at 1.76 bits per weight, but it is derived from Qwen3.8 27B and does not disclose its training method (PrismML), and DeepGrove's Maple-Preview 20B-A1B speed claims are unverified (AI Weekly). Microsoft's bitnet.cpp measures 1.37x to 6.17x speedups and 55 to 82% energy reductions on CPUs, but Intel's independent BITCOS work finds bitnet.cpp does not reach roofline and that optimized ternary kernels beat well-tuned 2-bit kernels by only 1.0x to 1.3x end to end (arXiv 2609.16338). Because no GPU has a ternary datapath, ternary's "multiplier-free" advantage only pays on custom silicon.
The quality risks are set by theory and practice together. The precision scaling law of Kumar et al. puts compute-optimal pretraining precision at ~7 to 8 bits, predicts that going below ~4 bits needs models more than 4x larger for equal loss, and finds that overtrained models are more sensitive to post-training quantization (arXiv 2411.04330). In practice, 2-bit PTQ is catastrophic for reasoning models: Qwen3-32B fell from 70.0 to 16.67 on AIME 2026, mean reasoning length grew ~6.8x (1,792 to 12,222 tokens), and raw 2-bit end-to-end task speed was 0.23x of FP16 (arXiv 2606.02011). That result is the clearest example of per-token savings turning into per-task losses.
KV quantization is more mature. FP8 KV halves memory and lowers the decode latency slope to 54% of BF16, at a cost of 128K NIAH 89% vs 91% and up to 0.7 to 2 points on reasoning (vLLM). NVFP4 KV is ~50% smaller than FP8 at under 1% loss by NVIDIA's measurement (NVIDIA Research), and on B200, W4A16 weights plus NVFP4 KV raised Qwen3-32B's maximum batch at 32K from 9 to 61 (arXiv 2606.02011). Weight sparsity, by contrast, is a minor lever. 2:4 Sparse Llama 3.1 8B recovers ~98% of accuracy but gives only 1.1x to 1.3x end to end alone, and 2x to 3x single-stream latency when combined with 4-bit weights (Red Hat). At a fixed 8x compression budget, "4-bit plus 50% sparsity" beats either method pushed alone, but the quality losses add (arXiv 2608.24070), and ternary checkpoints are already 30 to 52% zeros, so sparsity stacked on ternary has little headroom (arXiv 2609.16338).
Source: research/01-report-path-to-1000x.md, lines 41 to 47Edit on GitHub
Deep dive: the low-bit numerics note, with every finding, inference and gap.
~7x efficiency leverage; 5.2x vs naive TP serving
1.5-1.8x at batch 1; ~1x at scale
Frontier open models converged on 3 to 5% active parameters: Kimi K2.6 at 1T total and 32B active (Artificial Analysis), DeepSeek V4-Pro at 1.6T and 49B (aman.ai), gpt-oss-120b at 116.8B and 5.13B (arXiv 2508.10925), and Qwen3-Next at 80B and ~3B with 512 experts (VentureBeat). Kimi K3 reportedly reaches 2.8T with 16 of 896 experts active, but that figure comes from a secondary blog and is unverified (victorgrosser.nl). The relevant number is efficiency leverage at equal loss, not the total-to-active ratio. Ant Group's scaling laws, fit on more than 300 models up to 28B, predict efficiency leverage above 7x at 3.1% activation, rising with compute budget, and in validation a 0.85B-active MoE matched a 6.1B dense model on the same 1T tokens (arXiv 2507.17702). Returns are sublinear: halving activation again buys far less than 2x, and no law has been validated at frontier compute or below 2% activation.
Serving reality is the catch. SGLang's open reproduction served DeepSeek-V3 at ~$0.20 per million output tokens on 96 H100s, but only with wide expert parallelism, disaggregated prefill and decode and load balancing, which together ran up to 5.2x faster at decode than naive TP16 (LMSYS). By derived roofline arithmetic, an expert needs ~150 tokens to be compute-bound on BF16, so a 384-expert top-8 model needs about 7,000 concurrent tokens per expert-parallel group; below that, decode pays for total weight bytes rather than active FLOPs. MoE compounds well with low-bit experts, but with a knowledge cost. Allen-Zhu and Li measure ~2 bits of knowledge per parameter, with no loss at int8 but more than 2x loss at int4, while a 32-expert MoE stores ~1.3x less per total parameter while using ~9% of parameters at inference (HF Papers). Those are the quantitative overlaps between quantization and sparsity on knowledge capacity.
Activation sparsity and adaptive depth are small levers at datacenter scale. TEAL reaches 40 to 50% training-free activation sparsity for 1.53x to 1.8x at batch 1 (arXiv 2408.14690), and TurboSparse reaches ~90% FFN sparsity with 4.64x on a consumer GPU and 22x on mobile (arXiv 2406.05955), but both erode at large batch because the union of active neurons across a batch approaches dense. Early exit is fading: under an oracle MMLU tolerance, Llama3-8B can skip only 0.18% of layers against 4.69% for Llama2-7B (alphaXiv 2603.23701). Mixture-of-depths showed isoFLOP training wins only at 3B scale or below, with unproven serving benefits (Graphcore Research).
Source: research/01-report-path-to-1000x.md, lines 51 to 55Edit on GitHub
Deep dive: the moe and conditional compute note, with every finding, inference and gap.
~10x/yr fewer parameters at fixed benchmark
Capability per parameter has risen about 10x per year on saturating benchmarks, with the densing law finding maximum density doubling every ~3.3 to 3.5 months (arXiv 2412.04315). Distillation drives much of the 13x/yr price trend: DeepSeek-R1-Distill-Qwen-32B reached 72.6% on AIME 2024 against 47.0% for direct RL on the same base (arXiv 2501.12948). The limits are real, though. Post-hoc compression often loses density: pruned-and-distilled models such as Minitron-4B and Llama-3.2-1B/3B ended up less dense than their parents, and only Gemma-2-9B beat its source (arXiv 2412.04315). Apple's distillation scaling laws show distillation beats pretraining only when the teacher's cost is amortized, with the benefit stopping at a capacity gap (arXiv 2502.08606). Small 2026 models lack knowledge and spend tokens: Qwen3.5-9B scores 14.7% on AA-Omniscience with an 82% hallucination rate and uses 2x to 4x more output tokens than frontier models (Artificial Analysis). Order matters too, since quantizing before pruning or distilling pushed perplexity from ~3.2 to between 34 and 53 (arXiv 2511.19495). The implication is that reaching frontier quality with far fewer active parameters requires either sparse storage (MoE) or moving long-tail knowledge out of the weights into retrieval and tools, because the ~2 bits per parameter floor rules out holding frontier factual recall in a single-digit-billion dense model.
Source: research/01-report-path-to-1000x.md, line 59Edit on GitHub
Deep dive: the distillation and cost trends note, with every finding, inference and gap.
6-7x decode at 128K; ~50x KV at 1M
DeepSeek-V4 is the strongest measured data point. It combines 4x-compressed KV with a lightning-indexer top-k (CSA), 128x-compressed dense attention (HCA) and a sliding window, and at 1M context V4-Flash needs ~10% of V3.2's per-token FLOPs and ~7% of its KV, about 2% of the KV of a BF16 GQA-8 baseline (HF DeepSeek-V4). These are vendor figures, but the previous step has market evidence behind it: DSA in V3.2 claimed 3.5x cheaper prefill and 6 to 7x cheaper decode at 128K with benchmark parity, and DeepSeek cut output prices 75% (The Decoder).
Hybrids with about three linear layers per full-attention layer form the other camp. Kimi Linear cut time per output token at 1M from 11.48 ms to 1.84 ms, though only 2.3x at batch 1, because most of the gain comes from bigger batches (alphaXiv 2510.26692). Nemotron-H 56B reached 2.4x the per-GPU output throughput of Llama-3.1-70B at 64K input (arXiv 2504.03624), Nemotron 3 Nano reached 3.3x Qwen3-30B-A3B at nearly equal active parameters, which isolates the attention contribution (HF/NVIDIA), and Olmo Hybrid 7B led Olmo 3 on RULER at 64K, 85.0 vs 70.9 (AI2). The main caution comes from MiniMax, whose Lightning-attention hybrid matched full attention at small scale but showed "clear deficits in complex, multi-hop reasoning tasks" at larger scale. MiniMax returned to full attention for M2 and noted that prefix caching and speculative decoding were immature for linear layers (MiniMax). These gains mostly stack with MoE because they hit a different term, but within the KV family they overlap, since hybrid layers, sparse selection and KV quantization all chase the same bytes. They also fade at short context: vLLM finds KV reads dominate decode only beyond about 7K tokens (vLLM).
Source: research/01-report-path-to-1000x.md, lines 63 to 65Edit on GitHub
Deep dive: the attention and kv cache note, with every finding, inference and gap.
1.38x at batch 64; ~3x at 70-140 tok/s/user
5-15x per-user speed; ~1-2x cost
Speculative decoding and MTP are mainly latency levers. EAGLE-3 reaches up to 6.5x at batch 1 but 1.38x at batch 64 (arXiv 2503.01840). DeepSeek's DSpark raised V4 per-user speed 57 to 85% at matched aggregate throughput on live traffic, an interactivity gain rather than more tokens per GPU (vendor result, reported by Analytics India Magazine). On energy, EAGLE-3 at batch 1 saves 2.1x to 2.5x on 8B to 13B models but only 1.34x on Llama-70B, and classic draft models can increase energy, with factors of 0.63 to 0.80 (arXiv 2602.09113). InferenceX's 21x cost gap from switching MTP on (GB300, DeepSeek-R1 at 150 tok/s/user, $2.35/M off vs $0.11/M on) mostly reflects how badly underused the GPU is without speculation at that speed; at 70 to 140 tok/s/user the gain is about 3x (Silicon Report).
Diffusion LLMs deliver ~1,000 to 1,500 tok/s per user but have not shown per-token cost wins at serving batch. Same-lab comparisons show a quality gap: Gemini Diffusion trails Gemini 2.0 Flash-Lite on GPQA Diamond 40.4 vs 56.5 (DeepMind), and DiffusionGemma trails Gemma 4 on AIME 2026, 69.1 vs 88.3 (Decrypt). Inception's Mercury 3 parity claims against Haiku-class models are self-reported with no independent testing, and its $0.70/M output list price is only about 35% below rivals (Let's Data Science). LLaDA2.0's 1.9x speedup over autoregressive decoding is explicitly a small-batch result (LMSYS). Diffusion and speculation exploit the same idle compute, so they should not be multiplied together.
Source: research/01-report-path-to-1000x.md, lines 69 to 71Edit on GitHub
Deep dive: the decoding and test-time compute note, with every finding, inference and gap.
~2x (research); 4.4x spread at frontier
~8x cost for -4 pts
On the Artificial Analysis Index v5 (October 2026), median output per task rose from ~35k to ~59k tokens, and every model measured generated more tokens than in the previous version. Four agentic evaluations carry 55% of the index weight but produce ~90% of its tokens. Cost per task among top models spans more than 13x, and GPT-6.1 Sol (max) lands within 4 points of Opus 5.5 (max) at about one-eighth the cost per task (Artificial Analysis). Tokens-per-task levers are the only ones that multiply cleanly with every per-token lever, yet market pressure is pushing tokens per task up.
Research methods cut reasoning tokens by about half, mostly on small models. AdaptThink cut length 53% while gaining 2.4 points on a 1.5B model (arXiv 2505.13417), ThinkPrune halved length for a ~2% loss on AIME (arXiv 2504.01296), and selective latent thinking cut 58% at a 2.8-point cost, although naive latent compression on Qwen3-4B dropped GSM-Hard from 54.7% to 17.3% (alphaXiv 2605.25745). A small model plus test-time compute is a legitimate substitute for parameters on medium-difficulty problems: FLOPs-matched test-time compute beat a 14x larger model where the small model had non-trivial success, and lost on the hardest problems (arXiv 2408.03314). Routing is bounded by the share of hard tasks. If 30% of tasks need the flagship, no cascade can exceed ~3.3x however cheap the student becomes (derived).
Source: research/01-report-path-to-1000x.md, lines 75 to 77Edit on GitHub
Deep dive: the decoding and test-time compute note, with every finding, inference and gap.
1.2-2.8x on same GB300 in 6 months
On identical GB300 NVL72 hardware, audited MLPerf per-GPU throughput rose from v5.1 to v6.0 by 2.77x for DeepSeek-R1 Server but only 1.21x for Llama 3.1 405B Offline (StorageReview). NVIDIA attributes the gains to kernel fusion, Dynamo disaggregation, wide expert parallelism, MTP and KV-aware routing, but publishes no per-technique split. Software gains are largest for new sparse and reasoning-heavy architectures and smallest for mature dense models. Independent InferenceX data implies GB200 cost per token improved roughly 2.5x to 5x within a year at high interactivity, but only about 1.3x at low interactivity (SemiAnalysis). A new co-designed architecture should therefore expect a launch-time software penalty that closes over 6 to 12 months.
Source: research/01-report-path-to-1000x.md, line 81Edit on GitHub
Deep dive: the hardware and serving note, with every finding, inference and gap.
3.1x (H100 to B200); 2-5.4x (GB300 to Rubin) per MW
3-6x per-user speed; cost "comparable"
~50-74x per-user speed; ~1.4x cost at R1 scale
~10x power (simulated)
None published
On independent InferenceX data, B200 delivers ~3.1x the tokens per megawatt of H100 on GPT-OSS-120B, and GB200 NVL72 with FP4 delivers ~8x a single-node H200 with FP8 on DeepSeek-R1 (Silicon Report). Vera Rubin NVL72 reaches 2x GB300 up to 100 tok/s/user, ~4x near 200 and 5.4x at 300, where GB300 is barely viable. Cost improves less than energy, because owner TCO is $3.57 per GPU-hour for Rubin against $2.36 for GB300 and $1.84 for GB200. CoreWeave's 5.4x performance per MW over GB200 is an operator claim that SemiAnalysis did not verify (SemiAnalysis). Rubin adds a ~3.125-bit LUT weight format and 2.8x Blackwell Ultra's memory bandwidth, and its sparsity-labeled NVFP4 inference rating of 50 PFLOPS is 12.5x its 4 PFLOPS BF16 rating (Tom's Hardware). From H100 to Rubin the plausible gain is ~10x to 30x in tokens per MW at moderate interactivity, and less in dollars.
SRAM-centric and wafer-scale chips win on per-user speed. On 8B-class models, Cerebras runs ~2,000 tok/s/user, SambaNova ~900 and Groq ~600, against ~350 on Blackwell (EE Times). Cerebras's 981 tok/s on a trillion-parameter model comes with per-token cost it calls only "comparable to GPU providers" (Pulse), and Harvard's independent measurements found Cerebras, SambaNova and Gaudi idle at 10 to 60% higher power than GPUs (arXiv 2604.10852). TPU v7 has no independent per-chip inference benchmark (Silicon Report).
Hardwired weights are the most dramatic data point. Taalas HC1 etches Llama-3.1-8B into mask ROM at ~4 bits on an 815 mm² TSMC N6 die drawing ~250 W. EE Times observed 15,000+ tok/s/user hands-on, the company claims ~17,000 and ~73x H200 throughput, and no accuracy figures have been published. Taalas's own frontier-scale cost claim is 7.6 cents per million tokens for DeepSeek-R1 on ~30 chips (EE Times), which is only ~1.4x below GB300 with MTP ($0.11/M). Hardwiring is therefore a huge speed multiplier but a modest cost multiplier at frontier scale, because ~30 tape-outs per model must be amortized and each model change needs a ~2-month respin. AMD agreed to acquire Taalas on August 6, 2026 (SiliconANGLE). Analog in-memory compute is still at simulation grade: Sagence's claimed Llama2-70B system at 666,000 tok/s and 59 kW, against a 624 kW H100 system, is simulated (IEEE Spectrum), and IBM and ETH Zurich show that off-the-shelf LLMs cannot even reach W4A8 quality on noisy analog hardware without a dedicated training recipe (Semiconductor Engineering). Photonics reached its first LLM demo on November 17, 2026, when Q.ANT replaced one attention layer with optical Fourier transforms, but it published no size, accuracy, throughput or energy figures (The Quantum Insider).
Source: research/01-report-path-to-1000x.md, lines 85 to 89Edit on GitHub
Deep dive: the hardware and serving note, with every finding, inference and gap.
| Lever | Best realized gain (Nov 2026) | Quality cost | Overlaps with | Evidence grade |
|---|---|---|---|---|
| FP4 QAT (W4A4) | 3.6x weight bytes; 1.5-3x end to end | ~1% with QAT/native; PTQ worse when overtrained | Hardware generation, MoE memory | Vendor research, production adoption |
| Ternary / 2-bit | 1.0-1.3x over 2-bit kernels on CPU/GPU | 2-bit PTQ collapses reasoning; ternary at frontier scale unproven | Sparsity (30-52% zeros), custom silicon | Mostly vendor; Intel independent |
| KV quantization | FP8 2x; NVFP4 ~2x more | 0.7-2 pts (FP8); <1% (NVFP4, vendor) | Attention compression | Independent (vLLM) plus vendor |
| 2:4 weight sparsity | 1.1-1.3x alone | ~2-3% | Quantization (quality adds) | Vendor plus academic |
| MoE at ~3% activation | ~7x efficiency leverage; 5.2x vs naive TP serving | None by construction; ~1.3x less knowledge per total parameter | Batching, low-bit memory | Academic law, open reproduction |
| Activation sparsity / early exit | 1.5-1.8x at batch 1; ~1x at scale | Minimal | MoE, distillation | Academic |
| Distillation / density | ~10x/yr fewer parameters at fixed benchmark | Brittleness, hallucination, 2-4x tokens | MoE, routing | Independent (Epoch, AA) plus academic |
| Compressed / hybrid attention | 6-7x decode at 128K; ~50x KV at 1M | Multi-hop risk at scale (MiniMax) | KV quantization; prefix caching | Vendor reports, price evidence |
| Speculation / MTP | 1.38x at batch 64; ~3x at 70-140 tok/s/user | Lossless | Batching, low-bit, diffusion | Independent plus vendor |
| Diffusion LLMs | 5-15x per-user speed; ~1-2x cost | 10-20 pts on reasoning (same lab) | Speculation | Mostly vendor, self-reported |
| Reasoning efficiency | ~2x (research); 4.4x spread at frontier | ±3 pts | Multiplies with per-token levers | Independent (AA) plus academic |
| Routing / effort control | ~8x cost for -4 pts | Bounded by hard-task share | Density | Independent (AA) |
| Serving software | 1.2-2.8x on same GB300 in 6 months | None | MoE, MTP | Audited (MLPerf) |
| GPU generation | 3.1x (H100 to B200); 2-5.4x (GB300 to Rubin) per MW | None | Embedded FP4/LUT | Independent (InferenceX) |
| SRAM / wafer-scale | 3-6x per-user speed; cost "comparable" | None | Hardwired | AA speeds, vendor cost |
| Hardwired weights | ~50-74x per-user speed; ~1.4x cost at R1 scale | Aggressive 4-bit, unmeasured | Low-bit, MoE (30 tape-outs) | Vendor plus EE Times hands-on |
| Analog in-memory | ~10x power (simulated) | Needs noise-aware training | Low-bit | Simulation only |
| Photonics | None published | Unknown | Attention alternatives | Vendor demo only |
Source: research/01-report-path-to-1000x.md, lines 93 to 112Edit on GitHub
The tables below give each lever's standalone multiplier (low, central, high) and then combine them in four steps. Levers in the same family are grouped first. The model-side groups (weights versus KV) are then combined over their shares of bytes moved rather than multiplied together. Each idle-compute effect and each hardware-embedded effect is counted once. Finally, the group subtotals are multiplied, because they act on different factors of cost per task. A useful check is the ceiling the physics allows (derived). An MoE with ~20B active parameters in FP4, batch-amortized weights and sparse KV reads does ~40 GFLOP of weight math per token, about 20x less than the baseline's 0.8 TFLOP, and at H100 coefficients its energy floor is roughly 0.03 to 0.05 J per token, about 200x below B2024's ~9 J, before any hardware improvement. Realized model-side gains land well below that ceiling because of kernel efficiency, expert-parallel communication, prefill and idle power.
Multiplied naively the levers promise a million; overlap-adjusted, about a thousand
Each lever's standalone multiplier (low to high, dot at central), grouped by the term it attacks, then the overlap-adjusted subtotals and totals. Hollow dots are the naive product of every central row.
Every compounding factor here is the survey's inference: no source has measured these levers together at frontier scale.
Table B: against B2026 (DeepSeek-V4-Pro-class on GB300 NVL72, late-2026 software), fixed late-2026-frontier quality
| Lever | Low | Central | High | Overlap treatment |
|---|---|---|---|---|
| FP4 to ~3-bit LUT or ternary experts | 1x | 1.3x | 2x | Rubin LUT is ~1.36x fewer bytes than MXFP4 |
| MoE from ~3% to ~1.5% activation | 1.1x | 1.4x | 1.8x | Sublinear efficiency leverage |
| Frontier distillation plus retrieval offload of knowledge | 1.2x | 2x | 4x | Knowledge floor; brittleness risk |
| Activation sparsity / early exit | 1x | 1.05x | 1.2x | Near zero at scale |
| KV from FP8 to FP4 / 2-bit | 1.3x | 2x | 3x | KV already ~2% of GQA baseline, so its share is small |
| Hybrid linear layers on top of compressed-sparse attention | 1.1x | 1.5x | 2.5x | Same KV term |
| Model-side per token (overlap-adjusted) | 1.2x | 2x | 5x | Naive central ~11 |
| Serving maturity on new architecture plus DSpark-class drafting | 1.2x | 1.7x | 3x | MTP already in baseline |
| GB300 to Rubin (cost) | 1.5x | 2x | 4x | Energy central ~3x |
| Hardwired / SRAM tier for routed student (cost) | 1x | 1.3x | 3x | Energy central ~2x; up to ~10x on routed share |
| Reasoning-token efficiency (cost-aware RL, adaptive thinking) | 1.3x | 2x | 4x | Frontier spread 4.4x is the existence proof |
| Routing cascade and effort control | 1.5x | 3x | 6x | Capped by hard-task share; overlaps the row above |
| Tokens-per-task subtotal (overlap-adjusted) | 2x | 4x | 10x | |
| Naive product of all central rows | 420x | |||
| Overlap-adjusted total, cost per solved task | 4x | 35x | 1,800x | High is improbable: every lever must land |
| Overlap-adjusted total, energy per solved task | 6x | 80x | 5,000x |
Table A: against B2024 (dense ~405B, BF16, H100, 2024 software), fixed 2024-frontier quality, ~64K agentic mix
| Lever | Low | Central | High | Overlap treatment |
|---|---|---|---|---|
| FP4 weights and activations (QAT) | 1.5x | 2.5x | 3.6x | Weight group |
| Sub-4-bit (LUT / ternary) beyond FP4 | 1x | 1.2x | 2x | Weight group; GPU kernels cap it |
| MoE at ~3% activation | 3x | 6x | 10x | Weight group; needs wide-EP batch |
| Capability density (data, overtraining, distillation) | 4x | 10x | 30x | Weight group; overlaps MoE on active parameters |
| Activation sparsity / adaptive depth | 1x | 1.1x | 1.5x | Weight group; erodes at batch |
| Weight-group subtotal (overlap-adjusted) | 15x | 60x | 200x | Naive central ~200 |
| Attention architecture (MLA, hybrid, compressed-sparse) | 3x | 8x | 20x | KV group |
| KV quantization (BF16 to FP8/FP4) | 2x | 3x | 4x | KV group, same term |
| KV-group subtotal (overlap-adjusted) | 4x | 12x | 40x | Naive central ~24 |
| Model-side per token (byte-share combination plus batch synergy) | 10x | 30x | 80x | Shares ~65% weights / ~35% KV at capacity-limited batch |
| Serving software and scheduling | 1.5x | 2.5x | 5x | Utilization group |
| Speculation / MTP | 1.2x | 1.6x | 3x | Utilization group; same idle compute as batching |
| Diffusion decoding | 1x | 1x | 2x | Not credited (overlaps speculation; quality gap) |
| Utilization subtotal | 1.5x | 2.5x | 4x | |
| GPU generation H100 to Rubin (cost) | 3x | 5x | 8x | Energy central ~10x; FP4 counted above |
| Specialized silicon for routed share (cost) | 1x | 1.2x | 2x | Energy central ~2x |
| Tokens per task (reasoning efficiency, routing) | 0.7x | 1.5x | 3x | B2024 is a non-reasoning model, so small reasoning replacements can spend more tokens |
| Naive product of all central rows | 1,000,000x | |||
| Overlap-adjusted total, cost per solved task | 30x | 700x | 15,000x | High requires every lever at its best |
| Overlap-adjusted total, energy per solved task | 60x | 2,000x | 40,000x | Hardware energy improves faster than $/token |
| Lever | Low | Central | High | Overlap treatment |
|---|---|---|---|---|
| FP4 weights and activations (QAT) | 1.5 | 2.5 | 3.6 | Weight group |
| Sub-4-bit (LUT / ternary) beyond FP4 | 1.0 | 1.2 | 2.0 | Weight group; GPU kernels cap it |
| MoE at ~3% activation | 3 | 6 | 10 | Weight group; needs wide-EP batch |
| Capability density (data, overtraining, distillation) | 4 | 10 | 30 | Weight group; overlaps MoE on active parameters |
| Activation sparsity / adaptive depth | 1.0 | 1.1 | 1.5 | Weight group; erodes at batch |
| Weight-group subtotal (overlap-adjusted) | 15 | 60 | 200 | Naive central ~200 |
| Attention architecture (MLA, hybrid, compressed-sparse) | 3 | 8 | 20 | KV group |
| KV quantization (BF16 to FP8/FP4) | 2 | 3 | 4 | KV group, same term |
| KV-group subtotal (overlap-adjusted) | 4 | 12 | 40 | Naive central ~24 |
| Model-side per token (byte-share combination plus batch synergy) | 10 | 30 | 80 | Shares ~65% weights / ~35% KV at capacity-limited batch |
| Serving software and scheduling | 1.5 | 2.5 | 5 | Utilization group |
| Speculation / MTP | 1.2 | 1.6 | 3 | Utilization group; same idle compute as batching |
| Diffusion decoding | 1.0 | 1.0 | 2 | Not credited (overlaps speculation; quality gap) |
| Utilization subtotal | 1.5 | 2.5 | 4 | |
| GPU generation H100 to Rubin (cost) | 3 | 5 | 8 | Energy central ~10x; FP4 counted above |
| Specialized silicon for routed share (cost) | 1.0 | 1.2 | 2 | Energy central ~2x |
| Tokens per task (reasoning efficiency, routing) | 0.7 | 1.5 | 3 | B2024 is a non-reasoning model, so small reasoning replacements can spend more tokens |
| Naive product of all central rows | ~106 | |||
| Overlap-adjusted total, cost per solved task | ~30x | ~700x | ~15,000x | High requires every lever at its best |
| Overlap-adjusted total, energy per solved task | ~60x | ~2,000x | ~40,000x | Hardware energy improves faster than $/token |
Two cross-checks support Table A. The low case (~30x) matches the most conservative market trend over 2.4 years (~40x, open-weight Pareto), and the central case sits a few times above the headline trend (~250x to 470x), which is the expected gap between the market's average deployment and a fleet that uses Rubin, routing and token-efficient RL together. A derived roofline gives the same order of magnitude. B2024 moves ~92 GB per token at 64K, which bounds 16 H100s to a few hundred tokens per second of aggregate decode and puts its cost in the tens of dollars per million tokens. InferenceX measures DeepSeek-V4-Pro at $0.12/M on GB300 at much higher quality, a per-token gap of very roughly 100x to 250x before any tokens-per-task lever, consistent with the central row. 1000x against 2024 is credible today in energy and within about a year in dollars, and most of it is what the market already does.
Source: research/01-report-path-to-1000x.md, lines 118 to 143Edit on GitHub
| Lever | Low | Central | High | Overlap treatment |
|---|---|---|---|---|
| FP4 to ~3-bit LUT or ternary experts | 1.0 | 1.3 | 2.0 | Rubin LUT is ~1.36x fewer bytes than MXFP4 |
| MoE from ~3% to ~1.5% activation | 1.1 | 1.4 | 1.8 | Sublinear efficiency leverage |
| Frontier distillation plus retrieval offload of knowledge | 1.2 | 2.0 | 4.0 | Knowledge floor; brittleness risk |
| Activation sparsity / early exit | 1.0 | 1.05 | 1.2 | Near zero at scale |
| KV from FP8 to FP4 / 2-bit | 1.3 | 2.0 | 3.0 | KV already ~2% of GQA baseline, so its share is small |
| Hybrid linear layers on top of compressed-sparse attention | 1.1 | 1.5 | 2.5 | Same KV term |
| Model-side per token (overlap-adjusted) | 1.2 | 2.0 | 5.0 | Naive central ~11 |
| Serving maturity on new architecture plus DSpark-class drafting | 1.2 | 1.7 | 3.0 | MTP already in baseline |
| GB300 to Rubin (cost) | 1.5 | 2.0 | 4.0 | Energy central ~3x |
| Hardwired / SRAM tier for routed student (cost) | 1.0 | 1.3 | 3.0 | Energy central ~2x; up to ~10x on routed share |
| Reasoning-token efficiency (cost-aware RL, adaptive thinking) | 1.3 | 2.0 | 4.0 | Frontier spread 4.4x is the existence proof |
| Routing cascade and effort control | 1.5 | 3.0 | 6.0 | Capped by hard-task share; overlaps the row above |
| Tokens-per-task subtotal (overlap-adjusted) | 2 | 4 | 10 | |
| Naive product of all central rows | ~420x | |||
| Overlap-adjusted total, cost per solved task | ~4x | ~35x | ~1,800x | High is improbable: every lever must land |
| Overlap-adjusted total, energy per solved task | ~6x | ~80x | ~5,000x |
Against the late-2026 frontier, model-side headroom per token is only about 2x, because DeepSeek-V4 and its peers have already collected MoE, FP4 and most of the KV savings. What remains is spread across hardware (about 2x from Rubin), serving maturity, a specialized-silicon tier and, above all, tokens per task, which offers the largest remaining factor (about 4x) and is the only lever family on which the market is currently moving backward. Trend arithmetic converts these numbers into time. At 10x to 13x per year, 35x is about 1.4 to 1.5 years of progress. On trend, late-2026 frontier capability becomes ~1000x cheaper around 2029 with or without this program, so the program's value is reaching each step earlier and owning the cost structure. It does not multiply the trend, because the trend is other labs adopting the same levers.
Source: research/01-report-path-to-1000x.md, lines 145 to 166Edit on GitHub
Source: research/01-report-path-to-1000x.md, lines 114 to 166Edit on GitHub