Corerope

The levers

Stacked levers reach 1000x only against 2024

A roughly 1000x cut in cost and energy per solved task, at fixed quality, is credible against a 2024 baseline (a dense ~405B transformer in BF16 on H100s) but not, on current evidence, against the late-2026 frontier. Against 2024, the market has already delivered somewhere between ~40x and ~470x at constant capability (Epoch measures ~13x per year), and a deliberate full-stack program built only from techniques that already work in production reaches an overlap-adjusted central estimate of ~700x in dollars and ~2,000x in energy, so 1000x against 2024 is mostly a 2027 to 2029 market outcome rather than a research breakthrough. Against the harder baseline, a late-2026 DeepSeek-V4-class sparse MoE with FP4 experts, compressed attention and multi-token prediction on GB300 racks, the same program adds a central ~35x in dollars and ~80x in energy (range ~4x to above 1,000x, with the top end requiring every lever to land at once). That is worth about 1.5 years of the market trend: a well-run program puts a team ahead of the curve rather than multiplying it. The gap between these estimates and the naive product of each lever's best published figure (~106 against 2024, ~400x against 2026) is the central lesson, because most levers shrink the same bytes moved per token, the binding bottleneck rotates from KV cache to weights to arithmetic to communication, and speculation, diffusion and batching all spend the same idle compute. The most credible stack combines a fine-grained MoE at 1 to 2 percent activation trained natively in FP4, hybrid linear plus compressed-sparse attention with 4-bit KV, trained-in multi-token drafting, RL that prices tokens per task, and a routing cascade whose distilled student runs on hardwired or SRAM silicon while the flagship runs on Rubin-class GPUs with disaggregated serving. No absolute solution exists: the largest single risk is that tokens per task are rising faster than model-side levers cut them, and nobody has published a controlled measurement of how these levers compound.

Every lever attacks one of four terms, and the binding term rotates

Cost per solved task factors into four terms: tokens per task, bytes and FLOPs per token (weight bytes, KV bytes and arithmetic), cost or energy per byte and per FLOP (the hardware), and utilization (batching, scheduling and idle time). The physics explains why overlap is the rule. On H100-class silicon, reading one 8-bit weight from HBM costs about 90x the two FLOPs it feeds, and the batch size at which amortized weight reads equal arithmetic energy is about 90 for 8-bit weights and about 45 for 4-bit (derived from arXiv 2607.26571). Above that batch a GPU becomes arithmetic-bound, except for KV reads, which grow with context and are never shared across sequences. Landauer's limit sits 108 to 109 below today's per-token energy, so physics is far from binding; the practical limits are picojoules per bit of wire and memory traffic.

Batching hides weight reads only until arithmetic binds

Per-token energy to read a Dense 70B model's weights from HBM, shared across a batch, against the 0.073 J of arithmetic each token needs. Above the crossover batch, fewer bits stop helping on a GPU.

  • 16-bit weights
  • 8-bit weights
  • 4-bit weights
  • 1.58-bit weights
  • Arithmetic per token
Show the numbers
Crossover batch by weight precision
WeightsRead energy at batch 1Crossover batch
16-bit weights13.1 J180
8-bit weights6.54 J89.8
4-bit weights3.27 J44.9
1.58-bit weights1.29 J17.7

Agentic workloads push hard on the KV term. A September 2026 study of SWE-bench Pro, Terminal-Bench 2.0, GAIA and similar suites found that decode is 91.0 to 98.6% of LLM time, mean contexts run 69 to 80K tokens (maxima 146 to 166K), and prefix-cache hit rates run 84.6 to 99.5%. With Qwen3.6-27B, raising concurrency from 15 to 20 collapsed the hit rate from 98.85% to 16.78% (alphaXiv 2605.26297). KV capacity therefore sets batch size, and batch size sets cost.

Bottleneck rotation explains why naive multiplication fails. In B2024 at 64K context, HBM capacity binds: per-token bytes come out to roughly the per-sequence KV times HBM/(HBM minus weights) (derived), so cutting KV is the first lever that matters. Compressing KV 10x to 50x lets batch grow until weight reads amortize. Arithmetic then binds, and MoE active-parameter cuts and FP4 tensor math take over. After that, communication and scheduling bind, which expert-parallel MoE serving shows clearly: DeepEP dispatch costs ~0.17 ms per layer, expert load balancing alone was worth 1.49x on prefill and 2.54x on decode, and two-batch overlap hurt decode by 27% at 32 tokens per device while helping by 25.5% at 256 (LMSYS). A lever multiplies the total only while its term binds, and the overall gain is capped by the next floor.

Six overlap families follow. In the bytes-per-token family, low-bit weights, MoE at small batch, on-chip or hardwired weights, and batching all compete for the same weight-movement term. In the idle-compute family, speculation, MTP, diffusion decoding and large batches all convert unused FLOPs in a memory-bound decode into tokens, so they should be counted once. In the active-parameter family, MoE efficiency leverage, distillation and capability density, early exit and activation sparsity all shrink the parameters touched per token at fixed quality. In the KV family, MLA, hybrid linear layers, sparse or compressed attention and KV quantization compete for the same term. Gains already embedded in hardware, such as FP4 and LUT weight formats, are part of Blackwell and Rubin generational numbers, so counting them again against a Rubin FP4 baseline double counts (SemiAnalysis). Finally, the tokens-per-task family (reasoning efficiency, effort control and routing) is the only one that multiplies cleanly with all per-token levers, because it acts on a different factor of cost per task.

What November 2026 actually measures, lever by lever

Low-bit numerics: FP4 is solved, ternary is not

FP4 QAT (W4A4)

3.6x weight bytes; 1.5-3x end to end

Quality cost
~1% with QAT/native; PTQ worse when overtrained
Overlaps with
Hardware generation, MoE memory
Evidence
Vendor research, production adoption

Ternary / 2-bit

1.0-1.3x over 2-bit kernels on CPU/GPU

Quality cost
2-bit PTQ collapses reasoning; ternary at frontier scale unproven
Overlaps with
Sparsity (30-52% zeros), custom silicon
Evidence
Mostly vendor; Intel independent

KV quantization

FP8 2x; NVFP4 ~2x more

Quality cost
0.7-2 pts (FP8); <1% (NVFP4, vendor)
Overlaps with
Attention compression
Evidence
Independent (vLLM) plus vendor

2:4 weight sparsity

1.1-1.3x alone

Quality cost
~2-3%
Overlaps with
Quantization (quality adds)
Evidence
Vendor plus academic

FP4 is now a first-class inference and training format. NVIDIA's 12B hybrid Mamba-Transformer trained on 10T tokens in NVFP4, the longest publicly documented 4-bit pretraining run, held relative loss within 1% of FP8 in the stable phase and ~1.5% after decay at about 830 tokens per parameter, with ~16% of linear layers kept in BF16. Downstream scores were close but mixed (MMLU-Pro 62.62 vs 62.58, GSM8K 89.08 vs 92.27), with code showing the largest gap at 59.52 vs 56.67 (arXiv 2509.25149). Production adoption is broad: DeepSeek-V4 applies FP4 QAT to routed experts, GPT-OSS uses MXFP4 QAT for more than 90% of its parameters, and NVIDIA reports that NVFP4 PTQ of DeepSeek-R1 lands within ~1% of FP8 (AIME 2024 89 vs 91, a vendor measurement) (NVIDIA Research). The ceiling is clear but realization falls short of it. BF16 to NVFP4 (~4.5 effective bits) cuts weight bytes ~3.6x, Blackwell FP4 math is 4x (GB200) to 6x (GB300) over BF16, and NVIDIA cites ~3x FP8 peak inference throughput, yet published end-to-end gains are mostly 1.5x to 3x.

Ternary and 2-bit formats are where vendor claims and independent measurement diverge most. Microsoft's largest official native ternary model is still the 2.4B BitNet b1.58 2B4T, and its "100B on a CPU" figure is a runtime capability claim with no released checkpoint (microsoft/BitNet). Prism ML's Ternary Bonsai 2 27B claims 98.2% retention at 1.76 bits per weight, but it is derived from Qwen3.8 27B and does not disclose its training method (PrismML), and DeepGrove's Maple-Preview 20B-A1B speed claims are unverified (AI Weekly). Microsoft's bitnet.cpp measures 1.37x to 6.17x speedups and 55 to 82% energy reductions on CPUs, but Intel's independent BITCOS work finds bitnet.cpp does not reach roofline and that optimized ternary kernels beat well-tuned 2-bit kernels by only 1.0x to 1.3x end to end (arXiv 2609.16338). Because no GPU has a ternary datapath, ternary's "multiplier-free" advantage only pays on custom silicon.

The quality risks are set by theory and practice together. The precision scaling law of Kumar et al. puts compute-optimal pretraining precision at ~7 to 8 bits, predicts that going below ~4 bits needs models more than 4x larger for equal loss, and finds that overtrained models are more sensitive to post-training quantization (arXiv 2411.04330). In practice, 2-bit PTQ is catastrophic for reasoning models: Qwen3-32B fell from 70.0 to 16.67 on AIME 2026, mean reasoning length grew ~6.8x (1,792 to 12,222 tokens), and raw 2-bit end-to-end task speed was 0.23x of FP16 (arXiv 2606.02011). That result is the clearest example of per-token savings turning into per-task losses.

KV quantization is more mature. FP8 KV halves memory and lowers the decode latency slope to 54% of BF16, at a cost of 128K NIAH 89% vs 91% and up to 0.7 to 2 points on reasoning (vLLM). NVFP4 KV is ~50% smaller than FP8 at under 1% loss by NVIDIA's measurement (NVIDIA Research), and on B200, W4A16 weights plus NVFP4 KV raised Qwen3-32B's maximum batch at 32K from 9 to 61 (arXiv 2606.02011). Weight sparsity, by contrast, is a minor lever. 2:4 Sparse Llama 3.1 8B recovers ~98% of accuracy but gives only 1.1x to 1.3x end to end alone, and 2x to 3x single-stream latency when combined with 4-bit weights (Red Hat). At a fixed 8x compression budget, "4-bit plus 50% sparsity" beats either method pushed alone, but the quality losses add (arXiv 2608.24070), and ternary checkpoints are already 30 to 52% zeros, so sparsity stacked on ternary has little headroom (arXiv 2609.16338).

Deep dive: the low-bit numerics note, with every finding, inference and gap.

MoE is a ~7x lever at equal quality, realized only at hyperscale batch

MoE at ~3% activation

~7x efficiency leverage; 5.2x vs naive TP serving

Quality cost
None by construction; ~1.3x less knowledge per total parameter
Overlaps with
Batching, low-bit memory
Evidence
Academic law, open reproduction

Activation sparsity / early exit

1.5-1.8x at batch 1; ~1x at scale

Quality cost
Minimal
Overlaps with
MoE, distillation
Evidence
Academic

Frontier open models converged on 3 to 5% active parameters: Kimi K2.6 at 1T total and 32B active (Artificial Analysis), DeepSeek V4-Pro at 1.6T and 49B (aman.ai), gpt-oss-120b at 116.8B and 5.13B (arXiv 2508.10925), and Qwen3-Next at 80B and ~3B with 512 experts (VentureBeat). Kimi K3 reportedly reaches 2.8T with 16 of 896 experts active, but that figure comes from a secondary blog and is unverified (victorgrosser.nl). The relevant number is efficiency leverage at equal loss, not the total-to-active ratio. Ant Group's scaling laws, fit on more than 300 models up to 28B, predict efficiency leverage above 7x at 3.1% activation, rising with compute budget, and in validation a 0.85B-active MoE matched a 6.1B dense model on the same 1T tokens (arXiv 2507.17702). Returns are sublinear: halving activation again buys far less than 2x, and no law has been validated at frontier compute or below 2% activation.

Serving reality is the catch. SGLang's open reproduction served DeepSeek-V3 at ~$0.20 per million output tokens on 96 H100s, but only with wide expert parallelism, disaggregated prefill and decode and load balancing, which together ran up to 5.2x faster at decode than naive TP16 (LMSYS). By derived roofline arithmetic, an expert needs ~150 tokens to be compute-bound on BF16, so a 384-expert top-8 model needs about 7,000 concurrent tokens per expert-parallel group; below that, decode pays for total weight bytes rather than active FLOPs. MoE compounds well with low-bit experts, but with a knowledge cost. Allen-Zhu and Li measure ~2 bits of knowledge per parameter, with no loss at int8 but more than 2x loss at int4, while a 32-expert MoE stores ~1.3x less per total parameter while using ~9% of parameters at inference (HF Papers). Those are the quantitative overlaps between quantization and sparsity on knowledge capacity.

Activation sparsity and adaptive depth are small levers at datacenter scale. TEAL reaches 40 to 50% training-free activation sparsity for 1.53x to 1.8x at batch 1 (arXiv 2408.14690), and TurboSparse reaches ~90% FFN sparsity with 4.64x on a consumer GPU and 22x on mobile (arXiv 2406.05955), but both erode at large batch because the union of active neurons across a batch approaches dense. Early exit is fading: under an oracle MMLU tolerance, Llama3-8B can skip only 0.18% of layers against 4.69% for Llama2-7B (alphaXiv 2603.23701). Mixture-of-depths showed isoFLOP training wins only at 3B scale or below, with unproven serving benefits (Graphcore Research).

Deep dive: the moe and conditional compute note, with every finding, inference and gap.

Distillation and capability density: a fast trend against a hard knowledge floor

Distillation / density

~10x/yr fewer parameters at fixed benchmark

Quality cost
Brittleness, hallucination, 2-4x tokens
Overlaps with
MoE, routing
Evidence
Independent (Epoch, AA) plus academic

Capability per parameter has risen about 10x per year on saturating benchmarks, with the densing law finding maximum density doubling every ~3.3 to 3.5 months (arXiv 2412.04315). Distillation drives much of the 13x/yr price trend: DeepSeek-R1-Distill-Qwen-32B reached 72.6% on AIME 2024 against 47.0% for direct RL on the same base (arXiv 2501.12948). The limits are real, though. Post-hoc compression often loses density: pruned-and-distilled models such as Minitron-4B and Llama-3.2-1B/3B ended up less dense than their parents, and only Gemma-2-9B beat its source (arXiv 2412.04315). Apple's distillation scaling laws show distillation beats pretraining only when the teacher's cost is amortized, with the benefit stopping at a capacity gap (arXiv 2502.08606). Small 2026 models lack knowledge and spend tokens: Qwen3.5-9B scores 14.7% on AA-Omniscience with an 82% hallucination rate and uses 2x to 4x more output tokens than frontier models (Artificial Analysis). Order matters too, since quantizing before pruning or distilling pushed perplexity from ~3.2 to between 34 and 53 (arXiv 2511.19495). The implication is that reaching frontier quality with far fewer active parameters requires either sparse storage (MoE) or moving long-tail knowledge out of the weights into retrieval and tools, because the ~2 bits per parameter floor rules out holding frontier factual recall in a single-digit-billion dense model.

Deep dive: the distillation and cost trends note, with every finding, inference and gap.

Attention and KV: ~50x at a million tokens, ~1x to 2x at 8K

Compressed / hybrid attention

6-7x decode at 128K; ~50x KV at 1M

Quality cost
Multi-hop risk at scale (MiniMax)
Overlaps with
KV quantization; prefix caching
Evidence
Vendor reports, price evidence

DeepSeek-V4 is the strongest measured data point. It combines 4x-compressed KV with a lightning-indexer top-k (CSA), 128x-compressed dense attention (HCA) and a sliding window, and at 1M context V4-Flash needs ~10% of V3.2's per-token FLOPs and ~7% of its KV, about 2% of the KV of a BF16 GQA-8 baseline (HF DeepSeek-V4). These are vendor figures, but the previous step has market evidence behind it: DSA in V3.2 claimed 3.5x cheaper prefill and 6 to 7x cheaper decode at 128K with benchmark parity, and DeepSeek cut output prices 75% (The Decoder).

Hybrids with about three linear layers per full-attention layer form the other camp. Kimi Linear cut time per output token at 1M from 11.48 ms to 1.84 ms, though only 2.3x at batch 1, because most of the gain comes from bigger batches (alphaXiv 2510.26692). Nemotron-H 56B reached 2.4x the per-GPU output throughput of Llama-3.1-70B at 64K input (arXiv 2504.03624), Nemotron 3 Nano reached 3.3x Qwen3-30B-A3B at nearly equal active parameters, which isolates the attention contribution (HF/NVIDIA), and Olmo Hybrid 7B led Olmo 3 on RULER at 64K, 85.0 vs 70.9 (AI2). The main caution comes from MiniMax, whose Lightning-attention hybrid matched full attention at small scale but showed "clear deficits in complex, multi-hop reasoning tasks" at larger scale. MiniMax returned to full attention for M2 and noted that prefix caching and speculative decoding were immature for linear layers (MiniMax). These gains mostly stack with MoE because they hit a different term, but within the KV family they overlap, since hybrid layers, sparse selection and KV quantization all chase the same bytes. They also fade at short context: vLLM finds KV reads dominate decode only beyond about 7K tokens (vLLM).

Deep dive: the attention and kv cache note, with every finding, inference and gap.

Decoding: speculation buys latency, diffusion buys speed but not frontier quality

Speculation / MTP

1.38x at batch 64; ~3x at 70-140 tok/s/user

Quality cost
Lossless
Overlaps with
Batching, low-bit, diffusion
Evidence
Independent plus vendor

Diffusion LLMs

5-15x per-user speed; ~1-2x cost

Quality cost
10-20 pts on reasoning (same lab)
Overlaps with
Speculation
Evidence
Mostly vendor, self-reported

Speculative decoding and MTP are mainly latency levers. EAGLE-3 reaches up to 6.5x at batch 1 but 1.38x at batch 64 (arXiv 2503.01840). DeepSeek's DSpark raised V4 per-user speed 57 to 85% at matched aggregate throughput on live traffic, an interactivity gain rather than more tokens per GPU (vendor result, reported by Analytics India Magazine). On energy, EAGLE-3 at batch 1 saves 2.1x to 2.5x on 8B to 13B models but only 1.34x on Llama-70B, and classic draft models can increase energy, with factors of 0.63 to 0.80 (arXiv 2602.09113). InferenceX's 21x cost gap from switching MTP on (GB300, DeepSeek-R1 at 150 tok/s/user, $2.35/M off vs $0.11/M on) mostly reflects how badly underused the GPU is without speculation at that speed; at 70 to 140 tok/s/user the gain is about 3x (Silicon Report).

Diffusion LLMs deliver ~1,000 to 1,500 tok/s per user but have not shown per-token cost wins at serving batch. Same-lab comparisons show a quality gap: Gemini Diffusion trails Gemini 2.0 Flash-Lite on GPQA Diamond 40.4 vs 56.5 (DeepMind), and DiffusionGemma trails Gemma 4 on AIME 2026, 69.1 vs 88.3 (Decrypt). Inception's Mercury 3 parity claims against Haiku-class models are self-reported with no independent testing, and its $0.70/M output list price is only about 35% below rivals (Let's Data Science). LLaDA2.0's 1.9x speedup over autoregressive decoding is explicitly a small-batch result (LMSYS). Diffusion and speculation exploit the same idle compute, so they should not be multiplied together.

Deep dive: the decoding and test-time compute note, with every finding, inference and gap.

Tokens per task: the lever moving the wrong way

Reasoning efficiency

~2x (research); 4.4x spread at frontier

Quality cost
±3 pts
Overlaps with
Multiplies with per-token levers
Evidence
Independent (AA) plus academic

Routing / effort control

~8x cost for -4 pts

Quality cost
Bounded by hard-task share
Overlaps with
Density
Evidence
Independent (AA)

On the Artificial Analysis Index v5 (October 2026), median output per task rose from ~35k to ~59k tokens, and every model measured generated more tokens than in the previous version. Four agentic evaluations carry 55% of the index weight but produce ~90% of its tokens. Cost per task among top models spans more than 13x, and GPT-6.1 Sol (max) lands within 4 points of Opus 5.5 (max) at about one-eighth the cost per task (Artificial Analysis). Tokens-per-task levers are the only ones that multiply cleanly with every per-token lever, yet market pressure is pushing tokens per task up.

Research methods cut reasoning tokens by about half, mostly on small models. AdaptThink cut length 53% while gaining 2.4 points on a 1.5B model (arXiv 2505.13417), ThinkPrune halved length for a ~2% loss on AIME (arXiv 2504.01296), and selective latent thinking cut 58% at a 2.8-point cost, although naive latent compression on Qwen3-4B dropped GSM-Hard from 54.7% to 17.3% (alphaXiv 2605.25745). A small model plus test-time compute is a legitimate substitute for parameters on medium-difficulty problems: FLOPs-matched test-time compute beat a 14x larger model where the small model had non-trivial success, and lost on the hardest problems (arXiv 2408.03314). Routing is bounded by the share of hard tasks. If 30% of tasks need the flagship, no cascade can exceed ~3.3x however cheap the student becomes (derived).

Deep dive: the decoding and test-time compute note, with every finding, inference and gap.

Serving software: 1.2x to 2.8x in six months on the same silicon

Serving software

1.2-2.8x on same GB300 in 6 months

Quality cost
None
Overlaps with
MoE, MTP
Evidence
Audited (MLPerf)

On identical GB300 NVL72 hardware, audited MLPerf per-GPU throughput rose from v5.1 to v6.0 by 2.77x for DeepSeek-R1 Server but only 1.21x for Llama 3.1 405B Offline (StorageReview). NVIDIA attributes the gains to kernel fusion, Dynamo disaggregation, wide expert parallelism, MTP and KV-aware routing, but publishes no per-technique split. Software gains are largest for new sparse and reasoning-heavy architectures and smallest for mature dense models. Independent InferenceX data implies GB200 cost per token improved roughly 2.5x to 5x within a year at high interactivity, but only about 1.3x at low interactivity (SemiAnalysis). A new co-designed architecture should therefore expect a launch-time software penalty that closes over 6 to 12 months.

Deep dive: the hardware and serving note, with every finding, inference and gap.

Hardware: GPUs compound; exotic silicon mostly buys speed, not cost

GPU generation

3.1x (H100 to B200); 2-5.4x (GB300 to Rubin) per MW

Quality cost
None
Overlaps with
Embedded FP4/LUT
Evidence
Independent (InferenceX)

SRAM / wafer-scale

3-6x per-user speed; cost "comparable"

Quality cost
None
Overlaps with
Hardwired
Evidence
AA speeds, vendor cost

Hardwired weights

~50-74x per-user speed; ~1.4x cost at R1 scale

Quality cost
Aggressive 4-bit, unmeasured
Overlaps with
Low-bit, MoE (30 tape-outs)
Evidence
Vendor plus EE Times hands-on

Analog in-memory

~10x power (simulated)

Quality cost
Needs noise-aware training
Overlaps with
Low-bit
Evidence
Simulation only

Photonics

None published

Quality cost
Unknown
Overlaps with
Attention alternatives
Evidence
Vendor demo only

On independent InferenceX data, B200 delivers ~3.1x the tokens per megawatt of H100 on GPT-OSS-120B, and GB200 NVL72 with FP4 delivers ~8x a single-node H200 with FP8 on DeepSeek-R1 (Silicon Report). Vera Rubin NVL72 reaches 2x GB300 up to 100 tok/s/user, ~4x near 200 and 5.4x at 300, where GB300 is barely viable. Cost improves less than energy, because owner TCO is $3.57 per GPU-hour for Rubin against $2.36 for GB300 and $1.84 for GB200. CoreWeave's 5.4x performance per MW over GB200 is an operator claim that SemiAnalysis did not verify (SemiAnalysis). Rubin adds a ~3.125-bit LUT weight format and 2.8x Blackwell Ultra's memory bandwidth, and its sparsity-labeled NVFP4 inference rating of 50 PFLOPS is 12.5x its 4 PFLOPS BF16 rating (Tom's Hardware). From H100 to Rubin the plausible gain is ~10x to 30x in tokens per MW at moderate interactivity, and less in dollars.

SRAM-centric and wafer-scale chips win on per-user speed. On 8B-class models, Cerebras runs ~2,000 tok/s/user, SambaNova ~900 and Groq ~600, against ~350 on Blackwell (EE Times). Cerebras's 981 tok/s on a trillion-parameter model comes with per-token cost it calls only "comparable to GPU providers" (Pulse), and Harvard's independent measurements found Cerebras, SambaNova and Gaudi idle at 10 to 60% higher power than GPUs (arXiv 2604.10852). TPU v7 has no independent per-chip inference benchmark (Silicon Report).

Hardwired weights are the most dramatic data point. Taalas HC1 etches Llama-3.1-8B into mask ROM at ~4 bits on an 815 mm² TSMC N6 die drawing ~250 W. EE Times observed 15,000+ tok/s/user hands-on, the company claims ~17,000 and ~73x H200 throughput, and no accuracy figures have been published. Taalas's own frontier-scale cost claim is 7.6 cents per million tokens for DeepSeek-R1 on ~30 chips (EE Times), which is only ~1.4x below GB300 with MTP ($0.11/M). Hardwiring is therefore a huge speed multiplier but a modest cost multiplier at frontier scale, because ~30 tape-outs per model must be amortized and each model change needs a ~2-month respin. AMD agreed to acquire Taalas on August 6, 2026 (SiliconANGLE). Analog in-memory compute is still at simulation grade: Sagence's claimed Llama2-70B system at 666,000 tok/s and 59 kW, against a 624 kW H100 system, is simulated (IEEE Spectrum), and IBM and ETH Zurich show that off-the-shelf LLMs cannot even reach W4A8 quality on noisy analog hardware without a dedicated training recipe (Semiconductor Engineering). Photonics reached its first LLM demo on November 17, 2026, when Q.ANT replaced one attention layer with optical Fourier transforms, but it published no size, accuracy, throughput or energy figures (The Quantum Insider).

Deep dive: the hardware and serving note, with every finding, inference and gap.

Lever map and evidence grade

LeverBest realized gain (Nov 2026)Quality costOverlaps withEvidence grade
FP4 QAT (W4A4)3.6x weight bytes; 1.5-3x end to end~1% with QAT/native; PTQ worse when overtrainedHardware generation, MoE memoryVendor research, production adoption
Ternary / 2-bit1.0-1.3x over 2-bit kernels on CPU/GPU2-bit PTQ collapses reasoning; ternary at frontier scale unprovenSparsity (30-52% zeros), custom siliconMostly vendor; Intel independent
KV quantizationFP8 2x; NVFP4 ~2x more0.7-2 pts (FP8); <1% (NVFP4, vendor)Attention compressionIndependent (vLLM) plus vendor
2:4 weight sparsity1.1-1.3x alone~2-3%Quantization (quality adds)Vendor plus academic
MoE at ~3% activation~7x efficiency leverage; 5.2x vs naive TP servingNone by construction; ~1.3x less knowledge per total parameterBatching, low-bit memoryAcademic law, open reproduction
Activation sparsity / early exit1.5-1.8x at batch 1; ~1x at scaleMinimalMoE, distillationAcademic
Distillation / density~10x/yr fewer parameters at fixed benchmarkBrittleness, hallucination, 2-4x tokensMoE, routingIndependent (Epoch, AA) plus academic
Compressed / hybrid attention6-7x decode at 128K; ~50x KV at 1MMulti-hop risk at scale (MiniMax)KV quantization; prefix cachingVendor reports, price evidence
Speculation / MTP1.38x at batch 64; ~3x at 70-140 tok/s/userLosslessBatching, low-bit, diffusionIndependent plus vendor
Diffusion LLMs5-15x per-user speed; ~1-2x cost10-20 pts on reasoning (same lab)SpeculationMostly vendor, self-reported
Reasoning efficiency~2x (research); 4.4x spread at frontier±3 ptsMultiplies with per-token leversIndependent (AA) plus academic
Routing / effort control~8x cost for -4 ptsBounded by hard-task shareDensityIndependent (AA)
Serving software1.2-2.8x on same GB300 in 6 monthsNoneMoE, MTPAudited (MLPerf)
GPU generation3.1x (H100 to B200); 2-5.4x (GB300 to Rubin) per MWNoneEmbedded FP4/LUTIndependent (InferenceX)
SRAM / wafer-scale3-6x per-user speed; cost "comparable"NoneHardwiredAA speeds, vendor cost
Hardwired weights~50-74x per-user speed; ~1.4x cost at R1 scaleAggressive 4-bit, unmeasuredLow-bit, MoE (30 tape-outs)Vendor plus EE Times hands-on
Analog in-memory~10x power (simulated)Needs noise-aware trainingLow-bitSimulation only
PhotonicsNone publishedUnknownAttention alternativesVendor demo only

The multiplier budget: a million on paper, about a thousand in practice

The tables below give each lever's standalone multiplier (low, central, high) and then combine them in four steps. Levers in the same family are grouped first. The model-side groups (weights versus KV) are then combined over their shares of bytes moved rather than multiplied together. Each idle-compute effect and each hardware-embedded effect is counted once. Finally, the group subtotals are multiplied, because they act on different factors of cost per task. A useful check is the ceiling the physics allows (derived). An MoE with ~20B active parameters in FP4, batch-amortized weights and sparse KV reads does ~40 GFLOP of weight math per token, about 20x less than the baseline's 0.8 TFLOP, and at H100 coefficients its energy floor is roughly 0.03 to 0.05 J per token, about 200x below B2024's ~9 J, before any hardware improvement. Realized model-side gains land well below that ceiling because of kernel efficiency, expert-parallel communication, prefill and idle power.

Multiplied naively the levers promise a million; overlap-adjusted, about a thousand

Each lever's standalone multiplier (low to high, dot at central), grouped by the term it attacks, then the overlap-adjusted subtotals and totals. Hollow dots are the naive product of every central row.

Model side, per token
FP4 to ~3-bit LUT or ternary expertsRubin LUT is ~1.36x fewer bytes than MXFP4
MoE from ~3% to ~1.5% activationSublinear efficiency leverage
Frontier distillation plus retrieval offload of knowledgeKnowledge floor; brittleness risk
Activation sparsity / early exitNear zero at scale
KV from FP8 to FP4 / 2-bitKV already ~2% of GQA baseline, so its share is small
Hybrid linear layers on top of compressed-sparse attentionSame KV term
Model-side per token (overlap-adjusted)Naive central ~11
Utilization: serving and idle compute
Serving maturity on new architecture plus DSpark-class draftingMTP already in baseline
Hardware
GB300 to Rubin (cost)Energy central ~3x
Hardwired / SRAM tier for routed student (cost)Energy central ~2x; up to ~10x on routed share
Tokens per task
Reasoning-token efficiency (cost-aware RL, adaptive thinking)Frontier spread 4.4x is the existence proof
Routing cascade and effort controlCapped by hard-task share; overlaps the row above
Tokens-per-task subtotal (overlap-adjusted)
All levers together
Naive product of all central rows
Overlap-adjusted total, cost per solved taskHigh is improbable: every lever must land
Overlap-adjusted total, energy per solved task
Times cheaper per solved task (log scale)

Every compounding factor here is the survey's inference: no source has measured these levers together at frontier scale.

Show the numbers

Table B: against B2026 (DeepSeek-V4-Pro-class on GB300 NVL72, late-2026 software), fixed late-2026-frontier quality

Table B: against B2026 (DeepSeek-V4-Pro-class on GB300 NVL72, late-2026 software), fixed late-2026-frontier quality
LeverLowCentralHighOverlap treatment
FP4 to ~3-bit LUT or ternary experts1x1.3x2xRubin LUT is ~1.36x fewer bytes than MXFP4
MoE from ~3% to ~1.5% activation1.1x1.4x1.8xSublinear efficiency leverage
Frontier distillation plus retrieval offload of knowledge1.2x2x4xKnowledge floor; brittleness risk
Activation sparsity / early exit1x1.05x1.2xNear zero at scale
KV from FP8 to FP4 / 2-bit1.3x2x3xKV already ~2% of GQA baseline, so its share is small
Hybrid linear layers on top of compressed-sparse attention1.1x1.5x2.5xSame KV term
Model-side per token (overlap-adjusted)1.2x2x5xNaive central ~11
Serving maturity on new architecture plus DSpark-class drafting1.2x1.7x3xMTP already in baseline
GB300 to Rubin (cost)1.5x2x4xEnergy central ~3x
Hardwired / SRAM tier for routed student (cost)1x1.3x3xEnergy central ~2x; up to ~10x on routed share
Reasoning-token efficiency (cost-aware RL, adaptive thinking)1.3x2x4xFrontier spread 4.4x is the existence proof
Routing cascade and effort control1.5x3x6xCapped by hard-task share; overlaps the row above
Tokens-per-task subtotal (overlap-adjusted)2x4x10x
Naive product of all central rows420x
Overlap-adjusted total, cost per solved task4x35x1,800xHigh is improbable: every lever must land
Overlap-adjusted total, energy per solved task6x80x5,000x

Table A: against B2024 (dense ~405B, BF16, H100, 2024 software), fixed 2024-frontier quality, ~64K agentic mix

Table A: against B2024 (dense ~405B, BF16, H100, 2024 software), fixed 2024-frontier quality, ~64K agentic mix
LeverLowCentralHighOverlap treatment
FP4 weights and activations (QAT)1.5x2.5x3.6xWeight group
Sub-4-bit (LUT / ternary) beyond FP41x1.2x2xWeight group; GPU kernels cap it
MoE at ~3% activation3x6x10xWeight group; needs wide-EP batch
Capability density (data, overtraining, distillation)4x10x30xWeight group; overlaps MoE on active parameters
Activation sparsity / adaptive depth1x1.1x1.5xWeight group; erodes at batch
Weight-group subtotal (overlap-adjusted)15x60x200xNaive central ~200
Attention architecture (MLA, hybrid, compressed-sparse)3x8x20xKV group
KV quantization (BF16 to FP8/FP4)2x3x4xKV group, same term
KV-group subtotal (overlap-adjusted)4x12x40xNaive central ~24
Model-side per token (byte-share combination plus batch synergy)10x30x80xShares ~65% weights / ~35% KV at capacity-limited batch
Serving software and scheduling1.5x2.5x5xUtilization group
Speculation / MTP1.2x1.6x3xUtilization group; same idle compute as batching
Diffusion decoding1x1x2xNot credited (overlaps speculation; quality gap)
Utilization subtotal1.5x2.5x4x
GPU generation H100 to Rubin (cost)3x5x8xEnergy central ~10x; FP4 counted above
Specialized silicon for routed share (cost)1x1.2x2xEnergy central ~2x
Tokens per task (reasoning efficiency, routing)0.7x1.5x3xB2024 is a non-reasoning model, so small reasoning replacements can spend more tokens
Naive product of all central rows1,000,000x
Overlap-adjusted total, cost per solved task30x700x15,000xHigh requires every lever at its best
Overlap-adjusted total, energy per solved task60x2,000x40,000xHardware energy improves faster than $/token

Table A: against B2024 (dense ~405B, BF16, H100, 2024 software), fixed 2024-frontier quality, ~64K agentic mix#

LeverLowCentralHighOverlap treatment
FP4 weights and activations (QAT)1.52.53.6Weight group
Sub-4-bit (LUT / ternary) beyond FP41.01.22.0Weight group; GPU kernels cap it
MoE at ~3% activation3610Weight group; needs wide-EP batch
Capability density (data, overtraining, distillation)41030Weight group; overlaps MoE on active parameters
Activation sparsity / adaptive depth1.01.11.5Weight group; erodes at batch
Weight-group subtotal (overlap-adjusted)1560200Naive central ~200
Attention architecture (MLA, hybrid, compressed-sparse)3820KV group
KV quantization (BF16 to FP8/FP4)234KV group, same term
KV-group subtotal (overlap-adjusted)41240Naive central ~24
Model-side per token (byte-share combination plus batch synergy)103080Shares ~65% weights / ~35% KV at capacity-limited batch
Serving software and scheduling1.52.55Utilization group
Speculation / MTP1.21.63Utilization group; same idle compute as batching
Diffusion decoding1.01.02Not credited (overlaps speculation; quality gap)
Utilization subtotal1.52.54
GPU generation H100 to Rubin (cost)358Energy central ~10x; FP4 counted above
Specialized silicon for routed share (cost)1.01.22Energy central ~2x
Tokens per task (reasoning efficiency, routing)0.71.53B2024 is a non-reasoning model, so small reasoning replacements can spend more tokens
Naive product of all central rows~106
Overlap-adjusted total, cost per solved task~30x~700x~15,000xHigh requires every lever at its best
Overlap-adjusted total, energy per solved task~60x~2,000x~40,000xHardware energy improves faster than $/token

Two cross-checks support Table A. The low case (~30x) matches the most conservative market trend over 2.4 years (~40x, open-weight Pareto), and the central case sits a few times above the headline trend (~250x to 470x), which is the expected gap between the market's average deployment and a fleet that uses Rubin, routing and token-efficient RL together. A derived roofline gives the same order of magnitude. B2024 moves ~92 GB per token at 64K, which bounds 16 H100s to a few hundred tokens per second of aggregate decode and puts its cost in the tens of dollars per million tokens. InferenceX measures DeepSeek-V4-Pro at $0.12/M on GB300 at much higher quality, a per-token gap of very roughly 100x to 250x before any tokens-per-task lever, consistent with the central row. 1000x against 2024 is credible today in energy and within about a year in dollars, and most of it is what the market already does.

Table B: against B2026 (DeepSeek-V4-Pro-class on GB300 NVL72, late-2026 software), fixed late-2026-frontier quality#

LeverLowCentralHighOverlap treatment
FP4 to ~3-bit LUT or ternary experts1.01.32.0Rubin LUT is ~1.36x fewer bytes than MXFP4
MoE from ~3% to ~1.5% activation1.11.41.8Sublinear efficiency leverage
Frontier distillation plus retrieval offload of knowledge1.22.04.0Knowledge floor; brittleness risk
Activation sparsity / early exit1.01.051.2Near zero at scale
KV from FP8 to FP4 / 2-bit1.32.03.0KV already ~2% of GQA baseline, so its share is small
Hybrid linear layers on top of compressed-sparse attention1.11.52.5Same KV term
Model-side per token (overlap-adjusted)1.22.05.0Naive central ~11
Serving maturity on new architecture plus DSpark-class drafting1.21.73.0MTP already in baseline
GB300 to Rubin (cost)1.52.04.0Energy central ~3x
Hardwired / SRAM tier for routed student (cost)1.01.33.0Energy central ~2x; up to ~10x on routed share
Reasoning-token efficiency (cost-aware RL, adaptive thinking)1.32.04.0Frontier spread 4.4x is the existence proof
Routing cascade and effort control1.53.06.0Capped by hard-task share; overlaps the row above
Tokens-per-task subtotal (overlap-adjusted)2410
Naive product of all central rows~420x
Overlap-adjusted total, cost per solved task~4x~35x~1,800xHigh is improbable: every lever must land
Overlap-adjusted total, energy per solved task~6x~80x~5,000x

Against the late-2026 frontier, model-side headroom per token is only about 2x, because DeepSeek-V4 and its peers have already collected MoE, FP4 and most of the KV savings. What remains is spread across hardware (about 2x from Rubin), serving maturity, a specialized-silicon tier and, above all, tokens per task, which offers the largest remaining factor (about 4x) and is the only lever family on which the market is currently moving backward. Trend arithmetic converts these numbers into time. At 10x to 13x per year, 35x is about 1.4 to 1.5 years of progress. On trend, late-2026 frontier capability becomes ~1000x cheaper around 2029 with or without this program, so the program's value is reaching each step earlier and owning the cost structure. It does not multiply the trend, because the trend is other labs adopting the same levers.