A roughly 1000x cut in cost and energy per solved task, at fixed quality, is credible against a 2024 baseline (a dense ~405B transformer in BF16 on H100s) but not, on current evidence, against the late-2026 frontier. Against 2024, the market has already delivered somewhere between ~40x and ~470x at constant capability (Epoch measures ~13x per year), and a deliberate full-stack program built only from techniques that already work in production reaches an overlap-adjusted central estimate of ~700x in dollars and ~2,000x in energy, so 1000x against 2024 is mostly a 2027 to 2029 market outcome rather than a research breakthrough. Against the harder baseline, a late-2026 DeepSeek-V4-class sparse MoE with FP4 experts, compressed attention and multi-token prediction on GB300 racks, the same program adds a central ~35x in dollars and ~80x in energy (range ~4x to above 1,000x, with the top end requiring every lever to land at once). That is worth about 1.5 years of the market trend: a well-run program puts a team ahead of the curve rather than multiplying it. The gap between these estimates and the naive product of each lever's best published figure (~106 against 2024, ~400x against 2026) is the central lesson, because most levers shrink the same bytes moved per token, the binding bottleneck rotates from KV cache to weights to arithmetic to communication, and speculation, diffusion and batching all spend the same idle compute. The most credible stack combines a fine-grained MoE at 1 to 2 percent activation trained natively in FP4, hybrid linear plus compressed-sparse attention with 4-bit KV, trained-in multi-token drafting, RL that prices tokens per task, and a routing cascade whose distilled student runs on hardwired or SRAM silicon while the flagship runs on Rubin-class GPUs with disaggregated serving. No absolute solution exists: the largest single risk is that tokens per task are rising faster than model-side levers cut them, and nobody has published a controlled measurement of how these levers compound.
Two baselines and one metric: dollars and joules per solved task#
The "1000x" claim only means something once the baseline, the unit and the quality bar are fixed. This report measures dollars and joules per solved task at a fixed quality level on a fixed task suite, holding constant a per-user speed target (interactivity), a context-length distribution, and a power boundary that includes prefill GPUs, decode GPUs and idle power. Per-token figures mislead in both directions. At near-identical Artificial Analysis Intelligence Index v5 scores, Claude Opus 5.5 (max) emits ~119k output tokens per task while GPT-6 Astra emits ~27k, a 4.4x gap, and Opus 5.5's 20% per-token price cut was fully eaten by 1.6x more tokens than Opus 5 (Artificial Analysis). Small reasoning models behave the same way: Qwen3.5-9B used ~260M output tokens to run the index against 86M for Qwen3.5-397B (Artificial Analysis). Interactivity matters just as much. On one vendor's hardware, derived energy per token ranges from ~0.36 J (GPT-OSS-120B on B200) to ~10 to 14 J (DeepSeek-R1-class at 300 to 350 tok/s/user on Rubin), a ~40x spread as large as a hardware generation (SemiAnalysis; Silicon Report). Power accounting is not even consistent across sources, since InferenceX counts decode-GPU watts only while CoreWeave's Rubin figures count prefill and decode (SemiAnalysis).
Baseline B2024 is a Llama-3.1-405B-class dense transformer in BF16 on H100s with 2024 serving software (continuous batching, no speculation), measured on a decode-heavy agentic and reasoning mix with ~64K mean context. The following arithmetic is derived from the public model configuration and is approximate. BF16 weights occupy ~810 GB, which does not fit on one 8-GPU node, so the baseline needs two nodes (16 GPUs, 1,280 GB of HBM). With 126 layers and 8 KV heads of dimension 128, the BF16 KV cache is ~0.5 MB per token, or ~34 GB per 64K-token sequence, so only ~13 to 14 such sequences fit beside the weights. Each generated token therefore moves ~92 GB from HBM (weights amortized over the batch plus that sequence's entire KV). At measured H100 coefficients of 11.68 pJ per HBM bit and 0.52 pJ per tensor-core FLOP (arXiv 2607.26571), that is roughly 8.6 J of data movement plus ~0.7 J of arithmetic per token: more than 90% of the energy goes to moving bytes, before counting static power. No clean market price exists for this baseline, since Artificial Analysis no longer tracks any provider for Llama 3.1 405B (Artificial Analysis). It has to be rebuilt on rented hardware, which is Epoch's method and lands within 30% of API prices (Epoch AI).
Baseline B2026 is DeepSeek-V4-Pro-class serving at late-2026 frontier quality. The model has 1.6T total and 49B active parameters, FP4 routed experts, FP8 KV, CSA/HCA compressed attention and multi-token prediction (HF DeepSeek-V4), and it runs on GB300 NVL72 racks with Dynamo or SGLang disaggregated wide-expert-parallel serving. InferenceX measures V4-Pro at $0.12 per million output tokens at 27 tok/s/user on GB300 (Silicon Report). On quality, open trillion-parameter MoEs (Kimi K2.6, MiMo V2.5 Pro, DeepSeek V4 Pro) scored 52 to 54 on the April 2026 index against 57 to 60 for the top proprietary models, at "between half to one-sixth of the price" (Artificial Analysis). B2024 is the baseline the question proposes; B2026 is the one that tests whether a program does anything the market will not.
What the market trend already delivers#
| Estimate | Rate at constant capability | Window and scope | Source type |
|---|---|---|---|
| Epoch, "Plunging price of thought" (Sep 2026) | ~47% per quarter, ~13x/yr (math 16-19x, games 7-10x) | 2023 to 2026, five benchmarks | Independent analyst |
| Epoch, same report, at SOTA | ~75x/yr when a level first becomes SOTA, slowing to ~4.7x/yr two years later | Same | Independent analyst |
| Gundlach et al., "Price of Progress" (Nov 2025) | Frontier 5-10x/yr; GPQA Pareto 5.32x; open-weight 4.60x; ~3x/yr algorithmic after removing hardware | Apr 2024 to late 2025 | Academic (MIT/Epoch data) |
| a16z "LLMflation" (Nov 2024) | ~10x/yr; 1000x in 3 years at MMLU ≥42 | 2021 to 2024 | Investor analysis |
| Densing law (Tsinghua) | Capability per parameter doubles every ~3.3-3.5 months | Feb 2023 to Apr 2025, open base models | Academic |
Sources: Epoch AI, arXiv 2511.23455, a16z, arXiv 2412.04315.
Mid-2024 to November 2026 is about 2.4 years, so the headline trend implies ~250x to ~470x at constant capability, while the most conservative figure (open-weight Pareto at 4.6x/yr) implies only ~40x. The best-documented single case is ~75% on GPQA Diamond, which cost ~$0.30 per question with o3 in January 2025 and $0.0004 with GPT-5.6 Luna, a ~725x drop in under 18 months (Epoch AI). Three caveats matter for planning. These are prices, not costs, so margin compression inflates the headline, which is why Gundlach's hardware-adjusted algorithmic component is only ~3x/yr. Epoch also warns that distilled models score disproportionately well on benchmarks and seem more brittle, and that the decline may slow once models become too small for strong agentic work (Epoch AI). Finally, the trend is the industry adopting the same levers this report analyzes, so a program cannot count the trend and its own levers as separate multipliers.
Every lever attacks one of four terms, and the binding term rotates#
Cost per solved task factors into four terms: tokens per task, bytes and FLOPs per token (weight bytes, KV bytes and arithmetic), cost or energy per byte and per FLOP (the hardware), and utilization (batching, scheduling and idle time). The physics explains why overlap is the rule. On H100-class silicon, reading one 8-bit weight from HBM costs about 90x the two FLOPs it feeds, and the batch size at which amortized weight reads equal arithmetic energy is about 90 for 8-bit weights and about 45 for 4-bit (derived from arXiv 2607.26571). Above that batch a GPU becomes arithmetic-bound, except for KV reads, which grow with context and are never shared across sequences. Landauer's limit sits 108 to 109 below today's per-token energy, so physics is far from binding; the practical limits are picojoules per bit of wire and memory traffic.
Agentic workloads push hard on the KV term. A September 2026 study of SWE-bench Pro, Terminal-Bench 2.0, GAIA and similar suites found that decode is 91.0 to 98.6% of LLM time, mean contexts run 69 to 80K tokens (maxima 146 to 166K), and prefix-cache hit rates run 84.6 to 99.5%. With Qwen3.6-27B, raising concurrency from 15 to 20 collapsed the hit rate from 98.85% to 16.78% (alphaXiv 2605.26297). KV capacity therefore sets batch size, and batch size sets cost.
Bottleneck rotation explains why naive multiplication fails. In B2024 at 64K context, HBM capacity binds: per-token bytes come out to roughly the per-sequence KV times HBM/(HBM minus weights) (derived), so cutting KV is the first lever that matters. Compressing KV 10x to 50x lets batch grow until weight reads amortize. Arithmetic then binds, and MoE active-parameter cuts and FP4 tensor math take over. After that, communication and scheduling bind, which expert-parallel MoE serving shows clearly: DeepEP dispatch costs ~0.17 ms per layer, expert load balancing alone was worth 1.49x on prefill and 2.54x on decode, and two-batch overlap hurt decode by 27% at 32 tokens per device while helping by 25.5% at 256 (LMSYS). A lever multiplies the total only while its term binds, and the overall gain is capped by the next floor.
Six overlap families follow. In the bytes-per-token family, low-bit weights, MoE at small batch, on-chip or hardwired weights, and batching all compete for the same weight-movement term. In the idle-compute family, speculation, MTP, diffusion decoding and large batches all convert unused FLOPs in a memory-bound decode into tokens, so they should be counted once. In the active-parameter family, MoE efficiency leverage, distillation and capability density, early exit and activation sparsity all shrink the parameters touched per token at fixed quality. In the KV family, MLA, hybrid linear layers, sparse or compressed attention and KV quantization compete for the same term. Gains already embedded in hardware, such as FP4 and LUT weight formats, are part of Blackwell and Rubin generational numbers, so counting them again against a Rubin FP4 baseline double counts (SemiAnalysis). Finally, the tokens-per-task family (reasoning efficiency, effort control and routing) is the only one that multiplies cleanly with all per-token levers, because it acts on a different factor of cost per task.
What November 2026 actually measures, lever by lever#
Low-bit numerics: FP4 is solved, ternary is not#
FP4 is now a first-class inference and training format. NVIDIA's 12B hybrid Mamba-Transformer trained on 10T tokens in NVFP4, the longest publicly documented 4-bit pretraining run, held relative loss within 1% of FP8 in the stable phase and ~1.5% after decay at about 830 tokens per parameter, with ~16% of linear layers kept in BF16. Downstream scores were close but mixed (MMLU-Pro 62.62 vs 62.58, GSM8K 89.08 vs 92.27), with code showing the largest gap at 59.52 vs 56.67 (arXiv 2509.25149). Production adoption is broad: DeepSeek-V4 applies FP4 QAT to routed experts, GPT-OSS uses MXFP4 QAT for more than 90% of its parameters, and NVIDIA reports that NVFP4 PTQ of DeepSeek-R1 lands within ~1% of FP8 (AIME 2024 89 vs 91, a vendor measurement) (NVIDIA Research). The ceiling is clear but realization falls short of it. BF16 to NVFP4 (~4.5 effective bits) cuts weight bytes ~3.6x, Blackwell FP4 math is 4x (GB200) to 6x (GB300) over BF16, and NVIDIA cites ~3x FP8 peak inference throughput, yet published end-to-end gains are mostly 1.5x to 3x.
Ternary and 2-bit formats are where vendor claims and independent measurement diverge most. Microsoft's largest official native ternary model is still the 2.4B BitNet b1.58 2B4T, and its "100B on a CPU" figure is a runtime capability claim with no released checkpoint (microsoft/BitNet). Prism ML's Ternary Bonsai 2 27B claims 98.2% retention at 1.76 bits per weight, but it is derived from Qwen3.8 27B and does not disclose its training method (PrismML), and DeepGrove's Maple-Preview 20B-A1B speed claims are unverified (AI Weekly). Microsoft's bitnet.cpp measures 1.37x to 6.17x speedups and 55 to 82% energy reductions on CPUs, but Intel's independent BITCOS work finds bitnet.cpp does not reach roofline and that optimized ternary kernels beat well-tuned 2-bit kernels by only 1.0x to 1.3x end to end (arXiv 2609.16338). Because no GPU has a ternary datapath, ternary's "multiplier-free" advantage only pays on custom silicon.
The quality risks are set by theory and practice together. The precision scaling law of Kumar et al. puts compute-optimal pretraining precision at ~7 to 8 bits, predicts that going below ~4 bits needs models more than 4x larger for equal loss, and finds that overtrained models are more sensitive to post-training quantization (arXiv 2411.04330). In practice, 2-bit PTQ is catastrophic for reasoning models: Qwen3-32B fell from 70.0 to 16.67 on AIME 2026, mean reasoning length grew ~6.8x (1,792 to 12,222 tokens), and raw 2-bit end-to-end task speed was 0.23x of FP16 (arXiv 2606.02011). That result is the clearest example of per-token savings turning into per-task losses.
KV quantization is more mature. FP8 KV halves memory and lowers the decode latency slope to 54% of BF16, at a cost of 128K NIAH 89% vs 91% and up to 0.7 to 2 points on reasoning (vLLM). NVFP4 KV is ~50% smaller than FP8 at under 1% loss by NVIDIA's measurement (NVIDIA Research), and on B200, W4A16 weights plus NVFP4 KV raised Qwen3-32B's maximum batch at 32K from 9 to 61 (arXiv 2606.02011). Weight sparsity, by contrast, is a minor lever. 2:4 Sparse Llama 3.1 8B recovers ~98% of accuracy but gives only 1.1x to 1.3x end to end alone, and 2x to 3x single-stream latency when combined with 4-bit weights (Red Hat). At a fixed 8x compression budget, "4-bit plus 50% sparsity" beats either method pushed alone, but the quality losses add (arXiv 2608.24070), and ternary checkpoints are already 30 to 52% zeros, so sparsity stacked on ternary has little headroom (arXiv 2609.16338).
MoE is a ~7x lever at equal quality, realized only at hyperscale batch#
Frontier open models converged on 3 to 5% active parameters: Kimi K2.6 at 1T total and 32B active (Artificial Analysis), DeepSeek V4-Pro at 1.6T and 49B (aman.ai), gpt-oss-120b at 116.8B and 5.13B (arXiv 2508.10925), and Qwen3-Next at 80B and ~3B with 512 experts (VentureBeat). Kimi K3 reportedly reaches 2.8T with 16 of 896 experts active, but that figure comes from a secondary blog and is unverified (victorgrosser.nl). The relevant number is efficiency leverage at equal loss, not the total-to-active ratio. Ant Group's scaling laws, fit on more than 300 models up to 28B, predict efficiency leverage above 7x at 3.1% activation, rising with compute budget, and in validation a 0.85B-active MoE matched a 6.1B dense model on the same 1T tokens (arXiv 2507.17702). Returns are sublinear: halving activation again buys far less than 2x, and no law has been validated at frontier compute or below 2% activation.
Serving reality is the catch. SGLang's open reproduction served DeepSeek-V3 at ~$0.20 per million output tokens on 96 H100s, but only with wide expert parallelism, disaggregated prefill and decode and load balancing, which together ran up to 5.2x faster at decode than naive TP16 (LMSYS). By derived roofline arithmetic, an expert needs ~150 tokens to be compute-bound on BF16, so a 384-expert top-8 model needs about 7,000 concurrent tokens per expert-parallel group; below that, decode pays for total weight bytes rather than active FLOPs. MoE compounds well with low-bit experts, but with a knowledge cost. Allen-Zhu and Li measure ~2 bits of knowledge per parameter, with no loss at int8 but more than 2x loss at int4, while a 32-expert MoE stores ~1.3x less per total parameter while using ~9% of parameters at inference (HF Papers). Those are the quantitative overlaps between quantization and sparsity on knowledge capacity.
Activation sparsity and adaptive depth are small levers at datacenter scale. TEAL reaches 40 to 50% training-free activation sparsity for 1.53x to 1.8x at batch 1 (arXiv 2408.14690), and TurboSparse reaches ~90% FFN sparsity with 4.64x on a consumer GPU and 22x on mobile (arXiv 2406.05955), but both erode at large batch because the union of active neurons across a batch approaches dense. Early exit is fading: under an oracle MMLU tolerance, Llama3-8B can skip only 0.18% of layers against 4.69% for Llama2-7B (alphaXiv 2603.23701). Mixture-of-depths showed isoFLOP training wins only at 3B scale or below, with unproven serving benefits (Graphcore Research).
Distillation and capability density: a fast trend against a hard knowledge floor#
Capability per parameter has risen about 10x per year on saturating benchmarks, with the densing law finding maximum density doubling every ~3.3 to 3.5 months (arXiv 2412.04315). Distillation drives much of the 13x/yr price trend: DeepSeek-R1-Distill-Qwen-32B reached 72.6% on AIME 2024 against 47.0% for direct RL on the same base (arXiv 2501.12948). The limits are real, though. Post-hoc compression often loses density: pruned-and-distilled models such as Minitron-4B and Llama-3.2-1B/3B ended up less dense than their parents, and only Gemma-2-9B beat its source (arXiv 2412.04315). Apple's distillation scaling laws show distillation beats pretraining only when the teacher's cost is amortized, with the benefit stopping at a capacity gap (arXiv 2502.08606). Small 2026 models lack knowledge and spend tokens: Qwen3.5-9B scores 14.7% on AA-Omniscience with an 82% hallucination rate and uses 2x to 4x more output tokens than frontier models (Artificial Analysis). Order matters too, since quantizing before pruning or distilling pushed perplexity from ~3.2 to between 34 and 53 (arXiv 2511.19495). The implication is that reaching frontier quality with far fewer active parameters requires either sparse storage (MoE) or moving long-tail knowledge out of the weights into retrieval and tools, because the ~2 bits per parameter floor rules out holding frontier factual recall in a single-digit-billion dense model.
Attention and KV: ~50x at a million tokens, ~1x to 2x at 8K#
DeepSeek-V4 is the strongest measured data point. It combines 4x-compressed KV with a lightning-indexer top-k (CSA), 128x-compressed dense attention (HCA) and a sliding window, and at 1M context V4-Flash needs ~10% of V3.2's per-token FLOPs and ~7% of its KV, about 2% of the KV of a BF16 GQA-8 baseline (HF DeepSeek-V4). These are vendor figures, but the previous step has market evidence behind it: DSA in V3.2 claimed 3.5x cheaper prefill and 6 to 7x cheaper decode at 128K with benchmark parity, and DeepSeek cut output prices 75% (The Decoder).
Hybrids with about three linear layers per full-attention layer form the other camp. Kimi Linear cut time per output token at 1M from 11.48 ms to 1.84 ms, though only 2.3x at batch 1, because most of the gain comes from bigger batches (alphaXiv 2510.26692). Nemotron-H 56B reached 2.4x the per-GPU output throughput of Llama-3.1-70B at 64K input (arXiv 2504.03624), Nemotron 3 Nano reached 3.3x Qwen3-30B-A3B at nearly equal active parameters, which isolates the attention contribution (HF/NVIDIA), and Olmo Hybrid 7B led Olmo 3 on RULER at 64K, 85.0 vs 70.9 (AI2). The main caution comes from MiniMax, whose Lightning-attention hybrid matched full attention at small scale but showed "clear deficits in complex, multi-hop reasoning tasks" at larger scale. MiniMax returned to full attention for M2 and noted that prefix caching and speculative decoding were immature for linear layers (MiniMax). These gains mostly stack with MoE because they hit a different term, but within the KV family they overlap, since hybrid layers, sparse selection and KV quantization all chase the same bytes. They also fade at short context: vLLM finds KV reads dominate decode only beyond about 7K tokens (vLLM).
Decoding: speculation buys latency, diffusion buys speed but not frontier quality#
Speculative decoding and MTP are mainly latency levers. EAGLE-3 reaches up to 6.5x at batch 1 but 1.38x at batch 64 (arXiv 2503.01840). DeepSeek's DSpark raised V4 per-user speed 57 to 85% at matched aggregate throughput on live traffic, an interactivity gain rather than more tokens per GPU (vendor result, reported by Analytics India Magazine). On energy, EAGLE-3 at batch 1 saves 2.1x to 2.5x on 8B to 13B models but only 1.34x on Llama-70B, and classic draft models can increase energy, with factors of 0.63 to 0.80 (arXiv 2602.09113). InferenceX's 21x cost gap from switching MTP on (GB300, DeepSeek-R1 at 150 tok/s/user, $2.35/M off vs $0.11/M on) mostly reflects how badly underused the GPU is without speculation at that speed; at 70 to 140 tok/s/user the gain is about 3x (Silicon Report).
Diffusion LLMs deliver ~1,000 to 1,500 tok/s per user but have not shown per-token cost wins at serving batch. Same-lab comparisons show a quality gap: Gemini Diffusion trails Gemini 2.0 Flash-Lite on GPQA Diamond 40.4 vs 56.5 (DeepMind), and DiffusionGemma trails Gemma 4 on AIME 2026, 69.1 vs 88.3 (Decrypt). Inception's Mercury 3 parity claims against Haiku-class models are self-reported with no independent testing, and its $0.70/M output list price is only about 35% below rivals (Let's Data Science). LLaDA2.0's 1.9x speedup over autoregressive decoding is explicitly a small-batch result (LMSYS). Diffusion and speculation exploit the same idle compute, so they should not be multiplied together.
Tokens per task: the lever moving the wrong way#
On the Artificial Analysis Index v5 (October 2026), median output per task rose from ~35k to ~59k tokens, and every model measured generated more tokens than in the previous version. Four agentic evaluations carry 55% of the index weight but produce ~90% of its tokens. Cost per task among top models spans more than 13x, and GPT-6.1 Sol (max) lands within 4 points of Opus 5.5 (max) at about one-eighth the cost per task (Artificial Analysis). Tokens-per-task levers are the only ones that multiply cleanly with every per-token lever, yet market pressure is pushing tokens per task up.
Research methods cut reasoning tokens by about half, mostly on small models. AdaptThink cut length 53% while gaining 2.4 points on a 1.5B model (arXiv 2505.13417), ThinkPrune halved length for a ~2% loss on AIME (arXiv 2504.01296), and selective latent thinking cut 58% at a 2.8-point cost, although naive latent compression on Qwen3-4B dropped GSM-Hard from 54.7% to 17.3% (alphaXiv 2605.25745). A small model plus test-time compute is a legitimate substitute for parameters on medium-difficulty problems: FLOPs-matched test-time compute beat a 14x larger model where the small model had non-trivial success, and lost on the hardest problems (arXiv 2408.03314). Routing is bounded by the share of hard tasks. If 30% of tasks need the flagship, no cascade can exceed ~3.3x however cheap the student becomes (derived).
Serving software: 1.2x to 2.8x in six months on the same silicon#
On identical GB300 NVL72 hardware, audited MLPerf per-GPU throughput rose from v5.1 to v6.0 by 2.77x for DeepSeek-R1 Server but only 1.21x for Llama 3.1 405B Offline (StorageReview). NVIDIA attributes the gains to kernel fusion, Dynamo disaggregation, wide expert parallelism, MTP and KV-aware routing, but publishes no per-technique split. Software gains are largest for new sparse and reasoning-heavy architectures and smallest for mature dense models. Independent InferenceX data implies GB200 cost per token improved roughly 2.5x to 5x within a year at high interactivity, but only about 1.3x at low interactivity (SemiAnalysis). A new co-designed architecture should therefore expect a launch-time software penalty that closes over 6 to 12 months.
Hardware: GPUs compound; exotic silicon mostly buys speed, not cost#
On independent InferenceX data, B200 delivers ~3.1x the tokens per megawatt of H100 on GPT-OSS-120B, and GB200 NVL72 with FP4 delivers ~8x a single-node H200 with FP8 on DeepSeek-R1 (Silicon Report). Vera Rubin NVL72 reaches 2x GB300 up to 100 tok/s/user, ~4x near 200 and 5.4x at 300, where GB300 is barely viable. Cost improves less than energy, because owner TCO is $3.57 per GPU-hour for Rubin against $2.36 for GB300 and $1.84 for GB200. CoreWeave's 5.4x performance per MW over GB200 is an operator claim that SemiAnalysis did not verify (SemiAnalysis). Rubin adds a ~3.125-bit LUT weight format and 2.8x Blackwell Ultra's memory bandwidth, and its sparsity-labeled NVFP4 inference rating of 50 PFLOPS is 12.5x its 4 PFLOPS BF16 rating (Tom's Hardware). From H100 to Rubin the plausible gain is ~10x to 30x in tokens per MW at moderate interactivity, and less in dollars.
SRAM-centric and wafer-scale chips win on per-user speed. On 8B-class models, Cerebras runs ~2,000 tok/s/user, SambaNova ~900 and Groq ~600, against ~350 on Blackwell (EE Times). Cerebras's 981 tok/s on a trillion-parameter model comes with per-token cost it calls only "comparable to GPU providers" (Pulse), and Harvard's independent measurements found Cerebras, SambaNova and Gaudi idle at 10 to 60% higher power than GPUs (arXiv 2604.10852). TPU v7 has no independent per-chip inference benchmark (Silicon Report).
Hardwired weights are the most dramatic data point. Taalas HC1 etches Llama-3.1-8B into mask ROM at ~4 bits on an 815 mm² TSMC N6 die drawing ~250 W. EE Times observed 15,000+ tok/s/user hands-on, the company claims ~17,000 and ~73x H200 throughput, and no accuracy figures have been published. Taalas's own frontier-scale cost claim is 7.6 cents per million tokens for DeepSeek-R1 on ~30 chips (EE Times), which is only ~1.4x below GB300 with MTP ($0.11/M). Hardwiring is therefore a huge speed multiplier but a modest cost multiplier at frontier scale, because ~30 tape-outs per model must be amortized and each model change needs a ~2-month respin. AMD agreed to acquire Taalas on August 6, 2026 (SiliconANGLE). Analog in-memory compute is still at simulation grade: Sagence's claimed Llama2-70B system at 666,000 tok/s and 59 kW, against a 624 kW H100 system, is simulated (IEEE Spectrum), and IBM and ETH Zurich show that off-the-shelf LLMs cannot even reach W4A8 quality on noisy analog hardware without a dedicated training recipe (Semiconductor Engineering). Photonics reached its first LLM demo on November 17, 2026, when Q.ANT replaced one attention layer with optical Fourier transforms, but it published no size, accuracy, throughput or energy figures (The Quantum Insider).
Lever map and evidence grade#
| Lever | Best realized gain (Nov 2026) | Quality cost | Overlaps with | Evidence grade |
|---|---|---|---|---|
| FP4 QAT (W4A4) | 3.6x weight bytes; 1.5-3x end to end | ~1% with QAT/native; PTQ worse when overtrained | Hardware generation, MoE memory | Vendor research, production adoption |
| Ternary / 2-bit | 1.0-1.3x over 2-bit kernels on CPU/GPU | 2-bit PTQ collapses reasoning; ternary at frontier scale unproven | Sparsity (30-52% zeros), custom silicon | Mostly vendor; Intel independent |
| KV quantization | FP8 2x; NVFP4 ~2x more | 0.7-2 pts (FP8); <1% (NVFP4, vendor) | Attention compression | Independent (vLLM) plus vendor |
| 2:4 weight sparsity | 1.1-1.3x alone | ~2-3% | Quantization (quality adds) | Vendor plus academic |
| MoE at ~3% activation | ~7x efficiency leverage; 5.2x vs naive TP serving | None by construction; ~1.3x less knowledge per total parameter | Batching, low-bit memory | Academic law, open reproduction |
| Activation sparsity / early exit | 1.5-1.8x at batch 1; ~1x at scale | Minimal | MoE, distillation | Academic |
| Distillation / density | ~10x/yr fewer parameters at fixed benchmark | Brittleness, hallucination, 2-4x tokens | MoE, routing | Independent (Epoch, AA) plus academic |
| Compressed / hybrid attention | 6-7x decode at 128K; ~50x KV at 1M | Multi-hop risk at scale (MiniMax) | KV quantization; prefix caching | Vendor reports, price evidence |
| Speculation / MTP | 1.38x at batch 64; ~3x at 70-140 tok/s/user | Lossless | Batching, low-bit, diffusion | Independent plus vendor |
| Diffusion LLMs | 5-15x per-user speed; ~1-2x cost | 10-20 pts on reasoning (same lab) | Speculation | Mostly vendor, self-reported |
| Reasoning efficiency | ~2x (research); 4.4x spread at frontier | ±3 pts | Multiplies with per-token levers | Independent (AA) plus academic |
| Routing / effort control | ~8x cost for -4 pts | Bounded by hard-task share | Density | Independent (AA) |
| Serving software | 1.2-2.8x on same GB300 in 6 months | None | MoE, MTP | Audited (MLPerf) |
| GPU generation | 3.1x (H100 to B200); 2-5.4x (GB300 to Rubin) per MW | None | Embedded FP4/LUT | Independent (InferenceX) |
| SRAM / wafer-scale | 3-6x per-user speed; cost "comparable" | None | Hardwired | AA speeds, vendor cost |
| Hardwired weights | ~50-74x per-user speed; ~1.4x cost at R1 scale | Aggressive 4-bit, unmeasured | Low-bit, MoE (30 tape-outs) | Vendor plus EE Times hands-on |
| Analog in-memory | ~10x power (simulated) | Needs noise-aware training | Low-bit | Simulation only |
| Photonics | None published | Unknown | Attention alternatives | Vendor demo only |
The multiplier budget: a million on paper, about a thousand in practice#
The tables below give each lever's standalone multiplier (low, central, high) and then combine them in four steps. Levers in the same family are grouped first. The model-side groups (weights versus KV) are then combined over their shares of bytes moved rather than multiplied together. Each idle-compute effect and each hardware-embedded effect is counted once. Finally, the group subtotals are multiplied, because they act on different factors of cost per task. A useful check is the ceiling the physics allows (derived). An MoE with ~20B active parameters in FP4, batch-amortized weights and sparse KV reads does ~40 GFLOP of weight math per token, about 20x less than the baseline's 0.8 TFLOP, and at H100 coefficients its energy floor is roughly 0.03 to 0.05 J per token, about 200x below B2024's ~9 J, before any hardware improvement. Realized model-side gains land well below that ceiling because of kernel efficiency, expert-parallel communication, prefill and idle power.
Table A: against B2024 (dense ~405B, BF16, H100, 2024 software), fixed 2024-frontier quality, ~64K agentic mix#
| Lever | Low | Central | High | Overlap treatment |
|---|---|---|---|---|
| FP4 weights and activations (QAT) | 1.5 | 2.5 | 3.6 | Weight group |
| Sub-4-bit (LUT / ternary) beyond FP4 | 1.0 | 1.2 | 2.0 | Weight group; GPU kernels cap it |
| MoE at ~3% activation | 3 | 6 | 10 | Weight group; needs wide-EP batch |
| Capability density (data, overtraining, distillation) | 4 | 10 | 30 | Weight group; overlaps MoE on active parameters |
| Activation sparsity / adaptive depth | 1.0 | 1.1 | 1.5 | Weight group; erodes at batch |
| Weight-group subtotal (overlap-adjusted) | 15 | 60 | 200 | Naive central ~200 |
| Attention architecture (MLA, hybrid, compressed-sparse) | 3 | 8 | 20 | KV group |
| KV quantization (BF16 to FP8/FP4) | 2 | 3 | 4 | KV group, same term |
| KV-group subtotal (overlap-adjusted) | 4 | 12 | 40 | Naive central ~24 |
| Model-side per token (byte-share combination plus batch synergy) | 10 | 30 | 80 | Shares ~65% weights / ~35% KV at capacity-limited batch |
| Serving software and scheduling | 1.5 | 2.5 | 5 | Utilization group |
| Speculation / MTP | 1.2 | 1.6 | 3 | Utilization group; same idle compute as batching |
| Diffusion decoding | 1.0 | 1.0 | 2 | Not credited (overlaps speculation; quality gap) |
| Utilization subtotal | 1.5 | 2.5 | 4 | |
| GPU generation H100 to Rubin (cost) | 3 | 5 | 8 | Energy central ~10x; FP4 counted above |
| Specialized silicon for routed share (cost) | 1.0 | 1.2 | 2 | Energy central ~2x |
| Tokens per task (reasoning efficiency, routing) | 0.7 | 1.5 | 3 | B2024 is a non-reasoning model, so small reasoning replacements can spend more tokens |
| Naive product of all central rows | ~106 | |||
| Overlap-adjusted total, cost per solved task | ~30x | ~700x | ~15,000x | High requires every lever at its best |
| Overlap-adjusted total, energy per solved task | ~60x | ~2,000x | ~40,000x | Hardware energy improves faster than $/token |
Two cross-checks support Table A. The low case (~30x) matches the most conservative market trend over 2.4 years (~40x, open-weight Pareto), and the central case sits a few times above the headline trend (~250x to 470x), which is the expected gap between the market's average deployment and a fleet that uses Rubin, routing and token-efficient RL together. A derived roofline gives the same order of magnitude. B2024 moves ~92 GB per token at 64K, which bounds 16 H100s to a few hundred tokens per second of aggregate decode and puts its cost in the tens of dollars per million tokens. InferenceX measures DeepSeek-V4-Pro at $0.12/M on GB300 at much higher quality, a per-token gap of very roughly 100x to 250x before any tokens-per-task lever, consistent with the central row. 1000x against 2024 is credible today in energy and within about a year in dollars, and most of it is what the market already does.
Table B: against B2026 (DeepSeek-V4-Pro-class on GB300 NVL72, late-2026 software), fixed late-2026-frontier quality#
| Lever | Low | Central | High | Overlap treatment |
|---|---|---|---|---|
| FP4 to ~3-bit LUT or ternary experts | 1.0 | 1.3 | 2.0 | Rubin LUT is ~1.36x fewer bytes than MXFP4 |
| MoE from ~3% to ~1.5% activation | 1.1 | 1.4 | 1.8 | Sublinear efficiency leverage |
| Frontier distillation plus retrieval offload of knowledge | 1.2 | 2.0 | 4.0 | Knowledge floor; brittleness risk |
| Activation sparsity / early exit | 1.0 | 1.05 | 1.2 | Near zero at scale |
| KV from FP8 to FP4 / 2-bit | 1.3 | 2.0 | 3.0 | KV already ~2% of GQA baseline, so its share is small |
| Hybrid linear layers on top of compressed-sparse attention | 1.1 | 1.5 | 2.5 | Same KV term |
| Model-side per token (overlap-adjusted) | 1.2 | 2.0 | 5.0 | Naive central ~11 |
| Serving maturity on new architecture plus DSpark-class drafting | 1.2 | 1.7 | 3.0 | MTP already in baseline |
| GB300 to Rubin (cost) | 1.5 | 2.0 | 4.0 | Energy central ~3x |
| Hardwired / SRAM tier for routed student (cost) | 1.0 | 1.3 | 3.0 | Energy central ~2x; up to ~10x on routed share |
| Reasoning-token efficiency (cost-aware RL, adaptive thinking) | 1.3 | 2.0 | 4.0 | Frontier spread 4.4x is the existence proof |
| Routing cascade and effort control | 1.5 | 3.0 | 6.0 | Capped by hard-task share; overlaps the row above |
| Tokens-per-task subtotal (overlap-adjusted) | 2 | 4 | 10 | |
| Naive product of all central rows | ~420x | |||
| Overlap-adjusted total, cost per solved task | ~4x | ~35x | ~1,800x | High is improbable: every lever must land |
| Overlap-adjusted total, energy per solved task | ~6x | ~80x | ~5,000x |
Against the late-2026 frontier, model-side headroom per token is only about 2x, because DeepSeek-V4 and its peers have already collected MoE, FP4 and most of the KV savings. What remains is spread across hardware (about 2x from Rubin), serving maturity, a specialized-silicon tier and, above all, tokens per task, which offers the largest remaining factor (about 4x) and is the only lever family on which the market is currently moving backward. Trend arithmetic converts these numbers into time. At 10x to 13x per year, 35x is about 1.4 to 1.5 years of progress. On trend, late-2026 frontier capability becomes ~1000x cheaper around 2029 with or without this program, so the program's value is reaching each step earlier and owning the cost structure. It does not multiply the trend, because the trend is other labs adopting the same levers.
A ranked, four-phase program a well-resourced team could run#
The reference design follows from where the remaining headroom sits. The flagship is a fine-grained MoE of ~2T total and ~20 to 25B active parameters (about 1 to 1.25% activation), with expert granularity near the 8 to 12 optimum identified by the Ling scaling laws (arXiv 2507.17702). It is trained natively in NVFP4 with fewer than 15% of linear layers kept in higher precision, following NVIDIA's recipe (arXiv 2509.25149), and its experts move to Rubin's ~3.125-bit LUT format or to ternary only if Phase 1 clears the bar. Its attention stack uses three Gated DeltaNet or KDA layers for each global layer, where the global layer is CSA/HCA-style compressed-sparse attention with an FP4 indexer, MLA-style latents and NVFP4 KV; the global-attention ratio stays tunable until frontier-scale multi-hop evaluations pass, to guard against the MiniMax failure. It supports 1M-token context, with recurrent-state checkpoints at chunk boundaries so prefix caching works, and ships trained-in MTP heads plus a DSpark-style parallel drafter designed for state rollback. Post-training uses RL with an explicit compute-cost term in the reward, AdaptThink-style skip-thinking, loop detection and several effort levels, and offloads long-tail knowledge to retrieval and tools to stay under the ~2 bits per parameter ceiling. The student is a ~3 to 4B-active MoE distilled from the flagship, sharing its tokenizer and drafter, trained with native low-bit QAT and a small KV footprint, with frozen base weights so it can be hardwired and adapters held in SRAM. In the fleet, a small router plus verifier assigns each task to the student or the flagship and picks an effort level; the flagship serves on Rubin NVL72 with disaggregated prefill and decode, wide expert parallelism and KV-aware routing, while the student runs on hardwired-ROM or SRAM-dataflow silicon.
Research bets ranked by expected value against B2026#
| Rank | Bet | Credited central vs B2026 | Confidence | Why it ranks here |
|---|---|---|---|---|
| 1 | Per-task accounting, cost-aware reasoning RL, effort control, routing cascade | ~4x | High | Multiplies with everything; 4.4x token and ~8x cost spreads exist today at near-equal scores |
| 2 | Rubin-class deployment, disaggregated wide-EP serving, co-trained MTP/parallel drafter | ~3x | High | Independently measured 2-5x per generation; 1.2-2.8x software gains in six months |
| 3 | Hybrid linear plus compressed-sparse attention with 4-bit KV | ~1.5-2x at agentic contexts | Medium-high | DeepSeek and Kimi data are strong, but MiniMax's scale failure is unresolved |
| 4 | Native sub-4-bit experts and ~1.5% activation | ~1.5-2x | Medium | No frontier-scale equal-compute ternary result; precision law predicts a parameter penalty |
| 5 | Distilled student plus retrieval offload of knowledge | ~1.5-2x on routed share | Medium | Densing law is real; brittleness and the knowledge floor cap it |
| 6 | Hardwired or SRAM silicon for the frozen student | ~1.3x cost, 2-10x energy on routed share | Medium-low | Only vendor data plus one hands-on demo; respin cost and model churn |
| 7 | Analog in-memory and photonic tiers | Not budgeted | Low | Simulation and single-layer demos only |
| Dropped | Early exit, mixture-of-depths, unstructured pruning, diffusion as main decoder | Under 1.2x | Gains small, shrinking, or quality-limited at frontier |
Phases, milestones and kill criteria#
| Phase | Months | Work | Exit milestone | Kill or redirect criterion |
|---|---|---|---|---|
| 0. Measure | 0-4 | Rebuild B2024 and B2026 on rented H100 and GB300/Rubin. Build a fixed agentic plus reasoning suite. Meter prefill, decode and idle power. Ablate FP8 vs FP4 KV, MTP, effort and routing on existing open models (V4-Flash, Qwen3.5, gpt-oss) | Baselines reproduce InferenceX and Epoch within ~30%. Routing plus effort control gives ≥3x $/task at ≤1 point loss on held-out tasks | If routing gains fall below 2x, re-weight toward model-side work |
| 1. Factorial co-design at 1-10B active | 3-12 | Grid over activation {3%, 1.5%, 0.75%}, precision {FP8, NVFP4, 3-bit LUT, ternary native}, attention {3:1, 7:1 hybrid, CSA-only} and MTP depth. Train to ~800 tokens/param with equal-compute baselines. Evaluate multi-hop and retrieval at 128K | Interaction matrix showing which pairs compound. Sub-4-bit passes at ≤1.5% loss gap and ≤1.2x reasoning-length growth. Hybrid passes on multi-hop parity at 128K | Drop sub-4-bit if trace inflation appears (the 2-bit failure mode); fall back to CSA-only if hybrids fail multi-hop |
| 2. Flagship and student | 9-24 | Pretrain ~2T/20-25B-active flagship with the Phase 1 winners. Cost-aware RL post-training. Distill the 3-4B-active student. Ship state-checkpointed prefix caching | Within ~2 points of the contemporaneous open frontier at ≥10x lower $/task on the same Rubin hardware. Student solves ≥60% of routed tasks | Below 5x: hold the architecture and invest in serving software instead |
| 3. Silicon and fleet | 18-36 | Tape out a hardwired or SRAM chip for the frozen student (2-month respin cadence, adapters in SRAM). Run the production cascade. Pilot analog or photonic on one frozen expert bank or attention substitute | ≥30x $/task and ≥80x J/task against B2026 on the fixed suite. Silicon tier ≥5x J/task vs Rubin on its routed share | If student churn exceeds respin economics, keep the student on SRAM-dataflow or GPUs |
Honest odds, and the problems nobody has solved#
Against B2024 at constant quality, the program reaches an overlap-adjusted ~700x in dollars and ~2,000x in energy with today's techniques, and it passes 1000x in dollars as Rubin software matures through 2027. Against B2026 at late-2026 frontier quality, the central estimate is ~35x in dollars and ~80x in energy, with a plausible band of ~10x to ~100x. Against whatever the frontier is when the program finishes, the advantage over a typical deployment stays roughly 10x to 50x, about the size of the cost-per-task spread already visible among top models. Reaching 1000x against the contemporaneous frontier would need a hardware discontinuity, and the only candidate, hardwired silicon at frontier scale, shows ~1.4x in cost by its own vendor's figures, not 10x.
Vendor and independent evidence diverge in a consistent direction. Hardware vendors quote peak ratios (FP4 at 6x over BF16, 12.5x on Rubin spec sheets, 74x for Taalas), while independent and audited data show 1.5x to 3x for precision, 2x to 5x per GPU generation, and ~1.4x in cost for hardwiring at R1 scale. Model vendors quote equal-quality ternary and diffusion results (Bonsai's 98.2%, Mercury's parity), while same-lab or independent comparisons show either no frontier-scale evidence or 10 to 20-point reasoning gaps. Systems claims made at batch 1 (speculation at 6.5x, activation sparsity at 1.8x, early exit) shrink to 1.0x to 1.4x at serving batch. The most reliable sources are InferenceX, MLPerf, Artificial Analysis and Epoch, and they support the central estimates here rather than the vendor highs.
Several open problems decide whether the high case is reachable. First, no one has measured joint compounding: no source measures MoE, low-bit, compressed attention, speculation and routing together at frontier scale, so every compounding factor in this report, including the overlap adjustments, is inference, and Phase 1's factorial study is the program's most valuable output. Second, sub-4-bit at frontier scale is unproven. The precision law says going below 4 bits costs more than 4x in parameters, the only large ternary successes are derived from full-precision bases with undisclosed recipes, and the failure mode is subtle, because low-bit reasoning models loop and inflate traces until per-token savings become per-task losses. Third, hybrid attention leaves open questions at scale: multi-hop and agentic quality given MiniMax's degradation, prefix caching with recurrent state when agents rely on 85 to 99% prefix hits, and speculative rollback through recurrent layers. Fourth, MoE economics need hyperscale concurrency, which conflicts both with the per-user speed targets that drive premium pricing and with hardwiring, which favors small frozen models. Fifth, the knowledge floor limits distillation: at about 2 bits per parameter, int4 halves capacity and MoE loses ~1.3x per total parameter, and the workaround of moving knowledge into retrieval has not been quantified at frontier quality. Sixth, tokens per task are rising, with the AA median going from 35k to 59k in a single index revision, and agentic evaluations may push this up faster than any token-efficiency method cuts it. Finally, measurement itself is unsettled. Decode-only versus full-rack power, idle power that runs 10 to 60% higher on some accelerators, missing TPU v7 and Trainium data, and the gap between price and cost can each move a headline figure by 2x or more.
Conclusion#
Whether "1000x cheaper" is achievable depends mostly on which year's frontier serves as the baseline. Against 2024 it is an accounting exercise the market is already completing; against late 2026 it is out of reach for any single program on current evidence. The realistic goal is a sustained 10x to 50x lead over typical deployment at frontier quality, worth one to two years on the price curve. Most of that lead comes from the seams between layers (MTP heads trained in for the serving stack, weight formats chosen for the silicon, KV structure chosen for batch size, students frozen for ROM) rather than from a breakthrough in any single layer.
The least appreciated lever is also the cheapest to pursue. Every hardware and model-side gain multiplies through tokens per task, yet the industry let that term grow ~70% in a single index revision. A team that prices tokens per solved task into training, routing and evaluation captures the one multiplier that does not overlap with the others. Energy reaches 1000x before dollars do, because joules per operation fall faster than the capital cost of new silicon, so programs judged on energy should expect to declare success earlier than programs judged on price.
Source: research/01-report-path-to-1000x.md, lines 3 to 206Edit on GitHub