Corerope

Research note

Inference Hardware and Serving Systems: Energy and Cost per Token (state as of Nov 2026)

Research library / Hardware and serving · about 17 minutes

Notes on conventions: "Vendor claim" = stated by the company selling the hardware. "Independent" = SemiAnalysis InferenceX/InferenceMAX, EE Times hands-on demo, or academic measurement. "Audited" = MLPerf Inference result (the numbers are audited; the attribution of gains to particular software features is the vendor's framing). Derived numbers (my arithmetic) are flagged as such in Inferences. Unit conversion used: X tokens/s per MW = X/1e6 tokens per joule, i.e. J/token = 1e6 / X.

Q1. Measured tokens/sec/user and tokens per joule across GPU generations (H100, B200, GB200 NVL72, GB300, Rubin) and alternative accelerators (Groq, Cerebras, TPU, etc.)#

Takeaway#

On independent InferenceX data, each NVIDIA generation step delivers roughly 2x to 10x more tokens per MW at a fixed interactivity, with the multiplier largest at high tokens/s/user (where the older system can barely operate): B200 is about 3.1x H100 on GPT-OSS-120B, GB200 NVL72 FP4 about 8x a single-node H200 FP8 on DeepSeek R1, and Vera Rubin NVL72 about 2x to 5.4x GB300 and up to about 10x 2025-era GB200 on DeepSeek-R1-class models. Wafer-scale and SRAM-centric chips (Cerebras, Groq, SambaNova) win on per-user speed (roughly 600 to 2,000 tok/s/user on 8B-class models vs about 350 on Blackwell), but there is still no independent, apples-to-apples tokens-per-joule comparison between them and GPUs at matched quality, and TPU v7 Ironwood has no public independent per-chip inference benchmark.

Cited Findings#

GPU generations, independent (SemiAnalysis InferenceMAX/InferenceX)

  • B200 delivers about 2.8M GPT-OSS-120B tokens/s per provisioned MW vs about 900K for H100, i.e. about 3.1x; B200 is about 20% more efficient than AMD MI355X on the same metric (SemiAnalysis InferenceMAX via Silicon Report, July 2026) Silicon Report
  • GB200 NVL72 (FP4) vs single-node H200 (FP8), DeepSeek R1 without MTP: about 8x tokens per provisioned MW (SemiAnalysis InferenceX via Silicon Report) Silicon Report
  • InferenceX, July 2026, output tokens/s per total-GPU MW on DeepSeek-class frontier models: Rubin vs GB300 is about 2x up to 100 tok/s/user, peaks near 4x around 200 tok/s/user, and is 5.4x at 300 tok/s/user (where GB300 is barely viable); Rubin vs 2025-era GB200 is under 3x at low speeds, about 10x at 150 tok/s/user and about 6x at 200 tok/s/user; GB200 cannot reach 300 tok/s/user SemiAnalysis, Vera Rubin NVL72 vs GB200 NVL72
  • Rubin absolute efficiency: 96,446 output tok/s/MW at 300 tok/s/user and 70,703 tok/s/MW at 350 tok/s/user (DeepSeek-class, InferenceX-normalized; InferenceX counts decode-GPU watts only) SemiAnalysis
  • CoreWeave (operator-published, not independent) benchmark of Vera Rubin NVL72 on DeepSeek R1 671B NVFP4, 8k in/1k out, with MTP speculative decoding, Dynamo disaggregated serving, wide expert parallelism and TensorRT-LLM: headline 5.4x performance per MW and 5x performance per dollar vs GB200 NVL72; published July 22, 2026. SemiAnalysis says these CoreWeave figures are unverified by them SemiAnalysis
  • Rubin hardware: 288 GB HBM4 per package, 2.8x the memory bandwidth of Blackwell Ultra, 2x FP8/FP4 tensor throughput vs Blackwell; supports a LUT-based weight format at about 3.125 bits/weight (vs MXFP4 at about 4.25 effective bits), e.g. a 2.8T-parameter Kimi K3 needs about 1,094 GB vs about 1,487.5 GB in MXFP4 SemiAnalysis
  • Owner TCO per GPU-hour (SemiAnalysis scenario): Rubin $3.57, GB300 $2.36, GB200 $1.84 SemiAnalysis

Cost per million tokens, independent unless noted

  • Rubin at 350 tok/s/user: $4.18 per million output tokens (only Rubin can serve this point); Rubin is about 1.5x cheaper than July-2026 GB200 through 100 tok/s/user, about 3x at 200 to 250 tok/s/user, about 5x cheaper than GB300 at 300 tok/s/user, and up to about 8x cheaper than 2025-era GB200 at 150 tok/s/user SemiAnalysis
  • DeepSeek-V4-Pro at 27 tok/s/user: GB200 NVL72 $0.28/M vs GB300 NVL72 $0.12/M (InferenceX) Silicon Report
  • DeepSeek R1 at 150 tok/s/user on GB300 NVL72 (Dynamo + TensorRT-LLM): $2.35/M with MTP off vs $0.11/M with MTP on, about 21x (InferenceX) Silicon Report
  • GLM-5 FP8 with MTP at 18 tok/s/user (SGLang, May 2026): AMD MI355X $0.22/M vs NVIDIA B200 $0.30/M; Silicon Report's numbers imply about 27% cheaper, but the cited InferenceX URL slug says "40% cheaper" (internal inconsistency in the secondary source) Silicon Report
  • GPT-OSS-120B on B200: about $0.02/M (NVIDIA blog, vendor claim) Silicon Report
  • Package TDP: B200 1,000 W; MI355X 1,400 W Silicon Report

MLPerf Inference v6.0 (April 2026, audited)

  • GB300 NVL72 per-GPU throughput, v5.1 to v6.0 on identical hardware: DeepSeek-R1 Server 2,907 to 8,064 tok/s/GPU (2.77x); DeepSeek-R1 Offline 5,842 to 9,821 (1.68x); Llama 3.1 405B Server 170 to 259 (1.52x); Llama 3.1 405B Offline 224 to 271 (1.21x) StorageReview
  • Blackwell Ultra (GB300) vs Blackwell (GB200), DeepSeek-R1 per-GPU: +45% Offline, +25% Server Silicon Report
  • AMD MI355X, 87 GPUs / 11 nodes: Llama 2 70B 1.04M tok/s Offline, 1.02M tok/s Server; AMD frames this as 92 to 104% of B300 Silicon Report
  • No Google TPU or AMD numbers appear in NVIDIA's v6.0 summary; NVIDIA was the only vendor submitting on all new tests StorageReview
  • Secondary sources report Vera Rubin's MLPerf debut at 1.94x GB300 on DeepSeek-V3 and a v6.1 headline of "3.7x over GB300" (search-result titles only; not fetched, unverified) FourWeekMBA; Shattered.io

Non-GPU accelerators

  • Per-user decode speed on Llama-3.1-8B-class models: Cerebras about 2,000 tok/s/user, SambaNova about 900, Groq about 600 (Artificial Analysis figures as quoted by EE Times); NVIDIA Blackwell about 350 (Taalas internal test, vendor-adjacent) EE Times, Feb 19 2026
  • Cerebras (company claim, May 20, 2026): 981 tok/s on a trillion-parameter model (reportedly Kimi K2.6), about 6.7x the top GPU cloud and about 23x the median; per-token cost "comparable to GPU providers"; no power, wafer count, or measurement source disclosed Pulse
  • Harvard measurement study "The xPU-athalon" (Golden, Wu, Wei, Brooks; arXiv April 12, 2026) benchmarked Cerebras CS-3, SambaNova SN-40, Groq, Gaudi, TPUv5e, A100, H100, MI300X with power measurements across prefill and decode; abstract reports Cerebras, SambaNova and Gaudi have 10 to 60% higher idle power than NVIDIA/AMD GPUs and that the best platform depends on batch size, sequence length, and model size arXiv 2604.10852
  • TPU v7 Ironwood: no public independent per-chip inference throughput benchmark exists as of July 2026; Google's "4x+ over Trillium" claim is vendor-stated and not workload-specific Silicon Report; SemiAnalysis said TPUv7 InferenceX results were expected "in the next couple of months" (July 2026), and AMD committed to MI455X results SemiAnalysis

Inferences#

  • Derived J/token from InferenceX figures: H100 GPT-OSS-120B about 1.1 J/token (1e6/900K); B200 about 0.36 J/token (1e6/2.8M). Rubin on DeepSeek-R1-class at 300 tok/s/user about 10.4 J/output token (1e6/96,446), at 350 tok/s/user about 14 J/token. These are not comparable across rows (different models, interactivity, input/output lengths; R1 is reasoning with 8k-input prefill). The spread (0.36 J to 14 J per token) shows that interactivity target and model choice move energy per token by about 40x on the same vendor's hardware, which is as large as a generation jump.
  • Generational multipliers are interactivity-dependent: at low tok/s/user (throughput-optimized, big batches) the gap is about 1.5 to 3x per generation; at high tok/s/user the older system falls off a cliff, giving 5 to 10x. For a "1000x cheaper" target, the hardware-generation contribution from H100 to Rubin is plausibly about 10x to 30x in tokens/MW at moderate interactivity (3.1x H100 to B200, then about 2 to 4x across GB200/GB300 to Rubin), with cost per token improving less than energy because Rubin's $/GPU-hour is about 1.9x GB200's.
  • Since FP4 (GB200) and 3.125-bit LUT weights (Rubin) are already part of these generational gains, low-bit weight savings are partly already "spent" in the GPU baseline; a 1000x plan should not double-count them against a Rubin FP4 baseline.
  • The 21x MTP on/off gap at 150 tok/s/user is mostly a statement that without speculative decoding GB300 is badly underutilized at that interactivity; it is a model-co-design lever (MTP heads trained into the model) more than a hardware lever.

Gaps#

  • No independent tokens-per-joule numbers for Groq, Cerebras, SambaNova, TPU v7 Ironwood, AWS Trainium2/3 or Inferentia at matched model and interactivity were found in this session; the xPU-athalon full paper (which has measured power) was not read beyond the abstract.
  • Etched Sohu, MatX, d-Matrix (Corsair), Positron: not covered by any source fetched; no verified 2026 numbers collected. Status of NVIDIA's reported deal with Groq (late 2025) not verified in this session.
  • Vera Rubin MLPerf v6.1 numbers seen only as headlines.
  • Epoch AI hardware data not consulted.

Q2. Claimed and demonstrated efficiency of hardwired-weight chips (Taalas) and analog / in-memory / photonic compute for LLMs; maturity and accuracy#

Takeaway#

Taalas HC1 is the only hardwired-weight LLM chip with a third-party hands-on result: Llama-3.1-8B at 15,000+ tok/s/user observed by EE Times (company claims about 17,000) on a 250 W, 815 mm2 TSMC N6 die with weights in mask ROM at about 4 bits; AMD agreed to acquire Taalas on August 6, 2026. Analog in-memory compute (IBM, Sagence) and photonics (Q.ANT) remain at the simulation, small-model, or single-layer demo stage for LLMs, with no published measured energy-per-token on a full LLM.

Cited Findings#

Taalas (hardwired weights)

  • HC1 hardwires Llama3.1-8B only; weights stored in a mask-ROM-based "recall fabric"; about 4-bit parameters with multiplication on "a single transistor" (company claim; compute said to be fully digital); weights quantized "aggressively" by Taalas's own admission, no accuracy figures given; a small programmable SRAM holds fine-tuned weights (LoRA-style) and/or KV cache EE Times
  • Measured: EE Times demo test 15,000+ tok/s/user; company claims >16,000, internal tests close to 17,000 EE Times
  • Physical: TSMC N6, 815 mm2 die, about 250 W per chip, 10 cards per server about 2.5 kW, air-cooled EE Times
  • Company cost claims: 0.75 cents per 1M Llama3.1-8B tokens; DeepSeek-R1 on about 30 chips at 7.6 cents per 1M tokens, "less than half" a throughput-optimized GPU; simulated DeepSeek-R1-671B at about 12,000 tok/s/user vs GPU state of the art about 200 tok/s/user; about 20B parameters per chip in MXFP4 if SRAM is moved to a separate chip; full 671B would need about 30 custom tape-outs EE Times
  • Customization: two masks change per model; model-to-RTL about one week of effort ("not fully push-button"); expected two-month turnaround; verification by full-model simulation because error margin is "basically zero"; roadmap was a mid-size reasoning-model chip in early summer 2026 and frontier-level models by end of 2026 EE Times
  • Company claim (Feb 2026): HC1 about 73x the throughput of H200 at one-tenth the power; HC2 targets models of about 20B parameters (no status given as of Aug 2026) SiliconANGLE
  • AMD agreed to acquire Taalas on August 6, 2026 (price undisclosed); AMD plans to pair Taalas chips with Instinct GPUs in Helios racks, program via ROCm; Taalas had raised about $219M total (including $169M in Feb 2026), about 25 employees as of Feb 2026 SiliconANGLE; EE Times

Analog / in-memory compute

  • Sagence (analog in-flash): simulated Llama2-70B "Delphi" system at 666,000 tok/s at 59 kW vs a 624 kW H100 system; claims about 1/10 power, 1/20 cost and 1/20 space vs H100; claims 8 bits per flash cell and 1 flash transistor per 8-bit weight vs 48 transistors digital; generative chip supported only by simulation; noise, drift and temperature shift compensated by reference cells (no figures) (IEEE Spectrum, Nov 2024, older result) IEEE Spectrum
  • IBM Research / ETH Zurich "Analog Foundation Models" (arXiv 2505.09663, NeurIPS 2025): off-the-shelf LLMs cannot reach 4-bit-weight/8-bit-activation quality on noisy analog IMC hardware; their training method adapts Phi-3-mini and Llama-3.2-1B to retain performance "comparable to" W4A8 baselines and gives better test-time-compute scaling; no energy figures in the summary Semiconductor Engineering, Dec 2025; NeurIPS 2025

Photonic compute

  • Q.ANT (Nov 17, 2026, SC26): hybrid Q.ANT NPU Gen 2+ plus an AMD GPU ran an LLM in which an attention layer was replaced by optical Discrete Fourier Transforms; no model size, accuracy, throughput or energy figures published; Gen 2 launched 2025 (matmul + nonlinear ops); open-source SDK Sept 2026; combined Gen 2/Gen 2+ systems planned for 2H 2027 The Quantum Insider; Q.ANT deployed Gen 2 processors at LRZ in March 2026 Photonics Online

Inferences#

  • Derived Taalas energy: 250 W / about 16,000 tok/s is about 15.6 mJ/token for an 8B model at one stream. A batch-1 H100 reading 8 GB of FP8 weights per token at 11.68 pJ/bit spends about 0.75 J/token on HBM alone (8e9 x 8 bits x 11.68 pJ), so HC1 is about 50x better at batch 1. At GPU batch sizes of 64 to 128, weight-read energy per token on the GPU falls to roughly 6 to 12 mJ, so HC1's energy advantage depends heavily on how many concurrent streams one HC1 can serve (not disclosed). Its decisive advantage is per-user speed at low energy, not necessarily aggregate tokens/J vs a well-batched GPU.
  • Taalas's own DeepSeek-R1 cost claim (7.6 cents/M) is close to InferenceX's GB300 MTP-on figure ($0.11/M at 150 tok/s/user), i.e. only about 1.4x cheaper on cost, while offering about 80x the per-user speed. Hardwiring looks like a large speed multiplier and a modest cost multiplier at frontier scale, because about 30 tape-outs of 815 mm2 silicon per model must be amortized over a model's service life.
  • Hardwiring is the clearest example of a gain that requires model co-design: weights frozen in mask ROM, aggressive 4-bit quantization accepted, only a small SRAM for adapters/KV, and a 2-month respin per model. A model designed to tolerate ROM-frozen base weights plus small trainable adapters, and with a small KV footprint, fits this hardware best.
  • Analog IMC needs noise-aware training (IBM's work) to even match W4A8 digital quality, so its gains are inherently co-design gains; as of late 2026 no full-LLM measured silicon energy figure was found. Sagence's simulated 89 mJ/token for Llama2-70B (59 kW / 666k tok/s) is only about 4x better than the InferenceX-measured B200 figure for GPT-OSS-120B (about 0.36 J/token; different model, so a loose comparison). Even the simulated analog claim is therefore a single-digit multiple over 2025-26 digital GPUs, not 100x.
  • Photonics for LLMs is at a single-op demo stage (optical FFT replacing attention); it would require a co-designed architecture (FFT/Fourier mixing instead of softmax attention), and offers no quantified gain yet.

Gaps#

  • No independent accuracy measurements for Taalas HC1's quantized Llama 8B; no aggregate (multi-user) throughput per HC1 chip; no HC2 silicon results found.
  • EnCharge AI, Mythic, IBM NorthPole/HERMES measured LLM tokens/J, and Lightmatter/Lightelligence LLM compute results: not retrieved in this session.
  • ISSCC 2025/2026 CIM macro efficiency numbers (TOPS/W) not retrieved.

Q3. Energy breakdown of a decode step (HBM data movement vs arithmetic) and physical limits#

Takeaway#

On H100-class hardware, HBM access costs about 11.7 pJ/bit while tensor-core math costs about 0.52 pJ/FLOP, so reading one 8-bit weight (about 93 pJ) costs about 90x the 2 FLOPs (about 1 pJ) it feeds. Decode is therefore weight-movement-bound at small batch and KV-cache-movement-bound at long context; batching, on-chip storage (SRAM/ROM), and fewer bits per weight attack the dominant term, but arithmetic energy sets a floor roughly two orders of magnitude below today's per-token energy, and Landauer limits are many more orders below that.

Cited Findings#

  • Energy coefficients from microarchitectural measurements: HBM 11.68 pJ/bit (H100/GH200), 13.11 pJ/bit (A100); tensor core 0.52 pJ/FLOP (H100/GH200), 0.70 pJ/FLOP (A100) (Vartziotis et al., "From Tokens to Watt-hours", arXiv 2607.26571, 2026, citing Antepara et al.) arXiv 2607.26571
  • That paper's analytical model (single-request, batching only via a parameter-access amortization factor) finds compute dominant for most model sizes at 500/500 tokens and KV-cache attention reads growing super-linearly and becoming substantial at long outputs; its compute-only per-token estimates are 99.84 mJ (32B), 218.4 mJ (Llama 3.3 70B), 374.4 mJ (GPT-OSS 120B); request-energy errors vs measurements are 4.7% to 27%; static/idle power not modeled; authors say it does not replace physical measurement arXiv 2607.26571
  • Taalas's rationale is to place a model's weights and dataflow in transistors instead of moving them through HBM SiliconANGLE
  • Rubin's gains at high interactivity track its 2.8x memory bandwidth over Blackwell Ultra (decode at high tok/s/user is bandwidth-bound) SemiAnalysis
  • Idle power matters: Cerebras, SambaNova and Gaudi measured at 10 to 60% higher idle power than GPUs arXiv 2604.10852

Inferences#

  • Derived per-token breakdown (H100 coefficients, dense 70B, FP8 weights, batch 1): weight reads 70e9 x 8 bits x 11.68 pJ is about 6.5 J; math 1.4e11 FLOP x 0.52 pJ is about 0.073 J. Weight movement is about 90x arithmetic. The crossover batch where amortized weight-read energy equals math energy is about 90 (for 8-bit weights), about 45 for 4-bit. Above that, a GPU is compute-energy-bound at about 1 pJ per parameter per token (about 0.07 J/token for a dense 70B), plus KV-cache reads that grow with context length and are not amortized by batching (each sequence reads its own KV).
  • Implication for low-bit weights: going 8-bit to ternary (1.58 bits) cuts weight-read energy about 5x, which matters a lot at small batch, but at large batch the floor is set by MAC energy and KV reads. Ternary weights also cut MAC energy (multiply becomes add/sign), which is why low-bit gains compound only with hardware that has cheap low-bit datapaths.
  • Implication for on-chip memory: SRAM or mask ROM next to the MAC removes the 11.7 pJ/bit HBM term (on-chip SRAM access is commonly cited at well under 1 pJ/bit, though not sourced in this session). This is the physical basis for Groq/Cerebras/Taalas speed, but SRAM capacity per mm2 forces model sharding across many chips/wafers, and idle/static power plus interconnect energy can eat the gain.
  • MoE is a co-design lever on the same term: energy scales with active parameters (e.g. about 37B active for DeepSeek-class) not total, and InferenceX frontier results are already MoE, so a "1000x vs frontier dense on GPUs" baseline should be stated carefully: MoE plus FP4 plus Rubin is already about two orders of magnitude cheaper per token than a dense FP16 model on H100.
  • Landauer bound: kT ln2 is about 2.9e-21 J per bit erased at 300 K. A 1.4e11-FLOP token at about 0.5 pJ/FLOP (0.07 J) sits about 1e8 to 1e9 times above a naive Landauer floor, so physics is not binding; the practical limits are wire/memory energy (pJ/bit) and CMOS switching energy (fJ range per gate), meaning 10x to 100x more from hardware alone is plausible only by eliminating off-chip data movement and lowering precision.
  • Older background figures (Horowitz, ISSCC 2014, 45 nm: DRAM access about two orders of magnitude above an arithmetic op) are consistent with the 2026 H100 coefficients above but were not re-fetched this session.

Gaps#

  • No measured (rather than modeled) per-component decode energy split on B200/GB200/Rubin at production batch sizes found; the arXiv 2605.11999 phase-aware decode characterization was not read.
  • On-chip SRAM pJ/bit and NVLink/interconnect pJ/bit for current nodes not sourced.

Q4. How much have serving-software improvements alone cut cost per token on the same hardware?#

Takeaway#

Software alone produced 1.2x to 2.8x per-GPU throughput gains on identical GB300 hardware in six months (MLPerf v5.1 to v6.0, audited), and InferenceX shows 2025-era GB200 vs July-2026 GB200 differing by roughly 1.3x to 3x in cost per token at fixed interactivity. Speculative decoding via MTP is the single largest serving lever observed (about 3x throughput at 70 to 140 tok/s/user, up to 21x in cost at 150 tok/s/user on GB300), and it requires the model to ship MTP heads.

Cited Findings#

  • MLPerf v5.1 to v6.0, same GB300 NVL72 hardware: DeepSeek-R1 Server 2.77x, Offline 1.68x; Llama 3.1 405B Server 1.52x, Offline 1.21x; NVIDIA attributes gains to TensorRT-LLM kernel fusion and attention data parallelism, Dynamo disaggregated prefill/decode, Wide Expert Parallel, MTP, KV-aware routing; no per-technique breakdown published StorageReview, Apr 1 2026
  • InferenceX: MTP gives about 3x throughput at 70 to 140 tok/s/user across SKUs; GB300 DeepSeek R1 at 150 tok/s/user costs $2.35/M with MTP off vs $0.11/M with MTP on Silicon Report
  • NVIDIA vendor claims: 5x cost reduction for GPT-OSS-120B from software over two months; 15x lower cost per token for Blackwell vs the prior generation Silicon Report
  • Rubin advantage is about 10x over 2025-era GB200 but much smaller over July-2026 GB200 at 150 tok/s/user (cost: up to about 8x vs 2025 GB200 vs about 1.5 to 3x vs July 2026 GB200), and SemiAnalysis expects Rubin numbers to improve further as software matures, following the Blackwell pattern; Rubin can reuse Blackwell SM100 kernels, unlike Hopper-to-Blackwell SemiAnalysis
  • CoreWeave Rubin result already uses the full stack: MTP, Dynamo disaggregated serving, wide EP, TensorRT-LLM; CoreWeave's power counts prefill and decode GPUs while InferenceX counts decode GPU watts only SemiAnalysis
  • vLLM was used for Qwen3-VL in MLPerf v6.0; SGLang used for SemiAnalysis's GLM-5 FP8 MI355X vs B200 benchmark StorageReview; Silicon Report

Inferences#

  • Derived from SemiAnalysis's two GB200 baselines: the ratio of Rubin-vs-2025-GB200 to Rubin-vs-July-2026-GB200 at 150 tok/s/user implies GB200 software improved its cost per token by roughly 2.5x to 5x within about a year at that interactivity (8x / 1.5 to 3x); at low interactivity the improvement is smaller (about 1.3x).
  • Software gains are largest for new architectures (MoE, MLA, reasoning models) and at high interactivity, and smallest for mature dense models (Llama 405B Offline 1.21x). A new co-designed model architecture should expect an initial software penalty that closes over 6 to 12 months unless it maps onto existing kernels.
  • Serving levers that compound with model design: MTP/speculative decoding (needs draft heads trained in), prefix caching (needs workloads with shared prefixes and small KV), disaggregation (separates compute-bound prefill from bandwidth-bound decode so each runs on suitable hardware), wide EP (needs MoE). These are multiplicative with hardware generation gains in InferenceX data, i.e. stacking roughly 3x (MTP) x 2x (software maturity) x 3 to 10x (hardware generation) is observed in practice.

Gaps#

  • No independent ablation quantifying prefix caching, continuous batching or Dynamo disaggregation separately in 2026; no vLLM vs SGLang vs TensorRT-LLM head-to-head numbers retrieved.

Q5. Which multipliers require model co-design, and do hardware gains compound with model-side levers?#

Takeaway#

Most large multipliers beyond a generation step are co-design gains: low-bit/LUT weights (Rubin 3.125-bit format, Taalas 4-bit ROM), MTP heads, MoE sparsity, noise-tolerant training for analog, frozen weights for hardwiring, Fourier-style mixing for photonics. They compound partially, not fully, because they attack the same dominant term (bytes moved per token) and saturate once arithmetic energy, KV-cache traffic, idle power or amortized silicon cost become the limit.

Cited Findings#

  • Rubin's LUT weight format at about 3.125 bits/weight reduces a 2.8T model from about 1,487.5 GB (MXFP4) to about 1,094 GB, about 6 vs 4 HBM packages for weights alone SemiAnalysis
  • Taalas requires per-model silicon with aggressive quantization and frozen weights; only small SRAM for fine-tunes/KV EE Times
  • Analog IMC requires an adapted training recipe for LLMs to reach W4A8 parity Semiconductor Engineering
  • MTP yields about 3x throughput and is a model-provided feature Silicon Report
  • Q.ANT's LLM demo replaced attention with optical DFTs, i.e. architecture change required The Quantum Insider

Inferences#

  • Rough multiplier ledger vs "frontier dense transformer on H100 at FP8" (derived, order-of-magnitude): hardware H100 to Rubin about 10x to 30x tokens/MW; serving software maturity about 2x to 5x; MTP about 3x; MoE active/total ratio about 10x to 20x (model); weights 8-bit to about 1.6 to 3 bits about 2x to 5x on the memory term (model plus hardware support). Naive product exceeds 1000x, but the memory-related levers (low-bit, MoE, on-chip storage, batching) overlap because each reduces bytes/token, so real compounding is sub-multiplicative once the bottleneck shifts to MAC energy, KV reads, or fixed costs.
  • Hardwired/ROM chips plus a small, low-bit, small-KV model is the combination most likely to give an additional 10x+ energy beyond Rubin at high per-user speed, but cost per token is bounded by tape-out amortization and model churn (2-month respin); this suits a stable distilled model, not a fast-iterating frontier model.
  • KV cache is the term not addressed by weight-side hardware tricks; model-side KV reduction (MLA, linear/state-space attention, small heads) is required for long-context decode to benefit from on-chip-weight hardware.

Gaps#

  • No source quantified how much of InferenceX's frontier-model efficiency is due to MoE vs hardware; no measured head-to-head of a ternary model on ternary-native hardware vs FP4 on Rubin was found.