Corerope

Risks and prior art

What could kill it

The largest risk is quality: a ~12B ternary core may not reason at frontier level, however much memory sits beside it, and no experiment has tested that yet. The other risks are engineering and timing.

The risks

  • The baseline moves. The cost of a fixed capability level falls ~10 to 13x a year on GPUs. A stack taped out in 2029 is measured against the 2029 frontier, so the 749x central case shrinks with every year of delay unless the core's quality keeps pace.
  • Wires, not gates. Fully spatial designs are usually dominated by wire capacitance and leakage, which the gate counts here leave out. Taalas's ~2 pJ per parameter is the realistic reference until experiment 1 says otherwise.
  • Area and yield. ~1.5 x 1011 gates means several reticle-size dies or a wafer-scale part, with the packaging, yield and cooling risks that come with them.
  • Frozen flaws and oversight. A reasoning flaw in silicon outlives its discovery, and latent reasoning weakens how readable the model's thinking is. Both need the safety valves built in above, not added later.
  • Workloads not modeled. Multimodal input, very long single contexts and tool-heavy agents may shift the hard-task share and the per-user state budget.

Known gaps in the evidence

Honest odds, and the problems nobody has solved

Against B2024 at constant quality, the program reaches an overlap-adjusted ~700x in dollars and ~2,000x in energy with today's techniques, and it passes 1000x in dollars as Rubin software matures through 2027. Against B2026 at late-2026 frontier quality, the central estimate is ~35x in dollars and ~80x in energy, with a plausible band of ~10x to ~100x. Against whatever the frontier is when the program finishes, the advantage over a typical deployment stays roughly 10x to 50x, about the size of the cost-per-task spread already visible among top models. Reaching 1000x against the contemporaneous frontier would need a hardware discontinuity, and the only candidate, hardwired silicon at frontier scale, shows ~1.4x in cost by its own vendor's figures, not 10x.

Vendor and independent evidence diverge in a consistent direction. Hardware vendors quote peak ratios (FP4 at 6x over BF16, 12.5x on Rubin spec sheets, 74x for Taalas), while independent and audited data show 1.5x to 3x for precision, 2x to 5x per GPU generation, and ~1.4x in cost for hardwiring at R1 scale. Model vendors quote equal-quality ternary and diffusion results (Bonsai's 98.2%, Mercury's parity), while same-lab or independent comparisons show either no frontier-scale evidence or 10 to 20-point reasoning gaps. Systems claims made at batch 1 (speculation at 6.5x, activation sparsity at 1.8x, early exit) shrink to 1.0x to 1.4x at serving batch. The most reliable sources are InferenceX, MLPerf, Artificial Analysis and Epoch, and they support the central estimates here rather than the vendor highs.

Several open problems decide whether the high case is reachable. First, no one has measured joint compounding: no source measures MoE, low-bit, compressed attention, speculation and routing together at frontier scale, so every compounding factor in this report, including the overlap adjustments, is inference, and Phase 1's factorial study is the program's most valuable output. Second, sub-4-bit at frontier scale is unproven. The precision law says going below 4 bits costs more than 4x in parameters, the only large ternary successes are derived from full-precision bases with undisclosed recipes, and the failure mode is subtle, because low-bit reasoning models loop and inflate traces until per-token savings become per-task losses. Third, hybrid attention leaves open questions at scale: multi-hop and agentic quality given MiniMax's degradation, prefix caching with recurrent state when agents rely on 85 to 99% prefix hits, and speculative rollback through recurrent layers. Fourth, MoE economics need hyperscale concurrency, which conflicts both with the per-user speed targets that drive premium pricing and with hardwiring, which favors small frozen models. Fifth, the knowledge floor limits distillation: at about 2 bits per parameter, int4 halves capacity and MoE loses ~1.3x per total parameter, and the workaround of moving knowledge into retrieval has not been quantified at frontier quality. Sixth, tokens per task are rising, with the AA median going from 35k to 59k in a single index revision, and agentic evaluations may push this up faster than any token-efficiency method cuts it. Finally, measurement itself is unsettled. Decode-only versus full-rack power, idle power that runs 10 to 60% higher on some accelerators, missing TPU v7 and Trainium data, and the gap between price and cost can each move a headline figure by 2x or more.

What the notes could not find

Each research note ends every question with its gaps: what the sources did not measure, or what this research could not verify. They are collected here, unedited.

Low-bit numerics for LLM inference efficiency (quantization, QAT vs PTQ, pruning/N:M sparsity), state as of Nov 2026 6 questions

Q1. Largest natively low-bit models trained to date, and quality vs full precision at equal params / equal compute; scaling-law evidence

  • Could not fetch the BitNet b1.58 2B4T paper or model card (allowlist block); its headline memory (widely reported ~0.4 GB non-embedding), CPU latency and energy-per-token vs Qwen2.5 1.5B / Llama 3.2 1B are not verified here.
  • No published frontier-scale (>=30B dense, >=1T tokens) natively-ternary run with an equal-compute BF16 baseline found. Training method for Bonsai 2 and Maple undisclosed.
  • No 2025-2026 scaling-law paper re-fitting Kumar et al. with block-scaled FP4 at >10B was found in this session (QuEST, Quartet and ParetoQ scaling claims not directly read).
  • CAT-Q 235B ternary PTQ quality numbers not read (only its existence via Intel's survey).

Q2. Measured speedups and energy reductions on real hardware for ternary/FP4 vs BF16; theoretical vs realized

  • No independent measured energy-per-token comparison of FP4 vs BF16 on B200/GB300 for a large LLM at serving batch sizes was found.
  • MLPerf Inference v6.0 (2026) FP4 results for NVIDIA and AMD were found only as search titles (AMD ROCm blog, AMD press release) and not read; per-accelerator tokens/s for DeepSeek-R1 FP4 are therefore not recorded here.
  • Original BitNet b1.58 paper claims (e.g. 71.4x matmul arithmetic energy at 7 nm, 2.71x speed, 3.55x memory at 3B) could not be verified from a fetched source in this session.

Q3. Quantizing activations and KV cache to low bits: state of the art and quality cost

  • No primary-source numbers on 2-bit or ternary activations at >7B in production; BitNet's 8-bit (or 4-bit, BitNet a4.8 / v2) activation results were not fetched.
  • Read only titles of Kitty, Dual-QK and DeepSeek-V4.1-Flash; their accuracy/compression numbers are not recorded.

Q4. Does low-bit training/quantization interact badly with overtraining?

  • No ternary-specific scaling law at >10B with D/N >= 100 from primary sources found in this session (TriLM/Spectra and ParetoQ scaling claims known only via Intel's summary).
  • No quantified QAT-vs-PTQ gap for FP4 at frontier scale in fetched sources.

Q5. Pruning and structured sparsity: realized gains at scale, and whether it stacks with quantization

  • No 2025-2026 measured result of 2:4 sparsity combined with FP4 on Blackwell at >=70B found (Rubin/Blackwell sparse FP4 peak numbers exist only as spec sheets).
  • Activation-sparsity results (e.g. arXiv 2602.06183 on 2:4 activation sparsity in pretraining; arXiv 2511.04477 dynamic sparsity in quantized inference) surfaced only as search titles.

Q6. Hardware support: NVIDIA Blackwell/Rubin FP4, AMD MI-series, dedicated ternary chips

  • AMD MI355X/MI400-series MXFP4 peak numbers and MLPerf v6.0 throughput figures not read.
  • Whether Rubin's 50 PFLOPS NVFP4 inference requires 2:4 sparsity or a different "adaptive compression" mechanism is ambiguous; Tom's Hardware labels it "with sparsity", and NVIDIA's own spec page was not checked.
  • No production dedicated ternary ASIC with independently measured LLM tokens/J was found; TeLLMe (edge FPGA ternary accelerator) surfaced only as a title.

Read the whole note

Sparse and Conditional Computation for LLM Inference Efficiency (MoE, activation sparsity, early exit, layer skipping, mixture-of-depths) 5 questions

Q1. Highest total-to-active parameter ratios in frontier/near-frontier open models, and what MoE scaling laws say about optimal sparsity and efficiency leverage

  • No primary DeepSeek V4 expert counts (routed/shared/top-k) were retrieved. The aman.ai primer did not include them.
  • Kimi K3 parameters and expert counts are only from a secondary blog citing "reported" figures.
  • No validated scaling-law estimate of EL at frontier compute (above 1e24 FLOPs) or at activation below 2%.
  • The quantitative memory-constrained optimum from the Joint MoE Scaling Laws paper was not extracted (only the abstract was read).

Q2. Does MoE's compute saving translate to real cost-per-token savings at serving time (memory-bandwidth-bound decode, expert parallelism)? Measured numbers.

  • No apples-to-apples measured $/token for an MoE versus a quality-matched dense model on the same hardware and serving stack was found.
  • No public 2026 SGLang/vLLM benchmark for DeepSeek V4 or Kimi K2.6 serving on GB200/B200 was retrieved. The LMSYS InferenceMAX/GB200 post (Oct 2025) was found but not fetched: LMSYS
  • Epoch AI analyses of MoE serving economics were not retrieved.

Q3. Activation sparsity in dense models: achievable sparsity and realized speedups

  • No measured large-batch GPU serving speedups for activation sparsity were found.
  • No 2026 frontier model uses activation sparsity as a headline feature (none found).
  • ProSparse numbers were not retrieved.

Q4. Mixture-of-depths, early exit, layer skipping, and adaptive per-token compute: realized gains and quality cost

  • The often-quoted MoD claim of "up to 50% faster sampling steps" (from the original abstract, recalled) was not verified this session.
  • No production frontier model deploying MoD or early exit was found.
  • No wall-clock measurements for 2025-2026 layer-skipping methods were retrieved.

Q5. How MoE interacts with quantization and distillation (does it compound?)

  • No controlled study quantifying how far MoE expert weights can drop (e.g. 2-bit QAT) at frontier scale with less than 1% loss was retrieved. The NeurIPS 2026 low-bit MoE paper was not read.
  • No measured end-to-end $/token for FP4 MoE on Blackwell versus BF16 dense was found.
  • No source directly measured MoE + distillation interaction (e.g. a dense teacher to a sparse student at matched quality).
  • Epoch AI analyses on MoE inference economics were not retrieved.

Read the whole note

Attention Alternatives and KV-Cache Efficiency for LLM Inference (state as of Nov 2026) 5 questions

1. Measured throughput, memory and cost improvements of hybrid / linear / sparse models vs full attention at 8K, 32K, 128K, 1M

  • No vendor-neutral, apples-to-apples (same params, same data) throughput table at 8K / 32K / 128K / 1M for hybrid vs dense was found. Most comparisons confound MoE, model size and attention.
  • No measured numbers yet for Nemotron 3 Super/Ultra, Falcon-H1/H1R, MiniMax-Text-01 or Jamba 1.7 serving were collected this session.
  • DeepSeek-V4 pricing was not found in the sources read.
  • RWKV-7 and RetNet: no 2025-2026 frontier-scale serving data found. They appear not to be used in any top open-weight model as of Nov 2026, but this is not confirmed.

2. Capability gaps of pure linear/SSM models and how hybrids fix them; required attention ratio

  • No public frontier-scale (>200B active-equivalent) controlled ablation of hybrid vs full attention on agentic benchmarks was found.
  • How retrieval-heavy agentic tasks (e.g., SWE-bench over 100K+ token repos) behave in 3:1 hybrids vs DSA models has not been head-to-head benchmarked in the sources found.

3. KV-cache share of inference cost in realistic 2026 workloads (long reasoning, agents)

  • No published measurement was found that gives KV-cache reads as an explicit share of $ or GPU-seconds for frontier production traffic in 2026.

4. MLA and DeepSeek V3.2 sparse attention: cost-reduction claims and measured API price changes

  • V4 API pricing, and whether the V3.2-to-V4 compute savings were passed through to prices, were not found.
  • No independent (non-DeepSeek) measurement of DSA's 6-7x decode-cost claim was found.

5. Do these gains overlap with or stack on top of quantization and MoE?

  • No controlled study was found that measures the joint (multiplicative vs sub-multiplicative) effect of hybrid attention + KV FP4 + MoE on end-to-end $/token.
  • Token eviction methods (H2O, SnapKV and successors) and cross-layer KV sharing (CLA, YOCO) were not researched in depth this session. No 2025-2026 production deployment data was collected for them.
  • NVFP4 KV-cache accuracy data was not retrieved. The vLLM post explicitly excludes it, and a Spheron blog claims a gain in concurrent requests but was not fetched.

Read the whole note

Decoding Efficiency and Test-Time Compute Economics (state as of Nov 2026) 4 questions

Q1. Realized speedups of speculative decoding and MTP in production at large batch vs batch size 1; does it save energy or only latency?

  • No measured production J/token for speculation at large batch was found.
  • Medusa and lookahead (n-gram/Jacobi) decoding: no 2025-2026 production numbers were retrieved in this session. The 2026 energy paper covered Medusa, but its Medusa figures were not extracted.
  • DSpark absolute throughput, batch sizes and quality effect were not reported in the available coverage.
  • The DeepSeek-V3 MTP acceptance rate (85-90%) and 1.8x TPS were not re-verified from the primary text in this session.

Q2. Diffusion LLM throughput claims and quality vs similar-size autoregressive models; scaling evidence

  • No independent (e.g., Artificial Analysis) measurements of Mercury 2/3 speed or quality were retrieved.
  • No large-batch throughput or J/token data for any dLLM.
  • No controlled scaling-law study (dLLM vs AR, compute-matched) was retrieved. Inception's "scaling laws hold" claim is unquantified.
  • Mercury 3's parameter count and hardware are undisclosed.

Q3. Tokens per task for frontier reasoning models; how much can token-efficient reasoning cut this without quality loss?

  • Per-model output-token totals for all AA v5 models were not available in the text (only the median and four Claude/GPT figures).
  • Token-efficiency results for frontier-scale models (more than 100B) on agentic tasks are scarce. Most measured reductions are on 1.5B-4B models and math.
  • No Epoch data series on tokens per task over time was found. Epoch explicitly does not quantify it.
  • No numbers were retrieved for Claude/GPT "adaptive thinking" products beyond effort-level spreads.

Q4. Smaller model with more test-time compute vs larger model with less

  • No 2026 frontier-scale FLOPs-matched study (e.g., a 30B model with heavy test-time compute vs a 1T model with light compute on agentic tasks) was found.
  • No measurements of verifier/PRM overhead in production cost per task.
  • No study was found that measures interaction effects (e.g., speculation gains on quantized MoE at large batch, or dLLM plus reasoning-length reduction) in one controlled setup. Compounding claims above are inferences.

Read the whole note

Capability Density, Distillation, and Inference Cost Trends for LLMs (as of Nov 2026) 6 questions

Q1. How fast has the price of a fixed capability level fallen? (Epoch AI, a16z, MIT/Epoch "Price of Progress"; latest 2025/2026 figures)

  • No source gives a rigorous three-way split (model density vs. hardware vs. serving software). The Price of Progress paper only separates hardware from everything else.
  • Artificial Analysis's own price-vs-intelligence trend charts could not be retrieved (fetch blocked), so there are no AA-native 2026 frontier-price numbers here.

Q2. Densing law: capability density doubling time and follow-ups

  • I found no independent replication of the densing law after Apr 2025, and no 2026 update extending the series through Qwen3.5 or Gemma 4.

Q3. Best small models in 2026 and what fraction of frontier capability they reach

  • Gemma 4 (released around Apr 2026, Apache 2.0), Phi-4 successors, SmolLM3, and small Llama 2026 numbers could not be verified: the AA Gemma 4 article was not fetchable. No measured data here.
  • I found no quantified 2026 data on small-model performance on long-horizon agentic benchmarks (e.g., Terminal-Bench, tau-bench) relative to frontier models.

Q4. Limits of distillation: is the student capped at the teacher, and is knowledge storage bounded per parameter?

  • No measured estimate of the total knowledge content (in bits) of a frontier model.
  • The Physics of LM results are on synthetic data at sub-1B scale. I found no frontier-scale validation of the 2 bits/param ratio.

Q5. Evidence on compounding or overlap of compression methods

  • I found no rigorous large-scale (>30B) study measuring quality-matched compounding of MoE + distillation + low-bit quantization + speculative decoding together.
  • I found no quantified same-hardware software-improvement rate (e.g., % per quarter on H100 from vLLM/SGLang/TRT-LLM releases). InferenceMAX tracks this nightly, but the public article gives no rate.

Q6. How should the baseline be defined (e.g., cost per token of a frontier dense model on H100s in 2024)?

  • No measured $/M-token figure for Llama 3.1 405B on H100 was retrieved (AA shows no current providers, and the SemiAnalysis dollar TCO figures are paywalled).
  • Rented H100 $/GPU-hour for 2024 vs. 2026 was not retrieved.

Read the whole note

Inference Hardware and Serving Systems: Energy and Cost per Token (state as of Nov 2026) 5 questions

Q1. Measured tokens/sec/user and tokens per joule across GPU generations (H100, B200, GB200 NVL72, GB300, Rubin) and alternative accelerators (Groq, Cerebras, TPU, etc.)

  • No independent tokens-per-joule numbers for Groq, Cerebras, SambaNova, TPU v7 Ironwood, AWS Trainium2/3 or Inferentia at matched model and interactivity were found in this session; the xPU-athalon full paper (which has measured power) was not read beyond the abstract.
  • Etched Sohu, MatX, d-Matrix (Corsair), Positron: not covered by any source fetched; no verified 2026 numbers collected. Status of NVIDIA's reported deal with Groq (late 2025) not verified in this session.
  • Vera Rubin MLPerf v6.1 numbers seen only as headlines.
  • Epoch AI hardware data not consulted.

Q2. Claimed and demonstrated efficiency of hardwired-weight chips (Taalas) and analog / in-memory / photonic compute for LLMs; maturity and accuracy

  • No independent accuracy measurements for Taalas HC1's quantized Llama 8B; no aggregate (multi-user) throughput per HC1 chip; no HC2 silicon results found.
  • EnCharge AI, Mythic, IBM NorthPole/HERMES measured LLM tokens/J, and Lightmatter/Lightelligence LLM compute results: not retrieved in this session.
  • ISSCC 2025/2026 CIM macro efficiency numbers (TOPS/W) not retrieved.

Q3. Energy breakdown of a decode step (HBM data movement vs arithmetic) and physical limits

  • No measured (rather than modeled) per-component decode energy split on B200/GB200/Rubin at production batch sizes found; the arXiv 2605.11999 phase-aware decode characterization was not read.
  • On-chip SRAM pJ/bit and NVLink/interconnect pJ/bit for current nodes not sourced.

Q4. How much have serving-software improvements alone cut cost per token on the same hardware?

  • No independent ablation quantifying prefix caching, continuous batching or Dynamo disaggregation separately in 2026; no vLLM vs SGLang vs TensorRT-LLM head-to-head numbers retrieved.

Q5. Which multipliers require model co-design, and do hardware gains compound with model-side levers?

  • No source quantified how much of InferenceX's frontier-model efficiency is due to MoE vs hardware; no measured head-to-head of a ternary model on ternary-native hardware vs FP4 on Rubin was found.

Read the whole note

Prior art and sources

  • Engram: Conditional Memory via Scalable Lookup, DeepSeek, January 2026: hashed lookup memory beside MoE, table offloaded to host DRAM.
  • Memory Layers at Scale, Meta, ICML 2025: product-key memory layers at billion-entry scale.
  • Cartridges, Stanford, 2025, and Cartridges++, 2026: trained compact context caches.
  • Inference cost per token economics 2026, Silicon Report, citing SemiAnalysis InferenceX: the B2026 price.
  • arXiv 2607.26571: H100 energy per HBM bit and per tensor-core operation.
  • The companion report, Path to 1000x cheaper LLM inference (01-report-path-to-1000x.md): the Taalas HC1, BitNet, sparsity, tokens-per-task and Artificial Analysis figures used here.
  • The synthesis scripts (Yosys 0.71), add-count model and cost model in /experiments.

Every source the research cites, with where it is cited: the research library.