Persistent-megakernel compilers (TileRT, MIT-licensed, tile-ai) now hit Cerebras/Groq-class interactivity on stock NVIDIA nodes: 494-600 tok/s/user on 700B-1T MoE models (GLM-5.1, DeepSeek-V3.2) on 8x B200, 3.6x faster than conventional FP8 serving, and Xiaomi claims 1000+ TPS on a 1T model. Production at Xiaomi (MiMo UltraSpeed) and Z.ai (GLM HighSpeed) - both carved a premium speed tier out of GPU fleets they already owned rather than buying dataflow chips.
The structural argument is fungibility, not speed: a GPU pool reallocates between prefill / batch decode / ultra-interactive decode by scheduler config; an ASIC fleet freezes that ratio in silicon at PO time. This is the strongest single argument against the 'custom inference silicon takes share' scenario (merchant vendors: Groq, Cerebras, SambaNova - watch any Cerebras IPO paper hard).
NOT a new NVDA moat, though: TileRT is open source and the megakernel technique is already replicated on AMD MI300X (Kog) and commoditizing in research. Durable winners: owners of large fungible GPU fleets - Microsoft, Alphabet and Amazon - which get lower capex per interactive feature and are the hyperscaler expressions in the ticker list and the orchestration layer (vLLM as prefill/router). Mild NVDA positive (demand stickiness), neutral AMD (gets the trick too, but its conventional stack benchmarked 6x slower end-to-end: MI355X 18.2s vs TileRT 3.0s).
Fragility caveat: batch-size-1 only, 2 model families, ABI-locked builds, hand-tuned per architecture - today it is a racing engine, not a platform. Source: SemiAnalysis InferenceX (Aug 10 2026) + tile-ai/tilert repo.
Update 2026-08-28 (full subscriber article read):
- Production proof: Xiaomi (MiMo V2.5 Pro UltraSpeed) and Z.ai (GLM-5 HighSpeed) both shipped TileRT speed tiers WITHOUT buying dataflow chips — carved the tier out of GPU fleets they already owned. "Good enough on hardware you already own beats architecturally pure on hardware you have to buy."
- Silicon gap is real and permanent: Cerebras WSE-3 (44GB SRAM, 21 PB/s) sits ~100x above B200 (8 TB/s HBM) on batch-1 decode roofline. "Software can approach the HBM roofline, but it cannot raise it." TileRT does NOT kill the top of the speed market.
- Sharpest fungibility framing: GPU prefill/decode/speed-tier ratio is a software scheduler decision that follows demand hourly; ASIC fleet ratio is fixed the day the PO signs — guess wrong and you strand capital or turn away the premium traffic you bought the machines for.
- TCO teaser (paywalled detail): TileRT up to 1.9x faster interactivity at iso-cost vs traditional decode engines.
- CONFIRMED (web, Aug 2026): NVIDIA paid $20B for Groq in Dec 2025 (IP license + acquihire, structured to skirt HSR review). Groq 3 LPU / LP30 ships Q3 2026: Samsung 4nm, 512MB on-die SRAM, 150 TB/s per chip; LPX rack = 256 LPUs, 128GB SRAM, 40 PB/s. Slots into Vera Rubin as dedicated DECODE co-processor — NVDA now spans the whole interactivity spectrum: fungible GPU pools + TileRT-style software tiers + owned SRAM silicon for the roofline software can't reach. Independent-ASIC merchant story lost its strongest name; Cerebras/SambaNova are the remaining pure plays.
- Target workload is agentic coding: AgentX benchmark replays real Claude Code/Codex traces (median 140k-token inputs, 99.2% cache-hit roofline).
Watch 2026-08-31 — OpenAI Jalapeño is the first credible pressure on this framework. The OpenAI/Broadcom inference chip went from design start to tape-out in about nine months and, in benchmarks SemiAnalysis engineers ran themselves at OpenAI's labs, delivered 1.5-1.9x more work per watt than GB200/GB300 and 1.7-3.6x lower latency across three public models — WITHOUT speculative decoding, while the Nvidia comparison numbers used it. Two things cut at this thesis directly: the chip adopts the fungibility argument internally (one homogeneous pool, deliberately no prefill/decode disaggregation — an ASIC generalized enough to be partially fungible within inference), and the kernels were largely written by OpenAI's own coding models, which attacks the software-moat leg ("the CUDA moat is potentially dead" — SemiAnalysis). What still holds the framework together: inference-only (training stays on GPUs), benchmarks were fixed-shape chat workloads with no agentic traces — the very workload this thesis says stresses real serving — only three models tested, and meaningful volume is an end-2027 story. Note the category distinction: Jalapeño is captive first-party silicon like TPUs, not a merchant ASIC, so it does not trigger the merchant-vendor kill-signals below — but if the fastest-ASIC-cycle-ever playbook spreads (models designing their own chips in under a year), the fungibility premium erodes from a direction those kill-signals were not watching.
Watch 2026-10-25 — Trainium credibly attacks TRAINING, the moat Jalapeño never touched (SemiAnalysis, Oct 23 2026). Second blow from the captive-silicon direction, and an escalation: Jalapeño was inference-only; this is research training, the workload this framework treats as NVDA's hardest moat. Evidence: TorchNeuron = native eager PyTorch (.to("cuda") -> .to("neuron"), no XLA duct tape; AWS hired Google's native-Torch-on-TPU lead), TorchTitan works out of the box (Qwen3-32B on 16 Trn2: 23.9% MFU eager / 30.5% compiled), single Trn3 core beats GB300 MFU on mid-range matmul shapes, FlexAttention 89/96 pass, 80,000 physical cores on automated CI testing numerics parity vs Nvidia across top-100k HF models, and Anthropic running production inference + RL rollouts on Trn2/Trn3. Profiler (Neuron Explorer) called better than Nvidia's. What still holds: terrible compile times (27 min for gpt-oss; 30-55x first-step overhead — fix roadmapped), BF16-only (no MXFP8), no TokenSwitch, research-scale not frontier-scale, "nowhere close to CUDA ecosystem maturity." Same category note as Jalapeño: captive first-party silicon, does not trip the merchant kill-signals — but training-moat erosion at AWS plus inference erosion at OpenAI means the premium is now being attacked at both ends of the workload spectrum. Supports the hyperscaler-surplus AMZN leg (lower NVDA tax).