The “CUDA moat” is NVIDIA’s accumulated software stack and code corpus, which makes its GPUs the default compute target; its durability now hinges on whether LLM-based kernel synthesis can port performance to low-corpus silicon, and as of mid-2026 the published evidence says it cannot.
Summary
CUDA is NVIDIA’s proprietary programming model for its GPUs, plus the libraries, runtimes and domain platforms layered on top of it. The “moat” is not the language itself but three compounding effects. First, hand-tuned kernels and libraries that competitors must re-implement to match performance. Second, ecosystem default status: papers across unrelated fields ship CUDA-first implementations, from diagonal-sparse Hamiltonian simulation kernels in C++/CUDA to GPU fully homomorphic encryption via CUDA-Python bindings, and NVIDIA has extended the brand into adjacent domains with CUDA-Q for quantum simulation, used in protein-folding VQE, 40-qubit chemistry pipelines and quantum-network control studies. Third, and newest, corpus asymmetry: because almost all public GPU kernel code is CUDA, frontier language models are far better at writing CUDA than at writing kernels for anything else.
The third effect is what turned the moat from a commercial observation into a testable technical question. The bull case for challenger silicon (AMD, Ascend, NPUs, ASICs) is that porting cost is collapsing because agentic models can synthesise kernels and compilers on demand. The bear case is that models only look good where a large public corpus already exists, so the moat is self-reinforcing precisely in the dimension supposed to dissolve it.
The parameters that decide the outcome are: (a) how much of delivered inference performance actually rides on kernel quality rather than silicon, (b) whether LLM kernel synthesis generalises from CUDA to low-corpus instruction sets and programming models, (c) whether portable abstractions such as OpenMP target offloading, HIP or ONNX Runtime can reach hand-tuned CUDA performance without per-vendor work, and (d) whether hardware evolution (chiplets, confidential computing) invalidates CUDA’s own abstractions faster than the corpus accumulates.
On (a) the evidence is that kernel and runtime quality is worth large multiples of a percentage point: a task-based megakernel runtime on AMD MI350 cut Qwen3-8B decode latency 1.3-1.5x below vLLM at small batch and lifted L2 hit rate from 12% to 54% at batch 32, cutting HBM traffic by up to 37%. Software, not silicon, was the variable.
Evidence base
- 2025-11-20: KernelBench and TritonBench show state-of-the-art LLMs struggle to generate correct GPU kernels and perform substantially worse than expert-crafted kernels ref.
- 2026-03-15: GPT-5.2 scores 92% on CUDA level-1 kernel tasks but 14% on Ascend C, and fails entirely on level-2 tasks for emerging hardware ref.
- 2026-06-22: Dated arc of automated kernel synthesis: layer-norm 4.8x over PyTorch and Conv2D to about 180% (Jun 2025), but Flash Attention 9% and FP16 matmul 52%; Kevin-32B moved correctness 56% to 82% and mean speed 0.53x to 1.10x (Jul 2025); KForge targets non-NVIDIA accelerators (Nov 2025) ref.
- 2026-06-05: CUDA and HIP both expose a flat execution hierarchy that cannot express chiplet locality; a task-based runtime on AMD MI350 with Qwen3-8B cut decode latency 1.3-1.5x versus vLLM and raised L2 hit rate from 12% to 54% at batch 32, cutting HBM traffic up to 37%.
- 2026-06-24: On NVIDIA B300 confidential computing, BF16 matmul runs at 0.998x of non-confidential performance, yet LLM serving loses 13-27% of throughput because secure copies do not gain CUDA-stream concurrency; small alloc-and-copy operations are 44x slower.
- 2026-07-29: A portable LWE KEM implementation using OpenMP target offloading runs one source base on both NVIDIA and AMD accelerators, in contrast to most existing GPU implementations which rely on CUDA-specific optimisations.
- 2026-07-07: Graph-VQE responds to CUDA-Q lacking Qiskit support and restricting parallelisation by building a custom Qiskit-CUDA-Q integration layer rather than leaving the platform.
Open questions
- Do cold-start drafting plus continual refinement approaches close the 92%-to-14% CUDA/Ascend C gap, and how much target-specific human-authored kernel data does that still require?
- Can KForge-style synthesis for diverse accelerators (Metal, OpenCL) reach expert performance on hard kernels such as Flash Attention and FP16 matmul, where automated CUDA generation still hits only 9% and 52% of expert performance?
- If CUDA’s flat execution hierarchy is already inadequate for chiplet GPUs and confidential computing, does the corpus advantage transfer to the next programming model or reset with it?
- Do portable single-source routes such as OpenMP target offloading close enough of the performance gap on NVIDIA and AMD to remove the need for per-vendor hand-tuned kernels in production inference?
Assessment drafted 2026-08-31 from up to 18 KB sources using the technology-scorecard framework; scores are a draft read pending review.