An AI accelerator is purpose-built silicon that runs neural network and inference maths far more efficiently than a general-purpose CPU, and the current research frontier has moved away from raw multiply-accumulate throughput towards moving data less: memory locality, interconnect, and in-network or photonic fabrics.
Summary
An AI accelerator is a processor whose datapath is specialised to the arithmetic that machine learning actually performs, chiefly dense matrix multiplication (GEMM) plus a small set of nonlinearities, quantised to low precision. By dropping the generality of a CPU, an accelerator can spend most of its area on multiply-accumulate units and on the memory hierarchy that feeds them. The category spans several very different physical implementations: digital ASICs and GPUs at the datacentre end, FPGAs where flexibility or radiation tolerance dominates, tiny fixed-function blocks stacked directly onto image sensors, event-driven neuromorphic cores for spiking networks, and analogue optical tensor cores.
The parameter that now decides accelerator design is not FLOPS but where the bytes live. On multi-chiplet GPUs, memory is split into local and remote high-bandwidth memory across an interposer, and inter-chiplet traffic for the same GEMM shape varies by up to 58x depending on operand layout, thread-block traversal order and data placement. Fixing the memory layout so that chiplet-local data is contiguous cuts remote HBM traffic by an average of 13.0x on Qwen 3 30B and 20.7x on Llama 3.1 70B GEMMs relative to 4 KB page interleaving. Mixture-of-experts models make this worse: only a subset of experts fires per token, so expert loading latency, not compute, becomes the dominant cost.
Two escape routes from the memory wall appear in the sources. The first is to attack the beachfront constraint that fixes the memory-to-compute ratio on every current accelerator package, by replacing a local HBM stack with a photonic chiplet into a shared switched pool: a described appliance offers up to 32 TB of shared memory and 115 Tbps of all-to-all digital switching, with simulated gains of up to 3.66x throughput and 1.40x latency in LLM inference and no significant change to the GPU core. The second is to push computation into the network or memory side, a lineage running from Fetch-and-Add in the NYU Ultracomputer to NVIDIA SHARP and HPE Slingshot, and now to memory-side NIC instruction sets that collapse dependent round-trips, cutting 10-hop graph traversal latency by 2.85x at 3.4x higher throughput on an FPGA prototype.
Evidence base
Open questions
- Can a photonic tensor core hold multiplication error near 3.9% while scaling from a 4x2x1 unit to matrix dimensions used in production transformers, or does error grow with channel count?
- Do the simulated 3.66x throughput and 1.40x latency gains from replacing local HBM with a photonic memory fabric survive measurement on real silicon rather than CelestiSim?
- Will vendors expose per-rail energy telemetry that firmware already computes, so that accelerator efficiency claims become verifiable in procurement?
- If CPU-side processing accounts for up to 90.6% of agentic latency, how much of the accelerator gain is recoverable at system level rather than kernel level?
Assessment drafted 2026-08-31 from up to 18 KB sources using the technology-scorecard framework; scores are a draft read pending review.