Edge AI is the practice of running neural inference on the device that captures the data rather than in a data centre, and the evidence now shows the silicon largely works while the binding constraints have moved to memory hierarchies, heterogeneous scheduling and energy observability.
Summary
Edge AI covers any arrangement where a model runs on or near the sensor rather than in the cloud. In practice it spans four orders of magnitude of power budget. At the bottom are fixed-function inference chips: a 65 nm probabilistic decision-tree engine forecasting hypoglycaemia from continuous glucose monitoring at 11.3 nJ per inference, a FeFET compute-in-memory Bayesian engine reaching 185 TOPS/W/mm² with a 640 aJ per-sample Gaussian random number generator for drone search and rescue, and DNN accelerators stacked directly into a 3-layer CMOS image sensor. In the middle sit commodity accelerators and SoCs combining CPU, integrated GPU and NPU. At the top are desktop-class boxes: NVIDIA, Dell, HP, ASUS, MSI, Acer and Gigabyte were all shipping GB10-based desktop AI systems in 2026.
The mechanics are consistent across the range. Weights are quantised, usually to INT8 or lower, to fit constrained memory; the resulting numerical error is then managed. QMC splits weights into inliers stored in dense multi-level ReRAM and outliers preserved in high-precision on-chip MRAM, on the argument that SRAM is fast but low-density, DRAM has to serve static weights and dynamic KV caches simultaneously, and Flash sits idle during inference. Where the model is sparse, as in Mixture-of-Experts, on-chip memory limits force offloading and the off-chip access becomes the bottleneck; multi-chiplet designs with high die-to-die bandwidth are being used to schedule expert streams across dies instead. Chiplet partitioning is also a yield argument: monolithic SoCs are reported to fall below 16% yield at 360 mm² on advanced nodes, motivating designs such as a 7 nm RISC-V CPU chiplet paired with dual 5 nm 15 TOPS INT8 accelerators on a 30 mm interposer.
A second research thread pushes the computation off the device entirely without going to the cloud. Analog RF computing has a base station broadcast weight-encoded waveforms; each client reuses its passive mixer to multiply them against a locally generated input waveform, performing the matrix-vector multiplications in the radio front end at very low energy. Related work on integrated sensing and edge AI has devices sense a target, extract features locally, then aggregate them over the air via analog over-the-air computation for collaborative inference, with power allocation tuned either to minimise aggregation distortion or to maximise inter-class separability.
The parameters that decide any given deployment are: energy per inference against the available power budget; whether the accuracy loss from quantisation is tolerable for the task; memory bandwidth and the split between weights and activation cache; how well the runtime exploits a heterogeneous CPU/iGPU/NPU SoC; and what happens when the backhaul link is absent, since intermittent connectivity and resource exhaustion are part of the normal operating envelope at the edge rather than rare faults.
Evidence base
- A 65 nm probabilistic decision-tree engine achieved 11.3 nJ per inference and F1 0.825 on 30-minute hypoglycaemia forecasting from continuous glucose monitoring data, fabricated silicon rather than simulation.
- An audit published June 2026 found the ASUS Ascent GX10 (GB10 SoC) exposes no CPU energy counter, no INA power-rail monitor, no IPMI/BMC and no SCMI powercap interface, with only instantaneous GPU power via NVML, while NVIDIA, Dell, HP, ASUS, MSI, Acer and Gigabyte all shipped GB10-based desktop AI systems in 2026.
- SPARROW nodes combining a low-power GPU, solar power and LEO satellite or GSM backhaul sustained 24/7 autonomous operation across tropical, temperate and montane ecosystems in Colombia, Peru, Tanzania and the United States.
- Repurposing a commercial Hailo-8L accelerator for frozen INT8 backbone feature extraction, with only an FP32 head fine-tuned on the host CPU, gave up to 15.4x faster wall-clock on-device training than a Raspberry Pi 5 CPU baseline (July 2026).
- Monolithic SoC yields are reported below 16% at 360 mm² on advanced nodes, motivating a chiplet edge design pairing a 7 nm RISC-V CPU with dual 5 nm 15 TOPS INT8 accelerators and 16 GB HBM3 on a 30 mm interposer.
- On-earbud biometric key enrolment completed in 226 ms with 98.7% authentication accuracy and FAR below 1%, using fuzzy commitment and no classifier, indicating commodity edge silicon already meets the compute requirement for this class of task (Oct 2025) 2025 Earid Ear Canal Biometric Key Extraction.
Open questions
- Does the accuracy loss from aggressive quantisation and emerging-memory device noise stay within tolerance for regulated applications such as medical forecasting, or does it force back to higher-precision on-chip memory and erase the density advantage?
- Will heterogeneous CPU/iGPU/NPU scheduling gains like HeteroMosaic’s be absorbed into mainstream runtimes, or remain research prototypes while shipping SoCs continue to underutilise their NPUs?
- Do vendors unlock per-rail energy telemetry on edge AI platforms, and if not, how do buyers evaluate the energy cost of agentic edge workloads where CPU accounts for up to 44% of dynamic energy?
- Does analog RF computing or over-the-air aggregated inference produce any measured hardware demonstration, or does it stay a physical-layer optimisation exercise?
Assessment drafted 2026-08-31 from up to 18 KB sources using the technology-scorecard framework; scores are a draft read pending review.