Datacentre and cloud infrastructure is the physical substrate on which AI now runs, and the 2026 literature shows the binding constraints have moved from raw arithmetic to memory capacity and bandwidth, interconnect round-trips, power and thermal budgets, cross-layer fault propagation and multi-tenant isolation.
Summary
“Datacentre and cloud infrastructure” here means the warehouse-scale machine treated as a single computer: racks of accelerators, the memory hierarchy that feeds them, the switch fabric that connects them, the orchestration layer that places work on them, and the power and cooling plant that keeps them alive. The defining shift of the last decade is that this whole assembly, not the individual chip, is the unit of design. Google’s TPU line is the clearest documented case: across five generations from TPU v2 to Ironwood, HBM capacity and bandwidth per node rose roughly 10x and peak node performance roughly 100x, but supercomputer-level performance rose about 3600x, with optical circuit switches, built-in self test and hardware replay carrying the resilience burden that scale creates.
The mechanism is straightforward once you see where the time and energy go. A large model’s weights and activations must be moved repeatedly between DRAM or HBM, on-package caches and compute units, and across the network between thousands of accelerators. Every level of pointer indirection in a remote memory access costs a sequentially dependent network round-trip, a pattern that shows up in graph traversal, page-table walks and paged KV-cache lookups in disaggregated LLM inference. Every collective operation risks congestion at bottleneck switch ports, which in asynchronous training turns into lost updates and model staleness. And on modern multicore servers, active cores compete for a shared package-level power and thermal budget, so power rather than core occupancy becomes the scarce resource.
The parameters that decide outcomes are therefore: bytes per FLOP delivered from memory (and what a byte of HBM costs, given that hyperscale expansion now absorbs a significant share of global DRAM output); round-trips per useful operation across the fabric; watts per delivered token, including the delivery of weights, not just the arithmetic; mean time between correlated failures at 10,000-accelerator scale; and the degree to which multi-tenant isolation actually holds when tenants share the host page cache, the memory hierarchy or the local power electronics.
Research directions in the supplied sources split into four families: cheaper or exotic memory (optical recirculating delay-line “fiber memory” for immutable weights, plus better DRAM simulation and characterisation tooling); moving computation into the network or the memory-side NIC; managing heterogeneity and power in the scheduler; and hardening a substrate that is now shown to leak across container and VM boundaries and to be steerable into the electrical grid by a legitimate tenant.
Evidence base
- Five TPU generations from v2 to Ironwood delivered roughly 10x HBM capacity and bandwidth per node, 100x peak node performance and 3600x supercomputer performance over eight years, using optical circuit switches, built-in self test and hardware replay for resilience (2026-06-16).
- A hyperscale production platform accumulated 72,550 repair tickets over four years, 49% involving cross-layer fault propagation, while static per-layer test campaigns covered only 20-25% of modelled fault-propagation edges (2026-07-20).
- Generative AI memory demand plus hyperscale expansion now consumes a significant portion of global DRAM output and is raising contract pricing; a proposed optical delay-line memory claims removal of redundant weight copies across 10,000 accelerators and over 70% lower weight-delivery energy than HBM3e, on case-study evaluation (2026-07-20).
- An FPGA prototype of a memory-side NIC instruction set cut 10-hop graph-traversal latency 2.85x versus one-sided RDMA at 3.4x higher throughput and reduced page-table walk latency 62% (2026-06-15).
- Host page-cache residency was shown to leak across isolation boundaries on Docker, gVisor (systrap and KVM) and Kata Containers with QEMU and Cloud Hypervisor when tenants access host-backed filesystem state (2026-07-21).
- A legitimate cloud tenant can modulate GPU workloads to induce high-frequency power modulations that degrade local grid stability and feed back into computing services, with the distinctive signature concentrated in high-frequency components poorly captured by common telemetry (2026-07-08).
Open questions
- Does the fiber memory case study survive contact with silicon, specifically the latency variance and error rates of recirculating multi-core fibre with all-optical regeneration, or does the >70% weight-delivery energy claim hold only for immutable weights in a narrow serving regime?
- Do memory-side NIC offload ISAs like Tiara move from FPGA prototype into commercial NIC or switch silicon, and does the 2.85x latency and 3.4x throughput advantage survive at production line rates and multi-tenant scale?
- Can operators close the page-cache side channel across containers and VMs without giving up the shared host filesystem state that makes shared infrastructure economical?
- How much of the observed cross-layer fault load (49% of 72,550 tickets at one platform) is actually removable by adaptive fault-injection coverage, versus being intrinsic to multi-vendor hardware, firmware and orchestration stacks?
- Is the Bit2Watt workload-to-grid coupling detectable and mitigable with facility-side telemetry upgrades, or does it require contractual limits on tenant power modulation?
Assessment drafted 2026-08-31 from up to 18 KB sources using the technology-scorecard framework; scores are a draft read pending review.