Monthly Newsletter #2: Memory and Compute
Welcome to this month's edition of what I've been thinking about in the technology space. Rather than recap market movements, a few ideas that may seem underweighted in the current conversation.
Space for Time
A recent algorithmic breakthrough offers a useful window into how AI infrastructure actually works, and why the "memory bottleneck" narrative may be more nuanced than it appears.
Start with how inference works. When a model generates text, each new word depends on all the words that came before it. Naively, this means recomputing everything from scratch for every new word, which is expensive and slow. The standard solution is the KV cache, which stores intermediate calculations so they don't need to be repeated.
Here's the trade-off. Without the cache, you save memory but waste computation, recalculating the same thing over and over. With the cache, you spend memory to store those results, but generation becomes much faster. Most systems choose the latter. Memory is cheaper than making users wait.
But the real bottleneck today isn't computation. It's fetching that cached data from memory and moving it to the processor. The processor sits idle, waiting for data to arrive.
This is why a new compression framework from Google Research matters. It shrinks the cache by 6x or more, trading a small amount of computation (to compress and decompress) for a large reduction in data movement. The original trade-off was memory for compute. Now we're trading some compute back to relieve memory pressure.
The deeper point is that AI engineering is a multidimensional optimization problem. Computation, memory, bandwidth, power. Four pillars in constant tension. Solving one shifts pressure to another. Over the past three years, raw compute has improved dramatically. Memory, bandwidth, and power are now scrambling to keep pace. The interplay between them is where the real engineering happens.
The Head Node
The rise of agentic AI has sparked a renaissance for CPUs, which are now critical for orchestrating complex AI workflows. While GPUs handle the heavy parallel math, CPUs act as the head node, managing the logic and branching that decides which tools to call and when.
This trend not only counters the narrative of simple GPU cannibalization, but also marks a key distinction from basic chatbot inference: agentic AI is CPU-intensive by design. Nvidia is collaborating with Intel to develop tightly-coupled x86 CPUs for rack-scale AI systems. Arm is entering the market directly with its own "AGI CPU."
But demand eventually meets constraint. Hyperscalers are bidding up prices to secure inventory, and everyone else is getting squeezed out.
Intel is prioritizing internal wafer supply for data center over its PC business. Memory costs have doubled sequentially for some PC makers, now comprising 35% of their bill of materials. Some consumer hardware makers are dropping low-end product lines entirely. Scarcity doesn't distribute evenly. It concentrates pain where margins are thinnest.
Even if data center supply normalizes, consumer sectors and lower-end segments will take additional time to recover. Capacity doesn't instantly redirect. The AI boom isn't just creating winners in the data center. It's quietly redrawing the product roadmaps of everyone downstream.
Where do you see as the most under-weighted battlegrounds in tech for 2026? As always, I'd welcome your thoughts on any of this.
Until next month,
Jess Xu