Shepherd Newsletter #6: The Cost of the Commute

Welcome to this month's edition of what I've been thinking about in the technology space. Rather than recap market movements, this issue digs into data movement, one of the core problems engineers are fighting right now. Shepherd runs a long-only global technology strategy, and we build conviction by working through problems like this one ourselves.

Cheap Math, Expensive Delivery

Start with what data physically is. Inside a chip, a bit is charge held in a tiny capacitor. Moving it does not mean carrying those electrons from one place to another. It means opening a switch and letting current flood the metal wire, until the whole wire rises to a voltage the far end can read.

That wire is the problem. Every wire is itself a capacitor, whether you want it to be or not, so sending one bit means charging the entire length of the wire. The energy that takes scales with distance.

The computation at the end of the trip is a different story. Once the data arrives, multiplying two numbers is just a few microscopic switches flipping in place, with almost no wire to charge. Processing electrons is cheap.

If the calculation costs one unit of energy, fetching the data from memory sitting next to the chip costs roughly a hundred, and reaching across to another server costs thousands. By most estimates, 60-90% of the electricity in an AI accelerator goes to moving data. But a deeper cost is a chip worth tens of thousands of dollars sitting idle while it waits for data to arrive. The fight against data movement is really a fight for utilization, keeping the most expensive silicon in the world busy.

Through this lens, much of the recent hardware innovation starts to look like a single campaign. Advanced packaging drags memory onto the same silicon interposer as the processor, shortening the wire. Silicon photonics swaps electrons for photons where wires stop working. FlashAttention, perhaps the most cited algorithmic advance, reorders the math so intermediate results never leave fast memory at all. Even lower precision (quantization) means every number you move is a quarter the size it used to be.

Breaking the Memory Silo

Every processing chip comes with its own private memory: the GPU has its HBM stacked beside it, the CPU has its DRAM plugged in next door. Each chip can only read its own memory.

What happens when one server's memory needs to talk to another server's memory? The request has to go through the owner: one server's CPU copies the data and ships it out through its network card, across the cable, into the other server, whose CPU writes it into its own memory. Every step consumes electricity, the data ends up stored twice, and all of it to read something sitting a few meters away.

One of the industry's responses is to break the ownership: memory moves into a shared pool that many machines can reach, and a machine that needs the data receives an address rather than a copy. The industry calls this pooled memory. It is not a full escape, though: the pool sits outside the chip package, where bandwidth is a fraction of what HBM delivers.

Most data movement actually happens between compute and memory, and two facts drive it. First, the KV cache is too large for the closest memory to hold, so it has to sit at a distance. Second, to generate each new token, data travels back and forth between memory and compute around 100 times, once for each neural network layer refining what each token means in context.

The deeper question, to me, is whether the algorithm itself will change. The number of trips is set by the model's architecture, not by the hardware; a different architecture could simply demand less movement, and hybrid variants are already being tried. But for now, the protagonist is still the transformer. As long as that holds, data movement remains the constraint the industry is paying to escape.

If any of this resonates, I'd love to hear from you, and always happy to share more of what we're observing at Shepherd, whether that's infrastructure dynamics or overlooked corners of the stack.

Until next month,

Jess Xu