How Do We Fix Data-Movement Bottlenecks in AI Systems? An Interview with Christos Kozyrakis

An interview with Christos Kozyrakis, recipient of the 2026 Harry H. Goode Memorial Award.

Dr. Christos Kozyrakis is the Leonard Bosack and Sandy K. Lerner Professor of Engineering at Stanford University, whose seminal contributions to cloud systems research and hardware-software co-design have fundamentally reshaped the efficiency and reliability of modern digital infrastructures.

We connected with Dr. Kozyrakis to discuss the evolution of processing-in-memory integration, utilizing machine learning for cloud cluster orchestration, and the critical systems-level challenges of scaling modern generative AI workloads.

You were an early pioneer in combining vector processing with embedded DRAM. Looking back, how did that early exploration of processing-in-memory shape your long-term philosophy on avoiding data-movement bottlenecks?

The Berkeley IRAM project taught me to look at performance through the lens of energy efficiency. We were aiming to achieve both high performance and very low power for emerging mobile computing use cases. This is also the central challenge for AI accelerators today: they must deliver enormous compute throughput even though Moore's Law has slowed and chips are increasingly power-limited. IRAM was an early example of two ideas that are now core principles in accelerator design. First, we used vector processing because vectors amortize the overheads of instruction fetch, decode, and control across tens or hundreds of operations. Second, we placed processing close to DRAM because high memory bandwidth is only useful if it does not require excessive energy. We see this direction today with high-bandwidth memory in the same package as the accelerator, and I expect memory and compute to become even more tightly integrated. The time for aggressive memory-compute integration is now.

You shifted a major portion of your focus toward cloud computing systems, pioneering the use of machine learning for cloud management. What was the core inefficiency in traditional, static cloud orchestration that convinced you a dynamic, data-driven machine learning approach was necessary?

A catalyst for this work was my sabbatical at Google in 2011, where I saw how difficult it was to keep large cloud systems highly utilized while still meeting application performance goals. The problem was not just waste; it was complexity. Application teams had to reason about heterogeneous hardware, changing load, resource interference, and tail-latency risk. The conservative and understandable response was to reserve more resources than were typically needed. Those observations motivated a line of Stanford research on automating workload right-sizing and dynamic resource management. Paragon and Quasar used machine learning to predict how an application would behave on different hardware and under different interference conditions. The scheduler could then choose a suitable machine and resource allocation, or avoid co-locating workloads that would hurt each other. This was more effective than fixed rules because it used measurements from real applications, could adapt as applications and hardware changed, and could point to the source of performance problems, such as cores, caches, or memory bandwidth. Heracles pushed this idea further by managing resources dynamically and at fine granularity within a server. Together, these projects showed how to automate cluster management to achieve high utilization with good application performance. Similar techniques are now commonplace in cloud management.

High efficiency often comes at the cost of predictability. Your work introduced specialized operating systems designed specifically for low-latency computing. How do these modern, cloud-native OS architectures minimize tail latency while maintaining the high utilization rates cloud providers demand?

In the cloud, low latency only matters if it can be achieved at high utilization; otherwise the system is too expensive to operate. Our work started by looking for the sources of tail latency that also limited throughput. In IX, for example, we used run-to-completion processing for the common case, reducing scheduling events and interruptions, and improving cache locality. Those choices improved both latency and throughput because they removed overhead from the critical path. We then broadened the ideas to more kinds of applications. ReFlex applied similar principles to remote storage, while Shinjuku handled services with more complex service times and fan-out patterns by adding microsecond-scale preemption. That was an important missing mechanism: at the time, preemption was considered incompatible with low latency, but it was critical for avoiding the latency of head-of-line blocking. The results were strong, but we also learned an important practical lesson: custom operating systems are very hard to deploy and maintain in large cloud environments. That led us to ghOSt, where we brought many of the key scheduling ideas back into Linux through a more deployable framework. The lesson was to make the latency-critical path lean, while keeping the system practical to deploy.

Your current research focuses heavily on improving AI efficiency across the entire system stack. Why is relying solely on silicon scaling no longer sufficient to keep up with the processing demands of modern Large Language Models and generative AI?

Hardware matters enormously for AI, but the slowdown of Moore's Law and the scale of AI demand mean that hardware cannot carry the full burden alone. We also have to look at systems and algorithms. On the systems side, many of the hard problems are about managing state and parallelism: moving model state efficiently, feeding accelerators with data, coordinating large numbers of devices, and recovering from failures without destroying performance or cost. As accelerators evolve quickly, system software becomes essential for extracting the performance and efficiency that hardware alone cannot deliver. The next major challenge is to co-develop AI algorithms and AI hardware more deliberately. That is difficult because algorithms and hardware evolve on different time scales and require different expertise, but I think it is the critical missing link. I am excited about work that connects hardware and system design with the development of new AI techniques, including using AI itself to help design for end-to-end efficiency across the stack.

Hardware design is undergoing a massive renaissance due to the demand for AI computation. For students and young engineers entering the systems and architecture fields today, what do you believe is the most critical skill to master: is it hardware description languages, operating system design, deep learning theory, or something completely different?

Think across the stack. Hardware description languages, operating systems, and deep learning theory are all valuable, but the most interesting problems now sit at their boundaries. My advice is to learn the fundamentals, build real systems, and measure carefully. Measurement tells you where the problem actually is, sometimes one layer above or below your comfort zone, and surprising results often lead to the best insights. Building systems is equally important because it forces ideas to confront reality: performance bottlenecks, reliability issues, usability problems, and all the details that papers and diagrams can hide. AI can help in this process by making it easier to build ambitious prototypes that showcase new ideas. But the deeper habit is to keep learning from each system you build, especially when something does not work as expected, and to explain both positive and negative results with care.