Nvidia CTO Michael Kagan: Scaling Beyond Moore's Law to Million-GPU Clusters

[00:00] One of the interesting things about NVIDIA is the culture of win-win. We are not after taking a bigger piece of existing pie. We are after baking our... [00:13] a bigger pie for everybody. And our success is our customer's success. It's not... [00:21] Our success is not. [00:22] the failure of our competition. And I think fusing together conventional computing, one-human machines and accelerated computing that provided with NVIDIA, it actually gives NVIDIA and Intel channels to the market, expanding the market and serving the markets that otherwise was more challenging. [00:48] Bye. [01:05] - We are delighted to hear today from one of the legends of the semiconductor industry. [01:09] Michael Coggin, the CTO of NVIDIA. [01:12] Michael was formerly Chief Architect at Intel and then Co-Founder and CTO of Mellanox, which NVIDIA acquired for $7 million in March 2019. [01:21] In the time since, Michael has been a major driver of NVIDIA's dominance as the AI compute platform. [01:26] in large part due to the role of Mellanox and Interconnect in driving chips beyond Moore's Law.

[01:32] The AI race is ultimately a silicon race to squeeze the most intelligence possible out of each unit of silicon. [01:37] And Michael takes us down a journey about how the compute frontier has evolved, from squeezing more transistors onto a single chip, [01:44] to bringing together thousands and hundreds of thousands of chips into a single fabric connected by networking. [01:50] or an AI data center. [01:52] Michael has been driving the compute frontier forward for more than four decades, and we're honored to have him on today's show. [01:58] Okay, we're here with Michael Kagan, the CTO of NVIDIA, currently the world's most valuable company. Michael, thank you for joining us. [02:06] Thank you. My pleasure. [02:08] So I thought where we could start, our partner, Sean, [02:11] likes to make the case about every six months that NVIDIA would not be NVIDIA without Mellanox. Mellanox is the company that you co-founded some 25 years ago and have been a part of through this day. So can you kind of paint that picture for us? Why is it that the Mellanox acquisition... [02:29] was so critical to NVIDIA. [02:31] You know there is a huge transition In the world in terms of Computing and need for computing And it grows Grows exponentially And one of the Things that [02:46] We usually estimate linearly, but the world was exponential. And exponential growth now is actually accelerated. It used to be like Merlot, which is a basic silicon. And it was twice every other year. And regardless, the discussion that Merlot in terms of physics is not quite running anymore.

[03:16] Thank you. [03:16] 2010, 2011, it kicked in when GPU from... [03:21] Graphic processing unit became general processing unit, actually, where it's running. Workloads were the first time the AI workload was run on the GPU, taking advantage of the programmability and parallel nature of this machine. The requirements for performance started to grow up at much higher coefficient. [03:51] every... [03:53] three months which requires now 10x or 16x a year performance growth versus old school of 2x [04:05] twice every other year. And in order to grow this scale, you need to innovate and you need to develop solutions at much higher scale than just basic component. And that's where network kicks in. That's where network is. And there's multiple layers of scaling performance that requires high-speed networks and high-performance networks. The one is what we call scale-up. [04:34] Basically, if you're going back to the CPU days, scaling up was... [04:40] More low, more transistors, and also some advances in the microarchitecture, like auto-forestation, and at some point, multicore, and so on and so forth. So this is the basic building block of computing. In the GPU world, the basic building block is the GPU. And in order to scale it up more than you can do on a single piece of silicon with a lot of advances that we are doing with the microarchitecture and advanced technologies,

[05:10] you actually need to do something on the sort of multi-core CPU, but on much larger scale. And that's what we are doing with Envilink. This is a scale-up solution. So our GPU, what we call GPU today, is... [05:27] You need the forklift to lift it. So if you order just GPU on Amazon, just don't wonder that you'll show up this huge rack. People think chip, but it's really a system. Right. And that's just one GPU. Yeah. Okay. So. [05:45] Basic building block, a very basic computer that... [05:51] application software is running on is this GPU and it is not just [05:57] It's not just hardware. It's not just wires, but there is also a software layer that exposes CUDA as the API. And that's what actually enables to pretty much seamlessly scale. I'm simplifying a little bit the story, but seamlessly scale from one component that used to be single GPU all the way up to 72. [06:19] maintaining the same software interface. And once you... [06:24] get this building block as big as it conceivably can be built in terms of, uh, [06:31] you know, [06:33] power, cost, efficiency, then you start scaling out. And scale out means you take many of these building blocks, connect them together. And now on the algorithm level, on the application level, you actually split your application to multiple pieces running in parallel on these big machines. And that's again where network comes in and...

[06:59] So if you talk about scale up, we basically made memory [07:05] like domain to [07:07] to go beyond single compute node and a single GPU. And that's actually the first thing where Mellanox technology comes in because before Mellanox acquisition, the scaling up of NVIDIA with the Envilink was limited to a single node machine. Going outside of single compute node, it's 72 GPUs, it's actually 36 computers. [07:37] and all this as a single GPU. [07:41] connectivity outside of the single node. It's not just plugging in the wire to the connector. It's a lot of software. It's a lot of technology within the network, how to make multiple nodes to work as the single machine. And that's where Mellanox works. [07:59] First [08:00] just immediate in terms of the way we go upstream. That's the first one. The second one is how do you-- [08:08] split the operation across multiple machines. And the way to do it, if I have a task that it takes one GPU to do one second, if I want to accelerate it, I split it to 10, to, you know, thousand pieces and send each piece to different GPU. And now in one millisecond, I get done whatever I was

[08:38] split the task, then you need to consolidate the results. And every time you run this multiple times, you have multiple iterations or multiple applications running to us, so there is a part of doing communication, part of doing computation. Now, the thing is that you want to split it to [08:59] as many pieces. [09:01] as you possibly can because that's that's your speed up factor but then if your communication is actually blocking you you you you waste time you waste energy you waste everything so what you need to do you need to have a very fast communication so you split it to the many many pieces and so each piece takes very little time but then there is a another piece that is communicated and you need to feed it with this time so that's that's just pure bandwidth and [09:31] when you [09:32] Tune your application. You tune your application so that communication can be hidden behind computation. And it means that if communication for some reason gets longer, then everybody waits. So it means that what you need to do in the network, you need to have not only just raw performance like what's called hero numbers. [10:02] The time it takes is [10:04] distribution is very narrow. So if you look at other network technologies or other network products, you know, you go to the hero numbers, you know, sending bits from one place to another. It's basically physics. Yeah. So it's...

[10:19] pretty much close to everyone. You know, we are a little bit better, but that's not the big advantage. But when you do it thousands of times and it takes the same time to do it versus very wide distribution of other technologies, then machine becomes less efficient. So instead of being able to, [10:40] split your job to 1000 GPUs, you can split it only to 10 GPUs because you need to [10:47] accommodate for the jitter on the network within the communication, within the computation phase. So inherently, [10:54] Network determines the performance of this cluster. And we look at this data center as basically a single unit of computing. Yeah. Okay. Single unit of computing means that you look at this, you start architecting your components, your software and your hardware. [11:13] it. [11:13] the point where this is a data center, this is 100,000 GPUs that we want to make them work together. We need to [11:22] make multiple chips [11:25] Compute chips 2 [11:27] network chips five okay so this is a scale just you know in terms of you know what what's the what's the impact just and what's the investment you need to make to create this single unit of computing so that's that's where Mellanox technology came in and another aspect of this is there is a

[11:50] We talked about a network that connects the GPUs to run the task. But there is another side of this machine, which is customer facing. So this machine needs to serve multiple tenants. And this machine needs to run operating system. Every computer runs operating system. Another part of the Mellanox technologies is what we call Bluefield DPU, [12:20] processing unit, which is actually the [12:22] computing platform to run the operating system of the data center. In conventional computer, we have a CPU that runs operating system and runs application software. And there is many things we can talk about, you know, the advantage versus disadvantage, but there are two key things. One is how much [12:44] time do you spend on your general purpose computing to run the application? You want to maximize it. Another thing is how do you isolate your infrastructure computing from the application computing? Because, you know, viruses and cyber attacks and so on and so forth. And being able to run infrastructure, [13:06] infrastructure computing on the different computing platform. [13:11] actually reduces significantly the attack front versus [13:18] especially in the side channel attacks versus what happens if you run it on the same computer. If you remember, there was almost 10 years ago, there was this meltdown and all these cyber attacks of the side channel on CPUs, and this cannot happen, or the attack surface is reduced significantly when you run it on the different. So on the other side of the network, we have also technology.

[13:48] makes data center to be more efficient. And I, well, I may be not objective, but I do agree that this merger of Mellanox and NVIDIA, and it's actually goes both ways. I don't think that networking business of now it's NVIDIA, previously Mellanox could have been growing that significantly. - Yeah. - As it grew now, I think we are the fastest growing internet [14:15] internet business you know let alone NVLink and Infiniiband but just internet business is the fastest growing business ever yeah [14:26] What are the things that break as you get to... [14:29] 100,000, maybe eventually a million people. [14:31] GPU clusters and how do you use software to help design around that? It's a multi-stage challenge. One of the things that you need to [14:43] need to keep in mind, it's not very obvious for all the engineers that when you design the machine, they'll think how to operate it. Well, you know, you have these components and they're working and now just let's figure out. Okay, so the thing is that the hardware component works at 99.999, whatever. [15:02] percent of the time. And it's usually okay if you are dealing with single box with a couple of them. But if you are building 100,000 component machine, a 100 GPU machine, which means in terms of components, there is a millions of them, the chance that everything works is zero. So something is definitely broken. And you need to design it.

[15:27] both from hardware and from software perspective, to keep going, to keep going as efficient as you can, to keep your performance, to keep your power efficiency, and of course keep the service running. So this is challenge number one, even before we go to millions. This challenge actually starts at a few tens of thousands. [15:49] That's number one. Number two is when you are running this platform, [15:56] workloads, it is really important to, sometimes you run single job on the entire data center, and then it's, you know, you need to write the software and you need to provide all the interfaces to the software to place the different parts of the jobs more efficiently. Building networks at this scale is a very different story than building network, compute network on this [16:26] very different story than build just general purpose data center network. General purpose data center network is Ethernet. It's not a big deal. Well, it is a big deal, but it is a different deal. You are serving loosely coupled collaborative microservices that create the service that you see as a customer from outside. Here, you are running one single application on 100,000 machines. [16:56] Yeah, yeah.

[17:17] Copy. [17:18] This was another model on multiple machines, on multiple sets of machines, and run them then. [17:24] consolidated results and so on and so forth. On the inference, the story is a little bit different. But the thing is that [17:32] you need to provide the hooks [17:35] on the hardware and on your low-level system software for application and for scheduler to place the job and place the different parts of the job in the most efficient way. And as long as your machine fits a building, [17:54] which is about 100,000 GPUs now talking about gigawatts, all power driven. The challenge is that for many reasons you can't use it. [18:07] want to split your workloads across multiple data centers. And sometimes data centers are at a distance of [18:15] many kilometers, many miles. It may be across the continent. And this, you know, it comes with yet another challenge, which is speed of flight. Now the latency, [18:30] variance between different parts of your machine is dramatically different. And what is [18:36] even more challenging is that when you [18:41] Talk about networks, the congestion on the network is one of the key problems that

[18:48] deteriorate network performance. And managing congestion across such a latency difference, it's not like in old telco days you put some box at the edge of your data center with huge buffers and it's a shock absorber for congestion. Huge buffer is not good. [19:10] bigger is not better there is a famous statement from a very famous woman and so we need to these buffers are basically these buffers [19:22] devices are basically to isolate the external world from the internals. And [19:30] But when you want to run a single workload across data centers that are distant by kilometers, you need to be every machine on one side to be aware. But whom does it communicate to, whether it's short communication, long communication, and adjust all the communication patterns accordingly. So you don't need these big buffers because big buffers is a jitter. Yeah. Yeah. [19:55] And so we have a technology, we actually developed recently technology, you know, all our internet network is SpectrumX. And this is the device that we designed and developed based on the Spectrum switch that we put on the edge of the data center and it works. [20:16] Provides all the information and telemetry needed for a

[20:21] for the endpoints to adjust for the congestion. Can we talk a little bit more about training versus inference? How does the shape of workload differ when you're doing, I guess, [20:31] I guess backprop is a lot more computationally intensive, forward pass less so, but how does the workload differ? And then are you seeing customer demand start to shift from pre-training towards inference or do you think it's still very training heavy right now? And if I could just ask a quick follow-up question with that, will people be running inference workloads on the same data centers that they use for training or will these end up being two separate? Because they're different optimizations, people end up using two different sets of data centers. [21:00] Okay, yeah, that's a great question. And let me start with the first one. So training has... [21:08] two phases. One is inference [21:12] which is just for the propagation and then back propagation to adjust the weights. And for data parallel training, it's yet another phase to consolidate the results of the weights update across multiple model copies. So till recently it was... [21:32] main driver of the compute because [21:37] till not very long ago, it's maybe... [21:41] two years, which is ages in the AI era, the inference or AI was mainly perceptional. So, you show the picture, that's a dog. You show, you know, the photo of the person and here's that's Michael and that's Sonia. So, that's

[22:01] Single path [22:03] And that's it. Then became generative. [22:07] AI, where actually you get the [22:11] recursive generation. So when you [22:14] pause the prompt, then it's not just one inference. It's many inferences. Because for every token, when you generate text or generate picture for every new token, you need to go through the entire machine all over again. So instead of one shot interference, then there is more. And then now there is a... [22:35] A reasoning, which means machine starts, you know, sort of thinking. Yeah. If you ask me what time is it now, I can tell you it's easy, right? What time is it now? But if you ask me more complicated question, then I need to think. [22:50] probably need to wait or compare multiple solutions or multiple paths. And every such a thing is inference. Every such a thing is inference. And inference itself has actually two phases. One is much more computer intensive and the other one is memory intensive. It's what we call pre-fill because when you do the inference, you have some sort of background, right? [23:20] which is some relevant data that you need to process and create the context to generate the answer. And this is very compute intensive. It's not much memory intensive. And the other part is actually generating the answer, which is the decode part of the inference, where you generate token by token. Okay, well, there are some technologies that you can generate more than one token,

[23:50] single path is much less than the final answer. [23:55] So if you combine all these things together, inference demand for computing is actually not less than training. It's actually even more. And there are two reasons for this. One is that what I explained that there was a... [24:12] much more computing than it used to be for the inference. The other thing is, you know, you train model once, but you infer many times. You know, ChatGPT, you know, billion of people or it's almost billion of people, right? There's customers, they are pounding them all the time in the same model. They trained it once. Now they're making videos. Right, right. Now they're making videos and you can generate and, you know, everybody is doing the inference. [24:42] once she discovered this as her best friend. [24:48] So in terms of things, now to your question about machines, you can infer on the phone. Okay, so there is definitely going to be much smaller scale installations for inference. Yeah, it's like mobile devices. [25:12] data center scale and on the, it's a [25:18] Efficiency of the programming, programmability, is much more viable than optimizations for hardware. And, you know, every hardware...

[25:32] Yeah. [25:33] instance it has its own cost and its own drawback. So as long as you don't identify and I don't think-- besides this, we actually did-- it's very similar GPU. It's same programming model as the GPU for pre-fill versus decode. [25:57] I think, I don't remember when it happened, but actually we announced that we are building the GPU SKU that is [26:07] Optimize for pre-fill So you will have It can do [26:12] decode and decode gpu can do pre-fill so but you can equip your data center with the skews or that pre-fill versus skews that uh that are for decode to to optimize for like typical typical use but if your workload shifts for more decode or for more pre-fill you can use uh either one of [26:42] importance of programmability. The same interfaces for GPUs, it's based on code and up, [26:50] which is, that's what made Nvidia Nvidia before Mellanox. - Yeah, yeah. Can I ask you a question about data center scaling? [26:57] So for many decades, we had Moore's Law, and chips got more and more dense and produced better and better performance. And then we ran into the laws of physics, and chips just couldn't get more dense because their quantum mechanical properties caused them to break down. And so then we had to scale up to the rack level, and now we've got to scale out to the data center level.

[27:17] Is there some analogous law of data center scaling [27:22] that says when data centers get too big, the communication overhead causes the performance to break down? Or just said differently, or maybe said more simply, is there a natural limit to how big data centers can get? I think there is a practical limit of how much energy you can consume within a given size of the data center. If you were surrounded by nuclear power plants and the energy was available, would the data center itself perform? [27:52] I don't know. I'm not an expert in the construction even. But if you surround, there's energy coming in, now the heat is going out. So there is a whole, we are now basically moved pretty much entirely to the liquid cooling. And one of the reasons we did it is to enable much denser. [28:13] denser compute power. We couldn't build as dense computing as we're building now with air cooling. So there's a whole bunch of technologies coming to help this more and more. So now the last big data center, which is like XAI scale, is 100 or 150 megawatt. Now we're talking about gigawatt data centers, people talking about 10 gigawatt data centers. [28:43] So, you know, there's a looking forward to build much, much bigger data centers. Are you sending the data centers to outer space?

[28:50] Pretty cool. [28:57] determines the speed of data center deployment is how fast concrete gets stable. [29:06] Fair enough. So before starting Mellanox, you were at Intel. That's right. 16 years? 16 years. You became chief architect. Yeah. [29:15] NVIDIA and Intel recently announced a partnership [29:18] Can you share a little bit about what the vision for that might be? [29:21] The starting point is that computing changed [29:24] Thank you. [29:25] in last decade or a little bit more than decade. NVIDIA started as the accelerated computing company, and [29:35] video games was the first and then it evolved to AI which is the new way of data processing. So you cannot [29:45] just general one human machine just is not capable of of [29:52] being used as a platform to solve the problem like, you know, [29:57] Programming when human machine is just explaining Somebody what to do I can explain many things and I can Explain many people What to do but I can't Explain how to distinguish Between cat and dog So there is new challenges that AI solves and you need acceleration There and [30:18] our partnership with Intel is actually fusing accelerated computing with the general purpose computing because general purpose computing is not going away. Everything will be accelerated, but we accelerate the general purpose computing. We accelerate the applications. And, you know, x86 is the architecture that is dominant there. And, you know, it would...

[30:44] serve greatly both companies. That's actually one of the interesting things about NVIDIA. It's the culture of win-win. [30:52] We are not after taking a bigger piece of existing pie. We are after baking our pie. [31:02] a bigger pie for everybody. And the success, our success is our customer success. It's not, our success is not, [31:13] the failure of our competition, our success of our customers and success of our ecosystem. And I think fusing together conventional computing when human machines and accelerated computing that provide with NVIDIA is actually possible. [31:32] It's probably open yet another dimension that I'm not sure what it is, but it basically gives, you know, on the... [31:39] practical short term view is it gives [31:45] NVIDIA. [31:46] And Intel... [31:48] channels to the market or expanding the market and serving the markets that otherwise was more challenging. [31:56] You mentioned the culture of NVIDIA. So when Mellanox became part of NVIDIA in 2019, the market cap of the combined company was about $100 billion, which is no joke. But the market cap today is about $4.5 trillion. And so 45X growth in value in six years is pretty phenomenal. How has that changed the culture of NVIDIA? How is NVIDIA different today now that it's one of the

[32:26] ago. Yeah, but this, you know, when we just joined, Jensen was in Israel and I presented him, you know, that's, I believe that one plus one will be ten, and I actually was off by a factor of four. [32:42] So, but Mellanox and NVIDIA in a sense, it's sort of similar. The culture is very similar to begin with, but there are some, there is nothing [32:56] And I was the only founder that left in Mellanox after Real resigned a few months after the acquisition. And my main focus at the beginning, you know, things like what do you think about in the shower, was how to make sure that this acquisition will succeed. Yeah. You know, NVIDIA paid $7 billion for a company that I founded. [33:26] all the mixed feelings that were there, but once it's done, it's done. Now I have to make it successful. So eventually it worked. Most of the Israeli employees are stayed. I think, 85 or 90% of regional employees stayed. Actually, [33:49] Nvidia grew more than 2x in Israel in terms of manpower. Yeah. So we're growing and we are announcing that we're actually going to build a campus in Israel, a new campus for Nvidia. And so that's where I think the overall merger was very successful.

[34:16] I... [34:17] I did... [34:17] I did my best to make sure it succeeds and besides the technology that I was looking at this part of which is sort of technology but it's technology and theology and there's many other things to make sure that people are comfortable that from being in the... [34:40] center of Mellanox, which is the headquarters of Israel, don't feel left somewhere far away. And Jensen is [34:50] basically emphasizes the networking is the critical part of NVIDIA's success. And he's right. [35:01] So I think it's considered to be the most successful merger in the history of the technology. You guys probably track the things better than I am, but overall I think it was a great move. What are the science fiction things that you spend your time thinking about? Just even wondering, like for example, optical interconnects, do you think... [35:25] Do you think that will exist? Do you think AI will ever be better at physics than us and better at data science than us? Well, what I'm thinking, you know, if you look about... [35:36] science fiction is how to make history to be experimental science. You know, you can physics do, try something and then, you know, see what works and then try something else in history. Time goes one direction, but you have a good simulation of the world. You can make history experiments. We have an Earth 2 climate simulator. And with this type of technology, we can actually simulate how,

[36:06] now. Okay? [36:08] Experimental science. You try something, you see what happens 50 years later. [36:14] So... [36:16] That's the science fiction part. Yeah. And the physics, now we are moving from reasoning and so on and so forth. Now once we... [36:26] get AI models to understand physics, [36:29] We actually can learn physics. Yeah. [36:33] AI can teach us physics because the way we get to the laws of physics is that we observe, theoretical physics, right? You observe some phenomena and you generalize it and you compose the rule that basically the law, the physics law that... [36:53] stays underneath of this phenomena. And AI is really great of understanding [36:59] generalizing and data processing and observing so i can help us to get to know some laws of physics that we don't even imagine now yeah uh moore's law was 2x every two years [37:12] Huang plus Kagan's law is what is the slope and how long do you think you can sustain it? [37:17] Well, the slope is somewhere in the... [37:21] range of [37:22] 10x or few orders of magnitude a year and that's what we are doing by the way now we are we are we are since about two or three years ago we accelerated our product introduction from every other year to every year now we introduce a new new new wave of products every every year and it's a

[37:45] order of magnitude, higher performance. And it's not on the cheap level performance. It's on the machine that you can build with this performance. That's what we are looking at. It's a single unit of computing. And how long it will stay, I don't know. I don't know. But we'll do our best to maintain it as long as needed and probably even accelerate. It's all about exponent. It's all about exponent. It's hard to imagine. [38:15] Mulo curves or any row curves. They usually plot it on the logarithmic scale. So it looks like linear. But that's wrong thing to look at. You can't predict. [38:26] what's going to happen? Who could predict that when iPhone was first introduced or smartphone was first introduced, you know, [38:36] That's 15 years ago? 2007. Yeah. 2007. Oh, 17 years ago. Okay. Who could imagine that this... [38:45] smartphone, the least used function, at least for me, is a phone. It's e-commerce, it's texting, it's news, it's mail, it's basically running your life from this machine. So your authentication, your ID is there. [39:03] So, you know, now who can imagine what's going to happen, you know, 10 years from now with all these developments that we are doing today. But we are building the platform for innovation. [39:14] Thank you. [39:15] What is the...

[39:16] Your commentary on who can imagine notwithstanding, what is the most optimistic view of our future with AI? [39:24] that you like to think about. [39:25] Like what could AI do for the world 5, 10, 15 years from now? [39:30] Steve Jobs called computer to be the bicycle of mind. Yeah. Okay. So AI is, it's maybe... [39:40] I don't know if it's... It's probably a spaceship. [39:45] Because... [39:46] There's a lot of things that I would like to do, but... [39:49] I just don't have enough time, don't have enough resources to do it. With AI, I will have it. And it doesn't mean that, you know, I will do twice as much. Maybe I will do 10 times as much. But the thing is that I will want to do [40:05] 100 times as much as I want to do today. And that's where, and that's where, and you know, you go to any project leader and nobody says, you know, I have enough. I have enough power. I have enough resources. I don't need any more. Okay. If you give him a resource which is twice as efficient, he will do four times more. Yeah. And he will want to do 10 times more. So it's like electricity changes the world, right? [40:35] lamps and this infrastructure to use the gas as the source of energy. [40:43] Who could think that once this electricity was invented, it will change the world, that we can't live without electricity?

[40:51] The same with AA. [40:55] Thank you so much for joining us today. I love this conversation. Thank you. Thank you. Thank you for having me. [41:00] Music