Snowflake CEO Sridhar Ramaswamy on Using Data to Create Simple, Reliable AI for Businesses

[00:00] the product that makes [00:02] even. [00:03] The people that go, "I have GPT-4, I have an army of software engineers." The thing that even they struggle with is things like a reliable talk to your data application, [00:13] Because even with GPT-4 out of the box, you end up getting 45-odd percent reliability, meaning it gets half the questions wrong when it tries to answer it. We are well in the 90s, and we are racing to get to like 99% reliability on talk-to-your-data applications. Obviously, we restrict the domain and turn this into more of a software engineering problem than just like a pure AI model problem. [00:43] want that because even the people with the money and the resources to spend on software engineering teams very quickly realize that this is a wall that they are likely not going to break through. [00:55] *music* [01:11] Today, we're excited to welcome Sridhar Ramaswamy, CEO of Snowflake. [01:16] Snowflake is one of the most important enterprise companies in the public markets. It's the default cloud data platform. [01:23] But today... [01:25] The question is, [01:26] of what role does Snowflake have to play in the world of AI?

[01:30] looms large. [01:31] Sridhar is somebody we've known for a couple decades. He actually started [01:35] on the very same day as our partner Bill Korn at Google [01:38] back in April of 2003. [01:40] We backed Shridhar in his own startup, Neva, which was an AI-driven search engine. [01:45] Snowflake acquired Neva, which is how Sridhar became the successor to Frank Slutman. [01:50] rarely have we encountered somebody who is as in the weeds on the technology but also [01:56] as commercially savvy as Sridhar. [01:58] And he will join us today to talk about what AI means for Snowflake, [02:02] the importance of safety nets, [02:04] the open source community, [02:05] the competitive landscape, [02:07] and the practical applications of AI [02:10] that he's seeing in the enterprise through his lens as CEO of Snowflake. [02:14] We hope you enjoy. [02:15] All right, Shreda, we're excited to have you here with us today. You're a technologist by trade. You've spent a lot of time in the consumer world, and you are now at the helm of one of the most important enterprise companies of our generation. So before we jump in, we have a lot that we want to know about enterprise AI, what Snowflake is up to, some of your predictions on the world of AI. Before we jump in, though, just a level set, can you give us a couple words on your personal background? [02:45] but just for fun for people who aren't familiar what's snowflake so who should we are what's snowflake let's start there [02:51] That's great. Pat Sonja, super excited to be here at [02:56] iconic Sequoia, home to many, many legends I admire. Yeah, I'm a computer scientist by training, early career as an academic. I joke to people that I'm a reformed academic because I was like, I wanted to do things with more impact. Super lucky to be an early part of Google, where I joined one of the greatest businesses ever invented by humanity, which is the search

[03:26] in commerce at Google. [03:28] for five years, helped grow that business from a billion and a half to over $120 billion in revenue. And then, funded by Sequoia, did an ambitious startup called Neva, which wanted to modestly rethink what search meant before getting acquired by Snowflake and becoming its CEO. And Snowflake is the AI data cloud. [03:52] Our core thesis is that a cloud computing platform that puts data at its center [03:58] is going to be way better for enterprise customers to act on data than a generic cloud. And AI, of course, we think of as a transformational technology that is going to change every aspect of how data is stored, how it gets moved around, and of course, how it's accessed. We have over 10,000 customers, made 2.6 bill last year, but we're going to be able to do that. [04:25] at the center of everything enterprise and data. That's a super quick blurb. Perfect. Thank you. And so you have 10,000 or so customers. I know you've met at least 100, probably hundreds of them since you took over. I've met hundreds of them by now. So there you go. So I'm guessing you have a pretty decent read on what's going on in the world of enterprise AI. So maybe we'll just start there. What's going on in the world of enterprise AI? What are you seeing at your customers? First of all, [04:50] People get that this is going to be transformational. You know, lots of technologies have skeptics. I'm sure you've run into folks who are like, ah, mobile, it's not going to be a thing. This browser, like so lame. It takes a while for people to absorb. I think what's different about AI, first and foremost, is technology.

[05:08] People are like, [05:10] I get what this can do. [05:12] I think some of the power is just like honestly looking at the magic that ChatGPD is. Anyone that like has interacted with it, asked it to write a poem, asked it to create an image, knows like, wow, this is something that's very special. So the level of awareness is incredibly high. And we have thousands of customers that are in various stages of implementing AI solutions. [05:42] very excited by the idea of giving business users access to business data without going through like an elaborate, you need an analyst, you need a BI tool, you need blah, blah, blah, you need a week before a change can be made. They're like, I just want to put data into the hands of people that need it right now. But we also have dozens of people that are using AI as a transformation engine. So for example, if you have unstructured data, whether it's an image or let's say like [06:12] project to figure out [06:14] What's this image about? Now you feed into a model, ask it a question, and you get the answer. And so people are super excited by things like that. We have a product called Document AI, which extracts structured information from documents, say like contracts. All of us have contracts sitting around in our company folders that have all kinds of magic numbers. [06:32] that ideally you want to do analysis on. So there's a wide variety of cases that people are implementing and sending into production. But I would say stuff at the bottom, which is how do you transform data more effectively, more flexibly, and stuff at the top, which is how do you make data easily available to all kinds of business users in new ways, in interactive ways. I would say that's the sandwich in terms of what are people wanting to do with data. And can you say

[07:02] Snowflake's right to win. So some of the things you mentioned, like data transformation, for example, feels like that is very close to the core business of Snowflake. But then there are some things that are maybe a bit further afield. You know, if somebody wants to deploy an enterprise agent of some sort, they can use Snowflake to do it. But what's Snowflake's right to win in that situation? So can you just say a couple words about how Snowflake fits into this overall landscape and sort of the right to win? So first and foremost, the basic approach that we took [07:30] to AI sort of enabling or infusing AI into Snowflake is it should be an accelerant for everything that you do with Snowflake. That's what Cortex AI is. It's a model garden, but it's more than that. Snowflake prides itself on super tight integration of its various product features. And this is not another service that's part of Snowflake. It's built into Snowflake. This means [08:00] has access to SQL has access to AI. And so it's a massive democratizing mechanism. And then the early applications that we have built, like Document AI, are a very natural next in the progression of what people want to do, which is, hey, I want to act on the data that is within Snowflake's purview. By both expanding the data that Snowflake has access to via things like [08:30] but then providing things like document AI, we just make a whole bunch of AI applications that previously used to be software engineering projects into two commands that an analyst can issue. And so our first lens very much is that

[08:48] AI should become easy. [08:50] very trivial for data that is sitting in Snowflake. 100%. There are going to be applications that are, you know, cutting edge or going to involve many, many different services. But the angle that we bring to all of those customers is we make reliable AI. And there's a topic that we can get into. So, for example, I tell people you have no business believing the raw output of a language model for [09:20] it's ungrounded. It doesn't understand truth from falsehood, doesn't understand authority. So we make things like, you know, creating a grounded chatbot. Again, as I said, two commands, not a software engineering project. Similarly, with Cortex Analyst, which is our talk to your data API, we bring the full power of, we know everything about the schema, all the queries that have been run on the schema, the semantic context on the schema, we can produce a reliable, you know, [09:50] So we are leveraging our strengths in data to make AI products better. Are there going to be like specialist applications that can only be done with GPD Photo and a custom integration with a bunch of other stuff? Absolutely. But that's not what we are after. The bulk of our customers want to get work done. They're not in the business of doing research with AI. [10:13] And are you seeing customers bring net new data that maybe didn't sit inside Snowflake historically in Snowflake because of your AI services? And how do you think about your right to win as it comes to the data that's not in Snowflake yet? This is a broader question. I think one of the things that I've actually been a good part of is in expanding the lens of data that Snowflake should play in. Snowflake, as you know, is, first of all, it's closed source software for the most part. The code engine is closed source, just like search.

[10:43] But we also had a proprietary storage format where data was ingested into Snowflake and kept in this format. But what we consistently heard from customers, and I'm sure like you hear all the time, is there is 100 or 1,000 times more data sitting in cloud storage than there is inside a specialized player like Snowflake. And more and more industry trends have been towards interoperable data. [11:13] from multiple places. So, for example, if they want to write their own bespoke applications, most people don't want to do that, but the biggest ones do. They want the data to sit in cloud storage, where, yes, Snowflake perhaps can write it and read it, but other applications should also be able to read it. So we made a big push around Iceberg, which is the interoperable format. We also announced a cloud catalog recently. The idea is that in 10 years, data is going to be sitting on [11:42] mostly in the cloud, mostly in cloud storage, which is very cheap, mostly in interoperable formats, accessible via open catalogs. And this is the place where we see there being so much more access to data from Snowflake. So everything from data engineering and AI now comes into our purview. We have customers that, for example, are doing things like, oh, let's run a video model using Snowflake's container services on data that is sitting in S3, [12:12] into Snowflake. So it's just a very different world we are playing in. Makes sense. And then let's say for the data that's currently sitting in one of the hyperscalers, for example, you started the conversation by saying, you know, the core tenant of the company is that when you build your infrastructure kind of all around the processing of data, you can do better things. What are some of the ways that you're able to kind of offer better AI services around the data that doesn't currently sit in Snowflake, but that you're hoping customers will bring in versus, you know, what the hyperscalers are doing already?

[12:42] onto that real quick, because one of the things that we have heard from customers is at either end of the spectrum, you've got at one end of the spectrum, work directly with open AI, send your data, you know, into their, into their cloud, um, and maybe have some nervousness around whether that data is going to leak into the model or whether they have the right security and privacy sort of governance around it. Um, at the other end of the spectrum, you can just do everything yourself, grab a model off the hugging face, build it internally, you know, super safe, super secure, [13:12] to do all that. And then the middle ground, you've got [13:16] Amazon Bedrock, or you've got a Snowflake. And they both kind of have a value prop of best of both worlds. We're going to make it easy for you, but it's also safe and trusted and secure and all that good stuff. And so I think my angle on Sonia's question is like, for somebody who's making a practical decision about sort of what should I build in Snowflake versus what should I build on Bedrock or a comparable cloud service, what leans people in the direction of Snowflake? [13:46] that you want, whether it is data security, data governance, ease of use, all come out of the box. The incredible power that comes with core Snowflake's platform, including things like collaboration, other third-party applications. We make AI simple. [14:05] 100% there are those people that will say, I want to take data that's sitting in cloud storage or even in another application. I want to bring it into cloud storage. I want to recreate ACLs, you know, access control list. And then I want to create a vector index using a bespoke, you know, vector indexing solution.

[14:35] host myself. And then I will use Langchain and write custom routing logic for my application. [14:44] I can assure you that, you know, 99.9% of our customers want no part of this. [14:51] You know, that's just the reality. [14:53] All those poor people wanted was a chat bot to run on 100,000 docs that they have so that they can replace the annoying search box for FAQs on their site. [15:05] with, you know, here's a solution that just works. And our take is, yes, whatever governance you've had before works out of the box. And your data does not go anywhere else. You have the same rock solid guarantee that Snowflake will never use your data to train any data. [15:24] like cross-customer model. [15:27] And we will be very efficient and cost effective from just like overall cost of running the solution. [15:35] But Snowflake's magic, honestly, is we make the hard simple, and it's things like total cost of ownership. Many of our customers, you know, are banks. They are healthcare institutions. They are finance, you know, or other kind, like we play a lot in the media space as well. Most of our customers want to solve problems, not solve technology for the sake of technology. You know, we have a foundation model team.

[16:05] How do we, you know, get models that are better grounded generation? How do we get them to follow directions well? How do we get them to say no to questions that they should not be answering when it comes to, let's say, like talk to your data? You know, so we focus on specialized areas like that. But the biggest reason to use Snowflake for a lot of our customers is Snowflake. [16:28] 10% software engineering project with a whole lot of risk about data and security and what else can happen, turns into six hours of work for an analyst. We are good at that. We are proud of that. [16:40] So it sounds like the... [16:41] One-liner might be, it's kind of the level or the layer at which you're intersecting these products. If you're working with one of the public clouds, you're still very much at the infrastructure layer, building a lot yourself. Snowflake, you're at the platform layer. A lot of the hard work's been done for you. And our long-term bet, Pat and Sonia, is that ecosystems move upstream. [17:01] There was a time not so long ago where, I don't know, our parents, our grandparents knew every part of a car. They were like, oh, so manly to change a carburetor and get oil in between your nails. I got to be honest with you. I'm still impressed every time my dad knows exactly what is wrong with the car. Yes. You know, while I'm willing to go to, you know, go to strength training every day, getting oil in between my fingers, you know, with my car does not sound so attractive anymore.

[17:31] like I have a model garden here, I have a caching service there, I have a database here, I will stitch all of this together. [17:37] Um [17:38] is that everything turns into a software engineering project. For us, you're like, no, that's just a little data pipeline that you set up. And here is a beautiful UI that you get if you want a chatbot. [17:51] Obviously, you can do more, but you don't have to. [17:54] Yeah. Whether your customer is building on Snowflake and are there certain types of AI applications that are better suited to be built on Snowflake than others? As I said, the categories of AI applications come naturally from the kind of data that, you know, that are already there. I would say the broadest, broadest use case is really using Cortex AI via SQL in either interactive queries and dashboards or in jobs that people are running. [18:24] And so these span the gamut from, oh, let's do sentiment detection with a small model. It doesn't really have to even be that expensive. So that's just like literally it's one function call. Or let's do other kinds of data extraction where, as I said, you have things like a transcript or maybe clinician notes. You take that out and, you know, you get structured data from it. [18:49] Or the other thing that I talked about, document AI, which is you extract structured data from things like receipts, from contracts, so on and so forth. That's kind of our sweet spot. [19:00] But I have to say, the product that makes even products,

[19:04] the people that go, I have GPT-4, I have an army of software engineers. The thing that even they struggle with is things like a reliable talk to your data application, because even with GPT-4 out of the box, you end up getting 45 odd percent reliability, meaning it gets half the questions wrong when it tries to answer it. We are well in the 90s, and we are racing to get to like 99 percent [19:34] and turn this into more of a software engineering problem than just like a pure AI model problem. But that's the thing that makes every Snowflake customer perk up and go like, [19:45] I want that because even the people with the money and the resources to spend on software engineering teams very quickly realize that this is a wall that they are likely not going to break through. [19:56] And how do you accomplish that? Like maybe peel back for us how you're able to get to the 90s percent. Are you training your own models? Are you just tell us about, you know, how this all becomes possible? It's systems design. [20:09] Just like the magic of how you make a coding agent or an effective copilot work in practice, it's not always like the giant models. It is carefully breaking problems down so that you present the right context to the model. It's in deciding things like, oh, I see, the problem of whether to answer a question is different from how to answer the question.

[20:39] So you can specialize and have different models for these different subtasks. And also, what's the... [20:47] Basically, I call this like a problem definition, a product structure question. We structure the product of Cortex-Unless so that it is more restricted than a free-flow domain. What I mean by that is schemas are weird things. [21:05] People do random stuff. They have horrible column names that mean completely the opposite. Every company has its own definition for revenue. And if you take the best model on the planet and let it loose on an arbitrary schema, the likelihood that it's actually going to understand the nuance of what's in there is, [21:23] close to zero. [21:24] Like in our big deployments, for example, our customers have 200,000 tables. [21:29] And you can bet that there are several tens of thousands of tables with the word revenue in it. [21:34] They just don't have the same meaning. So it's really like problem definition. To me, by the way, this goes back to the magic of product. I think of like any amazing founder, any amazing product manager as someone that can visualize what's like the right tradeoff to be making in order to create something that has broad applicability. [22:04] can answer every question. But obviously, there's a precision recall tradeoff there. You can get 100% precision by answering no question. That's not the goal. You want to be useful, but still be precise. But it's a lot of software engineering.

[22:17] Um, [22:18] I want to go in a slightly different direction. Sure. Okay, which that reminded me of this, and I don't know why. But you guys, you seem, the product velocity at Snowflake seems to have inflected to the positive. Yeah, 100%. Even in the last six months or so. And we've worked with a lot of founders where, you know, the bigger the company gets, the slower and slower the velocity becomes. And so I guess I'm curious. [22:43] What have you guys done to positively inflect product velocity? Because that's hard to do when you're dealing with an organization at the scale of Snowflake. I've done this many times before. And the formula is always roughly the same, which is first and foremost, you make sure that you have a safety net that you believe in. Which is you have like regression tests so you don't blow up big functionalities. [23:09] Um, [23:10] But if you're pushing hard enough, you will make mistakes. [23:14] And so you have to distinguish between different kinds of mistakes. For a database company, there are catastrophic mistakes. Like if you write data badly, it's going to take you months to get out of that. So you need to understand what is risk. And then you build a safety net for things like, as I said, to detect problems before they happen. But in case you do have problems, how you get out quickly. [23:44] basically you would [23:46] you know, come up with a new experiment. All new change, all changes went through this experiment framework. And this thing would automatically say, I'm going to run this on a machine, watch it for 15 minutes, make sure that the machine doesn't crash. And then it rolled it out to 0.1%, 1%, 10% with measurement all along the way. All of a sudden you have velocity because someone can design, people can design a whole bunch of experiments. They're sort of now pushed out. So as I said, the first part is the safety network. And so we spend a lot of time on that.

[24:16] is the inner loop productivity, which is how quickly can you get, like, [24:20] a single change in quickly, because ultimately it ends up being the decider for how many changes are you going to get through. Another system design, Snowflake actually went through a process that predates me starting about two years ago. [24:36] of how to make the system extensible. As I said, at Snowflake, we are very proud of the single unified product. But that can become like, you know, something that gets in the way of speed. And so you have to design carefully for how do you make things extensible. So things like AI basically took advantage of that framework. And then to a certain extent, to be honest with you, [25:06] on what is important. How do you drive clarity? At all times with all teams, there is an infinity of work to be done. And driving that clarity, driving a sense of accountability with AI team, for example, I force every team to make promises for [25:22] Yes, over three months, but also what are you going to do the next two weeks and calibrate yourself on? Did you deliver on the things that you were doing that you said you were going to be doing? It's pretty much in my mind, if you want to get better and better, life boils down to say what you're going to do. [25:38] and do what you said you would do. Yeah. And examine and make things better. And so it's a bunch of things that I've been there that I've been building up at Snowflake. But certainly I bring this sense of

[25:54] Quality and speed are both requirements in what we do. It's a change, but people like the idea of just getting more things done. You and I have never met a software engineer that says, yep, I want to release that day after tomorrow. It's like, no, you want to get it done today. And so that itself builds momentum. When you release a bunch of products and you have a lot of customers that are using it, [26:24] to build on the good behavior that kind of got you there. And so I would say the team has responded very, very well. And I told them, hey, listen, this is the world of AI. [26:33] Stuff changes every week. [26:35] And you need to build with that speed. I'm very happy with how the team has responded. Is there anything in particular that you're most proud of in terms of what you guys have done in AI thus far? [26:47] As a Cortex Analyst is probably the hardest product that we have designed and launched. [26:54] So things like Cortex-AI, which is like our platform layer, I'm proud of it. But, you know, it is sort of predictable infrastructure work, even though there's a lot underneath in terms of, hey, should you use VLLM or something else? How do you optimize FUD inference? [27:13] How do you get capacity in like this annoyingly crazy world where it's very hard to get your hands on GPUs? There's a bunch of stuff. [27:20] But to me, that is a unique that things like that, things like document AI are a unique combination of our strengths being applied to new areas in ways that can make a big difference to our customers. And, you know, but you also know, Pat, that this is a little bit of like, you know, who's your favorite child? I can't really do that. And so there's a there's a bunch of stuff.

[27:50] Thank you. [27:51] And so I think there's a lot of energy within the team because, you know, it's a slow message, but it's getting through that you can have speed and quality. They're just different aspects of the same problem. And my firm belief all through my life is that, [28:10] virtuosity, [28:12] Trump's strategy all day long. [28:14] What does that mean? [28:15] Your speed of execution. [28:18] Your speed of reacting to situations is going to trump strategy very, very quickly. Yes, you need strategy, but life is never about fixed strategy because we live in a very, very dynamic world. [28:33] It's hard to predict which product is going to be wildly successful, what your competitor is going to do. Like we're going to talk about like GPT-5. It's like it's a big unknown whether it's going to come out and what impact that's going to have. So I place a huge amount of emphasis on. [28:49] You just need to be really, really quick at what you do. And I would say, like, that's the message that I'm trying to convey to the team. That's very – I see nice continuity from the Slootman era into the Shridhar era because I know I've heard Frank say at least a few times the general patent quotes, a good plan executed violently today is better than a perfect plan tomorrow. 100 percent. Yeah. 100 percent. [29:19] Napoleon has a famous quote, which roughly, I mean, it's not his, it predates him. It roughly translates into, you know, I commit.

[29:28] And I adapt. [29:29] Hmm. Which is you go into an important area knowing that you're not going to know everything and then you're adaptive to the situation that actually presents itself. Yeah. Are there any misconceptions about Snowflake and AI that you want to debunk? [29:43] We are a real player. It used to be that Snowflake used to be thought of as somebody that didn't really get get AI. And, you know, but like early on, we relied on things like more of a partnership oriented strategy for AI. But my big sort of. [30:02] observation, realization, is that [30:05] AI is a [30:08] Like it's a platform change in the sense that it is a new way in which you and I and everybody else in the world is going to get to software, is going to get to applications. And so once we had that realization, out came a bunch of product consequences, which is AI needs to be central to Snowflake. We need to make it super easy to both build applications, but also build the most important applications ourselves. [30:38] for example, is a direct to business user application. [30:42] We have never really done things like that before. It is driven by a strong belief that AI is going to disrupt how information is going to be consumed very, very broadly.

[31:12] on top of AI, [31:13] That, combined with things like broad data access, which is Polaris and, you know, Iceberg, I think puts us in a very, very good position. [31:25] Can we zoom out and ask a little bit about your, I guess, your hypotheses and your hot takes on the future of AI? Absolutely. I just think you are so well positioned. You probably built... [31:33] one of the first, if not the first, kind of LLM native consumer applications at Neva. And now, obviously, from your seat at Snowflake, you see so much. Maybe first on the LLM kind of race to scale. Like, what do you think about all that? Are we reaching the limits of scale? Like, what's next for those guys? I mean, obviously, this can go in a couple of different directions. I talk to a lot of experts. [32:03] in the horizon. [32:05] What I don't think anyone has a [32:09] Um, [32:11] like a clear bar, far, is what that's going to represent. Yeah. GBT4.0 was very cool, much faster. It also integrated multimodality natively in a way. That's pretty amazing. But when you think about reasoning capabilities, the ability to come up with plans for how to execute stuff, [32:33] it didn't feel like it represented a step change. [32:37] and while agents are [32:41] In all, [32:42] very hot, similar to Cortex, you know, until Cortex analysts came along, people didn't really believe that you could build reliable talk to your data application. They were always kind of hidden. And remember, the bar is very high. If you're giving data to a business user, like 75% accuracy is like one out of four wrong.

[33:01] And so I think the big unknown is whether these models are going to represent. [33:08] a big step forward in things like multi-step reasoning. [33:13] And if they can, they're going to unleash like a whole new class of applications that you and I just cannot imagine right now. [33:23] You know, on the other hand, I think when it comes to driving broad adoption, you know, [33:28] there is a lot that can be done with existing mods. So many things that are useful for you and me every single day, whether it's a piece of mail that we are looking at or looking through a PDF, just think about all the tedium that all of us have to go through. And so I think there is huge impact to be had simply in AI technology, just permeating software as we know it, especially the [33:58] So unlike other technologies, I think like there is enough that AI has already delivered that is going to have a meaningfully large impact on society. It's just going to take a while to run out. [34:10] You know, I sincerely hope we don't get to a phase where you need a billion dollars to train a great new model. I actually think that while what that model can do is cool, I think it also reduces the number of people that can have models like that to a very small number. And I think competition is just overall healthy.

[34:32] Um, so, but it's very hard to make a call. You mentioned this a little bit, but I'm curious to get your take on it a bit more. [34:40] You know, if GPT-5 is delayed or not a big step up or whatever the case might be, or if you just imagine a world in which the current capabilities of the foundation models, that's what we've got. [34:51] And it comes down to how do we implement those? How do we optimize those? How do we tune those? [34:57] One of the things that we hear from a lot of people building an AI, the first couple of weeks are like magic. Everything is amazing. This is great. And then the next few months are pretty painful. Oh, shoot. It can't do this corner case. It can't do that corner case. It's not quite accurate enough. And people get really frustrated. And sometimes they can engineer their way out of it. Sometimes they can't. But. [35:18] sometimes it leaves people feeling kind of disillusioned like, ah, this stuff's not as good as I thought it was, you know, maybe the time's not right. And so I'd love to get your take. If we froze the capabilities of the foundation models today, um, [35:29] What sort of changes will we see in the enterprise landscape over the next handful of years? What sort of stuff will we not see? Because we're just not ready for it yet. [35:38] To me, this is honestly the magic of software engineering. [35:42] Part of the [35:45] what I feel we have implicitly accepted with ChatGPT. [35:51] is it's sort of like [35:54] Because it's omniscience. [35:56] You're like, it can do everything. [35:59] They don't say it. In fact, they take pains to not say it. But just like Google search never tells you,

[36:06] That's a dumb query. [36:07] Ha ha ha. [36:09] Think about it, right? It'd be kind of fun if it did. Right. But there are lots of dumb queries that people type into it. Google's like, oh, yeah. I type lots of dumb queries. Yeah. They're like, oh, here are 100 million pages on the web, and here are the best pages for you, Pat, for your dumb query. [36:25] And so I think it's like it's some of it is good old fashioned, you know, AI enthusiasm. It can do everything. But some of it is just also plain dumb. [36:39] You should not be doing that. To me, [36:43] This is where things like, okay, let's actually make grounded chatbots the norm for... [36:51] you know, like interacting with information. Yeah. [36:53] The model is there. [36:57] This application should tell you where it got the information from. It should be very easy for you to verify said piece of information and feel good. [37:07] that you're actually getting something. Similarly, you need a test framework. [37:12] You know, like Harrison talked about an observability framework to do this on an ongoing basis. But I think sometimes when it comes to things like chatbots, people forget, wait, like there is such a thing as a set of regression tests. There is such a thing as acceptance criteria for software. Everything that we have, like if somebody were to build a new application, like one of your founders, your expectation is that, you know, they got their clue together and are actually testing stuff before they give it to customers.

[37:42] world of AI, we're like, no, no, no, no, no, it doesn't matter. And these models, you know, react pretty violently to the addition of a period in a prompt. [37:50] And so I think there needs to be this idea that you need good old-fashioned software engineering. [37:57] and you need to measure the performance of these things. And so I think this is where it goes, you know, away from these are hobby projects that can be hit or miss. [38:09] to, you know, here's somebody that can actually software engineer this for you. And we think of that as a core strength of what we bring to the table, which is like, [38:18] you should be able to have a predictable way to say, you know, this chatbot is going to work, or this agent-like application, this is the success rate that it's going to have, or this is what Cortex Analyst is going to do for you in your domain, so that you're like, okay, I feel good about deploying it. So even if GPT-5 did not happen, [38:41] I think there is a lot of magic to be done, but it's also... [38:46] just work. Yeah. Well put. Well, what's the, um, I forget who said it. There's a quote that we use every now and then, uh, [38:55] People miss most great opportunities because they tend to be wearing coveralls and they look like work. You know, I think this is one of those where, like anything else, if you want it to be great, you got to work pretty hard on it. You got to spread it out. And to me, this is also the place where the thinking of recall is.

[39:16] as something that you should tune. [39:18] Thinking of recall as an important part of how you think about these applications. Any ML engineer worth their salt will promptly come and tell you, it's like, okay, I have an AUC curve for you. What are they trying to say? They're basically trying to say there is a tradeoff between how much you squeeze the model to do and how good it is. There's no perfect answer. That's really what the AUC curve represents. And the more we think of AI applications as also having this AUC curve. [39:48] There are tradeoffs to be made between reliability and ability to respond. [39:53] And that's a very conscious factor in how you should think about things. I think the better off we are going to be in terms of where can they deliver value. Yeah. [40:03] I want to go back to the point you said a little bit earlier about reasoning and kind of that delivering the next big leap, hopefully for GPT-5 and Claude, et cetera. It seems like the approach that most folks are taking is kind of bringing in search at inference time and a lot of more inference time compute and kind of this AlphaGo style search stuff. [40:33] reasoning into these general models? [40:35] Give me a little bit more context. I can certainly see how search plays a role in how these models operate, but can you tell me a little bit more? Yeah, so if you take the example of AlphaGo and you're trying to decide what move to do next, if you can kind of create a branching tree of here are all the possible moves from here and do a search kind of over that of like here's what move I should do next,

[41:05] and into domains like, I don't know if you saw Devin's Cognition, [41:10] where they're effectively searching over different things that you can do in your coding as well. And so just like at inference time, just giving the model kind of the ability to like search possible paths to decide what to do. Yeah, there have been a number of papers, you know, on this. I think even NeurIPS had a bunch of papers about searching over domains as you come up with a plan. [41:33] What I don't have, to me, it's important to understand. I'm forgetting the name of the Newtips paper, but it also had the same problem. They were doing tree search. [41:42] is that they fundamentally rely on, [41:46] on a model, typically a neural network, being able to do things like grade a particular point in a, like a state space. Yeah. Basically, like AlphaGo, for example, you know, has pretty solid ideas about what is an advantageous position versus what is not. And the search is guided by, you know, by that. [42:16] very open-ended questions is as you come up with alternatives for the search space, you know, can you actually grade them effectively? If it's like an open-ended plan. Certainly, a number of these techniques work well for games that have structure in which you can actually learn what does optimal mean and you can begin to optimize towards it.

[42:39] What I don't have as good a feel for is, let's take something as simple as cooking. [42:45] um you would think it's you know it's simple but if you take [42:49] I don't know, 10 ingredients and 20 steps that you can take along the way and various things that you can do in each of these 20 steps and the steps themselves can be short, they can be long, you quickly end up with like this crazy combinatorial explosion of different ways of doing things. And yet there is just one perfect recipe or two or three. That's the part, honestly, I don't have a good feel for in terms of like, how do you even begin to measure the jump [43:19] terms of cognitive ability. It's easy in structured environments, but like out in the real world, where you're trying to do some pretty complex things, I think it becomes trickier. We've built prototypes for basically like agent analysts, but it's again a structured space. So what we do, one thing, I've done numbers like pretty much all my life. I used to do whatever household finances for my dad when I was 10, same, like we did it in a notebook. And over the past 20 years, every day, [43:49] I get like this email that tells me how my company did the previous day. [43:53] Used to be called bean counters at Google. Every day, you got a report card. Every few weeks, something would go wrong. [44:00] Like, you know, you made less money somewhere. And we would like start this predictable problem, like predictable exercise of some poor analysts would like go drill down into a bunch of different things, blah, blah, blah, blah, blah, look at sliced stuff. And then they would come back with like, oh, Sridhar, it was like Easter in Germany and Ascension Day in Brazil. And that's why our numbers were off.

[44:23] And it took like a decade to model all of these complex things in the world into like a prediction model. So you're like, OK, I can I can begin to predict. But if you think about it, the analysis that they do is constrained. [44:34] It's pretty much if a metric is wrong, go slice it by 10 different dimensions, go look at the results, see where likely the problem is. Certainly, we have built prototypes of this AI analyst that can remove 60, 70 percent of the work that is needed in actually diagnosing problems. [44:51] It's pretty free form, but you can make a language. If you can tell a language model, these are my attributes. I'll go call Cortex Analyst with all of these parameters, get the output, take a look at it, and then tell me what I should do next. So you can begin to automate some of it so that this is actually useful. [45:06] So you can do things like that, but a much more open-ended problem of here are a hundred different things, incomparable things you can do, and how do you judge and how do you prune, I think that's a part I honestly don't have good intuition for. [45:22] Totally. [45:23] I'm going to ask about search in a different sense, if that's okay. [45:26] Um, you obviously have a incredible point of view on search, given your time at Google and at Neva. And it seems like right now, you know, the consumer world is watching excitedly and nervously about, you know, is, um, is there going to be a new kind of search king crowned? Um, I'm curious your take on, on, on the whole kind of AI search space right now. How about a hot take on perplexity? Do you have a hot take on perplexity? Yeah. [45:52] Like, look, I'm happy for perplexity. And it reminds you again that right time, you know, right time, right place matters a lot. At Neva, you know, which converged on to a view of what search should be that was very similar to perplexity. We were just two, three years early.

[46:10] And timing ends up being everything. You can think of perplexity as like a consumer manifestation of how we want to deal with information. [46:24] Like, it's just... [46:25] Let's face it, I want to look through an eight-page doc to find the two lines that I really care about. [46:32] Said no one. [46:34] But that's search. [46:35] And so in that sense, it's absolutely the right place. I think the more important question is, [46:44] whether the business of search [46:46] which is carefully preserved. [46:49] with business contracts. [46:51] not with consumer choice. [46:53] Consumer choice is fiction. [46:55] in a whole bunch of things that we do. [46:58] We eat what's put in front of us and we will search with the default search engine that came in our browsers. [47:06] We might resist it, but on aggregate with humanity, that's the reality of the world. And so I... [47:15] you know, [47:16] I would say that that is the bigger challenge because search is mostly locked up by a few players that control the entry points. But I think that's the fundamental problem, which is it is very difficult to break into the business of search. [47:37] Consumers don't like doing stuff. [47:39] And this also gets to one of the kind of broader questions in the world of AI right now, which is incumbents versus startups. And historically, the battle is, can the incumbents with distribution build cool products before the startups with cool products build distribution? And I think search is a great example of that. You might have the coolest products in the world. It's awfully hard to change consumer behavior. That's right.

[48:02] AI is an interesting test case for this because so much of the coolness of the products is available through the open source world or through third party models. And so, [48:14] It feels like it might be a scenario in which incumbents are advantaged versus the startups. But do you have a point of view on that? I would take two different lens to this one. [48:24] One is what you said about models. [48:27] open source models, [48:29] plus players like Meta that... [48:32] basically have infinite budgets. Yeah. Under the link to open source models. I think the world of creating models from scratch, [48:41] Unless you have an attached hyperscaler, an attached business looks very, very hard. Yeah. And. [48:51] um, [48:52] So I think [48:53] As I said, I hope this doesn't go to like an ergo three GPT-5 class models that the world has, because I think that's a bad ending for the world. [49:03] So I would definitely say that foundation model companies without a strong business to accompany them, it can be a product. Like I think OpenAI has created a pretty solid product. It's not just a foundation model. Yeah. I think that's one thing to keep in mind. I'd answer your second question. [49:24] of sort of disruption slash innovation from a historical lens. I think of every generation of Silicon Valley companies.

[49:36] as learning from the previous ones. [49:39] They are smarter. They know the ways in which things can be disrupted, and they lean in pretty heavily. [49:48] You know, we all know, for example, the IBM to the mid-range computer sort of disruption, [49:58] And then the DEX and SGIs of the world then getting disrupted by the Microsofts of the world. And then the web coming along leading to the rise of companies like Google or mobile. [50:13] I would say that in each and every one of these transitions, powerful incumbents with very large pockets have shown an ability to lean in sooner, lean in faster. [50:27] At Google, for example, when I was there, we leaned in very heavily into the home assistants because Alexa was going to take over the world. [50:37] That was going to be the way in which you and I and everybody else searched. We were terrified. [50:42] Um, [50:43] And we put a pile of money into it. [50:46] And nothing came off it. [50:49] And it didn't matter. [50:51] Why? Because the cost of a disruption is way higher. [50:56] than the amount of investment that you have to make. I would say now this is generation five. [51:02] or something to that effect. Let's say all the incumbents are very aware of what can be disrupted, and they lean into it.

[51:11] There's a bunch of strategic thinkers. As I told you, I think of AI as basically shuffling the tiles on enterprise software. And a part of me goes like, you know, no way. Snowflake is going to be leading the charge when it comes to AI, not waiting for it to develop. But I think you see every enterprise AI company lean in the same way. And so this to me would be the question about how much disruption is AI going to drive? [51:38] in consumer software. Certainly there'll be new categories. [51:42] To me, if I were a startup, I'd feel a lot more comfortable that I'm creating a new category. Image creation, like done in a mass scale. [51:50] Clearly amazing. [51:51] But the same goes for videos, same goes for voice. There's a bunch of specialization that you can do here, adapt them to marketing. New things feel like a much safer bet in the AI world than, you know, take your pick. I can do XYZ faster because I am AI enabled. [52:10] I don't think of that as having a whole lot of legs. Yeah. Do you think ChatGPT has a chance of becoming the next Google? And to your point on consumer choice being a mirage and like, you know, business deals are where this stuff gets locked down. Like, I'm curious what you think of the Apple ChatGPT deal. [52:25] I think chat GPT, I mean, the phone is a pretty interesting place. [52:30] To me, the phone, because it's a controlled environment, [52:35] actually offers enormous potential for consumers. [52:40] I tell people something as ridiculous as copying I don't know an address or

[52:46] like from your calendar on a piece of email or to uber. [52:50] So dumb, so hard. [52:53] You know, like you would think Citi would like, you know, do this. [52:57] copy the address from this email from Pat and stick it into Uber so I can get an Uber. So to me, I think there's a huge amount of potential again in mundane applications. And because the mobile ecosystem [53:13] is a pretty closed one where Apple can mandate things like you must have APIs that make it possible to access your functionality using language models [53:24] Adults. [53:26] you might not get any traffic. That sounds like a pretty good incentive for everybody to kind of get in line. So I think there's a huge amount of potential there. I honestly wish there was more innovation in this space, because again, all of this is [53:40] Super doable technology. You and I can argue about, should this be done in the cloud? What can be done on the phone? But like, as a consumer, do you care? [53:48] Like you have great connections. I'm kind of like if this thing works only when I'm connected to the Internet, I'll take it. And so to me, those are those are sort of, you know, those those are details. [53:59] I actually think ChatGPT is an amazing product. [54:03] There's underlying technology, but in so many different ways, they've actually created a stunningly beautiful product experience. Yeah. That spans the gamut from, you know, they've turned pretty much like visually illiterate people like me into budding artists.

[54:22] I tell people, it's like, I'm good with words. I can talk all day long. I can write all day long. And the magic that I can do with ChatGP is truly amazing. [54:31] Um, that, or, uh, even, even things like I, you know, for example, like I'm on this language kick, I'm learning Hindi. And at some point I was like, oh, I'm struggling with these numbers. Um, but off comes a prompt that says, Hey, I want a CSV that translates numbers, just a string of numbers to Hindi. And can you do that? Can you just give me a CSV file that I can import into Quizlet that literally is faster for me to type than to describe to you. [54:57] I type it in, out comes a CSV file in 10 seconds. I download it into Quizlet. I have a quiz. [55:03] And so, and pretty much everything that I used to do with Python scripts on structured data, [55:08] I just do like with English. You just upload the CSV file and you're like, oh, add these two columns, do this other thing, format it into this nice table and get it out for me. It's magic. So I think there's absolutely a there there in terms of like, is it a great product and a great business? But, you know. [55:30] being the king of search is like a few more zeros. They don't come easy to people. Yeah. [55:36] All right. Should we close with a couple of quick fire questions, rapid fire questions? Let's do it. OK. Who do you admire most in the world of AI? [55:45] Who do I admire most in the world of AI?

[55:53] I admire the people that are [55:55] you know, [55:57] Likely. [55:58] working on things like foundation models. [56:01] that are able to do it on the cheap without the infinity of resources. So, for example, people like Arthur or Danny, [56:13] I think they've gotten Danny Yogatama from Rekha. I think they've gotten just like a remarkable amount of things done. Or from our own team, folks like Samyam and Yushang. To me, they represent so much creativity because I go and tell them, ah, limited budget. And what can you, you know, what can you do? [56:43] But it's the doers that are doing the work imagining our future that I'm a huge fan of. [56:51] What's your favorite AI application? [56:53] It's IGBT, by far. [56:55] Easy one. Easy one. Just the utility that I get from it day in and day out is just truly remarkable. Okay, follow up then. What's an AI app that you wish existed? [57:04] I'm like. [57:05] An actual talk to your phone? [57:07] that can actually mediate between apps. That would be super cool. Because remember, as I said, like just flipping between applications, doing very, you know, little things, such a pain. [57:18] All right, we're going to end on an optimistic question. [57:21] What is the best thing that can happen in the world of AI over the next five or 10 years? What would you be most excited to see coming out of the world of AI?

[57:30] software. [57:32] which you can think of as encoding our thinking. [57:37] capturing our ability to think. [57:40] and act in real world situation clearly has been transformational over the past 50 plus years. [57:47] To me, AI as an enabler, [57:54] of access both to the act of creating software, but using software. [58:00] to all of the people in the world. [58:02] would be a significant step up. And as I said, I don't think it's like lots of fancy new technology that you need. The newer technology can certainly help newer classes of applications. [58:15] You know, I was very proud of the fact that we put Google search, thanks to things like Android, into the hands of pretty much every human being on the planet. [58:24] You can be cynical about technology, but it's a genuine step forward for humanity. To me, just like AI models as like the new layer between humans and software, and software and software, is actually a significant step forward just in having this functionality be vastly more accessible to lots more people. As I said, both in the creation aspects, but also in the consumption aspects. [58:52] I think that's a pretty cool thing to look forward to. [58:53] Awesome. [58:54] Thank you, Sridhar. Thanks for doing this. [58:56] Thank you, Pat. Thank you, Sonia. [58:58] Thank you. [58:58] Music.