Prof. Anima Anandkumar — The future of AI and science

0:00 Hi and welcome to the Neil Ashton podcast. In each episode, we explained some of the fascinating ways that science and engineering are changing the world around us. We talk to leading engineers from elite level sports like cycling and Formula One to some of the world's top academics to understand how fluid dynamics, machine learning and supercomputing are bringing in a new era of discovery. We also hear some of their life stories, their career advice and lessons they've learned on the way that I hope will be helpful to you too. So sit back and enjoy this episode. Hi and welcome back to the Neil Ashton podcast. Today's episode is a continuation of the theme of the past few. This I guess mini

0:48 series focusing on AI for science. This idea that you can use machine learning and artificial intelligence to accelerate in hand, improve scientific discovery in fields such as fluid dynamics, computational fluid dynamics. So, you know, car design plane design that has been common throughout this podcast, but of course, also things like weather forecasting or drug discovery. And today's guest is most definitely one of those people who is seen as a leader and a pioneer and someone at the forefront of these methods. That person is Professor Anima Anandkumar, who is a Bren Professor at Caltech. But interestingly, and I think what has arguably made her such an important um person in this field

1:38 is that she has also been very close to, to industry. So she was first um at Amazon Web Services as a principal scientist focusing on, on AI and this was actually before, I suppose, the mad craze now, um, with AI and GenAI. Um and and recently was also senior director at NVIDIA for AI research. So she's spent not only time in academia but crucially at very senior roles at these tech companies that are at the forefront of machine learning. So that's why it's so interesting to get her perspective in this field because she has not purely been doing research in academics. And she's also been there in these companies and seen

2:27 um you know, its application to, to more real life problems. And um we, we discuss a number of things in, in this podcast as with any podcast, there's so many things that we didn't get to cover. So I'll be putting lots of links uh in into the comments on YouTube for you to, to read papers and look at other things that, that her group has done. But we talk quite a lot about one of the achievements and one of the things that she's arguably best known for which is uh neural operators, this this type of machine learning method that is uh very well suited to solving scientific problems like fluid dynamics. Uh And is um probably best known in a way for the work that was done for

3:11 FourCastNet. And if you look at the links below, you can read the paper that was one of the first papers or first approaches to really revolutionize um weather prediction. And so we talk a little bit about where things are at. You know, what, where does she see the current state? She focuses heavily on AI and science, not AI for science because she really sees them as complementary. And we dive into the usual questions of, you know, how close are we to being able to do in real life? How important is physics, some of these ideas of foundational models, some of the topics that I've been asking the other guests. But I,

3:51 I almost want to repeat the same question to the different guests because then you hear the different opinions and hopefully you as a listener can get a more rounded view of, you know, where the community feels it's at. Uh I'd say uh Anima is more bullish um on this. I think she has more confidence that we are closer to making some key breakthroughs. Um And, and we talk through that and we talk through some of the actual Pacific applications that, that have been made. Um interestingly, we also talk a little bit about her um career and her story so far. And I think it's very interesting her move through academia um to, to, to industry and,

4:27 yeah, really, um, really interesting. She's a fantastic speaker. She's actually just done a TED talk. So I'd highly encourage you to, to watch that because as we were discussing um more off air, I think it's hard to explain some of these things in either purely verbal, if you're just listening on the podcast or even just in a, in an interview fashion, you really want some slides and pictures to explain some of these topics. So if you are interested in AI for science or AI and Science, uh I would highly recommend you to watch her TED Talk cos I think it gives you a more visual uh impression of some of these topics.

5:01 So please sit back uh and en enjoy this episode. You're a full professor at Caltech, probably one of the most well known institutions in the world for, you know, science and engineering and many people, I'm sure have a dream of, of getting to that when they're, you know, when they're younger, when they're studying. Did you have academia in your mind from a, from a young age when you were in school? What was your, what did you want to be when you were growing up? Yeah. Yeah. No, I, you know, certainly Caltech was my dream too. I grew up being extremely inspired by Richard Feynman, you know, reading his books, but,

5:43 you know, his books on physics as well as his life stories. And, yeah, to me, you know, I wanted to be innovative. Right. And I kind of like, fully didn't think what that path would be because my background is kind of quite worried. My, both my parents are engineers and in fact, in India, it's quite rare for my mom, you know, in her generation as a woman to be an engineer. And that was just great because, you know, at home, it never felt like as a woman, you aren't good at something just because you're a woman. So it, it gave me a lot of uh just really good role model for someone who's a woman who is also a great engineer.

6:24 And both my parents um started a factories, manufacturing small components for automotive industry. And in the early nineties, they brought some of the first computerised machinery to my hometown in India. And so they really kind of like were forward looking and thinking. OK, how do we make this more efficient? How do we bring programming into manufacturing? Right. And uh so that was also a different way to get introduced to computers for me because it was always something more physical, like it did kind of like produce these components, it kind of manufacture them and machine them. And so that was also an aspect that was very interdisciplinary in what they did.

7:10 Um So that was great. Whereas, you know, my grandfather was a math teacher and I was always very excited about just solving puzzles and I would do that as fun. Although for a lot of people, math is like this homework they have to deal with. Um And I really liked him as a teacher and, uh you know, so I've had like role models both in industry and academia who, who were just great and I wasn't kind of set on one path, but it kind of led me to where kind of, I felt I would have the most access to doing minor work. Um You know, when I was in the process of finishing my PhD and going to the job market in 2008, you know,

7:55 those of us who know it was just such a difficult time, right? It was the first time to be on the job market. And at that point AI and machine learning was not a job description. So industry wasn't even an option. And of course, with all of this financial crisis, it was even less of an option. And, you know, I got a faculty position at the University of California at Irvine. And I was like, OK, you know, I can continue to do this work in machine learning, which right now many people think as science fiction, but that's what I want to do. I want to make that work. And here we are today. So I'm really glad how that that took me.

8:35 Uh And once deep learning started really taking off and there were all this industrial expat, I was also really lucky to be placed in many of the top industrial roles. You know, I uh went to Amazon Web Services, helped start the cloud AI group build some of the first AI products on the Cloud, then went to NVIDIA. So really kind of also following the journey of AI expansion in industry and first being in academia to build those foundations has been just a dream come true. Yeah, totally. And yeah, I really would love to dive into some of those uh those details. But what was it like moving to the US to do your PhD? That must have been

9:22 exciting daunting at the same time. Had you been to the US before or was that actually sort of a, a very big move for you to do that? Um You know, it always felt like I wanted to be in the top places to do A PhD, right? And do research and I had applied at various kind of schools. And Cornell is where I really connected with my advisor, you know, Lang Tong and I picked that uh but it was just uh culturally us is something that, you know, I've been close to, I've been to Europe several times. Before. Um And so, and also it's such an international place when you go to these universities. So, you know, that aspect wasn't an issue at all.

10:07 It really, he was like, ok, I can now learn from all the cultures around the world, meet people from all over the world. The only kind of issue was, oh, it's damn cold. Like, you know, coming from a tropical place it took in upstate New York was a, was a big, let's say surprise. But that also was, you know, I was like, what can we do about this? And I decided to take all kinds of winter sports. I took up ice climbing, which is really challenging, not to say I became an expert. But I was like, what can I do to really get out of my element and embrace this cold weather? Is that one of the reasons you then went to the

10:53 sunniest part of the US or was it completely separate? Yeah, I mean, like, as I mentioned in the height of the financial crisis, there weren't too many choices. So I was glad I got this faculty role and, you know, after that was Caltech and uh you know, there's a lot in California that's, you know, really great. So, yeah, happy to be here with them. Yeah. No, no, no, definitely. And did you, um at what point I, I've spoken to a few people about this so I'm kind of interested of this choice between academia and industry? Was it a conscious decision? Did you have a desire to um not only be a pure academic because you wanted to go

11:39 into tech or did you sort of fall into it by accident? Um You know, like I mentioned earlier, I wasn't like hard set ever to be in one versus the other and I don't see it as even an exclusive choice. Uh And, you know, like as I mentioned, uh straight out of PhD, industry wasn't even an option. So I had to go to academia, really build up the methods to show their work, right? That's when industry decides to start investing and when that happens, and there were roles where I could really contribute, like, you know, have this outsized impact of being able to start something entirely new, a new AI division or AI research,

12:21 you know, that was like to me, something that I was like, OK, I can really make a step change here and to the whole community and be able to publish open source. And that really was a great motivator as well for me to take that step while also keeping a foot in academia to really bridge the gap between industry and academia. And would you recommend that to people, do you think that's actually quite a wise and good thing to jump between academia and industry? You know, if, if you, if someone's listening to this now doing a PhD, would you, I know you said you kind of had to go into academia because 2008, 2009 was a

13:00 unique time. But would you actually recommend it in a way? It's really about personal choices? Right. And, and also in industry research as well, we're seeing more changes than before. It's not completely free for those guys. Complete freedom. Uh you know, at one point it used to be at least to some extent and we are seeing more of the closed models and uh kind of more, you know, targeted goals and for some people that's really what they want to do, right? And there are others who want to be working on areas that are like still new, still unexplored, making interdisciplinary connections. And perhaps for those cases, industry is not the best,

13:46 especially if you don't get assigned to a team that allows you to do that. So it's really one of like personal choices. So what is it that you value the most? And, and certainly in industry, there are more resources, of course, depends on the place. But in big tech, you do have a lot more GPUs. Uh but there is more of a focus, you know, aspect of, you can only work on a certain set of topics and areas that, that was actually, uh maybe before we move on to, to, to another topic. Uh where do you see that? Because I, I've read that that in a, in a strange way, academia in most other fields has been very much the developer of the fundamental science,

14:30 the fundamental methods. And then industry takes it once the technology readiness level is higher. When it comes to this current age of machine learning. And AI, there seems to be almost not the, not the opposite at all, but there is a huge amount of research and development going on in tech companies. And I'm, is there a brain drain essentially going on that because they have the resources to hire in the talent that in a way we're losing some people who would have naturally gone into academia. Um You know, for me, I don't like the term brain drying no matter between countries or between organizations, right? Because

15:14 there's always more brains to be brought in. So kind of like let me start with that. Uh you know, and, and sure, you know, like industrial roles speak to a broad set of students because there is resources and very well defined problems and you know, let's go work at it in academia. There's fewer resources, but that also means necessity is the mother of all inventions, right? Like how do we do more with less? So can we improve our training methods? Can we look at scenarios where there isn't as much data? How do we do learning in that? And it's always academia that works on problems that industry is typically not looking at because it's considered as too impractical

15:58 machine learning, all of machine learning. Was that at some point? Right. So I'm just saying that it's ideally in academia, it would be great to have more resources. But you know, that shouldn't be the only aspect when it comes to making the choices. So when did you first get into machine learning? And was that a conscious choice? When was it in your head that you really started to double down in that area compared to other areas of science and engineering? Yeah, I mean, during my undergrad too, I worked on signal processing which is essentially machine learning, right? Like either you work with the radars or image processing and

16:43 you know, there was not deep learning at that, although we knew neural networks as a concept, it wasn't considered practical, but the concepts were there those fundamentals were there, my undergraduate thesis was looking at Iris recognition biometrics, right? So it was really just these aspects that OK, there are all these important problems to be solved, maybe they're not practical today. But how do we start building the foundations of the algorithms? How do we frame learning as a problem? And what are the requirements in terms of data in terms of the right algorithms and constraints? Right. So we were doing a lot more of the theoretical analysis because

17:25 you couldn't like kind of run algorithms at scale at that point. Uh But that also meant we were thinking about it systematically and all of that led to the further developments. Hm. And what about, um, the movement into, I guess AI for science? So maybe pivoting a little bit more to, I guess what has been, um, I would argue you've been one of the leading voices in this field that's really pushed it along and given it more of a global um awareness because it a bit to your point of um companies work in some areas where there's more particular focus on others. And I guess the science one is arguably not immediately the stuff you see on the TV, the, you know,

18:15 the GenAI, the chatbots, et cetera. Um But I think voices like yours have helped to bring it out. Maybe we can, there is such a big topic. So maybe let's start by how would you define AI for science and why do you see it as being such an important area for us to, to work on? Yeah, I mean, to me, you know, being at Caltech, that was one of the first things I thought about when I came here, right? Like I came here thinking about Richard Feynman and all of the scientific developments that have happened here. And the natural question is what can AI do for enhancing and accelerating those scientific developments? And,

18:59 you know, back then in 2017, when I came here, there was still a lot of skepticism of even about AI or other areas, forget its usefulness to other areas. They were like, oh, is a, I, I even a real thing that was still early days and now of course, no one asked that question. Uh But I don't even think of it as AI for science instead. I think of it AI plus science. It's the deep integration of AI and what we call scientific research in all kinds of aspects. Um Because if you think about how scientific research is done since the time of Newton, you know, what do we call the scientific method? It's the aspect of coming up with ideas, right?

19:40 So this is where experts are better because either they get a eureka aha moment or they're just thinking and thinking and like saying, OK, I've kind of rejected all these hypothesis. So this one works. So, you know, there is this deep intuition and domain expertise that people develop over time and then that's not right. It's not just the great ideas that propel science, but the hard work spent in the labs to actually go test it out, validate it. And those experiments many times just disprove many of the theories that you have to go back and come up with new explanations, new ideas to test further. And this is great,

20:21 but it's extremely slow because the bottleneck is not ideas, you can keep generally thinking of all kinds of like, you know, our minds are creative. People have been coming up with all kinds of theories about this planet, but only a few of them are correct because you have to go measure and carefully validate those. And so to me, AI can help in all of these aspects, right? AI could come up with new ideas. You're seeing language models do that today. Uh You can come up with new proposals of materials, drugs, It can, you know, even design aircraft wings. If you ask DALL·E or stable diffusion, it can generate something that looks like an aircraft wing.

21:06 But you still have to go physically tested in the lab to validate that it's correct. And the problem of hallucination means many of those ideas that AI generates today is not that useful. You have to go through a lot of testing and discard a lot of the proposals made by deep learning before you get a success story. And that's why, you know, my kind of premises AI plus science when I say is not just using AI to come up with hypothesis or ideas, but AI that is deeply integrated with understanding the scientific models and processes that themselves, by which I mean, ideally one day we would reduce or completely remove the

21:49 physical testing that is needed because AI can internally simulate and really understand all the processes that are involved in that reasoning that yeah, I really would love to dive a little bit deeper into this. So could you maybe give some examples of where AI has already shown great promise in this area and there may be uh some areas where it's more challenging. Yeah, I mean, to me, like there's so many great success stories of AI and science coming together. Um by the way, also, I recently gave a TED talk that has come out so I encourage you to go check that out. Yeah. Yeah. No, I'll put it in a link. It's very, very good.

22:33 Yeah, I'll make sure if people are watching on YouTube, go to the comment section, the links and I'll put it there. OK. OK. Fantastic. Uh So to me like, you know, one of the really impressive success stories is the AI-based weather model, FourCastNet. That was the first AI-based model that we started working on more than three years ago and we released it first and other teams followed up and we now have a whole family of AI-based weather models, right? And what is impressive is, you know, there was a lot of skepticism as we were working on it because a lot of domain scientists felt oh there's decades of work that has gone into building these weather models through numerical methods.

23:15 And what those methods do is from ground up, try to simulate the physics, right? Like looking at fluid dynamics through NA Stokes equations, looking at heat transfer and all of these processes that you're simulating and using that to forecast the weather in the next time step. And there was a skepticism, how can AI learn all of this complexity? It's highly multi physics, high dimensional uh can AI really be able to do a good job in this? And to our surprise, you know, our very first attempt got us to being tens of thousands of times faster than weather numerical weather models. But not only that, it even ended up doing better on many aspects like extreme weather prediction.

24:02 Uh In fact, the recent Hurricane Beryl, our forecast model had like a better kind of uncertainty band around the where the true landfall happen compared to the traditional weather models. So we're not even having a question of trade off that AI is so much faster but worse. That's not the case AI is both faster and better, right? And why is this happening? So it's better because it's able to learn from all that historical data and able to adapt based on that. Whereas numerical models tend to be a bit more rigid and you can't easily adapt it with respect to the data. Um The other aspect why it's so much faster is instead

24:46 of like doing bottom up simulation of all of the processes, it's really learning to take, let's say bigger steps, right? You don't need the fine grid or the fine resolution that Nayer Stokes in a fluid simulation with the traditional methods would take, you can afford to take bigger jumps because AI learns nonlinear transformations. So it's able to learn what is the shortest path to get us to the correct answer rather than being forced to take these very small steps on a fine grid that numerical methods need to do because they're each time solving it from scratch and they have a fixed set of steps that is already presigned and not learned from data.

25:30 And maybe for people who are not so familiar, um what uh could you maybe describe a little bit the FourCastNet and the neural operators that are, that are behind it, the at a high level, the theory behind these these approaches. Yeah, absolutely. So you know, when it comes to training weather models on the data, right? There's historical weather data, we can ask, oh there's the current weather forecast, the pre the future weather, right? So you can do this uh forecasting model uh through training. But the question is of course, what is the right kind of model architecture to be used in these kinds of processes? So if you look at fluid dynamics, for instance, how the hurricane moves,

26:14 you can't just eyeball it and precisely predict where it's gonna move, right? If you just stare at a hurricane, we are not good at predicting where it's going to go. So this is a superhuman capability. So it's not just like, you know, we can do intuitive physics of very simple kind, but this is much more complex. And that's because it requires fine scale features, meaning you need to zoom into the details of the very what we call fine resolution or fine s and see how they move and how that relates to the macroscopic behavior. And if you use standard machine learning models like transformers, they have to be working on fixed size patches because the tokens are fixed there.

27:00 And that may be OK in some applications. But if you really want to get to the fine details of this fluid flow, you should be having the capability of working across resolutions. Meaning one model that learns information across multiple resolutions. And that's what neural operators that we designed are able to do because what they're learning is mapping between functions, meaning it represents data as continuous functions and maps them to answers that are also modeled as continuous functions. Meaning we are not just thinking of the globe as grid points but as continuous processes that happen everywhere along the globe, right,

27:40 not just at grid points. And by learning such a model, we can really be able to now capture these fine scale processes accurately. So one of them and it's fair to say that this really has been a seminal piece of work that is, as you say, seems like it's uh kicked off a a very positive and needed. Um I don't want to say race, but, you know, there's, there's a lot of people now trying to sort of outdo each other, which is sometimes good for, for science when that sort of thing happens. Um But one of the discussions I think people have, um and I'm sure you meet people like this all the time who are maybe slightly more skeptical of,

28:27 of machine learning is, is the physics angle. If I'm not mistaken that in that approach, you're not explicitly solving the PDEs, you're not explicitly encoding the boundary conditions. You, you're using a more data driven approach to do it. Is that correct? No, not entirely because you have the flexibility of doing it as a physics informed approach where you can add the physics losses along with data, right. So if you only did it on the physics losses, it's too difficult on optimization problems. So you just cannot solve it. So this is one of being practical that you can do both a mix of data and physics. But the aspect is no matter how you train the model,

29:14 you can always in each instance test whether it satisfies your loss of physics, right, or your your PDEs themselves. So you can always like test it for validity in physics, which is a great thing, especially for partial differential equations. We know what the ground through satisfied. So it's always easy to verify if the answer that's obtained is correct or not. Um So I don't get the skepticism because in cases where you can easily check if machine learning is correct or not, should be the ideal case to apply it because you can always say that, oh, this is not accurate enough for my application. In which case I can use that as a precondition

29:57 and initialize my software and further solve it even if you know, there are all kinds of ways to still use this model. But in areas like weather modeling, what you've seen is that it does better than what decades of numerical weather models have shown. And that's because it's able to learn and adapt from data. And what numerical models are not able to do is about the modeling error, right. So you can assume this is the model, but that's not how the planet is. There's always deviations from that model. So you should also account for modeling errors which is difficult with numerical methods compared to data driven approaches because you with data,

30:37 you can really bring down those errors and fit to the data. Well, I I think um what I was meaning more is uh for the weather, it seems a very ideal use case because there has been this collection of data for decades and at least currently openly available data. Um And so that's what I mean in in the FourCastNet, you don't have to use a physics informed type approach because you have a lot of data to learn it on. Um what a and I guess the geometry is always fixed because it's, it's always the earth. Um where if we now move to, let's say aircraft design or, or the bigger fluid dynamics,

31:25 uh where do you see, where do you see where we're at at the moment and where we can go, how do we, do we have, do we have enough data basically and therefore should it motivate other approaches? Yeah. So it's always a challenge of like, you know, how do you get enough data? Right? And we can use the current numerical solvers to do it. But the question is of course, what is the computational cost for it? And is there a way to reduce that? And this is where a number of techniques we've been developing has been helpful where we look at like progressive training. So you have a curriculum of training from simpler physics to more complex physics

32:08 and that simpler to complex can be in terms of, you know, thinking about lower no numbers and slow moving fluids and then fine tuning on like faster moving fluids. So that way you kind of reduce the requirements of data when it comes to the more complex physics that the numerical solvers need to generate. Um We also have like a recent work that will be releasing soon where we'll show that this approach is better than you know, trying to do closure modeling and other kinds of like traditional approaches, people do where they still keep a core skills, co solver and only use machine learning to refine that. Um Whereas this progressive approach just kind of

32:50 completely gets rid of any, let's say traditional solver and learns from data in a way that it takes it all the way to even getting to the complex. And along the way, you can always also do hybrid modeling where you add physics losses along with data. And of course, there is an art to it because you don't want to make the optimization landscape too difficult, right? So you need to kind of be much more thoughtful of the algorithms to make this work. And it's not as straightforward as text models where a lot of data is available. And there isn't a notion of curriculum, you just kind of take in all of that data and you just uh train the model.

33:32 Uh whereas in this case, it's more nuanced, but at the same time, there's an opportunity here to make it work other than just the brute force standard approach of like ingesting all of the data because it's just too expensive to do that. Hm. So where do you um I think what you're alluding to is this idea some would call and I'd be interested to know what you think about this more like a foundational models which you start to give it enough data from maybe other tasks that it is able to predict something quite, it's quite generalisable. Do you think that how, how, how broad can it truly be? How ambitious do you think we could, we could be in this area?

34:21 I mean, to me, the sky is the limit and just as we speak, my students are presenting at ICML. Uh you know, I don't know if anyone lives there right now. I know I in Vienna. But uh anyway, so they're presenting it right as we speak or around this time where we've created a, let's say, a GPT-2-sized model that is able to learn on multiple families of different partial differential equations. And also showing that one model can broadly learn across these domains rather than narrow models trained only on those narrow data sets. So you have this cross domain learning that gets that benefits from having that approach of learning multiple phenomena at the same time.

35:12 And do you think um one of the uh statements I've often heard um from, from various people is this idea of scale that um actually for a lot of the large language models, there's been a focus on any architecture that could scale for huge amounts of data, maybe at the expense of accuracy. I I would say I'd be interested to know you think, do you think the method we have now can truly scale because I've seen limited examples in academia or industry so far where it's at the scale of, you know, hundreds of millions of points of grids or, or thousands of different cases, is it just a matter of time to do it?

36:00 It's just a matter of time and resources. If you see our model, you know, there are two kind of aspects, right? You can think of like the weather model, which, like, the largest one is, let's say, a little less than a GPT-2-sized model is working very well for the narrow domain. And now we have a broader foundation model that is getting towards universal understanding of physics. But that's only GPT-2-sized, meaning it can't possibly have the ability to solve very complex three dimensional tasks, right. So it's kind of like able to do flow dynamics in a certain regime, but it's a limited regime because the model has only limited capacity.

36:39 But with this, what we are able to demonstrate is it can be scaled further and neural operators as a foundation can just work across resolutions in the same model. So you can have like different PDEs be queried in different grids, different resolutions and just one model can handle all of that, different geometries, all of that. So where does this bring to the future of traditional simulation methods? Traditional, you know, find it different, find an element in commercial or open source codes. So to me like it's there'll be a time where it won't be one versus the other. So like I mentioned AI plus science. It really be the ideas from these

37:25 numerical simulations would be deeply integrated into AI, right? Not just the solvers themselves in a black box, which I don't think it is a good idea for a variety of reasons. It's not a lot of researchers attempt to do it. I would just think that in future that's not a winning path for, you know, whole set of reasons that I'm happy to get into. Uh but it's really gonna be the aspects of how do we take the best of all the algorithmic ideas that have been developed for decades in numerical solvers and integrate it with AI algorithms together. And in fact, that was also our inspiration when we came up with neural operators,

38:03 right? This idea that numerical solvers are able to solve um on different grids, you can query any point in the domain. It need not be on a fixed grid or a fixed resolution. But neural operator before we invented neural operators, the neural networks that were standard didn't have that ability, right? They were always on a fixed resolution. So how do we bridge that gap? And how do we think about, like, numerical solvers like pseudospectral methods and then make them learn from data. So instead of a rigid like aspect of going between the frequency domain and the standard domain, we also have learnable parameters and nonlinear transformations in between.

38:46 And that came resulted in the Fourier neural operator. So we are continuing to take all kinds of inspirations from traditional methods to strengthen these methods. And to me, I think that is a better approach because by definition, that would be taking the best of everything. And so it wouldn't be just purely A I numerical methods deeply integrated into those ideas. So do you think that to um to achieve this goal? And I'm certainly interested to read that paper, maybe by the time this comes out, I can put a link to it, so maybe we can have a chat afterwards what you're allowed to share at the moment? Um Is it therefore a need for us

39:31 to somehow bring together the community to generate this training data? Because I guess a lot of this assumes that there is a broad enough set of problems and data to create these models or do you think this is going to be different than your large language models? Because people are not going to want to essentially release uh their data? Well, you know, in my view, so many of the solvers are open source, right? I mean, sure there are closed source solvers and there are some details of which one is that are capable of for a set of domain. But there's a lot that we can generate with open source solvers. And that's what we've done now recently and we've released many PD benchmarks,

40:16 other labs have done that and in our recent ICML paper, it's really the largest collection, right? We took all of the available PD benchmarks and trained a model on it. And to me, I think it's a continuing journey, I'm aware of different groups are gonna further working and are gonna release bigger data sets. So this is an exciting time. OK. That's good. And, and how about other domains? So we, you know, weather is obviously one. Um you were you were talking about, you know, some of the fluid dynamics, but are there any other areas that particularly excite you in terms of the potential of AI plus science? Yeah, I mean to me like if you look at partial differential equation,

40:58 there's so many different areas, right? So to give you some examples where there have been already success stories, uh nuclear fusion is one of them, we worked with UK Atomic Energy Agency and created these AI-based simulations of the tokamak— you know, how plasma evolution occurs in these nuclear fusion reactors. And how can we predict disruptions? Meaning when the plasma may escape confinement and could potentially damage the reactor, which you don't want it to happen and to be able to do that these AI models are should be very fast. In fact, they are faster than real time, right? That's why we can take corrective action.

41:38 On the other hand, if you try to run traditional magnetohydrodynamic simulations, which is what describes this plasma evolution that would be extremely slow. Uh In fact, our methods are a million times faster than what traditional simulations can do. And so that shows what we can kind of like now think about using these models from physics in applications that you earlier couldn't even conceive of. Right, like plasma is one control of drone is another, you wouldn't dream of putting a CFD solver on a drone because, first of all, it's kind of going to take a lot of time and energy to run anything and you know, and it's just going to be too slow and by the time the drone would have crashed.

42:22 But now we are in a place where we have used machine learning techniques that can help make the drone flights be better and safer. And so in all of these areas, speed is important, right? The cost of uh coming up with these predictions is important. The other aspect is going back to not just simulations but the aspect of design and discovery itself. So if you recall, I was mentioning the scientific method of like going back and forth between ideas and lab testing and the lab testing is the critical component. So it's important to come up with ideas that are already physically valid. And hopefully, you know, there's an internal simulation that certifies the model

43:04 to be the design to be correct. Otherwise you will spend a lot of time with going back and forth with lab testing. And we did that recently with the medical catheter where, uh we designed a better medical catheter than was previously available. Uh, so I'm not sure many of the listeners here know about the problems with medical catheter. Uh, it's a tube that takes fluids out of the human body. Very simple, but it's one of the, the most common cases of healthcare related infections, more than half a million cases just here in the US annually. And so, you know, this is a problem that is in fact described by physics pretty well.

43:44 Meaning bacteria tend to swim upstream near the wall of the pipe, right, where the slow fluid, the fluid outflow is slower and hence swim into the body and infect the human. And now we our collaborators in fluid dynamics had a simple idea. They were like, let's create like ridges, these triangular kind of shapes inside the wall. And with that, you can create vortices. So the bacteria doesn't have this ease of just swimming upstream and going into the body. But of course, the question is what is the optimal design for those shapes that best reduces bacterial contamination. And with our neural operator based model, because it's an AI model,

44:28 it's differentiable, meaning we can directly have gradients for improving our design and come up with an optimal design. So our AI model already proposed an optimal design and then we had to go to the lab and 3D print it just once and it resulted in 100 fold reduction in bacterial contamination. So, you know, think of like a future where AI imagines all kinds of new back designs, but it's not just hallucinations, it's not just a creative concept, it's physically grounded and valid and then we can actually bring it to the real world and show the impact in the real world. That's the future. That's very exciting to me.

45:10 Yeah, that, that, that is super exciting. And, and I think um it's probably the one that in, in some ways industry is most, not most interested, but I think the generative design I think is one, you know, most, whether it's an aircraft, a wind turbine, a fridge or whatever, you're trying to come up with a better design normally and simulation is just a tool to go that way. Um And I think it's there have been, you know, adjoint methods and, and other sort of methods that, that try to drive, but I think it's still not still not realized that dream of sort of a computer doing it for you. You know, there's always a very heavy human in the loop, et cetera.

45:54 And I think what you're alluding to is that potentially AI may help that come closer to reality because of the speed of the simulation. But one thing a previous guest mentioned to me, and it really got my sort of my thinking is in my head, I've split transformer large language models and you know, uh neural operators or PINNs or, or graph neural nets as if they're sort of two separate things. But how do you see because humans do ultimately like to verbalize something or the sort of prompts that we ask larger language models? How do you see the potential of integrating these together? Yeah. So first of all, I want to kind of make a clarification

46:43 that transformers are not separate from neural operators, right? Neural operators are the super class where you can now have transformers that work at all resolutions. That would be a neural operator. And in fact, if you one simple way to think of an example of that is you know, instead of the attention mechanism working on fixed tokens, you can work it in the four year space and like you know, go back to the standard domain with inverse Fourier transform. So if you do that now you can extend it to multiple resolutions, which essentially is a Fourier neural operator. But it's also a transformer if you parameters your attention with the four

47:24 Fourier transforms. So there's a lot of interesting math connections that mean that you know, they are not as separate and different as people may think it to be and neural operators are not as exotic as people may think it to be, right. So there's a lot of interesting nice connections. Another example is also like graph based methods also being operators when you have the flexibility to add new nodes to the graph and be able to adapt uh with new nodes, what are the new edges? And you can do that with like spatially defined graphs very easily. Uh So there's all these standard methods that you think of as quite separate that are really just special cases of neural operators.

48:08 So it's really that broader concept that we are describing. And uh now the question is can text be an interesting modality to me, like you can add all these modalities on top of a physics based model where there is physical understanding, right? I mean, text is a way for humans to interact. If you want to design an aircraft wing or a drone, you can kind of have a chat if you think that is the best interface. I mean of course, it depends on some certain people preferring that versus the other kinds of interfaces, right? So language can be one modality, there can be other modalities like looking at uh right, observational data from video feeds and so on and

48:50 all kinds of other aspects of multimodal model. But to me the foundation of all this is physical understanding. So you can build this and on top of it comes the other modalities, I guess what I'm I just want to see if you think this is purely fiction or is actually something that could happen, which is at the moment, you go to a, you know, various large language models. Um and you say create me a picture of a plane flying OV over Mars and it will create you something depending on how well you do the prompt, you know, pretty nice, pretty accurate. Now even more, create me a a movie of, of a plane going through. What about the reality of saying,

49:35 create me a design of an aircraft that can efficiently and then it goes off, creates a design, uses the model to go and simulate it. There's 1000 of those simulations come back and gives you the design that you go in 3d print. How much is that sort of science fiction? And how much could that actually happen? I mean, in fact, it's already happening, right? That's what we did with the medical catheter in a short smaller scale, I think to me, text is not the important aspect. Uh text is almost a distraction because if you can nicely describe and in fact, for, you know, design considerations, you want more clearly describe your design cost functions,

50:20 right? That you can go optimize with our neural operator foundation model as a way to get to those design goals. And sure you can try to specify it through text. But to me, like with text, you must still kind of end up making mistakes of translating it into the right design course. So if you could directly do it, I would just start with that because that is a much more cleanly defined problem. Yeah. Yeah. No, you, you, you're probably right. I think so. I think it's more the, um, the conceptual design, you know, the, the, the sort of market who just wants to, uh, very quickly go through designs, which I guess has been the dream of real time simulation, hasn't it,

51:02 this idea that a person in design studio? But, yeah, you're probably right. In reality, an engineer wouldn't just type it because it probably wouldn't be precise enough. Um You would, you would want to interface with, you know, some tool. Um So how what needs to be done then for this to happen at an industrial scale. So more practically now, where do you see the role of academia of industry of government funding? What's, what's, how is it now? And where would you like to see it go to really make this a reality? I mean, we certainly need more resources is the short answer, right? There's a lot of kind of attention paid to

51:47 language models and also now recently robotics, but you know, robotics is still kind of like uh data hungry and their data generation is even a harder problem than I would say with physics based simulations because you have the simulator, you can generate as much data as you want in our case. Uh But of course, the main constraint is the getting the resources to do that at scale. And you know, in various kind of aspects, we continue to scale these models and other teams are doing that. I think the question is how to get us to that next level where we can train it on much larger clusters and really get this to a place where

52:27 we can show those benefits of scale of being able to do cross domain models that can understand multiple physical phenomena as well as what we call multi physics stability to couple different partial differential equations together. Mm Yeah, that always seems like the holy grail that even with physics based simulations, it's still never really fully done. People do tend to be siloed, don't they into a fluids or structure or climate? Even though technically they are kind of connected at an engineering um level. Is that really your vision that there is—it's easier in a way to integrate it through an AI model rather than

53:09 these more traditional solvers. Yes, precisely. And, and to me that's the only fact to making also these AI models realizable and practical, right? Because you need to have the shared data from all of these different use cases to help learn a good representation. Otherwise the data needs would be enormous for narrow surrogates. So he here's a maybe a bit of a different question for you. But um you, you started doing your research and your academic career and you were arguably in the early days of ML where you were, you were moving into an incredibly exciting field that, you know, you've made massive contributions to,

53:53 if you were now giving advice to a PhD student or maybe an early postdoc, what, what area of, let's say, machine learning would you focus on bearing in mind? You know, the, the time it takes to, to sort of progress, are there certain areas that you see as being very promising that would be good for somebody to get into? Um Yeah, thanks Neil, I guess to me like, you know, there's the question I get a lot because also people are like, how do I distinguish myself? There's a million other people working on language models and how do I not have enough resources compared to people in the industry? Right? And it's not a level playing field. And to me, I think the main aspect is

54:37 doing a PhD that means you should be working on problems that most others are not paying their attention. And that could be anything, right? I mean, in my case in my lab, that would be AI plus science. And you know, not just uh you know, we're doing neural operators, but we're also looking at the fundamentals of learning itself. Like what are the optimization challenges and all the way to like problems in biology? You know how to train models for protein design for um being able to also generate new genome sequences of viruses and bacteria being able to like do drug discovery more efficiently. So there's all these different areas, right?

55:18 And there's application areas and then there's the fundamental problems within them and how do you formulate and make impact on those areas? And there's still such a green field. And to me, Caltech is an ideal place where because of its small size, it's highly interdisciplinary. And if you look at almost any of our pre papers in these different domains, they have people from those domains, right? So it's a very interdisciplinary collaboration and that means we are also tackling the right problem. We aren't just doing machine learning blindly putting a hammer, but looking at metrics that are relevant to the field.

55:54 And that's also what I think makes it difficult for many people because in machine learning L A there's one objective function, you optimize it, you're golden, but that's not how science works. Even if the weather model wasn't just like root mean square runner or just some metric, one metric, right? It's like, oh how does it do one extreme weather events of all kinds of things? Right? It's like all these different aspects of like, oh what happens if you keep running the model for a longer time? Is it stable? So you have to look at more than just one objective function. And that to me is a rich set of even foundational

56:33 questions to be answered. How do you formulate those kind of multi objective machine learning. How do you kind of prioritize different objectives because the optimization landscape becomes too difficult. And of course, in the, even in those application areas, there's a lot of deep thinking and working closely with the domain scientists to understand what matters for them. Um And so my advice would be like this is not something that's easily done in industry. Sure, in certain areas, there are industrial groups working on it. But you know, there's always some such broad areas of science and engineering that you know, academia has an upper hand.

57:13 And so to identify that uh uh but you know, similar argument could be also made for social sciences, for economics, right? Interdisciplinary problems are where I think you will be able to make unique contributions and have a Yeah, that's really uh that's, that's really good advice. And another because I like to make things quite practical. Where would, where would you recommend people go to stay abreast? Is it the standard? You know, ICLR, ICML, NeurIPS—are there other places that you're seeing emerging as where people should go to hear some of these newer ideas at? Yeah, very practical level conferences, symposiums.

58:00 Where would you recommend people go to, you know, to me, social media is not something I would recommend. Once in a while, it's still you may come up with something useful but days of course, a whole other conversation, it's too distracting for a variety of reasons. And so that leaves us with various of these academic conferences, right? So I, I think they are just still a great place to go. But the main events are quite crowded. We have, like, now NeurIPS with tens of thousands of people. So I would highly advise going to workshops, going to meetups, like smaller groups. And you know, it also not be very popular events worldwide.

58:43 It may be something local, you know, in New York City, different meetups, different uh just small workshops. So focus on attending the smaller events because that's when you can have real conversations and just, you know, get to know people who are working on related problems and what in their experience didn't work. I think that's the aspect that is very hard for people to get in an open forum like social media and you really have to go talk to people. And so I advise my students to just try and attend a lot of events locally here in California because they learn a lot from that. And what about um one of the things that sometimes come up in some of these workshops

59:26 is this idea of bringing the communities together, the domain specialist with the sort of ML specialist, do you think there is a need for the ML community to almost venture out from their comfort zone of NeurIPS and ICML and actually go to engineering conference or, or go to a, a Biology conference. I mean, I'm sure that is being done. But I, I was amazed when I went to, uh, ICLR so it was a huge number of people that then when you go to the big aerospace conferences or engineering, you see very little of, of those people yet they're both kind of working on the same problem. Do you think there are ways that we can improve this situation?

1:00:13 Yeah. No, I'm smiling because this was kind of my goal when I came here to Caltech, right? In 2017, we started the AI and Science initiative here. And that was one of the first across any campus to have that kind of an initiative to bring together people from many areas and really debate how AI can help them. You know, they may think, oh AI is not already, I don't need to worry about it and then there may be, you know, but we need to also kind of develop methods where it's currently not working. Like currently just doing a hammer approach, they didn't get a machine learning to work, doesn't mean it's not an interesting problem, right?

1:00:50 To me, those precisely are the interesting problems. So we created that initiative, we had regular workshops, we continue to have them. In fact, now we are doing it jointly with the University of Chicago. Thanks to a generous gift by the Pritzker Foundation. So, you know, the aspect I think is maybe the traditional venues are not so great for it. It would be my reaction to that because they're just too big, right? If you're going as a machine learning person, especially a student to this enormous event and you can't really make sense of like what aspects of it is relevant to you as a machine learning person. So you really have to go to talk to people who are invested in making that happen.

1:01:34 You know, as you said, there's a lot of skepticism in the field I have kind of, you know, face, let's say different kinds of personalities, you to pick people who are willing to disrupt their own methods. And here at Celtic, I'm lucky to kind of have collaborators in that realm who really are, you know, joint inventors of neural operators because they were willing to take this leap and say all they worked on numerical methods. Let me see what AI can do. And that's I think harder to combine a big conference. I think they should just talk to their colleagues in other areas, just grab a coffee, grab a drink, keep it casual, just go and learn and survey a lot of problems.

1:02:15 You know, I was lucky to do that because when I came into Caltech in 2017, I started this AI and science initiative. But I also had this generous gift from AWS to, you know, thanks to AWS have cloud credits and distribute that across campus. So I invited proposals and just learned everywhere. What are people doing, you know, how are they going to use the cloud, not just machine learning, right? All kinds of computational methods on the cloud. And you know that jump started a number of projects and collaborations. It also led people to thinking about using the cloud and machine learning methods more because the resources were available.

1:02:56 Um So I think we need to think of more creative initiatives like that at a smaller scale, you know, and then, you know, the question of course, how to go from there and do much bigger events. But at least at that point, that was a great way to jump start this initiative. Yeah, it definitely seems um the only way to truly convince a community is to work with them. And I, and I guess, you know, be, be inside of it rather than a than an outsider. I think people are always scared of change. And I think um it will be interesting to see and that, that's kind of what I was saying, you know, what, how close are we for things being ready? I, I guess the um

1:03:46 this is to some extent, which is why I was asking about the solvers in things like fluid. And I think weather is a bit different because I think weather codes are typically developed by the weather centers and, and so they have, they're in sort of control of their own destiny to a certain extent. Um Whereas I think it's, it's really interesting in the fluid dynamics or computational fluid dynamics community, because there is such a reliance on large commercial codes that in some ways there's a, there's a need for them to get on board to expose that if you know what I mean, like academia can do what it does, but you kind of need

1:04:27 those big powerhouse companies to sort of embrace it for it then for all these smaller companies to, to take advantage of it. Do you know what I mean? That there's, there's a, there's only a certain amount of academia you can do when you need these sort of software houses to, to get involved. Have, have you seen that or are you more optimistic? Now? I'm certainly more optimistic. Of course, I, I come from a different realm. Right. So academia has done wonders. So I wouldn't say that it's not possible. I mean, we continue to open source a lot of what we do and I'm also doing to, I think sometimes disruption happens from outside then with them.

1:05:06 Yeah. Yeah. Yeah. You know, maybe one way or the other but in the end progress will happen, which is great. Yeah. Yeah. Well, II, I would love to talk for hours and hours but, uh, I know you're a very busy person so I'll, uh, I'll make sure not to take too much of your time, but I wanted to really thank you for, for talking about this today. I I'm gonna share a lot of links because I think um I appreciate you stayed high level, you know, II I asked you to do that, so I appreciate it. But um I know there's a lot of content and papers that people could jump in to really dive into the work that your group's been doing.

1:05:43 So if people are watching this and you have a look, I'm gonna put a ton of links down there that you can dive into some of these papers and of course, watch the, the TED talk that I, we were discussing. Um, before we started that the only problem with doing a podcast and you talk about complex sciences, it's kind of hard to explain things without sort of illustrations and things. So I think your, your TED talk actually does a very good job of doing that. Thank you, Neil. No, this is so much fun and kind of took in all kinds of interesting directions. So I'm really glad that you're doing this and getting attention to this area.

1:06:18 You know, it's still a bit more niche, right? Compared to language models. But hopefully we can all together, bring more attention to this. Exactly. Thank you. Yes.