AI for Science — Personal thoughts and perspectives

0:00 Hi and welcome to the Neil Ashton podcast. In each episode, we explained some of the fascinating ways that science and engineering are changing the world around us. We talk to leading engineers from elite level sports like cycling and Formula One to some of the world's top academics. To understand how fluid dynamics, machine learning and supercomputing are bringing in a new era of discovery. We also hear some of their life stories, their career advice and lessons they've learned on the way that I hope will be helpful to you too. So sit back and enjoy this episode. Hi, and welcome back to the Neil Ashton podcast. Today's episode is really setting the scene for

0:46 what are gonna be a couple of really interesting interviews and discussions with some world leading experts on machine learning. What the next month's episodes are really gonna be focusing on is this topic of AI for science or how artificial intelligence, how machine learning can be used to solve scientific problems in addition to the way that machine learning is now increasingly being used, uh, for generative AI for these large language models that you can go in and you can ask questions. You can generate images. Increasingly, you can generate short clips. How or is it possible for some of those to also solve some of the scientific problems like

1:37 computational fluid dynamics, weather modelling, material design, drug discovery, et cetera, And I just wanted to set the scene. Really? Today is going to be a bit of a shorter episode, and I wanted to maybe, um, discuss some of the fundamentals in terms of why this is becoming interesting. And I'm gonna talk through the lens of Of of CFD of computational fluid dynamics, um such that hopefully the next few episodes when you listen to them with some of these world leading experts, uh, you'll get a little bit of another context because we may dive in straight away in those conversations. So I thought this one might be useful to

2:17 to really explain some of the high level concepts. So the way I would say it is this. So let's let's look at, uh, computational fluid dynamics. Let's look at what the previous you know, nine episodes have essentially been about, which is OK. I want to design a car or a plane or a wind turbine or a jet engine. And, uh, I could do it physically, as in, you know, I could build it, fly it and test it, but most of the time that's too expensive. So so people don't do it. That's that's really left, right? You know, to the end. It's not really a development thing. Sorry. I think I've got a bit of a cold or something. So my voice is going a bit,

2:57 Um, what is increasingly used more are either wind tunnel tests. So some sort of artificial environment that mimics the real world. But let's say in a very large warehouse where you blow air over like an object or you do some sort of lab test where you maybe have like a jet engine in S in, in inside a building which is detached from a plane and you do some experimental tests. Um, but even those are are are arguably still quite slow. And, um, limit the number of designs you can explore, and hence why? Simulation is so, uh, widely done now. And hence why? Computational fluid dynamics is this important area that is continues to grow in terms of its applicability.

3:47 And that's why I constantly talk about CFD because it really is used in so many of the things that that you're used to doing. But if you're doing uh, CFD and this applies also to, let's say, drug discovery or weather modelling or materials design. Typically you're solving some, you know, partial differential equations using some numerical methods that essentially have typically, uh, a quasi linear relationship between the accuracy of your simulation and the amount of time it takes to run it. I in general and different methods have different order, uh, of of of computational cost versus accuracy. But you know, in general, let's say, for the simulation of a car,

4:37 um, the to make it more accurate. I need to reduce the error in these numerical schemes by having um, less dissipation smaller digitalization to these sort of mesh points that you have more of them to be able to capture more of what's really going on. And as you add more of that fidelity in space and time, because most of the time these turbulent structures have a very small you know, time scale, and particularly true if we start getting into Multiphase and, uh, and combustion and things like that the time it takes you the number of iterations you need to run and the amount of memory and compute goes up and up and up.

5:19 So, um, that's why high-performance computing is such an important discussion. It's why I kind of link them together. Fluid dynamics HPC because they are inherently linked. But the third topic that I mentioned, I think, in the intro to every podcast is machine learning. And that's why I wanted these next few episodes to really dive in and help to be honest Me, I learn every time I speak to somebody, uh, but also yourselves to to get this sense of why I think machine learning has a strong potential in this area of, uh, Formula One and cycling and engineering and fluid dynamics. So what I said before was, in general,

6:04 all these simulations are getting more and more accurate, but they're also requiring more and more compute Now. Interestingly, machine learning has developed over the past decade, I would argue, but certainly has ramped up in the past 23 years, to the point that actually, and I'm gonna speak it at a high level because these next um, interviews are gonna be the chance where we can go in a little bit deeper. So this this is like, I guess, the sort of 101 class they developed to the point that, in theory, machine learning is all essentially about inputs and outputs, isn't it? You know, essentially, you've got some input, some data

6:51 that you can train a machine learning model, some sort of, you know, neural network, usually to come up with some weights, et cetera that then once it's trained, you can give something some new, unseen task that it can predict the output for. And to do that, you need some inputs and you need some outputs. So in the context of computational fluid dynamics, the inputs could be lots of cars or planes or wind turbines. It could be their geometries. It could be their volume measures. It could be a a variety of things, but essentially the inputs are usually the simulations of many cars or planes, or it could be more the same plane or the same car.

7:37 But lots of different boundary conditions, so different wind speeds different your angles and the outputs could be their lifts and drags, or it could be their flow fields of the lost and the pressure and the nodal points on the mesh. And the model is essentially learning that link between those out inputs and outputs such that now because it's learned that mapping. I can give it an input, let's say a new car or plane and it can go and predict me that output. But the real reason I guess that it's so appealing is because that actual prediction step that inference step

8:25 is essentially nearly real time seconds, typically less than a minute. And if you just focus on that pure imprint, step that pure prediction step and you say it's, let's say, worst case, let's say a minute a minute on what's often, uh, you know, a single GPU or even a CPU at this inference step compared to I don't know, um, 48 hours on 1,000 CPU cores 48 hours to a minute is a dramatic change, and that's why you're seeing many companies many papers claim 10,000 times faster and strictly That is true. It's 10,000 times faster. I think if you do the maths now, people will say rightly so. Well, what about the time to train the model?

9:19 Well, the time to train the model depending on what What methods you're using. And if you look in the literature is probably still in the in the hours, maybe maximum a day. So the training cost of time is is actually, um, for these sort of, um, methods typically not actually that long compared to the cost of a CFD simulation. Now I'm talking at a very high level here. I appreciate it. I'm just trying to set the scene. Really? There's nuances and there's different approaches, and that's what we're gonna explore over the next few episodes. But in general, the cost of training and the cost of inference is actually relatively cheap.

10:06 The big one, of course, is you have to have the training data, and this is the interesting similarities and differences between machine learning for scientific problems and machine learning for, um, I'm not sure how you would phrase it, but for large language models, for the systems you use, such as ChatGPT, Amazon Titan, Bard or Llama models, in those scenarios the training data is widely available because it it's and maybe it's getting harder now. But previously it was simply scraping off the Internet lots and lots of of texts and documents, et cetera. Now many of these companies have signed deals with, um, magazines or newspapers or various other entities to get the data.

10:51 But it's I would say, it's largely available and, um, not huge in size. So if you look at these models, the actual data itself is, you know, gigabytes or terabytes. It's not in the petabytes and the real cost of training these large language models that many of you are starting to use it is not the data collection per se, although there is work to do that, you know, labelling and preprocessing, but more the actual training time. So you know you might need 4000 or 10,000 GPUs running for weeks, and this is what's driven this huge boom in AI, uh, chips, because the training time is so massive. If you contrast that with, um,

11:35 this is why we jump into so many rabbit holes. So bear with me for many people at the moment. The big difference is that for scientific problems? Many people are not talking about foundational models, and I'll explain what I believe that stands for in a moment, but more that you are training a model each time, essentially, so the training time is for your specific 50 cars or 100 cars, or or your specific, um, Earth model or your specific drug discovery. And so the training time is actually not that much and not that expensive. What is the real challenge is collecting the training data foundational models. The whole point of a foundational model is it's meant to be so foundational

12:26 that you should train it once essentially, and then you can just use it, with the exception being that maybe you need to fine tune it. But when you go and use any of these commercially available large language models, you are only doing a prediction step. You are not training the model. It has already been trained on so much data that is considered almost foundational. There is a current debate in the scientific community, but I think it's fair to say that most of the stuff that's been done to date in the scientific side, whether it's weather modelling, or CFD are not foundational models because they only work for quite a specific,

13:05 um, use case that it has been trained for. So it's not able to predict any fluid dynamics or any weather scenario. It is still within the confines of a particular thing. One of the debates is. Could we build foundational models for science? And that's one of the topics I'll be discussing with some of my guest. And I'm purposely not saying the names of the, um of the guest to keep it. I'll I'll, I'll sort of leak that information, Um, once they're actually recorded. But the but the it's quite exciting, and I'm really pleased who I've managed to convince to to have a chat and share their their wisdom. Um, so the all of that was to essentially say, Why is it so interesting?

13:51 It's interesting because ultimately the promise is that you can get a method that, once trained, can run in quasi real time. So if you are a car manufacturer, a plane manufacturer wind turbine, you can design and explore thousands of designs in in in minutes. Now that all sounds too good to be true doesn't it? Well, the the truth is that, um at the moment, there is still a, um, active debate on how accurate are these methods? Uh, and what can they actually do? So there's been lots of I would say fundamental work, and there are various start ups in the space who are claiming they can do more, but it's it's It's a very active area of research that,

14:35 I would say, is nowhere near as mature as some of these large language models that have that people are using every day and you can actually see their usefulness. But there was a huge amount of promise. And so what I want to do over the next few episodes is tease in where these industry experts think we're at and and where we could go to. One of the things that you're probably thinking, and I just want to briefly explain and mention again as a context to these upcoming ones, is physics. When you're training these large language models, one of the interesting thing is that when it when you ask it a question, you ask it to write a sentence.

15:18 You have not told it explicitly. What are the laws of the English language I If we're talking about English now, you haven't told it these. These are the AL. This is specifically the alphabet. These are the rules. These are the grammar rules. Instead, it's actually learned it essentially from the data. When it comes to, let's say, fluid dynamics. There is one argument that, and this is typically called sort of data driven approaches that if you give it enough data, it inherently learns the laws of physics through the data that it ingests, similar to how a larger language model is able to essentially learn the language without explicitly knowing the rules.

16:05 And you could argue this is a little bit how a child learns at the beginning. They're speaking the language through sort of just observation through trial and error, without explicitly knowing the rules. Whereas an adult, maybe then you actually have to learn the rules. And that's a whole other topic, how the brain works. And anyone who learns a foreign language knows it's much harder when you're older to do it when you're, um, when you're younger. But um, when it comes to through dynamics, there is the opposite viewpoint, which is that? No, we must include somehow these physical laws in the machine learning model in the loss function

16:48 it some mechanism to force it to abide by these foundational physical laws, the conservation of mass conservation of energy that we know are real and and help us. And that is typically what are called more physics driven or physics informed approaches. And it's very interesting debate, I would say still a very ongoing debate in the literature on which one of these is essentially the winner. Um, which is the most correct? Or is it somehow something in the middle where, um, it's a little bit of data, a little bit of physics? This, um, in my mind is very interesting, and it's still not clear, uh, whether it will be one of these sort of models that have

17:36 been coming out Or is it going to be something completely different? So I hope that I've maybe just framed a little bit of what's coming. What really I want to tease out in the next few months is to hear from these people, first of all, where they see the current state of the art where things are going and their optimism to what a future will look like in 5 to 10 years. Will we ever be able to have these foundational models of science where we could essentially almost ask it in a prompt like fashion? Solve these equations and it can go off and do it just as I can tell it. Now write me a script that does this. Is it theoretically possible that if it could learn it,

18:22 that we could actually be solving some of these fantastic problems just in a A you know, a few, um, types of a keyboard. It's for me. It's super fascinating one. And I know it divides many people. Some people see machine learning as just a huge hype that is marketing. Um I. I felt that way a little bit at the beginning. Now I'm convinced, actually that it there are huge potential changes, Um, and something that everybody should be far more aware of because I think it has a potentially transformative effect, particularly in the engineering, um, industry that, um just as computers and simulation transform the sector from the previous

19:06 days of sort of trial and error, physically making things that the the question is, how much can machine learning bring out a new revolution in those industries? So, yeah, um, that's this episode. We're gonna keep it sweet. Gonna keep it short. And, um, please watch out, please subscribe, please. Like, look out at the next few interesting, uh, episodes I think you're gonna like. I think you're gonna learn a lot from And, um, I. I hope you'll join me for those. So thanks very much for listening to this and, uh, yeah, hopefully, uh, you'll join us for the next few ones too.

20:00 That