Prof. Nils Thuerey on Differentiable Physics and Foundation Models

0:00 Hi, and welcome to the Neil Ashton Podcast. In each episode, we explained some of the fascinating ways that science and engineering are changing the world around us. We talked to leading engineers from elite level sports like cycling and Formula One to some of the world's top academics to understand how fluid dynamics, machine learning, supercomputing are bringing in a new era of discovery. We also hear some of their life stories, their career advice, the lessons they've learned on the way that I hope will be helpful to you too. So sit back and enjoy this episode. Hi and welcome back to the Neil Ashton podcast. So today's guest is Professor

0:43 Neil Torre. He's an associate professor at the Technical University of Munich TUM and has been a pioneer in physics based deep learning. We had a really interesting discussion today on first of all, some of his background, Very interesting that he actually won an, an Oscar working in the visual effects industry for, for movies after his postdoc and PhD before he then turned to academia full time, which is kind of incredible actually, and is interesting because I've noticed the lessons learned from the visual effects industry in terms of photorealistic representations of, of, of the world is actually something we, we then talked right the end of

1:36 the, the podcast. So make sure you listen to the end about the link between foundation models and surrogates to world models. These world models that are being created essentially as a synthetic environment to train autonomous vehicles and robots. And we actually talked a lot about whether there needs to be more physics in these world models, which I actually thought was a very interesting discussion. And we only got to it at the end. But you know, he he's been one of these people who was doing deep learning in the mid twenty 10's and early twenty 20s before ChatGPT and the sort of big rise. And it was interesting to him

2:18 talk about how his work was looked in those times compared to now and some of the barriers, you know, and, and skepticism maybe people had. He's also been someone who's really pioneered open source and pushing the boundaries of differentiable physics and some of the codes that him and his team has developed and now have moved into both generating data sets like the Super wing data set, but also addressing the challenge of foundation models also from a sort of PDE point of view in the latest Tadpole paper. So we, we, we kind of go through some of those topics around those different strategies of sort of PDE based methods, you

3:02 know, pre training on those versus doing more of the traditional, I guess, building out large data sets. We talk about the role of startups and industry and academia. Interesting to get his perspectives on that and how all three can work together to solve some of these problems. And we also talk about the open source, why he's so pro open source and how we think that can help progress the, the, the field along. He's, he's actually contributed together with his team, many important papers covering not just on fluids, but also on the, on the weather side, interesting like weather bench. So I, I put a whole link list of papers in the chat notes that

3:42 you can in the show notes that you can have a look at. But I, I really enjoyed this conversation. He's someone who is very modest in, in what he's done. But if you actually read his papers and look through, you can see he's being quite influential in steering these topics and always seems to be one step ahead of where most of the field is AT. And so if you look at his 2025 and 2026 papers, you'll know what I mean by that. So hope you enjoy this episode with Professor Neil Torry. Well, yeah, thanks for thanks for joining today. I, I've been very keen to speak to you because as I've been getting up to speed myself over the past years on machine

4:24 learning and differentiable side. Your name is a, a constant, you know, top of the list in terms of influential papers, in terms of doing things what seemed to be earlier than most others have done, you know, setting the scene, which suggests to me that you have quite a good outlook and, and, and thought on it. So, yeah, thanks very much for for for joining. And yeah, maybe as a starting question, you took a diversion. Well, not diversion, but you spent some time doing visual effects after your PhD and postdoc maybe. What did you do your PhD? What was the postdoc? And then I'm I'm intrigued on on the movie stuff. Thanks for the invite.

5:09 First of all, Neil. Yeah, I'd be happy to tell about this. My background, right or over my career, I have switched fields a couple of times. My PhD was in a in a computational numerics computational physics lab, basically targeting multi grid methods and and fluids. And then I switched to computer graphics for a while. I guess that that actually was one of my driving my motivations from the start. I guess I'm a visual person. I'd like to see how things evolve. And I think that's that's still actually one of the main things that's fascinate me about fluids, these swirling motions and the things that actually do come out once it's working. So we also worked on computer

5:53 graphics, BASIC for computer based animation for quite a while. My post I get ETH and then afterwards also did actually work in industry for a while because I was curious how things would actually work out in industry in a practical environment. And partially also because I couldn't really get a good faculty position at the time. So right, not not completely voluntary in a while in a way, but right, it was good to see in the end. I noticed for me, research is the more I think the better field. But yes, that's that's why we did my poster basically. And then I think that if you 10 years ago, we started looking at machine learning. So previously we did in this ETH

6:36 group what was called data-driven approaches in computer graphics, essentially also trying to work with data and right, somewhat aligned, but we didn't have the tools back then. So it was, it didn't really work, to be honest, but we tried. And then when Alphago came along, I thought, oh, this has to be something that works. I didn't understand it at the time, but I thought this, there's got to be something there. And then, yeah, since 10 years, I think we've been working on, on basically finding these techniques to the reconciliations and food mechanics specifically. And I think in a way, it's a good time at the world. It's finally starting to work.

7:14 But right, it took a while. So you then joined TUM and you've been there ever since. That was your your period. And so where did you? What was it like? I'm trying to put it into the picture because it's very hard to the people who were working on these things before now, you know, before chat GB Teemo, what was the field like? What was the community like? What was the reception like when you were bringing up ideas of sort of data-driven and machine learning? Was there a lot of skepticism and push back? Yes, actually there was. So I mean, initially, especially in the early phases, all these AIO back then, it was called deep learning techniques were

8:01 focusing on, on learning descriptors. I remember our very first papers and graphics, they, they basically couldn't synthesize anything. You learned some reduced representation. They told you about what happens in high dimensional data, but right, it, it couldn't really generate anything. Also, if you, if you think about alpha gold, the early success stories, it was basically 8 by 8 with, with three states. That's a tiny domain and tiny state space. And back then that was challenging. So in a way, it really couldn't do much. And I, I remember quite well talking to colleagues and telling them, look, we're interested in this deep learning

8:36 stuff. We, we're doing simulations. And then some of my colleagues literally laugh me in the face like, you're doing learning for Pdes. What, what could be more ridiculous? If you have a PDE and you can solve it, why, why, why on earth would you do anything right learning based And I think so there was a lot of scepticism for years. I always had to defend in the very first slides of my talks that this makes sense at all, at least makes sense to consider this combination. And luckily I interestingly, I think for the for the white community, I think the picture really changed with chat GPTI think in computer science and

9:15 and computational fields. People have been aware earlier, but since chat ChatGPT it's really on everybody's plate and in every view but before. But you but you were interestingly was looking through some of the papers and you you had slightly earlier on probably some of the first ones are sort of like in the AI AA, the deep learning of like some random airfoils and so quite a few in the 20 twenties before ChatGPT. So what was it like sort of pre and post? Was it was the issue partly the compute side? Was it the the architectures? What? What was holding back? Exactly the the architectures were a fundamental issue.

9:59 I think the whole infrastructure right in the beginning, CNNS were were a big breakthrough, yes, for your MLPS. So CNNS definitely for feel like data worked quite nicely, but also right the, the, the shift from basically computer graphics to things like aerodynamics. The a paper you mentioned, we're a bit motivated by the observation that it's, it's really tough to synthesize stuff in 3D plus time. You essentially have 40 fields. Even if you simplify it and and treat regular geometry as regular grids, this is really difficult to do. It's still challenging these days actually with, with neural networks. So back then it was right bridge out of the question to do this.

10:41 And in graphics, you actually have extremely good approximations that aim for the visual aspect of, of things. So it, it was extremely tough to compete with that. And it was interestingly in the computer graphics field also, if you cannot directly apply it on a scale and, and with a fidelity that's fit for movies, then your papers typically get rejected. It's really, it's computer graphics makes sense, but it needs to look good in the end. So write this 2D proof of concept simulations. Even if they were fairly accurate, you you couldn't do much. It had had to be right movie quality basically. And and that just was extremely difficult.

11:19 Only also for niche applications like super resolution. This was was one that worked very nicely. But they can look at repeating structures and very local fields for synthesis on on larger scales, it was basically not possible. And that's why we also started actually doing these proof concept papers like the the Iran's predictions by AIA back then also these were tiny resolutions, right? 32128 square basically. So very small domains. And also I remember back then actually it's I'm, I'm glad it's got into a a, but there were also critical questions even back then. Why would you even do this at all? This is crude approximations. Does this make sense?

12:07 But yeah, basically we noticed in in aerodynamics and in engineering applications, there's much more demand for getting things right and really going forward converged and solutions that are accurate and can be evaluated. And there are many problems there that that are like really important basically for for a variety of industries. So we basically switched from from graphics trying to apply these for engineering applications. And where did the, you know, Phi flow when there's a differentiable physics, where did that start to come into the grid? Because that seems to be now quite a hot topic. But you were publishing in 2020

12:49 on these, so where did that first come from? I. Don't even remember, it seemed like a very obvious thing. We were working on computational and numerical methods, the the networks couldn't really generate a whole simulation. So why not start with the simulator, combining it with the network and then if you want to get a gradient through it actually need differentiability on on the solver side and right the the available ones couldn't do this. So we just started experimenting with on solvers since 5:00 flow became a good testing ground for very basic methods across different, different AP is back then, right there was 10s of flow and, and Pytorch and JAX

13:33 and the whole bandwidth of of different methods. So I think it was good for experimentation and flexibility. And ultimately we notice here it's a certain trade off. Flexibility and generality don't go too well together. And so now we're specialising a bit more. But five floor was one of these early testing grounds. But for us, it really seemed like in a way obvious, obvious way to go. And then later on, we notice, yeah, there's actually quite some work to do and also interesting questions. How do you get gradients through long chains of solver operations, which is still a topic? So maybe shifting a little bit towards the what's going on right now.

14:19 I mean, there's 22 fundamental questions that I see you've also been trying to tackle and be interested to get your thoughts on it. And that is around, I guess the foundation model topic. The idea of I guess it's logical, isn't it? You look at ChatGPT and you see it as essentially a data-driven problem that you know, applying up data, have a decent architecture and you solve it. But you well, two questions. One was on the idea of the ground truth and can you ever surpass the ground truth? And you know, one of your recent papers looked at can the model do better than essentially the quality of the training data? And the second one is more just a general question on where you

15:05 see the likelihood of foundation models going. How, how realistic is it? But on the first one, maybe you could explain that paper because I that really intrigued me actually, right? There was actually a great observation by Phoenix, one of one of the PhD students from my group, that he noticed in some situations. He also looked at basic combinations of solvers and networks and that's the networks really seem to give extremely good situations in certain situations extremely good results that seem to be even better than the the training data we essentially used for these cases. And looking into it a bit more, it's unfortunately not a topic

15:46 that I think can be generally applied and when analysing it in terms of numerical errors and, and how these pieces fit together. It's basically a combination of the priors imposed by neural networks in terms of smoothness, in terms of right, being able to over sharp more reproduce and be trained for producing certain parts of the solutions that benefits the the outcome. So in a way, if you can leverage or if you understand your solutions enough to leverage these particular behaviours of neural networks, then basically the outputs can be better than what you're trained with. But it's really this interplay of, of what neural networks do

16:29 as essentially also numerical methods to compute a solution and what, what's in your data and what you like to get out. So it also it raises some interesting questions on how to evaluate it because if especially in the numerical field, we're trusting the ground truth, there are also typically approximation errors in there. And it it can actually happen that solutions output by the network might be even better if you consider other solves or or other ways to approximate the same problems. But I think it's we also for this paper thought about it. It is very difficult to really do this in a target to way. So we had some new cases where

17:12 it worked for some, some basic infection fusion and burgers. Burgers equation was a good example actually because the the shock representation and smoothness bogus is non bogus equation. It's non trivial but seem to be a very good showcase for this. Directly applying it to other scenarios is is challenging so we haven't unfortunately we haven't gotten around to finding a good way to to leverage this for it. Reword never Stokes problems or so but but not to say. But what about foundation mods in general? Like, well, how was your group tackling this? What? What's your strategy in a sense, you know where, where, where do

17:53 you see this going? Recently we switched pretty much from these differential solvers to foundation models. I was quite skeptical for a TI for for quite a while. I think it's also there's also just quite some skepticism talking to other people in the field with foundation models. I think partly it's because of the name, right? Foundation model sounds super general and I'm sure you and and most people know these fields of Pdes are extremely diverse. The thought of having one model that just is everything out-of-the-box is yeah, but is is a bit unbelievable and I think also still out of quite a bit out of reach. Nonetheless, if you just right foundation was also by now I

18:39 think it's actually not so well defined. People use it in in all various forms and where I think it makes a lot of sense and in a way it's also the direction we are typically tech typically tackling is to see it as a model that's pre trained on on something and then applicable to other tasks, other problems. So there's just some disparity between training data and the actual application we saw. We want models to generalise. Also, generalisation is not a well defined topic, right? It's in distribution is clear, but how far out of distribution you need to go to have proper generalization. It's completely open. But in the past that's been really tricky.

19:19 Anyway, it's a classical transfer learning problem. Now, I think with these foundation model tools borrowing a lot from large vendors models, we can actually do this if we pre train on on one part and then apply it to something else. And how else how how different is is up for very specific to problems and and application domains. But that seems to work. And that's specifically what what I'm quite excited about is that we notice you can actually pre train on very cheaply generated data, offer us training foundation models in in 3D plus time. The amount of data needed seems very scary and also somewhat impractical given given the current cost of of doing these

20:03 training runs. But we noticed that in a way you don't really need these huge amounts of data. And I think that that really changes the picture, that changes how we can approach foundation model training. And I think in a way it makes it much more attractive as as a starting point. Could you maybe go into a little bit more detail when you when you talk about not needing as as much data or all the OR high quality data? Right. So you also thought about this quite a bit, right, as you recent paper discussing the the scaling of of of data and and computer requirements for fluids is is a good starting point to think about it.

20:43 What we noticed in this tadpole work, we we called it typically was basically that if you have a good solver. So it is actually also based in a way on on previous work on on this eight bench benchmark with very synthetic data. So if you and if you take the right solver, these spectral solvers, then you can extremely efficiently compute super good solutions for right, a class of basic PD, ES affection fusion, these ETDRK solvers in a way, they're really perfect. So also doing the learning on this on this level does not really make sense because the solvers are so good, the errors are so low, you can prove that they are basically as accurate

21:31 as it gets for for these basic solutions. So we can basically generate this data extremely quickly and very broadly if you bury the parameters and we notice that right, you can then pre train a model with this just generating a lot of different solutions for for varying parameter ranges for these different PD ES from affection diffusion to transport to some some higher order terms with chaos promoters, Yushinski and some some polynomial chemical reaction like constraints. And basically pre training this was on this very broad class of very basic synthetic solutions. Nonetheless, it's actually not not a very easy task and

22:19 benefits or seems to benefit in in quite a bit of downstream tasks. And the nice thing then is right, these, these solutions to these synthetic prones are really not interested in or interesting in a way you can compute them almost as quickly with the solver from from scratch basically. So you can train with it, you can run it through a model to get a gradient basically, but then you can throw it away. So it's basically just running the solver alongside you typically have written and all the four GPU. So one GPU does the data generation, the other 3 train and just generates data on the fly, throws it away right after it produces more data.

22:53 But the nice thing is it's produced on the fly and it's really just a continuous stream of data to train with. You don't, you don't have limitations on the bandwidth side on the hardware disk space to manage 10s of terabytes of or hundreds of terabytes to upload to HPC centres for for anyone who's tried this, it's, it's very painful actually to do this and it's not trivial. So right, having this online data generation with the training is is really quite neat and worked quite well in our experiments. But where? Do you, how do you see that progressing? I mean, I think this has often been my challenges and you've been on both sides of it.

23:39 You know, one side of it is, let's take realistic in well, in if we assume that the model should work for industrial scale problems, then you know, the the idea is you generate data like you did with the Super wing or, or now this highlight aero ML or any or company, you know, generating data. And then you you train on that. And you know that is one argument. And the other one is you're starting from the far left side of basic PD ES. But where do you see how can they go in the middle? The basic PDS are just can you go to Navier Stokes? How does that scale to more complexity? Yeah. So what we basically notice is that even if we pre trained also one of the one of the tricks

24:25 basically that I didn't mention just now was also that we don't train for a time evolution. It's really more of a of a latent space representation that is learned for for single spatial fields. So there's no time in the pre training and then the the basic architecture is basically so so general that you can just put time on top of that, right. So we learn in this latent space, one step to the next seems to work nicely. And also time in a way is also specific application already I would argue, right. There are steady state problems that that essentially go for an equilibrium already or an average solution where time does

25:05 not really the the evolution of time doesn't really play a role. It's more a mapping of some initial conditions in the geometry or so to the steady state. You might need time and other solutions, the occasions where you really want to resolve what happens over time. But in a way taking that out of pre training already makes sense in in retrospect I think. And then with the synthetic PD ES, probably you also might not get the out-of-the-box best accuracy for right high accuracy aerodynamic predictions. But I think it, it seems to be a very good starting point also for Navia Stokes. So what I see as a practical kind of approach in in the future would be to pre train

25:49 some very generic models as a starting point. And then step by step was first further pre training stages move towards more application specific models, aerodynamics, maybe other one, maybe aero acoustics, other one, maybe structural structural problems or branch off different directions. But pre training doesn't need to be restricted to these canonical PD ES, but could go in certain stages and then at a company you might for a certain use case then have 10 final models to fine tune the the last one for a certain application. I just think just this this outlook is basically not starting from scratch but pre training and then downstream needing less data.

26:33 I think it's potentially a good. So do you so because of that, do you feel that there can be, I mean how general can these models go do you think? I mean if you if you put a looking glass. Admittedly, that's an open question and in in a way also we were surprised that the bonus can learn something useful from these admittedly relatively useless synthetic solutions. But it's a really interesting question to analyse what they actually learned. So there are we had we had some hypothesis. Basically, do they really just learn or less random abstract fields or physical structures like the key modes? Or do they learn differential operators like Also, these PD ES

27:18 are built from certain derivatives and it's difficult to disentangle. But for example, the initial conditions are relatively easy to test. So the PD ES are initialized. If you only train with these random fields, the models actually also learn something, but it clearly does less good on the Stokes than if you pre train with the PD ES. So basically a transformation of initial conditions into actual states of a of APDE seems to be beneficial. I think this is just the first step. It's it's actually not right right now difficult to disentangle what they learned. And intuitively, I think the network network currently don't

27:57 has any reason to clearly disentangled modes or differential operators or so probably just mixes the most prominent structures that that come up. But in a way also most of the problems we're dealing with in in engineering and in real world applications are typically built from certain like building blocks to differential equations in a way have a have a very limited vocabulary. So my guess is that the structures that come from these actually in a way can be pre trended and reused. Yeah. And I guess the the other, I mean, so maybe to put it in a nutshell, in terms of your research and your thinking, do you think it is more better to

28:48 go down this sort of PDE route with an integrated solver in the loop rather than coming from the other direction, which is just generating lots and lots of, you know, if if we're trying to ultimately get to predict a wing, practically speaking, do you think it's better just to simulate a million wings in all different conditions or to sort of build up from the ground from a sort of PDE based that may be pre trained? And then you go to more complex, more complex, because they still do feel like different directions to go, right? The PD ES definitely aim for generality in a way I think also based on previous work, if your data set is large enough, if you

29:36 have enough data to really train the large scale model specifically for the task you're interested in, I don't think you actually need this this much more generic starting points. So if you can do this, I think probably it's not not necessary. It's really more motivated by the practical constraints. I think at the moment that's especially in in 3D for large scale Pdes, it's actually difficult to gather these really large collective, Yeah. I, I'm, I'm, I'm kind of intrigued on that point just because, you know, as that fluid intelligence paper tried to point out, there is a certain economic challenge, which is feels to be different than

30:27 ChatGPT where, you know, chat GBTS cost does not really include the, the data, the token cost per SE. And, and therefore it's all on the trading side. And I, I still, I still fundamentally wonder, is it massively inefficient? Because someone said to me, well, if you run cases, maybe I'll pose this question to you. So someone said, OK, so if you need to generate 2,000,000 cases in order to fully scope out what is a reasonable definition of problems within fluids, So all possible planes and cars and data centers and all the rest of it, and then you use all that to train a model. Haven't you already got the

31:18 solution from CFD for most of these problems anyway? Because you probably run it like, what? Where is it just a, you know, you're almost so massively in distribution that you almost could just map that solution to a new. Yeah. But you're right that that that might happen. So 2 million seems seems a little bit scarce for for actual geometry aside. But if if you could pretend enough of them, I guess you could hope that any new one you run would actually just hit close to one of the existing samples basically. Then I think a practical problem would still be that essentially you need to look up this data somehow, right? So these two all right, millions

32:02 of simulations would probably be I don't know how how many petabytes or gigantic lots of storage. So it could be beneficial to pre train just to have a reduced compressed representation to be able to efficiently look up these data points and maybe also give give some smoothness in between, right. That's even if your exact geometry is not in there, you get in in between solutions. So it, it could actually still be beneficial from a from a practical standpoint. But yeah, given how difficult it is and, and the amount of data involved with these online generated data sets, it seems that you can actually get away with much less data in a way. The the hope would be that you

32:49 do this as a first step and then right maybe you can get away with 1,000,000 off for CFD simulations or significantly less at least to to get to the same level of accuracy. Practically in your maybe one of the topics that also interests people and be interested to see how from a coding point of view you've seen this progress. So you raise the point of training the data, sorry, generating the data, throwing it away because you're ultimately using it. So maybe you could talk a little bit more what you saw at the challenges from an implementation from a coding point of view, because that that's I guess also part of the

33:30 problem if everybody's generating these hundreds of thousands of cases offline and then trying to bring into a model that also seems quite inefficient, so. The infrastructure work is quite substantial and I think it's getting better also largely thanks to all the methodology is converging on this LLM transformer style processing, which probably is is one of the reasons why now finally we're we're in a stage where it starts to work because also we have the the tools and and outlooks how to stay up things up to these larger resolutions. It's nonetheless still quite tricky in practice. So also in my group took quite a

34:19 while to to figure out how to put these pieces together. And there are numerous caveats and, and things that can go wrong. So actually, admittedly, one of the things we're still fighting with is the non linear scaling. So even with Transformers, which which are demonstrated to scale up to billions of parameters, if you take one model and you just try to increase the size of of layers and the overall capacity, it's highly non linear. So it's not guaranteed to to work if you rerun this, just longer with an increased size. Most likely at least need to adjust the learning rate, the additional hyper parameters like smoothing of of network states

34:59 over time. How to actually adjust how to scale up the size of the network in terms of this bedding space that the Transformers have or the the weights for the additional components that unfortunately or it seems it's necessary to revisit for, for every new case again and also for the for the PD case, unfortunately. So it's still not not right, not not really trivial to. Do this. And then right, also just ability, we fought quite a bit just with with the data management. So we rely on these official compute infrastructures here from the very end. And in Germany we now have supercomputers with a fair number of GPUs. But your someone need to get the

35:46 data over there and then you can just store it on on temporary drives. And if you don't pay attention then suddenly your trainer data has gone if you don't train often enough and issues issues like this basically. But how did you solve that with some of the just wanted to double click on the you know one GPU to generate the data, 3 to trade. How? How were you getting around the passing the data between? Yeah, I'm just trying to learn a little bit more how you approach that that topic of the online training. With the with the online training, so right, the classic approach would be right there. You, you have your huge data

36:24 sets on disk and eventually if you don't want to overfit to what fits into memory, you somehow need to get it from the disk, get it to the to the GPU's with this large data set that that becomes a bottleneck, at least for right, actually for even if you've hundreds of millions of parameters on HPC systems, I think we notice especially and the the hardware, the interconnects are good, but not fast enough to really get the data in quickly enough for training. So the standard set up would be right. You have your DP us they, they need to load the data from this, then shuffle it through the network to get a gradient and

37:00 they need to load the next sample basically. And for this online training, we are already basically forced to deal with a separate process that does the generation. So it's natural to put a buffer in between. So you basically have a, a buffer that is filled up by the simulator and then the training sets just pull the data from there or you take a random sample that's available, then train with it. And we try to replace it as quickly as possible. So in practice this this doesn't guarantee every sample is used only once, but you can measure right the the throughput of your simulator versus the training and it's typically at least below 2, so somewhere between

37:41 11.5. The South descendants get reused before they get replaced. So did you buffering? That's interesting. I mean, I need to look a little bit more at the the paper and the code, but that was, that was the bit that I guess I was assuming maybe for your PDE problem is easier. But if we're doing what is, let's say if we want to have something really accurate and we're doing an LES simulation, the actual LES simulation might need to run on 64 GPUs for like 8 hours, but the training is obviously way faster than that. So do you have any thoughts on that? How you would balance from a time from a loading point of view? So my guess is that that's why this online training so far

38:29 hasn't really taken off before because like you mentioned for all classic simulations, exactly you have the the warm up time until you get some equilibrium that's physically valid and that that could be used and it might take for for realistic reward case might take hours until you're in that regime and need fair number of of CPUs at least or DP US if you have a modern solver. So that's, that's really unattractive because the, the speed of generating the data is way below what you need for training, which is why I think it's actually so interesting to use these canonical PDS with these Spectra solvers because it, it's really, the solver is

39:10 basically as fast as, as a network or typically we run actually the, the training we run on little regions like 64 ^3 and the solver typically produce large, produces larger ones like 256 or so. And even those are pretty close to the, to the training speed. And then you can basically cut out different pieces for data augmentation. But it, it matches quite nicely. So I think without such a solver doing a large scale pre training is is infeasible. I think there are some approaches doing this, but you if you want to match the generation capacity with the training capacity you would need for a supercomputer to generate

39:52 data on the fly just to feed a decent sized model. Yeah, that though then my other thought was actually whilst it takes 8 hours on 64 GPUs, let's say, to do it, that is the entire simulation. But actually to generate one time step is probably only a second or two seconds. So part of my interest, and maybe some groups already doing this is training per time step. Now, the variation from one time step to another is pretty minimal. So you could argue that you're not really learning that much between, but you are. That's the only way I could see that the time scales being similar, You know, which I guess

40:42 ultimately addresses one of the challenges that I think many people have, which is almost all of the standard data sets and standard machine learning approaches do take some time averaged solution. You know, which if you're trying to get to real problems and real accuracy, turbulence is transient, you know, and giving just a steady state answer or time average answer is, you know, not, not really representing the ultimate goal of like weather forecasting. I guess you would have the temporal evolution of it. So I don't know how you how we can get over that. Issue. I think it's a great order to to really target time. I think there are some probably

41:32 what I see there are quite some hurdles for for practitioners because we have all these pipelines set up for for averaged quantities. All right, potentially I think this would be would be neat to have in place. Regarding your first point though, with the kind of exploiting the, the fast solves over time for data generation. Unfortunately he was tadpole with his online training. We have some, some, some data that is pointing to to problems there. So what we noticed even with this online generation if we don't replace the data fast enough. So if this reuse is too high of our our pre trained buffer basically the models do start to

42:16 overfit. So typically for foundation models you're dealing with fairly large models. So ours are not even extreme, but on the order of maybe 100 million parameters. And if the samples are reused too often, the bars already we saw some signs of of performance deteriorating if the generator didn't catch up. And basically this was just out of pure luck because of of scheduling on these high performance systems of right. If there's some one of the GPUs is is for some reason slower and suddenly on when training run you reuse the data more often. We saw a drop in performance and even there the the correlation between the samples was not overly strong.

42:57 So it's still generative fair amount of data. But I would be worried if you have these strongly correlated samples over time. Even if you're able to swap them out and reload reload different configurations for a solver, I think this would be would be difficult to ensure that the variability in the data is large enough to right not overfit a large model. Yeah, that's a very good point actually, that that's, that's the gang. Yeah. The drawback that we, the temporal evolution of the PDE needs to be small in for numerical reasons, you know, to, to avoid blowing up, you know, for because of the CFL constraints, etcetera.

43:41 So you you're kind of forced to slowly March in time, whereas I guess you're right, if you feed the machine learning model that it's just going to keep seeing essentially the same solution, very similar. Ones that. And and yeah, that's, but that's where I still see a bit of a fundamental. I just don't see. That's why I see almost as that's why I was interested in your differentiable physic, your your code writing essentially because I feel some of the breakthroughs in this probably will come through a clever, clever use of the training being linked to the data generation in a way that. I think right now our focus is on foundation was I'm also

44:28 confident that at some point it's going to make sense to bring the solvers back in the our our tests back back then a few years ago basically did did pretty short. If you have a solver, there's anything decent, the learning task is just simpler. So once once we have kind of reached the decent reasonable capacity for this pre training, I think it's might make a lot of sense to put a solver back into the loop to just make it more accurate in the end. Also, they are it's it is challenging at the moment at the at the scope given the current infrastructures and and large scale models. It's typically challenging to just train a single model and in a decent amount of time and then

45:17 trying to get a solver into the picture and maybe going over multiple time steps. Things like this are are tricky, but it's also going to change next year's. Yeah, yeah. No, I I would agree. And maybe the the other topic is around what were some of your lessons learned from doing, you know, weather bench and AP bench and, and some of these more benchmarking efforts because I guess that seems to have helped quite a bit on the weather and climate side. So what? What was some of the genesis behind those efforts? So the weather bench effort, yeah, in retrospect, it's, it's great that it's, it's doing so well.

46:06 I, I think it's especially, I guess I should think that so, right. What I'm trying to say is basically the, these benchmarks have a big impact in the field. I think by now that is widely accepted across all the fields. Back then it was probably more clear in the vision area where fair number of of benchmarks have been around for quite a while. But in other fields, like whether there was very little, basically having established benchmarks and libel evaluations is really important for for any discipline in the field. I think that's a nice pointer and right, it's civic. It's it's quite some work collecting the data, right and thinking about what should be in

46:48 there and how to evaluate it. So the way those efforts are really important for any subfield within any data-driven discipline. And I think by now it's great also in this scientific Yeah, I feel that this is noticed that by now more and more benchmarks are coming out. I think that's. Really important you you recently generated to your group the Super wing data set. I mean, where do you see this going? And it's a little bit of a philosophical debate around data because on one hand you could argue that we the progress in this field does seem limited by data to a certain extent. And, you know, does it mean that there should be some coordinated

47:38 effort to generate data? And if so, who should be doing that? And what should be the license attached to it, given that there is clearly also a commercial benefit to be had? That's a good question. I mean coming from from a university, I think it's great if it's all public and as open as possible. But given the commercial interest and also by now the the key outlook that this will be useful, that would of course be great to have support from from commercial partners in this. I think it's getting better. Also more, more companies see the need for AI and then also the need for or benchmarks and data sets on that front. So I think it's improving, but

48:23 definitely an issue where things could be done and could improve a lot. And unfortunately there's so much to quickly wrap this up. But I think there's so much proprietary data with IP rights and so on that are probably on some servers and companies that they that cannot be used. So it probably needs an effort to generate data that's free and and usable. Yeah, that I keep jumping between that because on one hand you could say that it is to the a bit like maybe open science experiments, you know, with, with astronomy or, or, or things where there's a sort of public good and a lot of it is funded through taxpayers. Ultimately, you know, through sort of science funding and the

49:06 data's made available. Part of it feels that that would help all companies, you know, to be doing it. But at the same time, is it really the responsibility of governments and science to do this? It it's it's kind of. Yeah, that's good to argue about the companies. So pay for it if they benefit from it, that's. Yeah, that's it. I just look at, I don't know about you, but I look at all the supercomputers in Europe, just as an example, you know, and you think how much data could be generated given that these systems now are quite large because they're being scaled up for the task of also, you know, large language model training,

49:56 etcetera. You know, if you've got a cluster with 10,000 GPUs, you think how much data could you generate, you know, from fluid simulations to a 10,000 GPUs, even just, you know, for a few weeks or, or you know, or a month, you know, that does feel like it could be a, a huge way of doing it. And if, if it's only a commercial company that does it, they probably have no incentive to release that data. And it then becomes hard to progress the field if only one company has that data where it feels like with large language models, the data has actually had quite an open movement, right? There's quite a lot of open data obviously around this.

50:47 And so, yes, you still could say you can only make models if you've got lots of compute, but there was already a movement now with open source LMS, you know, going on, whereas I feel now how can there be a movement of open source surrogate models if the data is not there in an open way, You know, it it it's. It's a good point. It would be a pity if that gets hidden away due to commercial interests. So it's almost the. Progression of science, I guess, and this is, I don't know if you've had this and certainly now my myself being a commercial company or the last two companies that you know that I'm at, there is always this debate

51:33 of open source versus commercial, and I still find that a tricky one. I wondered how you've seen it. You know what, what at what point does it help everybody to be open source? Yeah, OK, right. I have AI have a good. Argument for open source from from my career at least. We used to work in computer graphics basically with movie studios and I remember 1 of was my postdoc, one of the first papers at once at Arthur conference seeker there. We basically put up with an open source code and company DreamWorks at the time actually picked this up pretty quickly and directly used it for for one of the shots. And I said, oh, can't we get, or

52:16 maybe they can put us in the credit somewhere. They send us a poster in the end and also quite a few of my friends and kind of laugh or look all all you got for for this work. You put up your code, you got a poster for it. Why did you sell it or something? A couple of years later, these, all these movies where it was used were one of the key things to apply for one of these test tech Oscars. There's also application procedure. But in the end, we could show, look, it's been used in all these movies. And this this Oscar actually did help a lot also for applying for faculty positions and so on. So this is largely due to to open source availability.

52:56 It just might impact in the field. So I've had very good experience with this and I'm actually very happy that now that, yeah, I feel this is so obvious. 15 years ago it was very, it was actually the exception that papers came to code and it's great. It's changed so much. Yeah, it, it does seem an interesting 1 though, because it's, given the large amount of money that's required to develop models and develop training data, you know, there has to be some route to that money coming back. Which is why for a commercial company, if they spend $100 million on compute and they, you know, they have to think, well, how am I going to make a

53:40 business out of this? And so it's, it's kind of an interesting one that is the business that they, that the value to the company is not the model, but how you sort of tweak the model, so to speak. And therefore, just because there's a foundation model out there which is open source, the real value is that every single company needs to customize it, which I think is the value at the moment for open source LMS that ultimately companies still make money out of it because everybody realizes that it isn't fully ready in its base form. And that actually you take it and you tweak it and you optimize it and you know, you so that so there's money still to

54:25 be made. And that's what I kind of wonder on the fluids surrogate modelling side or even beyond that, it would it still be in the interest of some company to make it because ultimately the money will be made customizing it and therefore being open helps the whole community to develop it faster and make it more competitive against close models. Right, that's definitely. But I hope that, that we can go towards some foundation models or some generalizing models in the field that can then be fine-tuned and adaptive to adapted to different, different applications quite easily. I think that's, that's been super useful in the LLM field. And I, I do see potential there

55:15 on the, on the PDE front and through its front. So, and I think if, if at some point it's clear that you can get a really good result, right, if if you actually download this model and then right fine tune it on your couple of wing data set cases or so for certain regime you're interested in, then. But there could be a clear commercial also benefit for a company and the use case of supporting these for supporting these open models. Yeah. So that could imagine in an infrastructure environment working quite well also in this area, but that definitely needs a coordinated and a large scale effort to build it up right now. Yeah, that that's seems that

56:02 there's lots of separate efforts, I guess going on, but but not uncoordinated. I I mean, maybe moving to some of the final questions for you is where well, look, I, I guess a couple of things before we get to the future looking one. I'm just interested, where's your side on the like startups or academia industry? Because I've noticed there's a lot of, in recent years, there's even more interest in startups and, and, and the value of startups. But then that also in some ways can conflict with the open source academic, you know, mindset. Where, where have you seen that in the world? Have you been tempted with startups of, of industry?

56:48 Where do you see that role of academia, start-ups and and industry? Yeah, it's a good question. Exactly. Especially in last one or two years we've seen a very nice rise I think in terms of funding and and also just founding of all kinds of spin off companies that are pivots of quite existing companies towards this physically eye direction. So I think also in a way confirmation that now it's really starting to work, right, It's ready for practical applications. I've definitely toyed with the idea so far or also on my side, there are no immediate plans for this. In a way. I, I had my experience with industry I for, for visual attacks and before I'm quite

57:30 happy with the Open University research side. So I've, I've always been a big fan of open source and just being able to put out things and for, for research. I mean, it's also effectively a market with this openness and impact, right? You, if you can generate this later on, you can say, look, give me more money for research to do more of, of this. So it's not the commercial market, but also their own market in terms of research money. And but I like the openness of that. So for now, I think for me that's personally just because I like this openness the the better direction. But we also, by now we're working with all kinds of companies due to the commercial

58:14 interests. Yeah. But we were basically trying to do this on the more Open University side and then work with individual companies for maybe adopting or or adapting these these techniques. Yeah, I was going to say because the decking jury if all top academics create start-ups, then there will be no academics left to do that. The sort of so that that I'm sure that has been discussed a little bit in the broader LM space. If the only companies doing the latest stuff because of the scale are the big tech companies, then there is a risk to open science, I guess because everyone's tempted to go to a commercial company that maybe

58:56 aren't, as you said, incentivized to be open. I guess as academia, you are incentivized to be open because the reward structure is based around it publishing grant money. Like if everything's closed, I guess it may help the university a little bit because of grant capture or something, but it's not really the the core aim, is it? It's difficult actually for for universities. I think in the past quite a few labs had their in house solvers and then basically earned money and and funding. With direct we basically support or consulting contracts with companies. But right now I think especially in the fast moving AI field, it

59:39 was, it was closed solutions, you would see very little impact at the at the moment, Stephanie. Works nicely no I, I, I think that's why I'm always championing the academic side because I think people look at research coming out of tech companies or research coming out of the but forget that actually most of the things start in the university most most fundamental ideas and. They need to be kept being promoted and that's where I feel the data's the issue, because the they if you don't have open data, then you can't actually universities can't progress and can't publish and can't advance. The state-of-the-art. If everything's done closed

1:00:27 door, it actually hurts progress, doesn't it? Yeah. So LLMS will be interesting to some extent, right. That's there have been initiatives of training open LLMS, but some of them I guess did fairly well. But the top models are not open. And right now I think there are very few people who really can or try to train their own LLMS. That's really become a very tough field, at least to operate in. Yes, yes. So where do you see if you have your looking glass or your, you know, crystal ball? If we were to talk again in five years time, where where do you think the conversation will be going? What what do you think will be the breakthrough moments?

1:01:14 Do you feel like we're going to be just incrementally over the next five years or do you perceive some big breakthroughs or leap? Just hope that we're going to see this breakthrough in terms of adoption. So now also these all these initiatives on commercial side I think point towards the field really starting to work. So I hope that five years we we really see widespread adoption for for actual applications in a way sounds a bit boring, right, just being it used in in practice. But I think right also working on this for almost 10 years. I think it's about time that we that we do see that it's got it's really useful for for real world world things.

1:01:54 So I just hope that now we are at a stage where this really can be can be pulled off, but in the field is really progressing extremely quickly. And so one of the, one of the outlooks that I've been intrigued about are these world models, right? Like vision by now, they're already going beyond foundation models or specific language and, and just visual models towards ones that just capture the whole world. And right, also vision now notices actually physics is an important part. You only see so much. A lot of the complexity comes from things you don't directly see like air moving or so I think that's a really interesting outlook to to really

1:02:34 simulate basically the the whole world around us on a larger scale with the physics accurately. I think that could also be super useful for engineering and and real applications in a bunch of areas. But getting that right seems like a whole different level on top of what we currently dealing with just only doing the physics right with the foundation models. But I think it's a really interesting outlook. Yeah, it's actually a good point there. Yeah, I forgot to ask you about that. That I still don't fully get. I mean, I understand the world models are obviously one very big use case is robotics and driving drop, you know, autonomous vehicles and sort of

1:03:15 giving that synthetic data to help them to operate in the world, The bit that I guess I'm less sure on. And I wonder whether economically and scientifically is the physics side, you know, does it make a difference how good the physics is like, as long as it looks right? And this, This is why I'm interested, because your movie background, like does it, does it matter to the robots and to the autonomous vehicles? Actually, I do think that at least for robots, the physics play a big role because ultimately the, the action that you're generating is, is the control right of your, of the actuators of the actual things that the robot should do. And you need to estimate surface

1:04:07 properties, the weight of an object, weight distribution stability and all these things to at least in a, in a variable environment to interact with the, with the world. So think for robots by now rigid body type or in some somewhat deformable and physics are playing quite a role. So I would I would guess that it does make a difference for autonomous driving. We could argue at the end you only need to take right? Is it a dangerous situation or would something have gone wrong? You can probably do a lot without having to go through the whole crash or actually simulating how the car would tumble or crash into a building

1:04:50 or so. That might not be so important, right, As long as you can detect this would have gone wrong. But for robots actually having to really interact with objects, I can imagine that plays a larger role. And then again looking towards fluids, I, I think the challenge also in, in the current development pipelines is this interaction with the real world. I mean, I think in the end, the current validation and certification still goes through a lot of testing status and then you have all kinds of real world flights and and experience to make sure that the systems really do what they should in the real world. But if you could shift some of that into an earlier stage, I

1:05:32 think that that could also be definitely interesting. If right, you could have different environment conditions, different flow conditions, and you could have an have an object interact dynamically in in such an environment. Yeah, I guess maybe maybe you're right to maybe the examples I've looked at, you know, if you look at a robot walking around a factory floor or it has no the the drag, the lift, the thermal properties are very second or third order effects. They don't really influence how it's performing. But I guess you're right. If it was like a drone in the sky or a boat in the water, then the physics actually do make quite a bit of a difference in

1:06:18 and then I guess it really is depending what's the use of these world moles, if they really are like a true, true representation of the world for that drone or or helicopter or, you know, I guess the moment the autonomous vehicle, you're right, is it's more sensory. You know, like I see there's a person walking. It's not to design the car, right? It's not to say I see now that in this real world, by changing the shape, the drag is lower because it's actually I guess maybe that's where you're thinking, right, that the world model could be used as a true synthetic environment or cars, I guess if you could. Simulate a rainy and stormy situation and then really get

1:07:05 feedback on how the rain would splash around a car, or how gust of wind would influence driving stability in extreme conditions. Or right, all kinds of of landing and starting scenarios for planes. If you could really get feedback on how all the forming plane with straining conditions would would behave. I think that's that's still beyond even regular simulation capabilities that full structure interactions was changing conditions are are really challenging. You could get some feedback on these and then ideally put an optimization loop around them, right? You you would want to actually optimize your, you know, your wing profile or the shape of of

1:07:53 a vehicle tour behavior in the whole longer sequence. I feel like this is also the big debate, well not big debate, but a debate around the classic tool calling thing. You know, like isn't LLM good at adding numbers? No, so just go and call a calculator. That's like ultimately, I guess if you call Chachi BT to add 2 numbers together, it's just calling a tool of a calculator, adding the 2 numbers together. I kind of wonder it's sometimes in this world of, you know, world models or optimizing, you know, is it, is it really that it's truly integrated into a world model? Or is it more likely that there is a sort of agent workflow where it's calling a surrogate

1:08:37 model and maybe that's the where they're two separate models, if you know what I mean, rather than just a world model. It's it's more than a gentic way, if you know what I mean. Or not that that would already be a big step, right, If if you had any visual model that could on demand call some physics sub models surrogates to to get feedback on how these different pieces is recognised. Should should interact of original bodies. Actually these days you would just call it original body simulator. You would probably not need a surrogate. Yeah, that's. That's kind of where I always debate it and that and that's the traditional one that you

1:09:16 said if the solvers get so fast, like the spectral solver you mentioned or now with, you know, advancements in GPUs, some people are saying I can do a simulation in a minute. And then you think, well, do I need a surrogate? If if I can, if the solver can run so fast? Right, I think for World War, it's an interesting challenge will be to to quickly switch from from the proximate fidelity kind of for for rigid bodies or the former objects of winds or different feedbacks needed in the environment. Sure, if I know I need perfect rigid bodies for a certain case that might be ideal, but but right I need some feedback on on

1:09:59 wind or how right my plastic cup should deform or should it break predictions across the all these different physical phenomena. I could imagine that a flexible surrogate that at least gives a gives the first prediction of estimates of of what what might happen there could be beneficial. I think that would be tricky to do with classical simulators. Yes, yeah, yeah, yeah, that. I think this is where it's the classic what's the, what's the use case of real time? And sometimes if you go to an engineering team and I've had this where you tell them that a surrogate model prediction in one second, sometimes they'll say, well, we don't need it to

1:10:37 do in one second. You know, it's actually fine If it takes 5 minutes or even half an hour, it's OK. It's not a bottleneck. Whereas I guess if it's integrated in a world model or something, you need it to be essentially real time or else the whole value breaks of some simulation. You know, like testing a car moving. If you have to sort of pause the simulator for 30 minutes, then it's not that. Feel right? Yeah, I mean real time is is extremely challenging. Then you need I guess to to be a realistic virtual environment for a person. Yes, you need milliseconds. Yeah, very real. Yeah. So maybe a final question to you

1:11:26 looking more if, if there's a student listening to this or someone doing HD or early in their career, given, given all these changes, what would you recommend someone was would study as a PhD topic? You know what was going to future proof them in this world. Maybe I'm biased here, but I think these are really important techniques. I, I do see a lot of potential for AI based techniques. So I think it's important to have an understanding to know the classic, the classic physics numerics. But by now within these numerical tools, I think AI is, is a super important component. So I could, could highly recommend looking at at least combinations of classic and AI

1:12:09 based methods. And my, my guess is also that in the future or the next couple of of years, we were not going to be completely replaced by I don't think chap DPT is going to take over fluid simulation too soon. So having experts that we understand what's happening there, I think it's it's going to be needed for quite a while. So I think it's still a good field to to work on. Yeah, I guess that that's the the truth, I guess the, the short, medium, long term, I guess I see at least now the short term, there's even more need for specialists because frankly, it's almost become that simulation is more of a popular topic now. The fact that all these startups

1:12:53 are getting funded, the fact that, you know, there's more money in this space, there's more need to find out who's an expert in crash simulations, who's an expert in acoustics, who's an expert to help computer scientists, you know, to develop these models. I guess it's only in the very long term future that you could imagine. Maybe you don't need as many specialists because so much of that knowledge is now ingrained in these models. But there's a, in the short term, it's almost even more specialists to help. But deep, deep specialists, I guess you really understand it that, that at least that's the way I I'm seeing things at the moment.

1:13:36 And I think that's usually a great topic for a PhD, really getting to the bottom of things. So I just, it's a nice opportunity to work on one topic for a couple of years and yeah, get as much understanding as possible. So on. Right now I think PhD is in this area. Still a very good idea. Yeah, yeah. No, no, I agree. Well, thank you so much for taking the time to speak. I mean, I, I find all these topics fascinating and one of the things I'll do is put a list of the papers in the, in the show notes because I think actually there's a lot more to gather from looking in the details and, and looking at the codes that your, your, your group did.

1:14:12 So I know we didn't have time to cover all in, you know, full detail, but I'll, I'll put a list and hopefully people can find the time to read through the papers. That's a good idea. Oh, great discussion. Thanks for the invite. Yeah, interesting. We'll speak again in two years time and we'll see if the predictions are. That's interesting. Thank you. Thanks.