Hi, and welcome to the Neil Ashton Podcast.
In each episode, we explained some of the fascinating ways
that science and engineering are changing the world around us.
We talked to leading engineers from elite level sports like
cycling and Formula One to some of the world's top academics to
understand how fluid dynamics, machine learning, supercomputing
are bringing in a new era of discovery.
We also hear some of their life stories, their career advice,
the lessons they've learned on the way that I hope will be
helpful to you too. So sit back and enjoy this
episode. Hi and welcome back to the Neil
Ashton podcast. So today's guest is Professor
Neil Torre. He's an associate professor at
the Technical University of Munich TUM and has been a
pioneer in physics based deep learning.
We had a really interesting discussion today on first of
all, some of his background, Very interesting that he
actually won an, an Oscar working in the visual effects
industry for, for movies after his postdoc and PhD before he
then turned to academia full time, which is kind of
incredible actually, and is interesting because I've noticed
the lessons learned from the visual effects industry in terms
of photorealistic representations of, of, of the
world is actually something we, we then talked right the end of
the, the podcast. So make sure you listen to the
end about the link between foundation models and surrogates
to world models. These world models that are
being created essentially as a synthetic environment to train
autonomous vehicles and robots. And we actually talked a lot
about whether there needs to be more physics in these world
models, which I actually thought was a very interesting
discussion. And we only got to it at the
end. But you know, he he's been one
of these people who was doing deep learning in the mid twenty
10's and early twenty 20s before ChatGPT and the sort of big
rise. And it was interesting to him
talk about how his work was looked in those times compared
to now and some of the barriers, you know, and, and skepticism
maybe people had. He's also been someone who's
really pioneered open source and pushing the boundaries of
differentiable physics and some of the codes that him and his
team has developed and now have moved into both generating data
sets like the Super wing data set, but also addressing the
challenge of foundation models also from a sort of PDE point of
view in the latest Tadpole paper.
So we, we, we kind of go through some of those topics around
those different strategies of sort of PDE based methods, you
know, pre training on those versus doing more of the
traditional, I guess, building out large data sets.
We talk about the role of startups and industry and
academia. Interesting to get his
perspectives on that and how all three can work together to solve
some of these problems. And we also talk about the open
source, why he's so pro open source and how we think that can
help progress the, the, the field along.
He's, he's actually contributed together with his team, many
important papers covering not just on fluids, but also on the,
on the weather side, interesting like weather bench.
So I, I put a whole link list of papers in the chat notes that
you can in the show notes that you can have a look at.
But I, I really enjoyed this conversation.
He's someone who is very modest in, in what he's done.
But if you actually read his papers and look through, you can
see he's being quite influential in steering these topics and
always seems to be one step ahead of where most of the field
is AT. And so if you look at his 2025
and 2026 papers, you'll know what I mean by that.
So hope you enjoy this episode with Professor Neil Torry.
Well, yeah, thanks for thanks for joining today.
I, I've been very keen to speak to you because as I've been
getting up to speed myself over the past years on machine
learning and differentiable side.
Your name is a, a constant, you know, top of the list in terms
of influential papers, in terms of doing things what seemed to
be earlier than most others have done, you know, setting the
scene, which suggests to me that you have quite a good outlook
and, and, and thought on it. So, yeah, thanks very much for
for for joining. And yeah, maybe as a starting
question, you took a diversion. Well, not diversion, but you
spent some time doing visual effects after your PhD and
postdoc maybe. What did you do your PhD?
What was the postdoc? And then I'm I'm intrigued on on
the movie stuff. Thanks for the invite.
First of all, Neil. Yeah, I'd be happy to tell about
this. My background, right or over my
career, I have switched fields a couple of times.
My PhD was in a in a computational numerics
computational physics lab, basically targeting multi grid
methods and and fluids. And then I switched to computer
graphics for a while. I guess that that actually was
one of my driving my motivations from the start.
I guess I'm a visual person. I'd like to see how things
evolve. And I think that's that's still
actually one of the main things that's fascinate me about
fluids, these swirling motions and the things that actually do
come out once it's working. So we also worked on computer
graphics, BASIC for computer based animation for quite a
while. My post I get ETH and then
afterwards also did actually work in industry for a while
because I was curious how things would actually work out in
industry in a practical environment.
And partially also because I couldn't really get a good
faculty position at the time. So right, not not completely
voluntary in a while in a way, but right, it was good to see in
the end. I noticed for me, research is
the more I think the better field.
But yes, that's that's why we did my poster basically.
And then I think that if you 10 years ago, we started looking at
machine learning. So previously we did in this ETH
group what was called data-driven approaches in
computer graphics, essentially also trying to work with data
and right, somewhat aligned, but we didn't have the tools back
then. So it was, it didn't really
work, to be honest, but we tried.
And then when Alphago came along, I thought, oh, this has
to be something that works. I didn't understand it at the
time, but I thought this, there's got to be something
there. And then, yeah, since 10 years,
I think we've been working on, on basically finding these
techniques to the reconciliations and food
mechanics specifically. And I think in a way, it's a
good time at the world. It's finally starting to work.
But right, it took a while. So you then joined TUM and
you've been there ever since. That was your your period.
And so where did you? What was it like?
I'm trying to put it into the picture because it's very hard
to the people who were working on these things before now, you
know, before chat GB Teemo, what was the field like?
What was the community like? What was the reception like when
you were bringing up ideas of sort of data-driven and machine
learning? Was there a lot of skepticism
and push back? Yes, actually there was.
So I mean, initially, especially in the early phases, all these
AIO back then, it was called deep learning techniques were
focusing on, on learning descriptors.
I remember our very first papers and graphics, they, they
basically couldn't synthesize anything.
You learned some reduced representation.
They told you about what happens in high dimensional data, but
right, it, it couldn't really generate anything.
Also, if you, if you think about alpha gold, the early success
stories, it was basically 8 by 8 with, with three states.
That's a tiny domain and tiny state space.
And back then that was challenging.
So in a way, it really couldn't do much.
And I, I remember quite well talking to colleagues and
telling them, look, we're interested in this deep learning
stuff. We, we're doing simulations.
And then some of my colleagues literally laugh me in the face
like, you're doing learning for Pdes.
What, what could be more ridiculous?
If you have a PDE and you can solve it, why, why, why on earth
would you do anything right learning based And I think so
there was a lot of scepticism for years.
I always had to defend in the very first slides of my talks
that this makes sense at all, at least makes sense to consider
this combination. And luckily I interestingly, I
think for the for the white community, I think the picture
really changed with chat GPTI think in computer science and
and computational fields. People have been aware earlier,
but since chat ChatGPT it's really on everybody's plate and
in every view but before. But you but you were
interestingly was looking through some of the papers and
you you had slightly earlier on probably some of the first ones
are sort of like in the AI AA, the deep learning of like some
random airfoils and so quite a few in the 20 twenties before
ChatGPT. So what was it like sort of pre
and post? Was it was the issue partly the
compute side? Was it the the architectures?
What? What was holding back?
Exactly the the architectures were a fundamental issue.
I think the whole infrastructure right in the beginning, CNNS
were were a big breakthrough, yes, for your MLPS.
So CNNS definitely for feel like data worked quite nicely, but
also right the, the, the shift from basically computer graphics
to things like aerodynamics. The a paper you mentioned, we're
a bit motivated by the observation that it's, it's
really tough to synthesize stuff in 3D plus time.
You essentially have 40 fields. Even if you simplify it and and
treat regular geometry as regular grids, this is really
difficult to do. It's still challenging these
days actually with, with neural networks.
So back then it was right bridge out of the question to do this.
And in graphics, you actually have extremely good
approximations that aim for the visual aspect of, of things.
So it, it was extremely tough to compete with that.
And it was interestingly in the computer graphics field also, if
you cannot directly apply it on a scale and, and with a fidelity
that's fit for movies, then your papers typically get rejected.
It's really, it's computer graphics makes sense, but it
needs to look good in the end. So write this 2D proof of
concept simulations. Even if they were fairly
accurate, you you couldn't do much.
It had had to be right movie quality basically.
And and that just was extremely difficult.
Only also for niche applications like super resolution.
This was was one that worked very nicely.
But they can look at repeating structures and very local fields
for synthesis on on larger scales, it was basically not
possible. And that's why we also started
actually doing these proof concept papers like the the
Iran's predictions by AIA back then also these were tiny
resolutions, right? 32128 square basically.
So very small domains. And also I remember back then
actually it's I'm, I'm glad it's got into a a, but there were
also critical questions even back then.
Why would you even do this at all?
This is crude approximations. Does this make sense?
But yeah, basically we noticed in in aerodynamics and in
engineering applications, there's much more demand for
getting things right and really going forward converged and
solutions that are accurate and can be evaluated.
And there are many problems there that that are like really
important basically for for a variety of industries.
So we basically switched from from graphics trying to apply
these for engineering applications.
And where did the, you know, Phi flow when there's a
differentiable physics, where did that start to come into the
grid? Because that seems to be now
quite a hot topic. But you were publishing in 2020
on these, so where did that first come from?
I. Don't even remember, it seemed
like a very obvious thing. We were working on computational
and numerical methods, the the networks couldn't really
generate a whole simulation. So why not start with the
simulator, combining it with the network and then if you want to
get a gradient through it actually need differentiability
on on the solver side and right the the available ones couldn't
do this. So we just started experimenting
with on solvers since 5:00 flow became a good testing ground for
very basic methods across different, different AP is back
then, right there was 10s of flow and, and Pytorch and JAX
and the whole bandwidth of of different methods.
So I think it was good for experimentation and flexibility.
And ultimately we notice here it's a certain trade off.
Flexibility and generality don't go too well together.
And so now we're specialising a bit more.
But five floor was one of these early testing grounds.
But for us, it really seemed like in a way obvious, obvious
way to go. And then later on, we notice,
yeah, there's actually quite some work to do and also
interesting questions. How do you get gradients through
long chains of solver operations, which is still a
topic? So maybe shifting a little bit
towards the what's going on right now.
I mean, there's 22 fundamental questions that I see you've also
been trying to tackle and be interested to get your thoughts
on it. And that is around, I guess the
foundation model topic. The idea of I guess it's
logical, isn't it? You look at ChatGPT and you see
it as essentially a data-driven problem that you know, applying
up data, have a decent architecture and you solve it.
But you well, two questions. One was on the idea of the
ground truth and can you ever surpass the ground truth?
And you know, one of your recent papers looked at can the model
do better than essentially the quality of the training data?
And the second one is more just a general question on where you
see the likelihood of foundation models going.
How, how realistic is it? But on the first one, maybe you
could explain that paper because I that really intrigued me
actually, right? There was actually a great
observation by Phoenix, one of one of the PhD students from my
group, that he noticed in some situations.
He also looked at basic combinations of solvers and
networks and that's the networks really seem to give extremely
good situations in certain situations extremely good
results that seem to be even better than the the training
data we essentially used for these cases.
And looking into it a bit more, it's unfortunately not a topic
that I think can be generally applied and when analysing it in
terms of numerical errors and, and how these pieces fit
together. It's basically a combination of
the priors imposed by neural networks in terms of smoothness,
in terms of right, being able to over sharp more reproduce and be
trained for producing certain parts of the solutions that
benefits the the outcome. So in a way, if you can leverage
or if you understand your solutions enough to leverage
these particular behaviours of neural networks, then basically
the outputs can be better than what you're trained with.
But it's really this interplay of, of what neural networks do
as essentially also numerical methods to compute a solution
and what, what's in your data and what you like to get out.
So it also it raises some interesting questions on how to
evaluate it because if especially in the numerical
field, we're trusting the ground truth, there are also typically
approximation errors in there. And it it can actually happen
that solutions output by the network might be even better if
you consider other solves or or other ways to approximate the
same problems. But I think it's we also for
this paper thought about it. It is very difficult to really
do this in a target to way. So we had some new cases where
it worked for some, some basic infection fusion and burgers.
Burgers equation was a good example actually because the the
shock representation and smoothness bogus is non bogus
equation. It's non trivial but seem to be
a very good showcase for this. Directly applying it to other
scenarios is is challenging so we haven't unfortunately we
haven't gotten around to finding a good way to to leverage this
for it. Reword never Stokes problems or
so but but not to say. But what about foundation mods
in general? Like, well, how was your group
tackling this? What?
What's your strategy in a sense, you know where, where, where do
you see this going? Recently we switched pretty much
from these differential solvers to foundation models.
I was quite skeptical for a TI for for quite a while.
I think it's also there's also just quite some skepticism
talking to other people in the field with foundation models.
I think partly it's because of the name, right?
Foundation model sounds super general and I'm sure you and and
most people know these fields of Pdes are extremely diverse.
The thought of having one model that just is everything
out-of-the-box is yeah, but is is a bit unbelievable and I
think also still out of quite a bit out of reach.
Nonetheless, if you just right foundation was also by now I
think it's actually not so well defined.
People use it in in all various forms and where I think it makes
a lot of sense and in a way it's also the direction we are
typically tech typically tackling is to see it as a model
that's pre trained on on something and then applicable to
other tasks, other problems. So there's just some disparity
between training data and the actual application we saw.
We want models to generalise. Also, generalisation is not a
well defined topic, right? It's in distribution is clear,
but how far out of distribution you need to go to have proper
generalization. It's completely open.
But in the past that's been really tricky.
Anyway, it's a classical transfer learning problem.
Now, I think with these foundation model tools borrowing
a lot from large vendors models, we can actually do this if we
pre train on on one part and then apply it to something else.
And how else how how different is is up for very specific to
problems and and application domains.
But that seems to work. And that's specifically what
what I'm quite excited about is that we notice you can actually
pre train on very cheaply generated data, offer us
training foundation models in in 3D plus time.
The amount of data needed seems very scary and also somewhat
impractical given given the current cost of of doing these
training runs. But we noticed that in a way you
don't really need these huge amounts of data.
And I think that that really changes the picture, that
changes how we can approach foundation model training.
And I think in a way it makes it much more attractive as as a
starting point. Could you maybe go into a little
bit more detail when you when you talk about not needing as as
much data or all the OR high quality data?
Right. So you also thought about this
quite a bit, right, as you recent paper discussing the the
scaling of of of data and and computer requirements for fluids
is is a good starting point to think about it.
What we noticed in this tadpole work, we we called it typically
was basically that if you have a good solver.
So it is actually also based in a way on on previous work on on
this eight bench benchmark with very synthetic data.
So if you and if you take the right solver, these spectral
solvers, then you can extremely efficiently compute super good
solutions for right, a class of basic PD, ES affection fusion,
these ETDRK solvers in a way, they're really perfect.
So also doing the learning on this on this level does not
really make sense because the solvers are so good, the errors
are so low, you can prove that they are basically as accurate
as it gets for for these basic solutions.
So we can basically generate this data extremely quickly and
very broadly if you bury the parameters and we notice that
right, you can then pre train a model with this just generating
a lot of different solutions for for varying parameter ranges for
these different PD ES from affection diffusion to transport
to some some higher order terms with chaos promoters, Yushinski
and some some polynomial chemical reaction like
constraints. And basically pre training this
was on this very broad class of very basic synthetic solutions.
Nonetheless, it's actually not not a very easy task and
benefits or seems to benefit in in quite a bit of downstream
tasks. And the nice thing then is
right, these, these solutions to these synthetic prones are
really not interested in or interesting in a way you can
compute them almost as quickly with the solver from from
scratch basically. So you can train with it, you
can run it through a model to get a gradient basically, but
then you can throw it away. So it's basically just running
the solver alongside you typically have written and all
the four GPU. So one GPU does the data
generation, the other 3 train and just generates data on the
fly, throws it away right after it produces more data.
But the nice thing is it's produced on the fly and it's
really just a continuous stream of data to train with.
You don't, you don't have limitations on the bandwidth
side on the hardware disk space to manage 10s of terabytes of or
hundreds of terabytes to upload to HPC centres for for anyone
who's tried this, it's, it's very painful actually to do this
and it's not trivial. So right, having this online
data generation with the training is is really quite neat
and worked quite well in our experiments.
But where? Do you, how do you see that
progressing? I mean, I think this has often
been my challenges and you've been on both sides of it.
You know, one side of it is, let's take realistic in well, in
if we assume that the model should work for industrial scale
problems, then you know, the the idea is you generate data like
you did with the Super wing or, or now this highlight aero ML or
any or company, you know, generating data.
And then you you train on that. And you know that is one
argument. And the other one is you're
starting from the far left side of basic PD ES.
But where do you see how can they go in the middle?
The basic PDS are just can you go to Navier Stokes?
How does that scale to more complexity?
Yeah. So what we basically notice is
that even if we pre trained also one of the one of the tricks
basically that I didn't mention just now was also that we don't
train for a time evolution. It's really more of a of a
latent space representation that is learned for for single
spatial fields. So there's no time in the pre
training and then the the basic architecture is basically so so
general that you can just put time on top of that, right.
So we learn in this latent space, one step to the next
seems to work nicely. And also time in a way is also
specific application already I would argue, right.
There are steady state problems that that essentially go for an
equilibrium already or an average solution where time does
not really the the evolution of time doesn't really play a role.
It's more a mapping of some initial conditions in the
geometry or so to the steady state.
You might need time and other solutions, the occasions where
you really want to resolve what happens over time.
But in a way taking that out of pre training already makes sense
in in retrospect I think. And then with the synthetic PD
ES, probably you also might not get the out-of-the-box best
accuracy for right high accuracy aerodynamic predictions.
But I think it, it seems to be a very good starting point also
for Navia Stokes. So what I see as a practical
kind of approach in in the future would be to pre train
some very generic models as a starting point.
And then step by step was first further pre training stages move
towards more application specific models, aerodynamics,
maybe other one, maybe aero acoustics, other one, maybe
structural structural problems or branch off different
directions. But pre training doesn't need to
be restricted to these canonical PD ES, but could go in certain
stages and then at a company you might for a certain use case
then have 10 final models to fine tune the the last one for a
certain application. I just think just this this
outlook is basically not starting from scratch but pre
training and then downstream needing less data.
I think it's potentially a good. So do you so because of that, do
you feel that there can be, I mean how general can these
models go do you think? I mean if you if you put a
looking glass. Admittedly, that's an open
question and in in a way also we were surprised that the bonus
can learn something useful from these admittedly relatively
useless synthetic solutions. But it's a really interesting
question to analyse what they actually learned.
So there are we had we had some hypothesis.
Basically, do they really just learn or less random abstract
fields or physical structures like the key modes?
Or do they learn differential operators like Also, these PD ES
are built from certain derivatives and it's difficult
to disentangle. But for example, the initial
conditions are relatively easy to test.
So the PD ES are initialized. If you only train with these
random fields, the models actually also learn something,
but it clearly does less good on the Stokes than if you pre train
with the PD ES. So basically a transformation of
initial conditions into actual states of a of APDE seems to be
beneficial. I think this is just the first
step. It's it's actually not right
right now difficult to disentangle what they learned.
And intuitively, I think the network network currently don't
has any reason to clearly disentangled modes or
differential operators or so probably just mixes the most
prominent structures that that come up.
But in a way also most of the problems we're dealing with in
in engineering and in real world applications are typically built
from certain like building blocks to differential equations
in a way have a have a very limited vocabulary.
So my guess is that the structures that come from these
actually in a way can be pre trended and reused.
Yeah. And I guess the the other, I
mean, so maybe to put it in a nutshell, in terms of your
research and your thinking, do you think it is more better to
go down this sort of PDE route with an integrated solver in the
loop rather than coming from the other direction, which is just
generating lots and lots of, you know, if if we're trying to
ultimately get to predict a wing, practically speaking, do
you think it's better just to simulate a million wings in all
different conditions or to sort of build up from the ground from
a sort of PDE based that may be pre trained?
And then you go to more complex, more complex, because they still
do feel like different directions to go, right?
The PD ES definitely aim for generality in a way I think also
based on previous work, if your data set is large enough, if you
have enough data to really train the large scale model
specifically for the task you're interested in, I don't think you
actually need this this much more generic starting points.
So if you can do this, I think probably it's not not necessary.
It's really more motivated by the practical constraints.
I think at the moment that's especially in in 3D for large
scale Pdes, it's actually difficult to gather these really
large collective, Yeah. I, I'm, I'm, I'm kind of
intrigued on that point just because, you know, as that fluid
intelligence paper tried to point out, there is a certain
economic challenge, which is feels to be different than
ChatGPT where, you know, chat GBTS cost does not really
include the, the data, the token cost per SE.
And, and therefore it's all on the trading side.
And I, I still, I still fundamentally wonder, is it
massively inefficient? Because someone said to me,
well, if you run cases, maybe I'll pose this question to you.
So someone said, OK, so if you need to generate 2,000,000 cases
in order to fully scope out what is a reasonable definition of
problems within fluids, So all possible planes and cars and
data centers and all the rest of it, and then you use all that to
train a model. Haven't you already got the
solution from CFD for most of these problems anyway?
Because you probably run it like, what?
Where is it just a, you know, you're almost so massively in
distribution that you almost could just map that solution to
a new. Yeah.
But you're right that that that might happen.
So 2 million seems seems a little bit scarce for for actual
geometry aside. But if if you could pretend
enough of them, I guess you could hope that any new one you
run would actually just hit close to one of the existing
samples basically. Then I think a practical problem
would still be that essentially you need to look up this data
somehow, right? So these two all right, millions
of simulations would probably be I don't know how how many
petabytes or gigantic lots of storage.
So it could be beneficial to pre train just to have a reduced
compressed representation to be able to efficiently look up
these data points and maybe also give give some smoothness in
between, right. That's even if your exact
geometry is not in there, you get in in between solutions.
So it, it could actually still be beneficial from a from a
practical standpoint. But yeah, given how difficult it
is and, and the amount of data involved with these online
generated data sets, it seems that you can actually get away
with much less data in a way. The the hope would be that you
do this as a first step and then right maybe you can get away
with 1,000,000 off for CFD simulations or significantly
less at least to to get to the same level of accuracy.
Practically in your maybe one of the topics that also interests
people and be interested to see how from a coding point of view
you've seen this progress. So you raise the point of
training the data, sorry, generating the data, throwing it
away because you're ultimately using it.
So maybe you could talk a little bit more what you saw at the
challenges from an implementation from a coding
point of view, because that that's I guess also part of the
problem if everybody's generating these hundreds of
thousands of cases offline and then trying to bring into a
model that also seems quite inefficient, so.
The infrastructure work is quite substantial and I think it's
getting better also largely thanks to all the methodology is
converging on this LLM transformer style processing,
which probably is is one of the reasons why now finally we're
we're in a stage where it starts to work because also we have the
the tools and and outlooks how to stay up things up to these
larger resolutions. It's nonetheless still quite
tricky in practice. So also in my group took quite a
while to to figure out how to put these pieces together.
And there are numerous caveats and, and things that can go
wrong. So actually, admittedly, one of
the things we're still fighting with is the non linear scaling.
So even with Transformers, which which are demonstrated to scale
up to billions of parameters, if you take one model and you just
try to increase the size of of layers and the overall capacity,
it's highly non linear. So it's not guaranteed to to
work if you rerun this, just longer with an increased size.
Most likely at least need to adjust the learning rate, the
additional hyper parameters like smoothing of of network states
over time. How to actually adjust how to
scale up the size of the network in terms of this bedding space
that the Transformers have or the the weights for the
additional components that unfortunately or it seems it's
necessary to revisit for, for every new case again and also
for the for the PD case, unfortunately.
So it's still not not right, not not really trivial to.
Do this. And then right, also just
ability, we fought quite a bit just with with the data
management. So we rely on these official
compute infrastructures here from the very end.
And in Germany we now have supercomputers with a fair
number of GPUs. But your someone need to get the
data over there and then you can just store it on on temporary
drives. And if you don't pay attention
then suddenly your trainer data has gone if you don't train
often enough and issues issues like this basically.
But how did you solve that with some of the just wanted to
double click on the you know one GPU to generate the data, 3 to
trade. How?
How were you getting around the passing the data between?
Yeah, I'm just trying to learn a little bit more how you approach
that that topic of the online training.
With the with the online training, so right, the classic
approach would be right there. You, you have your huge data
sets on disk and eventually if you don't want to overfit to
what fits into memory, you somehow need to get it from the
disk, get it to the to the GPU's with this large data set that
that becomes a bottleneck, at least for right, actually for
even if you've hundreds of millions of parameters on HPC
systems, I think we notice especially and the the hardware,
the interconnects are good, but not fast enough to really get
the data in quickly enough for training.
So the standard set up would be right.
You have your DP us they, they need to load the data from this,
then shuffle it through the network to get a gradient and
they need to load the next sample basically.
And for this online training, we are already basically forced to
deal with a separate process that does the generation.
So it's natural to put a buffer in between.
So you basically have a, a buffer that is filled up by the
simulator and then the training sets just pull the data from
there or you take a random sample that's available, then
train with it. And we try to replace it as
quickly as possible. So in practice this this doesn't
guarantee every sample is used only once, but you can measure
right the the throughput of your simulator versus the training
and it's typically at least below 2, so somewhere between
11.5. The South descendants get reused
before they get replaced. So did you buffering?
That's interesting. I mean, I need to look a little
bit more at the the paper and the code, but that was, that was
the bit that I guess I was assuming maybe for your PDE
problem is easier. But if we're doing what is,
let's say if we want to have something really accurate and
we're doing an LES simulation, the actual LES simulation might
need to run on 64 GPUs for like 8 hours, but the training is
obviously way faster than that. So do you have any thoughts on
that? How you would balance from a
time from a loading point of view?
So my guess is that that's why this online training so far
hasn't really taken off before because like you mentioned for
all classic simulations, exactly you have the the warm up time
until you get some equilibrium that's physically valid and that
that could be used and it might take for for realistic reward
case might take hours until you're in that regime and need
fair number of of CPUs at least or DP US if you have a modern
solver. So that's, that's really
unattractive because the, the speed of generating the data is
way below what you need for training, which is why I think
it's actually so interesting to use these canonical PDS with
these Spectra solvers because it, it's really, the solver is
basically as fast as, as a network or typically we run
actually the, the training we run on little regions like 64 ^3
and the solver typically produce large, produces larger ones like
256 or so. And even those are pretty close
to the, to the training speed. And then you can basically cut
out different pieces for data augmentation.
But it, it matches quite nicely. So I think without such a solver
doing a large scale pre training is is infeasible.
I think there are some approaches doing this, but you
if you want to match the generation capacity with the
training capacity you would need for a supercomputer to generate
data on the fly just to feed a decent sized model.
Yeah, that though then my other thought was actually whilst it
takes 8 hours on 64 GPUs, let's say, to do it, that is the
entire simulation. But actually to generate one
time step is probably only a second or two seconds.
So part of my interest, and maybe some groups already doing
this is training per time step. Now, the variation from one time
step to another is pretty minimal.
So you could argue that you're not really learning that much
between, but you are. That's the only way I could see
that the time scales being similar, You know, which I guess
ultimately addresses one of the challenges that I think many
people have, which is almost all of the standard data sets and
standard machine learning approaches do take some time
averaged solution. You know, which if you're trying
to get to real problems and real accuracy, turbulence is
transient, you know, and giving just a steady state answer or
time average answer is, you know, not, not really
representing the ultimate goal of like weather forecasting.
I guess you would have the temporal evolution of it.
So I don't know how you how we can get over that.
Issue. I think it's a great order to to
really target time. I think there are some probably
what I see there are quite some hurdles for for practitioners
because we have all these pipelines set up for for
averaged quantities. All right, potentially I think
this would be would be neat to have in place.
Regarding your first point though, with the kind of
exploiting the, the fast solves over time for data generation.
Unfortunately he was tadpole with his online training.
We have some, some, some data that is pointing to to problems
there. So what we noticed even with
this online generation if we don't replace the data fast
enough. So if this reuse is too high of
our our pre trained buffer basically the models do start to
overfit. So typically for foundation
models you're dealing with fairly large models.
So ours are not even extreme, but on the order of maybe 100
million parameters. And if the samples are reused
too often, the bars already we saw some signs of of performance
deteriorating if the generator didn't catch up.
And basically this was just out of pure luck because of of
scheduling on these high performance systems of right.
If there's some one of the GPUs is is for some reason slower and
suddenly on when training run you reuse the data more often.
We saw a drop in performance and even there the the correlation
between the samples was not overly strong.
So it's still generative fair amount of data.
But I would be worried if you have these strongly correlated
samples over time. Even if you're able to swap them
out and reload reload different configurations for a solver, I
think this would be would be difficult to ensure that the
variability in the data is large enough to right not overfit a
large model. Yeah, that's a very good point
actually, that that's, that's the gang.
Yeah. The drawback that we, the
temporal evolution of the PDE needs to be small in for
numerical reasons, you know, to, to avoid blowing up, you know,
for because of the CFL constraints, etcetera.
So you you're kind of forced to slowly March in time, whereas I
guess you're right, if you feed the machine learning model that
it's just going to keep seeing essentially the same solution,
very similar. Ones that.
And and yeah, that's, but that's where I still see a bit of a
fundamental. I just don't see.
That's why I see almost as that's why I was interested in
your differentiable physic, your your code writing essentially
because I feel some of the breakthroughs in this probably
will come through a clever, clever use of the training being
linked to the data generation in a way that.
I think right now our focus is on foundation was I'm also
confident that at some point it's going to make sense to
bring the solvers back in the our our tests back back then a
few years ago basically did did pretty short.
If you have a solver, there's anything decent, the learning
task is just simpler. So once once we have kind of
reached the decent reasonable capacity for this pre training,
I think it's might make a lot of sense to put a solver back into
the loop to just make it more accurate in the end.
Also, they are it's it is challenging at the moment at the
at the scope given the current infrastructures and and large
scale models. It's typically challenging to
just train a single model and in a decent amount of time and then
trying to get a solver into the picture and maybe going over
multiple time steps. Things like this are are tricky,
but it's also going to change next year's.
Yeah, yeah. No, I I would agree.
And maybe the the other topic is around what were some of your
lessons learned from doing, you know, weather bench and AP bench
and, and some of these more benchmarking efforts because I
guess that seems to have helped quite a bit on the weather and
climate side. So what?
What was some of the genesis behind those efforts?
So the weather bench effort, yeah, in retrospect, it's, it's
great that it's, it's doing so well.
I, I think it's especially, I guess I should think that so,
right. What I'm trying to say is
basically the, these benchmarks have a big impact in the field.
I think by now that is widely accepted across all the fields.
Back then it was probably more clear in the vision area where
fair number of of benchmarks have been around for quite a
while. But in other fields, like
whether there was very little, basically having established
benchmarks and libel evaluations is really important for for any
discipline in the field. I think that's a nice pointer
and right, it's civic. It's it's quite some work
collecting the data, right and thinking about what should be in
there and how to evaluate it. So the way those efforts are
really important for any subfield within any data-driven
discipline. And I think by now it's great
also in this scientific Yeah, I feel that this is noticed that
by now more and more benchmarks are coming out.
I think that's. Really important you you
recently generated to your group the Super wing data set.
I mean, where do you see this going?
And it's a little bit of a philosophical debate around data
because on one hand you could argue that we the progress in
this field does seem limited by data to a certain extent.
And, you know, does it mean that there should be some coordinated
effort to generate data? And if so, who should be doing
that? And what should be the license
attached to it, given that there is clearly also a commercial
benefit to be had? That's a good question.
I mean coming from from a university, I think it's great
if it's all public and as open as possible.
But given the commercial interest and also by now the the
key outlook that this will be useful, that would of course be
great to have support from from commercial partners in this.
I think it's getting better. Also more, more companies see
the need for AI and then also the need for or benchmarks and
data sets on that front. So I think it's improving, but
definitely an issue where things could be done and could improve
a lot. And unfortunately there's so
much to quickly wrap this up. But I think there's so much
proprietary data with IP rights and so on that are probably on
some servers and companies that they that cannot be used.
So it probably needs an effort to generate data that's free and
and usable. Yeah, that I keep jumping
between that because on one hand you could say that it is to the
a bit like maybe open science experiments, you know, with,
with astronomy or, or, or things where there's a sort of public
good and a lot of it is funded through taxpayers.
Ultimately, you know, through sort of science funding and the
data's made available. Part of it feels that that would
help all companies, you know, to be doing it.
But at the same time, is it really the responsibility of
governments and science to do this?
It it's it's kind of. Yeah, that's good to argue about
the companies. So pay for it if they benefit
from it, that's. Yeah, that's it.
I just look at, I don't know about you, but I look at all the
supercomputers in Europe, just as an example, you know, and you
think how much data could be generated given that these
systems now are quite large because they're being scaled up
for the task of also, you know, large language model training,
etcetera. You know, if you've got a
cluster with 10,000 GPUs, you think how much data could you
generate, you know, from fluid simulations to a 10,000 GPUs,
even just, you know, for a few weeks or, or you know, or a
month, you know, that does feel like it could be a, a huge way
of doing it. And if, if it's only a
commercial company that does it, they probably have no incentive
to release that data. And it then becomes hard to
progress the field if only one company has that data where it
feels like with large language models, the data has actually
had quite an open movement, right?
There's quite a lot of open data obviously around this.
And so, yes, you still could say you can only make models if
you've got lots of compute, but there was already a movement now
with open source LMS, you know, going on, whereas I feel now how
can there be a movement of open source surrogate models if the
data is not there in an open way, You know, it it it's.
It's a good point. It would be a pity if that gets
hidden away due to commercial interests.
So it's almost the. Progression of science, I guess,
and this is, I don't know if you've had this and certainly
now my myself being a commercial company or the last two
companies that you know that I'm at, there is always this debate
of open source versus commercial, and I still find
that a tricky one. I wondered how you've seen it.
You know what, what at what point does it help everybody to
be open source? Yeah, OK, right.
I have AI have a good. Argument for open source from
from my career at least. We used to work in computer
graphics basically with movie studios and I remember 1 of was
my postdoc, one of the first papers at once at Arthur
conference seeker there. We basically put up with an open
source code and company DreamWorks at the time actually
picked this up pretty quickly and directly used it for for one
of the shots. And I said, oh, can't we get, or
maybe they can put us in the credit somewhere.
They send us a poster in the end and also quite a few of my
friends and kind of laugh or look all all you got for for
this work. You put up your code, you got a
poster for it. Why did you sell it or
something? A couple of years later, these,
all these movies where it was used were one of the key things
to apply for one of these test tech Oscars.
There's also application procedure.
But in the end, we could show, look, it's been used in all
these movies. And this this Oscar actually did
help a lot also for applying for faculty positions and so on.
So this is largely due to to open source availability.
It just might impact in the field.
So I've had very good experience with this and I'm actually very
happy that now that, yeah, I feel this is so obvious. 15
years ago it was very, it was actually the exception that
papers came to code and it's great.
It's changed so much. Yeah, it, it does seem an
interesting 1 though, because it's, given the large amount of
money that's required to develop models and develop training
data, you know, there has to be some route to that money coming
back. Which is why for a commercial
company, if they spend $100 million on compute and they, you
know, they have to think, well, how am I going to make a
business out of this? And so it's, it's kind of an
interesting one that is the business that they, that the
value to the company is not the model, but how you sort of tweak
the model, so to speak. And therefore, just because
there's a foundation model out there which is open source, the
real value is that every single company needs to customize it,
which I think is the value at the moment for open source LMS
that ultimately companies still make money out of it because
everybody realizes that it isn't fully ready in its base form.
And that actually you take it and you tweak it and you
optimize it and you know, you so that so there's money still to
be made. And that's what I kind of wonder
on the fluids surrogate modelling side or even beyond
that, it would it still be in the interest of some company to
make it because ultimately the money will be made customizing
it and therefore being open helps the whole community to
develop it faster and make it more competitive against close
models. Right, that's definitely.
But I hope that, that we can go towards some foundation models
or some generalizing models in the field that can then be
fine-tuned and adaptive to adapted to different, different
applications quite easily. I think that's, that's been
super useful in the LLM field. And I, I do see potential there
on the, on the PDE front and through its front.
So, and I think if, if at some point it's clear that you can
get a really good result, right, if if you actually download this
model and then right fine tune it on your couple of wing data
set cases or so for certain regime you're interested in,
then. But there could be a clear
commercial also benefit for a company and the use case of
supporting these for supporting these open models.
Yeah. So that could imagine in an
infrastructure environment working quite well also in this
area, but that definitely needs a coordinated and a large scale
effort to build it up right now. Yeah, that that's seems that
there's lots of separate efforts, I guess going on, but
but not uncoordinated. I I mean, maybe moving to some
of the final questions for you is where well, look, I, I guess
a couple of things before we get to the future looking one.
I'm just interested, where's your side on the like startups
or academia industry? Because I've noticed there's a
lot of, in recent years, there's even more interest in startups
and, and, and the value of startups.
But then that also in some ways can conflict with the open
source academic, you know, mindset.
Where, where have you seen that in the world?
Have you been tempted with startups of, of industry?
Where do you see that role of academia, start-ups and and
industry? Yeah, it's a good question.
Exactly. Especially in last one or two
years we've seen a very nice rise I think in terms of funding
and and also just founding of all kinds of spin off companies
that are pivots of quite existing companies towards this
physically eye direction. So I think also in a way
confirmation that now it's really starting to work, right,
It's ready for practical applications.
I've definitely toyed with the idea so far or also on my side,
there are no immediate plans for this.
In a way. I, I had my experience with
industry I for, for visual attacks and before I'm quite
happy with the Open University research side.
So I've, I've always been a big fan of open source and just
being able to put out things and for, for research.
I mean, it's also effectively a market with this openness and
impact, right? You, if you can generate this
later on, you can say, look, give me more money for research
to do more of, of this. So it's not the commercial
market, but also their own market in terms of research
money. And but I like the openness of
that. So for now, I think for me
that's personally just because I like this openness the the
better direction. But we also, by now we're
working with all kinds of companies due to the commercial
interests. Yeah.
But we were basically trying to do this on the more Open
University side and then work with individual companies for
maybe adopting or or adapting these these techniques.
Yeah, I was going to say because the decking jury if all top
academics create start-ups, then there will be no academics left
to do that. The sort of so that that I'm
sure that has been discussed a little bit in the broader LM
space. If the only companies doing the
latest stuff because of the scale are the big tech
companies, then there is a risk to open science, I guess because
everyone's tempted to go to a commercial company that maybe
aren't, as you said, incentivized to be open.
I guess as academia, you are incentivized to be open because
the reward structure is based around it publishing grant
money. Like if everything's closed, I
guess it may help the university a little bit because of grant
capture or something, but it's not really the the core aim, is
it? It's difficult actually for for
universities. I think in the past quite a few
labs had their in house solvers and then basically earned money
and and funding. With direct we basically support
or consulting contracts with companies.
But right now I think especially in the fast moving AI field, it
was, it was closed solutions, you would see very little impact
at the at the moment, Stephanie. Works nicely no I, I, I think
that's why I'm always championing the academic side
because I think people look at research coming out of tech
companies or research coming out of the but forget that actually
most of the things start in the university most most fundamental
ideas and. They need to be kept being
promoted and that's where I feel the data's the issue, because
the they if you don't have open data, then you can't actually
universities can't progress and can't publish and can't advance.
The state-of-the-art. If everything's done closed
door, it actually hurts progress, doesn't it?
Yeah. So LLMS will be interesting to
some extent, right. That's there have been
initiatives of training open LLMS, but some of them I guess
did fairly well. But the top models are not open.
And right now I think there are very few people who really can
or try to train their own LLMS. That's really become a very
tough field, at least to operate in.
Yes, yes. So where do you see if you have
your looking glass or your, you know, crystal ball?
If we were to talk again in five years time, where where do you
think the conversation will be going?
What what do you think will be the breakthrough moments?
Do you feel like we're going to be just incrementally over the
next five years or do you perceive some big breakthroughs
or leap? Just hope that we're going to
see this breakthrough in terms of adoption.
So now also these all these initiatives on commercial side I
think point towards the field really starting to work.
So I hope that five years we we really see widespread adoption
for for actual applications in a way sounds a bit boring, right,
just being it used in in practice.
But I think right also working on this for almost 10 years.
I think it's about time that we that we do see that it's got
it's really useful for for real world world things.
So I just hope that now we are at a stage where this really can
be can be pulled off, but in the field is really progressing
extremely quickly. And so one of the, one of the
outlooks that I've been intrigued about are these world
models, right? Like vision by now, they're
already going beyond foundation models or specific language and,
and just visual models towards ones that just capture the whole
world. And right, also vision now
notices actually physics is an important part.
You only see so much. A lot of the complexity comes
from things you don't directly see like air moving or so I
think that's a really interesting outlook to to really
simulate basically the the whole world around us on a larger
scale with the physics accurately.
I think that could also be super useful for engineering and and
real applications in a bunch of areas.
But getting that right seems like a whole different level on
top of what we currently dealing with just only doing the physics
right with the foundation models.
But I think it's a really interesting outlook.
Yeah, it's actually a good point there.
Yeah, I forgot to ask you about that.
That I still don't fully get. I mean, I understand the world
models are obviously one very big use case is robotics and
driving drop, you know, autonomous vehicles and sort of
giving that synthetic data to help them to operate in the
world, The bit that I guess I'm less sure on.
And I wonder whether economically and scientifically
is the physics side, you know, does it make a difference how
good the physics is like, as long as it looks right?
And this, This is why I'm interested, because your movie
background, like does it, does it matter to the robots and to
the autonomous vehicles? Actually, I do think that at
least for robots, the physics play a big role because
ultimately the, the action that you're generating is, is the
control right of your, of the actuators of the actual things
that the robot should do. And you need to estimate surface
properties, the weight of an object, weight distribution
stability and all these things to at least in a, in a variable
environment to interact with the, with the world.
So think for robots by now rigid body type or in some somewhat
deformable and physics are playing quite a role.
So I would I would guess that it does make a difference for
autonomous driving. We could argue at the end you
only need to take right? Is it a dangerous situation or
would something have gone wrong? You can probably do a lot
without having to go through the whole crash or actually
simulating how the car would tumble or crash into a building
or so. That might not be so important,
right, As long as you can detect this would have gone wrong.
But for robots actually having to really interact with objects,
I can imagine that plays a larger role.
And then again looking towards fluids, I, I think the challenge
also in, in the current development pipelines is this
interaction with the real world. I mean, I think in the end, the
current validation and certification still goes through
a lot of testing status and then you have all kinds of real world
flights and and experience to make sure that the systems
really do what they should in the real world.
But if you could shift some of that into an earlier stage, I
think that that could also be definitely interesting.
If right, you could have different environment
conditions, different flow conditions, and you could have
an have an object interact dynamically in in such an
environment. Yeah, I guess maybe maybe you're
right to maybe the examples I've looked at, you know, if you look
at a robot walking around a factory floor or it has no the
the drag, the lift, the thermal properties are very second or
third order effects. They don't really influence how
it's performing. But I guess you're right.
If it was like a drone in the sky or a boat in the water, then
the physics actually do make quite a bit of a difference in
and then I guess it really is depending what's the use of
these world moles, if they really are like a true, true
representation of the world for that drone or or helicopter or,
you know, I guess the moment the autonomous vehicle, you're
right, is it's more sensory. You know, like I see there's a
person walking. It's not to design the car,
right? It's not to say I see now that
in this real world, by changing the shape, the drag is lower
because it's actually I guess maybe that's where you're
thinking, right, that the world model could be used as a true
synthetic environment or cars, I guess if you could.
Simulate a rainy and stormy situation and then really get
feedback on how the rain would splash around a car, or how gust
of wind would influence driving stability in extreme conditions.
Or right, all kinds of of landing and starting scenarios
for planes. If you could really get feedback
on how all the forming plane with straining conditions would
would behave. I think that's that's still
beyond even regular simulation capabilities that full structure
interactions was changing conditions are are really
challenging. You could get some feedback on
these and then ideally put an optimization loop around them,
right? You you would want to actually
optimize your, you know, your wing profile or the shape of of
a vehicle tour behavior in the whole longer sequence.
I feel like this is also the big debate, well not big debate, but
a debate around the classic tool calling thing.
You know, like isn't LLM good at adding numbers?
No, so just go and call a calculator.
That's like ultimately, I guess if you call Chachi BT to add 2
numbers together, it's just calling a tool of a calculator,
adding the 2 numbers together. I kind of wonder it's sometimes
in this world of, you know, world models or optimizing, you
know, is it, is it really that it's truly integrated into a
world model? Or is it more likely that there
is a sort of agent workflow where it's calling a surrogate
model and maybe that's the where they're two separate models, if
you know what I mean, rather than just a world model.
It's it's more than a gentic way, if you know what I mean.
Or not that that would already be a big step, right, If if you
had any visual model that could on demand call some physics sub
models surrogates to to get feedback on how these different
pieces is recognised. Should should interact of
original bodies. Actually these days you would
just call it original body simulator.
You would probably not need a surrogate.
Yeah, that's. That's kind of where I always
debate it and that and that's the traditional one that you
said if the solvers get so fast, like the spectral solver you
mentioned or now with, you know, advancements in GPUs, some
people are saying I can do a simulation in a minute.
And then you think, well, do I need a surrogate?
If if I can, if the solver can run so fast?
Right, I think for World War, it's an interesting challenge
will be to to quickly switch from from the proximate fidelity
kind of for for rigid bodies or the former objects of winds or
different feedbacks needed in the environment.
Sure, if I know I need perfect rigid bodies for a certain case
that might be ideal, but but right I need some feedback on on
wind or how right my plastic cup should deform or should it break
predictions across the all these different physical phenomena.
I could imagine that a flexible surrogate that at least gives a
gives the first prediction of estimates of of what what might
happen there could be beneficial.
I think that would be tricky to do with classical simulators.
Yes, yeah, yeah, yeah, that. I think this is where it's the
classic what's the, what's the use case of real time?
And sometimes if you go to an engineering team and I've had
this where you tell them that a surrogate model prediction in
one second, sometimes they'll say, well, we don't need it to
do in one second. You know, it's actually fine If
it takes 5 minutes or even half an hour, it's OK.
It's not a bottleneck. Whereas I guess if it's
integrated in a world model or something, you need it to be
essentially real time or else the whole value breaks of some
simulation. You know, like testing a car
moving. If you have to sort of pause the
simulator for 30 minutes, then it's not that.
Feel right? Yeah, I mean real time is is
extremely challenging. Then you need I guess to to be a
realistic virtual environment for a person.
Yes, you need milliseconds. Yeah, very real.
Yeah. So maybe a final question to you
looking more if, if there's a student listening to this or
someone doing HD or early in their career, given, given all
these changes, what would you recommend someone was would
study as a PhD topic? You know what was going to
future proof them in this world. Maybe I'm biased here, but I
think these are really important techniques.
I, I do see a lot of potential for AI based techniques.
So I think it's important to have an understanding to know
the classic, the classic physics numerics.
But by now within these numerical tools, I think AI is,
is a super important component. So I could, could highly
recommend looking at at least combinations of classic and AI
based methods. And my, my guess is also that in
the future or the next couple of of years, we were not going to
be completely replaced by I don't think chap DPT is going to
take over fluid simulation too soon.
So having experts that we understand what's happening
there, I think it's it's going to be needed for quite a while.
So I think it's still a good field to to work on.
Yeah, I guess that that's the the truth, I guess the, the
short, medium, long term, I guess I see at least now the
short term, there's even more need for specialists because
frankly, it's almost become that simulation is more of a popular
topic now. The fact that all these startups
are getting funded, the fact that, you know, there's more
money in this space, there's more need to find out who's an
expert in crash simulations, who's an expert in acoustics,
who's an expert to help computer scientists, you know, to develop
these models. I guess it's only in the very
long term future that you could imagine.
Maybe you don't need as many specialists because so much of
that knowledge is now ingrained in these models.
But there's a, in the short term, it's almost even more
specialists to help. But deep, deep specialists, I
guess you really understand it that, that at least that's the
way I I'm seeing things at the moment.
And I think that's usually a great topic for a PhD, really
getting to the bottom of things. So I just, it's a nice
opportunity to work on one topic for a couple of years and yeah,
get as much understanding as possible.
So on. Right now I think PhD is in this
area. Still a very good idea.
Yeah, yeah. No, no, I agree.
Well, thank you so much for taking the time to speak.
I mean, I, I find all these topics fascinating and one of
the things I'll do is put a list of the papers in the, in the
show notes because I think actually there's a lot more to
gather from looking in the details and, and looking at the
codes that your, your, your group did.
So I know we didn't have time to cover all in, you know, full
detail, but I'll, I'll put a list and hopefully people can
find the time to read through the papers.
That's a good idea. Oh, great discussion.
Thanks for the invite. Yeah, interesting.
We'll speak again in two years time and we'll see if the
predictions are. That's interesting.
Thank you. Thanks.