Hi, and welcome to the Neil Ashton Podcast.
In each episode, we explained some of the fascinating ways
that science and engineering are changing the world around us.
We talked to leading engineers from elite level sports like
cycling in Formula One to some of the world's top academics to
understand how fluid dynamics, machine learning, supercomputing
are bringing in a new era discovery.
We also hear some of their life stories, their career advice,
and lessons they've learned on the way that I hope will be
helpful to you too. So sit back and enjoy this
episode. Hi, and welcome back to the Neil
Ashton Podcast. So today I wanted to talk about
the topic of foundational models for fluids or for computational
fluid dynamics. It's a topic that I brought up
and asked quite a few of the people that I interviewed
recently. But I, I wanted to give some
personal thoughts on this, partially because I had the
pleasure a couple of weeks ago of chairing a panel discussion
at the second data-driven fluid dynamics conference, which was
on a cough tac euromech thing that was done at Imperial Carter
London. I was involved in the first one
in Paris the previous year. And it's a great initiative that
has tried to bring the fluid community and I guess the AI
community together, particularly with a sort of European flavour,
I guess to it. But in this panel session, it
was really interesting because we had representatives from, you
know, Harvard, but also Caltech and, and some from industry and,
and really representing the experimental and the
computational side. And one of the things that I did
was to try and ask people about their opinion of foundational
models. But more interestingly or as
interesting was I tried to ask the audience some of the
questions and there was a couple of 100 people in the audience.
And so we use this Menti meter app that I have to confess I
haven't really used before, but it's great because you can
essentially do real time polling of people.
So I asked the question, do you believe that we will have
foundational models? And in fact, I'm just looking on
the on the screen now because I have the results up.
Well, I should tell you, the first thing I did ask was what's
your favorite British food? Bit of a tongue in chic really.
And people for fish and chips won, followed secondly by the
vomit emoji, which I was a bit offended by.
And then a full English breakfast.
So there we go. But more seriously, on the
actual topic of foundational models, I basically said, will
you know, can we get foundational morphin fluids?
How generalisable could it be? And what I asked was, could it?
Could this foundational model provide instantaneous pressure,
velocity for any boundary conditional geometry?
So as in you can say to it here's a geometry with a certain
boundary condition, can it predict it and give you the
instantaneous pressure and velocity?
A bit like ACFD solver. Now you give it a geometry, you
give it some boundary conditions, and it would go and
solve it. Second option I gave people is
exactly the same, but time averaged.
So not instantaneous, but time averaged, and I'll get into the
nuance of that in a little bit. Then the third option was yes,
but only for a specific use case, as in maybe like a road,
car or a plane. And finally, option was
essentially no physics. Sorry fluids are just too
complex for an AI model to learn.
And interestingly the out of the scores it was 38 and this is out
of 100 and something. So the the the most popular
answer was only for a specific use case. 38 people close 2nd.
37 was provide time average to any bound addition and geometry
followed by instantaneous pressure.
And finally 13 had fluids that are just too complex.
So out of that audience, which is biased because they are
researchers looking at fluids and AI, the sense was it really
is only possible for a specific use case.
And therefore the terminology of foundational model is perhaps
pushing it a little bit or needs redefining.
OK. So that's why essentially I
wanted to to talk about it and I wanted to share I guess some
opinions of my own, some stuff I've seen along the way.
I'm in a lucky position, I guess.
I'm working in the tech sector that I get to engage with a lot
of customers and academics from different groups.
So I'd like to and, and go to conferences.
And I'd like to say I'd therefore have a, a reasonable
good view of what's going on in the community.
And I would actually say that things are looking very
interesting as in only a couple of years ago there was really no
work into this idea of a foundational model.
And another way of thinking of foundational model is simply a
pre train model. So what most people were looking
at was simply can we come up with some sort of algorithm,
some architecture that could allow somebody to go and train
an AI model based on prior simulation data.
So we're not talking about accelerating an existing code,
we're talking about replacing the code with a surrogate model.
And that was certainly the main focus probably five years ago or
four years ago. And there was some Seminole work
done with things like mesh, graphnet and others which took a
graph neural net approach where you could essentially take the
nodes of the mesh as the input. And that was something or
particles I guess because it was all particle examples of this.
And this rapidly became one of the success stories in the
weather and climate, but also was showing potential for CFD.
And this led to a number of start-ups that that formed over
the time, the sort of Navastos, Nora concepts and others who
took some of these work or pioneered things themselves and,
and released some of those things that people could go and,
and try for themselves. And so you saw in the past, you
know, several years, people really wanting to test out those
methods and with some success for sure.
But obviously that requires you to have your own training data,
your own ability to train a model and to have, I guess, both
the software and the knowledge to do it.
And I guess if you can compare that against what you would
typically see in the LLM space where maturity of us are not
training our own LLMS, but we are simply doing a prompt and
asking a question, doing an inference, we're not training
it. The logical question was, well,
could it be possible to train a model and then just provide it
to people as a pre trained model?
And it's a really interesting discussion because it has pretty
large it, it could have a potentially very large impact on
the CFT community. Now, there are some of you out
there who still have very negative opinions of AI and sort
of C as a fad, and it will come and go.
But I would say those voices are slowly starting to be overtaken
by the realization that actually these methods have a lot of
potential. Only, like I said, five years
ago with this was basically not happening.
Nobody was even talking about it.
I remember I got involved maybe three years ago in this topic,
so still quite late to the game I guess.
But even three years ago I remember most people had no idea
what this was all about. They just thought of this as
like reduced order modelling ROMs, real, no sense of what was
going on, didn't know what architecture would work.
There was only maybe one or two start-ups in the space, wasn't
very prominent. Fast forward to now and I would
say that it is certainly rapidly expanding and I would say
companies can be categorised as either trying to lead in this
way, being very interested and the number who were just set
against it is now sort of getting lower.
But I would say that this is still a little bit split between
the industries. And like with there's a bit of a
parallel with high fidelity methods, I see the automotive
sector as being much faster to adopt these new technologies or
investigate these new technologies.
Automotive companies today use sort of hybrid around scale
resolving methods on a day-to-day basis, whereas the
aerospace sector is still sort of proving out that they think
they can be used in in production.
And some of that is because Reynolds numbers and
complexities. Another is just a mindset, I
believe. And interestingly, I see the
same thing with the AII, see that the automotive companies
are far more motivated to try and find AI approaches that can
be faster potentially and are embracing it.
Whereas I see aerospace is more in the mindset of I don't think
this can work for me. And I guess part of me, the
reason doing this podcast episode is really just to put
across that, and this is a genuine, this is not influenced
by the company I work for. I've I've these all sort of
independent opinions, personal thoughts, but I really do think
that these are looking incredibly interesting for the
obvious reason that, you know, now if you do an inference on a,
on a model, you're going to get an answer in seconds.
But the accuracy is debatable whether people have truly proved
out that these these AI methods are accurate enough.
They give a very good impression of, you know, approximately what
the flow field is or approximately what the lift or
drag is. But I would say that we are
still not at the case where they're widely accepted or there
are very clear accuracy. I would say it's a bit like CFD
in the early days where you could argue there was clear
points where it worked well, but there were many examples where
it didn't work. And hence why CFD took decades,
you know, to to progress and be trusted enough to be a dominant
design tool. Sai is still in that sense where
it's probably not trusted as a key design tool yet.
But I would say that the number of companies really investing in
this is significantly growing. The number of start-ups who are
emerging in this space is growing month on month.
And the interest from the big is VS is also growing month on
month. But the challenge is the data.
So do you have available data? And this has always been the
challenge for the idea of a foundational model, as probably
people know, you know, the reason that these LMS and others
can do so well is that they have a very large data source to, to
to, to get to. So typically scraping off the
Internet, but nowadays maybe doing commercial deals with a
certain repository of data. You know, if you want to do
something on journal papers, you might do a deal with Elsevier or
John Wiley. If you want to do something on
video, you might do a deal with a video, you know, distributor
or or a newspaper. If you want to do on text or
pictures, you might do with Adobe.
You know, there's a lot of commercial arrangements going on
now. And I think that's what's super
interesting on the CFD side because on one side you could
say the data is just not available.
If you want to do combustion modelling or multi phase or
hypersonic something. This is not publicly available
data. So how do you you know, how do
you solve that? And this is where I think
there's an interesting technical but also commercial angle to
this, which is, is it worth it for a company to independently
run lots and lots of simulations to the fact that they can pre
train a model and then supply the model and, and and charge it
at a certain rate. And we have seen some early
signs of this happening. You know, there's a start up
luminary cloud. Some people know that I spoke to
to Juan Alonso, the founder, and they recently released a sort of
foundational model for Rd. cars where they'd, you know, run a
whole bunch of simulations themselves in collaboration with
an end with an end customer of theirs and and then released
that and pre trained the model. Now that it's probably too early
to see, you know, how ultimately successful that that that will
be, But it's it's they've sort of fired the starting gun, so to
speak, on an actual company releasing a sort of pre trained
model. And I think it's really
interesting to observe how useful that is because on one
side, it removes a massive barrier for a company to to
collect all the day, to find all the day to have the expertise to
use an AI tool. It's much easier just to do
inference on a pre trade model. So I think that's the first
thing I predict that we're going to see many more of those come
out. Many more companies will will,
will do that both I think from a start up and a nice fee space.
But how, how much data can a company generate and how can
they incentivize it? Well, one interesting thing,
when I spoke with Priff, who's the CTO of ANSYS, he said in one
of the talks that I gave that, you know, maybe the ISV needs to
incentivize people to allow the ISV to have the data.
So I think that's a really interesting proposition that,
you know, what if you would take a box that was to say, well, you
know, as you run your CFD simulation, I give permission
for the ISV to use that data and perhaps in return they get, you
know, some commercial incentive to do it.
And I think that's a really interesting proposition.
And I'm, I'm, I'm keen to see if some of the Isvs go down that
route. And this actually links why I've
often spoken about the cloud being an important Ave. and the
cloud being an important Ave. links to this because of the SAS
bit. If you give somebody a on Prem
binary, even if they tick some box to say we're happy for you
to look at our data, what's the mechanism to share that file?
Well, it's pretty difficult because it's running on their
local network. How are they going to, you know,
send that over to you? It's not practical where with a
SAS solution, it's much easier because it's by definition
running in, let's say someone's cloud.
And so if you say I want to share some of my data with them,
it's actually much easier to do it.
So I predict that part of the motivation and I think we'll see
an acceleration of the sassification is to make this
sort of AI and data collection easier both.
If you're an existing AI start up, you know, you could say,
hey, use my AI tool and I'll give you a discount if you share
your training data with me and over time will collect more
data. And also it links into one of
the things I've said repeatedly on this podcast about fast CFD
solvers or CAE solvers. Because now one of the big
things is if you've got to generate 5000 CFD cases, if your
CFD solver is twice as fast as another CFD solver, that's a big
amount of money to save. Or for the same budget, you
could run twice as many cases. So if we assume that people to
build these foundational models will have to run hundreds of
thousands of cases, then actually there will be a huge
focus on on enabling the code to be as efficient as possible.
Now, one of the other things that I haven't brought up, but I
think is another interesting one is everything I've spoken about
now assumes that the CFD code is independent from the AI code.
But as anybody knows anything about, for example, in situ
visualisation, we'll know that there's a strong move now
towards this idea of doing it, you know, online rather than
saving everything to disk and then doing the visualisation.
Can you do it whilst it's running?
And this is obviously something that's not just me, you know,
saying this for the first time, it's known that I think there'll
be a big increase in the Ori is certainly in the academic world
of trying to build your CFD code in the same framework as your AI
code. Some groups are doing this in
Pytorch, you know, how do I write a code in there?
Or they're using some sort of Python code that you can
automatically, you know, differentiate and move between
to pass gradients along and to the neural networks.
But I predict this will be a will be important because if you
are trying to build some pre trained model, some sort of
foundational model, you don't really want to be doing your CFD
independently to your AI training.
You want them to be tightly coupled and potentially adding
more points as you need them and not having to be, you know,
bound by IO issues, having to keep the data.
Which is particularly true if you have the dream, and I think
this is a much longer term dream of being able to do
instantaneous. So time dependent, most training
now is done on time average solutions, even if it's a time
accurate simulation, because just the amount of data, I mean
having created with colleagues, you know the driver ML data set,
it's already 30 terabytes with one time step essentially as in
the time averaging, if you were going to try and do it for all
of them, 200,000 iterations, well, you can do the maths, it's
unbelievably large. But if at every time we were
running those simulations, we were passing that data to a
model to train, then the cost we we don't need to save the time
steps out. We're just passing it in memory.
So we actually could have done it because we were running the
simulations time after time dependent anyway.
So I think this is where there's a lot of interest in these.
I feel this is where new technologies are converging with
AI as the sort of motivating factor.
So foundational models. I, I really would encourage any
academic who's listening to this, I think this is the topic
to propose as a, you know, a big university project or European
project or government project or, you know, anything that is
big. Because if this could be made
and there's so many debates on should be open source, should be
closed source. It could transform the way that
we are doing CFD today. If it is possible, it would
change the commercial landscape, it would change the technical
landscape. And if you imagine that it's
possible for a certain class of applications, you've got to
imagine the transfer learning, the fine tuning.
You know, at some point how different is a car than a plane
or a city if you are starting with, if the model is able to
learn some of these interactions.
I haven't really spoken about the idea of including physics
into it, but I I really do feel that just as five years ago AI
was an interesting thing for people to look at, I think the
whole concept of foundational models is becoming interesting.
I believe at the beginning it will be for Pacific use cases as
the audience voted, but I perceive that there will be a
bit of an arms race between the highest fees and startups to
build this, and it'll be interesting to see whether
companies see this as their secret sauce.
Whether you know, an aerospace manufacturer or an automotive
manufacturer says, well, I don't need the ISV, I'm going to do it
myself. And this will be an interesting
balance between the software suppliers, the companies,
because on the other hand, the automotive company can say,
well, we're not a software, we're a car designer.
Why are we going to write our own codes?
And that's certainly been the case even in the aerospace.
There's a move towards commercial codes.
Well, you know, it used to be the case, everybody would write
their own codes and that was their IP.
And then they realised their IP is making cars or planes, not
writing software. So it, I still suspect that most
of this work, these foundations will still make their way into
commercial sort of big software companies IS VS.
But given that they don't have the data, it's the sort of
engineering companies who have the data and they need to
somehow get that data or produce that data.
There'll be an interesting dynamic, I think in
collaboration. So these are just my personal
thoughts. I could be completely wrong, but
I am quite bullish on the idea of some of these foundational
models. And, and certainly, you know, in
my day job, this is something I'm actively pursuing.
I'm I'm academically interested. I'm interested in, you know,
NVIDIA doing its bit to help things along the way.
And yeah, I'll be really interested to see what happens
in a few years time. So maybe I I'll try and set a
reminder in two years to record another one and see how much of
this. I was right on and maybe it was
it'll all not happen and I was completely wrong.
But I'm, I'm going to make a bet that in a couple of years we
will have progressed quite a bit further than than than we have
done to date. So with that, thanks for thanks
for listening. I would really enjoy your
comments. Let me know what you think if
I'm completely wrong, if you have a different viewpoint on
it, please put your comments in the YouTube or, or send me a
message. And yeah, thanks for listening
and hope you enjoyed it.