Prof. Johannes Brandstetter on AI for Computational Fluid Dynamics

0:00 Hi, and welcome to the Neil Ashton Podcast. In each episode, we explained some of the fascinating ways that science and engineering are changing the world around us. We talked to leading engineers from elite level sports like cycling in Formula One to some of the world's top academics to understand how fluid dynamics, machine learning, supercomputing are bringing in a new era discovery. We also hear some of their life stories, their career advice, the lessons they've learned on the way that I hope will be helpful to you too. So sit back and enjoy this episode. Hi, and welcome back to the Neil Ashton Podcast. So today's episode is with

0:45 somebody who I've actually really enjoyed getting to know better and chatting it various conferences that we've attended together and some mini symposia that we've done, which is Johannes brand set up. So he's got an interesting profile because not only is he a professor at England's at EU Johannes Kepler University in Austria, but he also founded or Co founded a startup recently called MEAI. His background is one that is very well positioned to help advance the state of machine learning for, for for CFD and I guess CAE more broadly. And yes, this is an episode on machine learning again, I had sometimes feel bad that I keep

1:29 doing this, but I think this season there's a mixture of of episodes and I still feel that there is value in going through the machine learning topic. And it's selfishly something that I I'm constantly interested in learning more about and learning more about. It is one thing that I always do when I speak to Yanis. And he has an interesting background because whilst he has AI guess a high energy physics background that he he, he did and he was at CERN and he worked in that sort of area. He then moved into the machine learning side, I guess, having spent time with Max Welling at the Amsterdam University in the machine learning lab.

2:13 And then I think he also, well, I don't think I know that he was then at Microsoft Research where Max was also out. And so the two of them collaborated a lot. You know, if you don't know who Max Welling is, well, first Google, his name is one of The Pioneers of machine learning. But I also did an episode with him in the last season that I think was really interesting and I think he must have been inspired by Max a little bit, even though we didn't talk about it in this episode, because Max also has been a serial start up founder. And the reason I said that he has an interesting background is because whilst he was at

2:47 Microsoft, he also worked on the Aurora machine learning for web and climate project. That I think is extremely useful when you have people who have gone through these major projects in another field because they bring with them lessons and learnings that are important. And, and what Johannes has been doing is first of all, bringing a very academic mindset to this academic in the terms of publishing and transparency, which I think is very welcome. So you'll find in the link in the YouTube a couple of the papers that he's published and and his startup has published a new sort of transformer based model that I, at least from my

3:27 reading, is unique and certainly seems to be at the one of the most cutting edge in state-of-the-art models out there today, both in terms of conceptual but also the accuracy on the data sets that they've shown. But I admire his vision and his willingness to try and solve the problem rather than being focused on the model, as in some people seem to, once they come up with a model, fixate on that being it, rather than being willing to consider that their model may be the right thing at the time, but then there'll be other models that get better. And he, his willingness to accept that, I think is a breath of fresh air and definitely will

4:06 help the community to, to involve, to evolve. So that's what we talked about today. Really we, we tried to go through and discuss, you know, more the general topics around machine learning, but really diving into this transformer based approach that he has. And I'm trying to understand some of his thoughts around the similarities to null operators, some of the slight differences, some of the links to I guess graph neural Nets and mesh graph Nets that probably most people are aware of. And then we talked a little bit about the process of forming the startup and some discussions around future topics. As with anything, I always feel bad because I finished the

4:45 episode and I think I really should have discussed more or, or, or dived into certain topics, but I'm conscious it was already over, you know, an hour 20 and people will, you know, fall asleep listening to some of these episodes. But hopefully it's enough to spark your interest and encourage you to read more about the topics that we discussed. And please leave any comments or let me know if you have any suggestions for for other topics. But for now, please sit back and enjoy what I found an extremely interesting episode with Johannes Bransetta. Cool. All right, thanks for joining me today. I wanted to have a chat with you

5:24 for a while. We've had some nice discussions over wine and and coffee, but nothing public. So this is the public version of our discussions, but maybe it'd be good for people to learn a little bit more about you. You know, where you're at today. Yeah. Who? Who is Johannes? Yeah, thanks first of all. Thanks really Neil for for having me. It's it's a great pleasure finally seeing this guitars here in a one to one setting, so to say. Yeah, my background is actually I'm I'm a learned physicist, some say a failed physicist. But after my PHDI switched to machine learning, as many people also did. I have spent time with

6:07 Sepulchreiter and three years in Amsterdam with Maxwelling. I was in industry for two years at Microsoft Research. And there at Microsoft Research I really, really discovered my likings for large scale simulation, especially for weather and climate modelling, because these are like some of the biggest problems you can have and you can tackle with machine learning. I also saw the transformative impact on this, on the systems, and that was then the point where I decided to to do my own thing to, but to apply it not to known problems like weather, but rather uncharted territories like engineering, simulation and this kind of thing.

6:48 And I decided to get my own group at university to build up my knowledge base back in Austria, where I'm coming from. And along the road. It also happened that I founded a start up MEII where we do this simulations at scale and where things are coming together with a bit more compute and a bit more resources. Nice. Well, we'll we need to dive into each of those, but maybe the first one was the Aurora project. I guess this is maybe it, it was subtaneously both a motivation, I think certainly for me and others who saw what was happening in weather and climate and was saying, you know, surely we could do that in engineering. But also it was an example of a,

7:37 yeah, a real large scale project. So what was it like working on that Aurora? What was the key lessons you learned both from maybe an ML point of view, but also a project and a team point of view? So the actual Aurora team was pretty small. I mean, all the credits to to Chris Bodner versus Pransma, Megan Stanley and a Lucic who did that, all the heavy lifting afterwards to finalize this project, to train to build the data loader. The Aurora was the learnings I think are threefold. First of all, with computer vision tricks, you can get very, very far in engineering or scientific application. In the end it was a swing transformer trained in in a 3D

8:25 swing transformer. Secondly that the the data engineering is potentially the hardest one and and thirdly that weather has a very unfair advantage for machine learning because nobody knows how weather is functioning. So you, you will always train a machine learning model on like actual data, which basically has the actual physical laws hidden. Whereas the, the the numerical methods, they have to come up with these laws by themselves. And that already brings these, these key interests of mine, which is that that neural surrogates will never replace numerics. They will just be a different branch which depending on how

9:13 you play the card acts in your favour. And that it's all about engineering and scaling these, these things to this, to this basically to this problems where they really have an impact and there really matters. And the impact for Aurora, you could see if you look through this, this small, through this downstream tasks that the larger these challenges are, the more impact you can basically generate. Yeah, it seemed to definitely spur on. And now it seems that there's been so many of the weather and climate models, it's almost got not congested, but there's there's, there's quite a few coming out. And so it's not obviously so

9:56 clear how each are progressing past each other and and how much are those of just an individual groups need to have their own model, if you know what I mean. But did you, I guess one of the big things that I want to discuss today with you in particular was to dive into, I guess where machine learning, how can I phrase this machine learning for engineering, for CFD, for CAE. You know, it's something I've spoken to a few people about and there's a lot going on, but it's hasn't always been so clear where the big breakthroughs will be or how, how good are we today and where we're going.

10:46 It it's it's still seems a congested space with different opinions, you know, with pins on one side and then these sort of mesh graph Nets and other things. So it's a bit of a, an open-ended question to you, but I guess like where do you see the state-of-the-art? What what's been your journey from the sort of mesh graph Nets towards your current thinking? Like how, how have you try to solve this problem, I guess. And let's, let's say, let's take CFD, maybe is the the the problem like automotive aerodynamics for example? Yeah, that's a very good, my favorite topic actually making the connections to weather. Let's let's say what what basically was driving this

11:32 weather modelling. I would say that NVIDIA was the first with the forecast net paper to to bring out a model which worked that was trained on error 5, which is a publicly available large scale data set. Depending on how you sample you get roughly a petabyte of data or a few 100 terabytes of data from it. And then you do basically mean squared error training that that was the the scene set and and as soon as this problem was defined of this input output relation of this metrics of of what to test, then people did what, what they do best. They optimize on this sort of problems. If you go to engineering, things are very different in, in many, many aspects.

12:17 So first of all, we don't have this data set. We don't have an error five data set. And even if you had an error five data set, I mean, there are now at least 2-3 publicly available CFD data sets. Needless to say that my favorite one is the Tri ML data set which are industrial standard. But even if these data sets are out there, people don't know how to train on them. And everyone trains differently. And this comes from the fact that we just don't know what you really want to optimize. There is people who sub sample parts of the surface and map to, to pressure values on the surface, people who sub sample part of surface and volume and so on and so forth.

12:59 So similarly to whether we have to figure out what's the right task, what's the right learning task, which we which we have to to do or like the tasks in, in order to really develop the right models in the right frameworks for that. Secondly, I think what is also very different to weather is that we have this input output relation. So in weather, it's always clear you take time T and you map to T + 1. That's basically a segmentation task on steroids because in the end every pixel on the Earth gets mapped to but with pixel on the earth like a time time point later, which you can basically you can take all the models from

13:48 computer vision. And obviously there is some tricks with resolution and and if you take the sphere into account, yes or not. But in the end it's it's a pixel segmentation task. And this is definitely not true for for many CFD related tasks because the task there is from a pure numerical perspective, you have a geometry which you can characterize with couple of parameters, probably less than 100 even. And then you get this full-fledged flow fields around and on the car so that the actual input to output ratio is very, very different. And we don't have architectures, we don't have frameworks, we don't have an understanding of how to tackle that.

14:34 But that makes it so exciting I would say. And maybe thirdly, if you, if you really look at these weather models and what they can do with hurricane predictions and so on, obviously there's always room for improvement and obviously you can go farther and farther. But if you just look what happened in the last two years and the resources and weather are I would say it's tiny, tiny fractions to the resources in LLM and and video generation. You see this tremendous progress and what what hard problems and hard multi scale problems we are already able to model and which we can able better than numerics. So it's very clear that if done

15:09 rightly and if the right incentives are there, this progress and this type of complexity we're able to do in in CFD. Obviously then we have to play the game numerics and ML together. Yeah, which is then I think the hardest task to answer. Yeah, that, that does seem, I guess the joke that the Earth is always the Earth. And so the geometry is sort of, you know, not so much the problem. And yeah, I guess it all depends on what is the data. You're right, in the CFD example, the geometries could be hugely varying and that link between the geometry and the volume is a Yeah, it it's so the geometry is geometry and the boundary conditions is ultimately what affects the

15:59 flows. So where have you seen, I guess a lot of people listening to this and myself included, probably had their first introduction with, you know, what the Deep Mind team did with with mesh graph Nets. I mean, there was obviously stuff before that, but that felt like the bit when everyone stood up and and noticed why? Why do you not feel that that is the right approach? It's always right or wrong. It's always hard to to to judge. But I think that the, the mesh craft net and, and, and and and graph neural network simulators from the Peter Battaglia group were tremendously important milestone in this field, mostly

16:47 because they, they were writing down this learning problem. And I myself went into this field of simulations because of these papers trying to model particle particle systems. What, what we noticed back in the days, I mean, this is like 5-6 years ago, that it's tremendously hard to model all particle particle interaction as numerics is doing. So if you have this 10,000 system of particles and they interact and you have you have something like a graph neural network and you have to make all these interactions correctly because otherwise the temporal integration will will just, yeah, just go, go, go crazy. That's just a very hard learning

17:29 problem. And this is not what machine learning is really good at. Machine learning is good at to understand the global dynamics, to understand the behavior of the system, to understand where things are going on a global scale, but not on a particle particle scale. So I, I think from, from, from setting up this learning problem, this was a tremendously important step. Obviously new ways of, of, of of models are coming the way for me personally, it was a big breakthrough that we started to, to see things as fields. So everything in my, in my head is a field. And because you a field is something which evolves over time.

18:06 It can be an occupancy field, which tells you where is mass, where is no mass. It can be assigned distance field. It can be a velocity field, it can be a displacement field, whatever field you want. But it's much easier for a, for a model to, to tackle, which is actually obviously the, the next step if you think of this whole scientific machine learning and no operator and, and, and so on and so forth. And in the end, I mean, this is also how you, you model the, the, the weather. So I, I think conceptually from me personally, from me, I don't know how it is for others. Understanding that you can model large scale systems as a field was, was breaking a lot of

18:42 boundaries in, in terms of scalability. Because if you model a system of 10 million, 100 million particles as a field, it's computation and not the problem. It's from a learning perspective, it's not the problem because you just don't need all these particles. You you'll learn the dynamics with the very heavily subsampled representation and and and and obtaining the full field is also not a problem because this this exists. So I would say for me understanding what machine learning is really good at and and and and and and what numerics is really good at and and and and and playing those two on, on on those two fronts was was was making the the

19:27 difference. Yeah. Well, one thing that I've maybe would appreciate you explaining to the audience and maybe let's first introduce some of the work that you've done in this recent paper of yours. So you you developed this transformer based approach, what you call it anchor based universal physics. Trump What? What? Well, maybe you explain what? Yeah, we. Have actually a couple of works of 1 is which we call Universal Physics transformer. That's basically how we do latent space modelling of logical system. Then we did something which is called Neural DM where we applied those to to multi flows and and and multi. Particle systems and then we, we, we recently applied this to

20:19 to large scale CFD similar framework, similar approach, which we call anchored branched because these are the two ingredients which which which we need to do in order to get this to work on large systems. So in that context, what I'm hoping you could do is maybe explain a little bit, because there's a lot of people who listen to this podcast who are maybe not ML specialists, but are CFD primarily specialists who are now, you know, more and more interested in ML and maybe actually developing it and, and yeah, sort of progressing. And you mentioned a really interesting point to me and I'm hoping you can explain this in a way that is understandable by

21:06 others is that I, and I guess some of this is a maths exercise, but you know, and I've been guilty of this. Sometimes there's this hierarchical thought process of being like, OK, you've got some sort of graph based thing where it's more like point to point. Then you've got a new operator where it's like more to solutions or fields. And then I initially thought of, OK, and then you have Transformers. And yet you see some people who say, well, everything is a new operator if you do the maths or design it. So can you explain and maybe dive a little bit deeper into that concept of solution mapping or field mapping and then how it lakes transforms?

21:47 Because I think it is a little bit confusing for for for people. So if yeah, if we could maybe get into that, that would be in in the context of let's say this drive air, ML or car, however you want to explain it. Yeah, happily to do so. So for me, in fact neural operate and this is also how it is introduced is something which maps between function spaces, which is very yeah, which is not very informative if you if you, because what does it mean? You map between function spaces. But I always think of it as you have two functions, one input function, 1 output function. And no matter what you sample from the input function and what

22:28 you predict on the output function, this has to be fulfilled. Meaning that if a sample 10 points from the input function, I should be able to predict as many points as I want from the output function and those points I predict are actually on the output function. If a sample with the same network with the same operator 6 points from the input function, I should also get the same 10 points and if I want more points, I get more points on the output function. Obviously there is a limit. If a sample to little input points, then obviously the approximation network is not able to really get the information needs. But if if this this this

23:06 sampling limit is is reached no matter how many points and where those points are spaced, it should always give you some sort of representation which allow you to construct the output function. So in the latent space this mapping is, then is then, so to say, resolution agnostic and and that with some small tricks you can do with convolutions, with full node operators, with Transformers and so on and so forth. And obviously Transformers with their. Can I if I've seen to it just for one second, just to make it even more clearer, when you say sampling the input function and the output function by input function, you're meaning like the geometry surface in this

23:49 conceptual sense and the output function being let's say the volume. Would that be 1 interpretation? Yes, for example you can. You can think of it that the input is, the is the geometry and the output is either the field on the geometry or the field in the volume, or both actually. And the idea of this so where you know, this sounds like a bit of AI. Remember when this first came out, I was I think it was billed as a resolution independent. So it which always to ACFD person feels like the no free lunch sort of theorem. Like how can this possibly Yeah, you know, how can that be true? You're basically saying I don't need to map the entire geometry,

24:33 I can just take a few and still get the same answers. So where's the catch? I I guess on this? Yeah. I mean that that's a very good point. And I think I can make my, my point very clear with, with really now going to CFD and, and, and before that, I really have to say, if people think of resolution, we always think again of this segmentation as, as we do with weather, right? You have a certain number of longitude and latitude points. And if we double those points, do we get higher resolution or do we just have an interpolation and so on and so forth. So can we just run our whatever unit and interpolate between this resolution or is there a

25:14 way to get this stuff resolved? So there's always this input output mapping and obviously it will end up with interpolation effect. So either you interpolate somewhere in your network or you interpolate on the output grid because as you said, no free lunch. But if we go to CFD, things are a bit different, especially that the input output resolution, as I said before, is very different or input output connection. And, and actually we show in this recent paper that if you subsample a few of the points, so both on geometry and surface and, and you do some, some training, there is a certain amount of points, but the performance stagnates. So if it's, it's for for tribe

25:56 ML, this is roughly 128 Ki think or 256 I, I, I don't know by heart. And if you then add more points, the performance is not getting better because all the, the information which in neural network needs is in those points. And, and if you sample it, let's say cleverly so, so that the distribution really covers the whole surface and the critical parts. And then also points at the closer to the surface and the volume are represented correctly. All this, this type of stuff. But there is a certain amount of points where you really capture the whole phenomenon. And that's the, the, the, the resolution invariant. So to say. Obviously, if you go lower with

26:39 the point to sample, you lose a bit of representation. So you don't, you don't resolve the full physics. So you're never able to really recover the whole physics. But there is a, a certain amount of points you need. And this is for, for each type of physics problem that you have the full information in the network. And then obviously depends how good your network is. And then obviously it makes much more sense to not have a network which again spits out this 128 points, but which conceptually is able to spit out as many points as you want because that's the new operator, right? That you can really get every point on the output function if if needed.

27:20 And this is something which is which is very, very interesting in CFD. Yeah. So I guess, again, correct me if I'm wrong, if I'm distilling this correctly, is you're saying if the surface in reality has 8,000,000 points or whatever the exact number is, you're saying that you should be able by training and sampling progressively different points, you shouldn't need more than 256,000 to actually do as good job as 8 million. Whereas I guess we've the analogy. I suppose what I'm trying to pick out is with mesh graph Nets or with graph neural Nets, people could in theory take a million, but they would normally like down sample, but they're

28:08 down sampling at least. Correct me if I'm wrong, you are then changing the problem statement in this it it will give you a different answer and and that I know was certain experience that we had and others have had where you go well, how much do I downsample and where do I pick the point to downsample? And so a lot of people take an 8,000,000 cell can't fit in AGP memory and take 500,000 but they're worried when they do inference they then would have to give the exact same mesh distribution or if they give a different mesh distribute. Do do you know what I'm getting? At that's lovely. I mean you, you say the most

28:48 important point, which I forgot. Obviously if you have to calculate something with drag or lift coefficient for which you you really need to get the correct value. You need the full simulation mesh, the full 8 to 9 million surface mesh. And obviously the network has to give you this full mesh because otherwise you're never able to calculate correct drag and lift coefficient. But the network is giving you this full network idea. Sorry, this full drag and lift coefficient on the full surface mesh, no matter if you give it 64,128 thousand 256,000 input points, it's just at some in number of input points the output is stagnating or the performance is stagnating

29:30 because you have reached the maximum performance and the the input and and obviously you can also give 8 million input points. The the the self attention will be terribly slow and I don't know what what parallelization tricks you need to do, but nevertheless it will give you the same answer in the output. So you can trustically reduce what you give as input as long as the output still gives you the correct answer. And that's, that's the magic we we got from transformer, which we cannot do with with other methods for. So for other methods we really had to make the problem much easier, which is obviously giving you the wrong physics. So if we down move on to and I

30:12 guess this is the the novelty that I saw in your approach and and I thought it was good to double click on a bit. It's why the Transformer, what specifically is that architecture giving you versus alternatives because yeah, and how and how that relate, if you could maybe go into a little bit because Transformers, I think most people know conceptually, you know, tokens, etcetera, LLMS, but maybe not in the context of CFD. Yeah, so transformer are tremendously flexible, tremendously optimised and and and tremendously well understood powerhouse power work, power horse in, in, in deep learning, right. They are they the the engine behind large language model

31:00 computer vision and and all these tricks are already done and they have a few certain properties which are super nice. The 1st is the the invariance with respect to sequence length. So no matter how long the the sequence length is which corresponds how many points you input to the network, it will do the same calculations, right? Which is very important for the scaling properties. And then they have this how to say so I call it discretization conversion. So if you sample more points in a in an error in an area, it is basically it will converge to some to some field. So, so the more points to sample, the, the better the

31:48 resolution gets, which is obviously extremely valid, important property for this new operator paradigm. They they are made. So the transformer paradigm is made and, and stress tested again and again and again for scaling, meaning that you can build larger models, that you can ingest larger data sets. Yeah. And, and, and all these things make them an ideal fit for us. I mean, I'm very happy that not everyone is using Transformers. That gives us some edge, but yeah. But specifically, So how do you tokenize the problem then? How could people conceptually understand, you know, mesh graph Nets conceptually was I have a node which corresponds to my

32:32 mesh, I have some edges and then I take that like you're not taking a token per node clearly or that wouldn't scale, right. So how could people conceptually think of that sort of tokenizing the the surface or the volume? I think that the way you have to approach this is not from a physicist perspective, but from a computer vision perspective. So what do people do in computer vision? So I think when do, when you do something like image generation where you type in a prompt, I want a horse in the, in the in the woods with whatever. So you have some text and then you mix that with some image which is generated, right? So you have basically 2 streams

33:17 and, and the second important part is this concept of patching. So the transformer for for text, it's very clear each word gets a token. So each word in a sentence or probably each each sign or whatever is tokenized for for images, small patches of the image will correspond to a token and a word. So everything is a token which then can interact with each other. But in in language, it's the token's words in, in envision, the tokens is small patches of of an image. So if we do the same thing in in CFD, what we actually have, we have basically three type of of information. We have the information of the geometry. So in general, what geometry we

34:07 have, we have the information of the volume and we have the information of basically the physics on the on the geometry. So it makes sense to treat this as either two or three modalities as we call it, right? So as as you have text and and and image and then you can tokenize either small areas, which makes sense. If you if you look, try to encode the geometry so small neighboring areas get pulled into one token so that you have this information which gets locally aggregated. Or you can for, for actually doing physics, you just take some samples of the mesh and see them as a token. And then you have the same setup, you have different tokens, they represent different

34:56 physic objects or physic parts of the physics. And then similar to what people do in computer vision, you let those tokens speak to each other. You like, like this volume tokens between each other and the volume tokens to the surface and so on and so forth. So it's really borrowing a lot of concepts from computer vision because that's what it works. So if I understand it correctly, essentially what you're doing is patching. So if you were to look at the surface mesh and if you have, you know, visually, if you looked at it and you had a clump of 5 by 5, then you, you would have, you know, 25 cells and however many edges and nodes, you would say, OK, well, that

35:48 can just become one token and then you go into the next, the next. So if you, so essentially you're able to go from 8,000,000 points and instead of taking 8 million tokens, you would have you know, 10,000 or what whatever the number is, is that. And the assumption is that the differences within that patch should be small enough to be not important. If you were to take too big a patch, I assume you would lose some accuracy because you're losing some local information. Would that be fair to say there comes a cut off point? Yeah. So if you if you talk about representing the geometry, what you correctly said, you can do two things, right.

36:30 You can treat each point of the geometry individually. It turns out this, this drive ML data set and so on so forth. The information is so rich. So in order to really represent the geometry, you need roughly, I don't know, half a million points. So it makes sense to pool them before, to aggregate them locally, and then to use those as, as, as tokens. That's just computationally much more efficient to have some sort of pooling that the number of tokens is not exploding. You could also obviously use representatives of the geometry as your points, but it makes sense to pull them before. And the same in the volume, right? You're essentially, instead of

37:13 taking 130 million nodes, you're breaking them into patches which themselves take into. Yeah, the The funny thing is so so there is this tool. So we always have the the number of points which it takes to represent physics correctly. This is 2 slightly different things. So for the geometry we really need a lot of tokens in order to really capture the geometry because the geometry is influencing the physics and so on and so forth. For the volume we don't need so many points. So we we only need roughly 128,000 or or even less to really capture the physics. So that's we are totally fine to pick those points individually and actually performance is not

37:56 degrading much if we go to 32 or 64,000 that will get more interesting if we get even larger data sets. But then you can also use some tricks. So there it's obviously fully fine to use individual points. So in that context, is it fair? A mental model is that you're saying that the number of points that are needed for a numerical solver should not be a one to one mapping. That you shouldn't think just because I need half a billion points to solve my PDE, that I should. I should need half a billion points for my mission, my machine learning model to learn the behaviour that we need to break from this concept of almost one to one mapping.

38:49 It's that part of your argument, because that's not most people that I think assumed in the GNN that I need to have this sort of 1 to one mapping that I'm, I'm sort of taking my notes and I'm putting that into my ML and that's the most logical way of learning the problem. Yeah. So I, I really like those questions, I have to say. So I would my argument is, and I mean you tell me you're the CFT expert, but I would say what a model has to be able to do is it has to produce the simulation mesh. So it has to give you predictions on the simulation mesh both on the on the surface and in the volume. And this is not only cars, it should be hold for airplanes and

39:30 so on and so forth. And if we, if we think of meshes which approach a billion of mesh points and where this is #1 condition, we cannot, we cannot use paralisation here because then we end up using thousand GPU's and this is not scaling very well. I'm looking at the actual information content. So it has to be the model has to be able to produce this this rich outputs in order to be a good tool to use. But on the other hand you don't need this huge output for training. You can train on much much smaller sub sampled version of the same problem. However, you have to make sure with null operator learning with function approximation that

40:14 those models you trained are then in inference or in test time. Able to give you the full answer on the full simulation measure. I think this is the true CFD problem or the true learning problem similar to the error 5 setup in Feather modelling which we have to do in CFD. So you yeah, so this is an interesting point that I it's good to maybe briefly discuss which is at imprint time. So we discussed about training time, you know what you take, how many of the points, how many do you agglomerate, you know, etcetera, etcetera. But I guess what you're getting to now is the inference time, which is something that maybe is

40:57 not so obvious to some people, but for me was always a bit of a conceptual challenge, which is how do you generate the mesh to do the inference on. So normally if you are running a normal CFD problem or you're taking the driver ML data set, you split it into a train and test, but both of them already have a mesh generated. You know, we're giving it you. So you already have the mesh. But the real life engineering problem is I want to predict a new geometry that I've never seen before. And the question is, do I have to mesh it like I would mesh ACFD problem or can I just take the CAD or can I just do some arbitrary case?

41:43 Am I understanding it right that this is sort of your argument of the neural operator or the field sort of base approach That if I take a new geometry of a new car that I haven't done anything before on and I mesh it 5 different ways and then ask your model to give me an A prediction, it should essentially give pretty much the same answer for those five different ways of meshing it. Yes, I think this is the true. To an end, to a point. I mean, two things. One, one thing I haven't said so, so I I was talking about the, the difference in in training and inference mesh.

42:33 I also should say for for us it's very important that the algorithm we are developing works for for cat geometry. So if you input interference only the cat geometry, you're able to obtain the full surface and full volume without giving any information of the surface and volume mesh in inference. The only thing the model gets the CFD input. Obviously the performance slightly degrades, but I'm amazed how well that works. And you can do that by using, obviously in training that the simulation mesh and, and, and using some tricks on the simulation mesh that because obviously you need this information in some way in training. But that you can tell your model

43:17 how to work with, with basically structured points in the in the volume and, and certain points on the on the surface such that interference it's the model is able to deal without any of those meshings. And I think this is somehow the true power of, of these deep learning models because they cannot only scale. You can, you can think of that the whole CFD compressed simulation is suddenly compressed to, to neural network weights. You just have to ask them the, the, the model, which region in the, in the 3D you want to have the prediction. But you can also do that from pure COD inputs. And so, so no storing, no whatever, right?

44:01 And yeah, this is truly exciting, I would say. Yeah, I think that's one of the bits that I think is the true test. Because if you have to generate a mesh, the real time element of the prediction starts to go. Because for many people, generating the mesh, the volume of the surface is itself quite a challenge. And I think most people would prefer to go straight from a CAD geometry or, or at least be less sensitive, you know, to, to, to, to the mesh. The, the only conceptual challenge to this, and I, I don't know if we've even discussed this before, but I'll just throw it out there, which I think still is part of the challenge in people's heads is

44:46 if you do ACFD simulation, you would expect there to be a difference with the mesh, right? It it should give a difference if I run that driver ML with a a grid that's half as course or twice as fine. I want it to give a difference because by having a course of mesh I have higher numerical error which should affect the flow field. And if I have a much finer mesh I should see a difference because I'm reducing the numerical error. I think where it gets a little bit harder with the notion of sort of resolution independence, is it sort of breaks from that mindset that, that I should be able to give you a mesh that's twice as fine or twice as coarse.

45:29 And yet I get the same answer. It it, it sort of messes with the CFD mind because you think it should be different because I I'm expecting in my normal PD solver for it to be different but it's not remember. Oh, it's, yeah. I mean, actually this was one of the reasons how we developed this, this anchor tokens, we call them anchor tokens approach because this very much is, I think the neural analogy of what you're saying. So what these anchor tokens are and basically they are the reasons how we can scale. So we train on a selected set of tokens on the surface and on the volume. And those tokens, as we already said, they need to capture the physics.

46:08 So it's, it needs to be a certain amount, but they are sufficiently well to train with self attention everything on Transformers and in fear and cyber. What we do, we, we, we pick these tokens. Those are representative points in the whole 3D volume from surface and the volume. And then basically those tokens span the weights of the keys and values. So in the in the Transformer, you have three different blocks and those span two blocks. So you do a first pass of the model with those tokens and then this is fixed. So you basically have what you've built is you've built a model which which is is a representation of of your full

46:46 flow and and then you can ask for specific points in the volume or on the surface to get another value. And these points can be arbitrary points, any point. And this point will just run through the the model with all these weights already fixed. So, so it only like a small part of the model is adjusted and it will give you an output. And obviously the more anchor points you have, the finer this this resolution gets, the better this prediction are obviously until a certain threshold, right. So if you think of 16,000 points in the volume, you can still reconstruct the full flow field, but it's it's not the full accuracy. If you go to 32,064 thousand at

47:30 some point, if the training allowed it, you will have the full field covered. And then with these, with these points, you can get arbitrary find resolution and similar to in CFD and you and you cover all the the points in space. I think I see this anchor points really as as some, some some sort of of of this computation cells in CFD. Obviously the the discrepancy between number of mesh cells you need in CFD to get the simulation to converge towards anchor points you need in order to cover the full mesh. Is is trusted different? But it's just because AI is trusted different, different than the Mac simulation. But there is this analogy. And obviously if you if you

48:11 don't use any of these points, it's very, very hard to construct these dynamics. Yeah. I think conceptually some of the interesting challenge of that is, correct me if I'm wrong, but you, your ground truth is your train data. So you feel that if you match the train data, that's the best it can be. Right. But the training data itself is dependent on the mesh. And so one of the, I guess research topics that people have discussed is, well, how about multi fidelity? So you, you know, you have some high fidelity data, some low fidelity data, but how?

49:05 I'm so going off a tangent on this a little bit, But one of the things I always say to people is if you have a concept of like a foundational model where you're saying, OK, I'm going to have a model that predicts cars. I'm more and unless my mind is just too static it it's a foundational model, but just for the mesh design that you've picked and the CFD method you've picked. If I then go and change that CFD method and regenerate the driver ML with like a RANS data set, I'm going to get a different answer. So it's sort of hard to conceptually imagine the ML being like a foundation model because the, the input data itself is so dependent on the

49:58 settings you have of the CFD. So ideally it should really be like experimental date should be the, the, the real ground truth. But it's, it's sort of a little bit of a philosophical thing where, yeah, the ML model is only as good as the training data that it has. And so if now you've sort of solved the ML problem, well, I'm not saying you've solved it. It's almost the data is the most important bit now. I couldn't agree more. So, so I, I, I would say talking about foundation models for engineering is very, very dangerous because you're kind of data agnostic in a way. You in the end, what goes into these models is a huge amount of

50:44 knowledge in this data generation. And as you said, data on numerics is not the ground truth. It's just one version of, of, of our reality, right. And, and machine learning is not replacing whatever this, this version of reality, it's giving a certain new access to this reality. So I, I think the power of machine learning comes when you, when you really pick certain areas where it's tremendously important to have this surrogates and to have this vast iteration to have whatever and then really think what it takes to build those surrogates. But I think from a modelling perspective we are especially in this static systems, we are

51:27 pretty far that we can give them enough data and high fidelity data and so on and so forth. We can build these systems with reasonable errors compared to the data is trained on. The question is then always how to interact with numeric. So if depends on what you want, but I think that the power and the true transformation comes when when numerics and CFD really go and deep networks really go hand in hand and not as as two different parties. Maybe I should also say to that we were doing this drive ML data set now for I don't know 10 months like quite intensively trying different things. We've reformulated the learning problem ourselves a couple of

52:09 times. So what we actually want to train, what we actually want to test and so on and so forth, because it's really, really hard to because it's, it's just not a replacement of the CFD. Your model is just doing something different and you have to frame that there's a problem. And and so that's why I think these two fields need to come closer together that these questions are answered. Yeah, no, I and I was just looking off to the side because I was just double checking if I was correct that the error five data set is a combination of actual satellite data, right observations with some sort of modelling. So it is quite different in and I think that's kind of my point

52:53 that the weather stuff has literally got what the satellites is seeing. Therefore you could argue is the real thing because it is the Earth. They saw what the weather's doing on the Earth, whereas in the CFD it's, it is a, a just a numerical simulation. And, and so that's the bit that I still think is a bit of the limitation, but we are so sparse when it comes to data. This is a bit of a segment into a next question to you, which has long been the argument of the pins people, which is are we ever going to have a vast quantity of data to train sort of data-driven models? Is is that a reasonable

53:44 assumption or is it better to say, well, actually we have pretty well defined equations and rules and models and is it not just about using ML to solve them faster? Like do we need, if we have such a a sparsity of data, do we not need to include physics into the models? Where do you sort of lie on that conundrum of just generating more data and let the models of the data or do we need to incorporate more physics into these models? I'm very pragmatic in this. I mean this, This was the same in weather modelling. Everyone was talking about physics informed, not physics informed. But people only talk for, for, for that amount of time until

54:32 they realise how heavy the data loading is and how hard of a machine learning problem that is. Because you don't want to talk about the physics in form neural network if you have to, to check it between loading gigabytes of data for one data point on the GPO and what to do. And in order to get this correct and in order to get the decent learning signal and so on and so forth. And I, I see it the same for this engineering simulations. We first have to be able to really run simulations. It's scale. And I'm not talking about 10,000 cells. I'm not talking about shape net, I'm not talking about even driving at the drive ML.

55:09 I'm talking about airplane simulation and I'm talking about transient simulation and that. And we are still far away from doing that both from how we generate, how we store, how we train data. And I think only if we, if we are able to really have a workflow there and know what it takes and know where it breaks and, and, and where all these problems are, we can then start thinking about which type of physics to include and and which not because, I mean, we, we also showed that in our paper, it's very easy to build a diversions free model for, for vorticity, because you can bake in the, the diversions free constraint, their construction a hard

55:51 constraint. We try the pin loss for the, for the mass conservation, which is much harder to just take the great performance. It in principle works. And you would need some tricks and I would say getting this to work is not the amount of time which needs to do all the scaling. But I would say that first the problems we have to solve first are the really how we interact with numerics, how we scale to these problems and then how we actually get industry ready. Well, you, you mentioned the beginning, the sort of T to T + 1 analogy when it came to weather. And clearly the bit that we missed out maybe in our introduction is that the

56:35 engineering, we're not doing that right. You're going from yeah, straight to a time average or something at least, at least in, at least in, well, in in the sort of way that most of these data sets for cars and planes have been to date. Given. How would you change the learning problem or the ML architecture if now you had a a full time history and would that help hindrance not make a difference? I mean same argument as as we have in in the spatial. If you if you look at the the neural network which has a few megabytes of weights and which compresses, I don't know the, the GB CFD simulation or 10 or 20, I don't know how many GB

57:22 actually CFD simulation, you can also play the same game over time, right? So neural network is just a very, very good way of of storing this, this information. And if you somehow manage to to store the temporal part in, in some sort of modelling, which allows you to give you the the temporal answer to a certain problem, this would be tremendously huge gain. Because suddenly you can really simulate stuff in, in, in, in real time, but also have the the ability to observe what's going on and, and, and to really understand the temporal dynamics. Whereas the training is just somehow pumping this simulation into the neural network weights, ideally without storing them.

58:06 But that's that's very, very hard. And then probably the next to, next to next to step. But yeah, I think first step is to get to get the decent neural network architectures for temporal modelling, which we are actually working on to be honest. OK. And, and would you have the sense to be something similar where you know, you may want to have a million time steps, but that you'd expect that there would be some way of, I don't know, is there, is there a sort of equivalency in terms of AD inference? Ideally you wouldn't need to go 1,000,000 steps, right? Do you have a sense of how much this is more question for people

58:59 who are generating data, You know, do I need to generate a million times steps of data so that you can learn the million? And do you expect that then you would be able to predict to the millionth but skip out every hundred? Because you, you don't need to, you know, to do that? Do you think you could learn the temporal problem without every single time step from the traditional PDE? Do you know what I'm trying to get a sense of? Like the how the temporal challenge would link to the spatial challenge? I think it's it's very much relatable. So in in space we see this phenomenon that you don't need all the points to to to capture

59:37 the physics and then depend. Then you have a huge advantage in learning because you can always sub sample different sets of these points which give you a huge data augmentation. And you have the same in a temporal domain. You don't need each time step in order to to present the full physics to the neural network. But obviously the the fine grain, the more the more fine graining you have in your in your solution, the better you can do the data augmentation in the temporal dimension. But obviously machine learning models can do much, much bigger time steps than numerical models. And yeah, this is something which which is a true advantage. They just don't work the same

1:00:17 and they don't have stability issues but also no stability guarantees well. I was going to say, isn't that one of the problems, the roll out problem that, you know, I guess if you do inference inference, inference imprints, you know, isn't, isn't these sort of instabilities going to build up or like, you know, how do you constrain it? Is this where some of the physics needs to be added? Do you think to sort of if you need to do 100,000 iterations, how much are you going to guarantee that just a small difference in that first is not going to, you know, bifurcate into different solutions? Yeah, I mean, there, there is 2

1:01:01 to answer. So that one is obviously you. Luckily people do video generation nowadays. So we get a lot and lot of tricks of video generation, which actually really, really start to make this thing stable. And one of these, these definitely modeling paradigms is this generative model that you model the distribution over time and that you kind of make sure the distribution always stays the same. And, and this boils down to a bit of, of physics understanding, right? If you, if you make sure the distribution is the same, if you make sure that that future and past they considered that there is a tension across the right axis and so on and so forth, we

1:01:35 get stability. It's always depends on how you define physics. For me, it's the information the model needs and the way you, you, you, you interact with future and past and so on. I truly think that this generative modelling is there's the breakthrough in this, this rollout stabilities. I remember when I ran first into this rollout problems, it was I think 4 years ago and, and, and basically I, I did this paper together with Daniel Worrell and I texted him and saying, hey, I don't know, we have to do something. Our rollouts always exploding. And then I have no idea. And, and he said, well, yeah, there has to be some, some tricks in literature and then

1:02:12 something smart and, and, and definitely people have thought about it. And I was like, yeah, I cannot find anything. There is like, why is nobody having this problem? And it's basically just a very different problem we have in in in this engineering task that the the input itself each time step is so huge from information content that every autoaggressive model, every every autoaggressive trick you you had before is just not working because they work with much smaller input vectors. But but now that computer vision is going to videos and that people are aware of this problem and that frequency spectrum need to be conserved and so on and so forth.

1:02:51 This guess really a lot of lot of progress. And I would say that that the having the frequency spectrum stable is is is what made many of these things stable. So it sounds like in a way there is still a lot of potential to learn from the other advances. Yeah, in ML, you know, the fact there's always an analogy problem I guess is what you're saying. There's the stuff you can learn in terms of video generation, text generation, weather, climate week. You can borrow ideas from other fields, but it suggests that you therefore need to have people with a a broader ML background. You know, if you just have a fluids background, you may struggle to solve this problem

1:03:39 because you need to know what or you need to have an understanding and an interest in looking at what's video generation doing or what's this doing. Would that be fair to say? This is why it's a multidisciplinary problem. Almost all you need a team that is multidisciplinary. I mean that that's happening in biotech and in other areas as well, right? We are just very in agnostic to that and language and vision because you don't need a linguist to build the LLM and you don't need basically, I don't know even what the computer vision like. What's that analogy there? But for, for, for, for all this alpha fold and so on so forth.

1:04:18 The surely had domain experts on the team and surely this is what, what's the exciting part about engineering you, you have problems which are so hard to correct that you need the top notch machine learning people that the top notch domain people, they need to speak the same language. And even if you can borrow a lot of concepts, they need to be heavily adjusted. Because the problem with this large mesh is with what's the input, what's the output? There's this huge ratio difference and, and, and, and turbulence and whatnot. This is just a, a totally different piece to, to crack. So that's what's the exciting part here. I, I wanted to maybe pivot it a

1:04:55 little bit to the process you went in creating a start up. I, I think a lot of people would be interested to know and I'm sure other people, you know, are in a pub or, you know, alcohol doesn't need to be involved, of course, but you know, there's somewhere and they've got ideas and, but not many people actually go out secure funding and, you know, create a start up. So obviously not giving away any sensitive details, but what was that experience like for you? Like how did it begin? How did you deal with venture capital companies, how to like? Yeah, I'd be interested if you could share some of your experience in founding a start

1:05:37 up, some of the lessons learnt maybe. Yeah, so I mean, I, I was really lucky. So, so first of all, a lot of knowledge comes up from my university. I, I have to know, I have to say that the people like Benedict Alkin and and Tobiascon Lachner who are really, really pushing the efforts, they have been with me from, from day one actually. Also people like Stefan Berka here in Linz University is a well known simulation expert who who knows all these tricks. When I met him the first day, basically after returning to Austria, we were 5 minutes and we were both like super excited that, that we do things in this area. And yeah, we, there was a lot of

1:06:16 momentum there. We were at a different company and XCI where at some point it was clear that the simulation team, which was was really working well, should branch off. I was also lucky to to have met people like Dennis Houston mix Mickelson's who who did this before graded startups, very successful startups and actually knew what to do. They were the first to listen to me when I told them, hey, this engineering like scaling these things up. What we do to this billion of mesh problems is actually what what industry needs. But just give us some time. We will figure that out. And so I think it was all very natural. It was not planned, but it's

1:06:56 obviously a huge excitement if you can, if you can do what you're really burning for in, in, in also in a, in a commercial set up where you you really can make an impact with what you're doing. So yeah, I would say in that sense I was very lucky. And I would also say that especially in this area, we, we see a lot of momentum generated and we're not the last start up and there is hopefully coming more it, it, it there is a momentum especially in Europe now in in this area. And yeah, just very exciting. I would do it exactly the same way again. And so how do you think you could have done this any other way? Do you think you could have come

1:07:38 up with what you're coming up at a company or at the university or is it is this is the start up really the only way of I asked this question because I asked the same thing to Max Welling and he was sort of saying, well, yeah, start-ups is the only place where this can happen. The university's either too small and doesn't have enough funding or companies too big and too slow. And the start up is this sort of, but it feels like in Europe, but often a bit risk averse. We almost feel like start-ups is for somebody else. But I'd be interested to know your thoughts on. That I'm not so sure if what we're pulling off. We just need the, the, the best people and, and, and, and some

1:08:22 compute. One thing which is really, really, really, really hard is the problems. So it's actually we are choosing our customer mostly by the, the, the, the scale and size of the problems they have because they're really interesting problems come from industry and the more challenging and those problems are, the better they are for us and, and for developing us. So obviously we now have have a customer which have huge temporal problems where you have, we have different modalities interacting and this can bootstrap our temporal modelling capabilities and and and and and and and and, and, and this is something which you wouldn't get in the industry I

1:09:06 at university. So I would say it's a mixture. I would say for me, it's, it's, it really helps to get hands on to the, to the real problems and, and on, on the other side, it's, it's just helps me to get into touch what industry really needs and, and not sit in my ivory tower. I wouldn't, I wouldn't stand doing things where I know people wouldn't use them. So this is a good, yeah, good mix. But I also have to say there is luckily some people like Neil Ashton who generate publicly available data sets that that people can really start looking at that We just have to give the community the urgency that that they really look at the hard data sets and not a shape, no

1:09:50 car. Yeah, it, it seems like the there is definitely a movement in the industry towards this machine learning problems. You know, every time I go to a conference, there's more and more people doing it, but it still feels like we haven't reached that inflection point. It still feels like you're either a start up or you're, I don't know. It still feels like we're not fully at that yeah, inflection point in terms of mass adoption of this. I feel like we're getting closer. I'm sure you feel the same. You know, lots of industries interested, but maybe it is the data that's the issue. Maybe it's the Era 5 was such a

1:10:39 large open data set that data wasn't the issue and therefore people just got into the modelling. Whereas I feel like now there's been quite a lot of advances in let's say Rd. car external aerodynamics. But Rd. car external aerodynamics, I don't know percentage wise, but it's it's a small percentage of the entire CFD domain. And yeah, yeah, it feels like we still need to convince maybe the whole CFD community to do more, to really advance it and make this a more systematic, I guess. Would that be true to say that you can only work on the problems that you updated for? I mean, it sounds like an obvious statement to make, but I assume if you had 10 times more

1:11:24 data, you could potentially go and hire more people and work on more problems. But you're not going to go and hire more people and work on other problems if there isn't the data or the willingness to do this. And, and most people also don't know what to, to optimise for, right. So, so I think the, the, the industry or the, the, the real problems need to need to set the, the, the, the, the, the stage they need to set the, the machine learning problem. Then we can start to think of how to solve it. Yeah, I, I, but there is a momentum. People are still reluctant because getting a nervous paper easier on a smaller data set

1:12:00 than on a larger data set. But but I think there is quite some movement and and and and things are are really changing. And I also from for me, it's, it's kind of our philosophy that to be a bit open source to show our models to show what we're doing to publish, because I think this is this is the way to go forward. Closed, closed doors policy is is is not, not what what what gets progress. I mean, we we see that the whole progress in LLM due to open source and then and so on and so forth. And I think this is also what will, not what should, but what will happen in in this space. Yeah, I must say that I do commend that you're releasing the data and the models.

1:12:48 Open source is is a novelty. And I think, yeah, it's good to see because I like particularly your paper now. I mean, I think that has hurt a little bit of trust in the CFD world so far. Instead, as you know, the tradition and I'm don't sure where the tradition came from. If you develop a turbans model or you develop a numerical method, whatever it is, it's almost always published. It's very rare actually for a model to make itself into a large commercial, even, you know, ISVCFD code and not to have a corresponding publication. It's sort of seen as like a thing that you don't make money on the method, you make money on the code.

1:13:38 That's sort of been the like thing like open foam or star CCM or anti fluid to others, typically the models themselves. You sort of know it's more the the the little tricks that you might do and but then more the support and the code and all that. Whereas it felt like so far in the ML4 CAE world, with the exception of a a couple of individuals or or companies, most of the start-ups have not published or made a thing of saying, here's our exact method, but we've implemented it in such a way that, do you know what I mean? And I feel like that's hurt a little bit of trust because it's hard for someone to believe results if they don't see a publication about it.

1:14:27 So yeah, I don't know if I'm alone on this, but I think your strategy of publishing more, I think actually helps at least CFD people to feel a little bit more like comfortable. Which I, I, I think it's the, it's the only way around. And I if I make a bold statement, if in the machine learning world where things are moving so fast you're afraid that others are are copying your stuff and overtaking you, you should probably reconsider your your capabilities as a company. Yeah. And the only thing I would say though, which is why I've asked you sometimes to up level the explanations, is I feel maybe even the biggest barrier is just

1:15:06 an understanding barrier that, you know, if you're ACFD person, you're probably going to read an ML paper and be like, I don't really understand this. That's almost the challenge. You know, it's like an education thing is that the people making decisions or higher up may not even understand because it's such a different field that that is the challenge. So anything that you can do to make like, yeah, simpler versions of papers or like summaries of papers or, or education, which I guess comes back to why having a dual affiliation between a university and a start up is probably a good thing. Because I feel that you need to

1:15:46 be teaching and producing like open source teaching material to educate people as much as also producing commercial solutions. I don't know if you've noticed that when you speak to customers that sometimes they're just knowledge of ML is part of the blocker. Yeah, but it's also the other way around, right? There's hardly any people who have the knowledge of the real domain expertise and knowledge of ML because only if you have the knowledge of domain expertise and you know what ML can do, you really find the problems where ML can really have an impact, right? And, and and this goes in both ways. So education both ways is super important and it's very, very

1:16:27 hard. I mean, I, I see that myself all the time that I can get lost in details and, and, and think things are obvious. And on the other hand, when I talk to real domain experts I get get really lost and have to ask questions and yeah. Yeah, yeah. Well, maybe This is why it still will take a few years to reach a greater maturity because both sides need to upskill themselves and, and, and sort of learn. So great. Well, really appreciate you chatting. I mean, what I'm going to do is put some links in at least on the YouTube to, to a couple of the papers and the models that you that we've referred to, because I think people should

1:17:05 really spend a bit of time looking at the papers. I have a feeling that, you know, we might need to chat again. I think this field is moving so quickly that everything that we, it'll be interesting to see if we listen to this conversation, a year's time, are we going to be completely wrong or in two years time? I very much hope so. But what we won't be wrong is that some things are moving fast and that we will have fast progress in two years. So we will be, oh, we didn't expect that to happen, so. Yeah. Yeah, that's going to happen. That's a good thing though. Maybe that maybe that's OK. But yeah, for now, then, until we speak again, thank you very

1:17:41 much and yeah, hope people that enjoy looking at your paper and your work. Thank you very much, Neil for having me.