Prof. Max Welling — Machine Learning Pioneer and AI for Science Visionary

0:00 Hi and welcome to the Neil Ashton podcast. In each episode, we explained some of the fascinating ways that science and engineering are changing the world around us. We talk to leading engineers from elite level sports like cycling and Formula One to some of the world's top academics to understand how fluid dynamics, machine learning and supercomputing are bringing in a new era of discovery. We also hear some of their life stories, their career advice and lessons they've learned on the way that I hope will be helpful to you too. So sit back and enjoy this episode. Welcome back to the Neil Ashton podcast. Today's guest is Professor Max Welling,

0:45 one of the foremost experts in machine learning. And I was speaking to him in this episode about AI for science, the use of machine learning and AI to solve some of the biggest challenges in science. Now, if you're in the machine learning world, you'll already know who Professor Welling is. But just in case you're, you're not, I'll just briefly give his background of why he is such an expert. Well, as we actually discussed in the podcast cos I I always like to ask people about their academic careers and how they got to, you know, where they got to, uh, he started off, you know, the, the standard academic track, I guess,

1:20 doing a master's and a PhD. Uh, and then as he said, he actually did many postdocs, um, moving between different universities. I, I guess, most notably he was with, um, Geoffrey Hinton who is seen as the sort of godfather of, um, of AI at U of T. Uh But then he went over to, to the US uh to UC Irvine and he started being a professor there. Um And then he ended up actually going um and being, being, yeah, an assistant professor there and then coming back to the University of Amsterdam to become a professor and he's actually stayed there uh pretty much ever since. But one of the interesting thing is that he has taken routes into industry and,

2:00 and start ups. So he was a VP at Qualcomm. Uh And actually very recently, he was also a VP and distinguished scientist at Microsoft. He recently left um that position and to start a new start up called CuspAI, which is working on machine learning for material discovery. In, in the topic of carbon capture, we actually go into what his startup's about what they're doing, why he's interested in this area. Um I probably could spend a long time going through all his achievements. Um But he, he's one of those people who is, you know, at the forefront of, of machine learning. So he's been the people in charge of all the major conference like

2:42 NeurIPS, ICML and ICLR conferences. Um He has been leading, if people haven't seen that, I would highly recommend that you do look some of these workshops on AI for science, uh, particularly at NeurIPS the past few years. So these bring together all the world leading experts on, you know, uh machine learning and its and its application to science and bring them all together in workshops that's has been really good. Um And what I like about him is he has that unique experience that we, that we touch about being not just a pure academic or being a pure industrialist, but someone who has jumped between. And I really do believe that that gives him uh somewhat a unique insight

3:23 into not just the science but the practicalities of how do you go about it and that's exactly what we try and discuss a little bit in this. We purposely keep it reasonably high level. So we're not going into the super super details but rather looking at the the big topics. So for example, we start off by discussing data driven versus physics driven approach. It as in do you need to include the physics, physical equations into the loss functions into machine learning? Or can you just let the data do it? We talk about uh foundational models as in, could you create a foundational model, the science that you can just use or, or just, or just fine tune.

4:04 And then we pivot a little bit and what I think anyway, at least have some interesting discussions around. Should you open source the models and the data? We have a good discussion about the merits of that um ethical considerations, et cetera. Uh And then we get into the different mode of operation, what the advantages of academic industry and start ups in terms of reaching these benefits. Um We talk about his start up about CuspAI we talk about what they're doing, why he's doing this, his excitement and motivation to go back to a start up. We briefly took on some of the funding, you know, what's it like getting funding from VCs in Europe versus the US?

4:41 Uh And then we, we sort of finish a little bit discussing career paths. He goes into more details about his personal journey through academia, through industry, his advice for people now. Uh and finishing really on what his uh thoughts are on the future of machine learning and how this could help scientific discovery. So a pretty wide ranging uh just topic like anything we probably could have spoke for many more hours. But I, I certainly learned a lot through this conversation and I, I hope you enjoy it too. So uh sit back and please listen to this episode with Professor Max Welling. do we start it now? We have? All right.

5:20 Uh Yeah. AI for science, I think, um, is the application of the tools that are being developed in machine learning and AI um to basically help the discovery process and the analysis process in the sciences. Um The interesting thing is that there's also signs for AI in a way which is the, you know, using principles from the sciences and the mathematics such as um you know, symmetries, um diffusion processes like thermodynamics to build better machine learning models. So this thing actually go goes two ways, the causal direction is going two ways. Um Now, what we see is that in the sciences um in in a very broad

6:12 spectrum of the science is going all the way from the very small uh you know, PICO secons uh femto meters, which is basically uh high energy physics where a lot of machine learning is u used to do data analysis all the way up. Uh you know, let's say going through molecules and liquids and fluids and then all the way to sort of earth sciences and maybe going all the way up to uh astronomy um which is the at the scale of the universe uh machine learning mo um models and methods are being used to, yeah, model that particular domain, but also analyze the data better. So it it it has been having a huge and very broad impact in the science as I would say,

7:03 and how much overlap is there, would you say between the work and the methods that have been used to develop, let's say some of the large language models and those which are in the sciences, is it a matter of applying the same tools and technologies? And it's just essentially a different focus area or do you see that fundamentally there are differences and new approaches that are need to be developed. There's a surprising amount of overlap I would say um which I find interesting. So the fir the first thing to say is that even within this incredibly broad range of um scientific disciplines, of course, the tools that are being used,

7:45 the mathematical tools that are being used are often very similar too. So you start from you, you know, it's either a partial differential equation or stochastic differential equation or ordinary differential equation. That's because physics is causal, you know, you try to predict the future from the past. Um It's typically continuous but not necessarily, but many, you know, processes are modeled continuously like fluids a me and it's local, which means that you, you typically don't have something at a very far distant place, instantaneously impacting things here. And so these three aspects make the tool set typically quite similar,

8:28 even even across all those scales. Um So now to the question, um some of these tools which are being developed for image generation um perhaps more than for large language models. So they're actually different tools, although they're all being, you know, put in the same bucket typically as generative AI but actually quite different. Um But especially the methods that have been used for video and and image generation, which you can actually somewhat think of as a physical object with pixels being microscopic degrees of freedom which have some dynamics. A me. So the tools and diffusion models for instance, which is the method that generates all

9:13 these beautiful images and videos these days, those can be almost literally used to generate molecules um and fluids. So um maybe not surprising because of fluid, you can think of it as a sort of a set of pixels or it's typically modeled as a set of pixels, you know, it evolves over time as well. Um For molecules you could argue, you know, those are, you know, it's more like a graph uh where the nodes are at certain locations. Um So maybe they're slightly more surprising, but it is almost literally the same tools. In fact, we developed uh what's called equi equivariant methods which incorporate the symmetries of the world,

9:55 which is that a molecule or an image. So it may be best for an image if you look at sort of a histopathology slide, which is a slide which has all all these kind of stain cells in them and some cells might be uh have a tumor in them and and other ones don't. And your task is to find the cells which are malignant. Um So, you know, if you turn the slide upside down, you know, you don't even know. Right. It's just the same set of cells, but this rotate a little bit and, but a neural network doesn't know that usually. And so we've built models which were called equi equivariant models, which build in sort of symmetries.

10:34 So for instance, you know, those methods, these symmetries which we've developed for fission very naturally also transfer to but back to physics. So they came from physics in some sense and we turned them into tools for, for, for images. And now, now, now we apply them again to molecules, right? And so a molecule upside down is the same molecule, a molecule displaced from here to here is still the same molecule with the same properties typically. So that's why these tools are very useful, you know, across the board. But one of the um topics that seemed to be very much an active debate in at least in the machine learning for

11:16 fluid dynamics, which is maybe what I'm most familiar with. But I, I'd be keen to get your opinion on this and more broadly to other scientific disciplines is this sort of physics driven versus data driven. I see on one side, the logical argument there are physics laws surely, you know, the solutions should bound by them. But I I saw you gave a talk recently and you were alluding to the idea of there's a sort of cut off point where if you don't have much data, maybe it makes sense to use them. But if you have a lot of data, you may even be constraining the problem and, and, and solving it. What, what are your current thoughts on this debate around

11:54 physics or data driven approaches? Yeah, very similar. So I, I'm still, I agree with my older self. So that's good, at least some level of consistency. Um But uh so, for instance, uh you know, when I was still working at Microsoft with a wonderful team there, um uh we worked on models for the atmosphere. Um And this was recently published as a paper called aurora. Um So there, there is a lot of data. So it's like a uh petabytes of data for because, you know, we predict the weather every day, every hour. Um and we store all that data and then of course, we know what the actual weather was, at least we have measurements.

12:38 And so it's a very rich um and outdated set of data in some sense. Um in that domain, you can basically let go of all the laws of physics. You can just say, let's think of this as a big machine learning problem. We just try to predict the state of the atmosphere, you know, a day ahead, you know, 10 days ahead, whatever, right? Um And you can just use your transformers and everything that you use for image and videos as well. Um And that worked really well. In fact, it's, it's basically as good as a me numerical soul verse. I should be a bit careful saying that. Um But it's as good um with if, if the, if the prediction stays within, you know, the,

13:27 the the sort of the input that it was trained on, if it goes far beyond that, you have to be much more careful, of course. But the big advantage is just it's way, way faster. So it can be up to 10,000 like four orders of magnitudes faster. Now, there's other domains where this is much harder where you don't have this huge amount of data in fact. And in that case, um you should really build in um the what we call inductive biases that are coming from the laws of physics. And, and I've seen some interesting examples where, for instance, we know the continuity equations should hold typically, if you write down sort of the Navier–Stokes equations,

14:09 many of them have to shape of some kind of continuity equation. So that, that's the, the partial differential equation that describes basically a velocity field over which the fluid moves um in three dimensions. And then there's a source term basically which you know, applies extra forces to that. And um and so um sort of if you do it with the transformers, you know, it has to learn all that. But um it makes a lot of sense to say, well, we know the world is three dimensional and, and, and it has this sort of, and we know how fluids move around. We just may not know precisely what the forces are that, that are acting on them.

14:47 And so we learn the forces we, but we constrain it to be a fluid that moves around uh according to the laws of physics. And there you know, if you have little data, then that's a very good prior. Now, the flip side is there's always a flip side. So if you, you know, we don't know the precise laws of physics at that scale, even for the weather, right, we have written down or we, the demetri ologists have written down and I don't know order 1020 equations which describe all the complicated interactions of the, you know, the atmosphere with, you know, the surface with, you know, the radiation with chemicals and all these things.

15:29 So everything is written down in these equations, but these are approximations. They are not actually, you know, we we might know the laws of physics at the very microscopic level, you know how one atom sort of interacts with another atom using quantum mechanics. And all that, we, we know it very precise, but it's completely hopeless to simulate the atmosphere at that level. So we have to Coors grain is what we call it. We have to go to a much more um sort of a less precise level of description. And at that level, things become approximate. And so if you impose more and more laws of physics, which is, you know, you've approximated

16:08 at some point, uh you might have so much data that, you know, you, you sort of build in a ceiling to this modeling, you say I want you to fulfill these constraints. But, but if these constraints are approximations, um at some point, that means that you cannot get out of that box. And if you have so much data that the model can actually learn something that's more precise than that box than those constraints you're giving it. Um Then you're constraining the model over constraining the model and that's, that's hurtful all of these, at least from what I can see most of us, when we use a large language model, we never need to train the model.

16:45 We are just doing an inference essentially on a pre trained model, a foundational model. And most of us never need to, to train it ourselves. Whereas if we take a fluid dynamics or, or a weather problem, do you think that we will ever be able to get to a truly foundational model which, which is trained on such massive data that it, it could work for a plane or a car or a building or the weather could be done in, you know, different parts of the world or anything or, or is, is science a problem that's just much more challenging than um than what we've done for the large language models to date. Yeah, it was a very interesting question, I think on the one.

17:29 So to me, this is kind of a, I think it does make a lot of sense to train a very large foundation model. And this means plenty of evidence now that if you collect data from diff from very different domains, different regimes um training such a pre training, such a very large foundation model um is a huge benefit, especially if you can then fine tune it to the particular problem that you're interested in. So with a little bit of data and a little bit of fine tuning, you can get a very good model. So I think that's a paradigm that has now been established. And I think that's, you know, I think that that that will just work for the foreseeable future.

18:07 Um But there's also this thing that um which are called the 8020 rule, which is that it's 80% of getting it right is quite easy, you can do it with 20% maybe of the effort, but then the last 20% is incredibly hard. Um And you may be you may you need a lot more effort, right? And so I think you can see this very clearly for self driving cars, right? So in self driving cars, you know, people get very enthusiastic for the 1st 80% because, hey, we can drive a car without hands on a, on a street. How wonderful. And then you realize that to get it to drive inside Amsterdam, um, you know, that's not gonna work with that first,

18:49 you know, with this first, you know, model you have. And so you now need an enormous amount of training, but actually probably just rules based methods as well. You know, to make this thing drive safely inside a complicated city like Amsterdam, maybe San Francisco works. I don't know, the streets are all rectangular and everything, but in Amsterdam it's all big chaos. Yeah, people run red lights, bikes everywhere, you know, it's like massively complicated. Um And so it will take 80% of the time maybe to get this long till of issues correct. And this is where humans really are very good somehow, this has to do with this generalization.

19:26 Somehow we do this very naturally and very good. Now, in large language models, I'm sort of seeing something somewhat similar, right? Which is we can get amazingly good at, you know, and, you know, I didn't predict this to be really honest, you know, to, to build these, these um these chat bots, but it is, it is uh perhaps 80% of the way. Um Now the last 20% is you should get it such that, you know, you can't trick it into saying all sorts of rubbish, like, you know, put glue in your pizza or eat rocks, right? These are the kind of things that are big tech companies are now struggling with because, you know, if you put this out there,

20:10 people will try to, you know, to game it. Um, and that's very negative publicity and all that. So, so this last 20% to make, to make it really understand common sense. Um, you know, it's, it's arguably whether this even can be done by just looking at text, maybe this needs to be some system that grows up among humans, you know, with the body and sort of interact with humans and understand social interactions and, you know, ethical rules and all these kinds of things. Maybe it's just, you can just learn that from text and that last 20% might be very, very hard. And so perhaps in, you know, in physics, it's the same, right?

20:48 So, you know, you can get it, you can get it right? 80% and that hopefully is useful. Um But then how do you protect yourself against that 20%? You know, an unseen initial condition with incredibly important consequences like OK, it might just, we might just get a heat wave, 50 °C, you know, do we, you know, do we warn the public um or do we not because we, we don't know precisely what's going on, right? And that I think is incredibly important to get Right. And we haven't really get our heads around how to do that. And I think uncertainty quantification and reliable, uncertainty quantification is going to be key for that.

21:31 Very interesting. I haven't really thought about it that way. It's true. The 8020 and now I think about it, how I use these large language models is I very rarely take the exact output. It gives me, I normally will iterate on it, but it enables me to get somewhere much faster, I suppose maybe from the physics side, we're expecting it to get perfect where really maybe our expectation should be more similar that it gets us someone closer and maybe then we feed that solution into a traditional solver precisely to do the next bit. Um And its initialization like a smart initialization or something. Yeah. Yeah. But all of this,

22:13 I still wonder about the data challenge because it seems because of the internet and maybe things have been clamped down, you know, so many images and so many text are publicly available that you can get. Whereas with science, it's very rare to have so much of that data in the public. To me, most companies have that behind firewalls. Even people publish papers, but they very rarely publish the entire data that created it. So how can we get around that issue of of the data? Yeah, that's the absolutely key. So that's the, that's the key for progress I think. So, so you can see that the domains were, you know, researchers have managed to do a lot more data sharing

23:02 or whether it's just more data available have progressed fast. Um certainly, you know, with machine learning tools. Um But for many other domains, uh companies are hoarding that data or even universities are hoarding their data because it's expensive to generate and they want to, you know, milk that cow for a couple of years in terms of papers before they give it to others. Um And so that is really um slowing down science and progress. Um There is, there is great initiatives where, you know, for instance, in, in bio where people aren't sharing a lot of data, there is a lot of omics, data available, et cetera. So there's, there's also good examples,

23:42 there's some materials project for instance, and other materials projects um around the world where data about materials are being collected. Um But um I think in the end, we will have to invent the technology that makes data sharing much easier. And, and I, you know, would envision some kind of marketplace um where people have their data behind a firewall. Um But um if I want to train a model, I just go to your, your data source and I'm going to say um I'm gonna pay you whatever amount of money to access your data source. Now, um note that I will do it in a differentially privacy private way which is which is to guarantee that

24:31 I'm gonna update my model parameters, but I'm gonna do it in such a way that I can never reconstruct data that were, that were, that were used inside your data source to, to build my model now, that can be guaranteed. But that technology also needs to be improved. So that means that, you know, you know, your agent and my agent, they negotiate a bit about the price and then I enter, you know, do my model updating and then I move to the next uh you know, data source, right? So now we can train really excellent models. Um but you know, there is kind of trading going on about how much that is worth. In fact, you could do it for your own data. So I could say

25:11 I've collected all my data over my lifetime, including my medical data, whatever I'm, I'm quite happy to help that, that hospital, let's say for free. Um But when Google knocks on my door, I want money because I want a fair share, right? And so basically this means or actually for, for open AI kind of, you know, ChatGPT models, right? Just like they now sort of there's lawsuits now, right? So basically, you know, it's unclear, you know, what these models like, you know, let's so I think there's a recent lawsuit against some of these companies that generate songs from, you know, from different rights from songs that are being created by artists.

25:49 And so these artists feel, you know, I've created this content and as being, you know, and somebody else runs away with it and it's, and you know, where am I in this process? This is a fundamental problem that we have to solve now. And it would be much better if an artist, you know, or a person who collects data in any way or generates content or data gets paid in a, in a reasonable way or can choose to make their data available to somebody who wants that data to train their models on. This is a fundamental problem across the whole machine learning. And AI think that space that needs to be solved. That's fascinating. I

26:30 it seems very interesting and maybe complimentary but in some ways also at odds in that, on one hand, there is a movement of open source open data, open source. So the idea being that everybody should make their data sets available. But and I believe that some of the um weather forecasting companies who released those data sets are now debating whether the next data should be closed and not open because it's true that many companies in our published models have have, you know, great success, but the people who generate the data have no financial reward for that. Um And I feel that for the sciences, particularly most applications of science get into industrial use

27:20 where there is a commercial side to it. So just saying open source your data, well, they're probably not gonna open up their data. So that's a very interesting point about it. A way of a way of people sharing their data but also being able to make money and privacy because I can't see a way of, let's say fluid dynamics. Why would Boeing release all the aircraft data for train? Why would Airbus, why would anybody else do it? It's a competitive advantage, it's a competitive advantage, which is why I'm always wondering and then universities typically do more fundamental problems. So if you train the data on just university data,

28:02 it's only gonna work the simple test cases because universities are incentivized to publish pure research, typically not full aircrafts. Um Yeah, I think the Blockchain could play an interesting role here somehow, right? So you, you could imagine that you could sort of ii I believe there should be some kind of marketplace for data. So first of all, we have to realize that data is perhaps the most important, you know, uh what do you say, resource? You know, it's, it's kind of the the the the new oil on which these machines, you know, and compute, I guess, right? So it's, it's computing and data that these machine learning methods need

28:45 and we just need to put a real value on it. We have to say, you know, it's, it is valuable if you share your data or give your data just pushing people to put it open source is I don't think it's gonna happen because everybody will always, you know, do what's good for them and we just need to make the incentive structures correct. I think that's the thing. You don't force people to do things to make the incentive structures correct. So people who are open sourcing their data probably, you know, they work at universities or they work at companies and then the open sourcing works for them somehow anyway, because even companies who are open sourcing their stuff probably do it

29:21 because in the long run, it will benefit them as well. What is your general opinion of that open source though? Versus close source? It seems to be a bit of a debate in the community of it. Does it help science to open source? But then how do companies fund themselves if everything's open source and how easily it can be copied? Do you have a, a thought on the, the optimum approach that you've seen different companies over the years take? Um I mean, are you open to talking about open sourcing data or open sourcing their models models more? But yes, I guess also, well, we maybe we just discussed the data, but how about the models themselves?

30:00 I guess the discussion is about whether it's dangerous to release a model, right? Um Because the, the model can be used for at the zeal purposes and is being used for a zeal purposes. You have, you know, you have to be careful that if you enable bad agents um to do horrible things. You know, you, you just have to really think about that. So, in other words, um if I would have the recipe to build in a lab, um you know, a terrible disease that can be, you know, unleashed on the world. I'm not going to argue for open sourcing that knowledge, you know, clearly, I just don't want that to be open out there. Um And so you could argue similar things will apply at some point

30:46 to machine learning models or actually maybe already, I mean, at some, you know, these models can be used for fishing attacks and all these kinds of things, right? So, so, or, or crime where you sort of um uh basically uh imitate somebody's voice and then you, you ask them to, you know, pretend to be your son and then, you know, ask to, to, um give some money. So you have to be extremely careful about, you know, when you open source something and maybe there just needs to be some kind of ethical board that decides whether a particular very powerful technology can be open source, yes or no. Now, within that, you know, once, you know, if you're within those bounds of safety,

31:30 um it's up to everyone to open source. It's, you know, it's a com you cannot tell a company to open source or not. Right. It's like it's, I think it's, it's not, not a very meaningful question to say companies should open source their stuff. It doesn't because of course they won't if it doesn't, you know, improve their bottom line. Um So it's up to them to open source, whatever they need to open or want to open source. But maybe a more specific question then is there a can methods move faster that have been open source because essentially you're outsourcing the development to a community effort or is it the case that

32:13 that sounds nice, but you can actually move faster by having an internal team with their own developers? Yeah. Yeah. So um so I think thinking very hard about the incentive structure. So if, if it's true that if you're a whole number of small play, you know, a small player cannot compete with a big player, right? So it does make a lot of sense to put a whole bunch of small players in a group, you know, uh share resources and compete with the big players. Let's say you all built one huge large language model uh within your group of smaller players and then everybody benefits from that large language model and now you can compete with the big players. So that make that's a clear incentive to

32:54 share in that group. Um Now there's also other reasons why people might want to share, which is like we're all stuck with a problem together, which is, uh, you know, climate change. Right. So for some strange reason we think that, um, we can pollute this earth, um, and, uh, you know, push carbon into an atmosphere which is a shared resource, um, but not pay the bill basically for doing that. Right. Um, we, we, we, we somehow think that's ok to do and, but I think there is a sort of ethical awareness within a lot of companies, big companies, big tech companies definitely who are saying, well, we should, we should all act together to do something about that, right?

33:40 Um So most of these companies have climate uh sort of um promises or what do you call them? Um sustainability, sustains and things like this. Um So to be carbon neutral or negative by 2030 all these kinds of things. So, so they, they're trying, although they're tied up in an AI arms race and so it's very hard to actually make those goals, but let's say that they're, you know, they're trying to do this. And so there it makes some sense to pool resources, I think and say, OK, let's be good actors in this world with good citizens and actually, you know, you know, release some of the data to the public so others can,

34:18 you know, help, for instance, a, you know, improve the technology to capture the carbon out of the atmosphere. Hm The um I wanted to maybe pivot a little bit um to, to, to talk about your new company, but maybe as a way of getting there, one of the things that I admire about what you've done and I'm always interested to hear the different logic is the role of academia, industry, big industry, like big tech companies and start ups. What, what can a university achieve? What's the advantage of a start up? Essentially? What, what, what, what drove you to create this new company? Is there a specific thing? I, I can just move faster by doing it as a start up versus

35:08 I'm interested to see how you've seen over your career, the sort of benefits of these three different places. Yeah, I find it very interesting because there really three modes of operation. I've tried all of them now so I can sort of sampled them a little bit. Um Now universities have a very special role of educating the next generation of talent that's really important and, you know, they use taxpayer money for that. So, you know, that's incredibly important, but they also do um so much more exploratory research, right? So they, they can, you know, uh if you like to think about the inside of a black hole, you know, fine, do it in a,

35:45 in a university setting. There's no company that will, well, there's very few companies that will fund that in some sense, basically. No, I think at this point. Um So, um but the impact is very diffuse. Right. So you, you, you write a paper, um, and that paper gets picked up maybe by different groups and they build on it and that's a very indirect, uh but important way to make impact and, you know, it's also, you know, of course, you, you, uh, it's also a different style of doing research and, and, and being engaged in intellectual activity. Um now the big companies and the start ups, you know, they can um command much bigger resources to really go after a specific target, right? So, um

36:29 no, but they are commercial, right? So they are tied to commercial goals. And so you have to really align the commercial goal with the thing that you want to achieve. Um Now some people like, you know, if you, if you like to achieve AGI, you know, great. So you can work for one of these big companies. Um that's a perfect alignment because that's what these big companies want. Do. They think they can make big money on that. So, you know, that's where you can sort of do that, that research and, and you can use the actual resources like massive amounts of GPUs um to achieve that. But big companies are also in a way slow and political

37:07 beasts, right? There's like, uh there's a lot of people that need to, uh you know, agree on something and, you know, a lot of legal involvement if you wanna change something small. And so it's, it's, it's, it's very uh fisc this whole thing talking about fluid. You know, it's like it's very fisc thing. So, um and a and a start, the great advantage of a start up is that um you're your own boss. So if you have a vision, you can execute on your own vision and you don't have to constantly calibrate with other people higher up in your organization, you just need a great set of co-founders and a great team to execute who are aligned with that vision. And then you can really go.

37:49 Um So to my surprise, um perhaps is that um the amount of resources you can get in a start up these days in the field of AI is no less than what you would get in a big tech company. So, in fact, um you know, if you can tap into VC capital, um I've had until now, incredibly positive experiences with this, which means that uh first of all, you know, these are very smart people, you know, of course, they, you know, they want, of course get return on investment. So there's no idealism here mostly. Um but they're very smart people, they help you find a market, they help you build a healthy business. Um And uh and there's lots of money available as well, right,

38:37 for that particular purpose that you want to pursue. Um And so, and so I think the agility and also the positive attitude that you find in start up land I really like. So I, I guess I've now converged on that particular model and I'll, I'll, I'll stick with that now for a while and, and just before we get into the Pacifics um of, of your company, is there any differences or challenges between, let's say, in your experience, us, Bay area, Europe, you know, other advantages, disadvantages in terms of finding the staff, finding the, the the money. Well, we actually do use uh American and somewhat and English VCs,

39:27 mostly there's also a few continental Europe, Dutch investors, but mostly it's uh we tapping into uh you know, um UK and US based VC firms, although our lead investor is actually is a UK based investor A me. So um so I don't think you need to make that distinction in some sense. The nice thing is that these big investors, they're turning their side to Europe. And the reason is that there's just incredible talent pool in Europe. Many of many of these talent pool likes to stay in Europe because Europe is just great, just, just fun place to be and the salaries are a lot lower than in Bay Area.

40:18 Um And so with relatively little investment, well, less investment than what you would have to invest in Bay Area, you can get a excellent team. Um And so it only makes a lot of sense that also these investors turn their eyes on, on Europe. So I think we will see a lot of growth and activity in these, in these sort of uh start up and scale ups in, in Europe in the coming years. I've often um I tease some of my American colleagues that why are they all based in Seattle? It seems like the most crazy time zone and position in the world to have such a, if you were going to invent a company to have a global reach, surely you wouldn't put it

40:58 there. I still don't understand how this, but yeah, something to do with ecosystems, right? So, um so, and, and that's really the thing. So uh if you build a particular nucleus or particular geographic area where, where it's lots of things are happening, right? In Silicon Valley, you have the big universities like Berkeley and Stanford, um which are providing a lot of talent and there is a lot of entrepreneurial spirit in those places and there's a whole, it think of it as a supply line sort of issue where there's a lot of, you know, other things which work for them. So there's a very small, you know, everybody knows each other in the VC.

41:39 So there's very, there's a lot of VCs which can fund these things and, you know, so I think it has to do with the fact that once you build a really good ecosystem locally, it then becomes very attractive to be there to do your start up because it's very easy to get good ideas from, you know, people help each other to grow these start ups. It's very collaborative atmosphere, it's easy. You know, if I need a new investment, I talk to a few people and they point me to other people and then you, you know, you before you know it, you're in front of new investors. So I think that kind of uh you know, lubricant in some sense is really good in, in these ecosystems.

42:17 So what we need to do in Europe is to build these ecosystems. And I think in Cambridge and in London, these ecosystems exist so they are very good as well. Um Amsterdam, Berlin, you know, some other places, Paris for sure. Um maybe Stockholm. So some of these places in Europe also start to develop these ecosystems, but we need to get better at that in Europe. Yeah, I I do definitely agree that there seems to be a lot of um a lot of very talented researchers and engineers and scientists in Europe that maybe feel a pressure to perhaps to move to, to the US where probably a lot of them would probably rather stay in Europe if they could do.

43:00 So. There is that I agree there seems to be a desire and if you could tap into that, you've got a big uh pool as and the other benefits of time zones and access to, to different markets. But yeah, I'd, I'd love to learn more a little bit um about your company. And, and maybe also, I think listeners to this podcast may be a little bit more familiar with uh some of the topics we discussed around fluid dynamics. So it would also be good perhaps for you to align how some, maybe the material discovery and material science has some overlaps and maybe uh differences to tonics. So the start up is about material discovery

43:39 um for carbon capture. Um And so we focus on uh uh um a sort of a family of molecules called uh metal–organic frameworks. Um These consist of uh sort of, you can think of them as a sort of a lattice structure with, on the nodes, you typically put some kind of uh you know, you know, small little molecule piece with a metal in it. Um And then they get connected by organic linkers is what they called. Um And so there's a huge design space, like many trillions of possible things that you can sort of even, you know, imagine uh with very different properties. And these um these frameworks uh if you blow, let's say air through them,

44:24 they tend to bind the molecules in the air uh to the to the framework. So for instance, carbon dioxide binds but hydrogen uh water, um sorry, I say water also binds nitrogen, binds. So all of these molecules bind to them and you want to find the uh metal organic framework that preferentially binds carbon dioxide um relative to water and nitrogen and all these other molecules. And then you want to do this at a particular temperature and pressure. Um And then you change the temperature and pressure. Um let's say you, you have less pressure and higher temperature that would sort of release then the carbon again and then you can sort

45:07 of sequester it and sort of put it away or reuse it for something. Um So, so that's the basic idea. And now we are going to use machine learning to accelerate the process a lot. All right. So we're gonna, we're gonna use the same generative models that are being used to generate images to generate new materials. Um And then we're gonna test those materials in a chemistry pipeline, but those, those are also um infused with machine learning methods to speed up this, this testing framework of how good this particular proposed material is. And then we have a search agent that tries to, you know, orchestrate this, this, this,

45:50 you know, searching through the space of molecules to find the best molecule with certain properties as quickly as possible. Um And you can, so how does this connect to fluids? Um Well, you a gas is not quite a fluid. Um but you can, you know, you could imagine actually a fluid in a, in a mo that's actually what happens. So you can sort of uh you can also put fluids in MOFs. Um But let's imagine a gas where you can sort of describe a gas but somewhat similar equations as a fluid. Um And, you know, there's sort of a, a, you know, an equation that describes how this gas would blow through this particular um framework and then, you know, and how it binds the molecules to the framework.

46:34 So that, that is not very unlike how you would uh simulate a fluid maybe um in, in some, you know, in some space with certain boundaries or something. And, and how do people do it now, you know, because there are companies trying to do these kind of, so are they relying on more traditional physics based simulations to do these discoveries? Yeah. So um there is already people who are um having these kind of pipelines. So uh they are mostly chemists. So it's, it's a chemistry pipeline or workflow um where um you know, there's a database of um you know, known um metal–organic frameworks and you push them through a chem chemistry

47:21 simulation pipeline that sort of simulates all the molecules as they wiggle around in this particular uh environment. And then you can sort of measure how many of them would would bind at certain temperatures and pressures. Um And that's, and that turns into a number that says how good this thing is. And then, you know, you have some, some sim simple way of searching through the space. Now, you can see that that can be massively improved with machine learning techniques. That's why we think it's, it's such an exciting field because they are not generating new materials with diffusion models. Um They don't use very sophisticated search algorithms either.

47:58 Um in order to um you know, to do, only do the simulations for the ones that are important. Um And you can even accelerate the simulation models by, you know, predicting the properties directly using machine learning models. Um or by what's called training force fields, these are methods that compute the forces on atoms as they wiggle. Typically, you have to do this with quantum mechanics, it's very expensive. Um but you could do it with a machine learned force field which is again way, way faster, many orders of magnitude faster than a normal uh force field. Uh if you use quantum mechanical calculations. Um but almost as accurate as that quantum mechanical calculation.

48:47 OK. Interesting. And what made you target this area? Is it more as the climate change sustainability? And is that a sort of a personal thing of yours that you've wanted to do? Cos I know you've had a focus in many areas of science. You know, I'm just wondering what made you focus on this area in particular. Yeah, it's definitely uh a personal uh sort of preference for me and my co-founder Chad Edwards a me. But also we think there is a big market for this because I think, you know, companies do want to be a good citizen and, you know, compensate their carbon emissions. And at some point, it might actually be enforced by government, which is what I hope,

49:31 you know, we need to tax pollution. Um And then having um a solution available to actually capture that carbon is, is there going to be a big market, I think as well? So it's, it's it's an alignment again between something you really want to do and work on with a commercial angle because otherwise you cannot build that company, that a healthy company that would actually make that impact. So that's the way I view that. Um But having said that, you know, there is many other directions in which this platform could be pointed even for metal–organic frameworks, um you can store hydrogen, you can catalyze, you can um um you can sort of deliver drugs, uh you can detect toxins, uh you can

50:20 clean water and that's only metal–organic frameworks, right? But then we can also go to other markets at some point um that are completely different materials. And how does it feel? I think you alluded to um what many people who work for large companies feel, which is the benefits of working for large companies, the, you know, access to a great talent pool, but maybe the slight slowing down and sort of meetings to decide other meetings. Have you felt a certain sort of energy of being uh back in a, in a start up again. Definitely, I've definitely felt a lot of energy being back in a start up. It is, uh I think it is no comparison

51:01 in terms of, uh you know, the, I don't know the way in which, you know, you can motivate people to all work on this problem. And um and it's, it's, it's somewhat small group at this point. Of course, when these companies grow much bigger, there's also, of course, you know, how do you maintain that spirit is not so easy, but, you know, as a start up, that's very special because you're small, um, you want to change the world, you have that vision to do that and you have, you know, you can do it quick and fast with a lot of money actually under your wings. So it's a big responsibility on the one hand, on the other hand,

51:38 it's also an incredible opportunity. And, yeah, and, and really a little bit of a different feel than working in a big company. Yeah, I wanted to pivot a little bit to get your advice really. And some of your, uh uh, yeah, maybe your advice, I guess to, to people who are early on in their careers, I think you've had a very interesting trajectory and, and, and kind of movement if you, what would you recommend? So if you're, you're going back, you're doing your undergraduate degree. Let's say you've already decided to do a PhD. Would you recommend go do a postdoc, try and go the academic route? Would you suggest to go to a start up?

52:22 How do you navigate this for people who wanted to sort of follow in your footsteps? What was, is there anything that you would do again or do differently? So, I should say I did many, many postdocs, right. So, um I did, uh I, I graduated in 98 and then I did a couple of years at Caltech and then I did and another three years with Geoffrey Hinton and do two different places and I just loved it. So, so the first thing I want to say, it depends a lot on who you are. Um And also, um so that really suited me at that stage in my life. It was incredibly interested in fundamental science and um I didn't get paid a lot at all.

53:06 Um But I didn't care at all, you know, it's just, it was a beautiful life, you know, uh having very little money to spend and just being, you know, being a scientist. Um So these days, but, but, but I didn't have that choice, right? So that was actually different and interesting. So these days it's much harder for young people because they see, you know, this other opportunity, clear clearly in front of them, right? And they say I could, I do that, you know, and earn like 10 times as much as I can earn there. Is it worth, is it worth for me to just go to this do this academic route? And you know, honestly academics

53:44 working in academia is also not all positive, right? At some point, you, you know, you're asked to juggle many balls, like you have to write grants, you have to teach, you have to basically, you know, run your own business. You have to be good people, a good manager. You know, you have to be good at research. You have to be good at everything basically. And then you, you don't have job security until you get tenure. It's really in that sense, a pretty lousy deal if you think about it. So you have, you have to really, you know, want, you want, you, you should be an educator. So the good thing about academia is it's great to work with young people,

54:22 right? It's fantastic to help people grow and I still sometimes get emails from people. I was in your master class then and then, and it's because of you that I took this route and I now have this fantastic job, right? That is, you know, unbelievably rewarding to get that. Um And that's what you have in academia, you can help people grow, you can turn them into great researchers and you can see them have stellar careers, which is very rewarding. I think so. So, so, you know, is that if that's you, you want to work on the inside of a black hole, you know what's going on on the inside of a black hole. And you like to work with young people

55:01 and you can put up with some annoying nuisances in, in academia. Then academia is your, is your thing, I honestly believe and you should, and especially, um, you should not worry about money. It's easy for me to say now. Um, but I didn't do it then I didn't worry about it. But it was the best thing I decided because I could really focus on, you know, building a, um, a good foundation for science learning. A lot about a lot about different topics and, and, and having a sort of a research strategy for myself. So I think having a postdoc on the Ariel Rings is, is really fantastic. And I don't think if you're good postdoc

55:40 and you publish well, you know, it, it's not a disadvantage either. I would say at, at all, but you can also do something else. You can also first work for a start up or work first work for, you know, for company and then go back to academia. You should, you should feel free to, you know, you're not closing any doors. I feel, you know, going back and forth between East East may maybe make sure you publish it now and then something exciting if you want to go back to academia. But do you think there should be more that the complaint I hear, uh, or the issue is that post doc, the funding model, the contract model is such that, as you said, for a certain time period,

56:25 you can sort of deal with it and I know from my own experiences but there comes a point where you, you're sort of, you want to buy a house or you want, you want to get married or something? There comes a point. Do you think universities should do more? Oh. Is there any way that you can do more to sort of keep those researchers who are great? But give them more security or, or does that break just the model of how universities work? Yeah. I don't know. It's a very hard one. Yeah. So, in Europe, you know, especially in the Netherlands tenure isn't, you know, it's very likely you'll get tenure, right. There's even something that was only recently introduced in some sense, but I've,

57:06 I've never seen people not get tenure. So it's, it's not at all like MIT, or Harvard where I know half of the people get kicked out or something. But then if you're kicked out of Harvard, you still have a great career afterwards, right? Because you work at Harvard in the first place. So, yeah, I've never really worried about tenure and all these things. I went through the 10 year process in the U.S. I just—maybe it's just blissful ignorance. I just never worried about it. I just, but whatever happened, it happened and just take the next step. Um, I don't know whether we can give postdoc permanent positions because that's not the definition of a postdoc. So,

57:40 what we have done at the University of Amsterdam a little bit is to allow people more like flexible contracts. Right. So, for instance, you could say half of my time I'm opposed or half of my time, you know, that particular construction doesn't exist, but it's a half time. You work in a start up, half your time, you work at the university or something like that. I think that could be very helpful because half of the time you can make a lot of money, you know, working in a start up or a big tech company or whatever or run your own business. And then the other half of the time you teach or, you know, do research at the university

58:14 that's already incredibly helpful and it doesn't cost university anything. So I was going to say that model, is it by chance or is it something? So if you look at an engineering company, look at Boeing, look at Airbus, look at Rolls Royce, look at Shell, look at almost any and I don't think there are very many people there unless I'm wrong who are, let's say their head of their science, but also a professor somewhere else. They seem to be purely at that company. Whereas I've noticed in tech companies and I guess your previous role of Microsoft was an example of that. They almost always is a, you know, extremely distinguished scientists like yourself who also has

58:56 a link to university because it enables them, by definition, they, that's how they still become distinguished because, you know, they, they're doing their own research. Do you think this is a model that should be taken more broadly or is it just something unique to the tech world with AI? If that's the case, I've just noticed this is a, a trend that, yeah, we do have some professorships in the Netherlands that are one day a week at the university um that are subsidized by a company. So I think it, it does exist a little bit more broadly, of course, in the tech world, it gives you also access to the talent pool, which is, which is a very scarce resource

59:35 in AI which may not be the same thing in other companies. So that, that's one way in which it's already helpful to have some leg in the university. Um Yeah, and the other thing is that this field is developing so fast, you just need to be at the front, you know, you know, frontier of all of that to know what's going on. And the best way to do that is to have, you know, to have students and you know, let them explain to you what the new trend is, right? So it's, it's really important that you're, you don't get, you know, sort of a, how do I say, um, fix it into your own, um, sort of ideology. Um But you keep listening to

1:00:16 young researchers which pick, pick up the new trends and want to work on new things and, and you, you go with that flow in some sense. So I think there's many advantages of having a foot in academia also for the company where you work for. Uh In fact, yeah, well, maybe um coming, coming to a final question, if in term, I think I heard you say in um one of the recent um talks that you gave that you had underestimated where machine learning could get to. So with that in mind, if we were to fast forward five years, where do you think we will be at and particularly from the AI for science side, do you think there will be a breakthrough,

1:01:05 huge change almost like, uh, OpenAI did with ChatGPT, which sort of sent this inflection point? Where do you see as being uh in the next few years? That's a tough one. Predicting the future is always extremely tough. Um I do see a very steady improvement and uh this field will grow definitely a lot over the next five years. So I see, but that's also because, you know, students need to get interested in it, you know, new start ups need to start, you know, big tech companies needs to get interested in this, which is what's happening now. Um And so that field will steadily grow, um whether there will be sort of a watershed moment.

1:01:44 Um Now, the reason that happened in, you know, in the other field is because there's a lot of data available there. Um So I think in some sense, the weather forecasting is a good example of a somewhat of a watershed moment. Um where, you know, these models are un you know, much better than people would have predicted. Um Now I'm betting myself on materials clearly. Um And I think my prediction would be that there's going to be enormous pool or pressure if you wish from society to work on, you know, problems and sustainability because we are running into a wall in this whole climate change problem, we are still increasing our

1:02:31 carbon output. Um you know, um since over the last 12% more, over the last five years, something like this. Um And there's a lot of countries which still have to go through sort of uh industrial revolution almost. So I I'm not seeing that politics will solve this. Um And the problem with this is that it's a shared resource and um it, it's, you know, there might be a dynamics where by the time it's in your face, it's too late because you're past tipping points and all these kinds of things and humans have the annoying, proper, you know, um as a property of only responding to things when it's right in their face, right?

1:03:16 You know, when the danger is right there in front of you and the lion is there to eat you, you know, that's when you run away. And so it's very hard to predict and say, oh, this is gonna be tricky, right? And so some people do that and some governments even do that. But two is that a global scale is incredibly hard. Now, what we will see is the impact of climate change, increasing and increasing. And that would mean that um this awareness also it's more in your face. So it become, you know, the awareness starts to increase as well. And that means that people are really going to look for solutions in the tech business as well.

1:03:50 So, you know, and there will be more and more investments and more and more people will go into that to try to do fusion or carbon capture or, you know, uh clean energy, all these kinds of things, right? And that will accelerate and that's a good thing. Um So in that sense, I think we will find amazing new technologies and you know, where indeed you you could actually find completely weird new materials that you know, weren't imaginable some time ago. But now with these new tools, we can actually create them. Um So I think we'll see amazing things where there's going to be an absolutely watershed moment that's really hard to predict.

1:04:29 Um But um yeah, I, I do think that with the new technology that we're building in AI um and maybe quantum computing as well uh in, you know, 5 to 10 years, this field will accelerate and, and, and, and produce very exciting results. Yeah. Do, do you feel almost that um one person told me we've globally tech companies and start ups have hired so many people to work on large language models. Do we almost need to wait for that to plateau for all those staff to then be given another direction to work on and then they can double down more on the scientific problems where for now, I guess most companies are trying to chase the current goal, which is,

1:05:14 you know, the the large language model type thing. Do, do you feel there's also a little bit of a resourcing and pivotal moment that we need to shift to focus on the science or can they be be done in parallel? Um I think they have to be done in parallel because I have no illusions that um you know, AGI will be pursued hotly by all these big companies because it's just, I don't know, you can make too much money if you find that, you know, this is too uh um you know, you can, how do you say too lucrative to, to not do it. Um But I think there is also a lot of talent in the market and I have found that a lot of young people in particular,

1:05:57 they want to do something that is meaningful with their life. And it's that subset of people that can align their goals in life, you know, contributing meaningfully to society with their talent and make a good, you know, salary on the side. And so I think there will be an increasing group of people who want to pursue these kinds of goals rather than work on advertisement placement or something like that. Well, yeah, I, I agree. I, I hope so. I think uh it's amazing what machine learning has been able to achieve. But I agree that if we could um push all that knowledge and that amazing thing to things that will help the world and society and the average individual,

1:06:41 then I think that will be seen as a very positive outcome from all this rather than, as you said, things that are maybe still useful, but maybe less an impact on society and on climate change. So, thank you so much. I know you're a very busy man. So I appreciate you taking the time to speak. I certainly learned a lot and I found it very interesting. So, yeah, thank you so much. My pleasure, Neil. It was, it was really fun talking to you

1:07:30 that