1
00:00:00,280 --> 00:00:02,400
Hi, and welcome to the Neil
Ashton Podcast.

2
00:00:03,080 --> 00:00:05,920
In each episode, we explained
some of the fascinating ways

3
00:00:05,920 --> 00:00:09,080
that science and engineering are
changing the world around us.

4
00:00:09,800 --> 00:00:12,760
We talked to leading engineers
from elite level sports like

5
00:00:12,840 --> 00:00:16,920
cycling and Formula One to some
of the world's top academics to

6
00:00:16,920 --> 00:00:21,000
understand how fluid dynamics,
machine learning, supercomputing

7
00:00:21,360 --> 00:00:23,000
are bringing in a new era of
discovery.

8
00:00:23,960 --> 00:00:27,080
We also hear some of their life
stories, their career advice,

9
00:00:27,640 --> 00:00:30,080
the lessons they've learned on
the way that I hope will be

10
00:00:30,080 --> 00:00:33,840
helpful to you too.
So sit back and enjoy this

11
00:00:33,840 --> 00:00:41,560
episode.
Hi, and welcome back to the Neil

12
00:00:41,560 --> 00:00:45,320
Ashton Podcast.
So on today's episode, we have

13
00:00:45,320 --> 00:00:48,760
Professor Michael Mahoney, who
is one of the world's leading

14
00:00:48,760 --> 00:00:54,640
experts on machine learning,
mathematics, and computer

15
00:00:54,640 --> 00:00:57,600
science.
He's also somebody who I've got

16
00:00:57,600 --> 00:01:00,480
to know over the past year and
has been a great help in

17
00:01:00,480 --> 00:01:04,040
understanding this space.
He's among other things, an

18
00:01:04,040 --> 00:01:05,800
Amazon scholar.
So we've managed to work

19
00:01:05,800 --> 00:01:09,160
together a little bit.
And hopefully, as you can tell

20
00:01:09,160 --> 00:01:12,440
from this episode, he's a a
really nice guy and has a great

21
00:01:12,440 --> 00:01:15,360
sense of humor and such an
intelligent person.

22
00:01:15,800 --> 00:01:18,880
Where do I start to talk about
what he's done?

23
00:01:18,880 --> 00:01:24,600
Well, he's a professor at UC
Berkeley in the Department of

24
00:01:24,600 --> 00:01:28,440
Statistics, but he's also at
Lawrence Berkeley National Lab.

25
00:01:28,680 --> 00:01:32,120
And as I mentioned, he's also an
Amazon scholar and, and has a

26
00:01:32,120 --> 00:01:38,320
few more hats as well.
He, if you look at his Google

27
00:01:38,320 --> 00:01:41,240
Scholar, which for a lot of
academics is a way of, you know,

28
00:01:41,320 --> 00:01:43,960
getting a look at what they've
done, he has some pretty

29
00:01:43,960 --> 00:01:47,160
impressive statistics.
And one that sort of comes to

30
00:01:47,160 --> 00:01:52,520
mind, not only does he have more
than 36,000 citations, which,

31
00:01:52,560 --> 00:01:57,600
which is a lot, his h-index is
84, which is very high.

32
00:01:58,240 --> 00:02:02,720
But what's more impressive is,
and again, this is probably

33
00:02:02,800 --> 00:02:04,600
going into the details, but it
makes a difference.

34
00:02:04,600 --> 00:02:06,200
If you go on Google Scholar and
you look at the number of

35
00:02:06,200 --> 00:02:10,479
citations, his is exponentially
growing.

36
00:02:10,479 --> 00:02:14,040
So not only does he have more
than, you know, 30,000

37
00:02:14,480 --> 00:02:19,040
citations, H&X of 84, which is
already in the very high, sort

38
00:02:19,040 --> 00:02:22,920
of top 1% or probably even
higher, it is increasing like

39
00:02:22,920 --> 00:02:27,600
this, which basically shows that
his research is becoming ever

40
00:02:27,600 --> 00:02:31,560
more relevant every single year.
It's not plateauing or going

41
00:02:31,560 --> 00:02:33,360
down.
That's actually very impressive

42
00:02:33,360 --> 00:02:37,000
and that shows why he is so
highly regarded because his work

43
00:02:37,000 --> 00:02:41,360
is really cutting edge.
There are many, I think it's

44
00:02:41,360 --> 00:02:44,920
fair to say that because he has
that mathematics background and

45
00:02:44,920 --> 00:02:47,280
that computer science
background, he's put his hand to

46
00:02:47,280 --> 00:02:50,200
many different things.
But one thing maybe relevant for

47
00:02:50,200 --> 00:02:54,000
the listeners or viewers of this
podcast is a paper he did with

48
00:02:54,000 --> 00:02:56,640
some colleagues on
characterizing possible failure

49
00:02:56,640 --> 00:02:58,680
modes in physics informed neural
networks.

50
00:02:58,680 --> 00:03:02,360
And this is, you hear many
people talk about pins, physics,

51
00:03:02,360 --> 00:03:04,440
informed neural networks.
And so they did a great paper a

52
00:03:04,440 --> 00:03:07,800
couple of years ago looking at
some places where it may not do

53
00:03:07,800 --> 00:03:11,080
so well.
We talk actually about that in

54
00:03:11,080 --> 00:03:12,840
the podcast.
We go through some of the themes

55
00:03:12,840 --> 00:03:16,120
I've discussed with other
leading ML experts.

56
00:03:16,920 --> 00:03:19,680
And we talk about, for example,
his opinion on, you know, do you

57
00:03:19,680 --> 00:03:23,120
really need to include the
physics in this AI for science

58
00:03:23,120 --> 00:03:25,560
regime or is it good enough just
to use data?

59
00:03:26,440 --> 00:03:29,200
We discussed this at length.
And we also get into the topic

60
00:03:29,200 --> 00:03:31,960
of foundational models for
science, something that he's

61
00:03:31,960 --> 00:03:34,400
actually really being a leading
voice on.

62
00:03:34,400 --> 00:03:37,200
He's given numerous keynotes,
important seminars, and

63
00:03:37,200 --> 00:03:39,880
published papers in this area.
So we have a good debate about

64
00:03:39,880 --> 00:03:41,840
that.
But we actually start off the

65
00:03:41,840 --> 00:03:44,680
conversation talking about
something he's also very well

66
00:03:44,680 --> 00:03:48,920
known for, which is randomized
linear numerical algebra, which

67
00:03:48,920 --> 00:03:52,120
is quite a complex topic, to be,
to be completely honest.

68
00:03:52,320 --> 00:03:55,280
And I don't think we got to the
bottom of it in this short talk,

69
00:03:56,040 --> 00:03:59,240
but it really shows that some of
his work from the pure math side

70
00:03:59,280 --> 00:04:03,200
or pure math side is coming
through and being relevant as we

71
00:04:03,200 --> 00:04:06,880
increasingly look towards lower
precision methods and ways of

72
00:04:07,320 --> 00:04:11,160
doing mathematical tricks to try
and improve the speed of many

73
00:04:11,880 --> 00:04:15,200
linear solvers, which are
common, of course in many, many

74
00:04:15,200 --> 00:04:17,720
fields of science, including
CFD.

75
00:04:18,920 --> 00:04:24,560
We also talk about his general
opinions or advice for people

76
00:04:24,560 --> 00:04:26,440
looking to move between academia
and industry.

77
00:04:26,440 --> 00:04:29,160
Just general career advice,
something he's well placed given

78
00:04:29,160 --> 00:04:31,720
that he has an incredible
academic record, but he's also

79
00:04:31,720 --> 00:04:36,040
very close to industry and sort
of industrial applications as

80
00:04:36,040 --> 00:04:39,480
well.
An hour or an hour and a bit is

81
00:04:39,840 --> 00:04:44,680
not really enough to go through
all of this, but I, I certainly

82
00:04:44,680 --> 00:04:47,600
learnt a lot and I really
enjoyed talking to him like I do

83
00:04:47,600 --> 00:04:50,520
every time I speak to him.
And I hope you'll, you'll find

84
00:04:50,520 --> 00:04:53,240
the same thing.
I will add some links if you're

85
00:04:53,240 --> 00:04:58,000
watching this on YouTube because
again, he's published so many

86
00:04:58,000 --> 00:05:01,160
papers and done so many great
talks that I really want you

87
00:05:01,720 --> 00:05:04,280
after watching this, listening
this to go to his website and

88
00:05:04,280 --> 00:05:08,920
and look through all of those.
And obviously if you are

89
00:05:08,920 --> 00:05:12,160
watching this on YouTube, you
may prefer to listen to it.

90
00:05:12,840 --> 00:05:16,360
Some people don't realise that
these podcasts are in video form

91
00:05:16,360 --> 00:05:18,920
and YouTube, but they're also on
Spotify and Apple for audio and

92
00:05:18,920 --> 00:05:20,440
vice versa.
If you normally listen to this

93
00:05:20,440 --> 00:05:23,240
and you didn't realise there's a
full video version, you can go

94
00:05:24,080 --> 00:05:26,960
to YouTube.
So yeah, I hope you enjoy this

95
00:05:26,960 --> 00:05:30,160
conversation with Professor
Michael Mahoney.

96
00:05:31,000 --> 00:05:34,800
One topic that you mentioned to
me when we were chatting in the

97
00:05:34,800 --> 00:05:38,600
past that I was fascinated by,
but if I'm being completely

98
00:05:38,600 --> 00:05:40,360
honest, I didn't fully
understand.

99
00:05:40,680 --> 00:05:44,200
So it's good for my purpose and
for everybody else was around

100
00:05:44,200 --> 00:05:49,560
the randomized linear algebra.
What exactly?

101
00:05:50,160 --> 00:05:52,640
For people who don't know what,
what is it and why is this

102
00:05:52,640 --> 00:05:56,600
becoming even mentioned on a
Netflix show?

103
00:05:58,640 --> 00:06:01,640
So that yeah.
So that's, this was a sign of

104
00:06:01,640 --> 00:06:05,640
success when it was finally
mentioned on a net no, the

105
00:06:05,640 --> 00:06:08,840
Lincoln Lawyer a couple of years
ago, someone sent it to me and

106
00:06:08,840 --> 00:06:11,560
it was mentioned in in one of
the courtroom scenes.

107
00:06:12,600 --> 00:06:17,000
So randomized linear algebra,
randomized numerical linear

108
00:06:17,000 --> 00:06:21,840
algebra is basically an area
that uses randomness as an

109
00:06:21,840 --> 00:06:24,920
algorithmic resource to solve
linear algebra problems.

110
00:06:26,440 --> 00:06:29,280
A lot, a lot of linear algebra
problems are under the hood,

111
00:06:29,280 --> 00:06:31,160
whether you're doing machine
learning or scientific

112
00:06:31,160 --> 00:06:34,760
computing, they manifest
themselves in different ways.

113
00:06:34,760 --> 00:06:37,360
So the exact questions and
numerical issues and so on are,

114
00:06:37,360 --> 00:06:41,200
are different in those areas.
And that's the source of maybe

115
00:06:41,200 --> 00:06:44,680
tension in the area, but also
synergy and, and, and, and, you

116
00:06:44,680 --> 00:06:47,080
know, part of the reason a lot
of people are interested in it

117
00:06:47,080 --> 00:06:48,760
because it, it holds the
potential to solve a lot of

118
00:06:48,760 --> 00:06:52,200
problems people look at, But
it's, it's solving core linear

119
00:06:52,200 --> 00:06:55,000
algebra problems and, and linear
algebra ones appear in a lot of

120
00:06:55,000 --> 00:06:56,280
places.
If you're solving partial

121
00:06:56,280 --> 00:06:59,680
differential equations, it's,
you know, linear operators or

122
00:06:59,680 --> 00:07:01,240
iterated former linear operators
appear.

123
00:07:01,240 --> 00:07:04,560
If you're solving machine
learning, you may be interested

124
00:07:04,560 --> 00:07:07,680
in support vector machines or
ensembling methods or these days

125
00:07:07,920 --> 00:07:11,880
deep neural networks.
And so matrix multiplications

126
00:07:11,880 --> 00:07:13,040
are at the core of a lot of that
stuff.

127
00:07:13,040 --> 00:07:14,960
So very core linear algebra
problems.

128
00:07:15,840 --> 00:07:17,840
I think historically the way
people thought about the

129
00:07:17,840 --> 00:07:21,840
relationship between linear
algebra and and randomness or

130
00:07:21,840 --> 00:07:24,000
noise was that there's
randomness and noise in the

131
00:07:24,000 --> 00:07:27,360
world, meaning the data you
measure, think of least squares

132
00:07:27,840 --> 00:07:29,680
and it's your job to clean it
up.

133
00:07:29,840 --> 00:07:34,040
And then you call a linear
algebra problem least squares or

134
00:07:34,040 --> 00:07:36,440
low rank approximation.
And you get more or less a

135
00:07:36,440 --> 00:07:39,840
deterministic answer and you get
more or less the exact answer.

136
00:07:39,920 --> 00:07:42,840
I mean, and I say exact and
scare and scare quotes because

137
00:07:43,160 --> 00:07:46,040
there's numerical issues and you
can't represent sqrt 2 on a

138
00:07:46,040 --> 00:07:48,800
computer, but you know, in so
far as machine precision is

139
00:07:48,800 --> 00:07:51,280
exact and you get an exact
deterministic answer.

140
00:07:52,200 --> 00:07:53,840
And for a lot of things that's
just overkill.

141
00:07:53,880 --> 00:07:58,280
And so you can use randomness as
an algorithmic resource, meaning

142
00:07:58,280 --> 00:08:00,920
inside the algorithm to speed up
computation.

143
00:08:01,560 --> 00:08:06,200
And this may be most notable
historically, like in Monte

144
00:08:06,200 --> 00:08:09,840
Carlo or Markov Chain Monte
Carlo, where you run simulations

145
00:08:09,840 --> 00:08:12,000
of of fluid dynamics.
I mean, the Metropolis algorithm

146
00:08:12,000 --> 00:08:16,440
was developed in that context,
but this is for core numerical

147
00:08:16,440 --> 00:08:19,440
linear algebra problems.
And so most people, if they're

148
00:08:19,440 --> 00:08:23,720
running computations, sit on top
of Glass and LA Pack and, and,

149
00:08:23,720 --> 00:08:27,840
and related software.
And if you're calling Python,

150
00:08:27,840 --> 00:08:29,480
you're calling something else,
you're calling something that's

151
00:08:29,480 --> 00:08:31,200
calling something that's calling
them typically.

152
00:08:31,680 --> 00:08:36,679
And so these are core libraries.
And so the question is, can you,

153
00:08:36,880 --> 00:08:41,000
you know, use good theory from
randomness and measure

154
00:08:41,000 --> 00:08:44,080
concentration and high
dimensional probability to

155
00:08:44,080 --> 00:08:46,880
improve those algorithms or
improve, you know, variance of

156
00:08:46,880 --> 00:08:48,280
those algorithms?
The short answer is that you

157
00:08:48,280 --> 00:08:49,840
can.
So how?

158
00:08:49,840 --> 00:08:53,160
But so how does it work in
practice then?

159
00:08:53,200 --> 00:08:57,480
So if I'm sold in APDE and I'm
normally using some sort of

160
00:08:57,480 --> 00:09:02,120
linear algebra library and it
takes me this time or this

161
00:09:02,120 --> 00:09:06,280
amount of flops or this compute,
how is it that the randomness

162
00:09:06,600 --> 00:09:12,280
reduces that?
Yeah, I mean, when you're

163
00:09:12,280 --> 00:09:16,080
solving APDE, you're typically
solving it for a particular

164
00:09:16,080 --> 00:09:21,480
application and you're calling
certain core linear algebraic

165
00:09:21,680 --> 00:09:23,800
primitives in in one way or
another.

166
00:09:23,800 --> 00:09:26,240
So for example, if you're
solving it with a splitting

167
00:09:26,240 --> 00:09:29,240
method or predictor corrector
method, or you're solving a

168
00:09:29,240 --> 00:09:32,560
finite element or finite volume
or you're solving a second order

169
00:09:32,560 --> 00:09:35,640
optimization to as a piece of
that PDE solver.

170
00:09:37,000 --> 00:09:40,840
These have least squares, you
know, linear solves things like

171
00:09:40,840 --> 00:09:43,080
this under the hood.
So typically you're improving

172
00:09:43,080 --> 00:09:44,400
those.
You could, you could improve the

173
00:09:44,400 --> 00:09:48,000
modeling setup for the PDE also.
But but you know, you could, you

174
00:09:48,000 --> 00:09:50,680
could try and solve the, you
know, the improve the core

175
00:09:50,680 --> 00:09:53,480
primitive.
So take least squares as an

176
00:09:53,480 --> 00:09:55,320
example, a low rank
approximation.

177
00:09:55,840 --> 00:10:00,400
A common motif in in a lot of
these algorithms is you have the

178
00:10:00,400 --> 00:10:02,720
data and you want to solve the
least squares of the low rank

179
00:10:02,720 --> 00:10:06,440
problem.
And you could solve it exactly

180
00:10:06,640 --> 00:10:11,000
machine precision or whatever.
But you might want to on the

181
00:10:11,000 --> 00:10:13,880
other hand, get what they call a
sketch of the of the data, which

182
00:10:13,880 --> 00:10:15,680
is roughly a small number of
data points.

183
00:10:15,680 --> 00:10:19,000
It could be a small number of
actual data points, or it could

184
00:10:19,000 --> 00:10:20,880
be what they call a random
projection.

185
00:10:20,880 --> 00:10:23,720
And if you're familiar with like
if you feel signal processing

186
00:10:23,720 --> 00:10:27,840
and electrical engineering and,
and physics, think of this as

187
00:10:27,880 --> 00:10:31,120
like a randomized version of a,
of a Fourier transform.

188
00:10:31,120 --> 00:10:34,840
So it takes some signal that
that might be localized in space

189
00:10:34,840 --> 00:10:36,280
and, and spreads it out
everywhere.

190
00:10:36,920 --> 00:10:38,800
But you don't need this to be a
physical space.

191
00:10:38,800 --> 00:10:41,120
This is just a linear algebraic
problem.

192
00:10:41,120 --> 00:10:46,280
And so you have, you know, you
have columns and rows, so you

193
00:10:46,280 --> 00:10:49,480
could select the important
columns or rows, or you could do

194
00:10:49,480 --> 00:10:52,560
this random projection, which is
essentially a, a basically a

195
00:10:52,560 --> 00:10:55,440
random type rotation that
spreads the information out and

196
00:10:55,440 --> 00:10:57,600
then sample uniformly in that
rotated space.

197
00:10:57,880 --> 00:11:00,840
And so you take this sketch and
you can do one of a couple

198
00:11:00,840 --> 00:11:03,280
things that so some communities,
the more theoretically inclined

199
00:11:03,280 --> 00:11:05,760
communities want to take that
sketch and solve the sub problem

200
00:11:06,040 --> 00:11:07,920
exactly.
And there's a range of

201
00:11:07,920 --> 00:11:10,920
theoretical work that says the
solution to the exact, the exact

202
00:11:10,920 --> 00:11:13,720
solution to that sub problem
computed any which way.

203
00:11:13,720 --> 00:11:16,840
A traditional solver or
something else is epsilon close

204
00:11:16,840 --> 00:11:19,080
to the exact solution to the
original problem if you set

205
00:11:19,080 --> 00:11:22,320
things up right.
Now, in a lot of cases that is

206
00:11:22,320 --> 00:11:24,800
sort of course because you
can't.

207
00:11:24,800 --> 00:11:27,120
It's hard to get machine
precision just by drawing a

208
00:11:27,120 --> 00:11:30,120
sample and solving the sub
problem unless you take a huge

209
00:11:30,120 --> 00:11:33,600
number of samples, just because
Monte Carlo methods tend to

210
00:11:33,600 --> 00:11:36,280
converge slowly as a function of
the error parameter.

211
00:11:36,840 --> 00:11:40,440
So you could take that, you
know, low quality, low

212
00:11:40,440 --> 00:11:44,560
precision, but not trivially bad
solution and ask yourself what

213
00:11:44,560 --> 00:11:46,640
is a preconditioner?
And by preconditioner I just

214
00:11:46,640 --> 00:11:50,880
mean a preconditioner for APDE
solver or or least squares or a

215
00:11:50,880 --> 00:11:52,640
linear solver.
And a preconditioner is

216
00:11:52,640 --> 00:11:55,000
basically a low quality solution
that you refine.

217
00:11:55,360 --> 00:11:58,920
So you can actually take this
preconditioner that the that

218
00:11:58,920 --> 00:12:00,760
this is a sketch.
The theoretically inclined

219
00:12:00,760 --> 00:12:03,640
people just say good, done M
epsilon good.

220
00:12:04,120 --> 00:12:06,240
And you can say, now I'm going
to take that epsilon good where

221
00:12:06,240 --> 00:12:09,320
epsilon is .1 and just use any
of a range of traditional

222
00:12:09,320 --> 00:12:12,120
iterative solvers and drive that
epsilon down to 10 to the -8 or

223
00:12:12,120 --> 00:12:15,960
10 to the -16.
And, and just like most

224
00:12:15,960 --> 00:12:20,400
preconditioners, if, if the cost
to construct it is less than the

225
00:12:20,400 --> 00:12:23,880
cost to, you know, the cost to
construct it plus iterate is

226
00:12:23,880 --> 00:12:26,880
less than you know, the, the
algorithm you're competing with,

227
00:12:26,880 --> 00:12:28,360
you win.
And it's a good preconditioner.

228
00:12:29,120 --> 00:12:31,520
So there's a range of ways
people solve it that way.

229
00:12:31,520 --> 00:12:33,720
So those are called sketch and
solve and sketch and

230
00:12:33,720 --> 00:12:37,000
precondition, respectively.
Increasingly for machine

231
00:12:37,000 --> 00:12:41,360
learning applications that want
medium precision, also relevant

232
00:12:41,360 --> 00:12:45,200
for scientific computing when
you're interested in low

233
00:12:45,200 --> 00:12:47,840
precision data representations,
you know, going to half or

234
00:12:47,840 --> 00:12:50,840
quarter precision, which is, is
increasingly seen in hardware.

235
00:12:51,360 --> 00:12:54,080
There's a more subtle interplay
where you can get the solution

236
00:12:54,080 --> 00:12:56,360
and then you can iterate it and
get a second sketch and toggle

237
00:12:56,360 --> 00:12:57,960
back and forth.
And if you set the parameters

238
00:12:57,960 --> 00:13:00,960
differently, you can get
intermediate solutions 10 to the

239
00:13:00,960 --> 00:13:03,480
-210 to the -410 to the -6
quality.

240
00:13:03,480 --> 00:13:05,360
So there's a range of ways you
can use the sketches, but

241
00:13:05,360 --> 00:13:08,400
roughly the idea is you get most
of the information in the sketch

242
00:13:08,400 --> 00:13:10,040
and and solve it or do something
with it.

243
00:13:11,040 --> 00:13:15,440
And is there any, you know,
orders of magnitude of the

244
00:13:15,440 --> 00:13:20,720
savings that you could get?
You know, if I'm doing a Nabia

245
00:13:20,720 --> 00:13:25,440
Stokes solver or solving some
neural net using these

246
00:13:25,800 --> 00:13:28,880
randomized, you know, numerical
linear algebra versus a blast

247
00:13:28,880 --> 00:13:33,600
rolling back, Are we talking,
you know, a 10% saving?

248
00:13:33,600 --> 00:13:38,720
Is it an order of magnitude or
is it still under research?

249
00:13:38,720 --> 00:13:41,080
And so it's not delivering the
full yet.

250
00:13:41,960 --> 00:13:43,800
Yeah.
I mean, I think the question of

251
00:13:43,920 --> 00:13:48,120
of how well you'd improve upon
something depends strongly on

252
00:13:48,120 --> 00:13:51,440
what your baseline is.
And if you feed this into a big

253
00:13:51,440 --> 00:13:53,480
PD solver, there's a lot of
moving parts and you're

254
00:13:53,480 --> 00:13:57,000
competing with very mature code
and they're doing a little bit

255
00:13:57,000 --> 00:13:59,760
better or even a lot better on a
solver may or may not matter for

256
00:13:59,760 --> 00:14:02,400
the downstream use case that
that the scientist that is

257
00:14:02,400 --> 00:14:05,560
looking at the PD solver is
interested in roll gold to the

258
00:14:05,560 --> 00:14:08,000
other extreme.
And you want to say, I want to

259
00:14:08,000 --> 00:14:10,200
compete with blast or a late
pack.

260
00:14:10,920 --> 00:14:14,800
These are extremely optimized
pieces of code and it's, you

261
00:14:14,800 --> 00:14:17,360
know, it's hard to beat them.
So one of the big successes in

262
00:14:17,360 --> 00:14:19,560
the area was with blend and
pick.

263
00:14:19,560 --> 00:14:22,560
And then soon after that was LSR
and blend and pick, which said

264
00:14:22,560 --> 00:14:26,960
we wanted to ask whether these
randomized sketches, you know,

265
00:14:27,320 --> 00:14:30,640
not in theory, not Big O
notation, not whatever can they

266
00:14:30,640 --> 00:14:34,880
beat LA pack Because this is not
boutique code you have or I have

267
00:14:34,880 --> 00:14:36,920
or a solver that you don't share
with the community.

268
00:14:36,920 --> 00:14:39,720
So well, this is something
that's been stress test for

269
00:14:39,720 --> 00:14:41,880
decades.
And the answer is yes.

270
00:14:41,880 --> 00:14:45,480
I mean, basically on any tall
dense matrix, you can beat LA

271
00:14:45,480 --> 00:14:47,880
pack with these techniques.
You got to set parameters right

272
00:14:47,880 --> 00:14:49,600
and be a little bit careful.
But, but the short answer is

273
00:14:49,600 --> 00:14:52,600
yes, how much you improve it by
depends on the aspect ratio and

274
00:14:52,600 --> 00:14:54,920
depends on properties of of the
matrix.

275
00:14:54,920 --> 00:14:57,160
But think of it as ballpark 2 to
10.

276
00:14:57,520 --> 00:15:02,440
Now this was back in 2010 and so
a lot's changed the since then,

277
00:15:02,440 --> 00:15:05,440
in particular in the hardware
landscape with respect to GP us

278
00:15:05,440 --> 00:15:08,400
and hardware heterogeneity and
the Sun sending of Moore's law

279
00:15:08,400 --> 00:15:10,120
and Dennard scaling and these
sorts of things.

280
00:15:10,800 --> 00:15:15,040
And so it is the case that you
know, you can be a factor of 10

281
00:15:15,040 --> 00:15:18,800
better.
Similarly in in in low rank type

282
00:15:18,800 --> 00:15:22,080
approximations.
I think the the the way

283
00:15:22,560 --> 00:15:24,960
scientists versus machine
learners use low rank

284
00:15:24,960 --> 00:15:26,240
approximations is very
different.

285
00:15:26,240 --> 00:15:31,520
Scientists tend to say low rank
means you know 99% of the

286
00:15:31,520 --> 00:15:36,000
Frobenius norm, meaning 99.9% of
the Frobenius norm and 99.99 I

287
00:15:36,040 --> 00:15:39,320
mean basically the whole matrix.
When machine learners say low

288
00:15:39,320 --> 00:15:41,440
rank, they mean sorry, sorry,
the spectral norm.

289
00:15:41,760 --> 00:15:44,920
When machine learners say low
rank, they mean, you know, 50%

290
00:15:44,920 --> 00:15:48,640
of the Frobenius norm, in which
case, you know, you lose a lot

291
00:15:48,640 --> 00:15:51,440
of information, but you might
iterate and and do something

292
00:15:51,440 --> 00:15:53,000
better.
So think of sort of scaling laws

293
00:15:53,000 --> 00:15:56,760
and the neural scaling context
with, with neural networks.

294
00:15:57,160 --> 00:16:00,520
And so the way you'd use those
low rank algorithms and feed

295
00:16:00,520 --> 00:16:02,800
this them into other solvers
would be very different.

296
00:16:03,120 --> 00:16:05,320
And then that would lead to very
different levels of improvement.

297
00:16:05,400 --> 00:16:07,240
And so I think that's largely
open.

298
00:16:07,240 --> 00:16:10,040
And then with the memory wall
that you're increasingly seeing

299
00:16:10,520 --> 00:16:13,400
in, in general, but most
egregiously in the machine

300
00:16:13,400 --> 00:16:17,240
learning applications, one of
the big wins and it's just

301
00:16:17,240 --> 00:16:19,360
starting to be explored for the
randomized techniques is

302
00:16:19,360 --> 00:16:22,360
basically improving the memory
properties.

303
00:16:22,880 --> 00:16:25,080
And so this could be just
reordering algorithms in the

304
00:16:25,080 --> 00:16:27,320
sense of communication, avoiding
linear algebra could be much

305
00:16:27,320 --> 00:16:30,720
broader because the randomness
sort of decouples things, makes

306
00:16:30,720 --> 00:16:33,080
it easier to paralyze certain
sorts of computations.

307
00:16:33,400 --> 00:16:36,480
And so you see much bigger than
factors of 2 or 10 improvement

308
00:16:36,480 --> 00:16:38,000
there.
But then again the baselines

309
00:16:38,000 --> 00:16:41,360
changing year to year as as as
people explore different

310
00:16:41,440 --> 00:16:44,960
representations and and low
precision and intermediate

311
00:16:44,960 --> 00:16:48,600
precision and stuff.
Interesting, so how was it

312
00:16:48,600 --> 00:16:51,360
mentioned in Netflix then going
back to the former?

313
00:16:52,520 --> 00:16:55,880
Well, the particular I, I think
so, I'm not a movie star or a

314
00:16:55,880 --> 00:16:59,880
movie producer, but I, I have a
sense that they, they want to,

315
00:16:59,880 --> 00:17:01,720
I, I mentioned this to someone
and they said, you know what

316
00:17:01,720 --> 00:17:03,000
that means?
They tried to take the most

317
00:17:03,000 --> 00:17:06,800
exotic thing that wouldn't make
sense to anyone and, and, and,

318
00:17:06,800 --> 00:17:09,640
and cited it.
So the context was I, I think it

319
00:17:09,640 --> 00:17:12,640
was early on and the, the, the
main character, who was this,

320
00:17:12,640 --> 00:17:16,440
the attorney needed to represent
someone and the person had to

321
00:17:16,440 --> 00:17:21,280
get out of, of, of whatever they
were charged with, which was not

322
00:17:21,280 --> 00:17:24,520
a, a, you know, relatively minor
thing, but it was the 10th time

323
00:17:24,520 --> 00:17:26,920
they did it.
And so they, the, the, the

324
00:17:26,920 --> 00:17:28,280
Lincoln lawyer said, why do you
need to get out?

325
00:17:28,280 --> 00:17:29,720
And they said, I have a defense
on Thursday.

326
00:17:29,720 --> 00:17:31,280
I need to get out.
And they asked what's the,

327
00:17:31,280 --> 00:17:34,120
what's the topic?
And they said randomization and

328
00:17:34,120 --> 00:17:37,680
numerical in your algebra.
So that was the context of how

329
00:17:37,680 --> 00:17:40,960
it was mentioned there.
I wonder how they found that

330
00:17:40,960 --> 00:17:41,920
out.
I wouldn't know how they

331
00:17:41,920 --> 00:17:43,520
actually.
Yeah, I don't know.

332
00:17:43,600 --> 00:17:45,120
They they, they, they didn't
call me up.

333
00:17:45,120 --> 00:17:48,240
So they must have had someone in
movie producer land or something

334
00:17:48,240 --> 00:17:52,240
that did did a Google search on
on something exotic and came

335
00:17:52,240 --> 00:17:53,120
across.
It that's fine.

336
00:17:53,120 --> 00:17:55,320
Yeah.
That's like reminds me of the

337
00:17:55,600 --> 00:17:59,040
Did you ever like Star Trek?
Yeah, I used to see if there's a

338
00:17:59,040 --> 00:18:01,840
bifurcation which maybe would
get us in into a separate

339
00:18:01,840 --> 00:18:06,160
discussion whether you like the
original version or the the

340
00:18:06,640 --> 00:18:07,760
second version.
But then there was this

341
00:18:07,760 --> 00:18:09,800
explosion of different,
different versions of it.

342
00:18:10,920 --> 00:18:16,000
Yeah, I was more the next
generation person, but I the

343
00:18:16,000 --> 00:18:19,840
reason I mention it is it was
often whenever they wanted to

344
00:18:19,840 --> 00:18:23,480
explain how they were breaking
the laws of physics, some

345
00:18:24,040 --> 00:18:26,960
stupidly complicated phrase was
used.

346
00:18:26,960 --> 00:18:30,800
And I'm sure it exists somewhere
and it just reminds me of that,

347
00:18:31,320 --> 00:18:33,480
you know, explanation of how
they're exceeding the warp

348
00:18:33,480 --> 00:18:35,600
drive.
Oh, it's because the something

349
00:18:35,600 --> 00:18:37,040
something.
Yeah.

350
00:18:37,040 --> 00:18:38,240
So I think it was a little bit
like that.

351
00:18:38,280 --> 00:18:40,200
I mean, I, I think it's a sign
of the times that what they're

352
00:18:40,200 --> 00:18:42,800
citing is not warp Dr. and
quantum gravity, but

353
00:18:42,800 --> 00:18:46,440
randomization numerical.
This is a core statement about

354
00:18:46,440 --> 00:18:48,480
how areas are progressing and so
on.

355
00:18:49,360 --> 00:18:50,680
Exactly.
Exactly.

356
00:18:50,680 --> 00:18:56,800
Yeah, yeah.
So maybe one of the topics would

357
00:18:56,800 --> 00:19:00,920
be good to chat with you about,
and it's definitely a hot topic

358
00:19:00,920 --> 00:19:07,520
at the moment, is the the
foundational models for science.

359
00:19:07,520 --> 00:19:11,440
I've seen quite different
viewpoints.

360
00:19:11,440 --> 00:19:16,920
There's one argument that is,
well, large language models take

361
00:19:16,920 --> 00:19:19,600
all this data from the Internet.
They train and they're

362
00:19:19,600 --> 00:19:22,000
remarkably good at doing what
they do.

363
00:19:24,160 --> 00:19:27,560
What about if we could do it for
scientific disciplines?

364
00:19:27,680 --> 00:19:31,560
Wouldn't we be able to just, you
know, ask her a prompt to go and

365
00:19:31,640 --> 00:19:33,960
simulate me a plane or, or, or
do something?

366
00:19:34,680 --> 00:19:39,600
And then there's the the other
Ave., which seems to be more the

367
00:19:41,200 --> 00:19:45,080
replacing simulation tools or
augmenting simulation tools like

368
00:19:45,080 --> 00:19:48,440
graph cast or forecast net or,
you know, these sort of Seminole

369
00:19:48,440 --> 00:19:51,200
pieces of work that have come
out, but they were for very,

370
00:19:51,640 --> 00:19:54,120
they had to be trained, you
know, for something.

371
00:19:56,200 --> 00:20:01,400
Where do you see this?
Where do you see this going?

372
00:20:01,760 --> 00:20:05,680
Are foundational models
possible?

373
00:20:05,960 --> 00:20:08,960
Is it we just don't have enough
data?

374
00:20:09,080 --> 00:20:11,880
This is a big topic.
So yeah, I'm not sure how we

375
00:20:11,880 --> 00:20:14,680
want to start it.
I mean, we're in the middle of

376
00:20:14,680 --> 00:20:18,120
this fray and I guess that's why
we were asking about this.

377
00:20:19,080 --> 00:20:20,960
And I should start off with
saying I don't know what a

378
00:20:20,960 --> 00:20:23,280
foundation model is.
Actually.

379
00:20:23,280 --> 00:20:26,360
I know what different people say
a foundation model is and, and

380
00:20:26,360 --> 00:20:29,080
it means very different things
to to different communities.

381
00:20:29,880 --> 00:20:33,880
And so I think articulating that
a little bit helps articulate

382
00:20:36,680 --> 00:20:38,800
some of the possible directions
and some of the questions you're

383
00:20:38,800 --> 00:20:41,520
asking.
I mean, I think one way to think

384
00:20:41,520 --> 00:20:44,920
about a lot of these machine
learning methods and it's

385
00:20:45,120 --> 00:20:47,680
highlighted particularly cleanly
with these foundation models is

386
00:20:50,200 --> 00:20:51,760
machine learning methods.
And in particular, these

387
00:20:51,760 --> 00:20:53,920
foundation models are, they're
an infrastructure, right?

388
00:20:53,920 --> 00:20:58,280
The, the, the, the Stanford
report said, call it foundation

389
00:20:58,280 --> 00:21:01,320
models, not foundational models,
because it's a foundation in

390
00:21:01,320 --> 00:21:05,200
which you build things and as
and as opposed to a half dozen

391
00:21:05,200 --> 00:21:06,400
other terms they could have
used.

392
00:21:06,400 --> 00:21:08,600
Now, whether that's right or
not, I mean, but that, that's

393
00:21:08,600 --> 00:21:11,320
what they said.
And so since then people have

394
00:21:11,560 --> 00:21:13,960
said, well, it has to be this
big and that big and, and

395
00:21:13,960 --> 00:21:17,440
whatever.
And, and then people want to get

396
00:21:17,440 --> 00:21:21,400
a foundation model, not for all
of science, but for area A&B&C

397
00:21:21,400 --> 00:21:24,480
and sub area D&E&F and get finer
and finer granularity.

398
00:21:24,840 --> 00:21:26,560
So in a sense it's, it's a it's
a term.

399
00:21:26,560 --> 00:21:29,520
And, and if you ask people, most
people using it what it means,

400
00:21:29,520 --> 00:21:31,640
they sort of will acknowledge
that, that they don't know.

401
00:21:31,640 --> 00:21:32,960
And it's not a standard
definition.

402
00:21:32,960 --> 00:21:35,080
I think probably the best way to
think about it is it's, it's,

403
00:21:35,120 --> 00:21:36,960
it's infrastructure, it's a
foundation what you build

404
00:21:36,960 --> 00:21:41,160
things.
And so the computer's

405
00:21:41,160 --> 00:21:43,880
infrastructure, I mean, you can
do different things with the

406
00:21:43,880 --> 00:21:46,120
computer than you can do with
pencil and paper.

407
00:21:46,680 --> 00:21:50,880
And so it wasn't obvious the way
computer science, even before it

408
00:21:50,880 --> 00:21:53,720
existed, would evolve.
And at one point, post World War

409
00:21:53,720 --> 00:21:56,280
2, the US thought they'd need 5
computers and that would be it.

410
00:21:56,400 --> 00:21:58,280
And then it would solve all the
nation's needs, right?

411
00:21:58,280 --> 00:22:00,040
So it evolved in ways different
than expected.

412
00:22:00,520 --> 00:22:03,480
And it involved in particular
because it, it could complement

413
00:22:03,480 --> 00:22:06,360
what people did.
I mean, you could ask different

414
00:22:06,360 --> 00:22:09,280
scientific questions than you
could with pencil and paper and

415
00:22:09,280 --> 00:22:11,000
you could ask different
engineering questions than you

416
00:22:11,000 --> 00:22:12,320
could.
I mean, now you can simulate

417
00:22:12,440 --> 00:22:15,200
whole things and, and the very,
very last step is building it,

418
00:22:15,200 --> 00:22:17,240
right.
But in addition, you could do

419
00:22:17,240 --> 00:22:19,160
lots of other things that were
driven by industry.

420
00:22:19,160 --> 00:22:21,320
And, and so there's a
bifurcation in the area whether

421
00:22:21,320 --> 00:22:23,560
you're doing more numerical
things that are continuous or

422
00:22:23,560 --> 00:22:25,760
whether you're doing discrete
things that, that, that are not

423
00:22:25,760 --> 00:22:28,320
continuous that were driven
largely cited by business needs.

424
00:22:29,080 --> 00:22:31,800
So I think the, the right way to
think about this is it's that

425
00:22:31,800 --> 00:22:36,000
sort of infrastructure and it's
tied to the data more closely

426
00:22:36,000 --> 00:22:39,440
than than, you know, database
systems because the foundation

427
00:22:39,440 --> 00:22:41,280
models have information cooked
into them.

428
00:22:42,000 --> 00:22:45,240
And so it wasn't obvious some
number of years ago that just

429
00:22:45,240 --> 00:22:47,560
with language you, you could do
what you know, has caught the

430
00:22:47,560 --> 00:22:49,480
popular attention in the last
couple of years.

431
00:22:50,160 --> 00:22:53,680
So I think so to anyone who says
that confidently, you, you can

432
00:22:53,680 --> 00:22:58,320
or you can't with science, I
mean, probably you can't say

433
00:22:58,320 --> 00:23:01,680
that they're sort of reliably
saying what what will hold for

434
00:23:01,680 --> 00:23:02,840
the future 'cause I think it's
not at all.

435
00:23:02,960 --> 00:23:04,280
I think it's not all clear for
two reasons.

436
00:23:04,280 --> 00:23:07,360
One, there's certain technical
things that really do matter.

437
00:23:07,400 --> 00:23:11,320
I mean that that are different
about scientific data than, you

438
00:23:11,320 --> 00:23:12,880
know, language and, and vision
data.

439
00:23:12,880 --> 00:23:15,200
And then there's just very
different cultural, there's very

440
00:23:15,200 --> 00:23:17,800
different cultural things.
I think every scientist thinks

441
00:23:17,800 --> 00:23:21,120
that their data is special and
unique just like they think that

442
00:23:21,120 --> 00:23:24,640
they're special and unique.
M LS taught you anything.

443
00:23:24,640 --> 00:23:26,800
These algorithms can predict
what movie you'll watch and what

444
00:23:26,800 --> 00:23:28,320
stuff you'll buy better than you
can.

445
00:23:28,320 --> 00:23:31,240
And so you're maybe slightly
less unique, you know, than than

446
00:23:31,240 --> 00:23:33,960
you thought in some sense.
And so I think there's a

447
00:23:33,960 --> 00:23:37,040
question, you know, what does
foundation model mean and how

448
00:23:37,040 --> 00:23:40,040
can it be used so narrowly?
You know, I have a big model I

449
00:23:40,040 --> 00:23:43,600
can train it in in domain A
domain A could be weather and

450
00:23:43,600 --> 00:23:46,240
climate have gotten attention,
but it could be fluid dynamics.

451
00:23:46,240 --> 00:23:48,640
It could be something else about
stylist formation.

452
00:23:48,640 --> 00:23:51,920
It could be properties and
materials and doing density

453
00:23:51,920 --> 00:23:54,400
functional theory and so on.
And then there's a question

454
00:23:54,400 --> 00:23:57,480
about whether you could maybe
learn broad based models that

455
00:23:57,480 --> 00:23:59,280
cut across domains, which would
of course be the more

456
00:23:59,280 --> 00:24:03,440
interesting thing.
So there's something Kronos

457
00:24:03,440 --> 00:24:06,360
said, I was involved with the
AWS people that sort of says, I,

458
00:24:06,360 --> 00:24:10,400
I don't want to be the best at
at the most extreme things.

459
00:24:10,400 --> 00:24:12,800
I don't want to be the 1% that's
predicting the most extreme

460
00:24:12,800 --> 00:24:16,120
things I want to be.
I want to do as well as as the

461
00:24:16,120 --> 00:24:20,000
majority of users for for time
series analysis and oversimplify

462
00:24:20,000 --> 00:24:21,240
the story.
I mean, time series is a

463
00:24:21,240 --> 00:24:24,680
complicated area clearly of
interest in a lot of cases, but

464
00:24:24,680 --> 00:24:26,320
it's it's a little bit, you
know, you got to be careful

465
00:24:26,320 --> 00:24:28,880
about off by 1 errors and a
range of other technical things.

466
00:24:29,440 --> 00:24:33,880
And what that says basically is
take a language model, language

467
00:24:33,880 --> 00:24:37,600
model structure with a bunch of
time series and just mean center

468
00:24:37,600 --> 00:24:40,640
it and variance normalize it.
So just do the simplest possible

469
00:24:40,640 --> 00:24:44,600
things you could and you got to
do some data augmentation stuff

470
00:24:44,600 --> 00:24:47,120
and boom, you know, you do sort
of comparably and, and better

471
00:24:47,120 --> 00:24:49,200
than a a wide range of public
benchmarks.

472
00:24:49,200 --> 00:24:52,960
Now, the public benchmarks are
not extraordinarily high because

473
00:24:52,960 --> 00:24:55,320
a lot of time series data is
valuable and so companies tend

474
00:24:55,320 --> 00:24:58,600
not to release it.
But but the fact that you can do

475
00:24:58,600 --> 00:25:01,920
that just by sort of variance
normalizing and and mean

476
00:25:01,920 --> 00:25:04,920
centering is is, you know, not
obvious and sort of interesting.

477
00:25:05,600 --> 00:25:10,080
It turns out sort of on a
separate thread that you could

478
00:25:10,080 --> 00:25:12,760
say, could I do well compared to
the 1%, you know, the most

479
00:25:12,760 --> 00:25:16,480
extreme things and, and the the
answers that you can there too.

480
00:25:16,520 --> 00:25:18,400
You got to use different
techniques than you do there.

481
00:25:18,960 --> 00:25:23,400
And so that leads to questions
as to whether you could have a

482
00:25:23,400 --> 00:25:27,800
foundation model for sort of a
broad swaths of, of time series

483
00:25:27,800 --> 00:25:31,200
and forecasting analysis.
And so, as you know, I said, I'm

484
00:25:31,200 --> 00:25:33,480
an Amazon scholar working with
the Scott team and we're looking

485
00:25:33,480 --> 00:25:36,320
at that there.
And one of the interesting

486
00:25:36,320 --> 00:25:38,840
things I think about there, if
you look at the details of the

487
00:25:38,840 --> 00:25:42,120
model, not the chronos, but some
other things, why would you

488
00:25:42,120 --> 00:25:46,600
expect that article scraped from
Wikipedia, you know, public,

489
00:25:47,640 --> 00:25:49,360
publicly available language
models?

490
00:25:49,760 --> 00:25:52,760
What would, why would that allow
you to do better job predicting

491
00:25:52,760 --> 00:25:54,800
dog food demand?
You know, I mean, it's not

492
00:25:54,800 --> 00:25:56,320
obvious they have anything to do
with anything.

493
00:25:56,960 --> 00:25:59,920
You know, it turns out, I
suspect the hypothesis is that

494
00:26:00,400 --> 00:26:03,560
the text, the linguistic
structure that you're learning

495
00:26:03,880 --> 00:26:08,000
from the Wikipedia articles is
strongly related to sequence to

496
00:26:08,000 --> 00:26:10,320
sequence modeling, which, you
know, if you think of 1

497
00:26:10,320 --> 00:26:13,800
dimension as a metric space,
it's a very, very special metric

498
00:26:13,800 --> 00:26:15,920
space, a very specially
structured metric space.

499
00:26:16,600 --> 00:26:19,720
And so you can learn not just
recent information and not just

500
00:26:19,720 --> 00:26:22,880
low frequency information over
the past, but maybe in

501
00:26:23,520 --> 00:26:26,480
information at different scales.
Because, you know, sometimes

502
00:26:26,560 --> 00:26:30,760
articles in text refer back 10
words or 100 words or 1000 words

503
00:26:30,760 --> 00:26:32,240
or 10,000 words.
So you can learn these sort of

504
00:26:32,240 --> 00:26:35,720
heavy tailed sort of structures
and that gives you a better set

505
00:26:35,720 --> 00:26:38,800
of basis functions to learn dog
food demand or whatever else.

506
00:26:38,800 --> 00:26:42,000
So in a sense, these, the
language models give you better

507
00:26:42,000 --> 00:26:46,880
embeddings, you know, 4A
analysis and Laplace transforms

508
00:26:46,880 --> 00:26:49,080
and, and these sort of things
are great for pencil and paper.

509
00:26:49,080 --> 00:26:50,800
They're all developed in the
1800s, right?

510
00:26:52,080 --> 00:26:54,720
These are data-driven embeddings
that are good if you have

511
00:26:54,720 --> 00:26:57,440
computers and and data not
necessary for pencil and paper,

512
00:26:57,440 --> 00:26:59,840
but they're good for that.
And so that leads you to the

513
00:26:59,840 --> 00:27:03,880
question about if, if I'm really
careful about doing foundation

514
00:27:03,880 --> 00:27:05,880
model work in science, what do I
need to worry about?

515
00:27:06,160 --> 00:27:08,520
It's not the details of this PDE
or that PDE.

516
00:27:08,800 --> 00:27:12,480
It might be that I want to
reproduce what happens in the

517
00:27:12,480 --> 00:27:16,160
natural language processing the
NLP models, which is roughly

518
00:27:16,160 --> 00:27:18,120
scale model and data and
compute.

519
00:27:18,120 --> 00:27:20,480
So none of them saturate.
That's very different than the

520
00:27:20,480 --> 00:27:23,000
usual strong scaling and weak
scaling and high performance

521
00:27:23,000 --> 00:27:26,080
computing.
I want to change the amount of

522
00:27:26,080 --> 00:27:28,840
data I'm putting in.
I want to change the size of the

523
00:27:28,840 --> 00:27:30,160
model.
I want to change the size as it

524
00:27:30,160 --> 00:27:31,800
computes.
So none of them saturate and,

525
00:27:31,800 --> 00:27:34,920
and the conjecture would be that
if if any of them saturate, it's

526
00:27:34,920 --> 00:27:36,680
going to be harder to do
transfer learning.

527
00:27:36,680 --> 00:27:40,600
So you really need the non
saturation and you do that one

528
00:27:40,600 --> 00:27:44,240
way with NLP, natural language
processing and CV computer

529
00:27:44,240 --> 00:27:48,280
vision.
But an NLP and CV, there's much

530
00:27:48,280 --> 00:27:52,400
weaker control you have and that
you need on the spatiotemporal

531
00:27:52,400 --> 00:27:55,800
geometry than PD ES, right?
Is, is, you know, PD ES, if

532
00:27:55,800 --> 00:27:58,200
you're below a nice a Mach
transition or some other

533
00:27:58,200 --> 00:28:00,000
physical transition, things are
sort of smooth.

534
00:28:00,000 --> 00:28:01,640
If you're above it, things are
very messy.

535
00:28:02,040 --> 00:28:04,200
Dealing with that transition is
hard and people spend their

536
00:28:04,200 --> 00:28:06,840
whole careers dealing with that.
So can you come up with data

537
00:28:06,840 --> 00:28:10,120
generation or tokenization
mechanisms that that respect the

538
00:28:10,120 --> 00:28:12,960
spatio temporal properties?
So I think if you're going to

539
00:28:12,960 --> 00:28:16,160
have a foundation model that
applies across a raw broad range

540
00:28:16,160 --> 00:28:19,680
of sciences that's trained on
weather and, and, and climate

541
00:28:19,680 --> 00:28:23,160
and, and simulations of
different grid structure from

542
00:28:23,160 --> 00:28:24,640
the machine learning
perspective, it's not so

543
00:28:24,640 --> 00:28:28,680
different than satellite data
from satellites looking down,

544
00:28:28,800 --> 00:28:30,960
you know, it's a different grid
and it's an image.

545
00:28:31,280 --> 00:28:33,560
And then you say, how does that
couple with the spatiotemporal

546
00:28:33,560 --> 00:28:35,720
properties?
So I think if, if you want to

547
00:28:35,720 --> 00:28:38,360
deliver on the promise in the
same way is, is the is computer

548
00:28:38,360 --> 00:28:40,880
science had to work out a range
of numerical methods to really

549
00:28:41,520 --> 00:28:43,080
match and beat state-of-the-art,
you're going to have to do a

550
00:28:43,080 --> 00:28:46,000
similar thing here.
And so partly depends on these

551
00:28:46,000 --> 00:28:48,600
technical issues, but partly
depends on cultural issues.

552
00:28:49,080 --> 00:28:51,560
But how much do you think?
I mean you raised the what

553
00:28:51,560 --> 00:28:57,640
percent versus the 50%?
I guess that is one argument as

554
00:28:57,640 --> 00:29:02,600
well, which is how close does it
need to be to machine accuracy

555
00:29:02,680 --> 00:29:05,520
to be useful.
You could argue that any of

556
00:29:05,520 --> 00:29:09,520
these large language models, the
way that most people use them,

557
00:29:09,520 --> 00:29:12,080
there is still a correction you
need to add.

558
00:29:12,360 --> 00:29:15,640
You don't typically ask it to do
something right, document and

559
00:29:15,640 --> 00:29:18,040
literally take it word for word.
You usually go in and go.

560
00:29:18,040 --> 00:29:21,680
That's, that's remarkably close,
but I'm still going to go and

561
00:29:21,680 --> 00:29:24,240
fix it.
Or it writes you a Python code.

562
00:29:24,240 --> 00:29:26,200
It's unusual, isn't it, that
it's perfect.

563
00:29:26,240 --> 00:29:29,080
There's usually something you
have to correct.

564
00:29:29,080 --> 00:29:34,400
So I guess with the science
side, maybe the argument is how

565
00:29:34,400 --> 00:29:40,840
close does it need to be to be
useful And the cost of getting

566
00:29:40,840 --> 00:29:43,920
the incremental increase in
accuracy, you know, is it, is

567
00:29:43,920 --> 00:29:47,320
there a like a trade off where
you need so much more?

568
00:29:47,320 --> 00:29:51,360
Data, I mean, again, I think the
best analogy is look at the

569
00:29:51,360 --> 00:29:53,520
history of computer science and
how that of all that, I think, I

570
00:29:53,520 --> 00:29:56,120
think framing the question to
say how close does it need to be

571
00:29:56,120 --> 00:29:59,800
to be useful?
You're already framing it in a

572
00:29:59,800 --> 00:30:05,600
way that that makes certain one
group comfortable, the numerical

573
00:30:05,600 --> 00:30:08,240
analysts and the PD people that
frame a question a certain way.

574
00:30:08,640 --> 00:30:11,280
That's not how people would have
framed the question before.

575
00:30:11,320 --> 00:30:14,920
I mean that, you know, before
you had, you know, represented

576
00:30:15,000 --> 00:30:19,080
continuous numbers on, on a
computer discreetly take a step

577
00:30:19,080 --> 00:30:21,120
back and say, I want to solve a
certain problem.

578
00:30:21,120 --> 00:30:24,840
And, and the question is now
that I have very different trade

579
00:30:24,840 --> 00:30:28,320
off in terms of compute versus
data and I have this new

580
00:30:28,320 --> 00:30:29,960
infrastructure that's language
models.

581
00:30:30,280 --> 00:30:34,920
Can I ask a different question
and, and, and push the science

582
00:30:34,920 --> 00:30:37,440
for it?
I mean, so in, for example, in,

583
00:30:39,160 --> 00:30:42,240
in chemistry historically, but
also nucleus physics and, and in

584
00:30:42,240 --> 00:30:44,480
fluid mechanics, there's this
notion of a semi empirical

585
00:30:44,480 --> 00:30:48,520
theory, which is a theory sort
of derived, you know, it's not

586
00:30:48,520 --> 00:30:51,560
just curve fitting, it's derived
from an underlying more

587
00:30:51,560 --> 00:30:54,200
fundamental theory, maybe
phenomenologically with

588
00:30:54,200 --> 00:30:56,880
parameters that are then fit
empirically or semi empirically.

589
00:30:57,160 --> 00:31:01,920
And So what you need there is
not the theory to be right, but

590
00:31:01,920 --> 00:31:04,000
sort of right enough at the
level of in the, in the

591
00:31:04,000 --> 00:31:06,320
chemistry is chemical accuracy,
which is, is however many

592
00:31:06,320 --> 00:31:08,760
kilovolts or whatever depends on
the reaction you're interested

593
00:31:08,760 --> 00:31:11,320
in so on.
And so certain methods that

594
00:31:11,320 --> 00:31:14,000
maybe were more principled were
a bit too course for that.

595
00:31:14,000 --> 00:31:15,880
And other methods, when you
combined it with other

596
00:31:15,880 --> 00:31:18,040
techniques achieved chemical
accuracy.

597
00:31:18,320 --> 00:31:20,520
And so none of them were low
enough in the stack that you

598
00:31:20,520 --> 00:31:23,720
they were better QR codes for QR
from, from linear algebra.

599
00:31:24,200 --> 00:31:26,440
But they solved the downstream
problem at that level of

600
00:31:26,440 --> 00:31:28,200
actually, probably what you'd
see here is that right?

601
00:31:28,200 --> 00:31:31,800
So it's, it's probably not going
to be so successful to say I

602
00:31:31,800 --> 00:31:35,280
want to get 10 to the -16 or 10
to the -2 of my large language

603
00:31:35,280 --> 00:31:38,680
model, but but a foundation
model for science and need not

604
00:31:38,680 --> 00:31:40,200
be an LLM.
It, it, it, you could be

605
00:31:40,200 --> 00:31:42,800
literally learning the
embeddings from a range of PD

606
00:31:42,800 --> 00:31:44,720
ES.
You could, you could train to

607
00:31:44,720 --> 00:31:47,160
PDS of different types.
And you could say, I want to

608
00:31:47,160 --> 00:31:50,680
port if I, if I know the basics
of hyperbolic and parabolic.

609
00:31:50,680 --> 00:31:53,440
And I know that in the transport
equation, this can enter in a

610
00:31:53,440 --> 00:31:55,080
certain way.
And you have nonlinearities and

611
00:31:55,080 --> 00:31:57,560
certain types of forcing
functions, you know, model each

612
00:31:57,560 --> 00:32:00,760
of those and and solve each of
the components separately.

613
00:32:00,760 --> 00:32:02,200
So I think the question is,
what's the right level of

614
00:32:02,200 --> 00:32:05,160
abstraction to do that?
And, and one of the modalities

615
00:32:05,160 --> 00:32:07,680
you could use is language, but
of course you could use image or

616
00:32:07,680 --> 00:32:09,960
PD ES or simulations, any of a
range of things.

617
00:32:10,400 --> 00:32:13,840
So I think in the context of, of
scientific problems that the

618
00:32:13,840 --> 00:32:18,280
foundation model need not be an
LLM based.

619
00:32:18,280 --> 00:32:20,080
I mean, there's some people are
pushing that, I think, but but

620
00:32:20,080 --> 00:32:21,920
it certainly need not be LLM
based.

621
00:32:21,920 --> 00:32:23,880
We just query the model and say,
you know, tell me what to look

622
00:32:23,880 --> 00:32:29,360
for, the top quark or whatever.
Well that that's the bit maybe

623
00:32:29,360 --> 00:32:31,400
you brought up would be
interesting to explore.

624
00:32:31,400 --> 00:32:36,280
There is, AI would say, a
relatively fierce or contested

625
00:32:36,400 --> 00:32:42,280
argument around physics informed
or physics in it versus

626
00:32:42,280 --> 00:32:48,600
data-driven.
Where do you sit on that, on the

627
00:32:48,600 --> 00:32:49,440
argument?
Do you, do you?

628
00:32:49,880 --> 00:32:53,240
Does the model need to
implicitly have some of the

629
00:32:53,240 --> 00:32:56,080
boundary conditions, some
awareness of continuity of

630
00:32:56,080 --> 00:32:57,880
energy?
Or is it good enough just to

631
00:32:57,880 --> 00:33:02,880
give it so much data that it
essentially learns those laws by

632
00:33:02,880 --> 00:33:06,840
itself?
Yeah, I mean, I, I think you

633
00:33:06,840 --> 00:33:09,640
could ask the same question
about in natural language

634
00:33:09,640 --> 00:33:15,280
processing, computer vision does
did you just, I mean, there's a,

635
00:33:15,320 --> 00:33:17,640
there's a common story people
tell which is just get more data

636
00:33:17,640 --> 00:33:20,800
and everything works.
But that ignores the fact as you

637
00:33:20,800 --> 00:33:23,200
know, that a lot of companies
and universities, a lot of

638
00:33:23,200 --> 00:33:27,800
people have put a lot of
resources into NLP and CV to try

639
00:33:27,800 --> 00:33:29,000
and figure out how to make it
work.

640
00:33:29,440 --> 00:33:33,400
And you know, convolutions
convolve things, they spread

641
00:33:33,400 --> 00:33:35,440
stuff out.
You know, if you have an image

642
00:33:35,440 --> 00:33:38,320
in 2D that may make sense, you
might want to average over

643
00:33:38,320 --> 00:33:40,120
nearby things.
But if there's, if there's a

644
00:33:40,120 --> 00:33:44,080
sharp corner like, you know, the
shirt against the background

645
00:33:44,080 --> 00:33:47,640
wall in this picture in my of
me, you may smudge stuff out.

646
00:33:47,640 --> 00:33:49,840
And so there's a range of ways
to try and sort of deal with

647
00:33:49,840 --> 00:33:51,680
that.
That's core structure.

648
00:33:51,680 --> 00:33:54,080
The 2D structure that you're
trying to average over is very

649
00:33:54,080 --> 00:33:56,080
different than sequence to
sequence learning.

650
00:33:56,560 --> 00:33:59,200
That's also much more discreet
in, in natural language

651
00:33:59,200 --> 00:34:00,800
processing.
So it's not like computer vision

652
00:34:00,800 --> 00:34:02,720
or natural language processing,
just learn stuff.

653
00:34:03,160 --> 00:34:05,440
You gave it very particular
architectures that had very

654
00:34:05,440 --> 00:34:08,520
particular adaptive biases and
then you're careful about the

655
00:34:08,520 --> 00:34:12,000
data and, and trying to get gobs
of it and, and then then it

656
00:34:12,000 --> 00:34:13,719
worked.
So I think if you, you know,

657
00:34:13,719 --> 00:34:15,080
you'd have to follow the same
path here.

658
00:34:15,080 --> 00:34:18,040
You can't just take an existing
architecture and press a button

659
00:34:18,040 --> 00:34:20,520
and hope it works.
I mean, time series, forecasting

660
00:34:20,520 --> 00:34:24,760
space, you temporal modelling,
these sorts of things will have

661
00:34:24,760 --> 00:34:27,440
to be, you have to figure that
out, how to do that.

662
00:34:27,719 --> 00:34:31,600
So I think the question, I mean,
some people, you know, I think

663
00:34:31,600 --> 00:34:35,040
are, and, and maybe sometimes
people that have a vested

664
00:34:35,040 --> 00:34:37,239
interest in, in moving forward
or not.

665
00:34:37,520 --> 00:34:39,360
I mean, some people say, you
know, just ignore everything,

666
00:34:39,360 --> 00:34:41,000
let the data do it all.
Some people say, Oh, you'll

667
00:34:41,000 --> 00:34:43,960
never match what I can do with
my careful PD solver.

668
00:34:45,159 --> 00:34:47,600
I mean, I think that's neither
of those questions is the right

669
00:34:47,600 --> 00:34:48,880
one.
I mean, the question is, given

670
00:34:48,880 --> 00:34:51,239
the fact that there's an
infrastructure of code and

671
00:34:51,239 --> 00:34:53,760
experience numerically, but but
you're also at a very different

672
00:34:53,760 --> 00:34:55,520
place.
You've been sitting on top of

673
00:34:55,520 --> 00:35:00,600
certain essentially hardware
and, and, and linear algebraic

674
00:35:00,600 --> 00:35:02,600
advances.
I mean, for years, for decades.

675
00:35:02,960 --> 00:35:07,280
People working on PDS call
certain linear algebra libraries

676
00:35:07,280 --> 00:35:10,080
and have essentially had not to
pay the technical debt

677
00:35:10,080 --> 00:35:13,080
associated with the fairly high
level of of complexity with

678
00:35:13,120 --> 00:35:15,360
developing new linear algebra
because just wait a year or two

679
00:35:15,360 --> 00:35:19,160
and the machines are faster.
And so if, if, if that's not the

680
00:35:19,160 --> 00:35:20,840
case, you're going to have to
start thinking a little bit more

681
00:35:20,840 --> 00:35:23,880
carefully about the underlying
linear algebraic computations.

682
00:35:24,120 --> 00:35:26,640
Maybe there's different
trade-offs points in the space

683
00:35:27,040 --> 00:35:28,600
and maybe bring something else
to bear.

684
00:35:28,600 --> 00:35:33,400
You know, a model that has
learned certain course type

685
00:35:33,400 --> 00:35:36,760
functions of one dimension, say
from the language model that I

686
00:35:36,760 --> 00:35:39,920
alluded to before that aren't
Fourier modes or aren't Laplace

687
00:35:39,920 --> 00:35:41,920
modes or aren't something else
like that that they're more

688
00:35:41,920 --> 00:35:45,360
familiar with, learn a different
grid type discretization.

689
00:35:45,800 --> 00:35:50,640
So I think the real challenge in
delivering on the promise of all

690
00:35:50,640 --> 00:35:53,440
this stuff is figuring out what
the right way to combine those

691
00:35:53,440 --> 00:35:54,680
two.
So you can imagine rather than

692
00:35:54,680 --> 00:35:57,520
just writing down a physics
informed loss and pressing a

693
00:35:57,520 --> 00:36:00,240
button saying, well, the way
people actually would solve this

694
00:36:00,240 --> 00:36:02,160
is to have some sort of
splitting method and deal with

695
00:36:02,160 --> 00:36:04,600
the two types of physical things
and the diffusion and invection

696
00:36:04,600 --> 00:36:09,000
in two slightly different ways.
And machine one and one and take

697
00:36:09,000 --> 00:36:10,800
the other as numerical.
And then you couple them

698
00:36:10,800 --> 00:36:13,400
together, like maybe in a
differentiable end to end.

699
00:36:13,400 --> 00:36:15,240
Yeah.
The, the reason I ask this is

700
00:36:15,240 --> 00:36:19,120
because it does feel certainly
at least in the CFD community,

701
00:36:19,120 --> 00:36:24,560
that there's a, there's a real
sort of split within the

702
00:36:24,560 --> 00:36:27,640
community.
And in a sense that on one hand

703
00:36:27,640 --> 00:36:31,280
you have people saying, well, we
have all these PDE solvers, you

704
00:36:31,280 --> 00:36:38,600
know, developed for decades.
And you know what convinced me

705
00:36:39,320 --> 00:36:45,080
how your machine learning or AI
method is going to be as good

706
00:36:46,200 --> 00:36:52,840
and faster and cheaper versus on
the other hand, you have quite

707
00:36:52,840 --> 00:36:58,280
sensational claims of, you know,
10,000 times faster, you know,

708
00:36:58,280 --> 00:37:01,000
sort of real time.
Well, of course, when you dig

709
00:37:01,000 --> 00:37:03,120
in, you know, you start asking
the questions, well, how much

710
00:37:03,120 --> 00:37:06,040
data did you have?
What was the time to create the

711
00:37:06,040 --> 00:37:08,920
data?
And but there is a bigger

712
00:37:08,920 --> 00:37:14,560
question, I think more when you
look to the future, which is, is

713
00:37:14,560 --> 00:37:19,800
it just a matter of time?
Is it a bit like in the 1980s,

714
00:37:20,440 --> 00:37:25,280
they could only simulate a plane
to a reasonably low accuracy

715
00:37:25,280 --> 00:37:27,360
because the computers just
weren't power enough.

716
00:37:27,680 --> 00:37:30,920
And as you said, you just almost
keep the same effort almost.

717
00:37:31,320 --> 00:37:35,280
And you just literally add 20
years of compute on top and now

718
00:37:35,280 --> 00:37:41,640
you can simulate.
Or is it a we'll never get there

719
00:37:41,640 --> 00:37:46,000
because we'll it's like an it's
like impossible to have that

720
00:37:46,000 --> 00:37:48,440
much data?
This is the bit I think a lot of

721
00:37:48,440 --> 00:37:52,560
BCS and start-ups are also
asking that question if we pump

722
00:37:52,560 --> 00:37:55,280
100 million into this or a
billion, will we fix it?

723
00:37:55,720 --> 00:37:58,240
Or is it a trillion dollar
problem and it's not worth

724
00:37:58,240 --> 00:37:59,960
doing?
Yeah, Yeah.

725
00:37:59,960 --> 00:38:01,600
I mean, OK.
So I think there's a there's a

726
00:38:01,600 --> 00:38:02,960
bunch of things in the question
there.

727
00:38:03,600 --> 00:38:06,600
Sorry.
We'll we'll CFD people to BC and

728
00:38:06,600 --> 00:38:10,280
and you know that they have two
very different utility.

729
00:38:10,680 --> 00:38:13,720
You.
Know, I think, OK, so it when

730
00:38:13,720 --> 00:38:16,640
someone says to me do this,
convince me of that I have these

731
00:38:16,640 --> 00:38:19,480
metrics that have been I've
worked on for decades, I just

732
00:38:19,480 --> 00:38:20,880
say thank you, good to talk to
you.

733
00:38:20,880 --> 00:38:23,120
I mean, because, because there's
no way you're going to win that

734
00:38:23,120 --> 00:38:24,920
battle because they're very
fine-tuned metrics.

735
00:38:24,920 --> 00:38:26,800
So they're very one little use
case.

736
00:38:27,280 --> 00:38:29,760
Even if you do win that battle,
they're not going to admit it

737
00:38:30,040 --> 00:38:32,320
because they'll, they'll tweak
their method and, and you know,

738
00:38:32,320 --> 00:38:35,280
be slightly better than you.
And there's greener pastures few

739
00:38:35,280 --> 00:38:37,720
everywhere.
And this is not a hypothetical

740
00:38:37,720 --> 00:38:39,120
statement.
This is a very practical thing.

741
00:38:39,120 --> 00:38:41,040
You're asking about this in the
context of PDS.

742
00:38:41,280 --> 00:38:43,440
We saw this before in randomized
linear algebra.

743
00:38:43,440 --> 00:38:46,520
I mean, so literally, I remember
it was very clear that the

744
00:38:46,520 --> 00:38:49,960
techniques could potentially be
useful, not just in theory, but

745
00:38:49,960 --> 00:38:53,120
in practice, and that the reason
the numerical people were

746
00:38:53,120 --> 00:38:54,680
saying, oh, they won't work, You
know, really.

747
00:38:54,680 --> 00:38:57,920
I mean, those are those number
of methods that clearly wouldn't

748
00:38:57,920 --> 00:38:59,560
work.
But but then there's the next

749
00:38:59,560 --> 00:39:01,080
generation of methods.
And that's sort of when I

750
00:39:01,080 --> 00:39:02,840
entered the the area and it was
clear that they could

751
00:39:02,840 --> 00:39:04,800
potentially work and the
objections people had before

752
00:39:04,800 --> 00:39:07,000
wouldn't work.
And so this was right around the

753
00:39:07,000 --> 00:39:09,200
time of the Blunden pic paper
that I mentioned that that

754
00:39:09,200 --> 00:39:10,680
basically said, can you beat LA
pack?

755
00:39:10,680 --> 00:39:13,920
And the short answer was yes.
So now, for example, if you look

756
00:39:13,920 --> 00:39:16,120
at the Simon linear Alger
meeting, half the talks are on

757
00:39:16,120 --> 00:39:18,480
this topic, but there's been
this process of creative

758
00:39:18,480 --> 00:39:21,720
destruction where this they're
still asking the same questions,

759
00:39:21,720 --> 00:39:24,120
but putting randomness there.
And so that's one group of

760
00:39:24,200 --> 00:39:25,640
community.
I mean, a very different

761
00:39:25,640 --> 00:39:28,360
community is the is the people
that say, I'm going to not do

762
00:39:28,360 --> 00:39:29,440
that.
I'm going to go and apply it to

763
00:39:29,440 --> 00:39:31,040
machine learning and 10 other
problems.

764
00:39:31,480 --> 00:39:35,120
And, and, and so there's that
tension I was alluding to

765
00:39:35,120 --> 00:39:38,320
before.
So I think if you, if you try

766
00:39:38,320 --> 00:39:42,560
and, and satisfy an old school
metric, it's, it's good to have

767
00:39:42,560 --> 00:39:44,520
a few examples of that as a
proof of principle.

768
00:39:44,520 --> 00:39:47,240
And, and this is 15 years later,
and I mentioned it twice here,

769
00:39:47,240 --> 00:39:48,560
right?
So this was a clear proof of

770
00:39:48,560 --> 00:39:52,800
principle of the area.
The area sort of accelerated

771
00:39:52,800 --> 00:39:54,520
after that.
And there's been a lot of theory

772
00:39:54,520 --> 00:39:57,360
and empirical development.
So I think there's been a few

773
00:39:57,360 --> 00:39:59,720
things that, if not that are
starting to look almost like

774
00:39:59,720 --> 00:40:02,040
that in this general scientific
ML area.

775
00:40:04,160 --> 00:40:06,440
I I think.
History is moving in that

776
00:40:06,440 --> 00:40:08,640
direction.
So it's clear then that, you

777
00:40:08,640 --> 00:40:10,440
know, randomness would be
important for linear algebra.

778
00:40:10,440 --> 00:40:12,680
Now it's even more so for all
the historical and hardware

779
00:40:12,680 --> 00:40:15,440
trends you were talking about.
I think likely the same thing's

780
00:40:15,560 --> 00:40:18,920
going to happen here, right?
And so a question you could ask

781
00:40:18,920 --> 00:40:22,280
is, are the people who know the
linear algebra, are the people

782
00:40:22,280 --> 00:40:24,400
who know the computational fluid
dynamics?

783
00:40:24,760 --> 00:40:26,400
Are they at the table developing
the methods?

784
00:40:26,400 --> 00:40:28,360
Are they just going to Pooh,
Pooh it and say, well, they're

785
00:40:28,360 --> 00:40:30,640
not going to work because it's
not going to satisfy my

786
00:40:30,640 --> 00:40:34,960
particular measure as opposed to
here's a broad class of

787
00:40:34,960 --> 00:40:38,000
techniques and it's hard to
imagine that there's nothing in

788
00:40:38,000 --> 00:40:39,640
my area that will be improved by
them.

789
00:40:40,320 --> 00:40:42,760
And so I think that's the that
that latter question is the

790
00:40:42,760 --> 00:40:44,280
question to ask.
And that latter is the question

791
00:40:44,280 --> 00:40:47,040
the questions VCs and and other
people will ask.

792
00:40:47,640 --> 00:40:53,200
And I think the, an important
maybe determinant of how this

793
00:40:53,200 --> 00:40:56,880
will evolve is, is how the
different players interact with

794
00:40:56,880 --> 00:41:01,280
this in, in the sense that a lot
of the machine learning, more

795
00:41:01,280 --> 00:41:03,720
broadly, scientific machine
learning, not just for the

796
00:41:03,720 --> 00:41:06,680
foundation model, but more
broadly machine learning really

797
00:41:06,680 --> 00:41:11,840
is, really is a horizontal.
I mean, it's, it's designed to

798
00:41:11,840 --> 00:41:15,480
say, what can I do with lots of
data and be relatively ignorant

799
00:41:15,480 --> 00:41:18,000
about your application area?
Because who knows why people

800
00:41:18,000 --> 00:41:20,800
click on ads or click on links
on their social media account,

801
00:41:20,800 --> 00:41:21,840
right?
I mean, you can tell the story,

802
00:41:21,840 --> 00:41:28,320
but really who knows why?
And so a lot of machine learners

803
00:41:28,320 --> 00:41:30,560
implicitly or explicitly will
say, I'm not interested in

804
00:41:30,560 --> 00:41:33,280
solving your scientific problem
if it's just a one off problem.

805
00:41:33,280 --> 00:41:34,960
I mean, if you're a scientist
using machine learning, you

806
00:41:34,960 --> 00:41:36,680
might be interested in that, but
that's because you're interested

807
00:41:36,680 --> 00:41:38,120
in the domain.
But if you're a machine learner,

808
00:41:38,520 --> 00:41:41,320
you might say, I mean, I see
some commonalities between your

809
00:41:41,320 --> 00:41:44,560
fluid dynamics and this other
solid mechanics problem.

810
00:41:44,960 --> 00:41:48,240
And I see some similarities
there between that and something

811
00:41:48,240 --> 00:41:51,880
in in 3DS in the atmosphere as
opposed to, you know, something

812
00:41:51,880 --> 00:41:54,640
very different.
And so identifying those

813
00:41:54,640 --> 00:41:56,080
commonalities I think is
important.

814
00:41:56,240 --> 00:41:58,880
What I see in, in in some cases,
and this is Hanford by

815
00:41:58,880 --> 00:42:02,240
universities and it's Hanford by
industry, and it's Hanford in a

816
00:42:02,240 --> 00:42:05,840
couple government labs in, in a
couple different ways, is a lot

817
00:42:05,840 --> 00:42:09,560
of people want to solve a
scientific problem.

818
00:42:09,560 --> 00:42:12,640
And so they're going to say, I'm
only going to invest, invest

819
00:42:12,640 --> 00:42:15,200
literally or figuratively in
machine learning in that one

820
00:42:15,200 --> 00:42:17,240
area.
And that's just not how machine

821
00:42:17,240 --> 00:42:19,720
learning works.
You don't see the, I mean, think

822
00:42:19,720 --> 00:42:22,080
of it as horizontal business
with horizontals and verticals.

823
00:42:22,360 --> 00:42:24,160
If machine learning is a
horizontal, like high

824
00:42:24,160 --> 00:42:26,760
performance computing that will
solve a wide range of problems,

825
00:42:27,120 --> 00:42:28,960
you got to apply it to a
vertical to justify it.

826
00:42:28,960 --> 00:42:32,120
That's that's where that that's
where you know, the rubber hits

827
00:42:32,120 --> 00:42:35,280
the road typically.
And so you don't see the

828
00:42:35,280 --> 00:42:39,280
successes in machine learning,
scientific machine learning at

829
00:42:39,920 --> 00:42:43,120
vertical companies, you know,
companies working in genetics or

830
00:42:43,120 --> 00:42:47,640
or oil exploration or any of a
range of other particular domain

831
00:42:47,640 --> 00:42:49,760
verticals.
You see it in the horizontal

832
00:42:49,760 --> 00:42:51,320
companies.
The companies have built out the

833
00:42:51,320 --> 00:42:53,960
technology infrastructure and
they get a team of people that

834
00:42:53,960 --> 00:42:56,120
know this is a vertical and
applies it there.

835
00:42:56,320 --> 00:42:59,240
And then, you know, a team that
knows a different vertical and

836
00:42:59,240 --> 00:43:01,680
applies it there.
And so I think the question is,

837
00:43:01,680 --> 00:43:03,360
how does that play out?
And that's going to be a non

838
00:43:03,360 --> 00:43:06,640
trivial evolution in terms of.
And your.

839
00:43:06,800 --> 00:43:09,120
That day.
Your comment was interesting as

840
00:43:09,120 --> 00:43:13,840
well about the difference
between sort of Transformers

841
00:43:13,840 --> 00:43:17,880
versus CNNS and you know, if you
just throw the whole world's

842
00:43:17,880 --> 00:43:22,480
Internet at CNN, that isn't you
needed to have the right

843
00:43:22,480 --> 00:43:26,440
architecture to be able to throw
a lot of data at it.

844
00:43:26,960 --> 00:43:29,120
I guess the question is more for
the science.

845
00:43:29,160 --> 00:43:33,760
Is there a common architecture
across the sciences that then

846
00:43:33,760 --> 00:43:37,720
would allow you to sort of build
out that big which could be

847
00:43:37,720 --> 00:43:41,240
applied to verticals?
Or is it the case that the needs

848
00:43:41,240 --> 00:43:47,160
for weather or genetics or
chemistry are so different that

849
00:43:47,160 --> 00:43:50,400
you would need a different
architecture for for each

850
00:43:50,400 --> 00:43:52,800
different size?
And therefore the, the, the

851
00:43:52,920 --> 00:43:54,960
dream of a true foundational
thing.

852
00:43:54,960 --> 00:43:58,040
Besides, it will never happen
because you know.

853
00:43:58,960 --> 00:44:00,200
Yeah.
So that's the big question.

854
00:44:00,200 --> 00:44:04,080
And I, I think there's a
technical component to that and

855
00:44:04,080 --> 00:44:06,080
then non-technical component in
terms of the areas, right?

856
00:44:06,080 --> 00:44:09,160
And so technically it's
open-ended.

857
00:44:09,160 --> 00:44:12,720
I don't know, right?
I mean, we're, we're in the

858
00:44:12,720 --> 00:44:14,920
middle of the fray and we're,
we're voting with our feet and

859
00:44:14,920 --> 00:44:17,920
placing a bet there that the,
that the answer is or could be

860
00:44:17,920 --> 00:44:20,640
yes.
To try and address that, you're

861
00:44:20,640 --> 00:44:22,320
going to have to deal with the
issues we were talking about

862
00:44:22,320 --> 00:44:24,920
before.
You know, one dimension is very,

863
00:44:24,920 --> 00:44:27,400
very special. 2 dimension, the
surface area to volume

864
00:44:27,400 --> 00:44:29,880
properties are very different. 3
dimensions is very different.

865
00:44:30,200 --> 00:44:33,160
By the time you get 4 in
scientific computing, that's

866
00:44:33,160 --> 00:44:36,080
sort of infinite, right?
Machine learners, by the time

867
00:44:36,080 --> 00:44:38,200
until you get to a million, you
know, it still seems pretty

868
00:44:38,200 --> 00:44:41,480
small.
So mapping the methods to one

869
00:44:41,480 --> 00:44:44,640
versus 2 versus 3 dimensions,
open question, maybe there'll be

870
00:44:44,640 --> 00:44:47,040
no trade off point where the
surface area to volume and the

871
00:44:47,040 --> 00:44:49,000
isoperimetry and other things
like that wins.

872
00:44:49,000 --> 00:44:51,360
I suspect not.
I suspect it could, but I mean,

873
00:44:51,760 --> 00:44:55,240
open question.
And then non technically, if you

874
00:44:55,240 --> 00:44:57,800
look at how computer science
evolved, numerical analysis and

875
00:44:57,800 --> 00:45:00,480
scientific computing was core to
computer science.

876
00:45:00,480 --> 00:45:02,560
Look at how numerical analysis
and computer science evolved.

877
00:45:03,200 --> 00:45:06,520
Yeah, these areas gave birth to
computer science and then

878
00:45:06,520 --> 00:45:08,520
computer science banished them
because it was easier to build

879
00:45:08,520 --> 00:45:11,400
the structure, the theory
around, around Turing machines

880
00:45:11,400 --> 00:45:13,920
and, and, and the Lambda
calculus and, and discreet

881
00:45:13,920 --> 00:45:16,720
notions that talked about
complexity independent of the

882
00:45:16,720 --> 00:45:17,560
data.
And if you're talking about

883
00:45:17,560 --> 00:45:19,240
complexity independent of the
data, you don't need

884
00:45:19,240 --> 00:45:21,320
preconditioners because it's
independent of the data.

885
00:45:22,080 --> 00:45:25,200
And so there's a cultural aspect
that that says, you know, does

886
00:45:25,200 --> 00:45:28,120
the vertical or the horizontal
culture win here?

887
00:45:28,120 --> 00:45:30,920
And so even if you stipulate
that the answer is technically

888
00:45:30,920 --> 00:45:32,760
yes, I think there's an open
question on how it'll look,

889
00:45:32,800 --> 00:45:36,040
it'll evolve and whether that'll
win the day and solve the

890
00:45:36,040 --> 00:45:38,880
problems that you're asking.
But I think it's not, it's not

891
00:45:38,880 --> 00:45:42,480
so much that I solved PDE type A
and applied it to PDE type B.

892
00:45:42,480 --> 00:45:43,720
It would be certain course
things.

893
00:45:43,720 --> 00:45:47,520
I mean, as you know, there's a
class of things for which finite

894
00:45:47,520 --> 00:45:49,240
element methods work.
There's a different class of

895
00:45:49,240 --> 00:45:51,240
things for which finite volume
methods work.

896
00:45:51,760 --> 00:45:55,160
And you know, you, you take a
course, you learn A1 O1 method

897
00:45:55,160 --> 00:45:57,160
and then boom, they bifurcate
after that.

898
00:45:57,160 --> 00:45:58,920
So what's the analogous taxonomy
here?

899
00:45:58,920 --> 00:45:59,960
And it's probably neither of
those.

900
00:45:59,960 --> 00:46:02,720
It's probably something that's
more data-driven, but what's the

901
00:46:02,720 --> 00:46:05,640
right way to taxonomize that?
And even if there's a

902
00:46:05,640 --> 00:46:08,640
technically a correct way, I
mean, you know, computer science

903
00:46:08,640 --> 00:46:11,520
is not computational science.
Computer science is an area of

904
00:46:11,520 --> 00:46:14,040
the coherent.
Computational science is spread

905
00:46:14,040 --> 00:46:15,920
over many departments and many
divisions.

906
00:46:15,920 --> 00:46:17,960
And so I think it's an open
question about how this area

907
00:46:17,960 --> 00:46:20,360
evolves.
Yeah, it is certainly.

908
00:46:21,520 --> 00:46:24,360
It is certainly interesting.
I think one of the sort of key

909
00:46:25,840 --> 00:46:33,680
topics to all of this though is
the data availability and

910
00:46:34,760 --> 00:46:42,160
licensing and financial reward.
So one thing that I've seen in I

911
00:46:42,160 --> 00:46:46,200
guess the early days is nobody
knew what was going on.

912
00:46:46,400 --> 00:46:49,320
So I guess the companies were
scraping half the Internet.

913
00:46:50,200 --> 00:46:53,040
There were a few captures or
there was a few sort of, you

914
00:46:53,040 --> 00:46:55,960
know, firewalls to stop thing.
But in general, it was a sort of

915
00:46:55,960 --> 00:46:59,880
free for all it seemed, Whereas
now people have, you know,

916
00:46:59,880 --> 00:47:04,320
cottoned on to that and some of
the companies having to do deals

917
00:47:04,320 --> 00:47:07,280
with publishing companies or do
deals with, you know, to try and

918
00:47:07,680 --> 00:47:10,640
legally get access or or
permissively get access to

919
00:47:10,640 --> 00:47:13,920
things.
I saw the example on the weather

920
00:47:14,760 --> 00:47:19,760
side that, you know, some of
those databases were released

921
00:47:19,840 --> 00:47:24,240
openly because they were
released openly enabled, you

922
00:47:24,240 --> 00:47:26,920
know, these various teams to go
and do things like, you know,

923
00:47:26,920 --> 00:47:32,320
forecast net and graph cast.
But those data may not be open

924
00:47:32,320 --> 00:47:38,280
in the future.
So how much you know, and you

925
00:47:38,280 --> 00:47:42,600
work in academia, you see people
want to have their own data.

926
00:47:42,600 --> 00:47:43,720
Some people want to release
later.

927
00:47:45,560 --> 00:47:48,400
Do you see that that will
ultimately be the blocker?

928
00:47:48,400 --> 00:47:51,840
I mean, how do you incentivize
people to want to make their

929
00:47:51,840 --> 00:47:56,000
data available and should it
even be done that way, you know,

930
00:47:56,000 --> 00:47:58,680
that the whole open source data
versus closed?

931
00:47:58,880 --> 00:48:01,920
You know, this is this, I think,
is the elephant in the room in

932
00:48:01,920 --> 00:48:06,360
terms of this area because a lot
of government grants require

933
00:48:06,360 --> 00:48:09,040
that you make the data publicly
available in some sense of the

934
00:48:09,040 --> 00:48:10,720
word.
Now, what does in some sense of

935
00:48:10,720 --> 00:48:12,680
the word mean for a lot of
scientists?

936
00:48:12,680 --> 00:48:14,760
If the data is more than a year
or two old, it's worthless

937
00:48:14,760 --> 00:48:16,880
because they're trying to push
the cutting edge of the science

938
00:48:17,480 --> 00:48:19,720
and then they don't maintain the
data for more than a couple

939
00:48:19,720 --> 00:48:21,480
years ago.
And so no one maintains and it's

940
00:48:21,480 --> 00:48:23,160
hard to access.
So there's a lot of public data

941
00:48:23,160 --> 00:48:27,680
that's not de facto public or
accessible, you know, in a lot

942
00:48:27,720 --> 00:48:31,000
in those cases and maybe in, in,
but, but certainly in, in

943
00:48:31,000 --> 00:48:33,640
machine learning sort of
applications more generally in,

944
00:48:33,640 --> 00:48:35,640
in Internet and social media and
so on.

945
00:48:36,880 --> 00:48:40,080
Having older data is good, but
you can't get more than a couple

946
00:48:40,080 --> 00:48:42,320
decades older, right?
And so often times the the most

947
00:48:42,320 --> 00:48:44,440
recent data is the most valuable
and you just design A method

948
00:48:44,440 --> 00:48:48,440
around that.
And so it's not obvious to me

949
00:48:48,440 --> 00:48:50,880
that the way the data are
generated and, and collected,

950
00:48:50,880 --> 00:48:54,080
distributed now will be the one
that is going developed going

951
00:48:54,080 --> 00:48:55,880
forward.
I mean, if you get better

952
00:48:56,040 --> 00:49:01,120
telescopes, if you get better,
you know, molecular sort of

953
00:49:01,120 --> 00:49:06,520
robotic systems to, to to make
molecules, I can easily imagine

954
00:49:06,520 --> 00:49:10,280
a situation where data that
exists now or prior to a few

955
00:49:10,280 --> 00:49:12,200
years ago is basically obsolete
and irrelevant.

956
00:49:12,200 --> 00:49:15,080
And so all of that data sits at
whoever generated it.

957
00:49:15,080 --> 00:49:17,520
And so then the question is, is
does a government lab generate

958
00:49:17,520 --> 00:49:20,040
it?
And if so, do they make it easy

959
00:49:20,040 --> 00:49:22,840
de facto to access?
Is it done at universities

960
00:49:23,600 --> 00:49:25,840
where, you know, you have a big
infrastructure capital

961
00:49:25,840 --> 00:49:29,280
investment in in particular,
essentially in particular

962
00:49:29,280 --> 00:49:30,640
science and verticals to
generate it?

963
00:49:31,080 --> 00:49:36,680
Is it done at companies that are
in different verticals or is

964
00:49:36,680 --> 00:49:40,320
there some sort of collaboration
or interaction with, you know,

965
00:49:40,360 --> 00:49:44,160
horizontal infrastructure
companies, hyper scalers and

966
00:49:44,160 --> 00:49:46,240
then they get their fingers in
that and have access to it?

967
00:49:46,920 --> 00:49:49,080
So I think these are all open
questions in in terms of how

968
00:49:49,080 --> 00:49:51,960
this evolves.
Yeah, it it seems to be very

969
00:49:51,960 --> 00:49:56,840
much, you know, traditionally
and I guess if this is for me,

970
00:49:56,840 --> 00:50:01,120
the difference between the pre
trained quote quote foundation

971
00:50:01,120 --> 00:50:04,080
models versus just letting
someone go and train them.

972
00:50:04,080 --> 00:50:08,440
Although I mean, I guess most
people you and I would not go

973
00:50:08,440 --> 00:50:11,800
and train our own LLM.
We're just going to go and use

974
00:50:11,800 --> 00:50:14,040
someone who's audit and we're
just doing an inference on it.

975
00:50:16,280 --> 00:50:22,240
That's and I could see if I'm
BMW and I have data from

976
00:50:22,240 --> 00:50:25,240
previous car designs or
simulations or whatever, how

977
00:50:25,240 --> 00:50:27,200
they could train a model using
their data.

978
00:50:30,760 --> 00:50:34,320
But I'm sort of wondering in my
head, how could it ever become

979
00:50:34,800 --> 00:50:40,480
the point where some person has
access to not only BM WS data,

980
00:50:40,880 --> 00:50:46,840
but Fords and Audi's and Volvos
and Boeings and Airbuses and you

981
00:50:46,840 --> 00:50:49,520
know, Exxon Mobil and Shell and
everyone else.

982
00:50:49,840 --> 00:50:53,000
Like how how could they collect
the data?

983
00:50:53,000 --> 00:50:57,800
What would be the incentive for
that to happen to or that's what

984
00:50:57,800 --> 00:51:00,560
I mean.
Or is it the case that that's

985
00:51:00,560 --> 00:51:02,880
why there will never be a
foundation and it will just be

986
00:51:02,880 --> 00:51:05,840
that individual people train
their own models because the

987
00:51:05,840 --> 00:51:08,760
data is just so sensitive and
difficult to collect in one?

988
00:51:09,560 --> 00:51:11,160
Yeah.
I mean, this is, I think this is

989
00:51:11,160 --> 00:51:13,520
the big elephant in the room.
I mean, it's not clear to me

990
00:51:13,520 --> 00:51:14,400
that one.
OK.

991
00:51:14,400 --> 00:51:17,960
So I think if you were to try
and generate a meaningful

992
00:51:17,960 --> 00:51:22,160
scientific foundation model,
you'd need data from a pretty

993
00:51:22,160 --> 00:51:25,640
broad range of use cases.
And so you gave, you know,

994
00:51:25,640 --> 00:51:31,000
several that in particular are
with is a strong sort of

995
00:51:31,000 --> 00:51:32,880
financial backing.
I'm always reminded of the

996
00:51:32,880 --> 00:51:37,680
story.
Gray was, I guess, the most

997
00:51:37,680 --> 00:51:40,800
prominent person in the
Microsoft's research division

998
00:51:40,800 --> 00:51:43,760
years ago, such that, you know,
he would have to have regular

999
00:51:43,760 --> 00:51:45,840
meetings with Bill Gates, the
CEO at the time.

1000
00:51:45,840 --> 00:51:48,040
And Bill Gates would say, why is
the most prominent person in my

1001
00:51:48,040 --> 00:51:51,160
research division applying
databases techniques to

1002
00:51:51,160 --> 00:51:53,640
astronomy?
This is when he started working

1003
00:51:53,640 --> 00:51:56,240
with Alex Silly and some others
on, on databases for, for

1004
00:51:56,240 --> 00:51:59,640
astronomy and, and cosmology.
Why don't you work on something

1005
00:51:59,640 --> 00:52:02,120
more useful like, you know, any
of our products?

1006
00:52:02,120 --> 00:52:04,720
And, and crazy answer was
basically, I'm working on it

1007
00:52:04,720 --> 00:52:07,920
because it's worthless, because
there's no financial sort of

1008
00:52:07,920 --> 00:52:09,400
incentive behind it.
I don't have to talk to the

1009
00:52:09,400 --> 00:52:11,680
lawyers and we can work at IP
and all these things.

1010
00:52:11,880 --> 00:52:14,880
It's public data and, and I can
work on database techniques and

1011
00:52:14,880 --> 00:52:17,200
figure out latency and bandwidth
and all these trade-offs.

1012
00:52:17,840 --> 00:52:20,600
So I don't know, but it would be
interesting to see.

1013
00:52:21,160 --> 00:52:22,760
Will you have an analogous thing
here?

1014
00:52:22,760 --> 00:52:26,520
So there's there, there is
public, there's, there's various

1015
00:52:26,520 --> 00:52:28,400
places where public data is
available.

1016
00:52:28,880 --> 00:52:31,360
And I think if you can
illustrate that you would have a

1017
00:52:31,360 --> 00:52:34,680
proof of principle foundation
like model.

1018
00:52:34,720 --> 00:52:37,840
Then the question is, will
people come to it?

1019
00:52:38,200 --> 00:52:40,720
Now clearly the incentives for
scientists will be different

1020
00:52:40,720 --> 00:52:44,400
than than for companies there.
But I could imagine a situation

1021
00:52:44,400 --> 00:52:48,120
where you build that out and
then companies could take it and

1022
00:52:48,120 --> 00:52:49,880
fine tune to their particular
data, right.

1023
00:52:50,120 --> 00:52:52,200
It's not obvious to me that any
of the companies that you

1024
00:52:52,200 --> 00:52:56,360
mentioned have data that that's,
that is that special in a

1025
00:52:56,360 --> 00:52:58,560
scientific sense?
I mean, clearly the particular

1026
00:52:58,560 --> 00:53:01,520
data they have is particular to
their particular situation and

1027
00:53:01,520 --> 00:53:03,960
then the particular, you know,
product they have.

1028
00:53:04,480 --> 00:53:09,520
But I, I can imagine that, you
know, the Pdes from early solar

1029
00:53:09,520 --> 00:53:14,520
system formation and, and
hydrogen bomb explosions and

1030
00:53:14,520 --> 00:53:18,600
hydrodynamics inside stars, you
know, is, has some similarities

1031
00:53:18,600 --> 00:53:20,720
and some differences with
weather and climate and some

1032
00:53:20,720 --> 00:53:22,480
similarities and some
differences with crack

1033
00:53:22,520 --> 00:53:25,120
propagation in the Earth.
And that'll have some

1034
00:53:25,120 --> 00:53:27,880
similarities and differences
with, with various sorts of

1035
00:53:27,880 --> 00:53:31,000
sensing modalities that you
would, that have a geospatial

1036
00:53:31,000 --> 00:53:33,240
component.
And so can you stress test that

1037
00:53:33,240 --> 00:53:36,440
and, and get even even a proof
of principle that, that if you

1038
00:53:36,440 --> 00:53:39,360
would attack those on, then the
answer would be yes, that, that

1039
00:53:39,360 --> 00:53:41,760
you could have a path to have a
more foundation model.

1040
00:53:41,760 --> 00:53:43,720
If the answer is yes, then the
question is you talk to various

1041
00:53:43,720 --> 00:53:45,400
stakeholders.
Some of them are government

1042
00:53:45,400 --> 00:53:48,080
entities or government sponsors,
some of them are corporate and

1043
00:53:48,080 --> 00:53:50,840
try and figure that out.
And I could, I could imagine

1044
00:53:50,840 --> 00:53:54,120
that that basically it's too big
a left in the area stalls.

1045
00:53:54,120 --> 00:53:56,120
So I think this is something to
be determined.

1046
00:53:56,120 --> 00:53:59,680
But I think having a nucleus of
something is going to be

1047
00:54:00,240 --> 00:54:03,560
important because right now I at
least what I've seen, I mean, in

1048
00:54:03,560 --> 00:54:06,480
most of companies like that,
there's, there's, there may be

1049
00:54:06,480 --> 00:54:08,920
some people a little bit
intrigued, but there's not a

1050
00:54:08,920 --> 00:54:11,520
sufficiently heavy lift that
they go screaming at their CEO

1051
00:54:11,520 --> 00:54:14,560
to say we need to do this,
probably because it's not

1052
00:54:14,560 --> 00:54:16,880
extremely strong baselines that
it can happen.

1053
00:54:17,280 --> 00:54:20,280
And their competitors by talking
to these public models and fine

1054
00:54:20,280 --> 00:54:23,000
tuning on their proprietary data
will do better.

1055
00:54:23,000 --> 00:54:25,600
I think at that point they'll be
you can see a change.

1056
00:54:26,440 --> 00:54:29,080
And I think this is, I mean, I
guess this is affecting the

1057
00:54:29,080 --> 00:54:33,280
current Gen.
AI and AI, which is how do you

1058
00:54:33,280 --> 00:54:36,160
make money out of this?
You know, like you can do it for

1059
00:54:36,160 --> 00:54:37,800
the sake of science.
That's nice.

1060
00:54:38,080 --> 00:54:41,920
I developed this thing.
But I'm always interested to see

1061
00:54:41,920 --> 00:54:46,640
like the money that you put in
to create the data to train the

1062
00:54:46,640 --> 00:54:53,280
model, can you ultimately make a
profitable thing out of it that

1063
00:54:53,280 --> 00:54:56,400
justifies its cost?
Now, with standard, I guess,

1064
00:54:57,280 --> 00:55:00,840
simulation software, there's an
argument, well, if I simulate

1065
00:55:00,840 --> 00:55:06,880
it, I don't have to build it.
If I can predict the weather, I

1066
00:55:06,880 --> 00:55:10,440
can help people figure out if
it's there's going to be a

1067
00:55:10,440 --> 00:55:11,880
hurricane.
And I can say, you know, there's

1068
00:55:11,880 --> 00:55:15,800
a, there's a clear thing.
But because we can already

1069
00:55:15,800 --> 00:55:18,960
simulate things, we can already
predict the weather.

1070
00:55:18,960 --> 00:55:21,560
We can always do it.
For me, the machine learning bit

1071
00:55:21,560 --> 00:55:25,640
is just the argument, well, it's
faster or, or, or it's cheaper.

1072
00:55:26,800 --> 00:55:28,400
That's where the amount of data
do.

1073
00:55:28,400 --> 00:55:30,800
Do you know what I mean?
Is there a commercial?

1074
00:55:31,440 --> 00:55:35,120
But I, I think, I think there's
a faster question and I think

1075
00:55:35,120 --> 00:55:36,640
it's tempting to say, can I be
faster?

1076
00:55:36,640 --> 00:55:38,680
Because faster means there's a
number you're comparing to.

1077
00:55:38,680 --> 00:55:40,840
So am I bigger?
Am I more on some axis?

1078
00:55:41,640 --> 00:55:46,080
And back to the, the, the
randomized linear algebra

1079
00:55:46,080 --> 00:55:48,240
example, I mean, I, I was
talking to various people and

1080
00:55:48,240 --> 00:55:49,680
saying, don't worry about the
randomness.

1081
00:55:49,680 --> 00:55:52,200
You know, if, if you do this and
that, that'll work.

1082
00:55:52,640 --> 00:55:55,720
And the people would, you know,
basically when you have a new

1083
00:55:55,720 --> 00:55:57,720
idea, they'll beat you up.
And so they beat us up and so

1084
00:55:57,800 --> 00:56:00,640
on.
And a few years later, one of

1085
00:56:00,640 --> 00:56:03,400
those people in particular had
some methods and showed it

1086
00:56:03,400 --> 00:56:05,560
worked, you know, not the blend
people, you know, something

1087
00:56:05,560 --> 00:56:08,200
around the same time and their
students came back to me and

1088
00:56:08,200 --> 00:56:10,000
didn't know who I was.
I was lecturing me and saying,

1089
00:56:10,000 --> 00:56:11,800
oh, don't worry about the
randomness, it'll all be fine.

1090
00:56:11,800 --> 00:56:13,520
And so on.
There's just a cultural shift.

1091
00:56:14,080 --> 00:56:19,040
And so there was a cultural
shift that really was not, is as

1092
00:56:19,040 --> 00:56:22,400
faster as this is as better.
And so I, I, I don't think

1093
00:56:22,400 --> 00:56:24,880
you're going to see the win here
just because something's faster

1094
00:56:24,880 --> 00:56:26,640
and better.
You might need that to point to,

1095
00:56:27,040 --> 00:56:29,240
but the win will happen because
you can solve other things that

1096
00:56:29,240 --> 00:56:32,440
you couldn't solve before.
So just as an example, say

1097
00:56:32,440 --> 00:56:34,600
you're, you know, you want a
better battery and you want to

1098
00:56:34,600 --> 00:56:37,760
understand the way crack
propagates better, you know, so

1099
00:56:37,760 --> 00:56:40,720
that I mean, you want, you want,
you want a longer lifetime, you

1100
00:56:40,720 --> 00:56:42,680
know, warranty.
So you, you're working on

1101
00:56:42,680 --> 00:56:45,240
developing batteries.
One of the ways this fits in,

1102
00:56:45,240 --> 00:56:47,320
not in a scientific, but in an
engineering or commercial

1103
00:56:47,320 --> 00:56:52,120
context is you want to write a
warranty to say that that the

1104
00:56:52,120 --> 00:56:53,600
batteries are going to last so
and so long.

1105
00:56:53,600 --> 00:56:55,880
And, and otherwise you'll, you
know, you'll eat the cost.

1106
00:56:56,280 --> 00:56:58,440
And so can you predict battery
lifetimes.

1107
00:56:58,440 --> 00:57:01,280
And maybe if you have a certain
crack propagation heterogeneity

1108
00:57:01,280 --> 00:57:04,160
structure, you know that it
correlates better with how long

1109
00:57:04,160 --> 00:57:05,720
the batteries will last.
So now there's a fair

1110
00:57:05,720 --> 00:57:08,200
variability and a particular run
on the extreme tails.

1111
00:57:08,680 --> 00:57:13,640
So that sounds sort of very
different than I want to do oil

1112
00:57:13,640 --> 00:57:15,480
exploration and better fracking
or something.

1113
00:57:15,480 --> 00:57:17,800
I send sound waves down into the
earth and I look at how they

1114
00:57:17,800 --> 00:57:19,920
bounce back and I try and learn
something about it.

1115
00:57:20,480 --> 00:57:22,880
But I could imagine that the
embeddings that you get from a

1116
00:57:23,280 --> 00:57:26,880
core model that cuts across a
couple different domains learn

1117
00:57:26,880 --> 00:57:30,240
spatiotemporal properties in
particular of complex systems

1118
00:57:30,240 --> 00:57:33,200
that have sort of long ranged,
non trivial, heavy tailed

1119
00:57:33,200 --> 00:57:36,640
correlations, which is what both
of those examples have.

1120
00:57:37,600 --> 00:57:40,400
And so say that you can come up
with a model that captures those

1121
00:57:40,400 --> 00:57:44,160
embeddings.
Now, a company here may tweak it

1122
00:57:44,160 --> 00:57:46,880
to their data, do some transfer
learning in the battery space.

1123
00:57:46,880 --> 00:57:51,080
A company over here can do a
better job adjusting it to their

1124
00:57:51,080 --> 00:57:53,760
data, which is, you know,
spatiotemporally very different

1125
00:57:53,760 --> 00:57:55,640
in terms of, you know, better
fracking.

1126
00:57:56,040 --> 00:57:58,080
And, you know, for all I know,
the same embeddings could be

1127
00:57:58,080 --> 00:58:00,360
useful for, you know, you had a
very different use case, right?

1128
00:58:00,360 --> 00:58:03,400
And certainly in the United
States, there's various places

1129
00:58:03,400 --> 00:58:07,440
where insurance markets don't
exist for floods or for

1130
00:58:07,520 --> 00:58:09,400
earthquakes or for a range of
things like that.

1131
00:58:09,800 --> 00:58:11,920
I could imagine that you get
these course embeddings.

1132
00:58:11,920 --> 00:58:15,120
These are course functions.
They're not sinusoids and 4:00

1133
00:58:15,120 --> 00:58:16,080
AM modes.
They're something that's

1134
00:58:16,080 --> 00:58:18,840
data-driven embeddings.
And you say on top of that, I

1135
00:58:18,840 --> 00:58:22,360
want to do transfer learning to
the relatively small amount of

1136
00:58:22,360 --> 00:58:24,280
data that I have.
That's very high touch.

1137
00:58:24,360 --> 00:58:27,480
You know, you have 10,000
stations across the country that

1138
00:58:27,480 --> 00:58:30,360
measure something in chemicals,
whatever I want to do transfer

1139
00:58:30,360 --> 00:58:33,120
learning to that and I model,
you know, model the system, the

1140
00:58:33,120 --> 00:58:36,240
way information flows in the
Mississippi Delta or something.

1141
00:58:36,720 --> 00:58:40,760
And, and if I do that, I can do
a better job, you know,

1142
00:58:40,760 --> 00:58:42,920
predicting the tails of flood
events.

1143
00:58:42,920 --> 00:58:45,120
Why?
Because as a model, I don't know

1144
00:58:45,120 --> 00:58:48,040
how things propagate, but you
know, subtle effects in terms of

1145
00:58:48,040 --> 00:58:52,120
the density of the soil or, or
the moisture of the soil coupled

1146
00:58:52,120 --> 00:58:55,280
with, you know, 10,000 other
things that the neural network

1147
00:58:55,280 --> 00:58:57,440
learns in a complicated way.
What's hard to reason about

1148
00:58:57,440 --> 00:59:00,440
scientifically means that now I
can create that market.

1149
00:59:00,520 --> 00:59:02,680
And that's, that's something
that any of those companies I

1150
00:59:02,680 --> 00:59:06,240
could imagine try and you know,
buy out a start up company that

1151
00:59:06,240 --> 00:59:07,600
does that.
So I think you're going to see a

1152
00:59:07,600 --> 00:59:09,680
win.
Not just I'm running faster,

1153
00:59:09,680 --> 00:59:11,960
something faster than you, but
but in a space like that.

1154
00:59:12,560 --> 00:59:17,400
Yeah, no, I, I, I think what
you're getting at is you, what

1155
00:59:17,400 --> 00:59:20,120
we've seen from current machine
that it seems to do things that

1156
00:59:20,120 --> 00:59:23,440
we don't fully understand, but
are like you, like you gave the

1157
00:59:23,440 --> 00:59:27,080
original example of the time
series that an LM that wasn't

1158
00:59:27,080 --> 00:59:29,240
explicitly trained on it
actually ended up being quite

1159
00:59:29,240 --> 00:59:32,200
useful.
So I guess we shouldn't

1160
00:59:32,200 --> 00:59:35,720
underestimate the potential of
how some of these things may do

1161
00:59:35,720 --> 00:59:40,400
something that we're, it's not
just about replicating what an

1162
00:59:40,400 --> 00:59:43,240
existing simulation tool could
do, but maybe it could help to

1163
00:59:44,400 --> 00:59:47,320
merge different disciplines
together and see the links and

1164
00:59:47,320 --> 00:59:48,640
the, and the, and the things
between them.

1165
00:59:50,320 --> 00:59:54,480
I did want to pivot to a
different topic, which is when

1166
00:59:54,480 --> 00:59:58,440
some of the other people I've
spoken to, I'm always really

1167
00:59:58,440 --> 01:00:03,760
fascinated by the curiosity or
the uniqueness or the

1168
01:00:03,760 --> 01:00:09,040
peculiarity of academia and, and
the sort of different people's

1169
01:00:09,040 --> 01:00:12,320
careers and, and, and
progression through.

1170
01:00:12,320 --> 01:00:16,360
And I think you have quite an
interesting one, because you are

1171
01:00:16,400 --> 01:00:20,920
a very pure.
Mathematician, academic, you

1172
01:00:20,920 --> 01:00:25,000
know, you, you are very much in
that, but you also have, you

1173
01:00:25,000 --> 01:00:27,760
know, the Amazon scholar
position you at the, you know,

1174
01:00:27,760 --> 01:00:35,000
Lawrence Berkeley.
Was this a a strategic thing in

1175
01:00:35,000 --> 01:00:39,520
the sense that you wanted to
work on real problems?

1176
01:00:40,240 --> 01:00:46,280
Was this a just you find more
interesting, like is there, How

1177
01:00:46,280 --> 01:00:48,320
has it benefited you?
And would you advise it to

1178
01:00:48,320 --> 01:00:48,960
others?
You know?

1179
01:00:49,760 --> 01:00:52,040
Yeah.
Yeah.

1180
01:00:52,040 --> 01:00:54,160
I mean, it's there's a lot of,
again, a lot of questions.

1181
01:00:54,200 --> 01:00:57,040
I mean, at least the narrow
questions, universities versus

1182
01:00:57,040 --> 01:00:59,040
not, yeah.
Yeah, there you go.

1183
01:00:59,080 --> 01:01:03,240
I'll start.
That but, but, but yeah, you

1184
01:01:03,240 --> 01:01:05,320
tapped enough stuff into the
question to make it Ford

1185
01:01:05,320 --> 01:01:07,240
compatible with lots of answers
as it sounds like.

1186
01:01:08,000 --> 01:01:12,320
I mean, so my, my, you know, OK,
so my PF2's in physics, I never

1187
01:01:12,320 --> 01:01:14,480
actually studied computer
science or statistics.

1188
01:01:14,480 --> 01:01:16,880
And it was in computational
statistical mechanics.

1189
01:01:17,600 --> 01:01:21,960
And at the end of the
dissertation I switched areas

1190
01:01:21,960 --> 01:01:24,280
basically to theoretical
computer science to work on

1191
01:01:25,880 --> 01:01:30,040
originally Markov chain
algorithms, some of the stuff

1192
01:01:30,040 --> 01:01:32,720
I've been working on
computationally, Markov chains

1193
01:01:32,720 --> 01:01:34,320
and molecular dynamics.
So it was a technical

1194
01:01:34,320 --> 01:01:36,160
connection, but it was it was
fairly loose.

1195
01:01:37,000 --> 01:01:40,160
And then the randomization
inside the Markov chains that

1196
01:01:40,160 --> 01:01:42,600
the people I was working with
were also working on these very

1197
01:01:42,600 --> 01:01:44,360
early versions of these
randomized linear algebra

1198
01:01:44,360 --> 01:01:46,800
algorithms.
So I got involved with that and

1199
01:01:46,800 --> 01:01:49,640
I'd done enough coding that was
clear certain ones are going to

1200
01:01:49,640 --> 01:01:50,760
be useful and certain ones
weren't.

1201
01:01:50,760 --> 01:01:54,960
And so at that point I decided
to switch areas partly because

1202
01:01:54,960 --> 01:01:59,000
of what I've been working on.
Well, two reasons there was a

1203
01:01:59,000 --> 01:02:01,120
forcing function.
This is I just mentioned this

1204
01:02:01,120 --> 01:02:03,720
because if you have younger
people listening, it's good to

1205
01:02:03,720 --> 01:02:06,720
know that sometimes older people
have had textured past, let's

1206
01:02:06,720 --> 01:02:08,760
say.
So Long story short, I had

1207
01:02:08,760 --> 01:02:11,280
career ending problems with the
dissertation advisor and and the

1208
01:02:11,280 --> 01:02:14,000
supply demand structure in the
natural sciences and engineering

1209
01:02:14,000 --> 01:02:15,640
is very different than in
computer science.

1210
01:02:16,160 --> 01:02:19,240
And so that was was motivation
to, you know, jump off the

1211
01:02:19,240 --> 01:02:25,240
Cliff.
I also realized that that the

1212
01:02:25,240 --> 01:02:27,320
particular work I've been doing
in computational system

1213
01:02:27,320 --> 01:02:29,520
mechanics, this is a great area.
I loved it and and it's informed

1214
01:02:29,520 --> 01:02:30,880
a lot of the stuff that I've
done since then.

1215
01:02:31,400 --> 01:02:36,720
But you know, it was a great
area to be in a 1970 because you

1216
01:02:36,720 --> 01:02:39,040
know, Ford compatible with a
huge expansion.

1217
01:02:39,040 --> 01:02:40,880
There's going to be Nobel prizes
given out.

1218
01:02:40,880 --> 01:02:42,480
I mean, just lots of good stuff
going on.

1219
01:02:43,080 --> 01:02:45,200
Not so much in 2000.
The area, it's sort of sad, but

1220
01:02:45,200 --> 01:02:50,040
it was clear that that the
future is going to be data

1221
01:02:50,040 --> 01:02:53,480
analysis and, and I joke that I
never knew more about algorithms

1222
01:02:53,480 --> 01:02:56,480
or data than the day I switched.
And then I've just been becoming

1223
01:02:56,480 --> 01:03:00,040
more ignorant since then.
So I don't know quite what I

1224
01:03:00,400 --> 01:03:05,520
meant by data analysis then, but
but but clearly I was right.

1225
01:03:05,520 --> 01:03:08,480
I mean, it was a fruitful area
for the next couple decades.

1226
01:03:08,920 --> 01:03:14,000
So I done work in theory of
algorithms and, and with respect

1227
01:03:14,000 --> 01:03:17,880
to the academic question, if
you're desperately wed to an

1228
01:03:17,880 --> 01:03:21,920
academic career, you shouldn't
do this because the academic

1229
01:03:21,920 --> 01:03:24,480
hiring is fairly siloed and, and
it's, it's not a good idea to

1230
01:03:24,480 --> 01:03:27,000
switch areas after your
dissertation and, and so on.

1231
01:03:28,080 --> 01:03:32,160
If, if you, if there's a forcing
function like I had, or you're

1232
01:03:32,160 --> 01:03:35,360
interested in sort of problems
more broadly, then you can think

1233
01:03:35,360 --> 01:03:37,120
a little bit more broadly.
So I've had a lot of students in

1234
01:03:37,120 --> 01:03:39,320
postdocs and I sort of give them
this advice and you try and

1235
01:03:39,320 --> 01:03:41,520
carve out various projects that
they're interested in.

1236
01:03:41,520 --> 01:03:43,640
Some want an academic route, so
they should work on slightly

1237
01:03:43,640 --> 01:03:45,400
more conservative things in the
general space.

1238
01:03:45,400 --> 01:03:48,680
Some definitely 1 industry and
some, you know, could go with

1239
01:03:48,680 --> 01:03:50,760
either and, and the ones that
could go with either, I've seen

1240
01:03:50,760 --> 01:03:52,360
some go one way and some go the
other.

1241
01:03:53,240 --> 01:03:56,480
As a general rule, this is going
to be a marketable space and

1242
01:03:56,480 --> 01:03:58,560
stuff I do.
And so maybe it's a little bit

1243
01:03:58,560 --> 01:04:03,080
less of an issue, but but, but
that's something that they

1244
01:04:03,080 --> 01:04:05,640
figure out.
So I had done work in in

1245
01:04:07,360 --> 01:04:09,440
theoretical computer science and
it was clear that these

1246
01:04:09,440 --> 01:04:13,120
algorithms would be useful more
broadly because the randomness

1247
01:04:13,120 --> 01:04:17,000
entered in a very different way
then classical algorithms.

1248
01:04:17,000 --> 01:04:20,520
And so then the question is, how
does it evolve?

1249
01:04:20,960 --> 01:04:23,560
And so it evolved.
I was at Yale in the mathematics

1250
01:04:23,560 --> 01:04:25,960
department as in one of the
junior faculty positions.

1251
01:04:25,960 --> 01:04:29,520
I spent time at Yahoo and
Stanford and moved to Berkeley.

1252
01:04:29,520 --> 01:04:33,920
I'm in the statistics department
there and at the International

1253
01:04:33,920 --> 01:04:36,360
Computer Science Institute of
Lawrence Berkeley National Lab.

1254
01:04:36,360 --> 01:04:39,920
And as you mentioned couple
years ago started with as an

1255
01:04:39,920 --> 01:04:42,560
Amazon scholar working in the
supply chain optimization

1256
01:04:42,560 --> 01:04:46,040
technology group, where, you
know, we work on better

1257
01:04:46,040 --> 01:04:49,080
forecasting, demand forecasting
algorithms and supply chain

1258
01:04:49,080 --> 01:04:51,640
decisions.
So I think this is, this is

1259
01:04:51,640 --> 01:04:53,320
something that's good for people
to think about.

1260
01:04:53,880 --> 01:04:56,840
And you know, there's at least,
I guess 4 hats there.

1261
01:04:56,840 --> 01:04:59,480
And each hat has pros and cons
and pluses and minuses.

1262
01:04:59,480 --> 01:05:03,240
And so figuring out how to
navigate that space is something

1263
01:05:03,240 --> 01:05:05,680
I try and encourage students and
post docs to think about sooner

1264
01:05:05,680 --> 01:05:08,280
rather than later.
How?

1265
01:05:09,760 --> 01:05:13,400
How much should people expect to
have to move?

1266
01:05:14,080 --> 01:05:19,640
You know, I always think this is
1 of this sad in a way, things

1267
01:05:19,640 --> 01:05:21,800
about academia, at least from
what I've observed, that the

1268
01:05:21,800 --> 01:05:27,560
fact that you do your PhD and
let's say you do a postdoc, how

1269
01:05:27,920 --> 01:05:30,720
realistic is it that that person
in the US or the UK should

1270
01:05:30,720 --> 01:05:33,360
assume that they can stay at
that same university and just

1271
01:05:33,360 --> 01:05:35,920
move up a chain?
Or how much should they expect

1272
01:05:35,920 --> 01:05:38,960
that they're basically going to
have to move somewhere else in

1273
01:05:38,960 --> 01:05:43,400
the country to find the
position?

1274
01:05:44,560 --> 01:05:47,640
Yeah.
I mean, the short answer is that

1275
01:05:48,560 --> 01:05:52,720
when you do a postdoc, it's
usually more targeted because

1276
01:05:52,720 --> 01:05:56,400
you're working with a particular
person, a particular group and

1277
01:05:58,240 --> 01:05:59,720
and so you could be at the same
place or not.

1278
01:05:59,720 --> 01:06:01,800
It's usually a good idea to go
elsewhere to get experience with

1279
01:06:01,800 --> 01:06:04,960
other people, the real filters
at the assistant professor

1280
01:06:04,960 --> 01:06:06,440
level.
And you got to move there

1281
01:06:06,440 --> 01:06:09,480
because, you know, except in our
cases, you don't get a job at

1282
01:06:09,480 --> 01:06:11,960
the same place.
And that's not a statement about

1283
01:06:11,960 --> 01:06:14,960
whether the place where you are
at is good or bad or whether

1284
01:06:14,960 --> 01:06:17,480
you're good or bad is just just
run the numbers, right.

1285
01:06:17,480 --> 01:06:21,000
If there's 10:20, 30-40 places
hiring and that you're looking

1286
01:06:21,000 --> 01:06:24,880
at and and the chances are one
and whatever that that you get

1287
01:06:24,880 --> 01:06:27,880
an offer, it's just unlikely to
happen at that same place even

1288
01:06:27,880 --> 01:06:29,880
if ultimately end up there.
Sometimes people go and leave

1289
01:06:29,880 --> 01:06:33,440
and come back.
So moving for better for us as

1290
01:06:33,440 --> 01:06:37,560
part of the part of the
equation, yeah.

1291
01:06:38,760 --> 01:06:44,000
But did you, some of the
questions I get is the should

1292
01:06:44,000 --> 01:06:47,720
they go into industry or, or
just let's put this two ways,

1293
01:06:47,760 --> 01:06:52,560
does academia value industry?
So if you've been an assistant

1294
01:06:52,560 --> 01:06:55,240
professor or you've been a
postdoc and you then say, you

1295
01:06:55,240 --> 01:06:59,320
know what, I'm going to go into
industry, do they value then?

1296
01:06:59,440 --> 01:07:05,560
And is the ability to come back
or have you been out of the game

1297
01:07:05,560 --> 01:07:08,240
too long?
You haven't sort of, you know,

1298
01:07:08,240 --> 01:07:11,320
got the publications or the
teaching experience, You know,

1299
01:07:11,520 --> 01:07:14,440
is that a good piece of advice
or would you say no, no, no, If

1300
01:07:14,440 --> 01:07:16,160
you really want to be an
academia, you basically need to

1301
01:07:16,160 --> 01:07:23,200
stay in in academia.
Yeah, I mean it, it's, it's more

1302
01:07:23,200 --> 01:07:25,480
texture than the following.
But I, I think the short answer

1303
01:07:25,480 --> 01:07:27,400
is that that that you need to
stay.

1304
01:07:28,840 --> 01:07:32,280
The slightly longer answer is it
depends on the area, like

1305
01:07:32,280 --> 01:07:35,040
computer science versus
statistics versus engineering

1306
01:07:35,320 --> 01:07:39,800
are rather different.
In some cases having a startup,

1307
01:07:41,320 --> 01:07:43,920
in some cases having a postdoc
in industry is good and you can

1308
01:07:43,920 --> 01:07:46,960
go back to universities and that
tends to correlate with computer

1309
01:07:46,960 --> 01:07:49,800
science.
I don't know as much in

1310
01:07:49,800 --> 01:07:52,880
engineering that may be changing
over time, but less so.

1311
01:07:52,880 --> 01:07:54,520
And maybe statistics and applied
math.

1312
01:07:56,280 --> 01:08:02,560
I think in a sense, on the one
hand, people value the

1313
01:08:02,560 --> 01:08:04,240
experience you might have, but
not really.

1314
01:08:04,240 --> 01:08:08,120
And by that I mean, you know,
if, if you, if you check all, if

1315
01:08:08,120 --> 01:08:11,000
if you check all my boxes.
And in addition, you have this

1316
01:08:11,000 --> 01:08:13,160
other stuff, good, but you got
to check all my boxes and the

1317
01:08:13,160 --> 01:08:14,440
boxes are the things you alluded
to.

1318
01:08:15,680 --> 01:08:19,479
And so there's sometimes there's
post docs that at industry that

1319
01:08:19,479 --> 01:08:21,640
are basically academic post
docs, you write papers.

1320
01:08:21,720 --> 01:08:24,279
So I'm not counting that because
that's, that's the fact of the

1321
01:08:24,279 --> 01:08:25,520
same sort of thing.
But if you're out more than a

1322
01:08:25,520 --> 01:08:29,600
couple years and you have fewer
papers, it just gets harder to

1323
01:08:29,600 --> 01:08:32,840
publish.
There are certainly cases where

1324
01:08:32,840 --> 01:08:35,279
people do that, especially if
they have some high profile ones

1325
01:08:35,960 --> 01:08:38,040
and, and can work their way back
one way or the other.

1326
01:08:38,040 --> 01:08:41,080
But it's, you know, you should
know going in that you're, it's

1327
01:08:41,080 --> 01:08:43,319
like a salmon swimming upstream.
And this is going to be a hard

1328
01:08:43,359 --> 01:08:49,640
one.
So how does he, this is maybe

1329
01:08:50,040 --> 01:08:52,200
people in the US are maybe a
little bit more familiar, but

1330
01:08:52,200 --> 01:08:54,800
particularly people who are not.
So how does it work with the

1331
01:08:54,800 --> 01:08:57,000
national labs?
So you have a position on the

1332
01:08:57,000 --> 01:09:00,840
national labs.
How, how does that work between

1333
01:09:00,840 --> 01:09:05,680
is this a, a quite common thing
in a way for these things to

1334
01:09:05,680 --> 01:09:07,200
happen?
I I can only imagine how you

1335
01:09:07,200 --> 01:09:12,240
juggle the e-mail inboxes and
the sort of meetings, but I

1336
01:09:12,240 --> 01:09:14,319
guess it's valuable.
And interesting off an hour

1337
01:09:14,319 --> 01:09:15,640
today and I haven't been hit by
e-mail.

1338
01:09:15,640 --> 01:09:18,439
That's what I've turned it off.
That's the accept e-mail is a

1339
01:09:18,439 --> 01:09:21,960
little overwhelming.
I mean, it's not so common.

1340
01:09:22,240 --> 01:09:26,439
So, so I, I with the Berkeley
hack, because I'm in the

1341
01:09:26,439 --> 01:09:30,040
statistic department there, I'm
at Lawrence Berkeley lab.

1342
01:09:30,040 --> 01:09:32,720
It's literally just up the hill.
And so I can, you can walk

1343
01:09:32,720 --> 01:09:37,040
there.
And so they worked in scientific

1344
01:09:37,040 --> 01:09:40,000
machine learning and, and I had
done work and was well known in

1345
01:09:40,000 --> 01:09:42,560
machine learning and, and
algorithms and statistics, large

1346
01:09:42,560 --> 01:09:45,200
scale statistics.
And I had various projects over

1347
01:09:45,200 --> 01:09:49,080
the years with people there and
elsewhere on scientific ML

1348
01:09:49,080 --> 01:09:53,120
problems.
And so started about two years

1349
01:09:53,120 --> 01:09:56,240
ago.
This is all we rename, but the

1350
01:09:56,240 --> 01:10:01,840
scientific machine learning
group, basically MLA machine

1351
01:10:01,840 --> 01:10:06,280
learning and analytics.
And and so it's it's it's part

1352
01:10:06,360 --> 01:10:09,600
was partly facilitating for the
fact that there there is a

1353
01:10:09,600 --> 01:10:13,800
moderate amount of interaction
between UC Berkeley campus and

1354
01:10:13,800 --> 01:10:18,360
LBNL Lawrence Berkeley National
Lab, Oak Ridge Argon.

1355
01:10:18,360 --> 01:10:20,360
Some of the other labs have
things like that, but it's not

1356
01:10:20,360 --> 01:10:23,880
so, so, so common.
Most people are there, meaning

1357
01:10:23,880 --> 01:10:27,360
just there and 100% of their FT
ES there and and there for long

1358
01:10:27,360 --> 01:10:29,880
term but so it's not.
I wouldn't say it's very common.

1359
01:10:30,800 --> 01:10:36,160
OK, which makes your experience
even more unique then to have

1360
01:10:36,160 --> 01:10:39,520
done it.
Yeah, what maybe is a sort of

1361
01:10:39,840 --> 01:10:47,520
finishing off comment or, or
topic is if you have somebody

1362
01:10:47,520 --> 01:10:53,240
now who was wanting to get into
scientific machine learning,

1363
01:10:53,920 --> 01:10:56,640
because this is I guess kind of
what we've been largely talking

1364
01:10:56,640 --> 01:11:00,880
about, what would you get them
to focus on?

1365
01:11:01,240 --> 01:11:03,480
They're doing a PhD or they're
doing a postdoc.

1366
01:11:03,480 --> 01:11:06,200
But is there any, doesn't have
to be a very, you know, Pacific,

1367
01:11:06,200 --> 01:11:10,840
Pacific, but what sort of area
we'd say, you know, this is

1368
01:11:10,920 --> 01:11:13,520
something you should look at.
This is an area that is ripe

1369
01:11:13,920 --> 01:11:20,480
for, you know, progress.
Yeah, that's a good question.

1370
01:11:20,800 --> 01:11:30,520
There's certain things, I mean,
that are less ripe for progress

1371
01:11:30,520 --> 01:11:33,280
in scientific ML, even if
they're important scientific

1372
01:11:33,280 --> 01:11:36,600
problems.
So what I, what I, what I would

1373
01:11:36,600 --> 01:11:39,680
try and say is work.
You should work on a problem

1374
01:11:39,680 --> 01:11:41,160
that's of interest to both
sides.

1375
01:11:41,160 --> 01:11:43,480
Otherwise you're coming at it
from one side or another.

1376
01:11:43,480 --> 01:11:47,120
And this is true whether you're
coming from CS or stats learning

1377
01:11:47,120 --> 01:11:49,440
scientific problems are coming
from one scientific area.

1378
01:11:50,640 --> 01:11:52,920
You should work on trying to
figure out how to frame what

1379
01:11:52,920 --> 01:11:54,520
you're doing as something of
interest to both sides.

1380
01:11:54,520 --> 01:11:58,520
So I have projects that, you
know, we, we, we construct

1381
01:11:58,520 --> 01:12:01,720
projects, you know, student, a
postdoc owns a piece of it.

1382
01:12:02,040 --> 01:12:06,200
And, and a goal typically is we
want a publication in the

1383
01:12:06,280 --> 01:12:08,760
particular scientific area, but
also in an ML venue.

1384
01:12:08,760 --> 01:12:12,000
And it need to be the same one.
Sometimes it is, it's a long

1385
01:12:12,000 --> 01:12:13,720
version, a short version.
Sometimes it's just two

1386
01:12:13,720 --> 01:12:16,960
different things, but work on a
method that's broad enough that

1387
01:12:16,960 --> 01:12:19,200
machine learning people are
interested in it and that by the

1388
01:12:19,200 --> 01:12:20,960
definition of machine learning,
people are interested.

1389
01:12:20,960 --> 01:12:22,640
You convince 3 reviewers to say
yes.

1390
01:12:23,320 --> 01:12:25,800
And so it's an imperfect
process, you know the review

1391
01:12:25,800 --> 01:12:28,480
process, etcetera, but boom, you
get it in a top machine learning

1392
01:12:28,480 --> 01:12:30,840
band room.
And similarly, on the scientific

1393
01:12:30,840 --> 01:12:33,400
side, you know, a, a statement
that it's a value of the

1394
01:12:33,400 --> 01:12:36,160
scientist is you get 3 reviewers
to say accept and then it's

1395
01:12:36,440 --> 01:12:40,680
accepted on the scientific side.
And oftentimes the way we try

1396
01:12:40,680 --> 01:12:43,040
and scope our projects is to
say, you know, we don't on the

1397
01:12:43,040 --> 01:12:46,640
machine learning side, we don't
want to work on your problem if

1398
01:12:46,640 --> 01:12:48,640
it's only of interest to you.
If I can't apply it to some

1399
01:12:48,640 --> 01:12:51,640
other area, someone else, some
other domain, Because if that's

1400
01:12:51,640 --> 01:12:53,600
the case, I probably need to
know so much about your area

1401
01:12:53,600 --> 01:12:56,480
that I become, you know, an
expert in your particular area.

1402
01:12:56,480 --> 01:12:59,520
And similarly, on the scientific
side, you know, if you have a

1403
01:12:59,520 --> 01:13:02,720
particular method that is only
that's so heavily tailored to

1404
01:13:02,720 --> 01:13:05,080
your domain that it's not going
to be useful more broadly to

1405
01:13:05,080 --> 01:13:07,360
machine learning people, that's
a much harder sell.

1406
01:13:07,360 --> 01:13:09,600
Then you're not doing something
that satisfies both sides.

1407
01:13:10,080 --> 01:13:12,800
So something that I knew about
from working on years ago was

1408
01:13:12,800 --> 01:13:15,480
like sequence alignment in in
genetics, right?

1409
01:13:16,040 --> 01:13:18,880
Clearly an important problem,
not something that's portable

1410
01:13:18,920 --> 01:13:20,680
the particular album, not
something that's portable to

1411
01:13:20,680 --> 01:13:23,680
lots of other scientific areas,
but better machine learning

1412
01:13:23,680 --> 01:13:26,680
methods for solving
spatiotemporal forecasting

1413
01:13:26,680 --> 01:13:29,240
problems with non trivial
boundary constraints.

1414
01:13:29,680 --> 01:13:32,760
That's clearly of interest to a
pretty wide range of

1415
01:13:32,760 --> 01:13:36,040
applications.
And also to solve it, you're

1416
01:13:36,040 --> 01:13:38,120
going to have to introduce
technical solutions that are

1417
01:13:38,120 --> 01:13:40,720
probably, you know, go, can you
come up with a differential

1418
01:13:40,720 --> 01:13:43,760
optimizer for an end to end
differentiable system where you

1419
01:13:43,760 --> 01:13:45,760
have hard constraints?
I mean, that's clearly something

1420
01:13:45,760 --> 01:13:48,120
of interest to machine learning
and optimization.

1421
01:13:48,120 --> 01:13:51,200
So I guess I'd try and focus on
something there.

1422
01:13:51,200 --> 01:13:54,000
And that is probably the way
that you're going to be.

1423
01:13:54,000 --> 01:13:56,920
And unless you know exactly what
you want to be doing 30 years

1424
01:13:56,920 --> 01:14:00,480
from now, that's probably the
way to make yourself most

1425
01:14:00,480 --> 01:14:02,320
forward compatible with with
however.

1426
01:14:02,320 --> 01:14:04,920
That's a really, that's a really
good point.

1427
01:14:04,920 --> 01:14:07,480
I guess what you're saying is,
yeah, it's true.

1428
01:14:07,480 --> 01:14:09,880
Like later on in your career
you've become established in a

1429
01:14:09,880 --> 01:14:12,840
certain area, but most people
during their PhD or the postdoc

1430
01:14:12,840 --> 01:14:15,360
at that time have no real
necessary idea.

1431
01:14:15,800 --> 01:14:20,120
So you're saying if you do
something broad enough that can

1432
01:14:20,120 --> 01:14:25,760
get you exposed to different
groups and and have a foundation

1433
01:14:26,400 --> 01:14:29,920
to your earlier comment, you can
more easily go into verticals

1434
01:14:29,920 --> 01:14:31,360
because you've built that
foundation.

1435
01:14:31,360 --> 01:14:34,360
Whereas I guess if you jump
directly into a vertical right

1436
01:14:34,360 --> 01:14:39,200
from the beginning for you to
sort of reverse out that it's

1437
01:14:39,200 --> 01:14:41,720
kind of harder, isn't it?
So I I think that's what you're

1438
01:14:41,720 --> 01:14:43,560
alluding to it, it sets you up
better.

1439
01:14:43,920 --> 01:14:45,760
Yeah, yeah, because it's not at
all.

1440
01:14:45,840 --> 01:14:47,560
I mean, all the questions you're
asking are fair ones.

1441
01:14:47,560 --> 01:14:49,880
And it's not at all clear how
the whole area will evolve.

1442
01:14:51,600 --> 01:14:54,960
And so depending on how it
evolves, having experience in

1443
01:14:54,960 --> 01:14:57,880
one versus another.
And, and I think I mean, having

1444
01:14:57,880 --> 01:15:00,440
worked in a bunch of areas, you
know, it's, it's, but, but

1445
01:15:00,760 --> 01:15:03,400
speaking substantially only
English, but a but a little bit

1446
01:15:03,400 --> 01:15:06,960
of other things I can imagine.
And I see, you know, you learn a

1447
01:15:06,960 --> 01:15:09,560
second language, it's hard.
You learn a third language, the,

1448
01:15:09,560 --> 01:15:11,960
the relative cost is a lot less.
You learn a fourth.

1449
01:15:11,960 --> 01:15:15,600
I mean, so there's a diminishing
returns in terms of the amount

1450
01:15:15,600 --> 01:15:19,160
of extra effort you need to do.
So you work in one area, switch

1451
01:15:19,160 --> 01:15:21,360
to the second.
It's very hard switch to the

1452
01:15:21,360 --> 01:15:23,960
third.
You know, you can, you know, you

1453
01:15:23,960 --> 01:15:27,520
can, you can do that.
And after that, one thing I've

1454
01:15:27,520 --> 01:15:30,880
sort of been always struck by is
it's amazing how little you, you

1455
01:15:30,880 --> 01:15:32,880
need to know about a particular
area, especially if you're

1456
01:15:32,880 --> 01:15:34,880
working with good people who'll
complement what you're doing.

1457
01:15:34,880 --> 01:15:37,280
You can learn from them, right?
And so it's amazing what you

1458
01:15:37,280 --> 01:15:40,560
don't need to know in order to,
to get interesting results in an

1459
01:15:40,560 --> 01:15:44,200
area.
And so, and so being learning

1460
01:15:44,200 --> 01:15:45,720
new things.
By the time you know two or

1461
01:15:45,720 --> 01:15:47,280
three things, it's easy to learn
the 4th.

1462
01:15:47,840 --> 01:15:49,720
Yeah, yeah, I know that that
makes sense.

1463
01:15:50,280 --> 01:15:54,920
Well, thank you so much.
I it's the day after Labour Day,

1464
01:15:54,920 --> 01:15:59,000
which means that probably
there's a whole bunch of emails

1465
01:15:59,000 --> 01:16:01,360
and stuff that's started to come
through on essentially your

1466
01:16:01,360 --> 01:16:03,960
first day back.
And I know we probably could

1467
01:16:03,960 --> 01:16:06,320
have carried on for for another
couple of hours.

1468
01:16:06,320 --> 01:16:10,080
But yeah, thank you.
Really, really appreciate it and

1469
01:16:10,080 --> 01:16:14,640
look forward to catching up in
person at some conference in the

1470
01:16:14,640 --> 01:16:16,160
future.
Sounds great, thanks for having

1471
01:16:16,160 --> 01:16:17,920
me, this has been fun.
All right.

1472
01:16:18,240 --> 01:16:41,320
Cheers.
None.