Hi and welcome to the Neil Ashton
podcast. In each episode we explain some
of the fascinating ways that science and
engineering are changing the world
around us.
We talk to leading engineers from elite
level sports like cycling and Formula 1
to some of the world's top academics to
understand how fluid dynamics, machine
learning, supercomputing are bringing in
a new era of discovery.
We also hear some of their life stories,
their career advice,
and lessons they've learned on the way
that I hope will be helpful to you, too.
So, sit back and enjoy this episode.
Welcome back to the Neil Ashton podcast.
I talked in the intro of this podcast
that we would talk about things like
fluid dynamics, we would talk about
aerodynamics, but one of the things I
also said was supercomputing and
high-performance computing. And I said
that because
nowadays with the huge increase in using
these simulation tools to simulate a car
or a plane or wind turbine, or develop
machine learning models like the ones
that you probably use day in day out now
with large language models, you need
computers to run them or to train them.
And that is what is known when it's done
in a very large scale and
high-performing way, high-performance
computing or supercomputing. And so I I
really wanted to have an episode that
discussed this so people who are very
familiar with this maybe got some extra
details and more insights in where the
future lies, or for people who are are
really not familiar at all just to make
you aware of this very important thing
in the world around us and that is so
important for scientific discovery.
So, I was very lucky that Professor Jack
Dongarra was willing to speak to me
because he is arguably one of the, you
know, most distinguished and and famous
people in the supercomputing and
high-performance computing world.
Um
that's because he
started
and really pushed this supercomputing
world, you know, he was there sort of in
the in the early days and is most well
known for two things. One is the
LINPACK, which is a software library
essentially for doing linear algebra,
but what it's used for is a way to
benchmark systems hardware. And still
today it is one of the main benchmarks
that are used to assess the TOP500
list, what are the TOP500 most powerful
supercomputers in the world. And we talk
about that in the episode. He has done
many other things, but he was also one
of the key contributors that developed
MPI, Message Passing Interface. And this
is a, you know, a function that is used
to program the majority of simulation
methods like CFD, like weather modeling.
But I think what he's also got to is
that point where he's been so aware and
been on so many advisory boards and
and also being, you know, a professor at
at Tennessee
and also at Oak Ridge National Lab, he's
so aware of what's going on in the
world. He's In recent years I found some
of his papers and commentary to be
incredibly insightful and very
interesting. And we talked through a
couple of those
in the interview with him. Really
focusing on a couple of key topics. One,
what is HPC?
What is the impact of machine learning
and AI on HPC?
Some of the discussions around the
leadership of the United States in
supercomputing, which for people who are
into this world is a sort of a hot topic
with the rise of China and other
countries. Um we also talk about just
the challenges and opportunities, what's
coming next.
And we finish with some more reflecting
on his career and advice to to people,
which is I think always a an important
thing. So, um we didn't have a huge
amount of time to discuss and so there
were many topics and and we didn't
really get into his whole career and his
life story, which, you know, I would
love to have done, but I'm hoping you
still find it um an interesting
discussion to listen to. So, yeah, sit
back and enjoy this discussion with
Professor Jack Dongarra.
I would start with maybe a very
high-level question that it seems
deceptively difficult for some people to
answer, which is how would you define
HPC?
So, high-performance computing, what is
it? High-performance computing is about
using the fastest computers that we have
to solve our problems, whatever problems
they are. So, high-performance computing
uses supercomputers
to help solve scientific problems as an
example. So, what's a supercomputer?
Supercomputers again would be the
fastest computer
that we have available to us for
for doing a computation. And now we have
to think about what's a computation.
So, in this context I would say if we're
thinking about scientific computations,
we're generally talking about doing
64-bit floating-point arithmetic.
Sometimes it's 32-bit floating-point
arithmetic.
With the advent of machine learning, we
have hardware now that can do 16-bit
floating point. They don't need as much
precision in terms of um
the computations they're doing. And we
even see the emergence of
8-bit floating-point arithmetic. And
maybe even less than that for some for
some machine learning. Now, those kind
of computations are hard to fit into
what I would call traditional scientific
computing,
but, you know, there is attempts being
made to do that. There's attempts being
made to
let's call it get an approximation to a
solution in a lower precision and then
do some do
some
mathematics to improve the accuracy and
come up with a solution that hopefully
is as good as what you would have gotten
doing 64-bit. Of course,
doing 64-bit is more expensive than
doing 32-bits and 32-bit is more
expensive than 16-bit. And that has to
do
with in some sense moving data and also
the hardware that's inherent in in the
floating-point units that are there.
It's been said that um
the one who has the fastest computer can
do the best science. Now, that's that's
perhaps an overreach, but there is some
truth in that.
Faster computer, the the supercomputers
that we have are
today in the
what's considered exascale range. So, we
have two examples of those that we know
about that are doing exascale computing.
So, exascale is defined as 10 to the 18
floating-point operations per second.
And that's 64-bit floating-point
operations.
So, that's that's a big that's a big
number, 10 to the 10 to the 18.
Yeah, um
there's two
two sides that I think people or at
least
myself, I'm always interested in where
the future lies a little bit. I I want
to know where the where things came
from, but I'm also always interested to
sort of see where things are going in
we've talked in in some other
discussions I've had with people around
the pure science. So, what's the future
of CFD or what's the future?
But it's interesting to maybe look at
that from a from a HPC point of view. Um
I saw I I really enjoyed the article
that you wrote, Reinventing HPC, the
challenges and opportunities. Um I I
know it's difficult task to summarize
that in a few minutes, but but what what
do you kind of see as the current
challenges but the opportunities
for for HPC over the coming years?
Right. So, let's see, the challenges.
Well, um
if we take a look at the computers that
we have today, they're
they're incredible devices, but
you know, they they suffer from the
standpoint of moving data. So, data
movement is the most expensive thing
that uh that we do on a computer today.
So, moving data from one part of the
machine to another part of the machine
where it's going to be used is very
expensive. So, it's much more expensive
than
doing floating-point operations. And
floating-point operations
today on our computers are really
over-provisioned, I feel. So, we have
too much floating-point capacity and we
have a hard time effectively using the
floating-point units because of passing
data to them. So, we spend most of the
time passing data. So, the challenge
would be to design a system that um
that can overcome some of that uh
some of that resistance in passing data.
You know, our our computers today, our
supercomputers today are bought or
purchased or acquired in a sort of a
strange way.
So, what we do is we we say we have a
desire to have a machine at this level
of performance
and we have this much money. So, then we
we tell the vendors we need a machine
that matches that performance with this
this um
this cap on it. So, I'm being very crude
in terms of the the the the parameters
here, but it's sort of like this. And
then a a vendor goes off and cobbles
together a machine usually out of
commodity parts and
bids that machine. And if they win the
bid, and you know, there's a lot of
things that go into winning a bid and
making making things work, then they put
together a machine and it basically gets
thrown over the fence.
and we on the other side of that fence
the people who who are intended to use
that computer sort of scramble for the
next three or four years to figure out
how to use it effectively.
And that that scrambling, you know, is
an incredibly intensive process of
optimizing
redoing algorithms and redesigning
algorithms, creating new algorithms that
can
effectively function in the parameters
of that new machine.
So, we're doing you know, in some sense
we're doing that incorrectly. So, we're
not we're not designing a machine that's
suited for the scientific problems that
we have. We're designing a machine based
on
how much money we have and what the
expected performance could be in terms
of peak performance of all things. So,
we end up with a computer that um
you know, for for some
for some outdated benchmark perhaps can
reach the highest levels we asked for,
but for realistic problems we run we run
much less than that. And the example
that I usually
drag out is something perhaps I'm
responsible for
is is the LINPACK benchmark. So, LINPACK
benchmark is is solving a dense system
of linear equations
using a standard algorithm. The
algorithm is Gaussian elimination with
partial pivoting. It's the algorithm I
learned when I was in high school for
solving a system of linear equations.
And the benchmark is to use that
algorithm to solve a dense matrix
problem and to look at the performance
with 64-bit accuracy.
So,
you know, that problem is at its core
doing a matrix multiply. So, if your
hardware can do matrix multiply well,
that algorithm is going to run very
well. Now, that that that algorithm
is not is not very relevant to today's
scientific computations. That is to say,
the computations we do on
supercomputers, they usually don't look
like matrix multiply. They look like
something with a sparse data structure.
So, it's solving the same problem by the
way. So,
solving a solving this LINPACK benchmark
solves
a system of linear equations Ax = b. So,
you're given A and you're given B and
you're asked to compute X and you're
told to use Gaussian elimination with
partial pivoting and that matrix A is
dense. So, all the elements are filled
in with nonzero
element nonzero components and you're
asked to use that algorithm. We have
another benchmark
that's called the high performance
conjugate gradients benchmark HPCG.
And that benchmark again solves that
same problem Ax = b, but the data
structures are different for the matrix
A. The data is sparse. So, we have a
sparse matrix representation for the
problem. And that sparse matrix is
similar the way in which it's formed is
similar to what we typically use
supercomputers to to do and that's
solve a three-dimensional partial
differential equation. So, when we
discretize that that differential
equation, we end up with a matrix
problem. That matrix problem is sparse.
And we use an iterative method to solve
it. So, it's not solved in the in the
sense that we solve that dense problem.
It's using a different approach to
solving it and that that approach does
not have matrix multiply at its core. It
has a sparse matrix vector product which
requires you to move data and get get it
gets very little reuse of the data. So,
the data is basically passed into the
functional units,
computations done and then
there's no reuse of that data and you
have to do it again for the next for the
next iteration. And just in terms of um
raw numbers,
the the dense matrix problem, we can get
probably 70 to 80% of the theoretical
peak performance in 64-bit precision.
For the sparse problem unfortunately
we get less than 3% of the theoretical
peak performance. So, problems that we
are intending to solve on these
supercomputers, we're ending up with 3%
of
of efficiency, let's call it. And
you know, is that is that a good or a
bad thing? Well, I would say that's not
a very good use of the resources that we
have. And you know, why is that the
case? Why do we only get 3%? It's
because we're we're we don't have a good
way to pass data around. And we've
cobbled together a machine based on
commodity parts
CPUs, commodity GPUs, commodity
interconnects,
commodity memory and all of that comes
together to give us a machine with a
very high theoretical peak performance,
but very very inefficient in terms of
how it's used.
How much I I guess you're alluding to
that all of one of the conclusions
around co-design. Are you essentially
saying that really the hardware and the
software should be more tightly sort of
coupled together rather than being
I mean, how how can that be addressed I
suppose pragmatically or practically?
Right. So, we had a mantra with some of
the Department of Energy projects. For
the mantra was to to do co-design. What
is co-design? Well, it's to get the
hardware people together with the with
the with the computational scientists
together with the guys designing
algorithms that will solve those
computational problems along with the
software people that are going to write
software for doing those for doing those
algorithms. So, it gets the
the architects together with the
scientists together with the computer
people
who are doing things. And that's they're
going to do co-design a machine based on
the aspects of the applications that
they have, how to make those
applications run as efficiently as
possible on some architecture. And
working with the architects come up with
a piece of hardware that can satisfy
that problem.
So, that's the mantra. In the end what
we got is somebody throwing a machine
over the fence and the computational
people scramble
for the next three years trying to
figure out how to use it. And there is
no real co-design of that machine. So,
again, we're stuck using commodity
processors. We've lost in some sense the
you know, I can remember back when I was
in school,
there were many universities doing
architectural work in terms of building
experimental machines
that
potentially could could solve
challenging problems. And I think we've
lost some of that. So, we've lost some
of the ability to do that
fundamental research in terms of
designing architectures
that would meet the needs of the
computational science community. So,
maybe we should go back
and
invest I'll say in that in that effort.
That is um
invest in terms of allowing those
universities to do architectural
research that would benefit scientific
computing. Architectural research is
being done of course in in universities,
but
with the focus being on scientific
computing I think is the thing that
we've lost.
Do you think um
that
you know, AI machine learning has been a
force for good for HPC in the sense that
it's obviously a user of HPC and has
driven huge new investment or has it
been negative because it's sort of
diverted that to from computational
science towards a different discipline
that maybe doesn't suit it, you know?
So, I would say AI is and machine
learning are are causing a revolution to
take place in how we look at scientific
problems.
Almost every scientific application that
I see today is embedding machine
learning and artificial intelligence in
it. And it's having a very positive
effect in terms of
allowing those applications to get a
better solution, better time to
solution, a more accurate solution. And
I'm not saying that they they replace
the traditional classical approaches.
What they do is they augment them.
They're a tool that's being used now by
the computational scientists. So, it's a
it's a tremendous positive influence in
terms of how we solve problems and that
that's going to be a great benefit as we
go forward and really understand how we
can use the uh the technology
and use it as a tool
to to solve our problems much better
than we uh
than we're doing today. So, it's it's
really a um
it's a thing which will help us amplify
our ability to solve these problems.
But do you um
I'm I'm apologies if I'm wrong on this
one, but
is there therefore a logic that some of
the
methods to evaluate the TOP500 should
be more skewed to include machine
learning given that increasingly a lot
of these supercomputers will run more
and more machine learning workloads with
different algorithms and methods?
So, so I guess you know, as a as a
person who has developed
accidentally benchmarks that have that
have gotten getting gotten take up in
the community, I'll say that we should
have a lot of benchmarks. We shouldn't
just have one or two benchmarks. We
should have a bunch of benchmarks. The
benchmarks should have should
reflect the applications that were going
to use these machines for. So, having
many benchmarks
allows us maybe to dial up
something
these benchmarks to get a handle on what
our what our application will look like
when run on these machines. So, we
should of course keep the existing
benchmarks. There is a lot of historical
information,
trends, and other things that we get out
of them.
So, it's been very informative and and
has been beneficial.
So, I would advocate keeping them, not
throwing them away, but I would say we
should augment them and you know, we've
tried to do that. I've tried to do that
just as as I go through trying to
understand how effectively they use the
machines. So, HPL was something that was
done, you know, to be honest, it was
done in the late 70s. So, we've got
something that has
you know, almost a what is that? 50-year
horizon here and
the TOP500's been around 30 plus years.
So, that that's an historical snapshot
of that that information in some sense
for high-performance computing. And now
we have this HPCG benchmark, which is
you know, again, looking at something a
little bit different but more relevant
perhaps to the applications that we do.
And you know, there are other machine
learning benchmarks that I think um
need to be understood and explored.
But you know, with these benchmarks, you
know, we want to keep them around for a
long period. We want to make sure they
reflect really what's going on in the
applications and what we should what we
can expect.
So,
some of that is unknown unknown
territory
today. So, I would say you know, we're
in a learning process and it'll take
some time for the right
benchmarks to um
perhaps to emerge. But I think you know,
we should be working towards a
whole
suite of benchmarks that can be used to
evaluate a system rather than just
focusing on one. We've tried to address
that in my group. We tried to address
that by um by tweaking, I'll say, um,
the HPL benchmark. So, the benchmark
that does a dense matrix computation.
Tweaking in the sense that we're trying
to understand what we can get out of
using mixed precision. So, the mixed
precision is can we get
can we get by with
low precision
start to the problem and then do some
mathematics to improve the solution
to the same that we would have gotten
had we done everything in 64-bit. So,
there's
there's a benchmark that goes by the
name of HPL-MxP.
MxP is for mixed precision. So, it it
allows the use of 16-bit 32-bit
arithmetic. It has a problem a dense
matrix problem that you're asked to
solve and there's a technique that we
have as a reference implementation which
uses a
iteration. It it constructs a
approximate solution in lower precision
and then uses an iterative technique on
that dense matrix problem to
converge to a much higher
accuracy result. And
we can show you know, very effective
very efficient speedups over time based
on that. And that's one of the
benchmarks that the guys at Argonne and
and also at Oak Ridge ran and a bunch of
other places have run
to show
significant speedups. So, there's
there's a need for many benchmarks and
those benchmarks should reflect the
applications that we're going to be
using on the machines.
But that's an interesting point about
the precision because um
that was kind of my question before a
little bit in terms of
the positivity, the negativity, the
direction with machine learning in the
sense
for many computational things, mixed
precision or double
seems to be what's required and and it
but it's good obviously there's been
work to see if there's a way around it.
But obviously a lot of machine learning
methods
can get huge speedups from using lower
precision
and there's a lot of money in that for
the chip vendors and the hyperscalers.
So, I guess
is there a slight
is there a need to sort of not get
overhyped by machine learning in the
sense you still have to remember that
there are some things that require
um
one chip cannot solve it all, I guess is
the
is that the right way?
Well, I'm I'm thankful that NVIDIA still
does 64-bit arithmetic in their in their
accelerators. So, that's that's the
first thing. Thank you very much for
keeping that real estate there
as opposed to selling out totally and
just doing 16-bit arithmetic, which is
what machine learning
needs to to do its to its problem. So,
yeah, I think that there's there's room
there to you know, I I think we're
we're trying to understand how we can
push the precision, the the accuracy,
and what we can do to enhance the
accuracy once we've gotten a result and
that's that's current research. So,
there's a lot of activity there. There's
a there's a
cottage industry, I'll call it,
trying to use mixed mixed precision
to solve scientific problems, trying to
exploit that 16-bit arithmetic as much
as possible. And and I've I've seen some
some reasonable results from that. So,
so I would say that's a positive thing
to exploit.
You know, again, I'm I want I'm thankful
that that we have 64-bit. And I saw a
paper recently which was quite
interesting. They were using um
integer arithmetic. So,
the NVIDIA GPUs have the ability to do
fixed-point arithmetic very fast.
So, it's you know, much faster than
floating-point 16-bit fixed-point
arithmetic, which gets you a 32-bit
result. Fixed-point
integer arithmetic and that's being used
to actually implement
the procedure for doing floating-point
matrix multiply. So, they essentially
think of you have two matrices you want
to multiply together and the algorithm
basically separates the numbers and bins
them in a certain way so that they can
do fixed-point arithmetic on those bins
of numbers and then
upscale that to floating-point and get a
get a floating-point result as a as a
result of that very efficiently. And it
can be encapsulated so that the user
doesn't even know that's happening. So,
it's under the covers. They're they
basically do a matrix multiply and they
they're expecting a 64-bit result and
under the covers it's being done in
integer arithmetic with the correct
precision with the correct ability to
detect errors in it with all the
floating-point
things we we would like to see from my
triple E arithmetic. In the end they get
a result which is done much faster but
using that energy unit. There are some
there are some hiccups along the way but
that's a that's a that's a very
interesting approach to dealing with
the situation of the hardware providing
a capability and exploiting that
hardware capability to get a results
that can be used.
Two questions for you. One,
is it really necessary
in the current age to have
a single system
with
50,000 GPUs or 100,000 GPUs? Does
anybody actually use the full system?
Or
is it actually
more economical or more effective to
have multiple systems, whether it's a
cloud or not,
because nobody runs on the full system
anyway. And or is it is there a bigger
picture of politics and and needing to
hit numbers? I'm I'd be interested to
get your take on the sort of
the need for just one massive machine.
Right.
So, right. So,
so the question is should I should I
build one exascale machine or get a
thousand petascale machines? That's the
equivalent of that question perhaps.
Yeah. So,
so let me say that the Department of
Energy is investing in exascale
computing and they've um they've
you know, put together a program called
the Exascale Computing
program, the ECP, along with
the Exascale Computing Initiative
and it's a it's a it's a program that
was over seven years and it was designed
to produce the those exascale computers.
So, they get three exascale computers.
The price tag is $600 million for each
one of those and the whole program is $4
billion. So, $4 billion over seven years
to get to exascale. That clock was set a
little over six seven years ago. The
program has run its course. It's spent
those $4 billion and we have three
exascale machines. One is at Oak Ridge
National Lab called Frontier. Another is
at Argonne National Laboratory that's
called Aurora. And uh third one is a
machine that's uh being constructed
today at Lawrence Livermore National Lab
called El Capitan. So, those are
hardware machines
uh that all have exascale plus
capabilities. And the uh the the uh the
purpose of those machines are to solve
some very challenging problems. So, uh
the Department of Energy in this program
defined, I think it was 20 uh 21 or 24
applications that they were going to
target to use the exascale machine. And
in the course of uh solving those
problems, not solving them, but using
those exascale machines to push the
boundaries for those problems forward,
um it's intended for those exascale
machines to be used at some point uh
during during the course of um its life
to be used the whole machine to be used
uh to solve or to be used in the
solution for some aspects of those 21
problems. So, yes, those machines are
intended to be used at uh at um at their
full
uh scale
um to solve one problem uh to help solve
that one problem. And um
uh when it's not being used to solve
that one problem, of course, it's being
used to solve many problems uh
simultaneously through a time-shared uh
through a uh system of using that
machine. And um
uh so, first So, that says there is a
need for the exascale machine. We
couldn't get by with a thousand
petascale machines uh to solve those
very challenging problems, which have
been designated as challenging problems.
Um uh the other um part of your uh
question, I think, is something that I
feel very envious about. So, the
hyperscalers can go off and build the
machine which is co-designed. It's it is
co-designed, you know, there Here's the
problem that we want to solve. We're
going to build hardware to help us solve
that problem. And they build hardware to
do that. So, you know, you go down the
list. Um
uh A- Amazon has their own hardware.
Microsoft has their hardware. Uh Google
has their hardware. So, they've invested
in that. They're They have enough
resources that they can do that. So, uh
you know, the statement that we make, I
think, in in one of the papers is the
exascale; hyperscalers uh hyperscalers uh
they're uh they have resources.
Uh they have tremendous amount of
resources, so they're exothermic. They
can they can do that investment in terms
of building hardware to help them solve
the specific problems that they have.
Where in the in the high-performance
computing scientific domain, which is uh
the research side with the DOE, NSF,
we're we're not we're endothermic. We
don't have enough resources to solve our
problems. And that leads us down that
path which says we're going to cobble
together a machine based on commodity
parts that are available and hopefully
get something that can be used to help
us solve our high-performance computing
problems. And um
you know, the machine is not co-designed
in the sense that the hyperscalers go
off and co-design a machine with very
specific requirements to solve the kind
of problem that they have. So, again,
I'm envious that we can't do that in the
high-performance computing. I think we
should uh invest in terms of making uh
allowing research to take place so that
we can investigate what architectures
would be the right
Is the US still the leader
in in HPC? I found it very interesting
the anal- I can't Sorry, I I will put
the a copy of it in the in the comments
of this whole thing, but but you you
were the
um I think the main author along with
some collaborators who who defined this.
I
I I think you know what I'm talking
about. I'm interested to get your
perspective Um
Yeah. on that and what should be done to
address
Right. So, so, so, this is AI was part
of a Department of Energy uh
uh study that was done for the Office of
Science on uh uh
US competi- competitiveness in this in
this area. So,
So, there's a number of issues there,
some of which we've touched on, uh but
the elephant in the room is
uh
does the US have uh the same or greater
capabilities than China uh in terms of
high-performance computing? Uh so, I
think we can say that um you know,
Europe is developing high-performance
computing, you know, they have a lot of
programs in place. Uh they're
European-centric in some sense. Uh they
want to use RISC-V. They have Arm
processors that are they're looking at.
They have a number of things that are
based on uh things which are more um
uh
things that that that would benefit the
European uh
Commission the European Union. And um
and I think that's fine. That's a great
thing, and you know, we'll we'll learn a
little bit from what they're we'll learn
from what they're doing. So, that's a
good thing. Japan, you know, has their
own hardware that they've developed very
impressive uh systems. Uh the Fugaku
system is their top-tier machine, which
has very uh very strong capabilities.
You know, it's one of the it's one of
the machines in the top 10
that are not using GPUs. So, they have
vector architecture uh at their
disposal, and uh they're using an Arm
processor. They've augmented it with
vector instructions, but their uh secret
sauce is the is their interconnect
network is very efficient. And um it's
through that efficiency uh that they can
get very good performance, and they can
out, you know, on tops for things that
matter in terms of data movement. Uh the
other one is China. China um
is interesting, you know, they've
developed
They They have been forced into
developing their own hardware. So, that
the forcing was the US putting sanctions
on
technology going to China. So, it was
initially processors, then GPUs, and now
uh you know, the the ability to use TSMC
for
uh for for fabric for fabrication is is
taken out of their hands.
Uh so, China, uh perhaps in a reaction
to that, has um
stopped uh submitting
uh benchmark numbers for uh Top500. So,
the So, we've seen no no new machines um
coming um
out of the out of the uh China
uh that are no new entries coming uh
from uh submitting uh numbers uh to the
Top500. So, that's um
you know, that's a that's a bit
you know, they're taking a very strong
uh stance, I would say, on that. And you
know, what's the reason for that? Well,
they're afraid the US will take more
action is the only thing I can think of.
Uh so, again, it's a situation where
they're reacting in that way. And you
know, again, China's pivoted, and
they're designing their own hardware for
their high-performance machines. They
have uh you know, their own chips that
they're designing. Uh question is, where
are those chips fabricated? Uh I would
guess they're fabricated in Taiwan, but
you know, they have fabrication
facilities in China now, which are, you
know, a little bit below what what one
can do in at TSMC. And uh those chips
will probably be fabricated there for
the next uh generation of uh hardware
that they're producing. So, it's
unfortunate that they've shut the
window, so we can't really see what's
going on. You know, occasionally they
write a scientific paper which describes
the hardware, so we get a glimpse of
what that hardware looks like, uh but we
don't have a real good uh benchmark
associated with it. We can't see how
well it does provide. So, so, that's a
that's a bit unfortunate, uh but that's
uh that's life. You know, we have this
thing called the Gordon Bell Prize. So,
Gordon Bell Pri- Gordon Bell passed away
just recently. Gordon uh set up this uh
prize that would uh uh set up a
competition to see who can improve on
real applications using high-performance
computing. It's sort of the the short
version of what the prize is about. And
uh it's uh you know, it's competed, and
uh people submit a paper that describes
the application and the results that
they got on their
uh supercomputer. Um and um we're now
looking at uh the current uh entries
that are being submitted. So, I'm I'm
one of the uh
uh judges who are looking at the entries
there. And you know, China's submitting
uh
results for uh applications that are
running on their current generation of
supercomputers. Those supercomputers are
described in the machine to some extent,
and uh we can see the performance that
they're getting on those applications,
not on a standard benchmark where we
have uh perhaps uh equal uh
understanding of what was done and how
it was done. So, so, you know, they're
they're making progress. Um they have
high-performance machines. The US is uh
certainly competitive uh to some extent.
Uh you know, there's claims that the
Chinese machines are faster for some of
these benchmarks. Uh you know, that may
be the case. Uh but uh you know, the
China's is competitive with uh what what
we see from the US systems. And if you
fast-forward, you mentioned before about
the exascale project. You know, it was a
seven-year project to to sort of reach
these frontier El Capitan, etc. If you
were to fast-forward seven years
from now,
is there the investment in place to make
an equal leap?
Or is it
the scenario that
we won't make the same progression that
we made, and other countries that have
doubled down will make a bigger
gain?
Right. So, one has to make an investment
to to get the advances in in hardware,
applications, algorithms, and software.
Um uh the ECP project was a wonderful
project. It uh it was uh seven years of
long-term funding. It in- involved 800
people uh working on high-performance
computing from applications, from
software, from algorithms, all received
funding to to advance things to further
as well as the hardware being purchased.
So, half the money went for
hardware and the other half went for
algorithms and software and and
applications. So, that was a interesting
split of that of that funding. But, it
ended. The project ended. And the the
sad part about that
is there's no follow-on project. So, we
had 800 people working on this this
effort and it ended. It ended in
December of last year and
those 800 people are scrambling for
jobs. So, we have very highly trained
people who understand high-performance
computing at various levels who are now
maybe out of a job and so they're
scrambling still to find find its way.
So, I think it's
it's a sad day for Department of Energy
who invested in in the
algorithm software applications as well
as the hardware and now those people who
designed those algorithms, applications,
and software
may be looking for a job. And you know,
I know that many of the people are being
sucked up by
the hyperscalers
companies like NVIDIA to develop their
their technology. So, that's a that's a
loss I would say for the community
and that loss is hard to replace. You
can't just spin that up overnight. If
you wanted to restart a program, you
can't restart the program. So, it's it's
a it's a
situation where we would have to follow
on. And you know, there there's an
attempt to follow on with AI for science
and you know, that that process is
undergoing and getting off the ground
now
but it it wasn't a clear follow-on. It
wasn't something that was dovetailed
into by the by the shutdown of the ECP
project. So, you know, I think it's
important. I think the investment needs
to be made. We need to have people who
can guarantee a job long-term. The
talents that they bring with them are
are hard to replace if they if they go
away and we're seeing some of that drain
today
in
in going to the in going to the
hyperscaler companies. You know, I I
feel DOE shouldn't be a minor league
team for the for the hyperscaler guys.
They they should be the major leagues as
well. The people should be funded for
doing that in the long-term.
Yeah.
Um
One
question
I guess still on the future
peering into the future a little bit is
obviously
depending on where you are in the
computer science or HPC world, either
you're very
full-on with GPUs and you've known about
it for decades like yourself or or it
still seems quite new.
But, that I guess will come to an end as
in the sense of the acceleration, the
performance improvement, the new Are
there any technologies that you see
coming that would just as GPUs has oddly
given a big boost from the traditional
x86 CPU? Do you see a new technology
around the horizon that people should be
sort of reading up about or being more
aware of than they currently are?
Right. So, if you look back and and look
at where we've been, you know, we've had
we've had originally we had machines
which were special purpose machines for
doing scientific computing. You go look
at companies like Cray, CDC,
Hitachi, Fujitsu
you know, built machines which were
specific for that but the market
couldn't sustain them. And we had
microprocessors come up and and be more
powerful. So, we had special purpose
machines, vector computers,
then we had multi-core computers which
was the architecture that sort of led to
the downfall of those vector-based
systems, the attack of the killer
micros. And then we had
parallel computing come into place and
then we had
GPUs. I'm sort of simplifying this but
that's sort of the the progression that
we had in terms of getting to the point
where we are now. So, what's next is
your question and you know, my crystal
ball isn't that good to to predict
what's the next big thing. But, you
know, machine learning comes into it.
It's a tool that we're going to use to
help us solve the problems but it's not
architectural um
uh advancement that we have. You know,
we think about
uh what what are the changes in the
architecture that we might envision we
can see
we can see things like um
uh
you know, neuromorphic computing. We see
things like um uh
you know, optical computing. We see
things like um
uh
you know, we see things like potentially
way in the future quantum-based
computing. Those are all things which
could could make you know, perhaps
make improvements where we get that leap
uh which we gotten
we've had in the past with those
generations of things. So, but you know,
quantum computing's way off into the
future is my my feeling. You know, a lot
of So, so it's a it's an important
topic. We should invest in the research
into it. It's something which you know,
has promise of course
but it's not going to replace replace
our existing machines in the near term
or not in my lifetime. So, it's it's
something which needs to be
worked on and figured out how we can
effectively use them. And it's not it's
not going to be the thing that that
replaces our high-performance machines.
It's going to be something that we add
to the to the ensemble that we use
to attack problems. So, again, we have
CPUs and GPUs and optical and
neuromorphic and
DNA-based stuff maybe and you know,
going going
adding quantum to that mix is something
that perhaps can happen and that would
then lead to
problems that could use those devices
perhaps giving a boost in terms of how
they
how they solve their problems. Mhm.
I I did want to change track a little
bit because you've um
you've had a an amazing
career. Obviously, you're very modest
in when you mentioned about oh, you
know, I kind of involved a little bit
with some benchmarking obviously with
LINPACK and and MPI and many other
things that you contributed to. I know
it's a difficult question to ask but do
you have any standout
sort of memorable things from your
career that were sort of a
wow, I
even just a day or a moment standing on
a stage or doing Is there anything that
you know, you sit when you
are having a drink and you sort of think
about what you've been doing? Is there
anything that stands out? Uh yes, that's
a hard question to answer.
You know, I like
I like to think I was I I've contributed
to three three things. And one thing is
developing software and algorithms for
solving
some standard problems in linear
algebra. So, that's that's something I
think I've I've contributed to.
And along with that goes portability and
performance. And the second thing is
tools for doing distributed computing.
So, there I'm I put in the basket
you know, MPI. So, I didn't I wasn't
solely around to do MPI. There was a
group of people of course who
contributed to the standard but you
know, we were all there together at
ground zero. And
the third thing is about performance
evaluation. And there again it's an
accident in some sense. So, the way I
think of this is
my true passion is with linear algebra
and developing software
that is portable and can be run
efficiently. So, that's that's the goal
and along the way we needed tools to
make that happen. So, we needed tools to
do the evaluation. So, that's where the
benchmarking comes in. So, we developed
tools which exposed how well that
software would would run on these
systems. And then we needed as the
architectures changed, went from
sequential to vector to parallel, we
needed some way to to engage with
parallel processing. You know, when we
started doing MPI, it was the wild west
in terms of message passing. Each vendor
had their own way of doing it. And we we
recognized there was a community that
recognized
we needed to standardize it. And it
wasn't going to happen by the vendors.
It was going to happen in a organic way
from the ground up. And we formed a
committee to a de facto committee to
look at what would we need in terms of
mechanisms for doing message passing on
those systems that could be portable and
efficient. And out of that came came
MPI. So, those are the those are sort of
the mixes. So, if I if I was saying my
contribution, what was I
what did I feel most proud about,
it must be the linear algebra software
and then these benchmarking things and
and the message passing stuff comes
along
as a necessity to doing that linear
algebra stuff.
What about
I don't want to say regrets but are
there anything you really wish I'd done
that or I missed that opportunity or you
know, is there anything where you look
back and you wish you'd done?
I wish I'd done. So, I don't I don't
have regrets in that context. I mean,
everything I have to say, you know, I
people come up and ask me I want to do
what you did. How did you do it? And I
say, well, it was serendipitous. I can't
give you a formula for it. You know, I
wanted to be a high school science
teacher. So, that was my ambition
when I started and then things changed
along the way and I can't reproduce
those changes. It just happened. I
happened to be chosen to work at Argonne
National Lab, spend a semester with a
scientist. And you know, that was
transformational. In terms of
looking at or following, you know, it
really opened my eyes and I saw passion
for doing these things and you know, I
changed the course of what I wanted to
do based on that. And I think I was in
the right place at the right time for
many many of the events that that have
occurred. So it's hard to
replicate that.
You know, I tell people
I tell people, you know, in research we
should expect to fail. So that's that's
an important lesson that everybody in
research should
going into research should understand.
You know, it's a process where I I don't
know the solution at a time. I can't I
can't give you what what it's going to
be, but I'm going to try things and
through that experiment experimental
process, I I may hit on something which
is going to lead to the right thing. So
you know, expect to fail if you're going
to do research. You know, follow your
passion, do something that you feel
passionate about
and and take take that forward. You
should
you know, networking is an important
thing.
Talking to other people, interacting
with people trying to
in solving a problem, we often need some
other stimulus and talking to people is
a good way to to get that. You know,
throw something up against the wall and
let other people see it and and how how
they react is is going to be important.
And I guess the other thing is you know,
aim high. Try not to solve try to solve
challenging problems. Try to solve
problems which
are stretching what you can do and what
others have tried to do. So it again
gives you an opportunity to
to advance the field and to do that. So
those are my recommendations I tell my
students. This for
And are you
are you optimistic about the future of
HPC and scientific computing?
Oh, absolutely. Yeah, this is a great
time. So um
and it's been a great time for a while.
So it continues to be a great time I
guess I should say. And it's a great
time because of all of the things that
that are happening in the field in high
performance computing and um
you know, AI is part of that.
Machine learning is certainly there.
You know, looking for the next big
advance in terms of architectural
features. That's a that's an important
thing.
For my standpoint, you know, how can I
bring
all this mathematical knowledge that's
been accumulated over time in solving
the next set of problems on the current
on the next
the problems that we have on the next
generation of architectures.
So that requires innovation and some of
that innovation comes about because of
not not creating something, but using
something that people have created in
the past and bringing it forward to to
be used on today's environment.
Well, thank you. I know that
you're a very busy person, so I want to
be respectful of your time, but I really
appreciate you coming to speak. I think
you very eloquently are They always say
Someone told me that you're a true
expert when you can explain things
simply. What I really like is that
you're Anytime I ask you a question, you
very I love how you you simplified it in
such a way and that's a real sign of of
expertise. So I know you're a university
professor explaining.
But thank you very much and Well, thank
you for that compliment. It was it was
fun to to talk to you and I look look
forward to to seeing seeing what what
you produce.
All right. Cheers.
Yeah.