Last week our CEO, David, shared some thoughts on mathematics and AI in an all staff email. We are sharing these in case they are of interest to our wider community.
Since ChatGPT came out, I have been giving it maths problems. A few years ago it struggled with even fairly simple mathematical reasoning tasks. But last week OpenAI announced that a near-unsupervised swarm of AI agents had produced a solution to one of the Millennium Prize problems.
This is yet to be verified and there are legitimate complaints to be heard about whether the solution really meets the spirit of the problem. But either way, we cannot understate the significance of this. If AI can solve problems that once defined the frontier of human mathematical achievement then what does that mean for mathematics? What does it mean for our pupils who are currently studying it?
Below are some of my thoughts. I don’t claim any of them to be correct – they are just musings. I’m sharing them as a contribution to our collective thinking.
The answer is only the beginning
As others have remarked (eg here and here), it has happened to be the case in mathematics to date that in order to solve hard novel problems one has had to generate new methods or understanding. This happy coincidence has meant that we have been comfortable prizing the solution of novel problems as the pinnacle of achievement in the discipline, even though what was most valuable was the generation of new ideas that could be applied beyond that specific problem.
AI is breaking this coincidence. We have had months of AI solving, or assisting humans to solve, novel problems. But so far, we have little novel understanding as a result.
I expect we are going to need to shift the incentive structure in mathematics away from solving problems into explaining solutions. What matters less is the computer-verified code of a proof; what matters more is the story of how one could recreate arriving at it.
Unfortunately this is not the current incentive structure. AI labs are rushing to announce solutions without doing the work needed for the discipline of mathematics to be enriched by those solutions. A recent notable proof was widely derided for being utterly incomprehensible, but the task of making it comprehensible fell on the community rather than the author (who had banked their proof and moved on). Kevin Buzzard, professor of pure mathematics at Imperial College London, makes this point in a dignified manner in his blog after Anthropic announced the Lean formalisation of Fermat’s Last Theorem, which he was very close to finishing.
We are at risk of knowing lots of answers but not knowing how to justify any of them. That has some value, but I’m not sure it has that much. This is not an inherent property of a future world with AI, rather it’s a reflection of our social norms and incentive structures. These need to change to reflect the new reality.
In this context I believe that our mantra at Axiom – the answer is only the beginning – is more important than ever.
Exhausting the field
What will we do when all the Millennium Prize problems (and other similar level problems that could’ve been picked for the prize but weren’t) get solved?
Terry Tao, professor of mathematics at UCLA, has mused on this here. The concern is that the manner in which we are AI-solving novel problems today (without generating novel understanding) means that we are not renewing the pool of worthwhile problems to solve.
School mathematics has long professed to focus on problem solving. Problem posing has been a much more niche pursuit. But arguably this will need to become an important part of how we train mathematicians. Our pupils should not be passive recipients of a production line of problems, but rather discerning choosers or creators of problems that they believe are worthy of their time.
Maths getting expensive
Kevin Buzzard makes the point that his research grant to formalise the proof of Fermat’s Last Theorem was £1m over five years. Claude estimates for me that Anthropic spent double or triple this in a couple of weeks to reach their proof (which was only part of what Kevin’s research grant was for). OpenAI used hundreds of billions of tokens on their most expensive internal model to solve the Millennium Prize problem, at an estimated compute cost of $20-30m (estimate according to Claude).
Maybe maths won’t stay this expensive. Open source and cheaper models are developing fast too, and tend to lag only 6-9 months behind the frontier. Two high school students, working with a postdoc through UCLA Maths Circle, used heavy AI assistance to solve a problem that Fields Medalist June Huh hadn’t been able to figure out. It is an early sign of what’s possible when frontier-level maths research becomes more accessible.
But moments like this are few and far between, and we don’t know if they’ll become the norm. Some of the hardest problems are always going to demand the best intelligence. If frontier problems will often be solved first by those with extremely deep pockets, what does that mean for academic mathematics? Is it sustainable for such solutions to be shared openly if they require so much investment, or will a return on that investment be required? At the moment labs are happy investing in maths as marketing spend. That cannot last forever.
Training for the new era
The people using frontier AI to solve maths problems were already frontier researchers before AI came out. They learned to do mathematics themselves or with other humans, and are now learning to work with AI to amplify what they’re able to achieve.
We need to reconceive of how we apprentice mathematicians. And we need to think about how we prepare young people to be able to wield these AI tools. Just as we have a theory of how and when to work with calculators, we need one for LLMs.
So what for our pupils?
Some have speculated that mathematics is dead. I think the opposite. We could be on the cusp of an explosion of discovery. I don’t know all the ways in which that will improve the world, but I know that it will. The “unreasonable effectiveness of mathematics” tells us that seemingly abstract or esoteric breakthroughs in mathematics turn out to be the foundations of major practical breakthroughs without which modern life would not be possible.
But I do not believe this is inevitable. Mathematics has only been unreasonably effective to date because there have been people who understood these esoteric discoveries, could spot their shadows in other disciplines, and then wrangle them into novel technologies that improve human life.
We are preparing the pupils who will bear this responsibility for the next generation. The power of AI is such that a single human who understands a field well enough to explore it – who knows its contours, who can feel where there is something exciting to be learned, and who seeks the reward of knowledge as well as the thrill of a solution – will be able to make more progress for human understanding than has ever been possible for humans before.
How many of these humans will there be? That’s up to us. We are responsible for raising them.