What does it mean that our data is used to improve model performance?

In year 2000, the Clay Mathematics Institute of Cambridge, Massachusetts established the Millenium Prize Problems. These are seven well-known complex unsolved mathematical problems, and the institute pledged to pay one million US dollars for the first correct sollution to each problem.

Tristan Buckmaster, an NYU mathematics professor, had been collaborating with Levent Alpöge, a prominent mathematician, on solving the same problem for almost a year making extensive use of Claude and GPT 5.6 Sol. On August 15th, they had a breakthrough. They also got rummors that news of their solution, which had not been publicly announced, somehow had reached OpenAI.

On September 8, 2026, OpenAI announced they had found a solution for the Navier-Stokes existence and smootheness problem, one of the Millenium Prize Problems. OpenAI used an internal advanced LLM for arriving at the solution. Buckmaster published a statement about the coincidence that OpenAI had solved the problem after having notice of Buckmaster and Levent breakthrough and that OpenAI’s solution seems to use the same approach.

At this time, there is an ongoing discussion on the internet, and there are several implications to an LLM solving complex mathematical problems. However, what caught my attention was Simon Willison’s post Some thoughts on the Navier-Stokes Millenium Prize Problem. At the end of his post, he poses some questions about what it means that our data is used to improve model performance:

My two favorite hypothetical questions regarding this used to be:

• If I’m running Codex and one of my API keys accidentally gets consumed in the context, what are the chances that someone else might ask for an API key in the future and get mine back? (I asked someone at OpenAI once and they called this the “regurgitation” problem and assured me that they take great pains to prevent that… but wouldn’t describe how.)
• If I brainstorm with ChatGPT about potential new directions for my company, what’s the chance that information might be exposed to a competitor in six months’ time who asks “what might company X plan to do next”?

My new preferred hypothetical for this is:

• If I use ChatGPT to help me partially solve a Millennium Prize problem, what are the chances that my work will influence training such that a later model helps someone else solve it first?