Google DeepMind researchers think they found a solution to AI's 'peak data' problem

• The AI industry has hit "peak data," OpenAI cofounder Ilya Sutskever said recently.
• DeepMind researchers see outputs from new "reasoning" models as a source of fresh AI training data.
• A new AI technique, known as test-time compute, will be put to the test in 2025.

OpenAI cofounder Ilya Sutskever announced something at a recent conference that should have had the AI industry trembling with fear.

"We've achieved peak data and there'll be no more," he intoned during a speech at the annual Neurips event in December.

All the useful data on the internet has already been used to train AI models. This process, known as pre-training, produced many recent generative AI gains, including ChatGPT. Improvements have slowed, though, and Sutskever said this era "will unquestionably end."

That's a frightening prospect because trillions of dollars in stock market value and AI investment are riding on models continuing to get better.

Yet, most AI experts don't seem that worried. Why?

Inference-time compute

There may be a way to get around this data wall. It's related to a relatively new technique that helps AI models "think" through challenging tasks for longer.

The approach, called test-time or inference-time compute, slices queries into smaller tasks, turning each into a new prompt that the model tackles. Each step requires running a new request, which is known as the inference stage in AI.

This produces a chain of reasoning in which each part of the problem is tackled. The model doesn't move on to the next stage until it gets each part right and ultimately comes up with a better final response.

OpenAI released a model called o1 in September that uses inference-time compute. That was followed swiftly by Google and Chinese AI lab DeepSeek, which rolled out similar "reasoning" models. 

"An iterative self-improvement loop"

Benchmark-based testing of these new models has shown that they often generate better outputs than the previous top AI crop, especially on math questions and similar tasks with clear final answers.

This is where things get interesting. What if these higher-quality outputs were used for new training data? This mountain of new information could be fed back into other AI model training runs to produce even better results.

Google DeepMind researchers published research on test-time compute in August and proposed this technique as a potential way to keep large language models improving through the peak-data wall.

"In the future, we envision that the outputs of applying additional test-time compute can be distilled back into the base LLM, enabling an iterative self-improvement loop," the researchers wrote. "To this end, future work should extend our findings and study how the outputs of applying test-time compute can be used to improve the base LLM itself."

A chat with a test-time researcher

The authors were Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Xu is still at Google, and Kumar spends some of his time at DeepMind, while Lee left to join OpenAI rival Anthropic.

Snell co-wrote the paper while interning at Google DeepMind. He's back at UC Berkeley now, so I called him up to ask what inspired the research.

"I was motivated by some of the things that have been preventing pre-training from continuing to scale, notably the finite supply of data," he told me in a recent interview. 

"If you can get an AI model to use extra inference-time compute and improve its outputs, that's a way for it to generate better synthetic data," he added. "That's a useful new source of training data. This seems to be a promising way to get around these pre-training data bottlenecks."

Satya satisfied

On a recent video podcast, Microsoft CEO Satya Nadella seemed unperturbed and even buoyant when asked about the slowdown in AI model improvement and the lack of new quality training data. 

He described inference-time compute as "another scaling law."

"So you have pre-training, and then you have effectively this test-time sampling that then creates the tokens that can go back into pre-training, creating even more powerful models that then are running on your inference," he explained.

"That's I think a fantastic way to increase model capability," Nadella added, with a smile. 

Sutskever also mentioned test-time compute as one possible solution to the peak-data problem, during his Neurips talk in early December.

Test time for test-time compute

2025 will see this approach put to the test. It's not a slam-dunk, although Snell is optimistic. 

"Over the last three years or so, it seemed more clear," he said of AI progress. "Now we're in this exploratory mode."

One open question: How well does this test-time compute technique generalize? Snell said it performs well with questions where the answer is knowable and you can check it, such as a math challenge.

"But a lot of things that need reasoning aren't easy to check. For instance, writing an essay. There's often no straight answer on how good this is," he explained. 

Still, there are early signs of success and Snell suspects outputs from these types of reasoning AI models are already being used to train new models. 

"There's a good chance that this synthetic data is better than what's out on the internet," he said.

If outputs from OpenAI's o1 model are better than GPT-4, the startup's previous top model, then these new outputs can in theory be reused for future AI model training, Snell explained. 

He shared a theoretical example: Say o1 gets a 90% score on a particular AI benchmark, you could take those answers and feed them to GPT-4 and get that model up to 90%, too. 

"If you have a large set of prompts, you could get a bunch of data from o1 and create a large set of training examples and pre-train a new model on them, or continue training GPT-4 to be better," Snell said. 

A TechCrunch report from late December suggested that DeepSeek may have used outputs from OpenAI's o1 to train its own AI model. Its latest offering, called DeepSeek V3, performs well on industry benchmarks. 

"They were probably the first ones to reproduce o1," Snell said. "I've asked people at OpenAI what they think of it. They say it looks like the same thing, but they don't how DeepSeek did this so fast."

OpenAI and DeepSeek didn't respond to requests for comment.