AI models are choking on junk data

How we get from ChatGPT to humanoid robots relies on one of the most consequential, but least discussed bottlenecks in artificial intelligence – the quality of the data that we feed these systems to learn from. 

Thus far, the AI industrial complex has operated on the idea that feeding models more data means smarter models. This worked brilliantly when researchers could simply vacuum up the internet to train large language models. But we’re on the cusp of the next frontier of AI — physical AI and world models – systems that will learn and ultimately operate in the physical world. Think about the cognition it takes to navigate roads and traffic, fold laundry, or assist in complicated medical surgeries. These all require something that can’t simply be downloaded. It requires rich and multifaceted data from which these world models can learn. 

There’s now a potential crisis in motion that could have major implications on the AI movement. If we aren’t able to stem the excess of junk data – data that isn’t able to move a model forward in development –  the entire promise of physical AI and world models may never achieve its full potential.  

A big part of the problem is the hunger for data to feed new and better models. AI companies are ravenous for that data, which has spawned a wave of multi-billion dollar AI data startups that provide these services like Scale AI, Surge AI, and Mercor. But catering to those insatiable appetites has produced a bounty of junk data that actually don’t advance AI models at all. 

Junk data is easier to produce, but the data needed for physical AI and world models requires much more time and effort. Because the physical world is very complex, training these models to understand the multi-dimensional world requires significantly more data — data that is also very hard to get. Machine learning engineers resort to simulating this data, and that requires hours upon hours of virtual reenactments of real world-scenarios to create the data that will ultimately train robots and self-driving cars. When AI models use junk data, it degrades performance, drags out the time to market, and could lead to unpredictable outcomes. 

For instance, to be considered safe, a fully autonomous car would require a system able to deal with all the unforeseen variables that people may encounter when driving, like a car driving on the wrong side of the road or high glare making it hard to detect a child about to run into the street. Junk data only makes it harder for such autonomous systems to learn what is typical from what is possible.

We’re already seeing the junk data problem rear its ugly head. OpenAI sunset its AI video app Sora while reassigning the team to other divisions. This at its core was a junk data problem because their world model lacked sufficient understanding of physics leading to realistic prediction. 

To achieve the real potential of AI capabilities, machine learning teams need the tooling and processes to cut junk data from their workflows. They must invest in technologies that analyze, clean, normalize, and correct training data. Distilling valuable insights and distinguishing them from the junk is how we train our AI models with the right information for success. 

The scaling hypothesis that feeding AI systems ever-larger quantities of data will produce ever-smarter systems turned out to be right, until it wasn’t. Quality data is now the constraint. The companies and research labs that recognize this first will build the AI systems that actually work in the world.

The opinions expressed in Fortune.com commentary pieces are solely the views of their authors and do not necessarily reflect the opinions and beliefs of Fortune.

This story was originally featured on Fortune.com