It's a confusing mess to compare the alphabet soup of AI models

• There are a lot of AI models, and it can be tricky to know which are best.
• Tech companies often use "benchmarks" to measure how an AI model performs.
• But industry observers are becoming increasingly wary of benchmark reliability.

It's hard to pick the best AI to help you in work and life. What about GPT-4o, 4.5, 4.1, o1, o1-pro, o3-mini, or o3-mini-high? If not OpenAI, you can go for one of the many models put out by Meta, Google, or Anthropic.

This year has already seen at least a dozen model releases from major AI companies, and it can be confusing to decipher which really have a competitive edge. Developers of most of those releases claimed their AI had superior "benchmark" results in some way.

But that way of comparing them has faced concerns that they might not be rigorous or reliable.

Earlier this month, Meta released two new models in its Llama family that it said delivered "better results" than comparably sized models from Google and Mistral. However, Meta then faced accusations that it had gamed a benchmark.

LMArena, an AI benchmark that crowdsources user votes on model performance, said that Meta "should have made it clearer" that it had submitted a version of Llama 4 Maverick that had been "customized" to perform better for its testing format.

"Meta's interpretation of our policy did not match what we expect from model providers," LMArena said in an X post.

A Meta spokesperson told Business Insider that "'Llama-4-Maverick-03-26-Experimental' is a chat-optimized version we experimented with that also performs well on LMArena."

They added: "We have now released our open source version and will see how developers customize Llama 4 for their own use cases."

The benchmark problem

The saga speaks to wider issues the AI industry has increasingly had with benchmarks.

Companies spending billions of dollars developing AI have a lot riding on releasing models that are more powerful than the last, which cognitive scientist and AI researcher Gary Marcus says can be problematic.

"Nowadays, with a lot of money resting on performance on benchmarks, it becomes very tempting for Big Tech companies to create training data that 'teaches to the test,' and then the benchmarks tend to lose even more validity," Marcus, who has criticized areas of the AI industry he sees as overhyped, told BI.

There's also the question of whether benchmarks are measuring the right things.

In a February paper titled "Can we trust AI Benchmarks? An interdisciplinary review of current issues in AI evaluation," researchers at the European Commission's Joint Research Center concluded that major issues exist in today's approach.

The researchers said there are "systemic flaws in current benchmarking practices," which are "fundamentally shaped by cultural, commercial and competitive dynamics that often prioritize state-of-the-art performance at the expense of broader societal concerns."

Similarly, Dean Valentine, cofounder and CEO of AI security startup ZeroPath, said a March blog post that "Recent AI model progress feels mostly like bullshit."

In his post, Valentine said that he and his team had been evaluating the performance of different models claiming to have "some sort of improvement" since the release of Anthropic's 3.5 Sonnet in June 2024.

None of the new models his team tried had made a "significant difference" in his company's internal benchmarks or in developers' abilities to find new bugs, he said. They might have been "more fun to talk to," he added, but they were "not reflective of economic usefulness or generality."

As he put it, "If the industry can't figure out how to measure even the intellectual ability of models now, while they are mostly confined to chatrooms," it's hard to see how more complex AI could be accurately measured in the future.

Benchmarks can be a 'good compass'

Nathan Habib, a machine learning engineer at Hugging Face, told BI that the problem with many arena-style benchmarks is that they skew towards human preference through crowdsourced votes, which means "you can optimize your model for likability rather than capability."

"For benchmarks to truly serve the community, we need several safeguards: up-to-date data, reproducible results, neutral third-party evaluations, and protection against answer contamination," Habib said, pointing to the GAIA benchmark as an example of a tool that does this.

He added that even if benchmarks aren't perfect, "they are still good compasses of where we should go."

According to Marcus, there's no immediate fix. "Making really good tests is hard, and keeping people from gaming those tests can be even harder," he told BI.

He said that many tests try to measure "language understanding," but "it turns out that you can fake out many of these tests by memorizing a lot of stuff, without having a deep understanding of language at all."

Marcus added, "The direct risk is that customers are told that the new systems are better and spend a bunch of money on that premise."

So, how should someone go about navigating the sprawling world of AI models? How can you know what's better out of DeepSeek-R1, DeepSeek-V3, Claude 3.5 Haiku, or Claude 3.7 Sonnet?

"When it comes to selecting the right model among countless 'state-of-the-art' claims, remember that the best model isn't the one that wins every benchmark; it's the one that solves your specific problem elegantly," Clémentine Fourrier, an AI research scientist at Hugging Face, told BI.

"Don't chase the model with the highest score; chase the model that scores highest on what matters to you," she said