The next generation of driverless cars will have to think about what’s on the road, not just see it

Autonomous vehicles have made remarkable progress over the past decade. Driverless cars and buses that once struggled to stay in lane can now navigate busy city streets, recognise pedestrians and cyclists, and respond smoothly to traffic signals.

Yet one challenge remains stubbornly difficult. The hardest situations on the road are not the common ones but the rare and unpredictable events – what AI researchers call “long-tail scenarios” or “edge cases”, because they occur as outliers on any event distribution curve.

Examples include unexpected roadworks, unusual behaviour from other road users, and other subtle situations where there is a very low probability of something happening – but which would have a significant impact on the vehicle and journey.

Addressing these issues needs more than just better sensors – it requires vehicles that can reason about uncertainty. The most promising class of AIs yet developed to do this are known as “vision-language-action” (VLA) models. These take visual inputs from sensors, form an internal reasoning process often described as “thinking in steps”, then (almost instantaneously) generate actions such as steering or braking.

VLA models are not new. In robotics research, they have been developed for years as a way of connecting perception, symbolic reasoning and physical behaviour. For example, my research group at the University of Leicester has been examining how robots can reason about ambiguous physical situations, rather than simply react to sensor inputs.

But the recent unveiling of an open-source platform of VLA models by Nvidia, the world’s leading AI chip-manufacturer and most valuable company, has brought global attention to whether this is the technological leap needed to make autonomous vehicles both safe enough and cheap enough to make them a common sight on all our roads.

What’s most notable about Nvidia’s VLA platform – called Alpamayo and launched by the company’s CEO, Jensen Huang, at the Las Vegas Consumer Electronics Show (CES) on January 5 – is the scale and levels of investment it brings: industrial-level data, simulation and computing applied directly to the complex and safety-critical task of driving.

Huang confirmed that German car manufacturer Mercedes will use Alpamayo technology in its new CLA models – but this does not mean these cars will be fully autonomous at launch. There again, I believe this technology is an important step towards a mobility future dominated by autonomous vehicles.

Why long-tail scenarios are so hard for AI

In machine learning, systems are typically trained on large volumes of representative data. For driving, this means countless examples of clear roads, standard junctions and predictable traffic flows. Autonomous vehicles perform well in such conditions because they closely resemble what the system has already seen.

The difficulty lies at the edges of this data. Long-tail scenarios occur infrequently but account for a disproportionate share of risk. A pedestrian stepping into the road from behind a parked van, a temporary lane closure that contradicts road markings, or an emergency vehicle approaching from an unexpected direction are all situations that demand judgement rather than rote response.

Human drivers handle these moments by reasoning. We slow down when something might happen, anticipate uncertainty and err on the side of caution. Most autonomous systems, by contrast, are built to react to recognised patterns. When these patterns break down, so can the system’s confidence.

How Alpamayo works

Alpamayo is neither a self-driving car nor a single AI model. It is an open-source ecosystem designed to support the development of reasoning-based autonomous systems. It combines three main elements: a large, open-source AI model (developed by Nvidia) that links perception, reasoning and vehicle actions; extensive real-world driving datasets from different countries and environments; and simulation tools for testing decisions in complex scenarios.

Alpamayo’s models are designed to produce “intermediate reasoning traces”: internal steps that reflect how a decision was reached. In practical terms, this means a system can explain (and learn from) why it chose to slow down, wait or change course in response to uncertainty.

In contrast, traditional autonomous driving software is usually organised as a pipeline. One system detects objects, another predicts their motion, and a third plans how the vehicle should respond. This structure is efficient and well understood, but can struggle when situations fall outside its predefined assumptions - particularly when multiple plausible outcomes must be considered, rather than a single predicted one.

The power of reasoning that Alpamayo is instilled with should be better able to deal with the unexpected. A system trained to think about what could happen, rather than what usually happens, has a better chance of coping with long-tail scenarios that fall outside its training data. It also makes the system more transparent, allowing engineers and regulators to inspect decisions rather than treating them as outputs from a black box.

However, despite the excitement around the recent Nvidia presentation, Alpamayo is not being presented as a finished self-driving solution. Large reasoning models are computationally demanding and unlikely to run directly in vehicles. Instead, they are intended as research tools: systems that can be trained, tested and refined offline, with their insights later distilled into smaller onboard computers in autonomous vehicles.

Seen this way, Alpamayo represents a shift in how autonomy is developed. Rather than hand-coding ever more rules for rare cases, the aim is to train systems that can reason their way through uncertainty.

This is just part of a wider trend in AI-centric approaches to autonomy. In the UK, autonomous vehicle tech company Wayve has attracted attention for its work on embodied AI. This is where a single learning system learns driving behaviour directly from experience, without relying heavily on detailed maps or hand-engineered rules.

While Wayve’s approach does not emphasise explicit reasoning traces in the same way as Alpamayo, both reflect a move away from rigid pipelines toward systems that can adapt more flexibly to new environments. Each, in its own way, is aimed at improving how autonomous vehicles cope with the long tail of real-world driving.

Daniel Zhou Hao does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.