When you might start speaking to robots

Last Wednesday, Google made a somewhat surprising announcement. It launched a version of its AI model, Gemini, that can do things not just in the digital realm of chatbots and internet search but out here in the physical world, via robots. 

Gemini Robotics fuses the power of large language models with spatial reasoning, allowing you to tell a robotic arm to do something like “put the grapes in the clear glass bowl.” These commands get filtered by the LLM, which identifies intentions from what you’re saying and then breaks them down into commands that the robot can carry out. For more details about how it all works, read the full story from my colleague Scott Mulligan.

You might be wondering if this means your home or workplace might one day be filled with robots you can bark orders at. More on that soon. 

But first, where did this come from? Google has not made big waves in the world of robotics so far. Alphabet acquired some robotics startups over the past decade, but in 2023 it shut down a unit working on robots to solve practical tasks like cleaning up trash. 

Despite that, the company’s move to bring AI into the physical world via robots is following the exact precedent set by other companies in the past two years (something that, I must humbly point out, MIT Technology Review has long seen coming). 

In short, two trends are converging from opposite directions: Robotics companies are increasingly leveraging AI, and AI giants are now building robots. OpenAI, for example, which shuttered its robotics team in 2021, started a new effort to build humanoid robots this year. In October, the chip giant Nvidia declared the next wave of artificial intelligence to be “physical AI.”

There are lots of ways to incorporate AI into robots, starting with improving how they are trained to do tasks. But using large language models to give instructions, as Google has done, is particularly interesting. 

It’s not the first. The robotics startup Figure went viral a year ago for a video in which humans gave instructions to a humanoid on how to put dishes away. Around the same time, a startup spun off from OpenAI, called Covariant, built something similar for robotic arms in warehouses. I saw a demo where you could give the robot instructions via images, text, or video to do things like “move the tennis balls from this bin to that one.” Covariant was acquired by Amazon just five months later. 

When you see such demos, you can’t help but wonder: When are these robots going to come to our workplaces? What about our homes?

If Figure’s plans offer a clue, the answer to the first question is soon. The company announced on Saturday that it is building a high-volume manufacturing facility set to manufacture 12,000 humanoid robots per year. But training and testing robots, especially to ensure they’re safe in places where they work near humans, still takes a long time. 

For example, Figure’s rival Agility Robotics claims it’s the only company in the US with paying customers for its humanoids. But industry safety standards for humanoids working alongside people aren’t fully formed yet, so the company’s robots have to work in separate areas.

This is why, despite recent progress, our homes will be the last frontier. Compared with factory floors, our homes are chaotic and unpredictable. Everyone’s crammed into relatively close quarters. Even impressive AI models like Gemini Robotics will still need to go through lots of tests both in the real world and in simulation, just like self-driving cars. This testing might happen in warehouses, hotels, and hospitals, where the robots may still receive help from remote human operators. It will take a long time before they’re given the privilege of putting away our dishes.  

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.