Inside Niantic Spatial’s audacious plan to scan the world

The robots are coming—and they’ll need a good map to get around, whether they’re delivering our packages, driving us around, or doing any of the myriad of other tasks robotic helpers may perform for us in the future.

“Without spatial intelligence, your robot really can’t do what you want it to do,” says John Hanke, the former CEO of Niantic Spatial who transitioned into the role of the company’s executive chairman a week ago. “Your Roomba is operating just in your house. But as we’re about to start operating at city scale, we need these big, large-scale, shared, accurate representations of the world.”

This week, Niantic is taking the next step towards building that map: The company is launching its new Scaniverse platform, which aims to give companies and individuals the tools to capture the world with phones, 360-degree cameras and drones, and then fuse that data into a massive 3D map that allows robots to navigate the world around us with centimeter-level accuracy.

It’s a very different approach from the way companies like Google have traditionally approached the problem, as Hanke knows firsthand from having led Google’s mapping efforts two decades ago. “The problem with the Google approach, having lived it, is: You send all the cars out, capture all the streets,” Hanke says. “Then a new road gets built. Everything changes. You’re always behind. Your data is always stale. It’s always old. And it’s very capital-intensive.”

And while an outdated map may be annoying to people, it’s all but worthless to a robot. “You want it to be a living map, not some static reconstruction,” Hanke says.

An app for enthusiasts becomes an enterprise platform

Niantic came up with its bottoms-up, distributed approach to mapping when it was working on the popular mobile game Pokémon Go, and needed to turn real-life landmarks into virtual battlegrounds for its millions of players. When the company sold Pokémon Go a year ago and rebranded as Niantic Spatial, it doubled down on its 3D mapping efforts, striking deals with robotics companies and others to take advantage of its geospatial data. 

The company’s new Scaniverse platform is based on an app for 3D enthusiasts the company acquired five years ago.

In its first iterations, the Scaniverse app primarily focused on letting people capture smaller objects in 3D. To do so, Scaniverse users would slowly move their phone camera around such an object, and then the app would turn the captured footage into a three-dimensional representation of it. Since the acquisition, Niantic expanded Scaniverse to also allow users to scan entire rooms—all while still processing all of the data locally on a user’s phone.

Now, Niantic is taking the next step with the launch of cloud-based video processing and a dedicated Scaniverse website. Enterprise customers can use that website to upload footage from a variety of devices, including not just phones but also 360-degree cameras and drones, and then have that footage turned into a 3D model.

Adding 360-degree camera support has been a game changer for the company. “Your camera sees what’s in front of you, what’s behind you, above you and below you,” says Niantic Spatial Senior Product Manager Azad Balabanian. “This is extremely important for being able to achieve very immersive-looking [scans].”

Last month, Niantic let me scan a large square in front of San Francisco’s Ferry Building, where the company is located. Capturing this area was extremely easy: I simply carried a 360-degree camera on a selfie stick while casually walking a big loop—a process that took maybe five minutes. Once the footage got uploaded to Scaniverse, the platform automatically removed any trace of me, and then rendered the entire square in 3D:

To turn the captured video footage into a 3D model, Scaniverse uses a technology known as Gaussian splatting. In essence, Gaussian splatting turns 2D images into a huge number of three-dimensional blobs that offer varying degrees of transparency, and can be viewed from all angles. Gaussian splatting has become one of the most promising ways to capture spaces in 3D. It is also being used by Meta to capture real-life rooms for virtual worlds.

Niantic released its own VR viewing app for Scaniverse scans last year. Now, the company is focused on not only making its scans look good to humans, but also make sense to robots and AI algorithms. “Gaussian splatting can be very tricky,” Balabanian explains. “Things [may be] looking right, but not actually have real surfaces.” 

Oversimplified, a wall that’s painted white can look nearly indistinguishable from a wall of fog to a 3D scanning app. That’s why Niantic researchers developed a depth estimation model for the Scaniverse cloud processing platform that accurately captures hard surfaces a robot may crash into. “We can achieve lidar-level-quality depth without needing Lidar,” Balabanian says, referencing the laser sensors used by self-driving cars to safely navigate streets.

3D for construction companies, Hollywood and AI

Some of the sectors that Niantic is targeting with its new Scaniverse platform include construction, logistics and utilities, all industries that handle large, complex sites. Having 3D models of these sites can help with planning and maintenance, but also enable the companies to eventually have robots navigate their warehouses and power plants.

Building and training robots, and the AI models used to make them work, also requires a bunch of 3D data. “Robots need a lot of training data, and [companies] want to simulate things in training environments before they deploy them,” Balabanian says. “If you want to make sure your robot works really well in a real-life place, the best way you’re going to do that is to map the place yourself.”

Even Hollywood can benefit from technologies like Gaussian splatting, and for instance scan film locations to plan shoots, and generate the kind of raw concept footage known among studio insiders as pre-visualization.

But while Niantic Spatial is now expanding its offerings to the enterprise, it hasn’t forgotten about the hobbyists and enthusiasts that turned Scaniverse into an early 3D scanning success story. For now, the company is offering separate consumer and enterprise accounts for Scaniverse. Eventually, it wants to merge those two versions of the platform, and give enthusiasts and prosumers access to cloud-based processing as well.

It’s a nod to Niantic’s roots in the consumer space—the same roots that led the company to bet on simple phone-based apps over an expensive fleet of camera-equipped cars to scan the world. “We developed quite a lot with very little, initially,” says Balabanian. “All the things [we] developed for consumer applications are now stepping stones for the enterprise.”