After a grueling twenty minutes of play and an eight-point gap, it was halftime for the Olympic basketball game between the U.S. and France on July 25. Spectators probably expected commercials or a plain break – but what they got instead was a seven-foot-tall Toyota robot in a Tokyo 2020 jersey that shot perfect baskets.   

First, it threw a three-point shot that swished right through the net. Then it rolled backward on its wheels to mid-court and shot another basket, complete with a flicked wrist.  

This robot’s name is CUE – and it serves as a cue for what’s coming next in Toyota robotics.

CUE won a Guinness World Record for shooting 2,020 consecutive baskets in 2019.

Most of us couldn’t use an enormous free-throwing robot. But what we could use are robots that help around the home, whether that means dusting, washing dishes, or even cooking. Those robots would require precise movements like those seen in CUE, but they also need something else that’s hard to achieve – advanced situational awareness. 

Thankfully, the Toyota Research Institute (TRI) is working on it.

 

How do you teach a robot how the world works?  
Researchers have already trained robots to recognize their surroundings through cameras and sensors. But what’s harder to recognize are complex images like reflections, which robots often recognize as three-dimensional objects themselves. This limitation makes at-home robots especially difficult to develop, as houses and apartments are typically filled with reflective surfaces like glass doors or windows, refrigerators, mirrors, and more.  

That’s why TRI is targeting this problem – and so far, it’s been a resounding success, as seen in a new TRI video. The new training method uses programmable data, which runs the computer through 3D simulations where it learns from its mistakes virtually instead of pulling from real-world data.

In the past, a team of humans had to individually label object data and program each possible scenario into the robot, which was a long and expensive process. With programmable data, however, the computer simulates a 3D world based on reality and learns ‘in its head’ instead of always relying on real-world, physical trial and error. It essentially plays through its own video game to decide what to do next. This solution also means the programmers can tweak the 3D world with simple data patches to provide new learning courses for the robots without changing anything in real life.  
This is what one of Toyota’s helper robots “sees” when it generates a 3D learning world based on reality.

The researchers used these simulated situations to test stereo sight, which uses two different images to determine an object's depth and three-dimensional structure. In short, it works a lot like human eyes do – and this approach combined with the new training method proved to change everything.  

What can robots do with programmable data training? 
TRI robots are no longer trying to grasp at reflections in the toaster oven. Instead, scientists have taught them to recognize and respond to transparent or reflective surfaces differently depending on the situation using programmable data training.  

As shown in TRI’s recent YouTube video, the robots have learned how to recognize a wipeable surface and move objects out of the way, even sensitive items like glass cups. They can clean floors, too, no matter how reflective the tile or wood is. 
This robot is trained to handle objects with care, whether it can or cannot identify what the object is.

The possible applications of this technology are endless. It could help elderly or disabled individuals with difficult household tasks. Busy people and parents could also benefit from having an extra pair of "hands" around the house.  

This isn’t Toyota’s first exploration into human-service robotics. Toyota designed robots to serve the Tokyo 2020 Olympics attendees, whether that be the scooter-like battery electric vehicles or the human support robot that delivers food objects ordered from a menu. Given Tokyo’s COVID-19 restrictions, most Olympic stands were empty in 2021 — but the technology remained.  

Robots aren’t ready to walk your dog just yet. But given enough time, Toyota’s research may just help change the way homes look and work in the future.