Driverless cars could get AI-powered heat vision for nighttime driving

Driverless cars can struggle to distinguish between a pedestrian and a cardboard cutout of a person when it is dark or particularly rainy. A system that uses AI to identify objects based on their heat emission patterns could help autonomous vehicles to operate more safely in all outdoor conditions.

Zubin Jacob at Purdue University in Indiana and his colleagues developed a heat-assisted detection and ranging (HADAR) system by training an AI to determine the temperature, energy signature and physical texture of such objects for each pixel in the thermal images.

To train the AI, the researchers captured data outdoors at night using sophisticated thermal-imaging cameras and imaging sensors capable of showing energy emissions across the electromagnetic spectrum. They also created a computer simulation of outdoor environments to allow for additional AI training.

HADAR learned to detect objects and estimate the distance from those objects with 10 times greater accuracy than relying only upon traditional night-vision technologies, says Jacob. The nighttime performance is also equivalent to daytime performance for traditional object-detection systems.

This proof-of-concept demonstration for HADAR is still years away from becoming viable on self-driving vehicles. The bulky and expensive camera and imaging equipment still needs to be manufactured in smaller form and at a much lower cost – the HADAR demonstration tested both a $10,000 thermal-imaging camera and a military-grade hyperspectral imager costing more than $1 million.

Another challenge is that the process of collecting and processing the data still takes about a minute, whereas that time would ideally be within milliseconds so that a driverless car could make use of such a system on the go.

The accuracy and reliability of such a system still need to be proven in a broad variety of different environments, says Miroslav Pajic at Duke University in North Carolina. But he described the HADAR concept as a potentially promising new capability to complement the existing cameras and sensors on self-driving cars.

“Having a new way of reasoning about the environment, especially in situations where cameras do not perform well when it’s dark, is definitely a plus,” says Pajic.

The technology could prove more immediately useful in helping to monitor wildlife at night or in future biomedical applications. “I believe in the next five to seven years, we’re going to see a lot of breakthroughs on the thermal front,” says Jacob.