Issue No. 05 - September/October (1999 vol. 19)
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/38.788791
You've probably been there one way or another: Driving down a dark mountain highway or deserted suburban street, musing about the day or slightly hypnotized by the reflective lines. Even when paying close attention, you can't always see that deer or pedestrian entering your lane up ahead. Usually you get lucky and spot them in time. Sometimes you don't.
Driving after dark can be hazardous. Raytheon and General Motors hope to make it less so with Night Vision, a thermal imaging system that gives drivers a better view of the road beyond their headlights. The system, which won Popular Science magazine's Grand Award in its annual "Best of What's New" in automotive technology, will be offered as an option in the 2000 Cadillac DeVille, due out in August of this year.
The Night Vision system uses a grill-mounted infrared camera and a head-up display unit that projects a virtual image through the windshield. The system gives drivers three-to-five times the visual range of typical low-beam headlights (see Figure 1). For a motorist traveling at 60 miles per hour, this increases the low-beam reaction time of 3.5 seconds to up to 17.5 seconds in Night Vision's range. Night Vision can also serve as an alternative view for drivers temporarily blinded by the glare of oncoming car headlights. In such situations, drivers can glance down at the virtual image to check for pedestrians or animals that the approaching lights might obscure.
Although only around one-fourth of all US driving is done at night, more than half of all traffic deaths occur then, according to the National Highway Traffic Safety Administration. Night Vision can't keep drivers awake (future systems might, as the sidebar " Drivers Optional?" explains), but the new system can enhance safety by expanding drivers' views of the road ahead.
The Raytheon-GM collaboration began in 1984, when the two companies first discussed using thermal imaging to make night driving safer. Although conceptually simple, engineering was a different matter. When the project began, the system was huge, both in size and cost. According to Stuart Klapper, Raytheon System's director of automotive programs, in 1984 the thermal imaging systems were "about the size of a large projection TV." It took 15 years, $100 million, and a core GM-Raytheon team of 60 to 70 people to create a system viable for the automotive domain.
Pricing is not yet set for the unit, but Debbie Frakes, Cadillac's product publicity manager, says they expect it to cost somewhere between $1,600 and $2,500—the price range of a high-end stereo or navigation system. The cost may seem more than worth it to some drivers, particularly those who live or drive in areas with large wildlife or roaming livestock populations. According to the Michigan Deer Crash Coalition, for example, the state received more than 65,000 reports of deer-vehicle collisions in 1997 alone.
Night Vision Components
The Night Vision system uses a head-up display to project real-time thermal images through the lower part of the driver's windshield. The image, which appears to float just above the hood and below the drivers' line of sight, is created by a thermal sensor that translates infrared energy into monochromatic, real-time video images. As Figure 2 shows, the projected image resembles a photographic negative in various shades of gray, from very light, almost white images (hot objects) to very dark ones (cool objects).
The Night Vision camera is mounted behind the Cadillac's front grill and covered by a protective window. Because it rests behind the grill, washing the car keeps it clean and avoids a build up of dust; the glass is also electronically heated to melt snow and ice. Behind the window lies a set of refractive optics with a 12-degree field of view. The Night Vision sensor range falls between 8 and 14 microns. The refractive optics maximize the focal range, and a special optical glass admits infrared from 8 to 12 microns; this limits the detector range, but still keeps it within GM's target.
As Richard Seoane, a Raytheon resident engineer at the GM site explained, the system's micron range is ideal for detecting the human body, which emits maximum heat at 9.3 microns; the body temperature of animals, which is slightly higher; and other heat-emitting objects, such as automobiles. Seoane said Night Vision's micron range is also the least likely to be affected by weather. Although some weather-related performance degradation is possible—mainly in the system's ability to distinguish between things like the road and the roadside, which tend to get uniformly cold in stormy conditions—people, animals, and heat-generating objects will still stand out.
"Water, snow, and dense fog all cut down on the transmission of the IR energy, but this micron band is least affected by moisture in the air," he said. "So, if you're driving through a rain storm, then yes, your performance is going to be degraded. But you're still going to see the hot objects."
Once infrared energy is emitted, the camera's optics focus it through a chopper disk to a one-inch barium-strontium-titanate detector array, which acts as the system's brain (see Figure 3). The chopper disk, a spinning, spiral-patterned disk mounted above the detector, modulates the signal, letting energy pass through and defuse. Each of the pixels in the detector resembles an independent capacitor that creates a charge depending on IR input, creating a gray-scale image rather than one with sharp contrast. When the array senses the infrared energy, it creates an electrical pulse.
"Electronically, what we're doing is sensing the difference in that signal. It's done at video rates—a 30 Hertz frame rate—so it's just like a camcorder," said Seoane. "We do some minor digital signal processing to clean up the signal, then we turn it into a video signal and send it to the head-up display."
The Night Vision detector delivers a 320 × 240-pixel image to a head-up display. The display unit consists of a mirror and projector embedded in the dashboard just in front of the steering wheel. The unit uses a liquid crystal display, similar to a camcorder. The backlit 2.5-inch LCD image bounces off an aspheric mirror designed to match the curvature of windshield, creating a rectangular image. The windshield requires no special treatment.
The 4 × 10-inch virtual image rests just below the drivers' line of sight, appearing to "float" at the end of the hood. Because this range aligns with drivers' view of the road, they can check the image without refocusing their eyes. Objects in the virtual image match up one-to-one with their real-world counterparts, which makes it easier to map from the virtual image to the real one.
"It's pretty basic. We're taking a display and projecting it off of a mirror," said Seoane, adding that the tricky part was designing a mirror that would project an image onto the windshield without distorting it. Although he wouldn't say how they did it, they did accomplish their goal. "The image is one-to-one: If I see a man in the image and he's one-inch tall, when I look out the windshield and I see that same man, he's going to be an inch tall."
Seoane said the size of the mirror itself posed a bit of a problem, but was key to getting the one-to-one match up. Thus, although the camera displays a 12-degree field of view, they had to limit the actual display view to 11 degrees—though they're working to develop a display with the full 12-degree view.
For users, getting used to the display takes some adjustment. According to Frakes, in Cadillac tests it typically took users from 10 to 20 minutes to get accustomed to the display, after which they treated it much like a rearview mirror, checking it occasionally with a fairly seamless transition between the display and the road. "This display is not intended to be your windshield," she explained. "It's designed to be another visual input, just like your rearview mirror."
The Night Vision system started out as a military technology, gaining popular attention for its widespread and much-televised use during the Gulf War. The idea of a low-cost IR technology first became feasible in 1978, when Raytheon introduced uncooled IR imaging to the US Army's Night Vision Laboratory. Prior to that, cryogenic coolers were required, and the systems were costly both to produce and maintain.
"Military systems traditionally have been very large and very expensive," said Frakes. "Several years ago, systems like this had to be cryogenically cooled and sold for over a million dollars a piece. Obviously we had to make it smaller and less expensive."
Among the requirements to meet, Raytheon had to reduce the bulky system to one the size of a soda can. "That was their challenge, and they met that challenge," says Frakes. "And that will continue to be a challenge as we go forward—to make the system smaller and less expensive still."
In addition to reducing the size and cost, project engineers had to simplify the technology for typical drivers. "This is a military system, and the military is big into performance and features," said Seoane. "In the automotive market, quality is very important, but features are less important. We wanted to make it user friendly so that anyone who gets in the car can understand how to operate the system within a matter of minutes."
The Night Vision system powers up automatically when the headlamps go on and a photocell indicates that it's dark outside. Most other features, including contrast on the electrical signal, are also controlled automatically. Users have two control switches. One lets them turn the display off and on manually and control the image's brightness, the other adjusts the display's vertical position.
Packaging caused problems for both the sensor and the display. Seoane said that, for the display unit, they had to get involved at the start of the DeVille's development cycle to claim dashboard space that might otherwise go to air ducts and defroster grills. For the sensor, claiming space was less a problem—not much happens on the grill aside from the Cadillac crest—though the placement did increase durability requirements. Engineers had to ensure that the imaging unit would survive a nine-mph impact, as well as flying rocks, dirt, and dust. The protective window, which is made of silicon and has a diamond-like coating, proved key to meeting these requirements.
Prior to paring Night Vision costs to fit the mainstream automotive market, Raytheon fielded IR imaging technology to public safety and industrial domains. One of its first applications was in US law enforcement agencies. Today, hundreds of US and international agencies use Raytheon's NightSight thermal imaging technology for routine patrol.
NightSight units can produce real-time video pictures in total darkness and detect people up to 2,400 feet away. Rather than a head-up display, NightSight uses a monitor mounted separately from the camera. The NightSight C200 camera has a 12 × 9-degree field of view and, like Night Vision, produces 320 × 240-pixel images. Adding pan and tilt features to the same camera reduces the field of view to 12 × 6 and the image resolution to 320 × 164 pixels. In exchange, flexibility and user control increase—users can pan up to 360 degrees and adjust the scan range from 2 to 180 degrees.
NightSight also finds routine use in marine rescue operations where dark and choppy waters make locating people difficult (see Figures 4a and 4b). The system has a weatherproof housing and a defrosting heater and wiper system to minimize the effects of bad weather. Other applications of Raytheon's IR technology include handheld units used for a variety of purposes, from detecting failing components ( Figures 4c and 4d) and the liquid level in commercial storage tanks ( Figures 4e and 4f) to diagnosing equine injuries ( Figures 4g and 4h).
Raytheon's PalmIR 250 uses an uncooled ferrorelectric detector with a 320 × 240-pixel array. Like the NightSight, it can detect a person up to 2,400 feet away. The PalmIR 250 is also one of several of Raytheon's thermal imaging products used in the firefighting domain, where it can locate people and hot wires in smoke-filled buildings and find hot spots in walls and roofs.
The Road Ahead
Despite the many uses of the thermal imaging technology outside Raytheon's more traditional military domain, Seoane said the automotive application is "ideal" and that they plan to extend its capabilities within the car—though what they have planned and how they'll do it remains under wraps for now.
"Raytheon is really a military company, but we're definitely in the automotive business for this Night Vision system," he said. "As the quantity goes up and the price drops, there will be a whole new set of markets to go after."
As for Cadillac and GM in general, Frakes said that early reaction to the system has been great. "We expect to sell quite a few," she said. "Depending on how reaction continues, I think you can expect to see it on other models in the future."