# Night Vision: Infrared Takes to the Road

Keri

Pages: pp. 6-10

You've probably been there one way or another: Driving down a dark mountain highway or deserted suburban street, musing about the day or slightly hypnotized by the reflective lines. Even when paying close attention, you can't always see that deer or pedestrian entering your lane up ahead. Usually you get lucky and spot them in time. Sometimes you don't.

Driving after dark can be hazardous. Raytheon and General Motors hope to make it less so with Night Vision, a thermal imaging system that gives drivers a better view of the road beyond their headlights. The system, which won Popular Science magazine's Grand Award in its annual "Best of What's New" in automotive technology, will be offered as an option in the 2000 Cadillac DeVille, due out in August of this year.

The Night Vision system uses a grill-mounted infrared camera and a head-up display unit that projects a virtual image through the windshield. The system gives drivers three-to-five times the visual range of typical low-beam headlights (see Figure 1). For a motorist traveling at 60 miles per hour, this increases the low-beam reaction time of 3.5 seconds to up to 17.5 seconds in Night Vision's range. Night Vision can also serve as an alternative view for drivers temporarily blinded by the glare of oncoming car headlights. In such situations, drivers can glance down at the virtual image to check for pedestrians or animals that the approaching lights might obscure.

Figure 1   The Night Vision system gives drivers up to five times the visual range of low-beam headlights, increasing potential reaction time from 3.5 to 17.5 seconds.

Although only around one-fourth of all US driving is done at night, more than half of all traffic deaths occur then, according to the National Highway Traffic Safety Administration. Night Vision can't keep drivers awake (future systems might, as the sidebar " Drivers Optional?" explains), but the new system can enhance safety by expanding drivers' views of the road ahead.

The Raytheon-GM collaboration began in 1984, when the two companies first discussed using thermal imaging to make night driving safer. Although conceptually simple, engineering was a different matter. When the project began, the system was huge, both in size and cost. According to Stuart Klapper, Raytheon System's director of automotive programs, in 1984 the thermal imaging systems were "about the size of a large projection TV." It took 15 years, $100 million, and a core GM-Raytheon team of 60 to 70 people to create a system viable for the automotive domain. Pricing is not yet set for the unit, but Debbie Frakes, Cadillac's product publicity manager, says they expect it to cost somewhere between$1,600 and \$2,500—the price range of a high-end stereo or navigation system. The cost may seem more than worth it to some drivers, particularly those who live or drive in areas with large wildlife or roaming livestock populations. According to the Michigan Deer Crash Coalition, for example, the state received more than 65,000 reports of deer-vehicle collisions in 1997 alone.

## NIGHT VISION COMPONENTS

The Night Vision system uses a head-up display to project real-time thermal images through the lower part of the driver's windshield. The image, which appears to float just above the hood and below the drivers' line of sight, is created by a thermal sensor that translates infrared energy into monochromatic, real-time video images. As Figure 2 shows, the projected image resembles a photographic negative in various shades of gray, from very light, almost white images (hot objects) to very dark ones (cool objects).

Figure 2   The road ahead (a) without and (b) with the Night Vision system.

### The camera

The Night Vision camera is mounted behind the Cadillac's front grill and covered by a protective window. Because it rests behind the grill, washing the car keeps it clean and avoids a build up of dust; the glass is also electronically heated to melt snow and ice. Behind the window lies a set of refractive optics with a 12-degree field of view. The Night Vision sensor range falls between 8 and 14 microns. The refractive optics maximize the focal range, and a special optical glass admits infrared from 8 to 12 microns; this limits the detector range, but still keeps it within GM's target.

As Richard Seoane, a Raytheon resident engineer at the GM site explained, the system's micron range is ideal for detecting the human body, which emits maximum heat at 9.3 microns; the body temperature of animals, which is slightly higher; and other heat-emitting objects, such as automobiles. Seoane said Night Vision's micron range is also the least likely to be affected by weather. Although some weather-related performance degradation is possible—mainly in the system's ability to distinguish between things like the road and the roadside, which tend to get uniformly cold in stormy conditions—people, animals, and heat-generating objects will still stand out.

"Water, snow, and dense fog all cut down on the transmission of the IR energy, but this micron band is least affected by moisture in the air," he said. "So, if you're driving through a rain storm, then yes, your performance is going to be degraded. But you're still going to see the hot objects."

Once infrared energy is emitted, the camera's optics focus it through a chopper disk to a one-inch barium-strontium-titanate detector array, which acts as the system's brain (see Figure 3). The chopper disk, a spinning, spiral-patterned disk mounted above the detector, modulates the signal, letting energy pass through and defuse. Each of the pixels in the detector resembles an independent capacitor that creates a charge depending on IR input, creating a gray-scale image rather than one with sharp contrast. When the array senses the infrared energy, it creates an electrical pulse.

Figure 3   A cross-section of the one-inch, vacuum-sealed barium-strontium-titanate detector. Each BST pixel is an independent capacitor, changing based on IR input.

"Electronically, what we're doing is sensing the difference in that signal. It's done at video rates—a 30 Hertz frame rate—so it's just like a camcorder," said Seoane. "We do some minor digital signal processing to clean up the signal, then we turn it into a video signal and send it to the head-up display."

### The display

The Night Vision detector delivers a 320 × 240-pixel image to a head-up display. The display unit consists of a mirror and projector embedded in the dashboard just in front of the steering wheel. The unit uses a liquid crystal display, similar to a camcorder. The backlit 2.5-inch LCD image bounces off an aspheric mirror designed to match the curvature of windshield, creating a rectangular image. The windshield requires no special treatment.

The 4 × 10-inch virtual image rests just below the drivers' line of sight, appearing to "float" at the end of the hood. Because this range aligns with drivers' view of the road, they can check the image without refocusing their eyes. Objects in the virtual image match up one-to-one with their real-world counterparts, which makes it easier to map from the virtual image to the real one.

"It's pretty basic. We're taking a display and projecting it off of a mirror," said Seoane, adding that the tricky part was designing a mirror that would project an image onto the windshield without distorting it. Although he wouldn't say how they did it, they did accomplish their goal. "The image is one-to-one: If I see a man in the image and he's one-inch tall, when I look out the windshield and I see that same man, he's going to be an inch tall."

Seoane said the size of the mirror itself posed a bit of a problem, but was key to getting the one-to-one match up. Thus, although the camera displays a 12-degree field of view, they had to limit the actual display view to 11 degrees—though they're working to develop a display with the full 12-degree view.

For users, getting used to the display takes some adjustment. According to Frakes, in Cadillac tests it typically took users from 10 to 20 minutes to get accustomed to the display, after which they treated it much like a rearview mirror, checking it occasionally with a fairly seamless transition between the display and the road. "This display is not intended to be your windshield," she explained. "It's designed to be another visual input, just like your rearview mirror."

### Development issues

The Night Vision system started out as a military technology, gaining popular attention for its widespread and much-televised use during the Gulf War. The idea of a low-cost IR technology first became feasible in 1978, when Raytheon introduced uncooled IR imaging to the US Army's Night Vision Laboratory. Prior to that, cryogenic coolers were required, and the systems were costly both to produce and maintain.

"Military systems traditionally have been very large and very expensive," said Frakes. "Several years ago, systems like this had to be cryogenically cooled and sold for over a million dollars a piece. Obviously we had to make it smaller and less expensive."

Among the requirements to meet, Raytheon had to reduce the bulky system to one the size of a soda can. "That was their challenge, and they met that challenge," says Frakes. "And that will continue to be a challenge as we go forward—to make the system smaller and less expensive still."

In addition to reducing the size and cost, project engineers had to simplify the technology for typical drivers. "This is a military system, and the military is big into performance and features," said Seoane. "In the automotive market, quality is very important, but features are less important. We wanted to make it user friendly so that anyone who gets in the car can understand how to operate the system within a matter of minutes."

The Night Vision system powers up automatically when the headlamps go on and a photocell indicates that it's dark outside. Most other features, including contrast on the electrical signal, are also controlled automatically. Users have two control switches. One lets them turn the display off and on manually and control the image's brightness, the other adjusts the display's vertical position.

Packaging caused problems for both the sensor and the display. Seoane said that, for the display unit, they had to get involved at the start of the DeVille's development cycle to claim dashboard space that might otherwise go to air ducts and defroster grills. For the sensor, claiming space was less a problem—not much happens on the grill aside from the Cadillac crest—though the placement did increase durability requirements. Engineers had to ensure that the imaging unit would survive a nine-mph impact, as well as flying rocks, dirt, and dust. The protective window, which is made of silicon and has a diamond-like coating, proved key to meeting these requirements.

Prior to paring Night Vision costs to fit the mainstream automotive market, Raytheon fielded IR imaging technology to public safety and industrial domains. One of its first applications was in US law enforcement agencies. Today, hundreds of US and international agencies use Raytheon's NightSight thermal imaging technology for routine patrol.

NightSight units can produce real-time video pictures in total darkness and detect people up to 2,400 feet away. Rather than a head-up display, NightSight uses a monitor mounted separately from the camera. The NightSight C200 camera has a 12 × 9-degree field of view and, like Night Vision, produces 320 × 240-pixel images. Adding pan and tilt features to the same camera reduces the field of view to 12 × 6 and the image resolution to 320 × 164 pixels. In exchange, flexibility and user control increase—users can pan up to 360 degrees and adjust the scan range from 2 to 180 degrees.

NightSight also finds routine use in marine rescue operations where dark and choppy waters make locating people difficult (see Figures 4a and 4b). The system has a weatherproof housing and a defrosting heater and wiper system to minimize the effects of bad weather. Other applications of Raytheon's IR technology include handheld units used for a variety of purposes, from detecting failing components ( Figures 4c and 4d) and the liquid level in commercial storage tanks ( Figures 4e and 4f) to diagnosing equine injuries ( Figures 4g and 4h).

Figure 4   Raytheon's commercial thermal-imaging technology is implemented in hand-held and mounted cameras used in many domains, including manufacturing, law enforcement, and marine rescue. A view of the water at night (a) without thermal imaging; with the technology (b) two stranded swimmers are clearly visible. A thermal image can determine which of several circuits (c) is likely to go bad next (d). As (e) and (f) show, thermal imaging can also reveal the amount of fluid in an opaque tank. The horse looks fine (g), but doesn't want to run. A thermal image (h) shows irregularities in both its front hooves.

Raytheon's PalmIR 250 uses an uncooled ferrorelectric detector with a 320 × 240-pixel array. Like the NightSight, it can detect a person up to 2,400 feet away. The PalmIR 250 is also one of several of Raytheon's thermal imaging products used in the firefighting domain, where it can locate people and hot wires in smoke-filled buildings and find hot spots in walls and roofs.

Despite the many uses of the thermal imaging technology outside Raytheon's more traditional military domain, Seoane said the automotive application is "ideal" and that they plan to extend its capabilities within the car—though what they have planned and how they'll do it remains under wraps for now.

"Raytheon is really a military company, but we're definitely in the automotive business for this Night Vision system," he said. "As the quantity goes up and the price drops, there will be a whole new set of markets to go after."

As for Cadillac and GM in general, Frakes said that early reaction to the system has been great. "We expect to sell quite a few," she said. "Depending on how reaction continues, I think you can expect to see it on other models in the future."

## Drivers Optional? Looking Down the Road

#### IQ INCREASE

Automotive safety technologies and navigational aids have gotten smarter. For example, Mercedes-Benz will begin offering Distronic, an advanced cruise-control system, in its new S-class cars. Distronic regulates the distance between cars using a radar sensor (see Figure A). The sensor, located in the grill, scans up to 150 meters of the road ahead. A display located in the dashboard control panel gives drivers an electronic image of the distance. If the space between the driver's car and the car ahead gets too narrow, Distronic automatically eases up on the throttle and, if necessary, activates the brake. If the situation becomes too hazardous to entrust braking to the computer, a signal warns the driver.

Figure A   Distronic, a new cruise-control system for Mercedes-Benz S-class cars, uses a radar sensor to detect the distance between cars. If the vehicle gets too close, Distronic eases up on the throttle, applies brakes, or sounds an alarm, depending on the situation.

Most car manufactures are also taking a second look at the airbag. Although automobile airbags have saved thousands of lives, they've claimed a few as well, in some cases because of overpowering inflation rates. Up until 1998, manufacturers engineered airbags to protect an unbelted adult male in a 30-mph head-on crash. Modified test standards now permit airbags to deploy with up to 35 percent less force—good news for lighter and slighter passengers. Future airbags will check passengers' weight and location, seatbelt use, and the vehicle's speed before deploying.

Navigation systems have also advanced. Dashboard map displays are being upstaged by a whole new class of system, linking drivers with 24-hour cell-phone assistance that serves as a combination help line and event planner. With GM's Onstar system users can order concert tickets, get hotel reservations and directions, and have their car doors opened if they lock their keys inside. If an Onstar owner is in an accident severe enough to trigger an airbag, the system automatically summons emergency help.

#### AN EVOLVING VISION

Stephen A. Benton, Allen Professor of Media Arts and Sciences and head of MIT's CC++ Car Research Group, said the two most important research areas for the future are the human-car interface and car connectivity.

"These are both highly speculative areas, which is why the Media Lab is interested in them," he said. He added that it's also why lab sponsors are happy—this type of long-term, high-risk research is not something most companies want to pursue.

In the auto-interface area, the CC++ group wants to bring two decades of study of human-computer interface issues to bear in the automotive domain. Their goal is to model the user for cars, giving them "a sense of who the drivers are as individuals," explained Benton, as well as how they are behaving in relation to expectations. For example, cars might one day detect road rage, fatigue, information overload, or illness by determining the users' emotional state using sensors and pattern-recognition technology.

In the near term, the car will likely continue to evolve as a driver's personal assistant and information source, delivering news, checking e-mail and voice mail, and offering directions to the nearest seafood restaurant. Benton said he also anticipates car "personality modules" becoming transportable, so drivers can migrate to new cars without having to reprogram their quirks and preferences each time.

The CC++ group is also looking for ways to take advantage of car connectedness that deviate from the obvious. Benton said cars will soon communicate with each other as intelligent highways and systems like Onstar evolve, eventually making a transient network possible.

Such advances in wireless technology could enable message passing along the highway, he predicted, alerting people of obstacles and accidents, enabling e-commerce ("Have my groceries and FedEx meet me at my gas station"), and letting cars compare notes. When one car attempts to pass another, for example, it might one day get a message like, "Better wait; my driver is in a bad mood and driving erratically."

#### GOVERNMENT AND RESEARCH INSTITUTIONS

The US Department of Transportation's Intelligent Transportation System Web site, including information on the safety-centered Intelligent Vehicle Initiative: http://www.its.dot.gov/.Information on the European Commission's Telematics for Transport program, including details on current projects and future plans: http://www.echo.lu/telematics/transp/transport.html.Information about air bags, research projects, accident statistics, and more at the National Highway Traffic Safety Administration Web site: http://www.nhtsa.dot.gov/.The US Transportation Research Board Web site, including a report on in-vehicle computing: http://www.nas.edu/trb/index.html.The IEEE Intelligent Transportation Systems Council home page: http://www.ewh.ieee.org/tc/its/.The Japanese Intelligent Transportation Institute Web site, including extensive links to intelligent transportation system projects in Asia, Europe, and the US: http://www.nihon.net/ITS/index.html.The University of Southhampton's Transportation Research Group Web site, with research and resources, including a list of UK transportation links: http://www.soton.ac.uk/~trgwww/.The Massachusetts Institute of Technology's Intelligent Transportation System Web site: http://hippo.mit.edu/; the MIT Media Lab's CC+ Car Consortium Web site is at http://www.media.mit.edu/cc++/.The Virginia Tech Center for Transportation Research site, including research on safety, human factors, and technology deployment: http://www.ctr.vt.edu/.The University of Michigan's Transportation Research Institute, with automotive research in several areas, including safety, bioengineering, human factors and psychology, and mechanical engineering: http://www.umtri.umich.edu/.California's Partners for Advanced Traffic and Highways (PATH), a multidisciplinary project administered by the University of California at Berkeley's Institute of Transportation Studies: http://www.path.berkeley.edu/.

#### COMMERCIAL SITES

The Hansen Report, a subscription-based newsletter on automotive electronics with sample issues available through their Web site: http://www.hansenreport.com/home.cfm.An overview of Raytheon's commercial IR technologies, applications, and products: http://www.raytheon.com/rsc/ses/spr/spr_cir/spr_cir.htm.Information about General Motors' innovations and advances, including Onstar: http://www.gm.com/vehicles/us/programs.html.Highlights of Mercedes-Benz's innovations and future plans: http://www.mercedes.com/e/innovation/fmobil/default.htm; an overview of the Distronic system and other S-class electronic advances: http://www.mercedes-benz.com/e/cars/s-class/facts_e.htm.Outlines of BMW's plans for navigation and automobile-assistant technologies: http://www.bmw.com/bmwe/pulse/aid/index.shtml.Ford Motor Company, including new and planned safety and security advances: http://www2.ford.com/default.asp?pageid=102.Details on Mitsubishi's vision for the car of the future: http://www.mitsucars.com/corp/abt/index.php3?corp=abt&subnav=conc.