Issue No.02 - March-April (1997 vol.17)
Published by the IEEE Computer Society
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/38.574652
It all started when Stan Honey (vice president of technology for News Corporation) was talking with David Hill (then-president of Fox Sports, a NewsCorp company). Stan was describing an exciting, emerging technology—virtual electronic billboards—that he felt could benefit Fox Sports. Though impressed with this prospect, David had just acquired the rights to produce National Hockey League games. He knew the sport had tremendous potential, currently not realized. Aware that the viewers' most common complaint was not being able to follow the puck on TV, he said to Stan, "What I'd really like is a system that can track and highlight the hockey puck—but I'm sure that's not possible." Stan, who had worked on similar problems in an earlier career at SRI International, replied that it was possible but that David couldn't afford it. When David asked what Stan meant by unaffordable, Stan took a stab at the cost. This was his first mistake—David immediately asked him to build the system.
Stan proceeded to pull together a development team. He needed a group bright enough to ensure a reasonable chance of success, but not so bright that they would know what they were up against. He initially approached Silicon Graphics' software division with the idea that they could be contracted to develop the necessary software. After careful consideration, SGI declined. However, one of their developers did express interest in the project. Tim Heidmann, who developed SailTrack software (the system for enhancing ESPN's TV coverage of the America's Cup races), began to talk to Stan about forming a company to develop the necessary software.
Stan felt the project management and part of the development should remain within NewsCorp. So he went fishing at Etak, a company acquired by NewsCorp that he had founded 12 years earlier to produce in vehicle moving map navigators and digital map databases. Here I entered the picture. My boss at Etak suggested me as a candidate to manage and contribute to the development. After talking with Stan and thinking long and hard about it (I knew his propensity for tackling only difficult problems), I jumped aboard.
Stan also enlisted the services of Jud Heinzmann, an acquaintance from his past life at SRI, where they did advanced radar research. Jud would develop the prototype radars to track the "electronic" puck. Finally, we approached Vista Research, an outfit made up of ex-SRI people with whom Stan had worked. They would develop a system for tracking the orientation of the broadcast cameras.
After weeks of negotiation that made the Middle East peace talks look like two guys trying to decide where to have lunch, Tim left SGI to form the new venture, Shoreline Studios. Additionally, Stan enlisted all the support he could get from within Fox Sports. Most notably, Jerry Gepner, Fox Sports' V.P. of field operations, acted as our liaison and made the system possible from the production point of view.
It was now February 1995. We had 11 months to develop a system to track and highlight a frozen hockey puck traveling at times in excess of 100 mph after being walloped by angry 250-pound men with sticks. The system, to be developed in secret, would debut at the NHL All-Star game in Boston on January 20, 1996. During those 11 months our group (see Figure 1) spent a lot of time sliding around on the ice and climbing in the rafters of the San Jose Arena.
And all for what?
The idea was, if TV viewers could easily follow the puck, more people would watch—which Fox Sports wanted. We hoped to add a "blue glow" to the hockey puck, so a viewer could follow the game without straining to see the puck (which on a good slapshot could actually be torn apart by, or disappear between, television scan lines). The enhancement would show the puck's location even when it was hidden by players or sticks. An X-ray effect would appear when the puck was hidden by a known object, like a wall. We could also add a comet tail—a graphic to indicate puck speed in excess of a predetermined threshold during slapshots. A small rectangular graphic in the lower right hand corner of the screen would also be used to show a digital readout of the puck speed. Of course, Fox knew that changing the game in any way might offend long-time hockey fans. We gave a lot of thought to developing an effect that would let would-be fans follow the game more easily without driving away those already watching.
Since we could not change the game in any way, the puck would have to look, weigh, and respond exactly as always to the players, officials, and fans. Furthermore, the system would not be used to judge goals or make official calls, but strictly to improve the production.
Interestingly, people occasionally saw us experimenting at the arena and conjectured as to our purpose. Camera operators consistently assumed we were developing a system to replace them. Some folks thought we were putting a camera in the puck. Of course, the camera operator does much more than center the puck in the view—he must frame the developing play. In fact, we went to considerable effort to develop tripod heads that would encode camera motion without interfering with the operators. (Because we could find only servo-driven heads that output state, we had to develop our own.)
How would we do it?
Initially, we planned to put a radar repeater in the puck itself and four microwave radars around the arena. Each unit would send a "chirp" (a brief radio transmission in which the frequency increases rapidly) repeated by the puck. By comparing the frequency of the outgoing signal with that of the repeated signal, we could determine the time it took for the signal to reach the puck and return. Thus each unit could compute its distance to the puck. Knowing the locations of each radar unit and the puck's distance from at least three of them, we could compute the position of the puck in three-space.
Additionally, we would need to know the field of view (FOV) of the broadcast camera(s) to place the highlight on the resulting video. To this end we would mount an infrared (IR) camera on the lens of each broadcast camera and place IR beacons (infrared-emitting diodes) in known patterns around the rink. By locating the beacons in the IR cameras' FOV, we could compute the pan and tilt of the broadcast camera. Finally, we would either outfit the broadcast camera lens with encoders or take analog voltage readouts to determine its zoom and complete our knowledge of the camera's FOV.
Sounds fine—what happened?
Well, Jud built a truly impressive radar unit and repeater that would later be miniaturized to fit in the puck. We did some testing in the open and were impressed with the accuracy and robustness of this solution. So we took it to San Jose Arena and turned it on. This is when we realized that the structure of hockey arenas reflects radio waves quite efficiently. The whole building basically started to hum, and the ice attenuated the signal significantly more than expected. We found a signal-to-noise ratio (SNR) that was unacceptably low.
What did we do next? We got scared! Our only approach looked like it wouldn't work. Then Stan suggested something that couldn't possibly make sense. While we were testing the radars, Phil Evans and Alan Burns from Vista Research were testing the IR beacon camera-pointing system. Stan noted that this was working well and suggested we put IR emitters in the puck (exposed but flush with the surface) and look for it in the IR cameras. Of course, I knew this could never work; they'd never take the punishment.
But what choice did we have? We ran home, put a puck on a drill press, and stuck some IR emitters in it. Jud continued to work on improving the radar, and we investigated passive IR approaches (bouncing IR off retro-reflectors on the puck). We returned to the arena and learned that the IR emitters might actually last in a puck (and that even the coaches could take one slap at a puck and tell us it was an ounce light). Because the IR emitters would be detected by imaging sensors (which give us both azimuth and elevation) rather than ranging sensors, this approach also offered the distinct advantage that only two sensors would need line of sight to the puck to compute a 3D fix.
The radar system had been improved, but things still looked dicey. We continued to investigate and develop each approach. On our third trip to the arena, it appeared that any of our approaches could work.
Among other fixes, Jud implemented a delay in the repeater to allow the ringing in the arena to subside before sending a return signal. We could account for this "false distance" and still find the puck. Still, the repeater unit looked like it might be too big, fragile, and expensive to put in a puck (10 to 30 pucks are used in a game—electronics or not). We also began to worry about making an acceptable puck after learning that even the practice pucks (all rubber) broke the glass around the ice when hit by the pros. Our pucks would have to act exactly like NHL game pucks.
Beacons in the puck looked promising, but we knew it would be difficult to find batteries and emitters to fit in a puck, compete with thousands of watts of arena lighting, and not change the puck's physical properties. We realized that if we put retro-reflectors on the puck, we could bounce limitless IR off it because we could plug the IR "flood lights" into the wall outlets. However, this meant changing the appearance and surface of the puck by adding retro-reflective material. And because we were working against an inverse-fourth law when bouncing energy off a puck, the IR requirements would be tremendous. Finally, anything reasonably reflective (like some jogging suits) might be confused for the puck.
We considered a modified "radar" approach as well. Rather than returning a radar signal, we might put a free-running transmitter in the puck. This transmitter would chirp 30 times per second, significantly improving the SNR because it could run at full output at all times (compared with a repeater, which responds proportionally to the incoming signal). This looked promising; it would use significantly less RF (reducing FCC concerns) and let us use less expensive radar receivers in the arena.
However, each unit would report only a pseudo-range (so we could not directly compute the distance from a given unit to the puck, but for any two units we could compute their differential distances to the puck). This meant four units would need a clear line of sight to the puck at all times (and at significantly different angles from one another). This concerned us, since the puck frequently gets trapped in the corners behind five or six players. Furthermore, the problem involved in comparing the signal arrival time at the different units was daunting—similar to building a global positioning system in the arena.
Time and money were running out, so we had to choose the most promising approach without collecting all the data we'd like. We decided the active IR approach (IR emitters in the puck) looked the best. At this point a key question remained. For reasons I'll explain later, the puck would have to send out pulses, and the IR camera shutters would have to be timed to coincide with these pulses. If we used an RF signal to tell the puck when to pulse, we could avoid building a sophisticated sync detection system, and we could turn individual pucks on and off remotely (solving the problem of detecting pucks that have gone into the stands). However, this would mean putting additional electronics in the puck. Also, we were already gun shy about the RF environment in the arena. This, along with our desire to avoid needing FCC approval, convinced us to go with the free-running, IR-emitting puck.
The final architecture
We had settled on putting IR emitters in the puck and detecting them with IR cameras attached to local computers. The computers would digest the pixel data from the camera and send that data to a Silicon Graphics Indigo II Impact computer on the Puck Truck (described later). As mentioned, we would also need to know the FOV of the broadcast camera(s).
By this time we had decided it would be simpler and more robust to rig the broadcast camera tripod heads with optical shaft encoders to report pan and tilt rather than using the IR beacon approach initially planned (see Figure 2). For this purpose the local computer would also interpret the output of the pan and tilt encoders. Finally, we would need to know the broadcast camera zoom state. For this we put an analog-to-digital converter (A/D board) in the computer to read the analog voltage describing the current zoom. This data too was relayed to the Impact computer in the Puck Truck.
The Impact computer would assimilate all of the incoming data to compute the three space coordinates of the puck; use the pan, tilt, and zoom data from each broadcast camera to determine where the puck would appear in each video frame; and render the graphic highlight to be sent to a linear keyer. This keyer (controlled by the alpha-channel output of the Impact computer) would combine the broadcast camera video and the graphic effect to produce the final product.
Because the computations would require as many as five video frames (one-sixth of a second), we would have to store video frames in a frame buffer while they waited to be combined with the graphic effect. As long as we had bought into this minimal delay, we got just a bit greedy and took 10 frames of delay. This would let us place the highlight on video even if the sensors lost the puck for as many as five frames (by interpolating past puck position once the puck was reacquired).
The sync system
We still faced one major complication: As a first step in achieving the SNR that would help the puck compete with the powerful arena lights, the puck must pulse, and the IR camera shutters must be synchronized to that pulsing. By pulsing the puck in very powerful short bursts, we could overdrive the IR emitters 10-fold (with a 1/300 duty cycle). If the IR cameras "looked at" the puck only while it was turned on, they would integrate the background light only over the 1/8,000th of a second during which the puck pulsed (and not over the remaining 1/30th). This would improve our SNR by as much as 300 times and also let our tiny batteries power the puck long enough to be practical. This meant building a sync detection system.
Building the sync detection system became a primary task for Vista Research. Initially, we planned to use four "pulse detectors" located on the top of the glass at the four corners of the ice. Each would be based on a photop (photo-diode and op-amp combination) and would use a narrow-band optical filter to block all but the narrow band of IR energy emitted by the puck. The detectors would connect to a "collector box" that would report the pulse sync data to a "sync computer" in the Puck Truck. This sync computer would send the sync signal back into the arena to each IR camera, causing their shutters to open for 1/8,000th of a second (about 30 times per second, but not synchronized with the broadcast cameras).
By the first season's deployment we built (and regularly used) 20 pulse detectors to achieve adequate coverage. This ensured that the puck was always in view of at least one pulse detector within range. The rafters held 10 looking down at the ice, and 10 were mounted atop the glass around the rink. We also built three flash detectors. These detected the high-power strobe flashes that came from the rafters when the still photographers took pictures. By detecting these signals, we could "blind" the sync system briefly to avoid confusing the phase-locked-loop circuitry (thus losing sync).
This season (1996-1997) we will continue to deploy 20 pulse detectors, but all located in the rafters. Because Vista Research developed an improved sync system for this season, we can now track the puck even in very noisy environments without flash detectors. Also, the sync system can "coast" for several seconds even when the puck is hidden.
The IR cameras
Initially we planned to deploy four IR cameras, one mounted to the lens of each broadcast camera (we could then produce the effect on one to four broadcast cameras). This, we theorized, would maximize our coverage, since the broadcast cameras tend to follow the puck. Thus we could use relatively narrow FOV IR cameras (for high resolution) and still see the puck at all times. Of course, the pan and tilt encoders on the broadcast camera tripod head (see Figure 2) would also give us the orientation of the IR cameras.
Because of concerns of compounding errors (compliance in the tripod heads, flexibility of the platforms on which the broadcast cameras are placed), and because the necessary software was not ready for one of the first system tests, we originally mounted the IR cameras rigidly to the arena structure. We continued this practice through the first season; however, we used eight IR cameras (four in the rafters and four at the game cameras) instead of four as originally planned. This gave us better coverage and a more robust system. If one of eight cameras failed during a game, system performance would suffer less than if we'd lost one of four. In these types of systems more is better.
This season we will use 10 IR cameras (see Figure 3). With the cameras mounted rigidly, we obtain a resolution of two to four inches of ice to one pixel depending on the location of the IR camera and length of lens used (we use 6.5mm, 8mm, 12mm, and 16mm lenses depending on IR camera placement and coverage requirements).
Like the pulse detectors, the IR cameras use narrow-band IR filters to pass only the narrow band of IR produced by the puck. This, in addition to the pulsing behavior, increases our SNR (we fought for every last dB of SNR, frequently wondering if we'd achieve a practical system).
One of the first problems we had to solve with the IR cameras was that of lens distortion. Shoreline Studios mapped the distortion of each of our IR lenses so that the software could compute a corrected position for each lit pixel in the IR camera. We initially planned to use a temporal filtering method to boost our SNR as well. To do this, we'd take two frames of IR data every 30th of a second—one during a puck pulse, and one immediately after. The only difference in the two frames would be the puck itself; the frames would be taken so close together that the background should match in each, and when subtracted, cancel out. This would require mounting two IR cameras on each platform with subpixel pointing accuracy. This worked "in the lab" but was not practical for regular deployment. To do this properly, we would have to mount two charge-coupled devices on a single prism to assure proper and solid alignment.
Vista Research also developed the "front-end processor" boards in the PCs at each IR camera position. These boards accept the progressive scan IR video data from each IR camera and buffer it for reading by the PC-resident software. Additionally, these boards allow software control to prevent passing any pixels that fall below a specified brightness or outside of a specified X- Y window.
These parameters can be controlled locally or from the Puck Truck. They let us fine-tune the system for particular installations where the arena lighting varies, where a scoreboard might fall in the FOV of an IR sensor, and so forth. They also reduce the necessary bandwidth between each of the PCs and the Puck Truck.
Our system uses 12 PCs, 10 associated with the IR cameras (some also connected to broadcast cameras) and two associated with the collector boxes (for the sync system). The computers located with each IR camera are industrial rack-mount, 66-MHz, 486 PCs in road cases. You'd think this meant we never had problems with them—but you'd be wrong. Initially we had to open every PC at every arena and reseat the boards and connectors. Since then, we have secured these items more satisfactorily.
These PCs collect the IR data along with the pan-tilt-zoom data and send it to the Impact computer in the Puck Truck. The PC software finds groupings or "clusters" of bright pixels and sends down information describing them rather than reporting on individual pixels from the IR camera. This reduces the bandwidth requirements and distributes the computational load, so the Impact computer does not have to perform this function for each IR camera. We put watchdog timer boards in each PC so that they will automatically reboot and restart their software if they hang.
Initially we planned to return all the data from the PC using fiber optics. However, because the budget and schedule were running out, and because there was still real science to do, we opted to use good old copper for communications (RS-422 over twisted pairs). Most arenas had twisted pairs available for our use, and running our own fiber would break the bank. Though we expected no problems with the copper, we actually spent more time debugging wiring problems than any other single issue.
The tripod heads
To determine the broadcast camera's FOV, we purchased the best tripod heads available and instrumented them in both pan and tilt with optical shaft encoders. These encoders output 10,000 "true" counts per revolution (we obtain 40,000 counts per revolution by reading the phase of the two optical tracks in the encoder). This translates to a resolution of about 0.6 inches at 300 feet. Unfortunately, we don't achieve quite this level of accuracy because the platforms on which the broadcast cameras are stationed (and from which the angles are measured) aren't perfectly stable.
In addition to pan and tilt, we need the zoom or angle of view. Unfortunately, the focus adjustment produces a second-order effect in the angle of view. Additionally, the lenses we use have a built in "extender," a 2X multiplier that can be selected "in" or "out" at the touch of a button. Fortunately, the lens has a port on the side that gives us access to analog voltages indicating the zoom setting, focus setting, and extender state. We made cables that connect this port to A/D boards in the PCs. The software on the PC reads this data and sends it to the Puck Truck along with the data from the associated IR camera.
Finally, the broadcast lenses, like the IR lenses, are not perfect. They distort the video image in ways that depend on the zoom, focus, and extender positions. Shoreline Studios characterized this distortion and uses this "distortion map" to improve the highlight placement.
Developing an electronic puck that would put out the necessary IR signature, stand up to the beating, and satisfy the players and NHL was another interesting challenge. Both Fox Sports and the NHL worked with us from the start; we never could have succeeded if they hadn't. However, this didn't diminish the fact that the puck had to meet the NHL's rules, and look and perform exactly like the NHL-approved pucks made by a single vendor in Canada. This included the puck's size, weight, balance, rebound, and coefficient of friction.
What we did not realize initially is that the pucks are stored on ice for several hours before being put into play. We needed to develop a puck exhibiting the same rebound as an existing puck both frozen and at room temperature (and yes, the players can tell). Furthermore, our electronics would have to take tremendous accelerations and compression loads, and perform satisfactorily at low temperatures. The latter was of particular concern, since batteries lose both capacity and current capability at low temperatures. And we felt it was pretty important that our pucks not split apart and spew electronic components on a good slapshot.
Probably the most interesting challenge of the puck development was the search for an adhesive that would give the proper weight and flexibility—and would NOT let go. We initially considered molding our electronics into the puck, but decided to revisit this problem later due to schedule constraints (and some concerns of thermally insulating the batteries while exposed to molten rubber). Consequently, we decided to make each puck by purchasing two official pucks and machining them into two halves with cavities to accommodate our electronics, batteries, and emitters (see Figure 4).
Machining the rubber to the necessary tolerances to avoid creating seams and to meet the necessary dimensions was a challenge of its own. To bond the halves back together, we tried many types of adhesives. We then hit pucks with sledge hammers, squeezed them in vises, shot them out of cannons at 105 mph—then repeated these tests with soaked and frozen pucks. We ended up settling on a custom compound of flexible epoxy and filler.
The puck contains 20 IR emitting diodes (IREDs—they look like LEDs) configured as five strings of four IREDS with four on top, four on the bottom, and 12 around the perimeter of the puck. Each string carries about 1 amp of current during a pulse. The placement of the IREDs ensures that if one string is damaged, the lost emitters are not neighboring. The puck is turned on with an internal shock sensor and remains on for 45 seconds from the most recent shock. Thus, when the official throws the puck down, it turns on and remains on until the players stop beating it (when it goes over the glass, for example).
We did encounter our share of problems in this development. Early on, when the electronics were failing in practice games, we began to build pucks with very small wires sticking out for later analysis, and it wasn't until days before our first NHL game that we figured out how to make a puck without filling the molded texture on the side of the puck with our "super adhesive."
The Puck Truck
The Puck Truck is a 40-foot "semi," both sides of which expand out on the rear third to produce a VIP room. The front of the truck contains a wall full of monitors and four equipment racks full of computers, video, and audio equipment (see Figure 5). This truck meets the regular production truck at any game for which Fox plans to use this effect. It is wired to the arena to collect all the necessary data and to the production truck to receive the video signal and information as to which camera or replay deck is on the air (so it can produce the appropriate effect). When a replay deck is on air, the computers must access the logged data from the sensors to produce an effect to be placed over the "clean" video recorded for replay.
Operating this system requires knowing the location of each IR camera, the broadcast cameras, and the dasher boards that surround the ice. To do this, we establish a 3D coordinate system in the arena. The origin of this system is placed on the center face-off circle with the X and Y axes on the ice and the Z axis pointing straight up.
To set up this coordinate system, we drill nine small holes in the ice and fill them with freezing ice paint. We then measure the distance between each of these marks and from several of these marks to the dasher boards with a laser range finder. This establishes the dimensions of the rink and our coordinate system relative to the rink. We then place an active puck on each mark in turn while taking note of its image in each of the IR cameras. These screen-based points in the IR camera are fed into a multivariate optimizing program to compute the position and orientation of each IR camera. Similarly, we find and mark the optical axis of each broadcast camera (in 1X and 2X mode), point it at each ice mark, and solve for its location with software on the SGI computer. The whole registration process, including drilling and measuring the ice, takes about three hours, while the complete setup including wiring the arena typically takes two long days.
When the system registration is complete, we position the 2D and 3D software filters to block out spurious IR sources and to reduce bandwidth and computation requirements. The most basic 3D filter is implied. The system will not compute a 3D fix for the puck unless that fix is on or several feet above the ice. Additional 3D filters can be placed about the goal lights, commercial lights, and so on.
The first deployments
The system was finally "ready for prime time"—or so we thought. We had performed three demos for Fox Sports executives at various stages of development. Despite many long nights and a well-founded fear of failed demos while living in the rafters at the San Jose Arena, our final demo in San Jose was a genuine success.
Next we would experience fear and loathing in New Jersey: The electronic puck was scheduled for its first official NHL game (though the effect was not to be aired) when we learned that the real world is truly a cruel place. We had hoped to do arena surveys during development, but simply did not have the time. Now we found the east coast shut down by the biggest snowstorm in anyone's memory and could not get ourselves or the trucks to the Continental Arena in N.J.
When we finally arrived, we wished we hadn't. The entire development team put in a 24-hour shift culminating in a live demo that did not go well. Even during the game, we were modifying software and hardware to try and salvage the system. By the end of the event—when all we could track was a Zamboni—we realized that the IR environment at the Continental Arena (like some of the other older arenas) was truly hateful to our system.
We packed up and escaped like thieves in the night, headed for the much newer Boston Fleet Center, where the system was to debut in one week during the NHL All-Star game. While the environment was generally much better in Boston, we had not anticipated everything from laser lights to fireworks. Nevertheless, with some delicate conditioning we had the system up and running with two days of margin before the game. David Hill used this time to call a press conference where the system was demonstrated publicly for the first time. The resulting national coverage over the next three days was breathtaking. Two days after the system operated successfully at the press conference, it performed well at the All-Star game—and everyone exhaled.
Though we had our share of problems developing and deploying this puck-tracking system, it proved to be an overall success. It resulted in one patent (for the puck) and five patents pending for the system as a whole. Also, Fox Sports' NHL coverage enjoys significantly higher ratings—in large part, we think, because of the tracking system.
While we certainly had moments of doubt, the Puck Truck proved successful at every event it attended. It's true that some fans don't care for the effect, but many think it's an improvement (see Figures 6 and 7). Over the last year we've been working to improve the system so that it will track the puck more accurately. Additional improvements that may show up this season should make the system more palatable to all fans.