In the wake of hurricanes Katrina and Rita in the southeastern United States, the computational capabilities of weather data networks are receiving new attention. Market experts say this industry segment is ripe for meteoric growth, whether developing better models for forecasting cataclysmic events—models that can better calculate risk for actuaryal and development purposes—or constructing a modern weather sensor network that dynamically feeds raw data into large databases that enable those models to work.

"I think businesses are starting to realize more and more the advantage of using accurate weather information for strategic business decisions," says Ken Reeves, senior meteorologist and director of forecasting operations at AccuWeather (http://www.accuweather.com), the world's biggest commercial weather service with 15,000 clients. "That part of the field of meteorology is in its infancy, but it's on the verge of explosion."

Weather's economic impact can be immense. Risk Management Solutions, a risk modeling firm, estimated Hurricane Katrina's total cost at US$125 billion. But the losses from noncatastrophic weather phenomena can also be significant. For example, executives from just one utility, Duke Power, estimate the company can save several million dollars annually by using precipitation forecast information to regulate the amount of water going through hydro plant spillways and so prevent "wasted" water—that is, water moved past hydroelectric generating facilities without being used.

The benefits of more accurate weather models and more timely forecasting are universally acknowledged. Still, the infrastructure necessary to deliver them—from raw data to granular modeling algorithms—is lagging. Furthermore, no lingua franca exists among researchers attempting to enlarge and integrate regional climatological and oceanographic data networks, nor with commercial weather services devising new forecasting technologies and markets. So, in a worst-case scenario, the current industry boom could produce a balkanized landscape of incompatible weather data formats and an awkward combination of open and proprietary data.

The National Weather Service (http://www.nws.noaa.gov), according to its own mission statement, says its data and products form "a national information database and infrastructure which can be used by other governmental agencies, the private sector, the public, and the global community." However, private sector meteorologists as well as an NWS document calling for modernization both contend the NWS isn't fulfilling its role as a fully capable data provider.

"Data is a core mission element of the National Weather Service," says AccuWeather's Reeves. "The general infrastructure that goes to support the distribution of weather data has been not well maintained, and there's a variety of reasons we could look at. Suffice to say the NWS has not done a good job keeping up with the volume of data, the quality of data, and the distributing capabilities now out there as well."

A new pool of weather data has come into existence as modern meteorological instruments become more capable and ubiquitous. Furthermore, these instruments are being incorporated into regional networks. Many television stations deploy observing stations at schools, for instance, providing both a regional weather network for viewers and educational opportunities for host schools.

"We have more people at more locations creating more measurements, and they want to distribute that out," says Steven Smith, AccuWeather's director of systems engineering. "So not only are you talking in the broad sense of distributed computing, where modelers are trying to improve forecasts using the capabilities of distributed computing, we're also trying to figure out how do we handle not megabytes or gigabytes of data, but terabytes of data. And when you're going to 24 different sources, who have potentially completely different operating procedures, that creates a headache."

The self-described backbone of US weather data collection is the NWS Cooperative Observer Program (http://www.nws.noaa.gov/om/coop/). Instituted in 1890, COOP consists of 11,400 volunteer weather observers located at nonairport locations throughout the US. This network, plus about 1,000 Automated Surface Observing System airport stations, forms the federal government's surface weather and climate observing network for the US. Its supply of raw data supports applications in water resource management, building design and maintenance, crop yield predictions, economic decision making, flood and drought monitoring and forecasting, and climate variability studies. Researchers also use COOP data in building forecasting models. However, the program's often obsolete instrumentation and nonautomated data reporting procedures result in substandard data quality and no real-time access.

The latest plan to modernize COOP (http://www.nws.noaa.gov/om/coop/reference/PDP4COOP.pdf), released in 2004, concedes this: "Processing of the data is labor intensive and does not occur in real time. Quality control and archiving of COOP data are cumbersome and inefficient. The basic observing equipment is, for the most part, unchanged since the program's inception. While the data does meet the most basic demands, the COOP system does not meet the expanded needs of the modern world."

Calls to modernize COOP, going back to a 1993 NWS recommendation, have inched along with little success. Last year's revision states an overriding goal of creating a National Cooperative Mesonet (mesoscale network) that will integrate and verify the quality of observations from an array of surface-observing systems that monitor the weather, water, and climate variability across the US. "In effect," the plan says, "the NCM will become a 'network of networks.'"

However, with the COOP modernization plan not calling for deployment until 2008, both commercial and academic meteorologists are designing their own systems to fill in the data gaps.

"Because there are holes within the NWS and the federal government in the data arena," says AccuWeather's Smith, "it has created, as you would expect in an economy such as ours, opportunities." Private companies have started offering to build private mesonets to support their clients. The problem is that the network data is also private. "What if you need data on Hurricane X," Smith asks, "and the only way to get information on that is through a licensing or royalty fee from a private company? It seems pretty petty when it comes down to life and death, but I have the feeling we're going down that path."

The National Oceanographic and Atmospheric Administration (NOAA) and NWS public affairs officials had no comment. According to information released by the NWS Employees' Organization, the Department of Commerce has issued a memo restricting comment from NWS employees without approval from DOC. One public affairs official didn't respond to an interview request, a second referred the request to another office, and a third was unable to schedule an interview.

Academic researchers are already deploying their own regional mesonets. Some of these are advanced systems. Oklahoma's mesonet (http://okmesonet.ocs.ou.edu/), for example, has about 110 reporting stations and has been completely operational since 1993.

Others are in their infancy. Systke Kimball, an assistant professor of meteorology at the University of South Alabama, is installing a mesonet on the Alabama coast, with a goal of enhancing data used in hurricane research. Kimball has two reporting stations running, is in the process of installing two more, and has partnered with the Dauphin Island Sea Lab on a fifth. Each station is located on a school campus and contains sensors at two and 10 meters that measure ambient temperature, barometric pressure, precipitation, relative humidity, soil surface temperature, solar radiation, and soil temperature and moisture at different levels. The stations transmit the data to a computer inside the host school via radio; the school's computer then uploads it to the Internet.

Kimball's network is connected to a server at the Louisiana State University Agricultural Center, which uploads the data to the Meteorological Assimilation Data Ingest System (MADIS) database (http://www-sdd.fsl.noaa.gov/MADIS/) run by NOAA.

LSU researchers have undertaken a project that might someday be able to use Kimball's data. In fact, Ed Seidel, director of the university's Center for Computation and Technology, says a grant proposal his team wrote up before Hurricane Katrina hit was eerily prescient.

"We wrote precisely a scenario about a hurricane approaching New Orleans and the need for better forecasting, particularly of storm surge models," Seidel says. "If you had advanced computational sciences in place, you could—by responding to data from sensor networks deployed across the gulf—feed those data directly into models, and then run ensembles of models—perhaps dozens or even hundreds at the same time—and compare them with the actual data that are being collected."

Researchers could revise the models on the fly, Seidel explains, "perhaps invoking special algorithms as the hurricane approached the coast. For instance, as it approaches sandy beaches or muddy beaches or specific areas where it might be headed, we could plug those factors in and refine the forecast. It envelops all this on-demand computing, because you have to do it exactly as the hurricane approaches."

To enable this network, LSU researchers are partnering with colleagues in the Southeastern Universities Research Association's Southeastern Coastal Ocean Observing Program (SCOOP) initiative (http://www1.sura.org/3000/3310_Scoop.html). SURA proposes an open-access network of distributed sensors and linked computer models for the southeastern coastal zone. Existing systems, including several in the SURA region, are not integrated and are not fully compatible.

Gabrielle Allen is the principal investigator for LSU's effort to introduce grid capabilities (http://www.scoop.lsu.edu/gridsphere/gridsphere) to SCOOP. She says the hardware resources are almost in place, so it's time to emphasize the software to make it work. "It all comes down to how you describe the problems, how you can actually search for the data you need," Allen says. "I am astounded by the number of models that go into hurricane modeling itself. They model the cyclone tracks, the atmospheric winds, the ocean surges, the waves on the surges, the surf on the waves on the surges, and so on. And all these different components have been done by different groups, which in the past were working in isolation. So they're very different data formats, very different data descriptions."

The AccuWeather scientists believe researchers and forecasters might be able to use advanced generic modeling technologies as more sensors are deployed globally. The sensors will provide specific regional data to differentiate likely effects of a weather pattern or storm.

"With all the data explosion going on, we might have every coastline around the world littered with observation points," Smith says. "Modeling almost becomes a modular approach. One model fits all, the only difference being that you make sure you have a specific observation network underneath to use as your input."

LSU's Allen says the post-Katrina groundswell of interest should prod government officials to take the lead in coordinating network integration and data standardization.

"They certainly have to provide the coordination to do that," she says. "NOAA are involved in the SCOOP project. The community has to slowly build up to this. You have to build the need in the community to want it, so it's a relatively slow progress, but you can see it happening."