Statistical Analysis of Massive Data Streams: Proceedings of a Workshop (2004)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

EXPLORATORY CLIMATE ANALYSIS TOOLS FOR ENVIRONMENTAL SATELLITE AND WEATHER RADAR DATA 16 TRANSCRIPT OF PRESENTATION MR. BATES: Thank you. I didn't think we would be in the big room. It is nice to be in this building. I am going to mainly talk about some of our larger so-called massive data sets that we acquire now over the wire from both environmental satellites—the ones you see on the television news every night, the geostationary satellites. NOAA, as well as Department of Defense, also operate polar-orbiting—that is, satellites that go pole to pole and scan across a swath of data on a daily basis. Also, right now, our biggest data stream coming in is actually the weather radar data, the precipitation animations that you see now nightly on your local news. In talking in terms of what we just heard, in terms of the different data sets that come in, they come in from all different sources. The National Climatic Data Center is the official repository in the United States of all atmospheric weather- related data. As such, we get things like simple data streams, the automatic observing systems that give you temperature, moisture, cloud height at the Weather Service field offices. Those are mostly co-located now at major airports for terminal forecasting, in particular. We have, in the United States, a set of what are called cooperative observers, about 3,000 people who have their own little backyard weather station, but it is actually an officially calibrated station. They take reports. Some of them phone them in, and deposit the data, and that is a rather old style way of doing things. We have data that comes in throughout the globe, reports like that, upper-air reports from radiosondes, and then the higher data now are the satellite and weather radar data. The United States operates nominally two geostationary satellites, one at 75 watts, one at 135 watts. The Japanese satellite, which is at 155 degrees East, is failing. So, we are in the process of actually moving one of our United States satellites out there. Then, of course, these polar-orbiting satellites. I am mostly going to talk about the polar-orbiting satellites and some techniques we have used to analyze those data for climate signals. Those data sets started in about 1979, late 1978, and then go through the present. 

EXPLORATORY CLIMATE ANALYSIS TOOLS FOR ENVIRONMENTAL SATELLITE AND WEATHER RADAR DATA 17 This is what I want to talk about today, just give you a brief introduction of what we are thinking about as massive data is coming in, and we are responsible for the past data as well as the future planning for data coming in from the next generation of satellites. A couple, three examples of how we use some techniques, sometimes rather simplistic, but powerful, to look at the long-term climate trends, some time-space analysis—that is, when you have these very high spatial and temporal data sets, you would like to reduce the dimensionality, but yet still get something meaningful out about the system that we are trying to study. Then, just briefly talk about amplification of the radar data. I just inherited the radar data science there, and so, that is new stuff, and it has just really begun in terms of data mining. So, when you have rare events in the radar such as tornadic thunderstorms, how can we detect those. Then, just a couple of quick conclusions. That is what we are talking about in terms of massive here. So, the scale is time from about 2002 projecting out about the next 15 years or so. This is probably conservative because we are re-examining this and looking at more data, probably, more products being generated than we had considered before. On the axis here is terabytes, because people aren't really thinking of pedabytes. Those numbers are really 10, 20, 30 pedabytes. Right now, we have got a little over one pedabyte and daily we are probably ingesting something like a terabyte. The biggest data set coming in now is that we are getting the next rad data—this is the weather radar data—from about 120 sites throughout the United States. We are getting about a third of that in real time. They used to come in on the little Xabite 8 millimeter cassettes. For years, we used to just have boxes of those because there wasn't 

EXPLORATORY CLIMATE ANALYSIS TOOLS FOR ENVIRONMENTAL SATELLITE AND WEATHER RADAR DATA 18 a funded project to put this on mass store. In the last two years, we have had eight PC work stations with each having eight readers, tape readers, on them, to read back through all the data and get it into mass store. So, now we can get our hands on it. So, the first lesson is accessibility of the data, and we are really just in a position now to be going through the data, because it is accessible. We are looking at data rates by 2010, on the order of the entire archive—this is cumulative archives. So, that is not data read per year, so it is cumulative archive building up, of something over 30 pedabytes by the year 2010 or so. So, that is getting fairly massive. In terms of remote sensing, there is a philosophical approach, and I am not sure how many of you have worked with remote sensing data. There are two ways of looking at the data, sort of the data in the satellite observation coordinates or the geophysical space of a problem you want to deal with. These are referred to variously as the forward problem. Just very briefly, the forward problem, you have geophysical variables—temperature and moisture profiles of the atmosphere, the surface temperature, and your satellite is looking down into that system. So, using a forward model—a forward model being a radiative transfer model, physical model for radio transfer in the atmosphere—you can simulate so-called radiances. The radiances are what the satellite will actually observe. In the middle are those ovals that we want to actually work on, understanding the satellite data and then understanding the processes of climate, and then, in fact, improving modeling. As an operational service and product agency, NOAA is responsible for not just analyzing what is going on but, foolishly, we are attempting to predict things. Analogous to other businesses, we are in the warning business. The National Weather Service, of course, is bold enough to issue warnings. However, when you issue warnings, you also want to look at things like false alarm rate. That is, you don't want to issue warnings when, in fact, you don't have severe weather, tornadoes, etc. The other aspect of the problem, the so-called inverse problem—so, starting from the bottom there—you take the satellite radiances and we have an inverse model that is usually a mathematical expression for the radio transfer equation which is non-linear. We linearize the problem. Then we have a linear set of equations. The inverse model, then, is an inverse set of equations. The matrix is usually ill conditioned. So, we go to those yellow boxes and condition the matrix by adding a priori information, a forecast 

EXPLORATORY CLIMATE ANALYSIS TOOLS FOR ENVIRONMENTAL SATELLITE AND WEATHER RADAR DATA 19 first guess, other a priori data, and then biases to somehow normalize the data set. We invert that to get geophysical retrieval. Then we can retrieve temperature and moisture profiles. We can retrieve surface properties, surface temperature, ocean wind speed, other geophysical quantities of interest. So, the first application, detection of long-term climate trends using environmental satellite data, the issue of global warming has really surfaced in the last 10 years. We would like to know, is the Earth warming, how much. Are systems changing? How much? Is there more severe weather? That would just be an issue with the extremes in a distribution. You know, certain weather events are normal distributions. Certain aren't. Precipitation is not normally distributed by any sense of the imagination. We get far fewer events of extreme rainfall—precipitation—than we do of light precipitation. So, it is more of a log normal distribution. We would like to know, are the extremes changing. So, that is a small portion of those distributions. With satellite data, we face a couple of unique problems. First, we are sensing the atmosphere with satellites that have a lifetime of three to five years. So, we need to create a so-called seamless time series so that, when you apply time-space analysis techniques, you are not just picking up artifacts of the data that have to do with a different satellite coming on line. We use a three-step approach to that, something we call the nominal calibration. That is where you take an individual satellite, do the best you can with that satellite in terms of calibrating it, normalizing the satellites, and I will show you what that means. We have different satellites with different biases. Often, different empirical techniques are used to stitch those together in a so-called seamless manner. We would like an absolute calibration. That, of course, is very difficult, because what is absolute, what is the truth? Then, we would like to apply some consistent algorithm. In the infrared, when you are remote sensing in the infrared, and you are looking down at the atmosphere from space, in the infrared, clouds are opaque. So, in order to send the temperature and moisture profile down to the surface, you have to choose or detect the cloud-free samples. So, you have to have a threshold that tells you, this is cloudy, this is clear. You can base that on a number of different characteristics about the data, usually time and spatial characteristics, time and space variability. Clouds move, the surface tends to be more constant in temperature. Not always, but the oceans certainly do. So, you use those statistical properties about changes in time and space of the data set, to allow you to identify clouds. You have to look at navigation. You have to do all kinds of error 

EXPLORATORY CLIMATE ANALYSIS TOOLS FOR ENVIRONMENTAL SATELLITE AND WEATHER RADAR DATA 20 checks, and then build retrievals to go from your radiant space to your geophysical space. Then, we get, finally, into the fun part, exploratory data analysis. I tend to view this as sort of my tool kit out there in the shop working on a data set where, you know, you throw things at it and see what sticks. Once you get something that looks interesting, you start to formulate hypotheses about the physical system, how it works, and how your data set compares to what physics of the problems say are possible solutions. Then you might go on to look at data analysis and confirm your hypothesis. Anyway, let's go through the first step here. I am going to take more time with this first example and a little less with the second and just briefly go into the third one. So, creation of seamless time series, you have here three different channels of data from a satellite, channel 8, that is an infrared window, channel 10 is actually a moisture channel in the upper atmosphere, a so-called water vapor channel, and these channels—10, 11, 12—are all water vapor channels. We look at emission lines of water vapor in the atmosphere. Channel 12 in particular we are going to look at because it is involved with a so-called water vapor feedback mechanism in global warming. In global warming, we hear these numbers quoted—atmospheric, oh, the temperature is going to go up two degrees in 100 years. Actually, anthropogenic CO2 manmade gasses only contribute one of the two degrees there. The extra warming, the other one degree of warming, comes from a so-called water vapor feedback. So, there has been a lot of controversy in the community about, does this water vapor feedback, in fact, work this way or not. So, the different colors in these three charts, then, I am showing three things. One is the average global temperature over time, and this is a 20-year data set. So, the top line in each one of these is just your monthly mean data point for each of these satellites over time, about a 20-year time series on each one. These are four different channels. The mean—you see the march of the annual cycle up and down—the standard deviation of the data set, and just simply the number of observations, these are something like millions of observations a month—you can't read that scale here, this is times 106. So, on the order of, you know, 5 or 6 million or so observations a month coming down. This is from the polar-orbiting satellite. So, these have sampled the entire planet. The geostationary only sample that region that they are over. You can see a bit of the problem here, especially in this channel 12 where, number one, there are offsets between the different colors. The different colors are the different satellites. Over this time period, there are eight different satellites. They are designated by this NOAA-7, 8, 9, 10, 

EXPLORATORY CLIMATE ANALYSIS TOOLS FOR ENVIRONMENTAL SATELLITE AND WEATHER RADAR DATA 21 etc. These are a series of the NOAA polar-orbiting satellites. So, there are a couple of things you can pick out right away. There are biases between—this is supposed to be the same channel, but physically we know there are some differences. We can account for some of those, physically, it is just a matter of the system. We would like to seam these time series together. There are offsets and there is another problem here that you can probably see. This yellow one drifts in time, while the satellite crossing time is actually drifting in time later in the day. This can be problematic, depending on what you are trying to study. So, we would like to stitch those time series together to get a seamless time series, and then do time series analyses. So, it takes a lot of checking. Over here, these are individual swaths of data, so, swath one, two, three. This is the Middle East. This is Saudi Arabia, the Red Sea. This is Africa, South Africa here. The different colors denote the different temperatures that the system is radiating at. Then, we have several other things going on here. We have already applied the cloud detection algorithm. So, the spotty pixilation of these swaths, the dark spots are where we have detected clouds and then not put a color in. So, you only see color where we have detected already that it is clear. Then you have another process you are banding here. The instrument is calibrated every 40 lines. So, instead of looking at the Earth, it looks inside the housing at black bodies with constant temperatures, so it can get an absolute calibration every 40 lines. So, this is a couple of swaths. There are 14 swaths a day for each satellite. You start to composite them together in a global view now. So, this is the global area, this is the Americas here, North and South America, the outlines are in white of the continental land boundaries, the Pacific Ocean. Here, again, we have color-coded the radiant temperatures. Dark areas are missing. That is persistent cloudy areas. Then, we have gridded these together for a five-day period where we can start to evaluate that visually also for quality. Then, this is a long-term sort of diagram of the health of the satellite data set again. These are just simple statistical quantities, but very helpful for scanning out bad data. What we have here plotted are simply the mean in red, of each swath, for an entire year. So, we just compute the mean of each channel, pretty simple, the maximum that we detect and the minimum that we detect. You can start to see, when you do this, right away you get outliers, and those outliers occur preferentially in different seasons. As this satellite goes around, seasonally, you will see different things, and you tend to have problems. This already 

EXPLORATORY CLIMATE ANALYSIS TOOLS FOR ENVIRONMENTAL SATELLITE AND WEATHER RADAR DATA 22 allows us to throw a lot of data out that we have found is out of bounds. On the other hand, if you are looking for abnormal things, this may be the data you are interested in. For us, we know this is the bad data. We don't want that. So, we composite them, we obtain metadata, and we save that for further analysis. Now that we have seamed everything together, what do we want to do? Well, we want to try to get a handle on what the system is doing. What we have done here is composite a bunch of different analyses together about the system. The top two panels are the spatial patterns of empirical orthogonal analysis of precipitation, and then this is water vapor. What we have done here is that we have subtracted the annual harmonics from the time series of the data sets, so that we can look at interregnal variability. So, it is just a simple filtering technique. What we have done is, we have fit the first three harmonics in the annual cycle to all the data sets, to every point in the Tropics. So, these are 30 degrees North/South now. We subtracted those out. Then, we have done empirical orthogonal analysis on monthly mean data. This is precipitation. This is the so-called El Niño swing in the Pacific. So, during warm events with El Niño, you have less convection and precipitation in the west Pacific and more in the central and eastern Pacific. That signal shows up much more as a global signal in the upper tropospheric humidity. You have teleconnections, so-called teleconnections, where the specific pattern here, again, is much more moist in the central equatorial Pacific. At the same time, you have much drier areas north and south of that. So, you have a speeding up of the whole atmospheric cycle. These are typical indices of time series. So, this is a 22-year time series. These are El Niño events. These are 1982–1983, a large event. Then you tend to get a small cold event. This is a modest event in 1986–1987, a big cold event in 1989. Cold events for the United States, in particular, are noted for droughts. It bounces around in the early 1990s, and this is 1997–1998, when it got a lot of publicity. You see it is sort of a four-year periodicity. The other time series that I didn't show you—it is supposed to be over here and magically isn't, I don't think—this is the other time series I wanted to show you. These are just global average time series, 30 degrees South, where we have stitched these things all together, we have gone through about 1.5 terabytes of data. Now, we are just doing some really big-time summarizing of the data. These are just simple tropical time series, and here you see the speed of El Niño again. It is about every three to four years on these time series. The time series I am 

EXPLORATORY CLIMATE ANALYSIS TOOLS FOR ENVIRONMENTAL SATELLITE AND WEATHER RADAR DATA 23 interested in are these guys for the Tropics. This is this upper trop humidity. I have subtracted, still, these harmonics of the annual cycle. So, I was very surprised to see these time series sort of just look like white noise. I said, what in the world is this? I said, I know what these are. This is an El Niño event, this is an El Niño event, this is a cold event, here is the big 1997–1998 El Niño. So, this is sort of easy, when I see these beats of this time series. I know what those guys are. When I saw this I said—of course, the first thing you always say to yourself, did I screw up. Did I really subtract out the annual cycle from here to get interregnal variability? So, I went back. Yes, I did. What in the world is going on with this? Then I started noticing, while some of these peaks here are synchronous with these peaks here and some of these other ones are synchronous, there is a lot more going on. These time series, this one here, and this one which is a global radiation time series, there is a lot more going on. There is much more, if you will, of white noise. This is more red, or this is a period of three or four years. This has a lot of stuff going on. There are some synchronous events going on. Based on that, and some talking with colleagues, we formulated a hypothesis that involves a seasonality and an interregnal time scale. What we came up with is also knowledge of the system. That is, the dynamics of the atmosphere works such that, when you get strong westerlies across the equator in these El Niño cold events, you get westerlies, and this can lead to strong eddies. Big eddies are just big winter storms and they flex moisture up into the upper atmosphere. On the other hand, this leads to the possibility of a dynamical wave duct. In the atmosphere, you can get storms in this configuration of the atmosphere in northern winter and spring, or you can get storms that come down deep into the Tropics, and actually cross the Equator. In summer, you can't get that. That is why you don't see those extremes in summer. This one, you have the opposite. This is an El Niño warm event condition. You have deep convection extending out deep across the Pacific, along the Equator to the central, and even the eastern Pacific. What happens there is, you have strong westerlies, dynamically, no winds and then westerlies. That means that these storms actually can't go against the gradient of the planet here and against this sheer. So, no storms get into the subtropics. It gets very dry, and this helps balance the system in the warm and cold events. So, it is sort of a neat thing. On the longer time scale, we are interested in longer-term trends of the system. So, we want to take those long time series and, again, do some rather simple things. 

EXPLORATORY CLIMATE ANALYSIS TOOLS FOR ENVIRONMENTAL SATELLITE AND WEATHER RADAR DATA 24 Long-term climatology, that is just the long-term monthly mean for 20 years. This is your global pattern. In the Tropics, your blues are your monsoon regions, the reds are your desert regions, basically, and this is just a zonal average of that, that shows you that the Tropics are moist, the subtropics are dry, in midlatitudes, you have more of a constant temperature-moisture relationship. This is your linear trend, pretty simple, just a linear fit to the system. Subtropics, or the deep Tropics, because of those El Niño events of the last 20 years, are trending to tend slightly more moist in the subtropics to lower midlatitudes drying out a little bit. Of course, you would like to assess the statistical significance of any of this. This confidence interval is just computed at each grid point time series, and it is both the fit to the linear trend, plus a red noise persistence term. That is just a simple lag one auto-correlation, and then fit to the significance in the length of the time series. Example number two, I will try and speed up a little bit. We would like to try to reduce the dimensionality of massive time-space data sets. One of the easiest ways to do this is also to take advantage of one of our dynamical systems. Many of our systems propagate west to east basically along latitudes. So, what we can do is, we can take advantage of this by taking a cross section at any longitude, and then averaging data for latitudinal bands. By doing this, we have reduced the dimensionality. So, here is an example of radar echoes from these weather radar precipitation data sets. We take a particular longitude here. You are about 90 degrees West. Then, what you can do is average five-degree latitude swaths. From that, you end up with a diagram that you see on the bottom. We call these Hovmöller diagrams. The weather pioneers 

EXPLORATORY CLIMATE ANALYSIS TOOLS FOR ENVIRONMENTAL SATELLITE AND WEATHER RADAR DATA 25 did this. This is how they predicted—the first predictions were not numerical models, but were statistical techniques where we reduced the dimensionality of the data set, and then looked at the propagation speed. Since this is a time- space diagram, this is the degrees of longitude per time steps. Here are days on your ordinate here. We can actually, from these diagrams, just come off with a propagation speed of various phenomena here. These guys tend to propagate slower, and these guys are propagating a little faster. This, although simple, is a very powerful technique. I am going to skip that example and I am going to go right to the end here. What you would really like to do is analyze this in time-frequency space. You apply FFT to both time and space dimensions and you come out with a frequency wave number diagram that allows you to detect various atmospheric phenomena, so called Madden-Julian Oscillations, Kelvin Waves and other waves, their propagation direction, their wave length and their periodicity. 

EXPLORATORY CLIMATE ANALYSIS TOOLS FOR ENVIRONMENTAL SATELLITE AND WEATHER RADAR DATA 26 These are very powerful. We have used those now to look at onsets of changes in the monsoons and regimes that favor or suppress hurricane activity in both the Atlantic and Pacific Oceans. Just real quick, this is just data mining techniques. This is radar. This was a confirmed tornado. This is Doppler velocity sheer, small-scale signatures only. This is the large-scale outflow boundary and techniques are being developed to classify those schemes. As with any classification techniques—I have just inherited this one—the classification depends on your trainer and then your criteria for evaluating whether or not you have success, including probability of detection, false-alarm rate and so forth. So, some of those techniques are applicable to many different situations. Again, with the public, if you are issuing warnings, like the National Weather Service 

EXPLORATORY CLIMATE ANALYSIS TOOLS FOR ENVIRONMENTAL SATELLITE AND WEATHER RADAR DATA 27 does with severe weather, you want people to take those warnings seriously. So, you want to have not only a good success rate, but a low false-alarm rate. So, you need to balance all those different factors in evaluating any technique for detection. So, concluding, we have got these massive data streams that are going to continue increasing geometrically in the next 10 to 15 years. The statistical tools range from simple to complex but, because we are dealing with such a difficult phenomenon, I really like a lot of the simple tools to first get a handle on our system. The outlook for hardware is that the hardware will probably keep up with these massive data rates, but our investment, I think, and I think many people are finding this rather obvious, so maybe I am just stating the obvious, that additional investment in the people resource is required, to ensure that the future generations have the technical skills required to fully exploit these massive data sets. Thank you.