National Academies Press: OpenBook

Statistical Analysis of Massive Data Streams: Proceedings of a Workshop (2004)

Chapter: 6. STREAM COLLECTION AND NORMALIZATION

« Previous: 5. IUM HIGH-LEVEL ARCHITECTURE
Suggested Citation:"6. STREAM COLLECTION AND NORMALIZATION." National Research Council. 2004. Statistical Analysis of Massive Data Streams: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/11098.
×
Page 314

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

A STREAM PROCESSOR FOR EXTRACTING USAGE INTELLIGENCE FROM HIGH-MOMENTUM INTERNET DATA 314 IUM platform has been implemented in Java, which enables multiplatform operation. The architecture has been designed with high modularity and configurability from the start. Upon start-up each of the Collectors obtains its own configuration from a central configuration server and then builds itself with the proper components required. Figure 2. HP's Internet Usage Manager (IUM) enhanced with Dynamic Network Analysis is a distributed agent architecture. 6. STREAM COLLECTION AND NORMALIZATION The input streams are captured from the source devices by plug-in Encapsulator (Figure 3) components represented in two different colors. The gradient shaded purple to gold ones are configured to interpret the data from specific source device types. The gold encapsulators are configured to read normalized data. Over the past few years we have developed many preconfigured encapsulators for a wide range of devices mentioned above as well as different collection modes that include real-time streams, (e.g., NetFlow, sFlow, DDNS), polled data (e.g., SNMP), files and directories, and databases (via JDBC). Figure 3. Encapsulation plug-ins are tailored to collect different types of input streams.

Next: 7. STREAM RULE PROCESSING »
Statistical Analysis of Massive Data Streams: Proceedings of a Workshop Get This Book
×
 Statistical Analysis of Massive Data Streams: Proceedings of a Workshop
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Massive data streams, large quantities of data that arrive continuously, are becoming increasingly commonplace in many areas of science and technology. Consequently development of analytical methods for such streams is of growing importance. To address this issue, the National Security Agency asked the NRC to hold a workshop to explore methods for analysis of streams of data so as to stimulate progress in the field. This report presents the results of that workshop. It provides presentations that focused on five different research areas where massive data streams are present: atmospheric and meteorological data; high-energy physics; integrated data systems; network traffic; and mining commercial data streams. The goals of the report are to improve communication among researchers in the field and to increase relevant statistical science activity.

READ FREE ONLINE

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!