Database Legend: How Real-Time Data Analysis Will Transform Society - Page 2

By Lisa Vaas  |  Posted 2005-02-15 Print this article Print

In round numbers, were two orders of magnitude faster than the elephants. And the two orders of magnitude are on identical hardware. If you normalize for clock speed of our production application vs. theirs, were one order of magnitude faster. What accounts for this speed gain? There are three big reasons: One, the elephants store the data. Theres no need to store the data. One of the characteristics of real-time, streaming data, its like IT sushi. It has high value right now, and the value decays very quickly. Theres no need to keep the data around for the long term in some sort of repository. That just takes up time, latency and resources to do that.
Reason No. 2 is when youre looking for the inter-arrival time between ticks, thats a time-series notion. When youre doing real-time stream processing, we have time-oriented primitives in the bottom of the screen. … We have extended SQL to something we call StreamSQL, which has extra stuff in it. … Weve had to add another notion to SQL, the notion of time windows. You can do SQL-like calculations over time windows. Do them in real time as data is flying by. …
[Finally,] if you want to count to 100, which is what this [application] had to do in order to decide to ring the red phone, the most efficient way to do that is with four lines of C++. In this application, it makes sense to mix small amounts of code in a general-purpose environment with database-oriented processing steps. We can do that in our architecture: freely intermix C++ with our StreamSQL primitives. The relational guys all run client/server, and C++ code has to run in the client in a separate place from the server. So the client/server architecture slows you down on this style of application. What types of enterprises need this type of fast analysis? Financial services, industrial process control, monitoring oil refineries, the government: Military and homeland security is full of this style of application. Weve been talking to one of the three-letter agencies. The guys who wont give you their business cards. Theyre monitoring Arabic chatter. When the czar of homeland security says, "The chatter has changed," theres a real-time system processing incoming feeds, computing statistics on incoming Arabic language streams, to actually determine that. They started yakking with us on piloting that application. Another example: network monitoring, for DOS [denial of service] attacks. Fraud detection. Next Page: Financial firms seek to thwart identity theft.

Lisa Vaas is News Editor/Operations for and also serves as editor of the Database topic center. Since 1995, she has also been a Webcast news show anchorperson and a reporter covering the IT industry. She has focused on customer relationship management technology, IT salaries and careers, effects of the H1-B visa on the technology workforce, wireless technology, security, and, most recently, databases and the technologies that touch upon them. Her articles have appeared in eWEEK's print edition, on, and in the startup IT magazine PC Connection. Prior to becoming a journalist, Vaas experienced an array of eye-opening careers, including driving a cab in Boston, photographing cranky babies in shopping malls, selling cameras, typography and computer training. She stopped a hair short of finishing an M.A. in English at the University of Massachusetts in Boston. She earned a B.S. in Communications from Emerson College. She runs two open-mic reading series in Boston and currently keeps bees in her home in Mashpee, Mass.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel