At its annual investor meeting on May 13, IBM announced the commercialization of System S, the company’s stream computing software that advances parallelism to deliver real-time business analytics capability.
IBM also announced the opening of the IBM European Stream Computing Center, headquartered in Dublin, Ireland. The center will “serve as a hub of research, customer support and advanced testing for what is expected to be a growing base of European clients who wish to apply stream computing to their most challenging business problems,” IBM said in a news release.
Nagui Halim, chief scientist forIBM’s System S project, said the effort started as a project in IBM Research at the end of 2003 that became one of the largest software research projects ever conducted inside IBM Research. Halim said with System S and stream computing the focus is on delivering insight and foresight, not hindsight. According to the IBM release:
“System S is built for perpetual analytics-utilizing a new streaming architecture and breakthrough mathematical algorithms to create a forward-looking analysis of data from any source-narrowing down precisely what people are looking for and continuously refining the answer as additional data is made available.For example, System S can analyze hundreds or thousands of simultaneous data streams-stock prices, retail sales, weather reports, etc.-and deliver nearly instantaneous analysis to business leaders who need to make split-second decisions. The software can help all organizations that need to react to changing conditions in real time, such as government and law enforcement agencies, financial institutions, retailers, transportation companies, healthcare organizations, and more.“
Moreover, IBM is commercializing the technology at a time when clients need it most-during the global economic crisis. “Using computers to rapidly analyze multiple streams of diverse, unstructured and incompatible data sources in real time, enabling fast, accurate and insightful decisions,” as IBM described the potential of System S, can be a competitive advantage for companies.
For instance, global market data is growing at a rapid rate and “needs to be ingested, decoded, processed and responded to in short order,” and System S enables users to do that, IBM contended.
Indeed, Halim said TD Securities is using System S to ingest more than 5 million bits of trading data per microsecond to make faster financial trading decisions. To match the capacity of the system, a trader would have to be able to read the entire works of Shakespeare 10 times in less than 1 second and then identify and execute a stock trade faster than a hummingbird flaps its wings, he said.
“System S software is another example of IBM helping clients through our long-term investments in business analytics and advanced mathematics,” John Kelly III, IBM senior vice president and director of IBM Research, said in a statement. “The ability to manage and analyze incoming data in real time, and use it to make smarter decisions, can help businesses and other enterprises differentiate themselves.”
According to the release:
“IBM is making System S trial code available at no cost to help clients better understand the software’s capabilities and how they can take advantage of it for their business. This trial code includes developer tools, adapters and software to test applications.“
Halim said the System S software can be configured to run on a supercomputer, a cluster of blades or even a single computer. Its first iteration is aimed at commodity hardware, he said. And it can be configured to attack a broad set of problems across a wide range of industries, he said.
IBMs New Language, SPADE
Moreover, Halim said to make the System S concept work, IBM had to come up with a new language. However, as a computer scientist experienced in using available tools, he said, “I was reluctant to embark on creating a new language, but stream processing is a significant development and warrants a new language.”
That language is SPADE, which stands for Stream Processing Application Declarative Engine. “SPADE allows you to describe the topology of what you’re working on,” Halim said.
A description of SPADE on an IBM Research Web page reads, “SPADE … is a programming language and a compilation infrastructure, specifically built for streaming systems. It is designed to facilitate the programming of large streaming applications, as well as their efficient and effective mapping to a wide variety of target architectures, including clusters, multicore architectures and special processors such as the Cell processor. The SPADE programming language allows stream processing applications to be written with the finest granularity of operators that is meaningful to the application, and the SPADE compiler appropriately fuses operators and generates a stream processing graph to be run on the Stream Processing Core of System S.”
Halim said in addition to the SPADE language, developers can use the SPADE compiler and Eclipse-based IDE (integrated development environment) along with administration, configuration, and installation tools and adapters to build and deploy System S applications.
“Traditional computing models retrospectively analyze stored data and cannot continuously process massive amounts of incoming data streams that affect critical decision making. System S is designed to help clients become more ‘real-world aware,’ seeing and responding to changes across complex systems,” IBM said in the release.
According to IBM’s release, other early uses of System S include:
“Uppsala University and the Swedish Institute of Space Physics are using System S to better understand “space weather,” which can influence energy transmission over power lines, communications via radio and TV signals, airline and space travel, and satellites. By using the LOIS Space Center radio facility in Sweden to analyze radio emissions from space in three dimensions, scientists use this technology to compile endless amounts of data and extract predictions on activities in space. Since researchers need to measure signals from space over large time spans, the raw data generated by even one antenna quickly becomes too large to handle or store. System S analyzes the data immediately as it streams from sensors. Over the next year or so the project is expected to perform analytics on at least 6 gigabytes per second or 21,600 gigabytes per hour – the equivalent of all the Web pages on the Internet.The Marine Institute of Ireland is using System S to better understand fragile marine ecosystems. As a core component of this collaboration, a real-time distributed stream analytical fabric for environmental monitoring and management is under development. Acting on large volumes of underwater acoustic data and processing it in real-time, the Institute extracts useful information such as species identification of marine life, population count and location. […]IBM and the University of Ontario Institute of Technology (UOIT) are using System S to help doctors detect subtle changes in the condition of critically ill premature babies. The software ingests a constant stream of biomedical data, such as heart rate and respiration, along with clinical information about the babies. Monitoring “preemies” as a patient group is especially important as certain life-threatening conditions such as infection may be detected up to 24 hours in advance by observing changes in physiological data streams. The type of information that will come out of the use of System S is not available today. Currently, physicians monitoring preemies rely on a paper-based process that involves manually looking at the readings from various monitors and getting feedback from the nurses providing care.“