Why Enterprises Should Embrace Machine Data Analytics

In the Third Age of Data, organizations are beginning to realize the promise of large-scale machine data and how to store it.

Many familiar names in IT made their name during the First Age of Data. Traditional IT infrastructure was designed around data predominantly created by humans, from email and documents to business transactions, databases and records. Twenty to 30 years ago, the volume of data was driven primarily by business processes in the form of online transactions. It was for things such as a bank generating customer statements or similar documents. Those transactions were conducted through mainframes, stored in traditional databases, transferred across storage area networks and related infrastructure until they ended up in your mailbox.

Then a Second Age of Data arose, still human-centric but driven less by processes and more by an explosion in content: office documents, streaming audio/video, digital imaging and photography, email and websites, to name a few. The addition of this variety of file types, formats and sizes on top of traditional data volume soon led to a huge increase in storage requirements. Pioneering vendors from the previous transactional age were subsequently replaced by scale-out companies that could meet the need of scale.

Now, in the Third Age of Data, industries are faced with a rising tide of data being generated from machines—sensor data, imaging, data capture, logging or monitoring and more. This hyperscale growth in machine-generated data provides a wealth of opportunities for enterprises to find new insights from complex systems.

At its core, the challenge of machine data in the Third Age of Data involves dealing with new parameters: a previously unheard-of volume and variety, now compounded by the speed and frequency with which machines generate large-scale unstructured data. This volume, variety and velocity create a “data multiplier effect” that can translate into orders of magnitude inflation in the scale of the data being collected. In the next three slides, we explore the challenges of the Third Age of Data: volume, variety and velocity.

Embedded sensors in automobiles and roadways supply information about location, speed, direction and operation, allowing for everything from better traffic management to vehicle monitoring, routing and entertainment. Similarly, network packet, traffic, and call and log monitoring provide insight into service operations and security, keeping IT data centers or telecommunications networks safe and sound. The scale of this sensor and packet data is massive and growing every year.

In some industries, companies rely on constant tiny measurements from ground or equipment-mounted sensors and devices, as well as huge and complex satellite imagery, weather models geospatial data and more. Similarly, many companies have a mix of data sources (different machine systems and humans, plus others.), with corresponding differences in data size and type. In either case, systems and storage optimized for one end of the spectrum may not be able to readily handle the other.

This may be the most challenging aspect of the Third Age of Data. A good baseball analogy would be third base, known as the “hot corner” for the many hard-hit balls the area receives. Sensors, satellites, networked systems and connected vehicles all have one thing in common: they never sleep. These machines typically operate on the basis of continual measurement—24 hours a day, seven days a week, 365 days a year—constantly streaming data for processing and storage. Moreover, the flood of data can quickly spike. In life sciences, for example, large-scale systems or teams rapidly generate tens of millions of files—or multi terabyte-size models—in just a few hours. Keeping up with that data load, and more importantly, understanding its constant ebb and flow, is an equally massive challenge.

The modern combination of flash-first hybrid storage appliances and real-time analytics has a profound impact in managing the onslaught of machine data in this Third Age of Data. It establishes new levels of performance, scalability, efficiency and reliability, and delivers the data-aware visibility that’s crucial to make accurate decisions and assessments in moments. The age of machine data holds tremendous promise and opportunity across a broad range of industries, but only for those that have the ability to gain insight at scale.