Nearly every large organization today wants to take advantage of big data analytics to inform decisions and gain a competitive edge. For example, the 2019 NewVantage Big Data and AI Executive Survey Report found that nearly 92% of the Fortune 500 executives they surveyed are increasing their investments in big data and AI, and 55% are spending more than $50 million.
But while nearly 62% of Fortune 500 executives say they are seeing measurable results, only 31% say they have a data-driven organization and just 28% have created a data culture.
Go here to see a listing of eWEEK’s Top Predictive Analytics Companies.
A related survey from Interana may help explain this disparity. Interana surveyed 85 business leaders from large organizations in March. They found that:
- a significant percentage of organizations gain insights from big data exclusively through pre-defined dashboards; and
- posing iterative questions to the data is difficult, due to the delay introduced by needing to formulate new queries; involve technical specialists; extract, transform and load different data sets (ETL); and wait for results.
To create a data-driven culture, managers need fast, direct access to insights based on data, and it appears that, currently, most organizations cannot provide that to their employees.
In this eWEEK Data Points article, Bobby Johnson, co-founder and CTO of Interana, walks readers through these findings to explain these studies’ implications for the current state of big data analytics in the enterprise and building a data-driven organization.
Data Point No. 1: It takes a long time to get answers from big data
In the Interana survey, nearly two-thirds of respondents (63%) said they have to wait hours (34%) or days (29%) to build a new query, and 18% said it can take weeks (11%) or months (6%) to do so. That said, just less than half (49%) say they are dissatisfied with how long it takes to get answers. One might expect that number to be higher, but given that just 11% said they can currently get answers within minutes, it appears that people are accustomed to waiting for queries to be built; they’ve never experienced any other way.
Data Point No 2: Business adoption of big data continues to be a challenge
In the NewVantage survey, 77% of respondents said that business adoption of big data and AI initiatives poses a serious challenge. Most likely, this challenge exists not only because it takes a long time to get answers from data but also because a significant percentage (39%) of business leaders surveyed by Interana say that they only have access to reports based on pre-defined queries. Additionally, 41% need to involve IT or other technical resources to ask questions of data at all.
Data from pre-defined queries is useful for monitoring key performance indicators and to understand what’s happening day-to-day in the business, but when things are going wrong, in order to understand why problems are arising, managers need to be able to ask multiple questions of the data to identify the cause. Pre-defined queries don’t allow for this kind of questioning, and, while it’s possible to ask iterative questions over a period of weeks or months with successive rounds of query building and ETL cycles, few organizations can wait that long to make critical decisions.
Data Point No. 3: There’s a serious shortage of data scientists
If an organization depends on data scientists to create queries, creating a data-driven culture is going to be tough. Not only are these highly skilled individuals a limited resource within an organization, but they are also in extremely short supply. According to a report last year from LinkedIn, the United States alone has an immediate need for more than 151,000 additional data scientists. So, if a data scientist leaves your organization–and according to a 2019 survey from Burtch Works, they change jobs about every 2.6 years–recruiting a replacement will pose a serious challenge.
Data Point No. 4: Executives know they must create data-driven organizations
Just shy of 80% of Fortune 500 executives fear disruption from data-driven organizations, according to the NewVantage report. It’s a valid fear. For example, 10 years ago, Facebook was neck-and-neck with MySpace for dominance of the U.S. social media market, and it didn’t surpass its primary competitor until May 2009. There are many reasons why Facebook would become such a dominant social media force, but one of the biggest was its highly data-driven culture. More than half of all employees regularly used the company’s internal data analytics platform to get real-time answers about user behavior. They didn’t need to rely on intuition when making design decisions; they could pose questions to the data to see what caused people to interact longer with the site and become more engaged with the platform over time.
Data Point No. 5: Conclusion
Until business managers have real-time access to answers from data, creating a data-driven organization will remain extremely difficult. The pace of business moves too fast to wait hours or days for an answer to a single question. Until non-technical employees can meaningfully and independently use big data to get answers quickly, decisions will continue to be based on the gut feel of the highest-salaried person in the room. Those organizations that succeed in creating a data culture will have a significant competitive advantage over those who lag behind.
If you have a suggestion for an eWEEK Data Points article, email cpreimesberger@eweek.com.