eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Big changes are coming in the big data and analytics arena, and IT decision-makers should prepare for major advances in 2018. The most significant big data trend is the increased integration of artificial intelligence (AI) and machine learning, but metadata management and global data fabrics will also make an impact next year. Following are predictions from some prominent industry experts.
“Artificial intelligence—especially data science and machine learning (DS&ML)—will change the way we acquire, manage and analyze data,” said Jorgen Heizenberg, Gartner data and analytics research director. “Today, due to its complexity, this is mostly done by humans—often developers hired from an external service provider, he said.
“However, DS&ML are the engines of future data and analytics (D&A) services. At first, it will be more about automating simple and routine tasks like data extraction,” Heizenberg said. “In time, more complex and non-routine tasks will follow, leading to ‘intelligent’ automation. This will potentially scale the enterprise insights, as it allows more room for human-based analytics.”
“We will witness a shift from labor-based D&A services toward machine-based, often as part of a converged analytical solution of services and software or ‘servware.’ Companies should introduce testing and review boards on all models, algorithms and data used in order to build trust,” Heizenberg recommended.
Getting Better Outcomes From Big Data and AI
“There is no AI without IA—information architecture,” stated Rob Thomas, general manager, IBM Analytics. “If companies don’t have the right infrastructure, it’s hard to do AI right. To get meaningful results, data needs to be in an organized state, and the right technology has to be put into action.”
Thomas pointed out that the internet of things (IoT) has added significantly to data challenges by forcing companies to ingest massive amounts of data at incredibly high speeds in short periods of time. This is putting a strain on many organizations.
Most executives Thomas has spoken with understand that data is a source of competitive advantage, but they want to get better outcomes than the ones they’ve gotten to date. One solution he suggested is to create data catalogs in a consumable form, which can help by making it easier to find data and gain insights from it.
Metadata management plays a role here. “Companies need a single source of truth,” Thomas said. “Data catalogs provide that, and metadata management details what’s in the catalog.”
This type of data governance is essential for both regulatory compliance and self-service access to data, according to Thomas. He added that the European Union’s General Data Protection Regulation (GDPR) “will give a big boost to data governance in 2018.”
Another point Thomas stressed is the importance of democratizing data analytics so that even non-technical workers—not just data scientists and business analysts—can use these products.
“Analytics systems should be simple, elegant and have a great design,” he said, adding that this would make these products easier to install and use, thus making them available to a growing number of employees. “At IBM.com, for example, a non-tech person can download DB2 in less than 10 minutes,” he said.
Boosting Analytics With AI
“The world is moving from big data to all data—structured, unstructured and contextual—from sources such as sensors, social, video and chat,” said Jean-Luc Chatelain, chief technology officer at Accenture Applied Intelligence. “This makes it much harder to separate the signal from the noise, but you can’t get good insights from bad data. That’s why, in 2018, artificial intelligence will have a much bigger role to play in data preparation.
“But it’s not just data preparation: AI is offering businesses the opportunity to boost analytics across the patch. For example, [they can] detect and amplify ‘weak signals’ and identify patterns that are otherwise invisible. This will arm businesses with a new level of intelligence that can, for instance, tailor interactions and offerings for each customer or facilitate personalized drug development. We’ll see a lot more of AI- and analytics-powered hyper-personalization in the near future.”
Chatelain pointed out that “This new intelligence will impact the workforce. It’ll enhance human decision-making and allow rote tasks to be automated, giving employees the opportunity to take on more strategic and rewarding roles.
“Whether they deploy AI to power more precise insights from data or to automate processes, businesses should always put people first. This includes introducing AI in a way that is compatible with the wellness of their employees and customers,” he said.
Businesses, Chatelain said, “need to comply with ethical AI design standards and, as personal data will be used to derive unprecedented insights, ensure a higher level of protection and transparency.”
Building a Global Data Fabric
“Big data is becoming an essential asset, and enterprises are transforming into data-driven concerns,” said Ted Dunning, MapR’s chief application architect. “This transformation naturally leads to big data systems becoming the center of gravity for enterprises, in terms of data size, storage and access, as well as operations and analytics.”
“As a result, more businesses will be looking for ways to build a global data fabric that breaks down silos to give comprehensive access to data from many sources and to computation for truly multitenant systems.”
Dunning predicts that in 2018, “We will see more businesses treat computation in terms of data flows rather than data that is just processed and landed in a database. These data flows capture key business events and mirror business structure.”
He believes “a unified data fabric will be the foundation for building these large-scale flow-based systems. Such a fabric will support multiple kinds of computation that are appropriate in different contexts.”
Dunning believes that “databases will become the natural partner and complement of a dataflow. The emerging trend is to have a data fabric that provides data-in-motion and data-at-rest, which is needed for multi-cloud computation provided by things like Kubernetes.”
Challenges and Opportunities
Despite the enormous opportunities offered by big data and analytics, the field also presents some challenges. In SAP’s global study, “Data 2020: State of Big Data Study,” 74 percent of the IT decision-makers surveyed said their data landscape is so complex that it limits their agility, and half said that many employees and managers can’t access critical data.
In addition, 85 percent of the survey respondents struggle to deal with data that comes from a variety of locations, and 72 percent said their data landscape is complex because it includes a variety and number of data sources.
As AI and other technology advances are used to enhance and simplify big data and analytics, organizations will be better equipped to deal with these complex challenges.