Real-Time Streaming, Business Needs Drive Demand for ‘DataOps’ Pros

Both the massive growth and competitive impact of data are motivating companies to hire what are called “DataOps” workers, according to a recent survey from Nexla, developer of a data operations platform. Nexla defines DataOps as a management practice that controls the flow of data so that teams can more quickly maximize its value. With the growing use of real-time streaming data and other drivers, organizations are seeing major increases in data every day. But data professionals say they spend too much time on mundane tasks such as data preparation and too little time finding innovative ways to transform data into a revenue-generating asset. More than 265 data professionals took part in the research. The following slide show presents survey highlights, with charts provided courtesy of Nexla.

The top data format is JavaScript Object Notation (JSON), as cited by 67 percent of survey respondents. Comma-separated value files (CSVs) followed (cited by 59 percent of respondents), and then TEXT (52 percent) and Extensible Markup Language (XML) (46 percent).

Nearly three of five respondents said their company uses real-time streaming data. More than one-third indicate that at least one-half of their data is streamed in real time, and 63 percent said their organization’s use of real-time streaming data will increase over the next year.

Nearly three-quarters of respondents said their company has plans to hire DataOps professionals within the next year. The average company has one data engineer for every five front-end business users.

The report indicates that 54 percent of companies are seeing their data grow by at least 1GB per day. What’s more, 13 percent are seeing data growth of more than 1TB per day.

About two of five respondents said a lack of data format consistency presents their greatest challenge. Data integration ranked second (as cited by 36 percent of respondents) and then the unreliability of data pipelines (35 percent).

Data professionals spend 17 percent of their time doing data clean-up/prep. They spend 14 percent of their time doing analysis, and then managerial activities (12 percent of the time) and data modeling (also 12 percent).

Seven of 10 respondents said they wish they had more time to develop new solutions or improve current processes. Only 7 percent disagree with this sentiment.

When asked which of their activities would benefit the most from automation, 56 percent of respondents selected data clean-up/prep. Analytics ranked second (as selected by 47 percent of respondents), followed by data integration (46 percent).

Nexla reports that 85 percent of respondents said their organization is working on artificial intelligence (AI) and machine learning (ML) projects. That’s up from 70 percent who indicated the same last year.

Just over three-quarters of respondents said their company relies upon data from third parties. Another 8 percent said their organizations will do so within the next two years.