Close
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Logo
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Big Data and Analytics
    • Big Data and Analytics

    Five Reasons Why Your Data Science Project is Likely to Fail

    By
    Chris Preimesberger
    -
    March 27, 2019
    Share
    Facebook
    Twitter
    Linkedin
      eWEEK logo Data Points copy

      Companies are forging ahead with digital transformation at an unprecedented rate. A recent survey by Gartner Research found that 49 percent of CIOs are reporting that their enterprises have already changed their business models to scale their digital endeavors or are in the process of doing so.

      As companies forge ahead with these transformations, they are infusing data science and machine learning into various business functions. This is not an easy task. A typical enterprise data science project is highly complex and requires deployment of an interdisciplinary team that involves assembling data engineers, developers, data scientists, subject matter experts and individuals with other special skills and knowledge.

      Moreover, this talent is scarce and costly. In fact, only a small number of companies have succeeded in building an experienced data science practice. And, while building this team takes time and resources, there is an even larger problem faced by many of these companies: more than 85 percent of big data projects fail.

      A number of factors contribute to these failures, including human factors, and challenges with time, skill and impact. In this eWEEK Data Points article, Ryohei Fujimaki, Ph.D., founder and CEO of dotData, a Silicon Valley tech startup focused on data science automation for the enterprise, discusses five key factors that contribute to these failures. 

      Data Point No. 1: Lack of Resources to Execute Data Science Projects

      Data science is an interdisciplinary approach that involves mathematicians, statisticians, data engineering, software engineers, and importantly, subject matter experts. Depending on the size and scope of the project, companies might deploy numerous data engineers, a solution architect, a domain expert, a data scientist (or several), business analysts and perhaps additional resources. Many companies do not have and/or cannot afford to deploy sufficient resources because hiring such talents is becoming increasingly-challenging and also because companies often have many data science projects to execute, all of which take months to complete.

      Data Point No. 2: Long Turnaround Time and Upfront Effort Without Visibility into the Potential Value

      One of the biggest challenges of data science projects is the big upfront effort required, despite a lack of visibility into the eventual outcome and its business value. The traditional data science process takes months to complete until the outcome can be evaluated. In particular, data and feature engineering process to transform business data into a machine learning ready format takes big amount of iterative efforts. The long turnaround time and substantial upfront efforts associated with this approach often result in project failure after months of investment. As a result, business executives are hesitant to apply more resources.

      Data Point No. 3: Misalignment of Technical and Business Expectations

      Most data science projects are undertaken to provide important insights to the business team. However, often a project starts without clear alignment between the business and data science teams on the expectations and goals of the project, resulting that the data science team is focused mainly on model accuracy, while the business team is more interested in metrics such as the financial benefits, business insights, or model interpretability. At the end, the business team does not accept the outcomes from the data science team. 

      Data Point No. 4: Lack of Architectural Consideration for Production, Operationalization

      Many data science projects start without consideration for how the developed pipelines will be deployed in production. This occurs because the business pipeline is often managed by the IT team, which doesn’t have insight into the data science process, and the data science team is focused on validating its hypotheses, and doesn’t have an architectural view into production and solution integration. As a result, rather than getting integrated into the pipeline, many data science projects end up as one-time, proof-of-concept exercises that fail to deliver real business impact or causes significant cost-increases to productionalize the projects.

      Data Point No. 5: Heavy Dependency on Skills, Experiences of Particular Individuals

      Traditional data science heavily relies on skills, experiences and intuitions of experienced individuals. In particular, the data and feature engineering process now is mostly based on manual efforts and intuitions of domain experts and data scientists. Although such talented individuals are precious, the practices relying on these individuals are not sustainable for enterprise companies, given the hiring challenge of such experienced talents. As such, companies need to seek solutions to help democratize data science, enabling more participants with different skill levels to effectively execute on projects.

      Data Point No. 6: End-to-end Data Science Automation is a Solution

      The pressure to achieve greater ROI from artificial intelligence (AI) and machine-learning (ML) initiatives has pushed more business leaders to seek innovative solutions for their data science pipeline, such as machine learning automation. Choosing a right solution that delivers end-to-end automation of the data science process, including automated data and feature engineering, is the key to success for a data-driven company. Data science automation makes it possible to execute data science processes faster, often in days instead of months, with more transparency, and to deliver minimum viable pipelines that can be improved continuously. As a result, companies can rapidly scale their AI/ML initiatives to drive transformative business changes.

      Chris Preimesberger
      https://www.eweek.com/author/cpreimesberger/
      Chris J. Preimesberger is Editor Emeritus of eWEEK. In his 16 years and more than 5,000 articles at eWEEK, he distinguished himself in reporting and analysis of the business use of new-gen IT in a variety of sectors, including cloud computing, data center systems, storage, edge systems, security and others. In February 2017 and September 2018, Chris was named among the 250 most influential business journalists in the world (https://richtopia.com/inspirational-people/top-250-business-journalists/) by Richtopia, a UK research firm that used analytics to compile the ranking. He has won several national and regional awards for his work, including a 2011 Folio Award for a profile (https://www.eweek.com/cloud/marc-benioff-trend-seer-and-business-socialist/) of Salesforce founder/CEO Marc Benioff--the only time he has entered the competition. Previously, Chris was a founding editor of both IT Manager's Journal and DevX.com and was managing editor of Software Development magazine. He has been a stringer for the Associated Press since 1983 and resides in Silicon Valley.
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.

      MOST POPULAR ARTICLES

      Latest News

      Zeus Kerravala on Networking: Multicloud, 5G, and...

      James Maguire - December 16, 2022 0
      I spoke with Zeus Kerravala, industry analyst at ZK Research, about the rapid changes in enterprise networking, as tech advances and digital transformation prompt...
      Read more
      Applications

      Datadog President Amit Agarwal on Trends in...

      James Maguire - November 11, 2022 0
      I spoke with Amit Agarwal, President of Datadog, about infrastructure observability, from current trends to key challenges to the future of this rapidly growing...
      Read more
      IT Management

      Intuit’s Nhung Ho on AI for the...

      James Maguire - May 13, 2022 0
      I spoke with Nhung Ho, Vice President of AI at Intuit, about adoption of AI in the small and medium-sized business market, and how...
      Read more
      Applications

      Kyndryl’s Nicolas Sekkaki on Handling AI and...

      James Maguire - November 9, 2022 0
      I spoke with Nicolas Sekkaki, Group Practice Leader for Applications, Data and AI at Kyndryl, about how companies can boost both their AI and...
      Read more
      Cloud

      IGEL CEO Jed Ayres on Edge and...

      James Maguire - June 14, 2022 0
      I spoke with Jed Ayres, CEO of IGEL, about the endpoint sector, and an open source OS for the cloud; we also spoke about...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2022 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×