Close
  • Latest News
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Logo
  • Latest News
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Applications
    • Applications

    Too Much Data?

    By
    Peter Coffee
    -
    April 19, 2004
    Share
    Facebook
    Twitter
    Linkedin

      More information enables better decisions. Isnt that the premise behind almost every IT pitch? Yet adding data can actually worsen the decision-making process.

      Understanding how this can happen and knowing how to avoid it is the kind of thing that makes someone a real CIO—instead of just being a head of IT with a CxO title on the door.

      Small and midsize businesses are especially vulnerable to the more-IT-is-better mistake. SMBs already represent more than half of U.S. IT spending, according to a study last month by IDC, and their IT budgets are in the cross hairs of most vendors marketing campaigns.

      But before a small company bulks up on IT, it needs to understand the paradox of adding data while subtracting information value.

      The error is easy to see in the simplest cases. If you have two data values, such as sales volume at two different times, it may be useful to calculate their average and use it as an estimate of future performance. But if someone also notes the dates on which those sales were measured, fits a line to the two resulting points on a graph and proudly reports a perfect fit, the resulting trend has no statistical significance: Any two points will exactly determine a line, even if the actual behavior of the system is just a random fluctuation around some mean.

      Adding data has turned an average that meant something into a “trend line” that could point anywhere.

      You may think that Im being too conservative about the real-world use of statistics. If your sales last year were $5 million and your sales this year were $10 million, am I saying that its wrong to predict that next years sales will be $15 million?

      Well, yes, I am, and Im not talking about trivial noise in that prediction. Im talking about the kind of error that can kill a company.

      The actual behavior, out there in that real world, could be trial buys by customers who like the product and begin to buy more while also telling their friends. If I could determine that this was taking place, I might hope that my sales volumes happened to be on an exponential curve—at least, in the short run—so that next years potential sales could be $20 million.

      If I wrongly predict a smaller value and cant meet that higher level of demand, my error will be a self-fulfilling prophecy—and Ill have a third point on my misleading trend line, encouraging me to make the same mistake the following year. Ill also be leaving unfulfilled demand that competitors can exploit to enter the market.

      Alternatively, the actual behavior behind those first two years sales might be first-time buyers hating the product and swearing theyll never try it again. My sales growth from $5 million to $10 million, in that case, might be merely the result of advertising briefly outpacing bad word of mouth.

      If my advertising budget reaches the same number of people next year, but there are three times as many unhappy customers loudly offsetting that message, my sales next year could plummet back down to $5 million or even less—and Ill never know what hit me.

      These examples may seem too simple, but errors like these may well be buried under the deceptive complexity of an elaborate CRM or enterprise-forecasting project. These systems can generate lots of numbers and may give me the illusion of understanding more than I did before I had them.

      The process is seductive because we want to understand whats going on, and throwing additional measures into the pot seems to make our models more certain. Not sometimes, not usually but almost always, since modeling algorithms use added variables in ways that improve the overall fit, unless theyre so perfectly random in behavior that the algorithm ignores them completely. Thats an unlikely result. Adding variables will improve most measures of fit but often without boosting actual predictive power.

      Being an SMB means being close to your customer. Dont blow it by building a wall of numbers that merely blocks your view.

      Technology Editor Peter Coffee can be reached at peter_coffee@ziffdavis.com.

      Peter Coffee
      Peter Coffee is Director of Platform Research at salesforce.com, where he serves as a liaison with the developer community to define the opportunity and clarify developers' technical requirements on the company's evolving Apex Platform. Peter previously spent 18 years with eWEEK (formerly PC Week), the national news magazine of enterprise technology practice, where he reviewed software development tools and methods and wrote regular columns on emerging technologies and professional community issues.Before he began writing full-time in 1989, Peter spent eleven years in technical and management positions at Exxon and The Aerospace Corporation, including management of the latter company's first desktop computing planning team and applied research in applications of artificial intelligence techniques. He holds an engineering degree from MIT and an MBA from Pepperdine University, he has held teaching appointments in computer science, business analytics and information systems management at Pepperdine, UCLA, and Chapman College.

      MOST POPULAR ARTICLES

      Big Data and Analytics

      Alteryx’s Suresh Vittal on the Democratization of...

      James Maguire - May 31, 2022 0
      I spoke with Suresh Vittal, Chief Product Officer at Alteryx, about the industry mega-shift toward making data analytics tools accessible to a company’s complete...
      Read more
      Cybersecurity

      Visa’s Michael Jabbara on Cybersecurity and Digital...

      James Maguire - May 17, 2022 0
      I spoke with Michael Jabbara, VP and Global Head of Fraud Services at Visa, about the cybersecurity technology used to ensure the safe transfer...
      Read more
      Applications

      Cisco’s Thimaya Subaiya on Customer Experience in...

      James Maguire - May 10, 2022 0
      I spoke with Thimaya Subaiya, SVP and GM of Global Customer Experience at Cisco, about the factors that create good customer experience – and...
      Read more
      Big Data and Analytics

      GoodData CEO Roman Stanek on Business Intelligence...

      James Maguire - May 4, 2022 0
      I spoke with Roman Stanek, CEO of GoodData, about business intelligence, data as a service, and the frustration that many executives have with data...
      Read more
      Cloud

      Yotascale CEO Asim Razzaq on Controlling Multicloud...

      James Maguire - May 5, 2022 0
      Asim Razzaq, CEO of Yotascale, provides guidance on understanding—and containing—the complex cost structure of multicloud computing. Among the topics we covered:  As you survey the...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2021 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×