Close
  • Latest News
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Logo
  • Latest News
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Latest News
    • Storage

    The Sushi Principle: Why Data Is Better Raw Than ‘Cooked’

    By
    Chris Preimesberger
    -
    October 6, 2015
    Share
    Facebook
    Twitter
    Linkedin

      PrevNext

      1The Sushi Principle: Why Data Is Better Raw Than ‘Cooked’

      1 - The Sushi Principle: Why Data Is Better Raw Than 'Cooked'

      Organizations often need to use data in a different way than originally planned. For flexibility and accuracy, they need fast access to data in its raw state.

      2How Systems Are Currently Set Up

      2 - How Systems Are Currently Set Up

      Most data systems are made up of three systems: ETL (extract, transform and load) system, databases and a business intelligence layer. Here is how most companies build IT systems: They buy the three systems noted above, hire a database architect, a database administrator, a project manager and a dozen engineers. They then decide on data schema(s) and wait months while the engineers put everything together. Finally, they spend years finding bugs and fixing errors. Efficient? Not very.

      3Where Data Comes From and Where It’s Stored

      3 - Where Data Comes From and Where It's Stored

      Raw data comes from servers, production databases, the Internet of things sensors and devices, and other periodic measuring or reporting devices. Most businesses store this data in data warehouses, data marts or so-called data lakes (a huge central pool of data). The classic data warehouse splits data into facts about the world, dimensions that describe the facts, and aggregates or metadata that summarize the facts.

      4Data Warehouses Contain ‘Cooked’ Data

      4 - Data Warehouses Contain 'Cooked' Data

      Before they store data, businesses “cook” it by cleaning, compressing, de-duplicating, normalizing, filtering and summarizing it, all in the name of optimizing access to the data. Many businesses move normalized information from fact tables to dimension table indexes; they build bitmap and B-tree indices—often many per table. This leads to pre-computed aggregates and building of online analytical processing (OLAP) cubes, materialized views and other aggregates.

      5So Why Do We Cook Data?

      5 - So Why Do We Cook Data?

      Most cooking of data is actually an application-level optimization. Normalizing data, building indexes and pre-computing aggregates make data warehouses fast and compact, but they also make data warehouses error-prone, difficult to maintain and difficult to understand. With today’s distributed architectures and faster processors, it’s now more efficient—and accurate—to pull information from raw data than it is to depend on summary tables that pull from many different systems and apply obscure logic.

      6The Sushi Principle, Step 1: Don’t Pre-Process

      6 - The Sushi Principle, Step 1: Don't Pre-Process

      Don’t pre-process data when you pull it in, store it or query it. Use a well-tested pipeline and keep it simple. Resist the temptation to add business logic, or you’ll end up with concerns about accuracy, won’t be able to get back to the original data source, and are likely to break the pipeline and potentially lose data.

      7The Sushi Principle, Step 2: Store All Data

      7 - The Sushi Principle, Step 2: Store All Data

      Store all your data with all its raw details, but partitioned and sorted for fast analysis. They can be in different silos, but a centralized management system is a must.

      8The Sushi Principle, Step 3: Summarize and Sample

      8 - The Sushi Principle, Step 3: Summarize and Sample

      Summarize and sample at query time for direct access to complete, accurate and fresh data. This guarantees that you will have the most up-to-date information possible.

      9Use Cases: Facebook and LinkedIn Were Early Adopters

      9 - Use Cases: Facebook and LinkedIn Were Early Adopters

      At first, these two social networking giants built data lakes before figuring out how to make the data ready to use. They were among the first companies to realize that raw data had more potential value than data that had been cooked. They pulled in data with as little pre-processing as possible into data lakes. They then developed processes that analyzed raw data in response to queries, delivering fresh, customized, “ready-to-eat” information instead of just pulling up pre-built, black-box summaries and aggregates.

      PrevNext

      MOST POPULAR ARTICLES

      Big Data and Analytics

      Alteryx’s Suresh Vittal on the Democratization of...

      James Maguire - May 31, 2022 0
      I spoke with Suresh Vittal, Chief Product Officer at Alteryx, about the industry mega-shift toward making data analytics tools accessible to a company’s complete...
      Read more
      Cybersecurity

      Visa’s Michael Jabbara on Cybersecurity and Digital...

      James Maguire - May 17, 2022 0
      I spoke with Michael Jabbara, VP and Global Head of Fraud Services at Visa, about the cybersecurity technology used to ensure the safe transfer...
      Read more
      Applications

      Cisco’s Thimaya Subaiya on Customer Experience in...

      James Maguire - May 10, 2022 0
      I spoke with Thimaya Subaiya, SVP and GM of Global Customer Experience at Cisco, about the factors that create good customer experience – and...
      Read more
      Big Data and Analytics

      GoodData CEO Roman Stanek on Business Intelligence...

      James Maguire - May 4, 2022 0
      I spoke with Roman Stanek, CEO of GoodData, about business intelligence, data as a service, and the frustration that many executives have with data...
      Read more
      Cloud

      Yotascale CEO Asim Razzaq on Controlling Multicloud...

      James Maguire - May 5, 2022 0
      Asim Razzaq, CEO of Yotascale, provides guidance on understanding—and containing—the complex cost structure of multicloud computing. Among the topics we covered:  As you survey the...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2022 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×