Columnar Databases Gain Ground in Enterprises

Columnar database may be gaining steam in the data warehouse, but makers of row-based databases don’t need to lose sleep over it, analysts said.

A column-oriented database in some circumstances can improve performance in large data warehouses. Industry analyst Jasmine Noel gave the example of a database that has a table with 20 columns and millions of rows. If a database administrator does a query the database isn’t indexed for, the query can target a particular column to get the answer instead of sorting through every single record, speeding the process, Noel noted.

Data warehouses are optimized for OLAP (online analytical processing), which involves fast loading and query of large structured data sets, and for which columnar databases are often well-suited. The market for column-based databases is still relatively immature, populated by vendors such as Vertica Systems, Sybase with Sybase IQ, and ParAccel.

In an October report on the enterprise data warehousing market, Gartner analysts reported column-oriented DBMS engines have shown superior performance in analytic applications over the more traditional row-based database management systems.

“However, when used in a complex query workload—for example, with many columns returned in a SELECT (that is, complete rows containing all the columns) with complex joins (that is, greater than 10 to tables in a single SELECT)—this can result in column-based DBMSs performing no better, and on occasion worse, than a row-based DBMS,” the report noted.

Though gaining ground in terms of customer wins and deployments for data warehousing, columnar database vendors have not yet penetrated the DW (data warehouse) market, which is dominated by companies such as Teradata, IBM and Oracle, said Forrester Research analyst James Kobielus. Those companies base their enterprise DW offerings on relational databases tuned and scaled for OLAP through various approaches involving indexes, partitioning, clustering, grid computing and fast interconnects, he explained.

“So it’s not a situation where columnar databases are revolutionizing the enterprise DW market, though they do offer a promising approach for accelerating OLAP query-intensive workloads for domain-specific data marts and other tactical DW deployments,” Kobielus said.

To Noel, the database industry has reached a point where it can have different database structures living side by side in the market.

“I was just talking to a DBA who now has SQL programmers querying IDMS through JDBC/ODBC drivers the same way as they do their Oracle databases. But with specific types of queries IDMS performance is so much better,” she said. “So I see these column databases in the same way—a structure that can give you better performance in specific circumstances—and you don’t need to learn a new query language to take advantage of it.”