How to Achieve a Near Real-Time Data Warehouse

Data warehouses are no longer simply repositories of business data. In particular, the use of business intelligence applications, which rely heavily on data warehouses for their operation, is expanding exponentially. With companies of all sizes now using BI applications on a daily basis, near-real-time data warehouse updates are now essential. Here, Knowledge Center contributor Ben Rosenberg describes how to make use of job scheduling software to achieve a nearly real-time data warehouse.


Data warehouses are no longer simply the attics of the business computing world. The use of business intelligence applications is growing exponentially, making such repositories an essential part of daily business life.

Compounding the need is the democratization of BI applications. Employees at all levels-from sales to production to human resources-are using BI applications on a daily basis. Add in the new forms of data that are flooding in, from radio-frequency identification (RFID) readers, Web services applications, mobile devices, cloud-based sources and more, along with the fact that data latency requirements have dropped (below one hour in 60 percent of cases and under one minute in another 35 percent) and it's clear just how important near real-time data warehouse updates have become.

It's estimated that up to 90 percent of all Global 2000 companies have now established a link between their data warehouse and at least one mission-critical application used to produce revenues or control costs. That's a jump from just 25 percent in 2007. Without current and accurate information, it's impossible to produce actionable insights at the pace of modern business.

Many tools exist to automate the processes used to update data stores but the problem is that most have significant limitations. For example, a DBMS has job-scheduling capabilities (as do even major operating systems including Unix and Windows). But DBMS options typically focus on data maintenance only. And, while operating system tools are convenient, their workflows are limited to tasks occurring within a particular server or operating environment.