iLuminate 4.0 Overcomes Data Warehouse Hurdles - Page 3

Download the authoritative guide: Big Data: Mining Data for Revenue

Of course, before I could begin exploring my campaign data, I had to load it into the iLuminate engine, a task for which Illuminate offers two different tool options: the iLuminate Importer and iLook&Load.

With each tool, I identified my data source, the sort of delimiter that marked off my records, the data types of each column, the presence or absence of a header row, and the table into which my data should be loaded. I could also apply rules to modify the data as it was loaded into the repository.

The Importer tool worked more quickly, and iLook&Load tool provided more information about my data as it entered the system, such as the totals and averages of integer-typed columns, as well as the maximum and minimum values of individual columns. iLook&Load also accepts a broader range of input types, including ODBC sources, text and XML files, Access databases, and Excel spreadsheets. The stock importer accepts text files and ODBC sources.

I loaded my largest test table-the 15-million-record, CSV-formatted individual contributions table-with each tool. The Importer took about 1.5 hours to complete the load, and iLook&Load required about 5.5 hours to do the job.

Before I was able to load the data into my repository at all, I had to do some cleanup, such as dealing with nested quotation marks within some of my fields. Contributors who took pains to include their nicknames ("Tex") within their name entries gave me fits, for instance.

As with any other data warehouse project, organizations that select iLuminate will need to use an ETL (extraction, transformation and loading) tool to first clean up their data. For the purposes of my tests, I focused primarily on getting all of my records to load, but actual deployments will demand more attention to data cleansing.

iLuminate runs as a pair of Windows services-one for the core engine and another for its ODBC support. I could start, stop, create and switch among database files using a service monitor tool that would minimize to my system tray. The monitor also provided performance details, as well as information about open queries, active connections and storage characteristics.

Also from the service monitor, I was able to enable remote access to the engine for other iCorrelate clients or ODBC consumers on my network.

Executive Editor Jason Brooks can be reached at