You are likely to be a member of your organizations stealth IT group whether or not you are aware of it. If you have a desktop or laptop PC and a way to extract data from other sources and populate a Microsoft Excel spreadsheet, then chances are you belong to this less than exclusive club.
Now, the stealth IT organization doesnt care about building a sustainable infrastructure. It doesnt care about metadata management or data security. It cares about getting the job done. And why not? That is why we get the big bucks, is it not?
Of course, in this highly regulated environment we live in today, little things like data accuracy and data security tend to be important. Especially if we dont want to find our companys name spread across the front page of a newspaper in a most unflattering manner.
That can happen either because we reported poor results based on poor decisions or because sensitive data found its way into the wrong hands.
And that is only the tip of the proverbial iceberg with respect to the hidden costs of our stealth IT organization. Lets not forget the time wasted by highly paid individuals, wrangling data and massaging it manually for use in their spreadsheets.
Or the fact that spreadsheets are simultaneously the greatest software invention known to mankind and perhaps the most destructive if used incorrectly.
Lets face it, every organization uses spreadsheets incorrectly. Our stealth IT organization has stretched the boundaries of spreadsheet usage well beyond those of a mere desktop productivity tool. Spreadsheets have become ingrained within business processes to the point where they have been institutionalized.
The good thing about spreadsheets is they are so powerful yet easy to use. But those very attributes make them dangerous to the long-term information needs of the organization.
Any user can, within a short period of time, take some data, create some calculations, use a formatting wizard and produce some useful pieces of information. People with real knowledge of the almost limitless power of spreadsheets can create amazing quasi-applications that dont even look like spreadsheets.
Why is that so bad?
There has always been the balancing act between the stealth IT organizations short-term goals and the official IT organizations focus on long-term goals.
The very nature of desktop productivity tools is anathema to the shared, secure, accurate and auditable nature of the infrastructure that the official IT organization strives for.
Now Microsoft has announced that in its upcoming release of Office 12, Microsoft Excel, the most popular end-user tool on the planet, will have a server component. Now that is pretty exciting.
If you are in the stealth IT organization you should all be cheering, because there is hope! Of course most of the members of the stealth IT organization were not aware that they needed help. So, everyone in the official IT organization, stand up and cheer! Now, just the database administrators! Ok, now the compliance officers!
A server-based solution to BI development and productivity is long since overdue for any organization that is trying to become more analytically mature. The database is the key piece of infrastructure here, but its not enough.
End users will require the same sense of autonomy from the official IT organization. You know, the one that holds them back from getting their jobs done (so they tell me anyway). At the same time, the long-term issues of data accuracy/timeliness, security and maintainability cannot be ignored.
So how do we provide a tool that delivers both autonomy and innovation to end users while maintaining a single version of the truth, near-real-time loading and more?
The list of issues inherent in trying to make an individual productivity tool scale would be too long for this column.
One example of the problems with the heretofore disconnected spreadsheet world was the lack of control over how a metric is calculated.
Furthermore, spreadsheets are positional (A22:A54) in access versus the relative (Select Name from Customer…) access offered by a database, making them much more difficult to maintain.
In addition, unlike actual application code, which can be stepped through logically, spreadsheets, especially complex ones, are much more difficult to follow, let alone maintain the logic driving the results.
Will slapping Microsoft Excel on top of SQL Server address that issue?
Admittedly, I do not have the details yet as to how, specifically, the server edition of Microsoft Excel will work in Office 12.
I dont know if Microsoft Excel functions and formulas will be converted into SQL or T-SQL, or if that logic is maintained separately.
What I do know is that the real IT organizations better start looking at this seriously.
If done correctly, this could hold the answers to dramatically improving organizations overall data management processes. But it will take some close scrutiny.
I am, shall we say, skeptical of Microsofts commitment to improving the state of the analytic infrastructure.
Not because the company doesnt want to, but simply because it is difficult for a product or company that has focused on individual productivity to reposition itself as a shared collaborative tool.
I fear that old habits will be difficult to overcome, but overcome them we must. Somehow a balance must be struck between allowing users to use and manage data as they see fit, and providing a solid, scalable, secure and auditable infrastructure on which to do that.
It will take planning. After all, we have had BI tools for years that could sit on top of a relational database and provide all the power and ease of use that Microsoft Excel has. Of course, they were not pervasive enough on users desktops to make a difference.
It will be up to the organization itself to address the costs and risks inherent in the very existence of the stealth IT organization.
We have to start by recognizing that any information that is part of a process that is beyond the scope of a single individual is probably not a job for which a tool like Microsoft Excel should be used.
Perhaps the server edition will address that, but there will be no long term-solution until the front-end BI tool of choice is accessing a single source of corporate truth, an enterprise data warehouse.
More importantly, this should be an active enterprise data warehouse that is updated where appropriate on a real-time or near real-time basis. Only then do we begin to address the issues of accuracy, security and availability.
As for what tool should be used, a Web-based version of Microsoft Excel does sound promising, but it better sit on top of more than just SQL Server.
It would need to support Teradata, DB2 and others that are more likely to be the infrastructure on which the data warehouse is based.
For users of Oracle 10g, I would check out HTML DB, a Web-based declarative development tool that comes bundled with every edition Oracle offers.
Its easy enough for the average user to pick up, but can be leveraged by heavy duty development types as well, and its Web-based approach makes it instantly ubiquitous.
I would also consider existing reporting tools such as Cognos Inc.s or Business Objects products as well. Given some time, I think end users will find there are other tools that can offer the same power as the desktop tools but with added benefits like scalability and maintainability.
In the end, however, its about planning for proper analytic infrastructure. We want users to have autonomy. We need them to create and innovate. But autonomy alone can actually hurt an organizations ability to adapt because our processes are based on fragile, unsustainable infrastructure.
Now is the time to open the dialogue between stealth IT and official IT and solve this problem.
Charles Garry is an independent industry analyst based in Simsbury, Conn. He is a former vice president with META Groups Technology Research Services. He can be reached at email@example.com.