By eweek  |  Posted 2003-11-07 Print this article Print

: Subjective Measurement"> CIO Insight: Im hearing that everyones emphasizing the importance of alignment. Lets look at a numbers themselves. You were telling me about the ROI numbers that themselves are often subjective and even illusory. Could you talk more about that?
Asiff Hirji: I think that theres a vested interest in pushing ROI from many areas, because it gives decisions which are by their nature difficult to substantiate, and difficult to get a group of people to align on, the look and feel of specificity without, frankly, very much specificity in many cases. If you look at the way ROI typically came across, anytime you have a set of inputs-this is back to Denises point-that are subjective in their nature, the measurement is, even though its mathematical, still a subjective measurement. So if you have any sort of terminal value, if you have any sort of any long-term profit stream thats baked into there, you will create something that has this sort of taste and smell of precision without any of the actual benefit of precision. Anyone [who] comes to the table and says, yeah, this is a big number, its bigger than that number, definitely lets go for this, [is] deluding themselves. [The] objective here is how to make better resource allocation decisions. Whatever works for your firm, you have to find a way of matching the technology decision-making to that process. If your firm throws darts at a board, great, keep doing that for technology because thats part of how you make the technology decision-making in line with the decision-making of the business. And I think, again to the point that Tom was making, that better companies will have a very tight alignment. Theres no such thing as a technology prioritization. Its business prioritization across various business lines, that flows to a set of projects, which flows to a set of initiatives. The way we do it at Ameritrade, every six months we revisit the macro level allocation across all the businesses. Thats not just technology, but marketing, HR, the whole bit.

CIO Insight: Chris, do you accept that its impossible to have objective ROI measures?
Christopher Gardner: No, having written a book on the topic, we have strong opinions about this, and everything thats been said today is absolutely what weve encountered. Its a very hard problem to solve. There are problems of isolation, alignment, the economics of companies vary, so the ROI definition varies from company to company. In fact, ROI in my mind is more of a proxy for a broader issue which is what is the value of IT, and how you measure that can be done in quite a variety of ways, some of which follow financial guidelines, [and] some of which are internal indices that are more meant to be something that correlates with value.
Theres a bunch of additional problems with ROI. One problem is that the inputs tend to get done poor, that there isnt enough time assessing demand in a rigorous way. There isnt enough time looking at alternative designs and assessing those. The economics of the business are not assessed and analyzed so that the IT systems effects on the economics of that business are clear. I think also, aside from kind of the hard work in getting the inputs right, the numbers can be dangerous. If theyre not done properly, usually theyre wrong, and then people get held accountable to numbers that are wrong. And so you have this situation where, frankly, I think theres an obligation to get the numbers right if youre going to hold people accountable, and the fact that the numbers can be dangerous is another issue here. But the inputs are the most critical thing to get by here.

Julie A. Schwartz: Im listening to the conversation, and it strikes me that there still might be too much emphasis on the numbers, the ROI tool and calculation itself, and not enough on the process. Chris, I think youre absolutely right about the fact that sometimes the numbers can be wrong, and as part of the process whats missing is this test for reasonableness. Im on the panel representing the vendor point of view, and one of the things that the vendors can be very helpful with, by working with their clients and their customers, is getting benchmark data together so they can build a database to help companies who are struggling with this problem to look at what is possible, what is reasonable. The other part of the process thats always missing is the post-implementation evaluation, going back and saying, okay, how were our numbers? Did we reach what we were going to achieve? What we found is that [in] very few, you know, maybe 20, 30 [percent] of the projects, [do] people go back and say, well, what did we do with ROI based on what our inputs were and what we got out? The problem is, were not learning. Theres no learning from the experience, so were not improving the process in any way, shape or form.

Gardner: Its actually borne out in your CIO Insight survey. ROI analysis is done at the very beginning of a project, but after that, theres very little effort to follow through.

Cameron: Let me give you just another data point, some data we at Forester gathered at the start of this year. The numbers bear out very clearly what Julie just quoted. We saw on average only 37 percent of projects were measured postproduction, but the interesting thing is that when we took that sample [of] senior IT decision-makers at 174 $1 billion plus North American firms, and broke it into quartiles-in other words, those who did the most down to those who did the least measuring postproduction-what we found is that the top quartile measure 86 percent of their projects postproduction, 45 percent the second quarter, third quarter 13 and the bottom 1 percent. So my point is that those who learn how to measure and learn how to do it well do probably as much as they possibly can.

CIO Insight: I want to go back to Atti. You were talking before about vendors. We heard from Julie how they can help and what they can do. But do you feel that, in reality, theyre helping as much as they could or that theyre actually part of the problem?
Riazi: I think some of the way theyve sold IT has been the ROI concept, and I think they are part of the problem in many ways because they never talk about the real issues. They come in, they sell you a concept that just looks terrific and wonderful, and they promise the ROI without sharing the issues related to change in the organization with the CFOs. The CIO becomes the therapist or the catalyst to make that change happen. We are here discussing a very irrational, illogical issue in a very logical way. And thats the disconnect I feel, thats the lack of alignment. We are talking about people, and when you implement technology, you change the way they work. And that is very irrational. The reason ROI fails is because we forget these are people who are going to worry about their jobs and their training and experience, and whats going to happen to me. I havent seen any research data or statistics on what happens there. Because the analysis and the statistics, all of them, are really a logical view: The process was very long, we collapsed it; the hierarchy was high and we collapsed it. Meanwhile, hundreds of people lost jobs, but thats irrelevant; thats not a logical argument. So I think between the vendors and us CIOs there is a responsibility to look at the failure, and the failure is not a logical discussion. It is a discussion of what happens in an organization when that technology changes the way that organization works, and its a lot of soft issues that happen. The biggest thing is whats going to happen to me, and that is going to stop your ROI. Its going to stop the return because [people] are not going to want to change. But the vendors have underestimated the importance of that. They too have approached [technology changes] in a logical way. You put this ERP and its going to be great and its going to be less head count and more efficient and better-you put the CRM in and youre going to automatically increase customers. And its not real.

Next Page: Vendors and customers need to work together on ROI.


Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel