The following is a transcript from a mediated, ROI discussion in August 2003 by CIO Insight.
Participants:
- Joseph Barkley, Vice President and CFO for Global Information Technology, American International Group Inc. (AIG)
- Bobby Cameron, Principal, Forrester Research Inc.
- Denise K. Fletcher , Managing Director, FA Group
- Christopher Gardner, Partner and Cofounder, iValue
- Asiff Hirji, Executive Vice President, CIO, Ameritrade Holding Corp.
- Pamela Cohen Kalafut, Director, Intangibles Practice, Cap Gemini Ernst & Young
- Thomas H. Murphy, CIO, Royal Caribbean Cruises Ltd.
- Atefeh(Atti) Riazi, Senior Partner and CIO, Ogilvy & Mather Worldwide
- Julie A. Schwartz, Vice President of Research, Information Technology Services Marketing Association
CIO Insight: Let me start with the first question: What is ROI? What does it mean? Does everyone agree on what it means? Atti, could you start us off on that? How do people usually define ROI?
Atefeh(Atti) Riazi: It depends who defines it. Mostly, the CFOs define what return on investment is, and its defined from the financial terms. I think a lot of what we do in IT has changed and has evolved. For example, you cannot really use ROI in the way we know it in knowledge management. There are so many areas where you cannot apply the financial point of view to it because the return is not necessarily financial.
So whos defining it, why is it being defined? I personally, and Ive shared this with you, I would say most CIOs are so fed up with the vendors who come in and sell this virtual vaporware with this great return on investment, which is nonexistent. Youre kind of pushed to get to the CFOs office and say here, yes, there is an ROI. And I refuse to do it.
CIO Insight: Well get back to talking about the vendors later. Joe, youve been a finance executive, working with IT organizations for many years. Is there a clear definition of ROI?
Joseph Barkley: The answer is yes and no. ROI is an accounting indicator that CFOs are comfortable with, because they know how to define return on investment, there are several mathematical formulas to get to that, and its a number.
If you look at the value of IT and tie it to a financial number, that analysis is generally doomed to frustration and failure. ROI is merely one indicator that you can use to rank the project. One of the problems we have with ROI is we cant find it after we finish the project, because the project and the effect are melded into the rest of the budget and the operation. We havent figured out how to go back and find it, verify all of the savings supposedly or all of the effect. Unless youve got a discrete business, by the way; then you can do it.
CIO Insight: But is there a specific definition of ROI within the finance?
Denise K. Fletcher: Yes, of course there is one. However, just like every other kind of modeling, the input is critical to the output. So if you start off your analysis [by] defining a project very narrowly, it is not likely to be representative of the investment.
Let me explain what Im saying. If you look at a project, for example, [and say] you need to add storage capacity, and you say its going to cost X number of dollars [and], Im going to get Y return on this project, to me the weakness is that you have to step back to be able to get to the right place. You have to step back and say: How does this fit into the strategy of the company? You cant separate an IT project any more than you can separate a marketing project from the overall strategy. If you step back and you look at the overall strategy, I think that you will find that just about every project has a purpose and a return.
To be very specific, I was CFO of MasterCard until a little while ago. One of the major investments that we decided to make was to invest in brand. Investing further in brand meant that we were obviously going to get a lot more response from customers. But to be able to deal with that, we needed to add a lot of storage capacity. Now, I can look at my investment in brand and include in that investment my storage capacity, which is going to give a very positive return, or I can look at it separately which is not likely to give me a very high return.
So I think the most critical thing is to go back to a word that [CIO Insight Editor-in-Chief] Ellen Pearlman used earlier: its the alignment, the alignment of the investment with the overall strategy of the company. I think ROI is only a tool, and the tool is only as good as the input that goes into it and as good as the boundaries that are defined around it.
Bobby Cameron: May I make a comment?
CIO Insight: Go ahead, Bobby.
Cameron: I think the series of comments point out two really critical factors. The last one I think is the most important, and that is there is no such thing as an IT investment, with the sole exception of some pure infrastructure cost-saving kinds of things. The justification or the rationalization has to be a complete view, meaning the business and the IT parts. There is no such thing as a standalone IT business investment. The second is that the mechanism, whatever it is, that goes to [the] justification of an investment has to be one that sits with the overall business operation, meaning if ROI or MPV or EVA or whatever is used for business investments, that should be whats used for IT. The flip side, which is frequently the case, is that IT is held to criteria that the rest of the business isnt held to. So, if IT is asked to do ROI and the rest of the firm isnt, then CIOs feel like theyve run against the wall.
CIO Insight: Id like to get back to that, Bobby, but let me get Tom Murphy on the phone. Tom, you told me that ROI has to be very closely connected to alignment as well. Could you comment?
Thomas H. Murphy: Well, I agree with Denise and Bobby, and our approach at Royal Caribbean is [that] we have a very clear business strategy that Ive actually coauthored, and we have then aligned the IT strategy to support all of those business elements. Weve laid out a $200 million investment called Jumpstart that has been approved at the board level with no ROI.
CIO Insight: What is Jumpstart?
Murphy: Jumpstart is our core business systems rewrite over the next three years. There are elements of employee, guest and supply chain to all of those, and we have component projects that make up that investment, and that is aligned exactly with the strategy that the business has put forward. In fact, the board has accepted that the business cannot achieve their strategies if we do not execute these IT projects that are really business projects.
But each of those projects, as they come up, are prioritized by the business and justified by the business for their value. In some cases those [justifications] are very ROI-driven, but in most cases theyre strategic or fit into this jumpstart model. They still have to go through an approval process, but ROI is not the sole measure. There are many measures of that and, again, it aligns very neatly with the strategy thats been laid out for the next three to five years by the business.
Next Page: ROI is a “subjective measurement.”
ROI
: Subjective Measurement”> CIO Insight: Im hearing that everyones emphasizing the importance of alignment. Lets look at a numbers themselves. You were telling me about the ROI numbers that themselves are often subjective and even illusory. Could you talk more about that?
Asiff Hirji: I think that theres a vested interest in pushing ROI from many areas, because it gives decisions which are by their nature difficult to substantiate, and difficult to get a group of people to align on, the look and feel of specificity without, frankly, very much specificity in many cases. If you look at the way ROI typically came across, anytime you have a set of inputs-this is back to Denises point-that are subjective in their nature, the measurement is, even though its mathematical, still a subjective measurement. So if you have any sort of terminal value, if you have any sort of any long-term profit stream thats baked into there, you will create something that has this sort of taste and smell of precision without any of the actual benefit of precision. Anyone [who] comes to the table and says, yeah, this is a big number, its bigger than that number, definitely lets go for this, [is] deluding themselves.
[The] objective here is how to make better resource allocation decisions. Whatever works for your firm, you have to find a way of matching the technology decision-making to that process. If your firm throws darts at a board, great, keep doing that for technology because thats part of how you make the technology decision-making in line with the decision-making of the business. And I think, again to the point that Tom was making, that better companies will have a very tight alignment. Theres no such thing as a technology prioritization. Its business prioritization across various business lines, that flows to a set of projects, which flows to a set of initiatives. The way we do it at Ameritrade, every six months we revisit the macro level allocation across all the businesses. Thats not just technology, but marketing, HR, the whole bit.
CIO Insight: Chris, do you accept that its impossible to have objective ROI measures?
Christopher Gardner: No, having written a book on the topic, we have strong opinions about this, and everything thats been said today is absolutely what weve encountered. Its a very hard problem to solve. There are problems of isolation, alignment, the economics of companies vary, so the ROI definition varies from company to company. In fact, ROI in my mind is more of a proxy for a broader issue which is what is the value of IT, and how you measure that can be done in quite a variety of ways, some of which follow financial guidelines, [and] some of which are internal indices that are more meant to be something that correlates with value.
Theres a bunch of additional problems with ROI. One problem is that the inputs tend to get done poor, that there isnt enough time assessing demand in a rigorous way. There isnt enough time looking at alternative designs and assessing those. The economics of the business are not assessed and analyzed so that the IT systems effects on the economics of that business are clear.
I think also, aside from kind of the hard work in getting the inputs right, the numbers can be dangerous. If theyre not done properly, usually theyre wrong, and then people get held accountable to numbers that are wrong. And so you have this situation where, frankly, I think theres an obligation to get the numbers right if youre going to hold people accountable, and the fact that the numbers can be dangerous is another issue here. But the inputs are the most critical thing to get by here.
Julie A. Schwartz: Im listening to the conversation, and it strikes me that there still might be too much emphasis on the numbers, the ROI tool and calculation itself, and not enough on the process. Chris, I think youre absolutely right about the fact that sometimes the numbers can be wrong, and as part of the process whats missing is this test for reasonableness. Im on the panel representing the vendor point of view, and one of the things that the vendors can be very helpful with, by working with their clients and their customers, is getting benchmark data together so they can build a database to help companies who are struggling with this problem to look at what is possible, what is reasonable.
The other part of the process thats always missing is the post-implementation evaluation, going back and saying, okay, how were our numbers? Did we reach what we were going to achieve? What we found is that [in] very few, you know, maybe 20, 30 [percent] of the projects, [do] people go back and say, well, what did we do with ROI based on what our inputs were and what we got out? The problem is, were not learning. Theres no learning from the experience, so were not improving the process in any way, shape or form.
Gardner: Its actually borne out in your CIO Insight survey. ROI analysis is done at the very beginning of a project, but after that, theres very little effort to follow through.
Cameron: Let me give you just another data point, some data we at Forester gathered at the start of this year. The numbers bear out very clearly what Julie just quoted. We saw on average only 37 percent of projects were measured postproduction, but the interesting thing is that when we took that sample [of] senior IT decision-makers at 174 $1 billion plus North American firms, and broke it into quartiles-in other words, those who did the most down to those who did the least measuring postproduction-what we found is that the top quartile measure 86 percent of their projects postproduction, 45 percent the second quarter, third quarter 13 and the bottom 1 percent. So my point is that those who learn how to measure and learn how to do it well do probably as much as they possibly can.
CIO Insight: I want to go back to Atti. You were talking before about vendors. We heard from Julie how they can help and what they can do. But do you feel that, in reality, theyre helping as much as they could or that theyre actually part of the problem?
Riazi: I think some of the way theyve sold IT has been the ROI concept, and I think they are part of the problem in many ways because they never talk about the real issues. They come in, they sell you a concept that just looks terrific and wonderful, and they promise the ROI without sharing the issues related to change in the organization with the CFOs. The CIO becomes the therapist or the catalyst to make that change happen.
We are here discussing a very irrational, illogical issue in a very logical way. And thats the disconnect I feel, thats the lack of alignment. We are talking about people, and when you implement technology, you change the way they work. And that is very irrational. The reason ROI fails is because we forget these are people who are going to worry about their jobs and their training and experience, and whats going to happen to me. I havent seen any research data or statistics on what happens there. Because the analysis and the statistics, all of them, are really a logical view: The process was very long, we collapsed it; the hierarchy was high and we collapsed it. Meanwhile, hundreds of people lost jobs, but thats irrelevant; thats not a logical argument.
So I think between the vendors and us CIOs there is a responsibility to look at the failure, and the failure is not a logical discussion. It is a discussion of what happens in an organization when that technology changes the way that organization works, and its a lot of soft issues that happen. The biggest thing is whats going to happen to me, and that is going to stop your ROI. Its going to stop the return because [people] are not going to want to change. But the vendors have underestimated the importance of that. They too have approached [technology changes] in a logical way. You put this ERP and its going to be great and its going to be less head count and more efficient and better-you put the CRM in and youre going to automatically increase customers. And its not real.
Next Page: Vendors and customers need to work together on ROI.
Vendors and Customers Together
CIO Insight: Julie, Id like you to respond, and then, Pam, this would be a wonderful moment to get you in to the conversation.
Schwartz: I think you make some very good points, but what vendors have learned is that with very few exceptions they are not able to get the kind of partnership relationship with their customers that they would like, so that they could walk hand-in-hand with the customers through the process you describe of really understanding the change management that goes on in the organizations where their solutions are implemented. In many cases we dont know how. [Vendors] are engineers, and theyre very analytical, and they dont know how to deal with that kind of ambiguity.
But I think its important to keep in mind that there are really two aspects of how vendors use the ROI. One is that theyre trying to make a compelling value proposition to get the CIOs attention. You know, you sit there, every week you get hundreds and hundreds of unsolicited e-mails and phone calls. And [vendors] need to find a way to get in so that they can tell you about their solutions. So one thing they are doing is trying to come up with the most compelling value proposition, and that might very well include ROI that looks like smoke and mirrors or vaporware to you.
The other reason they need to have ROI models is to help their clients and customers with an internal justification to build a business case. Some vendors are going to do this very well, and others are not going to do this very well at all. Theyre learning as well. But the most important thing is for the customers to really open [up] and say [to the vendors] well work with you so that we can collect the data, we can build the models, we can look at the assumptions, and do this in a consistent way so that you can do this with other customers, so that we can build a database so that we can track whats reasonable, so that we can start learning from our experience going forward so that we can make this a better tool. Its more of a process than an output.
Pamela Cohen Kalafut: I think that all makes sense. One of the things we get asked about the most is how this measure or others are going to help businesses get some sort of competitive advantage, which, as we all know, is almost impossible to sustain in the present economy, except through measures that might not always be recorded by businesses.
Fletcher: If I could just make a comment. It seems to me, if you start off with a premise that ROI is part of alignment and part of a corporate strategy, then its very difficult to expect the vendor to even have an understanding of what is going to drive the return on investment. For one thing, the discount rate that is used on the return on investment must reflect the capital costs of that company, not of the vendor. Thats the first weakness. Second, I dont know how a vendor could possibly understand all of those soft costs that Atti was talking about, the cultural costs. Theyre hard enough for internal corporate management to assess, let alone a vendor. I think the weakness, as I hear this conversation, of the approach is that when an outsider is looking at [ROI] theyre looking at a little box, but when you look at an ROI calculation to do it right, you have to look, as I said earlier, to the whole [set of costs], which really drives the responsibility of the ROI to internal management and not to the vendor.
Schwartz: The vendor is there to help in any way that they can, but they cannot do [the ROI calculations] for the customer. They can provide data: Have you thought of XYZ? Have you thought of ABC? Other customers we have worked with have done it this way. But they cant do it. In fact, we have done research with customers and asked who do you want to do your ROI calculations, and the majority say that theyre going to do it themselves, and a few are saying theyre going to do it with vendors helping them. But for the most part, theyre going to do it themselves.
Next Page: Trust at the C-level and in methodology.
Trust
CIO Insight: I have a question. Is ROI, in general, a symptom of a bigger kind of problem: a lack of trust between the business side and the IT side, and [of] being able to prove that a technology investment is going to return something to the company?
If there was better communication, perhaps there wouldnt be so much focus on that. Is it a reflection of that, or [of] something else going on in the enterprise?
Barkley: Id like to get into that first. To quote one of our former presidents of our company, hes always fearful of “the system that ate Chicago,” because systems tend to grow. So youre right, theres a fair mistrust. Theres a fair mistrust of the financial analysis capabilities of information technology people. Dont take that the wrong way. And theres a fair mistrust of the technical competence of the financial people, which is one of the reasons theres now people like me in our information technology [organization]. Because at the end of the day, the CIO and the CFO have to agree on the strategy, agree on the costs, and agree on what were going to deliver. But theres a fair amount of mistrust.
Murphy: I think the trust issue goes beyond IT and finance. Our finance group requires ROIs for any capital expenditure beyond $150,000. That includes building a new ship, which is a little bit like putting in a custom application. Its hard to nail it down. Its a physical asset, but it has a tendency to morph and grow and become more expensive as people have new, clever ideas, and a lot of it comes down to effective process and project management. I think the trust part comes in that once I, as CIO, have established that I have a methodology, and my team has process and discipline, for doing these things effectively, that changes the conversation. But I dont think thats any different than any other business unit that has significant capital requirements to do their business.
CIO Insight: Asiff, you had a comment?
Hirji: I think the sentiment in that question [about lack of trust] is right. I think IT professionals probably brought this upon themselves in some way. There was a study about how many major systems initiatives actually deliver, and the vast majority of systems fail. When I was a consultant, it was not uncommon to have projects of $200 million to half a billion dollars in size, the vast majority of which failed. Forget didnt deliver ROI, didnt get turned on. [Projects] got three-quarters of the way through implementation, and didnt get turned on. So it was easy to calculate ROI on that-zero. I think those days, thankfully, are dead, and that we collectively have learned a lot. But if you have a process by which your business allocates resources, checks progress on the projects, makes effective decisions about whether theyre proceeding or not proceeding properly, reallocates and delivers shareholder value or earnings per share or whatever it is youre aiming for, it doesnt matter if you use ROI or not, it doesnt matter what tool you use to do that, so long as the process is working and you as the management team are working effectively with it.
I think ROI is great for vendors, and I dont blame them for using that. Its an effective mechanism. But if youre going to use it, you better use it in context of the way your organization makes decisions and in alignment with the way your organization makes decisions. Its like code: Its only as good as the inputs. You can write the best code in the world, if you feed a bad input, youll get bad output.
So, theres too much emphasis on ROI as a measurement mechanism, and theres not enough on whats it about. Go back to that study that I think our friend from Forrester, Bobby Cameron, was quoting, he said top-tier companies do track ROI and they do track return on investment and so on. But I think CIO Insights study said that people have different definitions of ROI. So my takeaway from that is companies that set some goals in advance, measure the projects process, then measure against those goals what the project actually did, they learn and they get better. It doesnt matter whether you use ROI as part of that or something else as long as you are setting some goals, doing the project, checking it against the goals afterwards.
And in our experience at Ameritrade, the more you shorten the lifecycle of that, the better off you are. We will not do anything that is longer then 12 weeks. If we cant do it in 12 weeks, its dead. It doesnt get off the ground. Why? Because as bad as we all are at estimating, if were estimating from six to nine months out versus six to nine weeks out, well take the risk on six to nine weeks. Its just human nature.
Barkley: Let me just pick up. We worry about delivery. Once we approve the project, ROI [is] one of the indicators, but were more interested in the alignment and technology for our strategy. Were interested in the delivery of the project in the budget, and dont be late. We worry about those two things more than anything else. The other thing we started to do this year is every time we develop or design a new application or a new project, we put a metric in there so we know if its working. Let me give you an example. If you build an application thats going to do policy issuance and its supposed to be able to issue a million policies a month and you run it and it issues a policy, [people] say it works. Well, it doesnt work. It only issued one policy. I want to see it issue a million policies. Then I know it works. We have not in the past had operational metrics. When we install them, we started to improve our delivery in our continued operations, and the value, by the way.
Fletcher: Well, a couple of thoughts. I think that there was a comment made earlier about specificity and what goes into the equation is kind of a little bit random. Id like to answer that. In my view, ROI is a good tool as long as it is used consistently throughout the company for all kinds of different decisions, whether its marketing, HR or anything else. You cant just say IT goes the ROI way but marketing does its own spiel. That doesnt work.
Second, I think ROI is a very useful tool for allocating funds if you have a lot of demand for those funds, and you have to make a judgment call between one [demand] versus another. If you dont have that kind of situation and you have all the funds that you need in the world, I think free cash flow is a much better tool. It too, though, is subject to some of what we call lack of specificity or subjectivity because, lets face it, were all human, and were all going to do projections, and we all make the projections from the time in history of which we are living right now. If you ask people to make a projection now about revenue growth, theyre going to give you one kind of projection, which would be very different from anybodys projections three years ago. We never have the ability to forecast cycles. Its always a straight line from whatever point were in, or a hockey stick.
Where Im going [with that] is I think the most important thing is always a strategic driver. Why are we making this investment? Is it because we want to increase knowledge? Okay, thats great, we want to have certain kinds of efficiency. Lets not kid ourselves that we need to have a return on investment. But if I have only $100 million dollars and I have to decide whether Im going to go with Attis project or Im going to go with a marketing guys project, and they each want $80 million, I cant give them $160 million if Ive only got $100 million. So Ive got to make some kind of an allocation decision. But if I dont have that challenge, as long as its free cash flow positive, lets go do them both.
Next Page: Getting ROI through people.
ROI
Through People”> CIO Insight: Atti, I was fascinated by your comment on culture, essentially the soft issues. I wonder if you could elaborate a little bit about this. Im hearing about credibility problems, Im hearing different approaches to who has leadership here. We also hear a lot about rogue IT projects. Im just wondering what this is a symptom of. Is this a leadership problem? How do we fix this?
Riazi: I wish I knew the answer. Ive struggled with this issue the last 20 years being in IT and being a CIO. I think theres one reason they say the life of a CIO is three years, because the first year is infrastructure, the second year is begging for money to do ERP applications, and the third year is when you actually do it. And by the third year, you have the whole organization ready to shoot you. You learn two things as a CIO [to be] able to separate those who hate you from those who are undecided. (laughter)
This is a very complex issue, and I think, and Im hoping through panels like this and the research companies [to] start going to the forbidden zone. We all hate to do [it]. Were all analytical IT, finance people, and I think weve lost focus on the real issues that I have faced in every project I have rolled out, every large project. I was training some of the finance people, I was in the room where they were being trained on a wonderful application, and three of them sat back and said but thats my job. I was showing them how it was going to be automated, and the light bulb went off, thats my job. Well, that was the job of another 50 percent in the room, but the light bulb hadnt gone off yet.
How do you get ROI? You get it through people. I mean, the biggest cost is people, so thats the first way youre going to go. Even if you dont let people go, youre going to make them more efficient, theyre going to have to learn something new and there is the evaluation on that. I think we [have to] start looking at our failures, because our success has been abysmal. Our credibility is really shot. And thats why I said earlier I refuse to go to see my CFO with an ROI case because hes going to laugh at me and is going to kick me out of his office.
CIO Insight: So how do you survive? How does any CIO survive?
Riazi: I dont use ROI. I dont think CIOs should use ROI because the “R” isnt in my department; the “I” is in my department. The “R” is in the business which I have absolutely no control. So why should I use it?
Cameron: I disagree. I think you yourself pointed out an example earlier where the expense sits on both sides, where the user was sitting down and recognizing thats my job, the investment really is from them, too. They have to put time in; they have to change their process. Good technology has not nearly the impact [of] good process change, with or without technology to support it. So the bottom line, what Im hearing you say is, not that ROI fails or that the projects fail or theres maybe a bad odor of IT history, its that we in IT have sat alone and tried to do these things alone, and that really the cultural change is for us to become part of the business so that we relate to the rest of the firm like, lets say, a manufacturing plants general manager does to the rest of the firm.
CIO Insight: Is ROI politics? Is it possible to say that ROI is a technique that business people use to put the onus and responsibility for IT failure on IT people? You come up with an ROI, you prove it to us, and if doesnt make it, its your fault.
Murphy: I cannot imagine [such] a scenario where we would have a project go forward at Royal Caribbean, where I was developing the ROI. It isnt done. The business unit has to be accountable for the return on investment, except [for] a few real core infrastructure-types of initiatives, [and] even then I can flip back and show them how the value is going to go to the business. Now, you know, alignment is a big part, culture is a big part, trust is a big part. I have a personal relationship with every businessperson in this organization, and I am one of the businesspeople. I dont think people view me as that IT guy who doesnt deliver on projects. Im in a partnership with all of these folks trying to drive the business to success. Its a fascinating conversation, and theres a lot of angst out there obviously, but I think a lot of it is in the relationships that you develop within the business and the trust and the process that you put into place.
Barkley: We require a business case on every project over $1 million that comes to my desk. Before it gets to my desk, its signed off by the business CFO. Even after he signs it, Ill call him up and say: [Do] you understand what you signed? Do you understand how youre going to find a savings? Because Im not the guy whos going to look for them, you are. We require that. In our company, the application development is decentralized; that is, we have CIOs in each of the businesses that are matrixed to my boss, the global CIO and also to the business executive, although we control all of the infrastructure centrally. So the businesses have a big stake in the strategy, what were going to do, and the returns as well. Again, ROI is merely one of the indicators.
Fletcher: I was very concerned by a comment that Atti made which was that the return is at the unit, the “I” [in ROI] is with her. That would concern me because I think that that, to me, would be a little bit of a cultural disconnect between the IT organization and the rest of the organization. The most important thing to remember is that the “R” and the “I” are owned by the company.
So you need to get over the sort of silo thing, and you need to go, I think, to the models that both Joe and Tom are talking about, where you have a really strong alignment between the business units and IT, whether the IT is decentralized or centralized. But I think there are some pitfalls thinking that the “R” and the “I” are disconnected, because it all ends up in the shareholder value, and the shareholder value is the company as a whole. You can lose sight of that in the silos.
Gardner: I have to say I agree with everything thats been said here. Im not doing it because Im trying to form any consensus; I think everything here is true. I like Attis comment about people problems. I mean some of these soft benefits or soft costs, how do you get at those? There are ways to do that, and we actually borrow techniques from the advertising industry to get at some of these people issues, primary market research where its not a precise number but its a good enough number, and you can start to see what the dynamics are between some of the choices and some of the design alternatives.
Next Page: Measuring the value of IT and claiming management.
Measuring IT Value
CIO Insight: But how do you get at the people issues? I just want to make sure I understand what those people issues are and how they relate to ROI.
Gardner: Let me just make a couple of other comments. One is that I dont think that we can escape this question of how we measure the value of IT, because management will always have a problem with IT so long as you cant measure its value. If you cant measure the value of IT, my sense is-[and] this gets back to your point about is there trust between upper management and IT-if you cant measure the value of IT, how can you claim management, and the very top levels of the organization, are held accountable for shareholder value?
Cameron: Is there a value for IT? I would argue there isnt. Theres a value for business execution or delivery or value delivery, with IT as a component.
Gardner: Which is fine. Youve got to look at IT in the context of the company, [at] what IT is doing to contribute to the overall value of the company. Maybe its helping to create various competitive advantages. IT can be costly; that can create a capital barrier. There are skills involved, some of the people issues relate to IT skills, and that can give you can advantage. So, yeah, IT has to look at the broader context that it sits in, and youve got to understand how the company makes money so that the ROI [analysis] that youre performing ties to the way the company is actually making money. But I think there are ways to get at some of these things. I think theres also an issue around precision. [With] ROI, I think, sometimes theres a number that people are looking for [with] several decimal points. I remember seeing a vendor ROI analysis that came up with a number like 3,287 percent for some system, and thats just not the way to think about these things.
Initially, you go in, you might do a back-of-the-envelope type of calculation, and just see if it even makes sense to pursue further. And then over time, as things proceed and get more serious and maybe more money is involved, you make the effort to do the primary market research, to do the careful estimate and adjust for the cost of capital and risk. You know, its amazing to me that [in the] companies that weve served, theres a lot of talk and focus on ROI, the return part of the equation, but you need to look at risk and return together. Risk is completely left out of the equation. For those things that you cant quantify and are very uncertain and ambiguous, one can at least make an estimate of whats the risk. Is it a code red? Is it something that could affect the entire success of the project, or will it have a negligible effect? You can at least do that kind of thing.
Kalafut: I just wanted to get at the question in terms of the human side [of ROI], because the [human side of technology] is one of the things we talked about in [our book] Invisible Advantage and as part of our value creation index. We know there is no magical power in the technology itself. But in terms of the human side of technology, we have the work processes that go into it which are run by the people that actually have the power to create the strategies around it, and [who] motivate others to actually use that [technology] and decide that theyre not going to be replaced by those processes.
Also, in terms of work place organization, creating the jobs and responsibilities, the change in how the training comes into play to actually absorb people with new talents within the work place, has been one of our measures. Erik Brynjolfsson and his colleagues at MIT have studied companies investments in IT, and found that theyre almost invariably accompanied by much larger investments in intangibles, such as training in the workplace, and also by empowerment of employees as well. We go through a variety of examples, but all of it actually comes down to how you empower people to use those technologies rather than the technologies themselves.
CIO Insight: I just want to step back now. Weve talked now for close to an hour about what the problems and the issues with ROI. The pressure is there to come up with an ROI, but its still difficult to calculate. It seems to me that this is still something that we need to do, at least as part of the larger issue of value, particularly in the financial community. Does that sound right?
Barkley: Its slightly different. Maybe youre being overly focused on ROI. Heres why. We suddenly realized in looking at our budget process over the last two years [that] we spend 85 percent of our time on 30 percent of the budget. Projects are important, capital investment is important. But 68 percent of our money to is to keep the lights on and the business running. We hadnt done a good job at budgeting and analyzing that. So we had to figure out a way to do a better job at budgeting lights on, and aligning lights on with the strategic intentions of the company. So weve actually changed our budget process where we do the lights-on budget first, and then late in the budget process, after the businesses have done their budget, we get the project budget from the businesses, and we go through a strategic-alignment value analysis where we measure, among other things, internal rate of return, business value risk and dependency. We do that in the last few weeks of the budget [process]. While [the project budget] is important and we need to have it to justify the projects, weve got a large part of the business that we realize we hadnt been paying attention to, and we have to analyze that too.
Riazi: My problem is that I think we have sold technology to the business in isolation. Here it is, you put it in, and its going to be wonderful. I call it the diet pill: you take it, and you lose 50 pounds, and you dont have to worry about anything else. Thats the underlying problem.
Barkley: Do you know where that pill is? (laughter)
Riazi: I think technology has a great deal of ROI, its just our approach to it, our mind-set [that is a problem]. The fact is [technology] enables you to do so many different things. It enables you to streamline your operation, if thats how you approach it. [But] if you approach it that were going to dump this and its going to be great and its going to give us the ROI, its going to fail. So Im not questioning the value of ROI. I believe it is [valuable] and I believe we havent even scratched the surface. But there are other issues.
Next Page: Taking ROI value out of isolation for allocation decisions.
ROI
for Allocating Decisions”> CIO Insight: What Im hearing is we need to take ROI value out of isolation for the rest of the business. We need to make this part of a process. We need to use that process to build trust. And that the real purpose of ROI is to make allocation decisions; that ultimately is really what its about.
So, how do we transform our narrow, isolated approaches to ROI to something thats going to really get at the question of value, and is really going to help us make better decisions? Where do we start on that? Chris, can we learn something from Wall Street and how Wall Street looks at valuation of companies?
Gardner: Absolutely. My feeling is the top of the company is measured based on Wall Street style valuation models, and that should be reflected elsewhere in the company. Those [valuation models for IT] can be built, theyre complicated and theyre hard, but it can be done, and at least you have some assurance that your analysis ties back to something that is meaningful from a top executive standpoint.
CIO Insight: What would be the basic things needed to achieve that?
Gardner: Well, you need the free cash-flow approach that Denise was mentioning. Its ultimately about cash flow. Weve talked about ROI here, but my feeling is ROI is more of a general term that encompasses the concept of value for IT. The free cash-flow approach, theres strong evidence that shows it correlates with share price better than earnings per share and a bunch of other things. But this gets back to the inputs. If youve got a system that involves a lot of customer adoption, you need to understand the customers response to the design thats been proposed. Youve got to work out those costs in detail. You have to be complete about your costing. Youve got to make sure training is in there and change management, [and the] inefficiencies, the delays and that kind of thing. And some of these may not be known going into it, but you can see, well, what if were off here by 50 percent or 10 percent or 100 percent? Does it make any difference?
You can start to create something that you can use to manage the whole process of building a system. What were talking about isnt something you just use at the beginning of a project to decide whether to invest in it or not; its something of a tool that you use throughout the lifecycle of a project. So during the development you would know what are the get-rights. You know, if youve got a narrow market-opportunity window, schedules will be paramount. You can show that and model that. Then [theres] the operation of the system itself. This is really something that should be an operational tool and tie back to operational measures.
Its hard, and theres a lot of uncertainty here. The key is to always ask questions and challenge it. Credibility is low right now, but a tool like this should start to predict that some projects are not worth doing and some projects are worth doing. IT should start to be able to show that the business might be recommending projects that dont make sense. This is a politically difficult kind of thing, but you should start to see some projects being turned down. A test is: What am I seeing coming out of this? Does it calibrate with reality? Am I getting some responses that look halfway credible? Can someone show me the evidence, make the case, you know, sometimes you have to put it on the line.
CIO Insight: Bobby, what do you think? Is looking at IT investments kind of the way investors look at portfolios a helpful way to look and think about ROI? Is that something we all need to be doing as part of changing and transforming ROI?
Cameron: Let me make a couple of comments before I answer that directly. One is that it really is about the process. No matter what you do, if the process doesnt involve everybody fully, meaning business and IT equally, and the justifications dont cover everything, nothing works. So just put that on the table and push it aside. The second overall statement is that weve focused on ROI as if it were the be all and end all, but not everything either merits [justification] or can be justified, and so we need to make sure that we put some clarity there. Most firms that identify minimum pain around prioritization isolate strategic mandatory and operational investments into those categories where strategic justification is different from mandatory, tax regulatory, biggest-customer-tells-me-I-got-to [justifications], and neither of those is justified based on ROI.
ROI is really calculated for a portion of the investment stream that has to do with things that are justified based on changes to operations. In other words, I expect a business impact. And then there are firms who are committed to ROI who also justify projects based on softer metrics, EMC being one, for instance, where they use soft metrics for things that might improve customer satisfaction but would be very difficult to prove an ROI.
So now thats sort of laid the ground for what Im about to say, which is that most firms want to invest, as someone said earlier, to drive competitive advantage and improve stakeholder value. Frequently firms do their darndest to understand what the strategies are, what the directions are that they need to accomplish these goals. The problem is [that] theres no linkage between that planning and thinking and investments usually in the prioritization process. What Ive seen emerging is, in fact, a portfolio mechanism, meaning we look at everything as a group of investments and we compare them to each other, and the process attempts to adjust the prioritization [of] strategic needs.
CIO Insight: So what are the first steps then that we need to do to change and improve the way we do ROI?
Cameron: Well, ROI is best used when its a relative comparison tool. It has no absolute characterization. What I mean by that is if I want to adjust the direction of my spend to parallel what Im trying to accomplish, I need to put [in] some funds behind [them]. The best firms start by committing capital-investment dollars in specific directions around customer value and customer impact, around creating value, which is manufacturing or back-office systems, or enabling [systems]. By picking three buckets, I then immediately isolate some of the worst arguments between the plant-committed guys and the customer-committed guys who start throwing ROI around. By breaking them into buckets, you start to clean up the conversation. For instance, if I have a $100 million to invest in capital projects, I say 60 percent is going to customers, 30 to value creation and 10 to enabling. Right there Ive gotten rid of a lot of most absurd argument.
The second thing that then occurs is those buckets have specific business goals theyre trying to achieve-you know, improving employee productivity might be a good one in the value creation [bucket], improving customer satisfaction in the customer facing [bucket]. Then the third step is to prioritize projects based on their impact on those objectives, some of which are ROI and some of which are soft.
Kalafut: Can I just add to that a little bit? A value creation index which links measures like those you just mentioned back to things like market value, and getting estimates of how much a change of those will actually impact bottom-line performance, is another next step in the actual handling and treating those intangible drivers. Some of [those measures] are those that you mentioned, which are where we find three or four proxies to represent each of those drivers, and then link them back to market value so that they can be scored. [Theres] quite a difference [between] being able to actually put together several of those intangible drivers, particularly in this area of IT where so much of that is previously considered immeasurable.
CIO Insight: How do you generate trust that these drivers, these indicators actually work?
Kalafut: Well, this is a ten-year study of various organizations. We compiled various measures that tend to work in certain industries. We correlate them back to things like market value, [and] then watch them across time to see how those measures change. So its not a one point in time analysis that makes you decide that those measures are the correct ones, but actually watching them for several years. Once we were confident that those indicators were working and could be gathered for a variety of different industries and companies, we were able to go back and then start designing unique models, adding in additional characteristics for specific company types and build models around those, including various intangible drivers that might be measured in disparate parts of an organization but have never been pulled together into one model.
Next Page: ROI value as sharholder value.
ROI
as Stock Value”> CIO Insight: Denise, weve talked about Wall Street and portfolios, about separating out different kinds of value and analyzing them separately. Would this pass muster with you sitting in the CFO seat?
Fletcher: Well, I guess I start off back at what Christopher Gardner said earlier. I start off with shareholder value. That is the driver, nothing else is. Then the next step is: What drives shareholder value? The only thing that studies have shown actually truly correlate to shareholder value is free cash flow. Free cash flow is an element in the ROI analysis. It is, as I said earlier, subject to a lot of vagaries because youre doing projections. In my view, the whole key is not bucketing or anything like that. In my view, the key is to start your process, [as] Joe Barkley said earlier, with a strategy. Where are you taking the company? Once youve identified where youre taking the company, the next step is to identify the strategic drivers that are actually going to shift the company to where you want it to go, and where youre going to increase shareholder value. Then you look at the strategic driver and you say, what is the role of IT, what IT do I need? Its not really the role, but its really what infrastructure, what do I need to make the strategic driver happy? Well, I need to have a CRM system, I need to have 250 salespeople, I need to get rid of 25 number-crunchers, etc. Put all of that together.
Once youve put the together, you then say, what is the impact of this on my company, and that has to be added to the budget. And if I cant swing that because maybe for the first couple of years Im going to have negative free cash flow [before] its going to turn around, I obviously have to get rid of some of the things that Ive been doing until now. Joe was alluding to [this] before, when he talked about looking at how we do things every day in our business.
So as I see it, the benefit of going through a process like this, where you identify the strategy first, is that IT flows in as an enabler of the strategy, not as somebody who is running around trying to implement a specific thing. Why do I want to have this enabler? ROI, and I have to really push back on this, ROI is just a tool. As you say in IT, if the input is lousy, the output is lousy.
Barkley: Garbage in, garbage out.
Fletcher: Thank you, sir. But garbage in, garbage out is also true of ROI. Again, going back to what Joe said, whether its ROI [or] any number of other techniques, you just really need them because you have to set priorities at some point. But if theyre all going to be positive and youre a very cash-flow rich company, you can afford to do all of them, and sometimes your gut tells you the numbers are there.
CIO Insight: Is there an example from companies where this works very well?
Fletcher: Well, in my last position at MasterCard we did implement a process that we called a two-phase budgeting process. Phase one was in the first and second quarters of the year, where we really only focused on strategy. And then the goal in the fall was that the numbers would flow from the strategy, as opposed to the numbers driving the strategy, which is what I think a lot of companies do. It becomes a mechanistic budgeting process. Who cares about the decimal points? The CEO doesnt make decisions based on decimal points. The reality is, if youre going to have an after-tax return and its a positive return, unless you are bidding one project against the other, you really dont care if its 15 percent after-tax or 25 percent after-tax. And Ill be honest with you: When I get a number like 3,000 [percent ROI] or whatever Christopher was talking about, I do the exact same thing Christopher does-I slash. I say thats nonsense. I discard it and I say, okay, its greater than my cost of capital, and thats really all I care about.
CIO Insight: I want to continue going around the table getting reactions and specific suggestions on what we need to do to do a better job of looking at value. Whats the optimal approach that we need to take? Tom, let me get your response.
Murphy: Well, Ive kind of articulated what our approach is. We have aligned our strategy, but each element of that Jumpstart initiative does go through a review process. We do go through a process of evaluating value. We do some bucketing. We have a lot of safety, environmental and regulatory issues as a company that we have to address. Obviously, those become high-priority items that go onto the project list with relatively little review. We do have revenue-benefit, cost-efficiency and business-expansion projects that get a little bit more financial focus in terms of ROI than some other projects.
Then we have our strategic and goodwill projects, and under goodwill we have guest and employee satisfaction, which are harder to measure with pure financial elements, but we spend a little bit more time focused on business objectives. And those go into the formal evaluation process, and we do go back six months later and determine if they, in fact, had a return. Were the objectives met? Were the actual financial goals met? We meet on a quarterly basis with all of the senior officers of the company and review the project prioritization and modify it as we go. And then, depending on how well the company is doing, we can pull back on our capital investment or increase the capital investment relatively easily, based on our project portfolio process and prioritization process that we use.
Theres a very strong sense of engagement on the part of the business leaders that they are part of the decision-making process, that theyre part of the valuation process as opposed to IT being in that cat bird seat.
Hirji: I think the cardinal rule has to be to keep it simple. Anyone can take something and make it overly complex. It looks very precise, tons of man-hours get chewed up doing it, and everyone feels better, but the reality is youre no further ahead. So the real genius is turning decision-making and making it simple. So let me walk you through how we do it. We think we do a reasonable job.
Its a little strange to me to have a conversation about IT not being connected with the business, because in our case IT is the business. So theres never a disconnect. We are the product, therefore, for us to not be connected with the business is impossible. Second, my role, for example, is half shaping and executing the strategy of the firm, and half running the IT organization. So there is no major decision that is made without IT being at the table; similarly, IT would not make any major decision without the business being there. We never have the situation where the “R” and the “I” are in different places. Third, we have very specifically identified what our strategy imperatives are. So getting back to making sure IT is aligned with the strategy, in our case, is very simple. We are the low-cost producer with the highest margins, therefore, priority Number one is things that bring that cost structure down. Number two, we provide the best tools in the industry for our traders, so were always looking at creating the next generation toolset. And, third, were creating new tool sets. So, again, we have three very clear goals. We can align all the projects against one of those three goals. If it doesnt meet one of the three goals, its off the table.
The next thing that happens is [that] every single week the business and technology folks get together and run through the [project] prioritization. They basically list it one through ten in terms of what they want. We do that weekly. Why? Because our projects are small [and] because we dont bother trying to estimate something thats a year or two out. The marketplace changes; what was important last week may not be important this week. The one rule we have is [that once] a project is being coded, its too late to call it back. But that creates the trust in the business; if you change your mind about what you want to do, you are at worst four weeks away from getting it implemented. Before any project gets on that list, it has had a validation or business case; it has a number of things in it that say this is why its important. Sometimes its ROI, sometimes its customer [service], sometimes its a bunch of things. The important point is that that process gets both the IT folks and the business folks to say, we jointly believe this should be on the list and its important, and we think it should be number X out of 50 or whatever the number is. Thats why you do the process. Its irrelevant what the number is. Whats more relevant is getting alignment.
CIO Insight: Does this begin to address some of the concerns you have about bringing back the human element to looking at value?
Riazi: Im not certain. I mean, Ive heard a lot of great ideas here. We always fall back on our comfort zone, and I think in a debate we fall back to our analytical approach and a logical approach to a problem