Can We Trust the Analysts?
Can We Trust the Analysts?
One of the most important decisions each of us makes—and we do this hundreds, maybe thousands of times each day—is who or what to trust.
Who or what can I trust? Thats the question I ask myself when someone presents me with a research study or when I seek the opinion of an industry analyst. Its the same question you should be asking about the information you consume.
Is the information were presented "honest" or is it tainted by a particular vendors interest? More bluntly: Are we being lied to so people can take our money?
In the grand scheme of things, probably. But I dont think the computer industry is worse than any other, and it may be a good deal better. For example, can the analysts we so often see quoted be trusted? And should we ever trust the results of a study that a vendor paid to have done?
The answer to both these questions is "Yes." But its a qualified yes, and Id like to explain both the value and limitations of using analysts and their reports as sources. If you are a regular consumer of the computer trades this should be important information. Much of what you read either contains quotes from an industry analyst or in some ways reflects their thinking.
The good news: The established players are pretty trustworthy. The companies—and news organizations—whose names you recognize have been around a long time and have earned the industrys trust. They work hard to maintain it. That doesnt mean they are always right, but it does mean they arent corrupt.
Seeking to understand this issue, I did what I often do: I called a few analyst friends, people who have earned my respect over a period of years, and asked them how my readers could be better consumers of the information analysts generate.
Heres some of what I heard:
Dont expect a crystal ball—Analysts are professional watchers of a particular technology or market segment. Ideally, they know their areas better than anyone else, including executives of the companies they cover. Many sign nondisclosure agreements, which sometimes allow them to "predict" trends with 100 percent accuracy. But dont confuse that with omniscience. Also, remember the tendency for analysts to fall into the trap of seeing their world as the center of the universe, and making predictions accordingly.
If you see an analyst endorsing something, watch out—Reputable analysts will rarely recommend one product or vendor over another without solid research to back it up. The big-name analysts work for companies that get money from all the major vendors, and there is no value to them in angering all but one of their clients.
Just because a vendor pays doesnt mean the vendor calls the shots—One of my friends is working on a survey that is being sponsored by 10 vendors. One vendor asked if it was possible to drop a particular question if the vendor didnt like the results. My friend offered to let the vendor have its money back and not be included.
But thats at a major firm. My analyst friends stopped short of saying small firms cant be trusted, but do be wary. The research business is highly competitive and vendors are not beyond taking advantage of a startup or a company that is in trouble as the research business again consolidates. There is also a tendency of some small shops to be more cheerleader than steely-eyed research company. Especially when the vendor pays the bills.
Next Page: Evangelism, perspective, lies and ideals.
Evangelism, Perspective, Lies and
Ideals">
This should not, however, be confused with good old-fashioned evangelism, in which people promote a cause because they deeply believe in it, have a financial or professional stake in it, or both. These are the people who have, to use a common if unfortunate phrase, "drunk the Kool-aid," and thus become suspect.
It is to be expected, for example, that people who have tied their wagon to the Linux star will be fervent supporters and even propagandists. If you agree with their viewpoints, these people can be excellent sources of information. But be careful of accepting their global view of how things fit together, as their perspective can be both skewed and narrow. In short: There are lots of people who need to get out more often and see how the world really works.
Read the fine print—If you see survey results, try to understand who was questioned and what they were asked before accepting the conclusions offered. Polling can be skewed by a number of factors, resulting in some of the wildly different results we see. All polling is fraught with risk and few of us are really expert enough to separate good polls from bad ones. Often, the polling organizations dont seem to know.
Likewise, if you are looking at product test results or performance claims, look at how the numbers were generated. The major publications and organizations that routinely do testing take pride in their methodologies. Some of their test results end up in product advertisements.
What about vendor-sponsored testing, as in Microsoft comparing its products to Linux in its advertisements? Assuming the information is true, would anyone believe it if Microsoft did the testing in-house? And how is a testing lab supposed to fund its work if vendors dont pay? This is especially true of enterprise products, where testing can be complex and expensive.
My own belief is that vendors publicize the most favorable numbers they can find short of making them up. And may disregard less favorable results along with way. Reading the fine print may show telltale signs of testing being, well, "rigged" isnt quite the word, but presented so as to produce results more favorable to the vendor.
I filter these ads this way: They arent lies, but may compare a companys best case to a competitors worst case.
In an ideal world, all research would be paid for by its ultimate consumer. This would align the researchers interests with those of the customer. Likewise, the ideal news organization would receive its funding only from readers, much in the way Consumer Reports eschews advertising.
The problem with this is twofold: First, readers and customers dont like to pay for as much information as theyd happily consume. And, secondly, vendors need to get their messages to potential customers, and advertising is a legitimate way to accomplish this.
I think probably 90 percent of the information I read from mainstream sources is, if not totally reliable, at least not intentionally slanted. The further the source is from mainstream, the more skeptical I become.
And thats pretty much the conclusion of my analyst friends, who all said their own companies could be trusted but to treat everyone else with at least a wee bit of suspicion—and sometimes a whole lot more.
Check out eWEEK.coms for the latest news, reviews and analysis on IT management from CIO Insight.
