Beware of Analysts Bearing Surveys

We all love a good survey. I especially like those little survey factoids they have in USA Today that provide useful information like how many gallons of water got flushed during commercials on "American Idol."

I often wonder how useful survey data is. To the uninitiated, a survey seems like cold hard data—"just the facts, maam." The truth is that survey data is up to as much interpretation as anything.

The famous economist John Kenneth Galbraith was the man who coined the phrase "conventional wisdom." People forget however that he used that term in a derisive fashion. He noted that humans often associate truth with convenience and self-interest; we are more likely to believe a version of truth that can avoid any effort or discomfort on our part.

Resource Library:

The dirty secret is that survey data does not contain truth—it only contains a version of the truth. In other words, its opinion.

So when an analyst company publishes survey results, it also publishes its opinion of what the results mean. Other interested parties such as vendors or product champions within an enterprise will add their spin. The problem is that survey data becomes the conventional wisdom simply because we have been taught that figures never lie.

For the uninitiated, most IT analyst groups have traditionally fallen into one of two camps. The first is the qualitative research model. This model attempts to take some smart, knowledgeable people—hopefully with a background in IT—and let them speak with lots of users. From this interaction the analysts form an opinion and publish it in the form of a research note.

The second camp is the quantitative model, which relies heavily on survey work to show trends in behavior. But numbers do not speak for themselves. So analysis, otherwise known as opinion, is then applied. The important point here, folks, is that both models still ultimately represent opinion.

As a practitioner of the qualitative approach to research, I still love to look at survey data because all data points provide more insight into a subject. There is, however, no perfect model. I am always interested in how one analyst views the data versus how I would view the data. This is a practice not unlike playing devils advocate, and one I highly recommend that users try when presented with survey results produced by analysts and used by vendors for their own purposes.

Gartner accused Oracle of misusing Gartner research data in an ad in The Wall Street Journal. Click here to read more.

I was reviewing a survey recently published by Gartner on database management systems spending and deployment characteristics. It caught my eye not only because the database market is my particular area of interest, but also because I was fascinated by the analysis of what the numbers meant, so much so that I had a conversation with Colleen Graham, one of the authors of the report.

Graham works in Gartners Dataquest division, which is the decidedly quantitative side of Gartner. She also works on the annual market share survey for the database market. As a professional at conducting surveys, Ms Graham commiserated with me that she is often surprised at how the numbers she produces are used and spun beyond recognition in some cases.

In the report, 704 respondents were interviewed about their intentions. One part of the survey asked users which databases they planned to deploy in the future, which ones were already installed, and which did they have no plans to deploy. I found the results and the analysis to be very interesting.

Essentially every respondent answered the same questions for each of six database platforms plus a seventh "other" category. The written analysis emphasized the surprise that 19 percent of respondents indicated plans to deploy more DB2 compared with only 13 percent for Oracle.

Next Page: Putting a different spin on things.