Lies, Damn Lies, and Radica...oops I Mean Statistics

Imagine if you will a scenario. A scenario where you want to do a survey of an installed software user base to get a handle on the future of the software in question. Go further and assume that the software is used by 60,000 customers worldwide. So you develop a survey that contains questions that do not include choices that everyone can answer, but limit the answers to those that drive some unknown internal agenda. You then, instead of choosing a statistically valid sample of that n=60,000 universe, you post the survey on your company web site so that visitors to your web site, few as they may be, can self-select their participation.

The quantitative results, according to John Zogby of Zogby International in a phone interview today, "would not be valid if extended to be representative of the overall universe, and would not be useful at all". No, if you read this survey, you have not arrived in the Twilight Zone, but in The Radicati Zone.

Now before Dan Lyons of Forbes jumps all over my case and tries to misconstrue what I am writing here, not that he would ever do a thing like that, I need to make something perfectly clear. Mr. Zogby did not know what company I was referring to or what survey. He was simply addressing the issue of the validity and usefulness of a survey instrument in the context of the very real scenario described above. John did say that even though an instrument designed and administered this way might not have quantitative value, it "could have a qualitative value if the right level of people participated in the survey". More about this later. How can John make a statement like this? Simply put, he owns the pre-eminent survey research firm in the United States and has methodologies that other firms only wish they had at their disposal. He has no dog in this hunt and had no idea of the context of my interview questions with him. I do want to thank him for taking the time to talk with me as he was preparing to leave for a trip to Shanghai in the morning.

And let me also make perfectly clear that my problem with this report being given away free by the Radicati Group, well free if you give them your email address, does nothing but instill "Fear, Uncertainty and Doubt" (FUD), which is one of my soapbox issues. It just so happens that this company seems to have an axe to grind against IBM, but that the media and competition will still take this information as gospel. The key is that it is a good thing they are not charging for this document,  because then someone might have to report them to the better business bureau.

"We Surveyed 32 Lotus Domino Customers Chosen Completely By Random"

There is no other word that can appear to be used hear except for "balderdash". This statement assumes that the Radicati Group really did a statistically valid random sample, on that would give a 95% confidence interval, with an error margin of +/- 5%. But to do this, they would have to have received survey results from 383 customers. Their sample size has a margin of error of +/- 17.4%, which, as Zogby points out, is totally inadequate if you wanted to extend results to the full universe of customers.

No, the respondents were not chosen at random. It was small self-selected sample that, I believe, included some of my colleagues in the blogging world. The respondents chose to fill in the survey, they were not invited. This is an open challenge to the Radicati group: prove me wrong and republish this report outlining your specific methodology. Note that I did e-mail the Radicati Group with my concerns and as of yet have not received a reply.

"Only High Level Executives and IT Professionals Allowed"

The report states that "Only high level executives and IT professionals were allowed to answer the survey. Hold on while I search for another word so I am not redundant. Ah, here it is: BUNK! There were no controls of any kind about who was allowed to reply and at what level they had to be at. How does the Radicati Group even know who answered and if their answers were truthful? They don't. How do they know that they did not receive multiple responses from the same company? they don't.

Radicati Just Does Not Get It

I do not want to get into too much detail about the questions, but issuing a report saying that "33% of the respondents have decided NOT to move to Lotus Workplace..." is like saying, oh heck I don't know. The reality is that there is no need to migrate and unless IBM is bald-face lying, that is the answer most firms should give. It just adds more FUD. It may make great headlines, but the story has no meat or legs.

The Benefit of Qualitative Data

Zogby, as I stated earlier, said that even though a poorly designed survey instrument is not useful from a qualitative standpoint, it may have qualitative value if people at the right level of an organization responded or other factors played into the review. The example he gave was the The Hite Report: A Nationwide Study of Female Sexuality, issued in 1976. If you want to know more about the duality of this report, you can read that here. However, for this Radicati Group document, this would be the decision-makers and possibly the internal influencers. This would be C-level people to me and possibly managers. If you look at the percentages of respondents, it might look high. But only 30% of the respondents (9.6 of the 32) were CTOs or IT Managers, may of who may have other issues on their plate like Sarbanes-Oxley or HIPAA.

The qualitative value may be here and there are valid concerns that IBM does have to address, but only if they make sense and are presented in a report that does not have an agenda or negative overtones based on this history of the company issuing the report. I will let others dissect that part.

This Document Should Be Shredded

If you are a CIO or an IT Manager and someone from Microsoft or another competitor hands it to you as evidence that "Notes is Dead", put the report where it belongs: file 13 or the shredder. Stop relying on outside analysts that may be pimping themselves to the highest bidder and exercise your own due diligence. Buy the software that meets your specific business objectives and makes the most sense. Only you know that, not the analysts for hire. You will nit et a link to the report here.