Lies, Damn Lies, and Radica...oops I Mean Statistics
Imagine if you will a scenario. A scenario
where you want to do a survey of an installed software user base to get
a handle on the future of the software in question. Go further and assume
that the software is used by 60,000 customers worldwide. So you develop
a survey that contains questions that do not include choices that everyone
can answer, but limit the answers to those that drive some unknown internal
agenda. You then, instead of choosing a statistically valid sample of that
n=60,000 universe, you post the survey on your company web site so that
visitors to your web site, few as they may be, can self-select their participation.
The quantitative results, according to John
Zogby of Zogby
International in a phone interview
today, "would not be valid if extended to be representative of the
overall universe, and would not be useful at all". No, if you read
this survey, you have not arrived in the Twilight
Zone, but in The
Radicati Zone.
Now before Dan
Lyons of Forbes jumps all over my case and tries to misconstrue what I
am writing here, not that he
would ever do a thing like that, I need to make something perfectly clear.
Mr. Zogby did not know what company I was referring to or what survey.
He was simply addressing the issue of the validity and usefulness of a
survey instrument in the context of the very real scenario described above.
John did say that even though an instrument designed and administered this
way might not have quantitative value, it "could have a qualitative
value if the right level of people participated in the survey". More
about this later. How can John make a statement like this? Simply put,
he owns the
pre-eminent survey research firm
in the United States and has methodologies that other firms only wish they
had at their disposal. He has no dog in this hunt and had no idea of the
context of my interview questions with him. I do want to thank him for
taking the time to talk with me as he was preparing to leave for a trip
to Shanghai in the morning.
And let me also make perfectly clear that my problem with this report being
given away free by the Radicati
Group, well free if you give
them your email address, does nothing but instill
"Fear, Uncertainty and Doubt" (FUD),
which is one of my soapbox issues. It just so happens that this company
seems to have an axe to grind against IBM, but that the media and competition
will still take this information as gospel. The key is that it is a good
thing they are not charging for this document, because then someone
might have to report them to the better business bureau.
"We Surveyed 32 Lotus Domino Customers Chosen Completely By Random"
There is no other word that can appear to be used hear except for "balderdash".
This statement assumes that the Radicati Group really did a statistically
valid random sample, on that would give a 95% confidence interval, with
an error margin of +/- 5%. But to do this, they would have to have received
survey results from 383 customers. Their sample size has a margin of error
of +/- 17.4%, which, as Zogby points out, is totally inadequate if you
wanted to extend results to the full universe of customers.
No, the respondents were not chosen at random. It was small self-selected
sample that, I believe, included some of my colleagues in the blogging
world. The respondents chose to fill in the survey, they were not invited.
This is an open challenge to the Radicati group: prove me wrong and republish
this report outlining your specific methodology. Note that I did e-mail
the Radicati Group with my concerns and as of yet have not received a reply.
"Only High Level Executives and IT Professionals Allowed"
The report states that "Only high level executives and IT professionals
were allowed to answer the survey. Hold on while I search for another word
so I am not redundant. Ah, here it is: BUNK! There were no controls of
any kind about who was allowed to reply and at what level they had to be
at. How does the Radicati Group even know who answered and if their answers
were truthful? They don't. How do they know that they did not receive multiple
responses from the same company? they don't.
Radicati Just Does Not Get It
I do not want to get into too much detail about the questions, but issuing
a report saying that "33% of the respondents have decided NOT to move
to Lotus Workplace..." is like saying, oh heck I don't know. The reality
is that there is no need to migrate and unless IBM is bald-face lying,
that is the answer most firms should give. It just adds more FUD. It may
make great headlines, but the story has no meat or legs.
The Benefit of Qualitative Data
Zogby, as I stated earlier, said that even though a poorly designed survey
instrument is not useful from a qualitative standpoint, it may have qualitative
value if people at the right level of an organization responded or other
factors played into the review. The example he gave was the The Hite
Report: A Nationwide Study of Female Sexuality, issued in 1976. If
you want to know more about the duality of this report, you can read
that here. However, for this
Radicati Group document, this would be the decision-makers and possibly
the internal influencers. This would be C-level people to me and possibly
managers. If you look at the percentages of respondents, it might look
high. But only 30% of the respondents (9.6 of the 32) were CTOs or IT Managers,
may of who may have other issues on their plate like Sarbanes-Oxley or
HIPAA.
The qualitative value may be here and there are valid concerns that IBM
does have to address, but only if they make sense and are presented in
a report that does not have an agenda or negative overtones based on this
history of the company issuing the report. I will let others dissect that
part.
This Document Should Be Shredded
If you are a CIO or an IT Manager and someone from Microsoft or another
competitor hands it to you as evidence that "Notes is Dead",
put the report where it belongs: file 13 or the shredder. Stop relying
on outside analysts that may be pimping themselves to the highest bidder
and exercise your own due diligence. Buy the software that meets your specific
business objectives and makes the most sense. Only you know that, not the
analysts for hire. You will nit et a link to the report here.
Comment posted by Nathan T. Freeman05/25/2005 03:30:38 AM
Glad to hear you went to Zogby for methodological feedback.
See my comments on the survey here: http://www.bruceelgort.com/blogs/be.nsf/plinks/BELT-6ARTGF
TrackBack From Gregg Eldred05/25/2005 01:36:54 PM
And so it goes . . . .
Linked to you, Chris. I really liked your thoughts on this topic.
Comment posted by Alan Bell05/25/2005 03:43:18 PM
Homepage: Http://www.dominux.co.uk
one thing that really puzzled me last time and again this time was their proud (and untrue) claim that:
"Only High Level Executives and IT Professionals Allowed"
why do they think this is a good thing? They are providing summary reports to high level executives based on a survey of high level executives, did it never occur to them that their function is to summarise the views of those who have a clue to those that don't. Here is what I wrote last time, I think it is just as valid as before.
http://www.dominux.co.uk/dominuxblog.nsf/d6plinks/FOUN-63LCJZ
Comment posted by Jack Ratcliff06/03/2005 01:11:18 PM
Homepage: http://jackratcliff.com
Great article Chris. I wish I could have found you and others back in 2001 when my company dropped Notes. I'm sure some of the reasons that helped them make the decision was from analysist reports that they took as the "gospel" truth.