customer-satisfaction survey - summary

[GILLIOTTE Valérie <VGILLIOTTE -at- MEGA -dot- COM>]

Thank you to all of you who shared my interest in customer-satisfaction
surveys and to those who offered their help. I got what I needed. I
understand making surveys is a very difficult task and the results are
not always useful.
Here are samples of the answers I received. Thank you again for your
cooperation.

1)
We document User Guides and reference information for a large retail
organization in Australia. We surveyed over 150 stores.

The survey provided mixed results. The biggest problem was collating the
results. Collating and analyzing the results took two technical writers
many months. I suggest you consider using an independent research
company to conduct the survey for you. I also suggest you test each
question thoroughly. We didn't and ended up collate responses to
ill-conceived questions.

2)
Here's a page from our latest customer satisfaction survey as
a sample. We also had some open-ended questions.

Rate XXX documentation in each of these areas:
  Appropriateness of reading level
  Appropriateness of level of technical detail
  Clarity of expression
  Organization of topics
  Accuracy
  Completeness
  Usefulness of indexes
  Use of graphics
  Number and appropriateness of examples
  Overall attractiveness
  Conceptual explanations
  Step-by-step procedural explanations
  Online help

Do you have specific suggestions for improving XXX documentation?

Please circle the best response to questions 6 through 10:
6.When you have a question about XXX, finding the answer in a manual is
     Impossible   Always difficult  Usually possible Often easy   A
breeze
7.With XXX documentation, the chances you?ll find the answer you need
are
     Zero   5%   20%   50%   90%
8.Having the current docs online (indexed) while you use XXX would be
     Useless   Slightly helpful   Helpful   Very helpful   A boon
9.Being able to jump to the relevant page in a manual (online) while
using XXX would be
     Useless   Slightly helpful   Helpful   Very helpful   A boon
10.The best thing about XXX online help  is:
     Nothing   Availability   Content   Ease of finding help


We found the hardest thing was to get people to give 10 minutes to
the task of responding to the survey.  I wish you luck.

3)
I understand the desire to run a questionnaire past all your users, but
I
don't think it will be an efficient use of your time.

Fact is, if someone is satisfied (or busy or complacent), they won't
respond.  That being said, you will have no wage to gauge how
representative your returned sample is.

My experience suggests you will get better, more usable information if
you
approach several users directly.  If someone is copmplaining, talk with
them.  Try to get a feel for what they are saying.  Decide if their
complaints are valid.  Determine who your typical user is and see if the
program and docs make sense from that particular point of view.  Often
programs are written to specifications developed by the folks that are
selling or buying the program.  Many times, the end user of the software
never gets a chance to interject an opinion.  take the approach that if
anyone says they are dissatisfied with any part of the program  (and the
program DOES include the docs), the ENTIRE program/project has to be
closely evaluated in terms of the complaints.

Also, consider that the problem may not be the documentation, but rather
the program itself.  A good writer and cover up a program's
inefficiencies
to some degree, but if the program is lacking a function, a user's first
inclination is to blame the documentation (it didn't discuss the
problem,
therefore the docs must be incomplete).

Good surveys are difficult to construct (as a matter of fact, it's a
whole
other industry) and they are only as reliable as the sample that returns
them.

4)
We've found that surveys aren't all that useful. It's hard to think
ahead
to all the things that a consumer might like or object to. Then you
always
find yourself having to call some of the survey-takers to clarity
things.

We like to just call people and ask. Talk to them long enough and you'll
get the true story. We've found that you can get a good picture from a
surprisingly low number of interviews..maybe as few as a dozen. At least
the interviews will give you a better idea of what to put in a bigger
survey.

5)
Here's the survey we recently designed -- it's too soon to know how many
responses we'll get.
http://www.seagatesoftware.com/ashwin/docsurvey/
Our documentation is online in PDFs, so users can get to the web page
survey by selecting a link in the PDF. Also, I included a copy of the
survey in the books so users can print it out or copy it. We also set up
an e-mail alias for the product that directs mail to my e-mail address.

I kept the form short (7 questions with room for comments) and asked the
questions that most interested us most (was the information correct?
complete? could you find the information you needed? were the new docs
an improvement?)

In the past, we had a copy of the survey in the back of our hardcopy
books only. Even though we offered a certificate for a free dinner for
respondents, we didn't get many responses at all. I think members of our
audience are more likely to use an electronic form than a hardcopy form.

- end of summary

Valérie GILLIOTTE
vgilliotte -at- mega -dot- com