Questionnaire Reliability If you are talking to other providers of questionnaires, look for statements in their materials or on their site about the reliability of the questionnaires they are selling.  Look in their "Frequently Asked Questions".  If you can actually find someone to ask, listen carefully to their answers.  Here are some things to look for and questions you can ask: Do they mention reliability?  You don't even see the word on the questionnaire sites developed by IT people. Are there any statements or guidelines about "sample size", or the minimum number of respondents necessary for good results.  If there is no concern about minimum sample sizes, there won't be any discussion about reliability. If they seem to know what reliability is, do they measure it?  And how do they measure it?  There are different approved ways — listen for words like "test-retest", "equivalent forms", "split-half", or "Cronbach's Alpha".  You don't have to know what those terms mean, but they certainly should. Ask if they will measure the reliability of your questionnaire's results.  This is far better than a test of reliability performed at another time and place.  It will give you concrete evidence of whether your results can be trusted. Ask them if their questionnaire controls for different response patterns of the respondents.  Did you ever notice that some teachers give mostly C's and B's with few A's, while other teachers give mostly A's and only a few B's and C's?  People fill out questionnaires the same way — some are "black and white" using lots of high and low responses, while others use two or three boxes only.  Just like an A grade from an easy teacher is different than an A from a hard teacher, a reliable questionnaire knows that a "3" from one respondent is not be the same as a "3" from another. Ask them if their questionnaire controls for the effect of a issue's position in the survey.  It is known that people spend more time thinking about their answers at the beginning and get tired toward the end of a survey.  Does their questionnaire present different versions so each issue appears both early and late in the survey? Ask if they are able to detect collusion — when a group of respondents agree in advance how they will answer the questionnaire, to make the results come out a particular way. Ask if they measure the variability of each question or issue, to know on which issues people agree and on which they disagree. Ask if they can identify respondents who have difficulty understanding the questions or instructions because of literacy or language issues. If you let us help you develop your questionnaire, the answer to each of the above questions is YES. You could be making business decisions based on survey results that don't mean anything. What is Reliability, and Why Does It Matter? If you buy an unreliable questionnaire, it's worse than buying an unreliable car.  At least with a car, you know when it is not running.  A questionnaire, on the other hand, will always produce numerical results, even if they're meaningless.  You could be making business decisions based on survey results that don't mean anything.  Only a test of reliability can tell you if you should trust the results. Non-technically speaking, a reliable questionnaire is one that that would give the same results if you used it repeatedly with the same group.  That may sound funny because most organizations don't administer a questionnaire to the same group twice.  But if they did, they would learn how reliable their questionnaire is, because a reliable survey will give the same results on Tuesday as it did the previous Monday.  Instead of doing it twice, statisticians have devised tests of reliability for questionnaires.  These tests let us know whether the results are meaningful. Reliability is a property of the measuring instrument.  If you are like many people, you probably get on your bathroom scale in the morning, look at the weight displayed, then step off, and do it again.  You have learned that what is displayed by a bathroom scale the first time is not always exactly the same as the second, but it is usually very close. What if one morning you weighed yourself, then a second time, and the second weight displayed was 5 lbs. heavier than the first?  You would probably step off, then weigh yourself a third time.  What if it was now 4 lbs. lighter than the first?  Would you still be concerned about your weight?  Or would you be more concerned about finding out what's "wrong" with the scale?  What's wrong is that your scale has become unreliable.  You can see unreliability by repeatedly measuring the same thing.  And when you know the scale is unreliable, you don't even try to measure your weight, you concentrate on fixing the scale first. Only when you know your questionnaire is reliable, can you begin to discuss validity.		Questionnaire Links . . . Sample of Clients Reliability and Validity Other Questionnaire Topics Frequently Asked Questions Online Questionnaire Examples Why Don't Our Surveys Look     Like Traditional Surveys? What's Wrong With     Rating Scales? How do I get more information?
posted 12:52 - 06.08.08   |    © 2002-2007 Evensen Web Design