The Questionnaire Design for Surveys, part III

This article is a follow-up to this one and it refers to the questions presented as an example here:

 Let’s take it step by step. First of all, methodologically. The first question uses a scale with an even number, which requires the responder to mark only one answer. The second item presents an odd number of choices out of which one, several or all can be picked by the responder. This creates confusion because it requires the responder to use a different rule for answering question 2 from the rule used to answer question 1. No matter how well you explain, it is subject to mistake. You must eliminate these mistakes, out of respect for your work, for your personal branding as a competent, non-time waster professional, for the responsibility which you have towards providing actionable intelligence and for the responders’ intellectual effort.

The second question uses an unclear scale for choices, which combined with the words used in the item may create confusion.

If it’s a scale, it must be from white to black, from dark to light, from alpha to omega. You cannot play around with the responders’ perception. If the first choice is “daily”, the other choices may be “2-3 times a week”, “weekly”, “2-3 times a month”, “about once a month”, “3-4 times a year” “Yearly”.

“Pretty often “is a poor choice because what is “pretty often” for someone can be very different than what is “pretty often” for someone else. It could range in the individual perception from once a month to twice a day to all the time. Don’t use relative language. Use precise words when you measure frequency or intensity. And if you want both frequency and intensity, for example, measure them with different items. Yes, that means putting the same questions again, with similar scales.

“Sometimes” is a different item for measuring frequency than “pretty often”, but it may be similar. You may use these terms together in rather a six points scale like “never”, “rarely”, “sometimes”, “pretty often” “very often”, “always”. You can also cut the extremes and have it a four points scale, conditioned that the rest of the items besides “pretty often” and “sometimes” are “rarely” and “very often”. I myself I wouldn’t use this kind of vague terms, but they are not wrong.

Let’s take a look at the first words of the item: “How often do you usually”, continued with option d), “rarely”. So the question the responder has to ask himself in order to understand what the item means, in this case, is: “How often do I usually rarely…? Do I rarely often usually… ?”. It’s a nonsense. Use logic. If you aren’t used to logical thinking, learn it or drop human resources, NOW.

Another issue with the first question is the word “feel”. It’s a tricky word to use in a subjective description. In general, people are already very subjective when completing a questionnaire. Especially when you measure subjective experience of frequency, it is rather preferable to use moderate, neutral words. Besides, some people could relate more to seeing or hearing than to feeling. In formulating an item, you must respect the reality that people have different perception styles which you have to respect in order to speak on their own language. For these two reasons, it is recommendable to use the word “find” instead of “feel”. It appears to describe a more objective experience, but in reality, the subjectivity comes from the term “tension”. When using subjective nouns in items, don’t enhance it by adding subjective verbs to them. “Find” is more objective than “feel”. Pay attention to the choice of words. “If you ask wrong questions, you will get wrong answers” (3).

“When starting work” describes an unclear event. It could mean the minute entering the building, it could mean the moment actually starting doing work (after coffee, talking to the boss or a morning conference), or it could mean coming back to work after the meal. To which moment does the researcher refer?

Another thing to pay attention is that the first question refers to “working with colleagues” when the second questions also refers to other departments or family. When you want to evaluate the organizational environment in a specific office, you focus your questions in such a manner to discover which are the elements inside the system which are not working. If the researcher wants to find out also the external elements, than item 1 must be restated, in order to integrate the external causes. Not to mention there is a high difference in perception whether the responder has an internal locus of control or an external locus of control in the context of working in that specific environment.

The first question uses the term “tension”, the second question uses the term “pressure”. Although they may be synonyms, if you used one term for the first item, then you must also use it in the second item, too. Some people might relate differently to “tension” than to “pressure”, not to mention that “tension” is often perceived as internal, and “pressure” external. It’s not a rule, but there are subtilties to which you have to pay attention.

The second question is wrong from the beginning. If the purpose is to find the cause of a distress, you must first consider all the reasons for the distress. Let’s say, for example, that an employee has a good chair at his own office, but when joining others to work on a common project, the table around which they sit has wrong chairs for his back. The person feels a psysical tension, but none of the variants are responsible for that. It’s the chair. It’s not someone’s fault, it’s something. So define well what you want to measure and what words you use for it.

Please also consider this list of articles in Romanian about research.

Author’s experience in questionnaire design. The Author has worked as a member of a research group developing organizational evaluation tools. Since November 2006 to June 2007, he contributed to five questionnaire designs in students’ groups. Also, he has designed the first psychological questionnaire for application in political branding using NLP (Iaşi, 2004), for which project he was awarded the first prize at the students’ contest EconomMix in 2005, the management-marketing section.

Besides that, he has designed psychological questionnaires for own researches on parental education (2005), self-esteem (2007, 2009, 2011), insurance (2009), memory (2009), he enhanced an evaluation questionnaire for career consulting (2008).

The author can be contacted for questionnaire design consulting services at [at] yahoo [dot] com


3 thoughts on “The Questionnaire Design for Surveys, part III

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.