When designing questions, a known problem is the façade effect. The responders tend to understand what the researcher wants to find out and give “politically correct answers”. Although it might be difficult, there are ways to circumvent that. It’ all in mastering the words and the language in such a manner that the answer reflects what you want to find out, without letting the responder understand what you’re after.
One of the most powerful things you can do, on which many forget to act, is the simple frame of the questionnaire. Mention, either verbally or in writing, that the questionnaire is not intended to measure correctness, but rather preferences and tendencies, if you’re doing a psychological survey rather than a knowledge or a performance review. This can be done more easily with structured or semi-structured interviews. But for the larger frame, I’ll refer to any kind of questionnaire design.
One way of measuring the impact of façade effect is to build into the instrument a lying scale, that is, something which will measure through specific items one’s tendency to lie. But that may be too complex for some. A simple way is to put key-questions, in different parts of the questionnaire. That is, to ask the same things twice, in two different ways. If you put two-three of these questions and the answers do not match, you have reason to believe the responder was not honest/serious in completing the questionnaire.
Another way is to measure multiple dimensions / psychological scales with the same item. The risk is that the item will not be interpreted the same way by everyone and it can’t generate any pure correlations, but CPI, one of the most appreciated personality inventories does it. And it does it very well.
You can also do it with different choices for the same item. For example:
1. Usually, at work, do you consider:
a) Who wakes up early gets a better job done [measures proactivity]
b) I let others do the planning for my tasks [measures planning type]
2. When handling multiple tasks:
a) Who runs after more than one rabbit catches none [measures planning type and locus of control]
b) I’m first doing my tasks [measures proactivity and locus of control]
3. When a colleague does a mistake:
a) I let him/her correct the mistake [measures reactivity]
b) I delegate him other tasks [measures planning type]
4. How do you know when a colleague made a mistake?
a) The schedule I have planned does not meet the results [measures planning type and management style]
b) He/she is responsible for his/her own thing and he/she can know when something is done right or wrong [measures reactivity and management style]
You can notice how easy it is to believe, by sometimes opposing choices which do not measure the same thing at the same item can create confusion by forcing a choice between the two alternatives. The important thing to keep in mind is to oppose the dimensions an even number of times for each dimension. For example, these 4 items measure: planning type, management style, locus of control and activity type. I have opposed planning type to activity four times, I have measured plain locus of control and plain management style once. Therefore, at the end of these four items, I will have a score for each of the categories, each having a chance to be scored one point based on any of the choices offered. In the software-designed questionnaires, you can also add questions if at the end of the test there are categories with the same number of points for one of the preferences. Let’s say in the end, the responder has an equal preference for internal locus of control and for proactivity. You may add two questions, either one measuring only locus of control and the other activity type or both measuring opposite preferences for the two dimensions, like for example proactivity opposite to external locus of control and reactivity opposite to internal locus of control. But this might be a little too complex.
Let’s have a simpler approach. Let’s say you measure locus of control. You want to have two equally balanced alternatives for internal and external locus of control.
When there’s a delay in the distribution system of our product:
I check the information at the source [measures external locus of control]
I check my own mistakes first [measures internal locus of control]
Both options have equal advantages and disadvantages for the employee. What is the “politically correct answer”? If (s)he states that (s)he checks the information at the source, it could be bad because (s)he’s not thinking to his/her own responsibility in evaluating the errors and bothering clients. If (s)he states (s)he checks her own mistakes it means (s)he’s not well informed on what to find or that (s)he is used to making errors.
The correct thinking for someone who honestly does what (s)he checks for this item is:
1. If the person wants to know what mistakes to correct, (s)he must first check whether there are her mistakes. Checking his/her mistakes without having clear information on what went wrong is wasting time for the company. Here it also matters how many deliveries are handled daily by the employee and what is the rate of failure and what is the rate of delay. These pieces of information must be known beforehand.
2. If the person wants to check the information at the source, (s)he must know what questions to ask so that (s)he would figure out what is the precise cause for the effect of the delay. It shows the person knows his/her job.
In both cases, there are equal advantages and disadvantages to answering any of the variants measuring locus of control.
Building such items requests flexibility in thinking, creativity, empathy, a wide vocabulary (also reading proverbs books helps in defining choices for specific dimensions) and practical knowledge about the specific organization and the business model.
In the end, let’s think of an example for selecting relevant criteria for predicting performance in a job. A psychologist was contracted to design a test for strip-tease dancers at a bar. Of course, the only thing all the candidates had in common was the fact that they were lying like breathing. So the logical criterion for performance in this position was sociability. Indeed, the sociability construct, measured through specific items, indicated those who would bring more money to the bar. The responders must know as little as possible about what you’re after. You can ask 10 questions and just follow the real answer for 2 of them. It may not be efficient, but it’s effective.
In conclusion, questionnaire design is not a joke, especially in HR. You must prove you respect your work, the results you provide, the responders’ position by stepping into their shoes and to the education/knowledge you’ve got. If you don’t understand some of the terms I used in this article, go google them, wiki them, read about them in books, research abstracts and online journals before conducting any kind of HR research in an organization.
Author’s experience in questionnaire design. The Author has worked as a member of a research group developing organizational evaluation tools. From November 2006 to June 2007, he contributed to five questionnaire designs in students’ groups. Also, he has designed the first psychological questionnaire for application in political branding using NLP (Iaşi, 2004), for which project he was awarded the first prize at the students’ contest EconomMix in 2005, the management-marketing section.
Besides that, he has designed psychological questionnaires for own researches on parental education (2005), self-esteem (2007, 2009, 2011), insurance (2009), memory (2009), he enhanced an evaluation questionnaire for career consulting (2008).
The author can be contacted for questionnaire design consulting services at marcus.victor.grant [at] gmail [dot] com
Copyright © Marcus Victor Grant, 2012-present