When designing questions, a known problem is the façade effect. The responders tend to understand what the researcher wants to discover and give “politically correct answers”. Although it might be difficult, there are ways to circumvent that. It’s all in mastering the words and the language so that the answer reflects what you want to find out without letting the responder understand what you’re after.
One of the most powerful things you can do, on which many forget to act, is the simple frame of the questionnaire. Mention, either verbally or in writing, that the questionnaire is not intended to measure correctness, but rather preferences and tendencies, if you’re doing a psychological survey rather than a knowledge or a performance review. Of course, this can be done more efficiently with structured or semi-structured interviews. But for the larger frame, I’ll refer to any kind of questionnaire design.
One way of measuring the impact of the façade effect is to build into the instrument a lying scale, which will measure one’s tendency to lie through specific items. But that may be too complex for some. So instead, a simple way is to put critical questions in different parts of the questionnaire. That is, to ask the same things twice, in two different ways. Then, if you put two-three of these questions and the answers do not match, you have reason to believe the responder was not honest/serious in completing the questionnaire.
Another way is to measure multiple dimensions / psychological scales with the same item. Of course, the risk is that the item will not be interpreted the same way by everyone, and it can’t generate any pure correlations. Still, CPI, one of the most appreciated personality inventories, does it. And it does it very well.
You can also do it with different choices for the same item. For example:
1. Usually, at work, do you consider:
a) Who wakes up early gets a better job done [measures proactivity]
b) I let others do the planning for my tasks [measures planning type]
2. When handling multiple tasks:
a) Who runs after more than one rabbit and catches none [measures planning type and locus of control]
b) I’m first doing my tasks [measures proactivity and locus of control]
3. When a colleague does a mistake:
a) I let him/her correct the mistake [measures reactivity]
b) I delegate him other tasks [measures planning type]
4. How do you know when a colleague made a mistake?
a) The schedule I have planned does not meet the results [measures planning type and management style]
b) He/she is responsible for his/her own thing, and he/she can know when something is done right or wrong [measures reactivity and management style]
You can notice how easy it is to believe that sometimes opposing choices that do not measure the same thing at the same item can create confusion by forcing a choice between the two alternatives. The important thing to remember is to oppose the dimensions an even number of times for each dimension. For example, these 4 items measure: planning type, management style, locus of control, and activity type. I have opposed the planning type to activity four times; I have measured plain locus of control and simple management style once. Therefore, at the end of these four items, I will have a score for each of the categories, each having a chance to be scored one point based on any of the choices offered. In the software-designed questionnaires, you can also add questions if, at the end of the test, there are categories with the same number of points for one of the preferences. For example, let lot, the responder has an equal preference for internal locus of control and proactivity. You may add two questions, either one measuring only the locus of control and the other activity type or both measuring opposite preferences for the two dimensions, for example, proactivity opposite to the external locus of control and reactivity opposite to the internal locus of control. But this might be a little too complex.
Let’s have a more straightforward approach. Let’s say you measure locus of control. You want to have two equally balanced internal and external locus of control alternatives.
When there’s a delay in the distribution system of our product:
-
I check the information at the source [measures external locus of control]
-
I check my own mistakes first [measures internal locus of control]
Both options have equal advantages and disadvantages for the employee. What is the “politically correct answer”? If (s)he states that (s)he checks the information at the source, it could be wrong because (s) he’s not thinking of his/her own responsibility in evaluating the errors and bothering clients. On the other hand, if (s)he states (s)he checks her own mistakes, it means (s) he’s not well informed on what to find or that (s)he is used to making errors.
The correct thinking for someone who honestly does what (s)he checks for this item is:
1. If the person wants to know what mistakes to correct, (s)he must first check whether there are her mistakes. Checking his/her mistakes without having clear information on what went wrong is wasting time for the company. Here it also matters how many deliveries are handled daily by the employee, the rate of failure, and the rate of delay. These pieces of information must be known beforehand.
2. If the person wants to check the information at the source, (s)he must know what questions to ask so that (s)he would figure out the precise cause for the effect of the delay. It shows the person knows his/her job.
In both cases, answering any of the variants measuring locus of control has equal advantages and disadvantages.
Building such items requires flexibility in thinking, creativity, empathy, a vast vocabulary (reading proverbs books helps define choices for specific dimensions), and practical knowledge about the organization and business model.
In the end, let’s consider an example of selecting relevant criteria for predicting performance in a job. A psychologist was contracted to design a test for strip-tease dancers at a bar. Of course, the only thing all the candidates had in common was that they were lying like breathing. So the logical criterion for performance in this position was sociability. Indeed, the sociability construct, measured through specific items, indicated those who would bring more money to the bar. The responders must know as little as possible about what you’re after. You can ask 10 questions and just follow the honest answer for 2 of them. It may not be efficient, but it’s effective.
In conclusion, questionnaire design is not a joke, especially in HR. You must prove you respect your work, the results you provide, and the responders’ position by stepping into their shoes and to the education/knowledge you’ve got. If you don’t understand some of the terms I used in this article, go look them up, and read about them in books, research abstracts, and online journals before conducting any kind of HR research in an organization.
Author’s experience in questionnaire design. The author has worked as a research group member developing organizational evaluation tools. From November 2006 to June 2007, he contributed to five questionnaire designs in students’ groups. Also, he designed the first psychological questionnaire for application in political branding using NLP (Iaşi, 2004). For that project, he was awarded the first prize at the students’ contest EconomMix in 2005, the management-marketing section.
Besides that, he has designed psychological questionnaires for his own research on parental education (2005), self-esteem (2007, 2009, 2011), insurance (2009), and memory (2009); he enhanced an evaluation questionnaire for career consulting (2008).
Copyright © Marcus Victor Grant 2012-present, all rights reserved.
The materials on this blog are subject to this disclaimer.
Analytic Vision 