Part 3 in our series, So Many Variables, So Little Time: A practical guide to what to worry about when conducting multi-country studies
In multi-country studies, untruthfulness presents the largest potential corrupting influence to accurately measuring any form of low incidence personal behavior, second only to basic cross-cultural response bias.
Our recently published paper, Dimensions of Online Survey Data Quality: What Really Matters?, discusses the results of two large-scale multi-country survey experiments interviewing more than 11,000 respondents in 15 countries that tested untruthfulness, along with several other factors, in a treatment versus control group approach. Our goal was to understand the impact of untruthfulness on the quality of survey data, and how to deal with this issue to accurately compare data across countries.
First, we needed to define “untruthful.” We based our definition on a technique I developed with Eli Drake, GMI’s Director of Modeling, to measure authenticity of responses from each respondent. A detailed explanation of this technique can be found in the white paper. Briefly, an untruthful respondent is one who answers yes to a high number of improbable questions, which we calibrate using known sample and population benchmarks. Our research found that truthfulness varies greatly by country and is directly related to culture. In cultures with high levels of corruption, as measured by the World Bank’s corruption index , higher percentages of online respondents fail our honesty detector.
We are exploring other factors that influence untruthfulness, such as incentive to lie from anticipation of rewards for qualification and completion, respondents’ views of market research, and whether participants care if their voices are heard. Research to date indicates lying varies between 3% and 30% from country-to-country and is a major risk factor to accurate data.
Some Question Types Suffer More Impact
This matter impacts some questions much more than others. It is most prominent when asking about a high status activity like owning iPads, visiting Harrods or reading Vogue magazine. In our experiment, we measured upwards of 100% overclaim among the untruthful for these types of questions compared to the totally truthful group.
It has less dramatic, but nevertheless significant general influence on any question surrounding personal behavior (e.g., have you washed your hair today), but is a fairly benign issue when recording attitudes toward issues. The question least influenced by removing those who over-report, interestingly, was where respondents had to bet some imaginary money on their choices.
How to Mitigate Untruthfulness
Because a significant proportion of the population will honestly answer yes to one, two or three of our low incidence questions, removing all of these respondents from the sample would actually introduce bias and remove honest survey takers. Our suggestion is to consider screening out respondents who answer yes to four or more low incidence questions and flag any respondent in the data who answers yes to more than three. Moreover, we would take note of any differences in how this group answers sensitive questions that may be effected by untruthful respondents.
This is part 3 in a series. If you would like to read more, visit:
Part 2: Vive la Difference, the impact of cultural factors on multi-country studies
- See more at: http://www.ls-gmi.com/data-quality/pants-on-fire-testing-untruthfulness/#sthash.PMqiKkPk.dpuf
