Kantar's Profiles Blog

A dozen years ago a debate raged in the marketing research community over the switch from probability sampling methods such as telephone RDD to nonprobability sampling methods as are typical with online access panels. In the interim years, most clients moved to online samples but there are still some that cling to probability methods. However, we now see the quality of probability samples being questioned because of low response rates for RDD. In an interesting twist, the very same techniques that nonprobability samples use to weight and model data now often need to be done on probability samples to account for nonresponse bias. 

Research has consistently shown that all panels are not the same. Recruitment sources and management practices vary, and this can cause differences among panels. Beyond panels, there are other sources of online survey respondents, such as river, dynamic, and social media sources – and these can produce data that is different from each other, as well as different from panels. Given the wide variety of sample sources, and their benefits and drawbacks in cost and quality, researchers often struggle with the question, “How can I blend in other sources without impacting my data?”

The conversation around conducting surveys with online panels rather than face to face or CATI is one that comes up every day for me in South East Asia. Whilst this is an entirely valid discussion, online actually adds ways to ensure trustworthy and valid respondents. In this mini-series, we unlock the truths behind these myths.

Knowing who someone really is isn’t exclusive to our industry. If we look at the everyday world, people can look us in the eye and lie. We filter this, both consciously and subconsciously and take forward what we have confidence in being true. Now back to research. As a student, I had friends who would get calls from focus group recruiters offering them 5000 Rs ($80 USD). The next question would be 'do you like/ consume ‘x’ product?' They would agree to anything to get into the focus group and take the money home, including saying they drank Malibu regularly (I’m not sure anyone can say that hand on heart!) Whilst these recruiters are not a fair reflection of all in the industry, online uses pre-screeners that are far more subtle than this tactic (and far less heavily incentivised, granted) which encourage a level of honesty from the outset.

According to a recent GRIT Sample Report* presented at SampleCon in January, 81% of industry buyers and sellers believe traditional panels are dying. At the same time, over half believe traditional panels are the gold standard. Traditional research panels do bring advantages that yield data quality; perhaps most important, they are not associated with loyalty programs. Rather, panelists are sequestered solely for the purpose of taking surveys which minimizes bias. Other important benefits include validation of people’s identities upon registration as well as extensive profiling that can be used not only for targeting, but also to shorten surveys by appending data. Double opt-in panels also facilitate the integration of behavioral and attitudinal data sets. 

As online research has transitioned over the past 15 years from random digit dialing or door-to-door methods to online, a shift from probability to non-probability sampling has occurred. To assuage the concerns of researchers, rigorous sampling methods were developed to ensure representative samples. Despite this, however, there has been a rapid change in sampling practices as respondent efficiency became the focus of fieldwork companies and respondent routing replaced traditional sampling for much of the market research industry.