We talked to over 100k respondents last year. We threw out over 25k because they didn’t meet our quality standards.
Hard-to-reach audiences are ripe for fraud. The incentives are high, so bad actors are constantly trying to game panels and find their way into our online surveys with bots and survey farms. If you are not evaluating and removing respondents throughout the fielding period, you likely have a significant amount of questionable (if not downright fraudulent) data.
Over the past 20 years working with highly specialized audiences, Illuminas has developed a series of criteria designed to identify potential bad respondents. No one is automatically excluded off a single metric – that’s not always indicative of quality issues – but as soon as one of these “wires” is tripped, we evaluate all their answers before deciding whether or not to include them in the data set:
Quality Alarms:
- Length of interview
- Straight-lining responses on scale questions
- Excessive use of “don’t know”
- Consistency of responses throughout the survey
- Inclusion of knowledge verification questions
- Review of open-ended responses
In addition, everyone answers all screening questions before being terminated so cheaters cannot easily determine a path to qualify.
Most difficult to reach audiences are not prevalent on double opt-in panels and are practically unreachable with river and router methods. So how do we find them?
The good news is there are still quality panels that specialize in niche audiences (such as B2B). Every research agency needs dedicated resources or partners that are constantly searching for and vetting traditional panels, specialty panels and expert networks. And, a good source is going to be transparent about how they source, incentivize and validate their panel. Most will run incidence tests or small field tests so you can see if the source is a fit for your audience.
TIP #1: When fielding with the most hard-to-reach audiences, you will get the best results when you do not put a large sample size requirement on any single panel but instead get 50-100 completes from different partners. Combining large double opt-in panels, specialty panels and expert networks can be very effective.
TIP #2: Direct email invitations to double opt-in panels are preferred – we do not permit the use of river or router. Research on research has shown that you get higher quality results and fewer quality removals from double opt-in, direct email invite panel sample.
TIP #3: Most panels aggregate their panels with others, and many providers of panel sample are only aggregating other sources. Don’t allow them to use partners unless you know who the partners are and that they have been vetted. Once you are informed you can decide if you want to use this as an opportunity to vet a new sample source.
Related Pages
Making complex data digestible
Let’s face it, 150 slides of bar charts and pie charts probably isn’t going to get you what you need. Complicated data sets demand custom visualizations to be digestible for an executive audience.
Being a partner, not a vendor
We view ourselves as an extension of your team—we know your industry, we know your company, and we understand your customers. Let us do the heavy lifting so you can focus on the 500 other things on your to-do list.