Why participant recruitment is the biggest risk in UX research
How to defend against growing threats to research quality.
Everyone’s talking about AI in research right now (including me!)
Research teams are under pressure to do something with AI. Stakeholders are asking how it will speed up research or improve insights. AI tools are emerging seemingly every day, creating both opportunities and threats.
This is the burning platform for research teams right now. They need to be seen using AI and finding practical applications for it.
But there’s another big challenge in UX research, which is perhaps under the radar of many people: participant recruitment.
Quality participants are everything
Participant recruitment is the essential ingredient in UX research. If you don’t have the right people to talk to, it doesn’t matter what technology you’re using. The insights you gain won’t be valid or will be of lesser quality.
It doesn’t matter if you’re using AI to speed up analysis if the people providing the insights aren't the right ones. All the technology and process optimisation in the world doesn’t matter if your participants aren’t suitable.
The quality of participants is the foundation upon which everything else is built. Get this wrong, and everything that follows is tainted.
Yet recruitment gets little attention because it’s seen as boring, solved and unsexy. Unlike AI, which is flashy and new, participant recruitment has been around forever and doesn’t change often. Plus, most researchers outsource it to panels or agencies, so they don’t fully understand the challenges until they’re face-to-face with a participant or watching an unmoderated research recording.
Big panels have their limits
Almost all established research teams have access to a major platform like UserTesting, giving them tools and access to a panel with millions of participants.
Yet when I’ve talked to research leaders, they often express their dissatisfaction with the quality of these panels, despite paying a six-figure annual fee.
Large panels are great for broad consumer products and when you need feedback quickly, but they have significant limitations:
Serial respondents can undermine quality. Many participants on large panels are serial responders. They’re there primarily to make money online, not because they match your criteria. Their motivation is financial first, fitting your requirements second.
When a traditional recruiter reaches out to someone new, their primary reason for selection is that they match the criteria. The incentive is just that: an incentive to participate. But on large panels, people sign up primarily to earn money.
Niche audiences can be poorly served. Big panels work well for broad consumer products, but struggle with niche requirements, especially B2B. If the people you need aren’t on the panel, you’re often left to bring your own participants. Some panels offer active recruitment but this is rare.
Fraud risk is increasing
As Dr Maria Panagiotidi notes, “several overlapping factors have made qualitative research more vulnerable to fraud”.
Remote research methods, accelerated by COVID and platforms like UserTesting, have reduced friction but also made deception easier. Without face-to-face interaction and ID checks, participants can more easily misrepresent themselves.
AI is making this worse. With tools like ChatGPT, people can generate believable screener responses or interview answers without real experience. This is especially bad for quant surveys, which AI can easily answer in a plausible way.
With AI moderation rolling out via various products and platforms, there’s even the prospect of AI participants talking to AI moderators.
Tactics to safeguard your participant recruitment
Don’t take recruitment for granted or assume that it’s someone else’s problem.
- Use multiple sources. Don't rely solely on one massive panel. Have a range of options so you can pick the best method for each use case. Sometimes you need speed and quality is less critical. Other times, especially for strategic research, you want more bespoke recruitment where you can better guarantee quality.
- Tighten your screeners. Think carefully about how to prevent fraudulent participants from misrepresenting themselves. Avoid screener questions that can be easily guessed. Assume bad faith when you’re reviewing them.
- Check how participants are verified. How is the platform or recruiter you’re using validating that people are who they say they are? Some do this a lot more rigorously than others. For example, if you are looking for owners of a car brand, how do you know that they actually own that car? Platforms might rely on self-reporting, while traditional recruiters will ask for proof.
- Be cautious with unmoderated and quant tools. When you’re not directly interacting with participants, it’s harder for you to detect fraud. Tools have varying levels of fraud-prevention. Make sure you understand what defences your tools have in place against fraud (e.g. detecting when someone pastes a response into a text box).
- Consider expert networks for B2B. While these have the same issue with serial respondents, they often verify people better than mass-market panels. However, they can be expensive and restrictive about which tools and methods you can use.
- Build your own panel. For B2B especially, intercepting your customers and getting them to sign up for research can provide better access to the right people. Tools like Ethnio can help, although this requires significant investment.
The future of participant research
What research teams need when it comes to participant recruitment hasn’t changed:
- Quality: the right people, who are who they say they are.
- Speed: so researchers can keep up with the pace of product development.
- Scale: so they don’t have to manage 10 different vendors.
- Cost effectiveness: sensible prices that scale with demand.
- Tools: to run research in different ways, using different methods.
You can’t get all of this from one platform yet – although there are companies like Askable that are heading that way – so research teams inevitably need to have a number of options at their disposal.
Research teams are understandably focused on everything AI at the moment, but while this is a valid thing to explore, they also need to not take their eye off the ball with participant recruitment. Unlike AI integration – which everyone's talking about – recruitment quality is often taken for granted.
Yet the people you gather insight from are the foundation of good research. If your participant recruitment process is flawed, the entire design process built on top of it is undermined.
