Why I never use panels provided by remote usability testing services

User research. It’s right there in the name. A user is someone who actually uses your service. Equally valid is the idea of a “target user” – someone who doesn’t yet use your service, but has a genuine need that it would fulfil.

User research has to involve these people. Otherwise, by definition, it’s not user research.

When you use a panel provided by remote usability testing service, you end up gathering data from a bunch of freelancers – professional participants who know the kind of things they’re expected to say. At best, they’ll make a real effort to take the tasks seriously and you feedback. At worst, they’re just actors, going through the motions to get paid.

Real user research involves recruiting participants who honestly care about the problem your service is trying to solve. They’ll engage with the information, weigh up their options in difficult decisions, and carefully consider implications. Most importantly, they’ll draw upon their life experiences to weigh up the benefits of your service against their current practices and other competitors.

One of the biggest strengths of remote research is the fact that you can cast your recruitment net far more widely than with face-to-face research. Remote research is well suited to sourcing real users. If you use remote research techniques and then test irrelevant participants, you’re missing the point of it all.


Edit 1: This post isn’t intended as a criticism of remote usability testing services per se. Most services offer a predefined panel alongside the option to recruit your own users. I’m advising you to recruit your own, and to take care in doing so – it’s as simple as that.

Edit 2: Nate Bolt has reminded me that ethnio is a good tool for recruiting your own users, including the ability to intercept users live off your own site. I’ve used it a few times, and it gets my vote.