User research. It’s right there in the name. A user is someone who actually uses your service. Equally valid is the idea of a “target user” – someone who doesn’t yet use your service, but has a genuine need that it would fulfil.
User research has to involve these people. Otherwise, by definition, it’s not user research.
When you use a panel provided by remote usability testing service, you end up gathering data from a bunch of freelancers – professional participants who know the kind of things they’re expected to say. At best, they’ll make a real effort to take the tasks seriously and you feedback. At worst, they’re just actors, going through the motions to get paid.
Real user research involves recruiting participants who honestly care about the problem your service is trying to solve. They’ll engage with the information, weigh up their options in difficult decisions, and carefully consider implications. Most importantly, they’ll draw upon their life experiences to weigh up the benefits of your service against their current practices and other competitors.
One of the biggest strengths of remote research is the fact that you can cast your recruitment net far more widely than with face-to-face research. Remote research is well suited to sourcing real users. If you use remote research techniques and then test irrelevant participants, you’re missing the point of it all.
——
Edit 1: This post isn’t intended as a criticism of remote usability testing services per se. Most services offer a predefined panel alongside the option to recruit your own users. I’m advising you to recruit your own, and to take care in doing so – it’s as simple as that.
Edit 2: Nate Bolt has reminded me that ethnio is a good tool for recruiting your own users, including the ability to intercept users live off your own site. I’ve used it a few times, and it gets my vote.
Hi Harry,
Interesting post. I agree with you in that whenever we have used remote testing services, it seems that the tests have almost always been completed by professional participants who spend a lot of time testing sites. While this is obviously NOT ideal in terms of usability testing with representative end users, this can still be quite useful in terms of getting early non-domain specific, UI/IxD feedback that can catch any general usability issues. These can then be fixed and the solution tested with REAL end-users, rather than via a remote service – this is useful therefore if your access to real end users is in limited supply. Steve Krug actually makes the case for this approach in his latest usability testing book, Rocket Surgery Made Easy, when getting enough end users is a problem.
Otherwise, I totally agree with you. Testing should be with real end users completing tasks based on real, research-based, user needs identified early in UCD process and our experiences are that this isn’t typically the case with remote testing. Thanks for the link to ethnio though – I will definitely be checking that out! :)
Pingback: Marketing Day: January 18, 2012
I would also recommend doing your own recruitment and refraining from outsourcing to recruitment agencies.
There’s little point testing with participants that are not the target audience. UX researchers will always have a better understanding of the target audience over an external recruiter. The researcher also has a vested interested in engaging with the right users – as you don’t want to waste your time and money on duff users.
I’m not a fan of remote testing and will avoid it unless it’s backed up with real research. By real research I mean the use of a qualified researcher who interviews users. As someone who likes to probe and question user’s behaviour to uncover underlying motivations, I find it very frustrating watching a video, seeing behaviour or hearing an opinion and not being able to probe further. You certainly don’t get quality results.
I too would question the people recruited – in fact I met a guy recently who excitedly offered himself as a willing tester for our agency, claiming to be a regular tester for a certain remote testing site I won’t name. The very fact that he’s now a regular tester should rule him out of any further research. I’ve seen it myself in videos where participants, knowing it’s a usability test’ keep mentioning ‘usability’ and their focus on the task is hindered by their focus on finding usability issues.
Great research should never just be about finding usability problems, it should be about understanding expectations, mental models, current behaviour, wants, needs, experiences…. It’s only then that you can uncover the much bigger picture.