I too would question the people recruited – in fact I met a guy recently who excitedly offered himself as a willing tester for our agency, claiming to be a regular tester for a certain remote testing site I won’t name. The very fact that he’s now a regular tester should rule him out of any further research. I’ve seen it myself in videos where participants, knowing it’s a usability test’ keep mentioning ‘usability’ and their focus on the task is hindered by their focus on finding usability issues.

Great research should never just be about finding usability problems, it should be about understanding expectations, mental models, current behaviour, wants, needs, experiences…. It’s only then that you can uncover the much bigger picture.

There’s little point testing with participants that are not the target audience. UX researchers will always have a better understanding of the target audience over an external recruiter. The researcher also has a vested interested in engaging with the right users – as you don’t want to waste your time and money on duff users.

Interesting post. I agree with you in that whenever we have used remote testing services, it seems that the tests have almost always been completed by professional participants who spend a lot of time testing sites. While this is obviously NOT ideal in terms of usability testing with representative end users, this can still be quite useful in terms of getting early non-domain specific, UI/IxD feedback that can catch any general usability issues. These can then be fixed and the solution tested with REAL end-users, rather than via a remote service – this is useful therefore if your access to real end users is in limited supply. Steve Krug actually makes the case for this approach in his latest usability testing book, Rocket Surgery Made Easy, when getting enough end users is a problem.

Otherwise, I totally agree with you. Testing should be with real end users completing tasks based on real, research-based, user needs identified early in UCD process and our experiences are that this isn’t typically the case with remote testing. Thanks for the link to ethnio though – I will definitely be checking that out! :)