If you genuinely are in the business of designing for a very wide user base, you should segment your users to make sure you are addressing all their needs.

Having said that, if you have really poor usability, pretty much any old random person can be used to point that out. Right?

The big problem I experience is clients taking a market research attitude to recruitment. They then get all intricate about things that won’t make a blind bit of difference.

It depends entirely what you’re trying to develop. Especially with websites built for non-specialized audiences, there is a mountain of useful information you can get from non-representative users.

You bother if you don’t have the budget or time to perform a formal test. In other words, if it’s the difference between doing no test and a test with non-representative users, perform the test.

There are many clients who still subscribe to the myth of the genius designer (or the genius in themselves), so they’re reluctant to invest the time and $$ for testing.

Your point about requirements analysis is of course true – but the idea that requirements analysis is a phase that is timeboxed right at the beginning of a project, then is never thought about again – that’s an old school software engineering concept. When you’re using a UCD process, requirements analysis is done mostly at the beginning of a project, but then continues through the course of the project in a diminishing manner as you iteratively prototype your product. Starting off by showing real users early low-fi prototypes is an excellent way of validating and nuancing your requirements.

In fact, the case study seems more like a case of poor requirements analysis. Building software that has no use is a different failure to that of building useful, but difficult to use, software.