90 percent of everything : Usability Blog

Written by Harry Brignull

Objectified is a feature-length independent documentary about industrial design by Gary Hustwit, the same guy who did Helvetica. If you haven’t already seen the trailer, here it is.

Read more here.

In a recent post on Boing Boing, Clay Shirky talks about the user research approach used at meetup.com:

[…] Scott pulled me into a room by the elevators, where a couple of product people were watching a live webcam feed of someone using Meetup. Said user was having a hard time figuring out a new feature, and the product people, riveted, were taking notes. It was the simplest setup I’d ever seen for user feedback, and I asked Scott how often they did that sort of thing. “Every day” came the reply.
Every day. That’s not user testing as a task to be checked off on the way to launch. That’s learning from users as a way of life.
Andres Glusman and Karina van Schaardenburg designed Meetup’s set-up to be simple and cheap: no dedicated room, no two-way mirrors, just a webcam and a volunteer. This goal is to look for obvious improvements continuously, rather than running outsourced, large-N testing every eighteen months. As important, these tests turn into live task lists, not archived reports. As Glusman describes the goal, it’s “Have people who build stuff watch others use the stuff they build.”
Mark Hurst, the user experience expert, talks about Tesla — “time elapsed since labs attended” — a measure of how long it’s been since a company’s decision-makers (not help desk) last saw a real user dealing with their product or service. Measured in days, Meetup approaches a Tesla of 1.
Glusman and van Schaardenburg have also made it possible to take Jacob Nielsen’s user-testing advice — “Test with five users” — and add “…every week.” Obstacles to getting real feedback are now mainly cultural, not technological; any business that isn’t learning from their users doesn’t want to learn from their users. […]”

While reading this I found myself nodding. Outsourcing a small lab-based user research project can cost £10-12,000 (depending on your supplier and the project details), so it you don’t need to many each year before it makes sense to bring it in-house. Rapid, iterative research is something I’ve blogged about enthusiastically before, and it’s a very effective approach. But there were a couple of points in Clay’s article that I just couldn’t swallow.

Test with 5 users every week.
5 users a week? Or every day? This sounds a bit like announcing that everyone should bench-press 500 lbs as part of their gym work-out just because you do. 5 users a week is impressive, but there’s no need to feel intimidated by it - the scale of your research should match your needs. For many web businesses 5 a week is too many and you’d be doing great if you achieved half or a quarter of that. Let’s think it through - to achieve 5 a week we’re talking at least one solid day of research sessions each week. As well as the researcher and the user, you also need decision-making stakeholders in the viewing room (otherwise you are back to the old fashioned giant-report-and-video-analysis scenario). This means you have to take some of the best members of your team out of their normal working life for 1/5 of their working time. That’s pretty expensive and you’d need to consider expanding your team to make up the loss in man-hours. Research and design are like Yin and Yang: they need to be balanced and they need to work together. In the rush to address the imbalance, you can push the scale back over too far onto the research side.

TESLA - Time elapsed since labs attended
If your TESLA is 6 months, you know you’re being naughty. But should you just be counting lab research? Isn’t that a bit like only doing bench-presses to stay healthy? There are lots of other exercises you should be doing.

In response, I propose RMU: “Range of Methods Used”
If your RMU=1, then you know you aren’t being as effective as you could be. Face to face user testing is just one approach amongst masses of others - and there are many times when these other approaches are more appropriate. In fact, I’m pretty certain that Meetup must be doing lots of other kinds of research, but it didn’t come across in Clay’s short article.

So what other research methods am I talking about here? Off the top of my head - diary studies, cultural probes, A/B testing, multivariate testing, analytics, co-discovery sessions, participatory design workshops, surveys, heuristic evaluation, walkthroughs, open, closed and reverse card sorting, remote usability testing, eyetracking (sometimes), telephone interviewing, ethnographic field work, NPS and other one-click feedback tools. All of these have many variants - the list goes on and on.

Now don’t get me wrong - face to face user testing is a wonderful method, but sometimes it’s inefficient, expensive and can even deliver weak findings if misused. Here’s an example. Last week a client came to me complaining that he needed to validate a large category scheme for his site. The scheme covered about 35 different industry sectors, and so he said he would need to run the testing on 35 different types of users (after all, you would need someone in accounting to tell you whether your accounting IA made sense to them). He was complaining that as a result, he simply couldn’t afford to do user testing. The solution to his problem was of course to escape from the mindset that “research = face to face user testing”. In his case, he could save a lot of time and get more reliable findings by running a remote user test, using a tool like User Zoom’s self serve.

Heading into the 2009 recession, we have to be smarter about the way we use our research budgets. If your RMU=1 for 2008, then it’s definitely time to break out some new methods.

Chalkmark aims to address the “challenge of user testing when there is minimal budget or time” - a worthy goal heading into cash strapped 2009. It’s made by Optimal Usability, a New Zealand based company who also make OptimalSort. Chalkmark is currently free and in still Beta, though the site blurb implies that it will be a freemium service once it takes off.

How does Chalkmark work?

• You upload a series of still images as the stimulus for your study.
• These can be scanned paper prototypes, wireframes, or anything else you can convert into a still image.
• For each one, you are able to ask a question, of the form “Where would you click to do X?”.
• When the user clicks on their desired location, the coordinates of the click are saved, and the next question/image pair is loaded.
• The test is given a URL which you can distribute any way you please for recruitment purposes (it would probably work quite well in tandem with Ethnio).
• The output is a heatmap for each question, showing where the users clicked.

Chalkmark’s current limitations:

• It seems to be still in Beta (I’ve experienced a few bugs, but I’m sure they’ll be ironed out soon).
• There’s no way of screening, profiling or segmenting you participants.
• You can’t add any other kinds of questions (there’s no comment or questionnaire component).
• You’re limited to still images, with one question per image. You can’t add branching logic, so interactive prototypes aren’t possible.

This clearly isn’t a one-size-fits-all research tool, but does look like a clean, lean and well designed web app that I’m sure would be very helpful in certain situations. Also coming soon from Optimal Usability is TreeJack, a reverse cardsorting tool that helps you validate large IAs by giving tasks to real end users. I have to admit, though, personally I’m holding out for Webnographer from Fera Labs to be launched so I can start doing some full fat remote usability testing in the new year.

Having problems finding the budget for costly usability testing? Here’s your solution.

If your users can solve this CAPTCHA, they will be able to cope with anything you throw at them!

Thanks to James Wragg for the link!

Looking back at my last 12 months of analytics, here are some of my most popular posts:

My presentation on Out of Box Experience Design: this is all about designing the moments between peeling off the shrink-wrap and getting your new gadget up and running. I gave this talk with Pete Gale of CogApp at UX-Brighton a few months ago.

User Centred Design is dead? Which bits?: this debate seems to go on and on. We’ve been talking about it at some of the UX-Brighton gatherings - a lot of the confusion seems to tie into how you define UCD and the alternatives. Have you read We tried baseball and it didn’t work by Ron Jeffries? It’s amusingly relevant to this debate.

The Boxing Glove wireframing technique: be proud of your barely legible scrawling and your inability to draw straight lines without a ruler. It’s a method, see!

Are you doing your user research on the right people?: this article is a bit of a backlash against all the articles that claim it’s a good idea to recruit random people to participate in your user research. It’s not. Although it’s better than nothing, street recruitment is the bottom of the barrel.

A review of Clearleft’s Silverback: I’ve recently met a few people who say the only usability testing software they’ve ever used is Silverback. This is pretty impressive since it’s so new - and it seems to be all down to it having a well placed price point and having a dead simple set-up process.

Which one of these shopping basket pages perform best?: this article reports on the findings of a multivariate test that Maxymiser carried out for Laura Ashley.

Designing end-to-end user experiences: why it’s worth looking beyond the “ends” of a designed experience.

Rapid Iterative User Testing: clients often tend to like the idea of “one big usability study”. It’s actually quite inefficient. Research and design work best when they dance together.

I remember when I was a lad, Google was the world leader in minimalist UI design. Mind blowingly, paradigm shiftingly, amazingly clean, elegant design, where every unnecessary pixel was carefully whittled away using Occam’s own razor, leaving nothing but perfection.

Looking at the new Google Reader, I can only hold my head in my hands and weep. What is going on with all the rollovers? It practically needs an epilepsy warning!

(The short clip above shows the new Google Reader on Chrome 0.4.154)