Alertbox 9 - User Testing is Not Entertainment

Click on the link bellow to go to the article's web site.

Summary

Don't worry about entertaining the people in the observation room, so they will focus in your prototype/website and you can observe them going through it. Test them to find out about the design. Sometimes the tasks that they will perform may seem boring to watch, but it has to be done.

Typical Errors

Screening for outgoing, articulate test participants so that you can get more elaborate quotes and comments. While it's certainly more engaging for observers to watch users who talk a mile a minute, it's also potentially biasing. Out of the 233 guidelines for recruiting test participants, the number one guideline is to get a representative sample of customers. If you're trying to sell, say, networked storage to enterprise customers, you need to test a broad range of system administrators. You can't just screen out introverted nerds, since they could be the key influencers and might well behave differently than more easygoing sysadmins.
Focusing the test on the site's sexy parts, while skipping parts that require users to read detailed descriptions of products and services. But such boring behaviors often close the sale, particularly for B2B sites where detailed product specs and whitepapers are a key part of the user experience.
Prompting users to talk in great detail about the problem your design team is currently debating. Yes, it's tempting to get input for your next decision. Such feedback isn't reliable, however, if you've turned users' attention away from their tasks to grill them about multiple, alternative design ideas. You have to watch users do things -- not listen to them talk about hypothetical designs they aren't using. It's fine to test two or three alternative paper prototypes that you've quickly mocked up. It's not fine to ask users to speculate on "how would you like it if the site did this...," because what people say has no relation to what they'd do if they actually used such a user interface.

Usability As Showbiz

In a business environment suffocated by information pollution, you must wow and persuade people to keep their attention on your message. When you're selecting highlights from usability study videos, you shouldn't include a 10-minute clip of a user visiting 20 erroneous pages. Show the first one or two wrong clicks, then throw up a transition title that says "nine minutes later...," and conclude with the clip where the user says, "This is a horrible website. I can't find anything. I'm never coming back."

To present findings, it's okay to emphasize material that engages and motivates your audience. For example, don't show a bunch of clips of your most introverted participants.
To discover findings, however, you can't bias your study and skip directly to the good parts for three reasons.
- First, if you don't let users go through those 20 wrong page views, you won't learn as much about the flaws in navigation design.
- Second, if you guide users past "boring" pages, you won't learn how users interpret those parts of the site.
- Third, if you take users directly to the site's "interesting" parts, they'll arrive at those areas with a substantially different mental model and attitude than people who've had to struggle through page after irrelevant page to get there. Users are more inclined to quickly leave a section if they've struggled to reach it. In contrast, users easily guided there will have a more positive attitude and will dig harder; the section will (falsely) appear to be more successful as a result.
To document findings, your report should focus on the importance of the usability issues for the overall business, not on which issues generated the most colorful quotes.

In the long run, money talks, and you optimize ROI by emphasizing unbiased research and by watching behavior. Usability's role in a design project is to be the source of truth about what really works in the world, as opposed to the team's hopes for what might work. The less thoroughly you uncover the truth, the less your company will need you in the long run.

Sunday, March 30, 2008 1:32 AM