How many sessions should a stakeholder observe?
In an ideal world, the stakeholder would be able to observe all of the usability test sessions live. The stakeholder would get to see those that succeed with ease, and those that may have a fair bit of difficulty. Often, however, this doesn’t happen. The stakeholder has limited time and can only sit in on a few sessions.
Who ends up being a good participant?
It’s not a good idea to try to predict how good a participant will be at the usability test before they begin, for this preconceived expectation can lead to moderator bias. Besides, the predictions would often be wrong anyway. Sometimes those who on paper seem like they’d be the ideal users end up struggling, and those that have a background that would seem to lead to more difficulty with the site end up succeeding.
I did a usability study recently, and the client only got to observe one participant. While most of the participants were able to use many features of the site, the one time slot that the client chose to observe ended up being with the participant who had the absolute most difficulty using the site. I felt bad for the client during the session, imagining how discouraging this might seem. I told the client after the session that while the participant did hit on some common points of confusion, she shouldn’t be too discouraged because this participant did have an unusual amount of difficulty. The client responded that she understood, and even found the session very interesting. In some ways, I guess the exaggerated errors that this participant was having probably did make for more interesting observation.
If only I could pick the participant for the client to observe!
It got me thinking, though – if I could know beforehand which participants were going to succeed with ease, which were going to crash and burn on the tasks, and which participants were going to be middle of the road, which one would I tell the client to observe if the client could only observe one session?
The perfectly successful participant would not be of much value. A report detailing problems that other participants had might be a little harder to accept. The crash and burn participant might be the most interesting to observe, but could also be discouraging. So, I’d select the middle of the road participant. Actually, I’d select two or three middle of the road participants because they are likely to have at least some differences in the issues that they experience but may have some useful similarities as well. I can’t predict, of course, so I can’t tell the client who the best participants will be to observe. I’d recommend, though, selecting a minimum of two or three, which will at least give a better sampling of the kinds of experiences that participants are having with the site.
A highlights tape could do the trick.
That said, particularly when stakeholders can’t observe many of the sessions, and time/budget permits, I like to suggest a highlights tape. While a highlights tape does add some extra effort on my part, it is often a great way to show both the highs and lows of the activities across many of the participants. The highlights tape can often by ordered in the same way as a report too, meaning that it’s easy to cross-reference observed issues.
Make sure to set expectations.
Ultimately, though, the expectation that should be set with any stakeholder who chooses to observe a study is that while it may be easy to jump to immediate conclusions after a single study session, the point of the analysis afterwards is to look at issues across participants and see what really jumps out as key issues.
Image Courtesy of Teerapun / Shutterstock
