“You created a bad survey!” exclaimed a participant as we started off day 1 of a recent usability study. I tried to explain gently that the web application she was working on wasn’t actually a survey we had created but the product we were testing, and the reason that we brought her in (and were paying her $200 for one hour of her time) was that we wanted to figure out how to make it better before it actually goes live. “Well,” she said in frustration, “you shouldn’t let people try it until it’s much better.”
Although she, perhaps, had less patience than other participants, her level of frustration was indicative of not only a complex web application that needed usability testing, but a site that was riddled with bugs and some data errors in the test data as well. There were so many issues, in fact, beyond just straight usability problems, that the two of us on-site were feeling uncomfortable with how stressed all the participants looked as they participated in this study.
That got me thinking about how important it is that participants feel comfortable and safe during a usability study. Participating in a usability study should not be causing the level of frustration that these participants felt.
Not only is it uncomfortable as a moderator to have participants who feel uncomfortable, but this discomfort can hurt the integrity of the data. Their discomfort distracts them from the tasks at hand, and in this case, the numerous bugs and errors with the sample data were at times so much of a distraction that it was harder to identify all the actual usability problems vs. problems caused when participants got sidetracked by bugs or other unexpected anomalies.
So what can the usability team do to prevent this level of frustration?
Set expectations with the development team
It’s true that usability testing can happen with any level of design – from design comps, to low-fidelity click-through prototypes, to high-fidelity fully functional sites. But particularly with those fully functional sites, bugs should be squashed before testing begins.
Explain to development teams that while it’s always possible that we’ll encounter a bug or two when doing usability testing since participants may do something entirely unanticipated, the usability testing itself is not the place to inventory the bugs. So, to whatever extent possible, the team should do their best to reduce these potential distractions to participants before testing begins.
Thoroughly review the system against the tasks
I had seen an early version of the system when I flew in to be onsite a month before testing, so based on that, we were able to pull together a first draft test plan. I had worked out with the client that given the complexity of the system, I wanted to spend a few days with it on my own to fully validate the test plan. We agreed that they would move the application to a public facing server a week prior. While they were able to do this, they weren’t able to get the test data (including a login for me) loaded and working properly until right before testing began, so I had a very limited opportunity to test most things out.
Given that I didn’t have the anticipated (and critical) advanced access, in retrospect, a solution could have been using a shared screen with the development team a few days prior via GoToMeeting, WebEx or Hangouts. It would certainly have been more clunky than me doing it on my own, but at least it would have uncovered some problems with the site and test data in advance, before it was put in front of participants. The caveat, however, is that some bugs could have emerged when the system was moved to a new public-facing server and access to the application on that server is what took the development team until the final hours.
Users should be fully representative
While the recruit for this study was very narrowly targeted on a specific user base with a specific set of knowledge, it turned out that for three of the tasks, only an even narrower subset of people had the requisite knowledge. Knowing that only this narrower subset had the knowledge would not have helped in the recruit (they would have been nearly impossible to find) but it would have helped me know not to include those tasks that ended up frustrating participants for being so far out of their wheelhouse.
As soon as we realized how out of sync those three tasks were for the recruited participants, we adjusted to supply more context. But even supplying additional context wasn’t enough, and in discussions with the client, we eventually decided to eliminate these tasks for the remaining participants.
The lesson here is that participants need to at least have the background knowledge and experience to understand what they are being asked. If that fails, consider supplying additional knowledge. Although you run the risk of biasing results by supplying too much knowledge, results from participants who don’t have knowledge that real users would actually have are potentially invalid anyhow.
Avoid distractions in the physical space
For the purposes of this study, we rented two hotel conference rooms: one for testing and the other as an observation room for the client (with high-quality picture-in-picture streaming to a big-screen monitor via Morae software). By and large, aside from some HVAC issues heating up the room as the day went on, this was a good space, and at least participants could focus on the computer and not be distracted by other goings on around us. We did try to open the door to the testing room at several points to cool things off, but with a boisterous religious convention also taking place in the hotel, this solution never lasted too long.
I’d caution that while I’ve seen people do studies as “mall intercepts” or ad-hoc recruits at a local Starbucks, these always come with a caution that the distraction of the space can mess with result validity. I frequently do testing “in the field,” but even then I still try to find a quiet space without visual or auditory distraction.
Put participants at ease
After the first few sessions of this four-day study uncovered many unexpected site and data setup failures, we realized that there was only so much we could do to prevent participant frustration with the system, and we did the best we could with the situation in which we found ourselves. While I always try to present questions and comments neutrally (e.g., “What do you like or not like about…”) in this case, the bugs and data errors were so prevalent that we let participants know in the introduction, in general terms, the kinds of frustrations that they would encounter. During testing, when these frustrations did occur, we did our best to keep a light tone to the conversation and reinforced to participants over and over that it wasn’t them – the system was extremely buggy. Yes, we knew that we risked bias, but given the high level of frustration that was leading to the point of distraction anyway, we felt that this was the best thing to do in a difficult situation.
On a more routine note, we also had the hotel supply coffee, water bottles, and snacks for participants. While never absolutely necessary, this is always a good thing when feasible to help make for more comfortable participants. My only caution here: always think about how crumby or greasy food items are!
Perfection is rare, so go with the flow
No matter how much advance planning and checking you do, remember that perfection is rare. It’s unlikely that everything will go absolutely as smoothly as you’d like. So plan for what you can, and include contingency plans when possible.
Simultaneously, set stakeholder expectations appropriately to whatever extent you can, letting them know that no matter how much planning everyone does, there may be things in the study that need to be adapted or modified during testing.
I believe that in-progress study modifications to help understand actual usability can be far more valuable than task analytics, for example knowing exactly how many people successfully completed the exact same task. In fact, for multi-day tests, I always let clients know that if they see low-hanging fruit that they want to adjust between testing days, that’s okay – just as long as they keep us on the testing team informed about what changes they have made.
And finally, I’ll end by saying that while everything going smoothly is wonderful, there is also something satisfying about dealing successfully with unexpected or surprising situations, even if initially frustrating. With this study, even with all the frustrations we encountered, we were still able to give the client a full report with valuable information and only a few additional caveats, primarily suggesting running a few more participants once the bugs plus the identified batch of usability issues have been addressed.
Comfortable participant image for this post composed by and featuring Eliana Lebson.
