One of the most exciting things for me when doing user research is having opportunities to interact with both stakeholders and participants in person. Thus, I gravitate to the kinds of projects where these interactions occur.

Most frequently, at least some of the stakeholders who are involved in the product design attend the usability test sessions that I moderate. If stakeholders are on the fence about attending or are pressed for time, I suggest that they attend as many of the sessions as is possible, particularly the earlier sessions if they are able. This allows them a greater opportunity to let me know if the test plan, once put into real use, is giving them the kinds of answers they are looking for.

Here’s a summary of some of the recommendations that I make to stakeholders who want to observe sessions live and in person.

Stakeholder Observation in the “Lab”

When I rent a market-research type facility for a study, I will have a room with one-way viewing glass that makes it easy for several stakeholders to observe not only the session through the glass but also with a streaming view of each participants’ computer or mobile screen.

In-person logger triage

Typically, when studies are local or if the budget is big enough for two people to travel, I come to the facility with an on-site logger who will often sit in the observation room with the stakeholders.

In this controlled situation, the logger keeps a chat client open to communicate with me as I’m moderating the sessions. If an observer has a specific question or wants me to add an additional probe, the observer relays it to the logger who in turn can decide if it’s the right time to let me know; if not, the logger could let the observer know that it’s not the right time, either because it would bias a future task or be problematic for another reason, or because the logger knows that it’s going to be probed about later.

Direct stakeholder communication

Sometimes, whether or not the logger is in the room with them, observers want to communicate with me directly via a chat client, and I offer this with a few caveats:

  • They should limit to only what they consider essential communication.
  • I can’t promise that I’ll see the message immediately if I need to focus on the participant.
  • Even if I see a request, I may not be able to work it into the session immediately.

Beware of laughter, loud talking and bright lights

Even though I tell participants as part of the introductory script that others may be observing, unless they explicitly ask, I don’t go out of my way to explain how those people are observing. So I tell observers to be aware that semi-soundproof glass doesn’t mean that noise is completely isolated between the two rooms. Even a hollow ceiling can conduct sound to some degree.

Don’t laugh: I warn stakeholders that no matter what the participant says or does, and no matter how funny it seems to them, laughter can carry, and a participant could catch on quickly to observers laughing at a mistake that they realize they made.

Talk quietly: I also ask Stakeholders not to talk too loudly. While it’s more frequent that the words themselves won’t carry through, even the recognition that others are nearby and talking can be distracting.

Keep the lights low: One-way glass only works when the room the observers are in is significantly darker than the room that the participant is in. This means that the testing room should be bright, and the observation room should be as dark as possible. Even bright laptop monitors or a hallway door that stays open can shine enough light on stakeholders’ faces that they show up as ghostly shadows. While the participant isn’t apt to see these ghostly shadows unless they look carefully, it is still something that observers should be careful about.

Virtual Observation

When it’s not practical or is too costly to rent a facility with one-way glass or if the session itself is being conducted remotely, I often suggest a virtual in-person observation area. This typically involves using two conference rooms that are reasonably close together and involves streaming both an image of the screen and ideally an image of the participant as well. While not quite as “high-end” as rooms with one-way glass, this setup still lets stakeholders be part of the process; lets an on-site logger potentially have the option of sitting with stakeholders; and, like one-way glass, also still allows the moderator to join those in the observation room between sessions to gather their thoughts and feedback.

A virtual observation room could really be setup anywhere in the world as well. Although as a moderator I strongly prefer the opportunity to interact with stakeholders in-person, whether in rooms with one-way glass or without, having a virtual observation room is definitely a good backup plan that allows stakeholders to watch the sessions together.

As a fallback, particularly when stakeholders are not located in the same building, virtual observation can be accomplished by each person at their desk and on their own computer. Just be aware that stakeholders in this situation are most likely to get distracted with other business activities and not observe sessions as thoroughly.

No Observation Room

While I do frequently conduct studies with an observation room, there are also times when I’m in the field conducting the study in a less formal setting. In these cases, I’ll often set things up at a table, hopefully, but not always, with some walls around me.  Even when there is no second room, I still really value stakeholder observation.

While I am aware that observers in the testing room will likely reduce the level of scientific rigor of the study at least somewhat, as long as everyone is aware of that going in, and we discuss risks, potential impacts, and ground rules, I’m also okay with stakeholders observing in-person without a barrier of a wall or one-way glass.

If there are multiple stakeholders, however, I prefer to limit to one in-person observer at a time, even if this means that others might be sitting alone when it is not their turn.  Admittedly, though, I’ve written and spoken about flexibility in user research and, like everything else, each situation is a little bit different.

Ground rules when there is a single room

Here is a summary of ground rules for observers who want to observe live and in the same room as the sessions are taking place:

Positioning: Try to position yourself in a location that is outside of participants’ line of sight.

Introductions: It’s fine to introduce yourself to the participant, but beware of saying things that lead the participant to believe that you have a vested stake in this product or that this product is “your baby.”

Non-interference until the end: Once the session starts, don’t say anything, don’t whisper to other observers if you’re not alone, and don’t answer participant questions. Staying quiet will help participants forget that you’re there, and you will fade into the background. At the end of the session, however, you can discuss anything you want with the participant. If you have other tasks that you’d like to see the participant try or you want to ask them to replicate an experience from a prior task, you can do it then.

Laughter: Don’t even hint at laughter if the participant does something you find funny.

Body language: If you have no choice but to be within participants’ line of sight, suppress what might be a natural tendency to nod or use other body language when the participant takes the right path or takes the wrong path. Even a small head motion or micro-expression could inadvertently let someone know that they are on the right track or wrong track.

Taking notes: There is no problem with taking notes by hand or even with a laptop, but don’t let the participant hear you typing only when they do something wrong.

Communication with moderator: It’s fine to communicate with the moderator, but treat this the same way you would if you were in a separate room – communicate electronically.

When not to be present with the moderator in usability sessions

Moderator comfort: I’ve been doing usability testing for twenty years now, and I am reasonably comfortable with a number of permutations on a standard usability test.  However, as a note to stakeholders, if a moderator is not comfortable having you in the room, don’t push them into it, as doing so could impact the data collection effort.

Scientific rigor: If you are doing a summative study as final validation of product usability and/or are looking for rigorous research data, a controlled environment is much more important, and this would likely necessitate a room without stakeholders.

Know yourself: Finally, if you know that the ground rules listed will likely be difficult for you to follow, then definitely opt for observation from a secondary room if at all possible.

Do what works best for the team

There is no single correct answer for each situation. Your decision will ultimately be influenced by a combination of budget, convenience, practicality and comfort. Communicate openly about preferences and then figure out a solution that works best for everyone.

 

Image: Wavebreak Media Ltd / Bigstock.com