In the infamous words of Donald Rumsfeld, there are known unknowns and unknown unknowns. When it comes to website or app design, testing with representative users (a.k.a. usability testing) can uncover both.
Having conducted many, many user testing sessions over the past 10+ years I can confidently say that without exception they ALL produced important insights for improving the websites or apps being tested. But based on my experience, one method is significantly better than the other.
Unmoderated versus Moderated Testing
When planning a usability testing project, a series of tasks are created for users to try to complete. Then, without help or coaching, you see if they can accomplish those tasks without significant challenges along the way. (For more detail, see my post on how to do user testing.) Because you’re starting with various hypotheses, these tasks usually represent the known unknowns. For example, if a website is experiencing a lot of abandoned shopping carts, you may hypothesize that the checkout process has problems, so you write a task to ask users to put items in a shopping cart and check out.
Unmoderated testing is a methodology where users are sent a link to a website or prototype design, and they attempt to complete a series of tasks on their own. They make comments as they go which are recorded and later reviewed by an analyst. There are more and more good platforms available for unmoderated testing.
Moderated testing is similar with one important distinction. Instead of users being on their own, they are accompanied during the testing session by a moderator who asks them to accomplish the planned tasks and makes observations along the way. The testing participant and moderator can be together physically or virtually (using a platform like UserTesting.com or Zoom). The sessions are also recorded for later reference.
Which is better?
I posit that moderated testing is usually the better approach. Why? Simply because you get richer insights than you do from unmoderated testing. With moderated testing you are much more likely to learn about the unknown knowns. You learn things you didn’t even know to consider.
By contrast, unmoderated testing forces you to focus on the known unknowns and to ask specific questions with little opportunity to pivot or dig deeper. Plus, one of the biggest challenges with what you believe to be the known unknows is confirmation bias — the tendency to look for the answers that reinforce your preconceive notions.
Why is moderating testing better?
More than any other factor, moderated testing is advantageous because it lets you ask, “Why?” For example, “Why did you click that button?” You can also prompt users to think out loud, such as, “Tell me what you are thinking as you’re considering the options for how to proceed.”
What are user testing sessions like? Frankly, they can be tedious to watch, but this 1½ minute clip of a mobile website test demonstrates how prompting the user to “Tell me what you’re thinking” can produce meaningful insights.
Of course, that kind of interaction with the user during testing must be done very carefully so as to not help them accomplish the task or lead them to a particular outcome. Moderators be cautious when digging deeper to not provide clues to get to the answers they want to hear. For example, “What are you thinking about the available button options?” is much better than, “Why don’t you click the blue button?” Moderated testing can even allow you to make a full-scale pivot in the middle of the testing session. Something the user says or does may prompt you to ask, “How would you go about doing that?” or, “What would you do next?” In essence you can create an entirely new task on the fly. Unmoderated testing doesn’t allow that flexibility.
Any Testing is Better than No Testing
Even though I’ve promoted my preferred approach, I want to emphasize that any testing is always better than no testing at all. Test early and often, the maxim goes. Test formally or informally. Test during design, during development, after launch. Just do it — you will be surprised how much you learn that will improve user experiences.
And here’s some food for thought. Approximately 15-20% of your website traffic could come from people with some form of disability. Making sure that content is accessible by including people with disabilities in user testing should be a strong consideration for many websites.
And if you need assistance, please contact me to discuss how usability testing can improve an existing website or be incorporated into the process for your next design/development project.