User Research, Usability Testing

Screen Shot 2016-05-30 at 10.42.06 PM
This was an evaluative study to understand issues first time visitors of the website run into while searching for training and certification exams.

Tools Used: Ovo Studios, Powerpoint, SurveyMonkey

Team Size: 4

Data Analysis

In addition to understanding the experience of first time users, it was important for us to understand the users who were already visiting the site. We wanted to ensure that we focus our work on realistic users who would potentially visit the site on their own versus a complete non-user.

Microsoft shared detailed data from our sponsors. Based on this data, we created user groups. These user groups helped us in creating our personas for the system.




We carefully selected the persona’s for our study based on the demographics of users who have visited the site in the past. This helped us in two ways (1) Creating a baseline about what certain persona’s will do when confronted with different tasks on the website and (2) recruiting participants similar to the persona so that we understand how we can make the experience better.

The Image on the right has been modified to hide data.





We developed our screener to map our sample against it and select participants by considering factors such as eliminating professional user research participants, minimizing no-shows, eliminating the strong -silent types, learning about their familiarity with Microsoft Learning website and ensuring that people are physically able to participate.

We also aimed to be precise and drafted the sequence of our screener to get to the point quickly, asking the elimination questions first and avoiding questions that are strictly for information gathering. We also asked “what they do on the Internet” to learn about their Internet knowledge as our preference was participants who are more tech savvy.



 Usability Testing

We picked the think aloud method for testing the Microsoft Learning website. Our project required us to test how easy it is for a first time user to navigate the website and find information on it. Thinking aloud suits this kind of project the best because participants will be speaking their thoughts. This allowed us to understand what the participant is actually thinking instead of us guessing.
Each 60-minute session was broken into four sections:
  • Pre-session interview
  • A warm up task
  • Two sets of scenario-based tasks completed with the website
  • Post-session interview and debrief


For each task, we collected the following data:




We used affinity diagramming to find clusters of problem areas and sort them into related groups and sub-groups. This also helped us in finding areas with most severe problems.

The Image below has been modified to hide data.




The study revealed some interesting findings and we were able to complement our quantitative data with verbatim comments from our participants.  From these comments, we were able to dive deeper into the issues that caused task failure or difficulty. All the recommendations have been implemented by the Microsoft team and are live now.



The usability study was a great learning experience. Few of the high level learnings were:

  1. Test the scenario, prior to the actual test, on the same computer as you would use for the participant, and do not log in from the corporate account. There are some interfaces which are available to employees only and might impact your results.
  2. Keep a print out of the tasks for the participant and ask them to read it aloud and ask them to repeat it in their own words. It helps to see if the participant has understood the task correctly.
  3. Always test a new environment a few hours before the test is about to begin. Give enough room to fix any technical glitches
  4. Run a pilot study (at least) a day in advance. Have enough time to fix the script or other issues you might face.

Related Projects