This was my second project at IBM. I collaborated with cross-functional partners and conducted baseline benchmarking to test a guided tour in Modeler Flows, a drag & drop Machine Learning (ML) tool.
Due to the nature of this project, I have omitted detailed information and am focusing on the process.
Lead Researcher
Research, Surveying, Usability Testing
December 2021 - January 2022
Recommendations from that research included:
Reducing number of concepts required for users to learn
Exposing more capabilities by default
Streamlining what information is provided and when
The Modeler Design team built a Guided Tour that introduced first-time users to Modeler by guiding them through the process of building a classification model.
After conducting brainstorming sessions with Design and PM, I led an official Research Kick-Off presentation that outlined the reasoning behind a benchmark study, the research plan, outstanding questions, and next steps.
Slides from research kick-off presentation that established research objectives and guidelines.
Although I was targeting data scientists, I approved participants who had different job titles because they marked the necessary data science skills.
Task 1: Create a New Modeler Flow
Task 2: Build a classification model to predict which drug to give patients
Initially, I spent lot of time rewatching recordings and trying to match up the notes to what I was seeing. In addition to filling in the gaps of the notes, I was affinitizing the data in Mural.
Virtual whiteboarding to answer research objectives
Because I structured the benchmark study to capture quantitative data, in addition to qualitative, I needed to figure out way to handle to metrics data. I created a notetaking template to guide my documentation of each participants user path as they worked to complete the tasks.
Note-taking template I created that outlined the Golden Paths to complete Tasks 1 & 2.
Participants, overall had an easier time completing Task 2, building a classification model to predict which drug to give patients, when they had the support of the Guided Tour.
Time on Task
For Task 2, that the Guided Tour helped first-time users build classification models faster than without the tour.
Task Completion
None of the participants in the "Without Tour” group were able to complete Task 2 without assistance, while 25% of the "With Tour" group completed the task within time and unassisted.
Errors
For Task 2, the "Without Tour” group had an average of 2 errors while "With Tour" group, on average, had 6 errors, which were errors specific to the tour.
Assists
The "Without Tour” group required assistance, on average, 2 times for Task 2, and the "With Tour" group required less than 1 assist.
I placed the issues within the tour in an issue severity matrix and created 5 themes to help communicate the findings and tell a compelling story. I learned that the workflow was a violation of user's mental model and modeler created map shock. The guided tour's UI was inconsistent and only provided a surface level introduction to Modeler Flow. Finally, new users perceived Modeler to be for beginners, when it is really intended for users of all skill levels.
5 Issue Themes
First-time use improvements
Consider looking at the workflow of other IBM products
Work with content designers to craft more user-friendly descriptions for Type node
Update Guided Tour UI
Update Guided Tour UI to consistently use Next or X buttons on all tour pop-ups
Update Guided Tour panel UI to number steps in tour
Update tour options and complexity
Multiple tours based on experience level
After tour: Provide documentation & extra resources
Consider end-to-end integrated tour across Watson Studio
I ran a follow up unmoderated A/B usability study to solve the problems around type node understanding.
The most tedious part of this research was note-taking and mapping every user’s exact path to complete the task. I wish I had created the note-taking template sooner in the process, but it was all part of my journey to completing my first official user research project.
I spent a lot of time with the qualitative data, even though I decided not share it with my cross-functional partners. It was challenging to try to learn R Studio, but I had a lot of fun.
