🏆
Context
This was my second project at IBM. I collaborated with cross-functional partners and conducted baseline benchmarking to test a guided tour in Modeler Flows, a drag & drop Machine Learning (ML) tool.
Due to the nature of this project, I have omitted detailed information and am focusing on the process.
Previous research found that it was hard learn Modeler Flows.
Recommendations from that research included:
Reducing number of concepts required for users to learn
Exposing more capabilities by default
Streamlining what information is provided and when
The Modeler Design team built a Guided Tour that introduced first-time users to Modeler by guiding them through the process of building a classification model.
Since the request for research came from the design team, I started by doing stakeholder interviews to learn more about Modeler and the Guided Tour.
After conducting brainstorming sessions with Design and PM, I led an official Research Kick-Off presentation that outlined the reasoning behind a benchmark study, the research plan, outstanding questions, and next steps.
Slides from research kick-off presentation that established research objectives and guidelines.
I created a screener targeting data scientists and posted it to 2 recruiting platforms.
Although I was targeting data scientists, I approved participants who had different job titles because they marked the necessary data science skills.
The moderated tests asked participants to complete 2 tasks, a survey, and answer post-task questions.
Task 1: Create a New Modeler Flow
Task 2: Build a classification model to predict which drug to give patients
I spent 2 weeks reviewing the notes, re-watching the recordings, and creating insights.
Initially, I spent lot of time rewatching recordings and trying to match up the notes to what I was seeing. In addition to filling in the gaps of the notes, I was affinitizing the data in Mural.
Virtual whiteboarding to answer research objectives
Because I structured the benchmark study to capture quantitative data, in addition to qualitative, I needed to figure out way to handle to metrics data. I created a notetaking template to guide my documentation of each participants user path as they worked to complete the tasks.
Note-taking template I created that outlined the Golden Paths to complete Tasks 1 & 2.
Participants, overall had an easier time completing Task 2, building a classification model to predict which drug to give patients, when they had the support of the Guided Tour.
Time on Task
For Task 2, that the Guided Tour helped first-time users build classification models faster than without the tour.
Task Completion
None of the participants in the "Without Tour” group were able to complete Task 2 without assistance, while 25% of the "With Tour" group completed the task within time and unassisted.
Errors
For Task 2, the "Without Tour” group had an average of 2 errors while "With Tour" group, on average, had 6 errors, which were errors specific to the tour.
Assists
The "Without Tour” group required assistance, on average, 2 times for Task 2, and the "With Tour" group required less than 1 assist.
Although the Guided Tour helped participants complete the tasks, there were still opportunities to improve the Guided Tour for new users.
I placed the issues within the tour in an issue severity matrix and created 5 themes to help communicate the findings and tell a compelling story. I learned that the workflow was a violation of user's mental model and modeler created map shock. The guided tour's UI was inconsistent and only provided a surface level introduction to Modeler Flow. Finally, new users perceived Modeler to be for beginners, when it is really intended for users of all skill levels.
5 Issue Themes
I presented the findings and recommendations to my cross-functional partners.
First-time use improvements
Consider looking at the workflow of other IBM products
Work with content designers to craft more user-friendly descriptions for Type node
Update Guided Tour UI
Update Guided Tour UI to consistently use Next or X buttons on all tour pop-ups
Update Guided Tour panel UI to number steps in tour
Update tour options and complexity
Multiple tours based on experience level
After tour: Provide documentation & extra resources
Consider end-to-end integrated tour across Watson Studio
The recommendations were implemented, and the tour was updated to address the UI issues.
I ran a follow up unmoderated A/B usability study to solve the problems around type node understanding.
Behavioral Note-taking Template
The most tedious part of this research was note-taking and mapping every user’s exact path to complete the task. I wish I had created the note-taking template sooner in the process, but it was all part of my journey to completing my first official user research project.
Qualitative Research
I spent a lot of time with the qualitative data, even though I decided not share it with my cross-functional partners. It was challenging to try to learn R Studio, but I had a lot of fun.
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 18 16" xmlns="http://www.w3.org/2000/svg"><g d="M 0 7 L 15.5 7 M 9.395 0 L 16.395 7 L 9.395 14" fill="transparent" height="14px" id="ZmtGk4vvY" transform="translate(0.5 1)" width="16.394736842105267px"><path d="M 0 0 L 15.5 0" fill="transparent" height="1px" id="KUXTs4_nB" stroke-dasharray="0" stroke-linecap="round" stroke-linejoin="miter" stroke-miterlimit="4" stroke-width="1" stroke="rgb(94, 94, 94)" transform="translate(0 7)" width="15.5px"/><path d="M 0 0 L 7 7 L 0 14" fill="transparent" height="14px" id="zsb_KSZIt" stroke-dasharray="0" stroke-linecap="round" stroke-linejoin="round" stroke-width="1" stroke="rgb(94, 94, 94)" transform="translate(9.395 0)" width="7.0000000000000036px"/></g></svg>)
