Benchmarking • Longitudinal Study
Establishing a baseline user experience across Smartsheet product areas
Overview
Over 6 months me and my team conducted 8 usability benchmarks to formally assess the usability of various Smartsheet areas, tools, and functions. These studies provided a comprehensive measure of the usability of different product areas and set a baseline against which future nations and updates can be compared. In this project I personally conducted 128 participant sessions amounting to more than 140 hours of moderated testing and interviewing.
Role Researcher, Project manager
Methods Moderated Testing, Interviews
Tools Validately, Google Sheets
Project Goals
Working closely with the company’s in house research team, we laid out a roadmap that would set a clear expectation for current and future benchmark work:
Measure usability in each product area
Estalish a baseline usability score to measure future product progress.
Identify general usability issues.
Explore areas of improvement for future releases.
Find patterns across all product areas
Highlight common issues
Discover how discrepancies among different product areas may affect the overall product
Use insights from this summary to drive product strategy initiatives
Methods
With minor variations, the eight studies had the following structure:
Scope. Each study included 10-19 moderated sessions; each session lasted 60-90 minutes.
Sample. Participants were evenly split between Experienced and Novice groups. The specific criteria of experience were defined by UXR and PM owners for each product area.
Tasks. A representative list of ~15 core tasks for each area was created in collaboration between UXR and area PMs.
Metrics. Across studies, we collected the following metrics:
Task Completion Rate (% of participants who completed the task correctly),
Task Learning Time (length of time novices need to understand how to do a task), and
Area SUS (SUS for the specific product area of a study)
My role included the following responsibilities:
Before a study:
Coordinating task lists with product area owners
Writing participant-facing documents
Coordinate screening criteria with recruiters
Creating test environments
During a study:
Moderating benchmark tests with participants
Note-taking, collecting metrics
After a study:
Cleaning data
Analyzing quantitative metrics
Reviewing qualitative data for patterns
Selecting user quotes and creating impactful video clips
Assembling reports and visuals
Presenting findings to product area owners
Example task analysis, content blurred
Findings
Task completion. By industry standards, 85% or more task completion rate is considered ‘good’, 71-84% task completion rate falls under ‘needs improvement’, and 70% or less task completion rate is ‘poor’. Overall, the average SUS across the eight benchmarks was the same average SUS measured in previous research efforts.
Causes of friction. Categorizing failures across product areas revealed that features are usable but not intuitive. This was shown in the data in three ways:
Findability as the biggest cause for task failure or friction for both Novice and Experienced users shows that features are not available where users expect them to be located.
Awareness / Understanding is also a key factor in motivating users to learn about features and helping them to understand how to start using these features.
Confirmation is an important mechanism to help Novices understand whether they are using a feature correctly. This is key for Smartsheet since many features require source data to be formatted in specific ways.
SUS. In the SUS surveys, we found a large gap between the two ends of the product experience scale, Experienced and Novice.
Task Completion Rate by experience level, content blurred
Causes of Failure by experience level, content blurred
SUS across product areas, content blurred
Results & Recs
We categorized common causes of friction into the following high-level problem groups and offered recommendations for addressing them:
Information Architecture. Users struggle to understand where to find specific features and due to the fact there is usually more than one entry point, they cannot remember where they found it.
Unclear Entry Points. Offering too many or unclear entry points makes the product hard to learn, but after some amount of clicking around new users were generally able to find the start point, and experienced users were almost universally successful. Most errors were due to discovery issues and not specific usability problems.
Terminology Confusion. Smartsheet uses some terms that users have trouble interpreting, creating a barrier to learning the product. Once the terminology is known the users can extrapolate to other feature areas, but it makes the product less approachable.
Misunderstood Metaphors. Some of the processes and spaces are used in unconventional formats so that users cannot leverage their experiences from other similar products to their experience with Smartsheet.
Prerequisite Knowledge Needed for Advanced Features. In order to understand the more complex and powerful features in Smartsheet, the underlying technology needs to be understood by the user.
Limitations & Lessons
This study focused on deep dives into individual product areas as if they existed independently from each other. While this allowed us to explore tasks within each area in depth and compare issues across the areas, such a discrete approach limited insights into the usability of end-to-end customer scenarios. Future research should address the usability of prototypical end-to-end scenarios most relevant to specific types of our customers.