Fabulous for New Users
1. Overview
Fabulous is a habit tracking app incubated in Duke University’s Behavioral Economics Lab.
Many new users used to stop using the Fabulous app within the first week. To investigate the challenges faced by new users I remotely evaluated 1-week usage of the iPhone app with 8 participants. I identified 263 issues, which I rated based on criticality and frequency of occurrence. The issues were related to onboarding and tutorial, understanding the features of the app, navigation, adding and checking off habits, reading articles, etc. I provided design recommendations to the design team for addressing the identified issues.
Duration
August 2020 - November 2020
(4 months)
Team
I was the only UX researcher in the team
Managed by the CEO Sami Ben Hassine
Collaborated with the design and development teams
Duties
Frame research questions
Select evaluation methods
Design evaluation procedure
Prepare evaluation materials
Recruit participants
Conduct the evaluation study
Data analysis and reporting
Evaluation Methods
Remote diary study
Observation
Screen recording of app usage
Think aloud
Interviews
Surveys
Pilot study
Data Analsyis
Affinity analysis
Open, Axial, and
Selective coding
User journey maps
Personas
Scenario
Task and User flow
Triangulation
2. Introducing the Fabulous App
Fabulous is a habit tracking app incubated in Duke University’s Behavioral Economics Lab. It is a coach that uses insights from behavioral science to help users build healthy habits.
3. Research Focus
Research Goal
Identify why a lot of new users stop using the Fabulous app within the first few weeks.
Research Question
What are the challenges faced by new users of the Fabulous app in the first week?
4. Evaluation Methodology
I conducted a remote qualitative case study.
Evaluation Methods
I conducted remote and unmoderated evaluation involving a diary study with 10 participants who used the app daily for 7 days. I used the following methods for collecting data -
Screen recording: The participants recorded their daily usage of the app using screen recording.
Think aloud: While using the app the participants used unmoderated think-aloud protocol, for which I pre-trained them. I asked them to focus on thing slike
What you are trying to do and why?
What you expect will happen as a result of what you are planning to do next?
What challenges are you facing?
Online surveys: I used online surveys to collect demographic data as well as for daily check-ins. I used both closed and open-ended questions for the daily check-in questionnaire, such as
How challenging did you find to use Fabulous in the last 24 hours? (0- Not at all to 5- Extremely)
What are the top 3 challenges you faced when using the app in the last 24 hours or so?
The daily check-in also involved me communicating with them through emails to get clarification when any parts of the videos they submitted were ambiguous or missed details.
Interviews: After participants used the app for 7 days , I conducted semi-structured interviews with them individually where I asked open-ended questions such as -
When you started using Fabulous, what were your objectives behind using the app?
To what extent would you say the app met your expectations?
Out of all the challenging experiences you have had with the app, which do you consider to be the top 3?
Some of these questions were common across the 10 participants (examples above) and the rest were individually tailored to follow up on the crucial findings from the analysis of their think-aloud usage videos (examples below).
Why did you choose to not join any of the habit challenges on Fabulous?
On day 5 in your check-in questionnaire, you mentioned that it was confusing to track progress. Can you help me understand what you found to be confusing?
Why did you choose to use the app only once a day?
A participant using Fabulous while thinking aloud.
Evaluation Procedure
The evaluation procedure involved the following steps
Participant briefing: I briefed the participants over a Zoom call. I cross-checked participants' eligibility, introduced the study objectives and requirements, showed them how to do screen recording on iPhone and tested it, trained them to do think-aloud, asked them to do a dry run of think-aloud for a different app. I also shared a document including the briefing information for future reference.
Diary study: Participants used the Fabulous app daily for 7 days. To capture natural and contextual usage, I asked them to use the app as they like, whenever they wanted to, and for however long they desired to; however having to use it at least once daily for 7 consecutive days. While using the app the participants used unmoderated think-aloud protocol and recorded their usage using screen recording. They also filled the daily short survey as discussed earlier. I asked them to send me the videos on a daily basis so that I could analyze the data and follow-up with them for clarifications and provide them any required additional training or instructions. I considered such daily follow-up to be crucial since the study involved unmoderated think-aloud protocol. I recruited 2 to 3 participants in a week to be able to analyze their data and follow-up with them daily as required.
Interview: I interviewed the participants soon after the 7 days of using the app so that their experience of using the app was still fresh in their memory.
Remuneration: I provided the participants with a remuneration of $85 which was broken into $10 per day of the 7 days diary study and $15 for the interview to allow participants to drop out in between the study.
Pilot Study
I iteratively developed the evaluation methods, procedure, and materials discussed above by running a pilot study with 4 participants, following which I evaluated Fabulous with 10 participants. Some of the insights from the pilot study include -
Will think-aloud be overwhelming? 3 out of 4 participants mentioned that they did not find it difficult to think-aloud while using the app. Based on this I finalized to use think-aloud protocol since it provided more insights compared to when I asked pilots to provide comments only when they came across anything challenging.
Do they record complete usage? When pilots were instructed to record their screen only when they faced any challenge with using the app, it created issues since they were unable to go back to the previous screens/stages of the app which initiated the challenging experience. Hence I asked participants to record complete usage.
How long should the diary study be? I tried different lengths of usage from 3 days to 15 days and found 7 days to be an appropriate extent since the crucial features of the app were unlocked within this period and participants had sufficient time to use these. Also, some pilots found it difficult to continue the study after 7 days.
Is an interview required? The analysis of the usage videos submitted by the pilots highlighted that since the think-aloud was unmoderated, an interview will help gain deeper insights by following up on the crucial findings.
5. Participants
4
Pilot Participants
1 female, 3 male
25-34 years
10
Target Participants
5 female, 5 male
18-44 years
1
Country
USA
Recruiting
I designed eye-catching posters to recruit participants. I put up the posters on social media including 150 Facebook groups (e.g. buy & sell, paid gigs, user study), Facebook marketplace, Slack channels, Linkedin groups, and free user recruiting platforms such as Honeybee and Survey Circle. 1000+ people signed up for the study within 10 days.
Screening
I used a questionnaire to screen target users who met the following criteria
Never used Fabulous before.
Minimal experience of using other habit tracking apps.
Why? People new to habit tracking apps are likely to face more challenges while using Fabulous compacted to experienced habit tracking app users.
Strong current motives of using Fabulous.
Why? The evaluation involved natural usage of the app for 7 days, during which participants were asked to use the app as they like.
Born in the USA, lived 10+ years in the USA, and currently residing in the USA.
Why? USA is the target market of Fabulous and the majority of current users are from the USA.
After conducting the evaluation with 8 participants, I found that the data was repeating, and there was no new significant data. Hence I stopped the study after running it with 10 participants.
6. Data Analysis
Data Collected
Analysis Procedure
Using NVivo I did affinity analysis involving open coding, axial coding, selective coding for the video recordings of the app usage and the interview data. I used task flow, user activities, and apps' features to guide the axial coding. I triangulated the findings by referring to multiple data sources such as usage video, interview, and check-in surveys.
Sample of axial coding using NVivo.
To help prioritize the issues, for each identified issue I provided
Criticality rating on a 5 point scale based on impact factor.
Frequency of occurrence across the 10 participants.
Net severity on a 10 point scale calculated by giving 2:1 weightage to criticality and frequency resp.
I further organized the issues by indicating the involved user activity from task flow, associated features of the app, and location of occurrence which can be used to group, sort, and filter the issues.
For each identified issue I indicated the heuristic violated and provided design recommendations for addressing those.
Data Reporting
I reported the findings using Airtable and user journey maps involving persona, scenario, and, user flows. I provided evidence in the form of screenshots and video clips from app usage data and quotes from interviews and think-aloud videos.
Sample of data reporting using Airtable. Click for expanded view.
More Projects
