.png)
Noom- Health Tech App
Problem:
Solution:
Outcome:
Noom wants to transition from an email-style coaching model to a "Live Chat" model to improve user retention and satisfaction. However, the increased workload for coaches could lead to coach burnout and decreased coaching quality overtime.
​
Conduct A/B test and evaluative research to identify the impact on users and coaches.
​
Recommended to the Coaching Director that we put systems in place to reduce coach workload before rolling out new coaching model. Created a live appointment tracker to help supervisors monitor coach burnout. ​
Project
In May 2022, I collected feedback from 40 health coaches at Noom to assess their experience implementing a new coaching model.
Objective
The main goal was to identify how the "Live Chat" would impact Health Coaches productivity and wellbeing.
Tools
-
​Slack Groups- Recruiting
-
Google Form- Survey Design
-
Zoom- Focus Groups
My Process
Week 1-4: Coordinated research operations for A/B Testing
-
For part 1 of the study, I supported the experiment team by coordinating research operations including:
-
Recruiting and assigning participants to control vs variable group
-
Monitoring Noom's A/B testing platform
-
Managing communication with participants
-
Week 5-6: Designed and administered a feedback survey for Health Coaches
-
For part 2 of the study, I led the post experiment feedback and qualitative analysis.
​
-
Created a 30 question survey and sent to all coaches in the "Live Chats" experiment to evaluate the following topics:
-
Preparation: How well did the training team prepare coaches for the experiment?
-
Implementation: How well were the coaches able to implement the experiment protocol with real users?
-
Satisfaction: How much perceived value did coaches gain from the experiment?​​​​​​​​
-
​
-
When analyzing survey responses, I like to start by using the numerical data to get a sense of the big picture. Then, I use the open-ended responses to paint in the details.​ Here was my approach:
-
​Calculated the mean of each question to assess the average level of agreement.
-
​Determined which data sets best represented the main themes stakeholders would want to know such as coaches' overall level of preparedness and the percent of coaches that felt the protocol enhanced their relationship with users.​​
-
Read the open-ended questions and coded the responses as positive, negative, or neutral feedback. Then summarized the insights into major themes.
-
Analyzing open-ended questions for survey
Week 7-8: Conducted and analyzed data for 4 focus groups as a follow up to the survey data
-
​Collaborated with the quality assurance and training teams to review the survey analysis and collect a list of topics they wanted to explore further.
-
Created a discussion guide to organize my thoughts and set clear goals for each focus group. I use a flexible approach that allows participants the freedom to direct the conversation toward related topics that are most important to them while addressing stakeholders' questions.​
​
-
Recruited coaches from different teams on Slack and moderated the 4 focus groups.
Week 8: End Results and Deliverables
Based on the feedback collected from coaches, we learned that:
-
65% of coaches felt that live coaching improved their coach-client relationship.
​
-
55% of coaches expressed positive feelings toward live coaching, many of which received gratitude from their clients for dedicating time to focus on their needs
​
-
However, only 25% of coaches felt capable of maintaining their wellbeing throughout the experiment.
​
​The results showed that live chats have the potential to improve the value of coaching. However, the majority of coaches expressed that managing more than a few live chats for longer than 20 minutes would be unsustainable longterm.
Next Steps
-
Before rolling out this new coaching model to the entire company, I recommended putting systems in place to monitor coach workload in order to prevent burnout.
​
-
Additionally, I created a live appointment tracker that gives supervisors a daily update on their coaches' caseloads to actively monitor for burnout during future iterations of the experiment.
.png)
Automated appointment tracker to monitor for coach burnout