User research to increase retention rate (Part 1)

Illustration by someone I forgot the name. Sorry mate.

Author’s note:

I joined Habitify 2 years ago, without any prior knowledge in marketing, so I had quite a hard time figuring out what is supposed to be the most basic knowledge of any marketer. 

Therefore, this episode may sound fairly childish for some of you who are years ahead or even new but had a proper education in this field. 

However, I’m writing this with pride because I dug my way out of this, on my own, with what I have on hand. This is not only a story of how I increase my app day-1 retention rate from around 31% to over 40% (even 46% in key markets), but also the story of how I grow.

I still believe, though, that this series will be helpful for some junior product manager or UX designer, as, sadly but fortunately, there are very few documents out there, and even fewer case studies on this subject, especially in the domain of small-niche mobile app development.


Start with why

Just give me a reason

I didn’t suddenly go out of my way to spend 6 months straight to do user research unless it really compels me to. 

And yes, it did compel me.

Remember I said I was a newbie in this field? So my starting point for this whole journey is just as embryonic as me. 

When I was in charge of customer support, I had always been bugged by a lot of feature requests that, to me, sound perfectly plausible. Driven by the minimal and pragmatic mindset, my CEO appears to be very patient listening to all of my rantings about cool features that I (or the users) come up with, but he, 9 out of 10, declined to implement them. 

What is the impact if I implement this feature?

-Peter

Words can’t express my frustration as my ideas were turned down, yet at the same time, I understand his concern for our limited resources and the potentially negative impact a new feature could have had on our users.

By the time I was speaking with him, we just crossed 600,000 users.

 “If you implement A, then the normal scenario is 300,000 people are gonna love it, while the rest 300,000 are gonna hate it. What will you do? Do you implement a “fix” feature for the 300,000 haters or do you remove it to turn 300,000 lovers into 300,000 haters?“ 

-Peter

Feature development, in this way, is a vicious circle of release-fix that will drown us and steer us from innovating the product. 

This ignites the whole engine. I was pumped. Excited and curious. I set out to sail my boat. 

Going deeper 

The first question I asked myself was

If users are asking for features, what does that mean?

That means Habitify lacks certain things to keep users to stay. That means many have already left because they didn’t find what they want. The ones who are talking to me are tolerant of our inadequacy. 

That means user retention is in trouble. 

Indeed.

I took a look at the analytics and I figured out that in 2018, the day-1 retention rate of Habitify was 31.6%. This translates into 31 out of 100 people stay with Habitify until the second day they install the app.

Habitify User Retention in 2018 – Source: Amplitude

This is way below the industry standard:

Apps in Travel and Lifestyle category has 44% retention rate – Source: Localytics (2017)

So instead of asking “What features should I implement”, I go with

“How can I make them stay?“

And so my journey begins.

Choosing the tool

Now that I have the “why“, next is the “how“.

The Golden Circle

How can I know how I can make them stay? I need a tool to collect data! 

To answer this, I read some articles of basic UX design, and I learn that there are 2 popular ways to get data:

  • Quantitative research: We can do this in real-time or by looking at numbers. Since my company doesn’t have a behavioral tracking system deeply integrated, this option is not feasible.
  • Qualitative research (User survey or interview): Despite the fact that the answers might be subject to several factors (how I frame the questions, their mood, their memories), this will provide me with how the users “perceive” the app and since I can dig deeper into their daily life (which the first option fails miserably and inherently to do so), I can deduce what they actually need. 

Choosing the users

The best scenario is I can interview both free and Premium users (Yes, Premium users deserve a capital P). However, due to my limited time and resources, I went ahead with Premium users for these reasons:

  • They are the most supportive people, so picking their brain is not as hard as free users (who have a high tendency to be frustrated at something in the app, or just simply ignorant)
  • They use the app intensively. They can surely give me tons of insights as to how they actually use the app.
  • They PAID. Yes, that’s why they deserve such a big P. Paid users will definitely answer the question of “why” they purchase the product, or more specifically, “what” features they are paying for. This is key.
Illustration by Joanna on Dribbble

I purposefully chose Premium users who have just upgraded the app in the last 2-3 months to make sure they are still using the app and do not have a vague impression of the USP of Habitify. 

This method of selecting users is called Convenience Sampling, one of three Non-probability Sampling methods as introduced by Linh Giang.

Sampling needs methods! – UXPress – Medium

Building survey and interview questions

The Survey

A survey is suitable for collecting demographical information and also quantifiable information. I thought they were good for open-ended questions, too, but later I figured out analyzing over 300 answers was a real pain in the neck. 

I designed the survey mostly to collect pre-usage information, including search intent and search platform, besides their demographics. 

A majority of Habitify users are developers and/or working in IT industries – Source: Habitify Member Survey

The Interview questions

Meanwhile, bearing in mind that interview is best suited to collect qualified information, I focus on users’ in-app behavior and also their daily routine, which may sound too generic for marketers, but really insightful for my app in particular.  Here’s my original questionnaire.

What I learned right after a few interviews is that there are unnecessary questions about user’s background like “What’s the solution before Habitify” because, unlike other products, apps are pretty accessible in the beginning and so they are often the first choice of people when it comes to their problem. So, needless to say, the typical answer would be “Habitify is the first solution”. 

But the most redundant question title will be rewarded to “When do you upgrade?” 9 out of 10 people will forget why, and they will give me very generic (though truly heart-warming) answers like “Because I want to support you guys” or “I find it most suited to me among other apps I have tried”. 

Also, I found that this top-down approach doesn’t work as expected:

Problems > previous solutions > Habitify

The problem is this approach is too generic, it doesn’t have a “pivot” point on which the users can base on to tell their stories. It’s very challenging to make people “Triggered” because I’m asking them to recall memories in batch rather than in pieces.

So I changed the approach:

Habits > problems of one habit > personal problem in life > Habitify

This approach works like a charm because when the user is asked about a very specific habit, they can recall a lot of things in the past, which provide us with various materials to base on to (make hypothesis then) ask follow-up questions.

Here’s the revised list.

Getting things done ✅

I collected 315 answers via survey. As I had nearly a year of experience in sending emails and optimizing open rates, I didn’t have much difficulty in gaining that amount in 2 weeks. 

I conducted interviews a month later than the survey after I realized that the survey didn’t give me enough behavioral insights. 

Here are some lessons I learned after 40 hours on call

User interview is beyond getting insights

I wish that I realized this sooner…

User interview is a wonderful opportunity to watch and hear people express their love for our app in the most authentic way. This is the most convincing evidence to prove that Habitify is actually helpful for a person, and, if it were to be put on our landing page, it would definitely convert more users.

Except that I forgot to record the video call 😆

If you’re still on the fence about what I said. Watch this video

Tell me, do you want to buy an Apple Watch now? 

Similarly, I didn’t know I should do write-ups of the interview and published on our blog as success stories. We can use their testimonials and also their company as a charm to gives our page a boost of credentials. 

Take a look at this Todoist newly-branded landing page, and let me know how you feel when you see the app is used by a Lead Designer at Adobe? Or a senior product engineer at Buffer? 

Todoist new landing page

Effective is this method also for SEO. When you share a story of a user, they are more willing to share this on their own blog (if you get lucky, they will write the whole article for you), or social media. We got one juicy do-follow link from one of our users from this.

First lesson: Always collect testimonials in all forms

Preparation is a time saver 

The first few interviews were a real struggle for me. Things were getting out of hands when my users, instead of following my questionnaire, started sharing about their pain points with the app and requesting several features. Yes, that’s part of the plan, but not what I wanted the most. I wanted to know their background, the context they use the app and their frequency. 

It boils down to the fact that I didn’t have good preparation to be flexible enough. My questions were designed in a fixed, arrogantly presumed order that, if not followed correctly, will give me a hard time bringing them back on track and organizing the information later on. 

What I did afterward were to divide my questionnaires into several “topic” blocks, and devise plans to switch between them should the users go sideway. 

Second lesson: Always prepare all scenarios. We won’t have enough time to react when facing users.

Never trust your memory (too much)

I used to schedule  2-3 interviews next to each other because I think that I will be in the “interview mode” constantly. That’s true, only except the fact that I trouble myself with remembering all the nitty-gritty details of the talk, and “decoding” my own notes (I have a terrible note-taking skill, by the way) into meaningful sentences. 

I realized that I had a nightmare-ish memory on which I never should put total trust. It was too bad to the point that after some interviews I even have to send a very long follow-up email which will highly likely cost the users the exact duration they spent on my call. What a shame.

Third lesson: Take notes and consolidate after each interview. This will save a ton of time later.

Let your users help

I realized this after my talk with Anna, an experienced marketer who happens to also be a Habitify user. 

After my interview with Anna, I sent her the writeup. To my surprise, she sent me back not only the writeup with lots of corrections from her side but also page-long emails about how I should improve my interview skills and methodology. 

Her experience in marketing and her vast knowledge in different fields have taught me to look at my matter from a different perspective. And, who would have guessed to receive life and health lessons?

A touching opening in Anna’s email to me
Forth lesson: Let your users know your problem. Show them your weakness. You don’t know how much help they can give.

(To be continued)

0

5 thoughts on “User research to increase retention rate (Part 1)

  1. Thanh Sơn says:

    Great work! You already know that I care about user experience just as much as you do, but caring is easy. Carrying out an actual research, on the other hand, requires a whole new level of effort. The most tedious part of getting users’ answers is, fortunately, thoroughly reported in the blog post, and all your trials-and-errors will be of tremendous help to me in future projects for sure. Please write a part 2, I would love to learn from your experience.
    And yes, that Apple Watch commercial is pure heroine. I crave one, NOW.

    0
    Reply
    1. tuanmon says:

      Thanksss for your kind reply mate. We’ve got so much to learn from Apple mastery of UX, don’t we?

      When I first started, I was afraid of not carrying out this whole thing the-right-way, but then I realized standing still is worst because I would have learned nothing.

      Don’t forget to tag me when you write about your own project! ☀️

      0
      Reply
      1. Pun396 says:

        Hey, how you let go of the underlying needs of “carrying out this whole thing the-right-way”. I was stuck on that. Same story as you, but my “product” is actually services, so it was way more hard to define clearly

        0
        Reply
        1. tuanmon says:

          Hello! I’m not sure I understand your questions correctly. Mind repeating/rephrasing?

          0
          Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Scroll UpScroll Up