The Tenets of A/B Testing from Duolingo’s Master Growth Hacker

https://review.firstround.com/the-tenets-of-a-b-testing-from-duolingos-master-growth-hacker

A/B TEST #1: Delayed Sign-Up Several years ago, Duolingo began tackling perhaps the most existential question for an app startup: what was causing the leak at the top of their funnel — and how could they stop it?

“We were seeing a huge drop-off in the number of people who were visiting Duolingo or downloading the app and signing up. And it’s obviously really important for people to sign up — it suggests they’re going to come back and, second, signals our opportunity to turn one visit into a rewarding, continuous experience,” says Gotthilf. “That means we can send them notifications and emails. So we started thinking about what we could do to improve that.”

“We found that by allowing users to experience Duolingo without signing up — do a lesson, see the set of skills that you can run through — we could increase those sign-up metrics significantly,” says Gotthilf. “Simply moving the sign-up screen back a few steps led to about a 20% increase in DAUs. The first foray into delayed sign-ups was before my time, but we’ve since seen a positive impact.”

That change introduced “soft walls” — that is, optional pages that ask users to sign up, but allow them to keep going by hitting “Later” — and opened the door to a new arena for experimentation. “We have three of those soft walls now,” says Gotthilf. “Finally, there’s a hard wall, after several lessons, that basically says if you want to move forward, you have to sign up. Here’s what’s key: without those soft walls priming a sign-up as they’re ignored, those hard walls perform significantly worse.”

A/B TEST #2: Streaks Duolingo, like many other apps, wants to encourage users to develop a regular behavior — in this case, completing a language lesson every day. In fact, the product essentially doesn’t work unless a user visits regularly. “Over my life, I’ve studied four languages — and can speak five. One of my takeaways is that there’s no way one can learn a language on just Saturdays or Sundays,” says Gotthilf. “We need to get people to do it every day or every other day for languages to stick.”

For Duolingo, streaks — keeping track of how many days in a row a user has used the product — have been a game changer. And a metric that Gotthilf and her team continues to focus on. “It’s like in a video game, where you have to do something every day or you lose your rank,” she says.

They ran a number of tests, and ultimately landed on the metric that’s been in place ever since: 23.5 hours. “We assume that whenever you used our app was a good time for you to use the app that day. So we assume that it might be a good time for you to use the app the next day too. We want to help our users create a habit. We want to make it like brushing your teeth.”

A/B TEST #3: Badges The addition of badges to Duolingo has become one of Gotthilf’s favorite projects, though it wasn’t at first. She had wanted to test it for years, but was cautious of its return due to the high investment it’d require. But her team advocated for it, and it’s become a gamble that’s really paid off. Their hunch? Users would complete more lessons, more often, if they were rewarded for meaningful accomplishments.

That hunch was right, in a big way. Since rolling out V1 alone, they’ve seen a 2.4% increase in DAUs. “Session starts — how often people begin a lesson — increased by 4.1% and session ends — the number of lessons users are completing — have increased by 4.5%,” she says. “So that tells us that people are not only starting more lessons, but they’re also finishing lessons they would have given up on because they want to get a badge.”

A/B Test #4: The In-App Coach Enter Duo, Duolingo’s supportive owl mascot, who now doubles as a coach. “Sometimes you’re doing a very hard language. I just started learning Japanese. I make a lot of mistakes,’” says Gotthilf. “And when you make a lot of mistakes, you feel bad. You feel like you’re failing. And you’re prone to giving up. So we have this friendly character pop up on the side and say things like, ‘Even when you’re making mistakes, you’re learning.’”

Duo’s encouragement is somewhat controversial among the Duolingo team. “Some people dislike it and find it annoying,” says Gotthilf. Still, the results speak for themselves: They saw a 7.2% increase in D14. “So 14 days later, we’re getting 7.2% more people coming back to Duolingo because we added this language and cheerleading owl in the lesson. It’s crazy surprising. I’ll keep those results.”

“For new apps, you should focus on A/B tests that bring a 20–30% increase in your metrics,”

Refine and repeat.