HOFFMAN: Luis started feeling responsible for millions of hours of lost productivity. So we take from Duolingo, I mean we spend a lot of time its funny, you can see a lot of the different items in Duolingo. HOFFMAN: But they soon found a problem with this method. HOFFMAN: This is a problem that all mission-driven founders must face: how much people like your product is not the same as how effective it is in achieving your mission. And of course we just said, sure we can. And I believe this legal agreement between Duolingo and its users is a great innovation launching the company to the next level of scale. If you let them down, theyll lose faith in your commitment, and your product. It would harness the power of gym-goers to generate electricity. AHN: I kind of had forgotten my whole TOEFL experience when I had to take that test in Guatemala. But these are all nudges. And frustrating your users is never good. We realized actually we could get a good number of people helping us add languages to Duolingo. We didnt expect that, and it just kept growing and growing and people started asking for a ton of languages. So it was 10 seconds of very valuable work in some sense. And in general we want to do whats good for the users in terms of actually learning, and we dont want to get these people always learning the word for men a trillion times. KATIE CLARK GRAY: I had no idea until we were working on this episode that all those thousands of reCAPTCHAs Ive done helped digitize books for Google. exercise bikes hooked up to the grid, treadmills wired to power converters. that anachronistic English test he and millions of others have to take if they want to study in the US. Thats Luis von Ahn, founder of the language-learning app Duolingo, describing a game-changing idea he had at the age of 12 to disrupt the fitness world.
I mean, a CAPTCHA is one where you can just The incentive for the human to do it is because they want to get an email account or buy tickets or whatever. But thats not teaching better. AHN: I have now realized a lot of gyms make a lot of their money from people signing up and never showing up. You can trace a lot of the things we do based on what game our product managers are currently playing. Arguably, something for the greater good. Well ,Duolingo is full of chests these days. Indy has with him an ancient artifact called the Staff of Ra. The incentive is just this intrinsic thing. And then with my PhD advisor, a guy named Manuel Blum, whos amazing, we came up with a potential solution, which was these distorted characters that you have to type whenever youre getting an account. Imagine its a free gym. AHN: What we did is, we opened up our tools for creating languages for Duolingo, for adding languages to Duolingo, to the world. It works a lot. Udi knew that the way to recruit engineers wasnt to talk about how great it was to work for Yahoo. In part because it was a free way to learn languages.
So, yeah. Pay attention to how actually even a very simple game actually retains some people because of the mechanics that they use. Other big sites like CNN and BuzzFeed signed up for reCAPTCHAs translation service. But I quickly remembered, and when we looked at this business, its a pretty crazy business. So for example, when we started, we wanted to create a game to get people to be more connected. Right. After a while there was a couple of volunteers that started thinking, well wait, who owns this? Or, if youre in dire straits, the disgruntled many. I mean, one of the hardest things is finding a problem that is kind of a Goldilocks problem. Well ,Duolingo is full of chests these days. Its a blocker and they have to do it. Humans are not very good at generating electricity this way. I thought, how hard can it be? I mean, I have questions! Not something that has no relation to them. I now understand the reason he came to CMU. Now Ill tell you, dont get your product organization playing Clash Royale. HOFFMAN: What was it that made this idea so revolutionary? But even a 10-second blocker is frustrating. There was also another flaw. The first idea, actually it occurred to me, was something very similar to reCAPTCHA or very similar to the gym, which was okay, were not going to charge people because I dont want to charge people because this is kind of supposed to help the world. Am I going to get a credit? AHN: Apple named us the iPhone app of the year. if your efforts to keep quality up end up blocking more than they help, then your crowd will find somewhere else to realize their passion. Not at first, but eventually Im like, oh, this is just like that gym. And it was at CMU, when Luis was completing his PhD, that he caught an impactful lecture by Yahoos chief scientist, Udi Manber. For a 10-second task, it works. And I believe this legal agreement between Duolingo and its users is a great innovation launching the company to the next level of scale. So basically anything that computers are not super good at, they kind of turn into a CAPTCHA to get a lot of training data. ' And then it works, but yeah, I wish my phone could just be better at that. Can you handle high traffic? Would you be able to help? And they responded pretty much immediately saying, Id love to and I have friends and Im willing to help..
We didnt expect that, and it just kept growing and growing and people started asking for a ton of languages. And the power of it they are invested and will do their best to help you improve your product. Its amazing. But at the time, it was impossible for machines. I mean, I did a number of these. AHN: We built the first prototype and we actually gave it to Udi, and he put it in Yahoo within a week. And it was about this time that a lot of websites asked to use it. Its pretty similar. This is why Luis puts so much time and effort into ensuring that every user-generated course Duolingo puts out meets his rigorous standards. Crowdsourcing is a way to tap skills. Early on we realized the hardest thing about learning a language is staying motivated. But if you get the alignment wrong, you can end up with anything from a weak fizzle to a fiery disaster. HOFFMAN: So Luis left his home in Guatemala to study at Duke University. Luis solution? And so much of the time, and this is true for gamification as much as it is for games, when you have a system that isnt giving you the output that you want, that the real question is: have I named the correct goal? Its free. Just cant make that much money off of translation, it turns out.
So I thought I would do something that would teach people English for free. And then my co-founder is from Switzerland, Swiss German. makes them feel secure that Duolingo is aligned with their mission. This makes me hate these six seconds much less. The main concern for Luis is to keep Duolingo focused on the mission. For the first four years of Duolingo, the only thing we did was improve user retention. If you are crowdsourcing for a for-profit business, you need to demonstrate that your mission is the same as that of your contributors. It lets you scale faster and farther than you ever could on your own. So the CAPTCHA was a successful, if simple, form of crowdsourcing. But I think hes being far too hard on himself. This is why you need to be clear with your contributors. Its the lack of a shared mission bonding the crowd and the gym operator. It lasts one week, the top 10 people go to the silver league and then if youre the top 10 there, you get to the next league, et cetera. And its still harnessing its crowdsourcing power today. They were like, hey, were implementing a CAPTCHA service. But there are also other things you can do to keep your crowd motivated. Where it was kind of. I need to certify that I know English. We got enough of these emails that we started looking into this. That is brilliant. HOFFMAN: Heres something else thats pretty amazing: Luis and Manuel gave away the CAPTCHA technology for free. When it works, it opens extraordinary opportunities. AHN: It turns out this is actually not a very good idea. Elizabeth Sampat is a game designer and author of the book Empathy Engines: Design Games that Are Personal, Political and Profound. Fortunately, I had enough means to be able to fly to El Salvador to take that test. So its this funny thing. In fact, it was. AHN: Theyre using it to improve things like Street View, the recognition of the addresses. Whereas Luis wanted his gyms motivation to be aligned with his customers. Why? At first, the numbers told a story of success. as will their willingness to work with you. I carbon-date to an earlier era in the web and in technology. So I thought I would do something that would teach people English for free. And I also now understand that for a very large company to do something in one week, it means it was a serious problem for them. No offense. Elizabeth says Duolingo is a good example of gamification done right. Humans are not very good at generating electricity this way. I mean literally nobody, but its like Im trying to win that game. Will I ever know which specific books I helped with? HOFFMAN: This is another unexpected place where the crowd took Luis and Duolingo to. And misunderstanding can cause a buildup of bad sentiment that may poison your project. Maintaining quality is vital to keeping users and contributors engaged. It was around this time that Google embarked on its quest to digitize every book in existence. And was it really a good thing? Quickly, the subscription made more money than the ads. Bots beware. To picture what I mean, I want to remind you of a classic scene from the movie Raiders of the Lost Ark. And the way to implement that is this concept called leagues. You can trace a lot of our elements just to which game people were playing. And obviously as we get to Duolingo, itll be a pattern, but what are the kind of the ways that you put this toolset together where you say, oh, actually in fact theres ways that groups of humans can do this in collaboration with computers that make all of this ecosystem better. At least in all the systems that Ive done like this. HOFFMAN: Luis thought to himself: Well, humans can recognize those words. And literally, every day we would get requests, multiple requests for, hey, can you add this language or that language? AHN: Over time we pretty much realized, look, the main thing were going to do is something to teach a language, well figure out how to make money later. But that was the original business model for Duolingo. Pretty soon after that, a website that was starting to get big. Heres a news article that is related to what you just learned in the language that youre learning. But I think hes being far too hard on himself. So we take from Duolingo, I mean we spend a lot of time its funny, you can see a lot of the different items in Duolingo. Theyre using it to improve things like Street View, the recognition of the addresses. HOFFMAN: Its this strong aversion to inefficiency that got Luis thinking about how to iterate on the CAPTCHA. I would say its people who make games.
Can you translate it to your native language? Right? And I believe crowdsourcing can scale your business in unexpected ways, To picture what I mean, I want to remind you of a classic scene from the movie, And this is the kind of alignment you need to get crowdsourcing right. I had to take the test. It is impressively hard to do that. Your users will be invested in your shared mission, and they will help you course correct well before you hit the jagged rocks of disillusion. I mean, adding a course to learn Esperanto, youre just not going to get that many users. They had to be too complex for computers, but easy enough for humans to do in a few seconds. We have enough users that that was significant. It was this shared mission that let Duolingo hold off from needing to generate revenue, and focus on attracting mission-focused creators. AHN: We as a company have our own version of the test. Bots beware. AHN: There was again, a flaw kind of similar to the gym flaw. This is exactly the reaction one of our producers, Katie Clark Gray, had when she was working on this episode and realized for the first time that reCAPTHCAs were more than just an annoyance. And this is what you want when you take your product to the crowd. HOFFMAN: But even thoughtful gamification can spin out of control. I worry about this. And the part thats really painful, the part where all the friction is, is having to get to the point where you know the language. And theyd hit a snag. At the game dynamics to help them stay engaged, stay motivated, not to create a different outcome. So its this funny thing. Duolingo had tried to make money by capturing the efforts of its language learners to offer a translation service. I mean, people really saw it.
This is why you need to be clear with your contributors. Luis isnt the only person who fell under the electrifying spell of the power-pumping gym. But Luiss idea hit a snag.
I mean, the standard way to make money in education is just to charge for people learning. I want to spend the rest of this episode focusing on one way Luis does this. So it was 10 seconds of very valuable work in some sense. Listen to this, Okay Google, turn the lights red. And you re-type them in a box. For some fraction of the users, we would give them a little test on the app and then we would have them use the app for a while and then we would give them another test and we would see how much they learned. HOFFMAN: Its not just unworkable math that makes the gym idea an ineffective use of crowdsourcing. HOFFMAN: Keep this in mind when you reach out to users and offer to be partners with them in a shared mission: you raise the stakes. HOFFMAN: Weve all seen CAPTCHAs. And spammers would write programs to obtain millions of email accounts for free. You just ask them beforehand, you do a few questions, then you let them learn and then you ask a few questions afterwards. And when they found out, well, they werent always thrilled.
So we put your number of connections at the very top of your profile in a prominent way. And that would push Duolingo to become the worlds most popular language learning app. : One thing you wish your phone could do? And in the first week, about 50,000 people applied to add a language. And we had a protest when we later retrofitted this and said, Okay, at 500 it goes to 500-plus and were done.. as long as you perfectly align your mission with your crowds motivation.
When youre signing up for something online, youre presented with a set of random letters or numbers, bent out of shape as if theyre melting. So from then on, all kinds of websites started using it and it was pretty amazing. We were very mission-driven and people really saw that. And misunderstanding can cause a buildup of bad sentiment that may poison your project. And so the first thing I did is for one of these requests, randomly I just responded, I said, look, This is a very small language. The crowd is attracted by free gym time. Can you handle high traffic?
Its a decent alignment of crowd motivation and company mission. HOFFMAN: This handcrafted approach was an undeniable and unexpected success. CAPTCHA. And you can turn them into all kinds of different problems. To do this, he had to find a way to measure their effectiveness. And I came to college here in the U.S., but to come to college in the U.S. you have to take an English proficiency test. I mean, we make more money than any other education app, and we are the one that doesnt charge for the actual content. And I made the Spanish course. AHN: Because we had this mission that we didnt want to really charge for content, our hands were a little tied on what we could make money with. HOFFMAN: Did it occur to you this was like your gym idea? Its this thing called a CAPTCHA. And the part thats really painful, the part where all the friction is, is having to get to the point where you know the language. HOFFMAN: Luis has also been able to revisit his early ambition to kill the TOEFL that anachronistic English test he and millions of others have to take if they want to study in the US. Its about figuring out what the actual goal of something is and how to get there. Thats just like if you just make the course really frustrating. Yeah, I think leaderboards and progress bars, people just want to fill up progress bars.
A little less discipline than a math professor.
So it turns out to be pretty hard. JUNE COHEN: This is so odd because Katie and I agree on most things. And in the first week, about 50,000 people applied to add a language. that contributors retain ownership of all content they submit. With reCAPTCHA, users were actively helping achieve something. None shall pass. Luis thought: There has to be a better way. ELIZABETH SAMPAT: Gamification works best when its really tied deeply into the users needs or desires, and when it really fails is when the gamified solutions are actually about the needs and desires of the person making the product, as opposed to the person thats actually going to be using it. Yeah. I wholeheartedly believe that our business model is actually better than the just standard charging for content. We were very mission-driven and people really saw that. I wholeheartedly believe that our business model is actually better than the just standard charging for content. You can be a passionate language learner without caring a jot about providing translation services. Like generating electricity theyll never use, or helping to digitize books theyll never read. AHN: We realized actually we could get a good number of people helping us add languages to Duolingo. It stands for Test of English as a Foreign Language. I had to take this test, and I had the problem that the country of Guatemala ran out of seats for this test. HOFFMAN: Generating revenue didnt disrupt the mission to bring free education to as many people as possible. However, this turned out to be a non-starter. But so is getting out of the way. When youre signing up for something online, youre presented with a set of random letters or numbers, bent out of shape as if theyre melting. Cameo appearance: Elizabeth Sampat (game designer, author) . : I wanted to do something related to education that could help a country like Guatemala, which is why I thought of language learning. Udi wasnt just recruiting in a very clever manner he was also crowdsourcing for Yahoo. So weve started patching this and now if you redo any lesson more than twice, we stop giving you points. Using these measurements, Duolingo can test a course on a selection of users, and tweak it to perfection before releasing it. HOFFMAN: And this is where young Luiss navet shows through: traditional gyms are placing a bet that most of their customers will lose their drive. Would you be able to help? And they responded pretty much immediately saying, Id love to and I have friends and Im willing to help., Note here how Luis uncovered a hugely valuable resource. I read somewhere that you also wanted to become a professor. One of the hardest things is finding a Goldilocks problem that computers are not quite good at but humans are really good at..
AHN: And he came and gave a talk of 10 problems that they didnt know how to solve at Yahoo. Because, oftentimes, youve actually created a really solid system, youve just created a system for the wrong goal. Just last year we took a concept from gaming where the idea is you actually make a leaderboard with strangers but with strangers that are equally committed to you. HOFFMAN: Luis started looking around for problems that could be solved with 10 seconds of human brain power. During those 10 seconds, the human brain that was solving this was doing something that computers could not do. Instead, he presented these engineers with the one thing they couldnt resist: an unsolvable problem. Yes. HOFFMAN: Philosophy. HOFFMAN: Its the Holy Grail or should that be the Rosetta Stone? Feedback will not be coming from just the disgruntled few. I love it. Am I going to get free access to these books? Well, okay, youll definitely have heard of the next one. Now, I love books but I feel very conflicted that I wasnt even aware that I was helping. We have these little game dynamics. Luis thought: There has to be a better way. And the number I came up with was about 200 million times a day. Those CAPTCHAs only took 10 seconds to do. It was, yeah, Ill just tell him some stuff thats exciting and then maybe theyll come work for Yahoo. But by now we have much more sophisticated ways of measuring how well people learn; we have a number of measurements and we can tell you know the quality of the different courses. And it was about this time that a lot of websites asked to use it. Because you will learn things from them. In Duolingo, Luis had found a way to monetize the activity of his users in a way that furthered their mission, rather than exploiting them. A little less discipline than a math professor. motivated by the shared mission of making it easy and free for anyone to start learning a language.