Do or Do Not: The Doer Effect

Learning By Doing

Let's pretend you work for a software company, and your manager wants you to create a clickable prototype for an app you are about to launch. 

Part of your prototype includes a dropdown menu that appears when users hover over it; however, you've never mocked up a dropdown menu before. 

You have two learning paths available to you. Would you rather:

1. Watch a video of someone making a dropdown menu.
2. Find a written worked-out example and follow along with your favorite prototyping software. 

Based on your choice: 

• Which path do you think will be easier to follow in the short term? 
• Which learning path will result in longer-term learning? 
• Which learning path might generalize to other related tasks?

"Do…or do not. There is no try." —Master Yoda, The Empire Strikes Back (1980)

Since the early days of this blog, we've opened with a Learning By Doing activity. What is the point of that? Is there evidence that this is a useful thing to do? 

Before jumping to the data, the idea of "learning by doing" is not at all new. Origins of the idea can be found in quotes by Aristotle [1] and Confucius [2]. John Dewey popularized the idea in American education in his book, Democracy and education [3].

There are theoretical reasons to believe that learning by doing will result in more durable and lasting changes. For example, I remember my memories better than I remember yours. Why? Because our brains are selfish. It is advantageous to our survival to remember the things we've done, both in terms of our successes and mistakes. We can also teach ourselves new strategies for solving problems that we've encountered several times in the past [4].

There are also empirical reasons why we learn better by active engagement. For example, we know from memory research that if we have an active hand in generating items to remember, we have a better shot of remembering them later. This effect goes by the name the generation effect.

If you're in a MOOC, be a Doer!

It seems that "learning by doing" is an effective learning mechanism when we test people in the lab. What does it look like out in the wild? What does the evidence look like in the classroom? 

Fortunately, with the advent of online education, we now have a ready-made venue that offers the opportunity for naturalistic experiments. Many "massive open online courses" (or "MOOCs") offer the learner several different types of learning activities. Most MOOCs have video-based lectures, online textbook passages; some even offer active-learning resources such as computer tutors or simulations. For MOOCs that offer both, which learning activities offer the best learning outcomes? 

Dr. Ken Koedinger and his collaborators conducted a pair of studies to answer precisely this question [5, 6]. They categorized students into several groups, based on their in-class behavior. Students who primarily watched videos were categorized as "Watchers", those who read the text were called "Readers", and those who completed the interactive learning activities were categorized as "Doers". Then the researchers looked at their performance on both quizzes and the final exam. The learning outcomes were extremely consistent. No matter which outcome variable they used, students who were categorized as Doers outperformed the Readers and Watchers. 

To estimate the impact of engaging in more interactive learning resources, they computed a statistical model that looked at the impact of pretest, doing activities, watching videos, or reading text on the final exam grade. The magnitude of the impact of doing the activities was huge. Completing the learning activities was six times more impactful than just watching videos or reading the text. 

This is strong evidence that learning by doing is an effective learning mechanism in online classrooms.

The Classroom Connection

The implication for education is fairly straightforward since the evidence was taken straight from an online course. In fact, their results are completely consistent with the ICAP framework from a previous post. As you move from a passive learning experience (e.g., reading text or watching a video) to more interactive learning environment (e.g., solving a problem or drawing a diagram), then learning tends to improve.

The educational goal, obviously, is to make the "lesson come alive" by actively engaging students in their learning. Active learning can assume an unlimited number of forms. But the point is that you don't want to rely just on asking your students to watch a video. Instead, follow up with a series of questions. Or, better yet, ask them to engage in the same activities as the video (kind of like the standard "I do, we do, you do" sequence). The danger is, if videos (or text) aren't followed up with an activity, then students risk tricking themselves into thinking "they get it."

Thanks for reading. Now go do something! 

Share and Enjoy!

Dr. Bob

Going Beyond the Information Given

[1] “For the things we have to learn before we can do them, we learn by doing them.” 

― Aristotle, The Nicomachean Ethics

[2] I hear and I forget

     I see and I remember

     I do and I understand 

—Confucius

[3] Of course, merely acting does not guarantee learning. There has to a meaningful connection of action to its consequences for there to be any useful learning. Dewey, J. (1923). Democracy and education: An introduction to the philosophy of education. macmillan.

[4] Anzai, Y., & Simon, H. A. (1979). The theory of learning by doing. Psychological review, 86(2), 124.

[5] Koedinger, K. R., Kim, J., Jia, J. Z., McLaughlin, E. A., & Bier, N. L. (2015, March). Learning is not a spectator sport: Doing is better than watching for learning from a MOOC. In Proceedings of the second (2015) ACM conference on learning @ scale (pp. 111-120).

[6] Koedinger, K. R., McLaughlin, E. A., Jia, J. Z., & Bier, N. L. (2016, April). Is the doer effect a causal relationship? How can we tell and why it's important. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 388-397).

Clueless: The Illusion of Explanatory Depth

Learning By Doing

You've seen a bicycle before, right? Of course you have! You probably learned how to ride when you were a kid, although maybe it's been a while since you've last ridden. I'm guessing you're probably not a bike mechanic, but you are familiar with the general shape of a bike and roughly how it works.

To kick off this post, I would like you to do two things. First, I would like you to rate your knowledge and familiarity of bicycles on a scale from 1 ("I know nothing about bikes or how they work.") to 7 ("I have a complete understanding of how a bike works."). The second task requires a pen and some paper. Below is a partial sketch of a bike; however, you will notice that it's missing a couple of parts (Fig. 1). I would like you to finish my drawing. Specifically, I would like you to add the pedals, chain, and the missing pieces of the frame [1]. 

Ready? Let's get started.

Figure 1. Complete the drawing of a bike by adding the missing pieces of the 
frame, the pedals, and show where the chain goes (used with permission).

The answer to this task is probably parked in a bike rack not far from where you're sitting. But if you need to see an image of a basic bike, with no gears or brakes, here is a great example. Now that you've seen the answer, how did you do? Did you make any mistakes?

"I thought I knew more than I did."

Most people, myself included, walk around thinking that we have a pretty good understanding of the way the world works. But every once in a while, we are confronted with the uncomfortable realization that we don't know as much as we think we do. If I asked you to re-rate your knowledge about how a bike works on the same 7-point scale, would it go up, down, or stay the same? If I had to guess, I would say it probably went down. This task is likely harder than you thought it would be [2].

The reason it was so hard is because of the illusion of explanatory depth, which is the belief that you understand something better than you actually do [3]. The illusion doesn't usually happen with facts or procedures. In other words, we're pretty good at estimating when we don't know a piece of trivia (e.g., "When did Amelia Earhart become the first woman to fly across the Atlantic ocean?" ) or how to do something (e.g., "Take the first derivative of f(x) = 3x² + 4x - 5"). But with semi-complex mechanical objects (e.g., a lock or a crossbow), people are often overly confident when it comes to explaining how things work. 

How Does This happen? 

You might be asking yourself: How does the illusion of explanatory depth happen? 

There are several potential sources of the illusion of explanatory depth, but here are two. First, the illusion might arise from a confusion between familiarity and understanding. Since we have all seen many bikes in our daily lives, we come to think that we understand them. When we learn how to ride, we might also think that we understand how a bike works because we have experience interacting with them. 

Second, the illusion might be caused by the ease by which we can mentally simulate the mechanical device under question. For example, if you say, "Imagine a bike." I can do so easily. The detail of my mental image, however, is fairly sparse. The demands of the task don't require me to do anything more than envision something with wheels and a frame. Thus, my performance on the task seems adequate for the current purposes. Only when we raise the stakes do I stumble and discover my lack of understanding.

The S.T.E.M. Connection

The illusion of explanatory depth is a problem for education, partly because it seems inevitable. When you are learning something new, a necessary first step is to become familiar with the terminology and the concepts. You can't learn about the anatomical structure of a frog without first becoming familiar with the names of the organs. So what can be done about the illusion?

The most basic antidote for moving beyond a superficial understanding is to try and answer the question: Why? or How? Once you are tasked with explaining how something works, only then will you discover the gaps in your knowledge. It is both illuminating and humbling. Here are several examples that I've recently run into:

• When are you able to see a new moon? 
• Since the tension on a crossbow is strong, how do you draw back the cord?
• How do tumblers in a lock prevent the cylinder from turning? 
• How does a zipper work?
• Why does the toilet keep running?

The illusion of explanatory depth is strong, and it probably serves a purpose. We can't carry around knowledge of every mechanical device around in our heads. Otherwise, we would never have enough time to do anything else! But we also don't want to dupe ourselves into thinking we understand something when we clearly don't [4]. So the best treatment is to keep asking yourself: Can I explain what the parts are, where they go, and how they interact? In other words, keep asking yourself if you understand the way things work.


Share and Enjoy!

Dr. Bob

Going Beyond the Information Given

[1] The drawing task was taken from Lawson, R. (2006). The science of cycology: Failures to understand how everyday objects work. Memory & Cognition, 34(8), 1667-1675. I am grateful to Rebecca Lawson for allowing me to recreate Figure 1.

[2] Another good example of a deceptively difficult task – and one that we talked about in a previous post – is trying to remember what a penny looks like. We've seen hundreds, maybe thousands, of pennies. But it is surprisingly difficult to identify the real penny. Try it for yourself. 

[3] Rozenblit, L., & Keil, F. (2002). The misunderstood limits of folk science: An illusion of explanatory depth. Cognitive science, 26(5), 521-562.

[4] Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121.