Do or Do Not: The Doer Effect

Learning By Doing

Let's pretend you work for a software company, and your manager wants you to create a clickable prototype for an app you are about to launch. 

Part of your prototype includes a dropdown menu that appears when users hover over it; however, you've never mocked up a dropdown menu before. 

You have two learning paths available to you. Would you rather:

1. Watch a video of someone making a dropdown menu.
2. Find a written worked-out example and follow along with your favorite prototyping software. 

Based on your choice: 

• Which path do you think will be easier to follow in the short term? 
• Which learning path will result in longer-term learning? 
• Which learning path might generalize to other related tasks?

"Do…or do not. There is no try." —Master Yoda, The Empire Strikes Back (1980)

Since the early days of this blog, we've opened with a Learning By Doing activity. What is the point of that? Is there evidence that this is a useful thing to do? 

Before jumping to the data, the idea of "learning by doing" is not at all new. Origins of the idea can be found in quotes by Aristotle [1] and Confucius [2]. John Dewey popularized the idea in American education in his book, Democracy and education [3].

There are theoretical reasons to believe that learning by doing will result in more durable and lasting changes. For example, I remember my memories better than I remember yours. Why? Because our brains are selfish. It is advantageous to our survival to remember the things we've done, both in terms of our successes and mistakes. We can also teach ourselves new strategies for solving problems that we've encountered several times in the past [4].

There are also empirical reasons why we learn better by active engagement. For example, we know from memory research that if we have an active hand in generating items to remember, we have a better shot of remembering them later. This effect goes by the name the generation effect.

If you're in a MOOC, be a Doer!

It seems that "learning by doing" is an effective learning mechanism when we test people in the lab. What does it look like out in the wild? What does the evidence look like in the classroom? 

Fortunately, with the advent of online education, we now have a ready-made venue that offers the opportunity for naturalistic experiments. Many "massive open online courses" (or "MOOCs") offer the learner several different types of learning activities. Most MOOCs have video-based lectures, online textbook passages; some even offer active-learning resources such as computer tutors or simulations. For MOOCs that offer both, which learning activities offer the best learning outcomes? 

Dr. Ken Koedinger and his collaborators conducted a pair of studies to answer precisely this question [5, 6]. They categorized students into several groups, based on their in-class behavior. Students who primarily watched videos were categorized as "Watchers", those who read the text were called "Readers", and those who completed the interactive learning activities were categorized as "Doers". Then the researchers looked at their performance on both quizzes and the final exam. The learning outcomes were extremely consistent. No matter which outcome variable they used, students who were categorized as Doers outperformed the Readers and Watchers. 

To estimate the impact of engaging in more interactive learning resources, they computed a statistical model that looked at the impact of pretest, doing activities, watching videos, or reading text on the final exam grade. The magnitude of the impact of doing the activities was huge. Completing the learning activities was six times more impactful than just watching videos or reading the text. 

This is strong evidence that learning by doing is an effective learning mechanism in online classrooms.

The Classroom Connection

The implication for education is fairly straightforward since the evidence was taken straight from an online course. In fact, their results are completely consistent with the ICAP framework from a previous post. As you move from a passive learning experience (e.g., reading text or watching a video) to more interactive learning environment (e.g., solving a problem or drawing a diagram), then learning tends to improve.

The educational goal, obviously, is to make the "lesson come alive" by actively engaging students in their learning. Active learning can assume an unlimited number of forms. But the point is that you don't want to rely just on asking your students to watch a video. Instead, follow up with a series of questions. Or, better yet, ask them to engage in the same activities as the video (kind of like the standard "I do, we do, you do" sequence). The danger is, if videos (or text) aren't followed up with an activity, then students risk tricking themselves into thinking "they get it."

Thanks for reading. Now go do something! 

Share and Enjoy!

Dr. Bob

Going Beyond the Information Given

[1] “For the things we have to learn before we can do them, we learn by doing them.” 

― Aristotle, The Nicomachean Ethics

[2] I hear and I forget

     I see and I remember

     I do and I understand 

—Confucius

[3] Of course, merely acting does not guarantee learning. There has to a meaningful connection of action to its consequences for there to be any useful learning. Dewey, J. (1923). Democracy and education: An introduction to the philosophy of education. macmillan.

[4] Anzai, Y., & Simon, H. A. (1979). The theory of learning by doing. Psychological review, 86(2), 124.

[5] Koedinger, K. R., Kim, J., Jia, J. Z., McLaughlin, E. A., & Bier, N. L. (2015, March). Learning is not a spectator sport: Doing is better than watching for learning from a MOOC. In Proceedings of the second (2015) ACM conference on learning @ scale (pp. 111-120).

[6] Koedinger, K. R., McLaughlin, E. A., Jia, J. Z., & Bier, N. L. (2016, April). Is the doer effect a causal relationship? How can we tell and why it's important. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 388-397).

Criss Cross: Aptitude by Treatment Interaction

Learning By Doing

 Let's play a fun game called Guess Which One. The answers are provided in the next section. No cheating! 

1. Guess which list of word-pairs has more accurate recall:
     A. A list provided by an experimenter.
     B. A list that you personally generated.

2. Guess which study method leads to deeper learning:
     A. Re-reading the material
     B. Testing yourself on the material you just read.

3. Guess which instructional method is better: 
     A. One-on-one human tutoring
     B. An intelligent tutoring system (i.e., a computer tutor)

4. Guess which study strategy is more effective: 
     A. Paraphrasing an expository text
     B. Self-explaining an expository text

5. Guess which type of text leads to a better understanding of the subject matter: 
     A. A minimally coherent text
     B. A globally coherent text

"Criss cross" –Owen, Throw Momma From the Train

If you've been reading this blog for a while now, you may have noticed that some answers have been discussed in previous posts.

1. The generation effect would predict that personally generated items are more memorable than those provided by someone else; therefore, the answer is A. 

2. The research on desirable difficulties predicts that students are better off quizzing themselves than re-reading the material. The best answer is A. 

3. This is a tough one. If you believe the early research on Intelligent Tutoring Systems, humans were the gold standard. But then Kurt VanLehn called that conclusion into question. The answer is A (but I'll accept B if you cite VanLehn, 2011). 

4. The research on self-explaining pretty clearly indicates that students learn more when they self-explain because they are using their background knowledge and reasoning to repair their flawed mental models. The answer is unequivocally B.

5. The answer is A or B. Wait, what? That's right! The answer to #5 is "it depends." This post is about the conditions upon which learning outcomes depend. Read on.

The Aptitude x Treatment Interaction

To better understand what's going on with the fifth question, let's take a step back and talk a little bit about research methodology. One of the most common experimental studies is to contrast the outcome of an experimental group with a control group. But instead of just comparing the outcomes of an experimental condition with a control condition, you have two levels of each independent variable. 

To make this more concrete, suppose you hypothesize that listening to music hurts learning performance. However, you don't think that all music hurts. Instead, you hypothesize that lyrically complex music hurts learning lists of words; whereas, instrumental music doesn't have any impact at all. 

To test your hypothesis, you design a study where there are two types of music and two types of lists to memorize. For the musical manipulation, you play a lyrically complex song versus techno music without any words. For the item manipulation, the first is a list of only words, and the second list only contains numbers. When you run this experiment, you plot the results with a line graph (see Figure 1). 

Figure 1: A cross-over interaction between music type and item type. 

Notice that the impact of music depends on the interaction between the type of music and the item type. If you listen to techno music, then there isn't any improvement or cost to recall. If you listen to lyrically complex music, then you get a little boost when memorizing lists of numbers. But if you listen to a song with lyrics, then it completely wipes out a participant's ability to memorize words. 

Said another way, there is a music-by-item interaction. When we talk about learning manipulations, we need to be sensitive to potential interactions between a student's aptitude and the learning situation they are in. Why? Because their learning outcomes might depend on it! 

Going back to our rousing game of Guess Which One, the answer to question #5 is "it depends" because students who have lots of background knowledge learn better from a minimally coherent text while students who do not have the same background knowledge learn better from a globally coherent text [1]. In other words, there is an aptitude (i.e., high vs. low background knowledge) by treatment (i.e., high vs. low textual coherence) interaction.

Students with a large amount of background knowledge are better served by minimally coherent texts because they must supply the missing information. They need to do more generative work while reading the text. As we have seen in other contexts, being generative during learning is beneficial for deep learning. The low-prior knowledge students, however, require a maximally coherent text because they lack the background knowledge to generate the connections. Therefore, they need more support and scaffolding when learning a new topic. 

The S.T.E.M. Connection

The above finding underscores how important both formative assessments and personalized learning environments are. In theory, if a teacher had enough data to diagnose how well each student understood a topic, then they could assign each student a different text. A knowledgeable student would get a minimally coherent text, while a low-knowledge student would get a maximally coherent text. 

Unfortunately, in practice, things are much more tricky. It would be a lot to ask a teacher to come up with two (or more) versions of a textbook. However, some labs are applying latent semantic analysis (LSA) to help match a given student to a particular version of a text [2]. The goal is to select a text that maximizes the reading comprehension for a particular student. This is an exciting area of research and one to keep an eye on as more textbooks are distributed digitally.

Someday, perhaps we can synthesize all of the (right) answers to Guess Which One and develop a learning platform that can handle the multitude of interactions between all of the variables that influence learning. That would be extremely powerful (and wouldn't require anyone to be thrown from a train!).


Share and Enjoy!

Dr. Bob

Going Beyond the Information Given

[1]  McNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996). Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 14(1), 1-43.

[2] Wolfe, M. B., Schreiner, M. E., Rehder, B., Laham, D., Foltz, P. W., Kintsch, W., & Landauer, T. K. (1998). Learning from text: Matching readers and texts by latent semantic analysis. Discourse Processes, 25(2-3), 309-336.

Pop a Cap: The iCAP Framework

Learning By Doing

Before we begin, let's learn about how a jet engine works [1]. While you watch this 5-minute video, do your best to learn the contents of the video, while paying attention to your learning process. That is, make a mental note of what you're doing to learn the material. I know that's probably going to split your attention across two different sources of information (I therefore apologize!). Finally, while you are watching, remember not to fall prey to the illusion of explanatory depth! I know that's a lot to ask, but try your best. Here you go:

Friggin' jet engines...how do they work?

Pop Quiz! Do your best to answer the following questions:

• Why is cold air super-heated in the combustion chamber?
• What shape are the stator blades on the turbine?
• Why is the outlet narrower than the intake?
• We all know jet engines are extremely loud. What makes them so noisy?

Now it's time to introspect on your learning experience. While you were watching the video, what did you do to learn the material? Did you:

1. Passively listen to the voiceover and watch the animations?
2. Pause the video and take notes?
3. Ask yourself questions or attempt to connect the material to what you already know?
4. Talk to a friend about the video? 

Depending on the activities in which you engaged, we can make an educated guess about the likelihood of your learning the material. The iCAP Framework [2] makes the following predictions:

1. Shallow learning occurs when a student passively processes the material;
2. Better learning results when the student actively does something to learn the material;
3. We would observe even better learning if the student is making connections and constructively working with the material;
4. The best learning outcome would be observed in an interactive discussion.

Each of these four learning processes are defined in the sections that follow.

Passive

This is the easiest learning process to describe. Passive learning occurs when the student engages in no overt behavior. A great example is a student listening to a lecture. Presumably, the learner is listening to the words in the lecture and looking at the images (assuming there is a slide presentation). In other words, during passive learning, the information is being attended to and it passes through working memory. From there, it is anybody's guess as to the ultimate fate of that information. In the best case, the presented material is stored in long-term memory in such a way that it is available for later recall. However, from the outside observer's point of view, the student is not overtly doing anything to remember the material.

Active

Here's where things get a little more interesting. An active learning process requires that the student engage in an overtly observable behavior. Going back to our lecture example, a student would be said to be engaged in active learning if she is taking notes. Another example would be highlighting a passage in a textbook. The external behavior that occurs during active learning results in some external representation (e.g., notes or highlighting). Active learning is the hallmark of many other learning theories, which suggest that the student should be doing something while learning. In fact, active learning forms the basis of John Dewey's pragmatic educational philosophy [3]. 

Constructive

The problem with active learning is that it mainly focuses on the overt learning behavior instead of considering the content or quality of those behaviors. Thus, a constructive learning process is one in which the learner goes beyond the information that is immediately presented. For example, suppose I gave a student the following function: f(x) = ax² + bx + c, and I tell the student, "This is a quadratic function." Recognizing that the prefix "quad-" is the same as a class of shapes (i.e., quadrilaterals), the student points out that a square is a quadrilateral, and that the formula for calculating the area of a square is A = x². This student is wildly constructive because she has made connections to her previous knowledge and elaborated the original message. Thus, constructive learning takes active learning one step further by adding new information to the target material [4].

Interactive

Being constructive is a great learning strategy because a student is much more likely to remember something when he or she generates it for him- or herself (see the generation effect). However, construction typically happens while learning alone. Interactive learning says that the lesson or material will be better understood if it is done in the context of a learning partner. The reason for the advantage is that two different people typically have non-overlapping knowledge, in that they share some of the same knowledge, but they also know some things that the other person does not. We also see things differently. The reason interactive learning can be better than constructive learning is when collaborators infer new knowledge together. In the literature, this idea goes by many names, including co-construction or co-inference.

To summarize, Table 1 defines each learning process and provides a concrete example.

Learning Process	Definition	Example
Passive	No overt activity	Listening to a lecture
Active	Overt activity is observed Learning by doing	Taking notes during a lecture
Constructive	Going beyond the given information	Drawing a concept map
Interactive	Co-inferring new information with a partner	Collaboratively identifying differences and similarities

Table 1. A summary of each learning process.

The S.T.E.M. Connection

Marshall McLuhan famously said, "The medium is the message." When students are given a video to learn from, it is very tempting to sit back and assume a passive learning orientation. After all, that's what we do when we watch television. That might change with the rise in popularity of educational videos; however, most of us have been trained to treat videos as entertainment. One way to combat passive learning is to give the student a task while watching the video that pushes them toward the active/constructive end of the continuum. 

Another suggestion is to ask students to watch videos in pairs with the explicit instructions to pause the video and ask each other questions. This is a good way to structure collaborative learning because the students can learn from the video as well as from each other [5].

I realize it's not always possible, or even desirable, to ask students to work collaboratively and co-construct new information. But I think the iCAP Framework is a useful way of organizing the learning literature because it helps highlight learning processes that are more (or less) effective. Our goal as instructional designers is to use the framework to select the appropriate learning process for the task at hand.


Share and Enjoy!

Dr. Bob

Going Beyond the Information Given

[1] The video "Jet Engine, How it works ?" is produced by Learn Engineering. If you are interested in learning more about a variety of other engineering topics (e.g., wind turbines), this is a good resource.

[2] Chi, M. T. H., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49, 219-243.

[3] I find this idea so useful that I open each blog with the heading Learning By Doing.

[4] Again, I find this idea so compelling that each blog also has also has a section entitled Going Beyond the Information Given.

[5] Chi, M. T. H., Roy, M., & Hausmann, R.G.M. (2008) Observing tutorial dialogues collaboratively: Insights about human tutoring effectiveness from vicarious learning. Cognitive Science, 32(2), 301-341.

The Pain Teaches Me: Desirable Difficulties

No Pains, No gains. 
–Robert Herrick (1650)

It's a simple matter of logic. 
I'm not like other people. 
I can't stand pain, it hurts me. 
–Daffy Duck (1961)

Pop Quiz: We all have implicit theories about the conditions under which learning is maximized. Let's make your theory explicit. For each item below, which scenario will lead to more robust learning?

	Option A	or	Option B
1.	Studying in the same room where you are going to take the test		Studying in various environments
2.	Solving a bunch of problems that are similar		Switching between different types of problems
3.	Cramming for an exam the night before		Studying a little bit over a longer period of time
4.	Reading and rereading material until it becomes familiar		Reading material once and quizzing yourself
5.	Reading a text that contains gaps in the reasoning		Reading a highly coherent text that fills in all the gap

Recalibrating Our Intuitions

Sometimes our intuitions about what is best for learning are mis-calibrated. The reason is likely due to a distinction between storage strength and retrieval strength [1]. Storage strength is how well a memory is embedded within a larger network of related concepts. Retrieval strength refers to how quick or easily you can recall an item from long-term memory.  

When learning scenarios boost retrieval strength, we might be tempted to believe that we will remember the same material days, weeks, or maybe even months later. However, storage strength is more likely to be related to our successful recall. 

How, then, can we design learning events so that storage strength is maximized? One way is to introduce what is called a desirable difficulty into the learning situation. The difficulty is said to be "desirable" when it leads to more robust learning. Here are some desirable difficulties that we can introduce into the learning process.

Consistent vs. Inconsistent Settings

There is a semi-famous memory study where experimenters asked volunteers to memorize a list of words in one of two settings [2]. The first group studied their list on dry land. The second group donned scuba gear and jumped in a pool to study their list. Then the experimenters altered where the volunteers had to recall their list. The consistent group studied and recalled their list in the same setting (either both on land or both underwater) and the inconsistent group switched the context of study and recall (e.g., if they studied underwater, then they recalled on land). Guess what the experimenters found? The consistent group had better recall than those who switched settings.  

But what if we alter the amount of study that occurs? In other words, what if we asked our brave group of volunteers to study their list twice, either in the same setting or two different settings? Who would have better recall then? It turns out that studying the same material in two different settings provides a slight edge in performance. Why is this the case? It might be because the students are forced to generalize their knowledge so that idiosyncratic environmental cues are not used to help recall the information. 

Blocked vs. Intermixed Problem Solving

The above studies manipulated the context of study and retrieval. What if we manipulate the material instead? It might seem on the surface that you would want to study the same thing over and over until it becomes proceduralized. A more difficult study scenario would be to take concepts taught in different lessons and intermix them. For example, suppose you are teaching separate lessons on perimeter, area, and volume. Should you ask your students to solve a block of perimeter problems, then a block of area problems, and then a block of volume problems? Or should you teach all the lessons and then have students solve a mixture of problems from all three lessons? Suggestive evidence from a study that contrasted blocked versus intermixed problems found a consistent advantage for the intermixed problems [3].

Mass vs. Distributed Practice

As we saw in an earlier post, our memory seems to be designed such that information that isn't required tends to drop off in terms of its retrieval strength. Names of people who we haven't seen in five years don't come to mind as readily as the name of someone you saw last week. We also know that an attempt to retrieve a memory can help to reinforce that item in long-term memory. So, how do we optimize our study so that we counterbalance memory decay with making sure that we aren't wasting time on items that are already known? The best method is to increase the delay between study sessions, which will lead to a longer retention interval [4]. 

Presentation vs. Generation

When trying to learn something new, should I read and reread the material while taking notes? Or should I spend less time reading and taking notes and try testing myself instead? Perhaps a counterintuitive finding from the memory literature is that attempting to remember something has the effect of increasing that memory's strength. Therefore, it might make sense to spend less time reading and more time quizzing yourself.  

Why might this be the case? We read earlier about the generation effect, which states that memories are more likely to be recalled when we create them ourselves. When we quiz ourself on the material that we just read, it pushes our cognitive processing of the text toward the generation end of the spectrum (with the other end being repeated exposure). 

Incomplete vs. Complete Text Material

Do you learn better from a text passage that is complete or incomplete? It turns out that this is a trick question because it depends on your level of prior knowledge [5]. If you are a relative expert in a domain, you tend to learn more from an incomplete text because you must actively try to make sense of the material. If it is a more complete text, then the presentation becomes tedious and you tend to passively process the information. A low-knowledge reader, on the other hand, does not have the background to piece together an incomplete text. Instead, they need the extra information to help connect sentences together, as well as link together higher-order concepts.

The STEM Connection

Learning is hard. Why would we want to make it even more difficult? The main reason is to ensure that we don't fall into the trap of mistaking retrieval strength for storage strength. When we recognize that the goal of a learning event is to create a durable memory trace, then we can intentionally make our learning more difficult. Like exercise, our hard work will be rewarded when our learning withstands the test of time. 


Share and Enjoy!

Dr. Bob

For More Information

[1] Bjork, E. L., & Bjork, R. A. (2011). Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. Psychology and the real world: Essays illustrating fundamental contributions to society, 56-64.

[2] Godden, D; Baddeley, A. (1975). Context dependent memory in two natural environments. British Journal of Psychology 66 (3): 325–331.

[3] Rohrer, D., & Taylor, K. (2007). The shuffling of mathematics practice problems im- proves learning. Instructional Science, 35, 481–498.

[4] Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin, 132(3), 354-380.

[5] McNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996). Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 14(1), 1-43.