More Human Than Human: Human Tutoring

Editorial Note: As of April 1, 2018, our paper, Learning from human tutoring, has been cited over 1,000 times. That qualifies it as a "citation classic." Congratulations to my co-authors, especially Micki, for making this happen!

Learning By Doing

Let's start with a quiz. Rank order the following learning strategies from most effective to least effective.

1. Listening to a lecture
2. Participating in a large-group discussion
3. One-on-one human tutoring
4. On-the-job training or an apprenticeship 
5. Hands-on laboratory activity
6. Collaborative problem solving
7. Watching an online video
8. Studying worked-out examples
9. Solving problems with a computer tutor

"Sort It Out" –Vinny, Snatch (2000)

This is probably an impossible sorting task because it isn't always clear when one particular learning strategy should be employed. For example, it might depend on the learner's prior knowledge. If the material is completely new, then one-on-one human tutoring might be the best approach. Alternatively, if the student is familiar with the material and just needs a refresher, then watching a video might be the most efficient way to learn. In other words, there isn't a grand unified learning theory that can confidently recommend the exact learning activity for a student who is learning a specific topic.

That is not to say, however, we can't make an educated guess [1, 2]. We know that requiring a student to be active during the learning process (i.e., asking deep questions) is better than letting him or her be passive (i.e., watching a video). This general framework was covered in a previous post.

Thus, being active during learning is important. We also know that being interactive is also important for learning. What kind of instruction encourages interactivity? Both parents and students probably agree that one-on-one human tutoring is highly interactive and extremely effective. The open questions is: Why is human tutoring effective? 

Three Possible Explanations

Here are three possible explanations for why human tutoring is effective. 

The Tutor-centered Hypothesis

The first hypothesis suggests that human tutoring is effective because the tutor has a deep understanding of both the topic and pedagogical moves that are useful for eliciting learning from a student. For example, suppose a tutor is teaching how to solve projectile-motion problems. She knows that students frequently hold the misconception that motion in the x-direction depends on motion in the y-direction. Therefore, the tutor knows the exact question to ask when the student displays this confusion. Thus, tutoring is effective because the tutor knows exactly what to say and when to say it.

The Student-centered Hypothesis

In contrast, there is an equally valid way of looking at tutoring that focuses on the student. Instead of giving the tutor all the credit, the student is the reason why tutoring is effective. When the student is engaged with constructing his or her own understanding, then tutoring will be effective. We've seen in past research the importance of the generation effect and self-explaining. Tutoring will be effective in so far as the student is able to engage in these constructive learning activities.

The Interactive Hypothesis

The third explanation for the effectiveness of human tutoring takes a more holistic view of tutoring. Instead of crediting the student or tutor, the effectiveness of tutoring arises from their mutual interaction. In other words, the tutor appropriately sets the current learning task and the student then has the opportunity to engage in this task. The tutor is available to answer questions and give feedback as needed. The tutor is also available to back up and propose an easier task if the student gets stuck.

Let's Put It To a Test

To test these three equally plausible explanations, we conducted the following study [3]. The first part of the experiment asked tutors to teach eighth grade students about the human circulatory system. The tutors, who were nursing students, weren't restricted in any way. Their goal, of course, was to help prepare students to answer really tough questions about the circulatory system. In the second part of the experiment, we took the same tutors, gave them a fresh batch of students, and told them not to give away any information. Instead, we wanted them to get the students to do all the work.

Before we get to the results, we had to make sure that the tutors understood and followed our instructions. Sure enough, in the second round of tutoring, they explained a lot less and  asked a bunch more questions. Therefore, we were satisfied that our manipulation was a success. 

In the first study, we found a correlation between the number of tutorial explanations and shallow learning. I don't find this result very surprising. The student read the text and the tutor explained it, which was effectively like hearing the same lesson twice. The more interesting finding was that when students made reflective comments about their understanding or learning. Those types of comments were correlated with deep learning. So far, we have evidence for the first two hypotheses. What the tutor does is important for shallow learning (i.e., give tons of explanations) and what the students does is important for deep learning (i.e., be reflective or metacognitive).

In the second study, we found something else that was interesting. Recall that the tutors weren't allowed to explain or give feedback. Instead, they asked a lot of "scaffolding" questions (e.g., giving hints, asking fill-in-the-blank questions, or asking for an example). There was about three times as much scaffolding in the second study, and the number of explanations dropped precipitously (see Fig. 1). What makes this finding interesting is that the learning outcomes were the same in both studies. The students excelled on their post-tests. The reason why, we argue, is because the tutors created situations where the student could be generative, and when they engaged in the generative activities, they learned a great deal from the tutoring session.

Figure 1. The number of scaffolding and explanation
episodes for the first and second study.

The S.T.E.M. Connection

The results from the first study clearly indicate that tutors, when left to their own devices, will try and explain the material to the student. This isn't necessarily a bad thing. Repetition generally leads to learning. Unfortunately, that isn't the best way to learn the really hard material. Instead, it is a good idea to create a tutoring situation where the tutor is asking the student questions and pushing them to further develop a line of reasoning. Tutors are effective because when they ask a tough question, and when they hear the student struggle, then they back up and ask an easier question. That is where the metaphor of "scaffolding" comes from. On the flip side, it is important for the student to monitor his or her understanding. We've talked about the importance of metacognition and in this case, metacognitive monitoring was correlated with learning. To the extent possible, we should teach our students to monitor their evolving comprehension.

Human tutoring is effective for so many reasons, and it doesn't just boil down to one thing. But as we continue to investigate and probe, eventually we will be able to figure out tutoring's secret sauce. Once we do, then we can provide tutors with concrete suggestions to become even more effective. And maybe, just maybe, we can build a computer tutoring system that is as good as humans. Maybe even better [4]. 


Share and Enjoy!

Dr. Bob

Going Beyond the Information Given

[1] Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012). The Knowledge‐Learning‐Instruction framework: Bridging the science‐practice chasm to enhance robust student learning. Cognitive science, 36(5), 757-798.

[2] Chi, M. T. H., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49, 219-243.

[3] Chi, M. T., Siler, S. A., Jeong, H., Yamauchi, T., & Hausmann, R. G. (2001). Learning from human tutoring. Cognitive Science, 25(4), 471-533.

[4] VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems and other tutoring systems. Educational Psychologist, 46, 4, 197-221.

Pop a Cap: The iCAP Framework

Learning By Doing

Before we begin, let's learn about how a jet engine works [1]. While you watch this 5-minute video, do your best to learn the contents of the video, while paying attention to your learning process. That is, make a mental note of what you're doing to learn the material. I know that's probably going to split your attention across two different sources of information (I therefore apologize!). Finally, while you are watching, remember not to fall prey to the illusion of explanatory depth! I know that's a lot to ask, but try your best. Here you go:

Friggin' jet engines...how do they work?

Pop Quiz! Do your best to answer the following questions:

• Why is cold air super-heated in the combustion chamber?
• What shape are the stator blades on the turbine?
• Why is the outlet narrower than the intake?
• We all know jet engines are extremely loud. What makes them so noisy?

Now it's time to introspect on your learning experience. While you were watching the video, what did you do to learn the material? Did you:

1. Passively listen to the voiceover and watch the animations?
2. Pause the video and take notes?
3. Ask yourself questions or attempt to connect the material to what you already know?
4. Talk to a friend about the video? 

Depending on the activities in which you engaged, we can make an educated guess about the likelihood of your learning the material. The iCAP Framework [2] makes the following predictions:

1. Shallow learning occurs when a student passively processes the material;
2. Better learning results when the student actively does something to learn the material;
3. We would observe even better learning if the student is making connections and constructively working with the material;
4. The best learning outcome would be observed in an interactive discussion.

Each of these four learning processes are defined in the sections that follow.

Passive

This is the easiest learning process to describe. Passive learning occurs when the student engages in no overt behavior. A great example is a student listening to a lecture. Presumably, the learner is listening to the words in the lecture and looking at the images (assuming there is a slide presentation). In other words, during passive learning, the information is being attended to and it passes through working memory. From there, it is anybody's guess as to the ultimate fate of that information. In the best case, the presented material is stored in long-term memory in such a way that it is available for later recall. However, from the outside observer's point of view, the student is not overtly doing anything to remember the material.

Active

Here's where things get a little more interesting. An active learning process requires that the student engage in an overtly observable behavior. Going back to our lecture example, a student would be said to be engaged in active learning if she is taking notes. Another example would be highlighting a passage in a textbook. The external behavior that occurs during active learning results in some external representation (e.g., notes or highlighting). Active learning is the hallmark of many other learning theories, which suggest that the student should be doing something while learning. In fact, active learning forms the basis of John Dewey's pragmatic educational philosophy [3]. 

Constructive

The problem with active learning is that it mainly focuses on the overt learning behavior instead of considering the content or quality of those behaviors. Thus, a constructive learning process is one in which the learner goes beyond the information that is immediately presented. For example, suppose I gave a student the following function: f(x) = ax² + bx + c, and I tell the student, "This is a quadratic function." Recognizing that the prefix "quad-" is the same as a class of shapes (i.e., quadrilaterals), the student points out that a square is a quadrilateral, and that the formula for calculating the area of a square is A = x². This student is wildly constructive because she has made connections to her previous knowledge and elaborated the original message. Thus, constructive learning takes active learning one step further by adding new information to the target material [4].

Interactive

Being constructive is a great learning strategy because a student is much more likely to remember something when he or she generates it for him- or herself (see the generation effect). However, construction typically happens while learning alone. Interactive learning says that the lesson or material will be better understood if it is done in the context of a learning partner. The reason for the advantage is that two different people typically have non-overlapping knowledge, in that they share some of the same knowledge, but they also know some things that the other person does not. We also see things differently. The reason interactive learning can be better than constructive learning is when collaborators infer new knowledge together. In the literature, this idea goes by many names, including co-construction or co-inference.

To summarize, Table 1 defines each learning process and provides a concrete example.

Learning Process	Definition	Example
Passive	No overt activity	Listening to a lecture
Active	Overt activity is observed Learning by doing	Taking notes during a lecture
Constructive	Going beyond the given information	Drawing a concept map
Interactive	Co-inferring new information with a partner	Collaboratively identifying differences and similarities

Table 1. A summary of each learning process.

The S.T.E.M. Connection

Marshall McLuhan famously said, "The medium is the message." When students are given a video to learn from, it is very tempting to sit back and assume a passive learning orientation. After all, that's what we do when we watch television. That might change with the rise in popularity of educational videos; however, most of us have been trained to treat videos as entertainment. One way to combat passive learning is to give the student a task while watching the video that pushes them toward the active/constructive end of the continuum. 

Another suggestion is to ask students to watch videos in pairs with the explicit instructions to pause the video and ask each other questions. This is a good way to structure collaborative learning because the students can learn from the video as well as from each other [5].

I realize it's not always possible, or even desirable, to ask students to work collaboratively and co-construct new information. But I think the iCAP Framework is a useful way of organizing the learning literature because it helps highlight learning processes that are more (or less) effective. Our goal as instructional designers is to use the framework to select the appropriate learning process for the task at hand.


Share and Enjoy!

Dr. Bob

Going Beyond the Information Given

[1] The video "Jet Engine, How it works ?" is produced by Learn Engineering. If you are interested in learning more about a variety of other engineering topics (e.g., wind turbines), this is a good resource.

[2] Chi, M. T. H., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49, 219-243.

[3] I find this idea so useful that I open each blog with the heading Learning By Doing.

[4] Again, I find this idea so compelling that each blog also has also has a section entitled Going Beyond the Information Given.

[5] Chi, M. T. H., Roy, M., & Hausmann, R.G.M. (2008) Observing tutorial dialogues collaboratively: Insights about human tutoring effectiveness from vicarious learning. Cognitive Science, 32(2), 301-341.

The Pain Teaches Me: Desirable Difficulties

No Pains, No gains. 
–Robert Herrick (1650)

It's a simple matter of logic. 
I'm not like other people. 
I can't stand pain, it hurts me. 
–Daffy Duck (1961)

Pop Quiz: We all have implicit theories about the conditions under which learning is maximized. Let's make your theory explicit. For each item below, which scenario will lead to more robust learning?

	Option A	or	Option B
1.	Studying in the same room where you are going to take the test		Studying in various environments
2.	Solving a bunch of problems that are similar		Switching between different types of problems
3.	Cramming for an exam the night before		Studying a little bit over a longer period of time
4.	Reading and rereading material until it becomes familiar		Reading material once and quizzing yourself
5.	Reading a text that contains gaps in the reasoning		Reading a highly coherent text that fills in all the gap

Recalibrating Our Intuitions

Sometimes our intuitions about what is best for learning are mis-calibrated. The reason is likely due to a distinction between storage strength and retrieval strength [1]. Storage strength is how well a memory is embedded within a larger network of related concepts. Retrieval strength refers to how quick or easily you can recall an item from long-term memory.  

When learning scenarios boost retrieval strength, we might be tempted to believe that we will remember the same material days, weeks, or maybe even months later. However, storage strength is more likely to be related to our successful recall. 

How, then, can we design learning events so that storage strength is maximized? One way is to introduce what is called a desirable difficulty into the learning situation. The difficulty is said to be "desirable" when it leads to more robust learning. Here are some desirable difficulties that we can introduce into the learning process.

Consistent vs. Inconsistent Settings

There is a semi-famous memory study where experimenters asked volunteers to memorize a list of words in one of two settings [2]. The first group studied their list on dry land. The second group donned scuba gear and jumped in a pool to study their list. Then the experimenters altered where the volunteers had to recall their list. The consistent group studied and recalled their list in the same setting (either both on land or both underwater) and the inconsistent group switched the context of study and recall (e.g., if they studied underwater, then they recalled on land). Guess what the experimenters found? The consistent group had better recall than those who switched settings.  

But what if we alter the amount of study that occurs? In other words, what if we asked our brave group of volunteers to study their list twice, either in the same setting or two different settings? Who would have better recall then? It turns out that studying the same material in two different settings provides a slight edge in performance. Why is this the case? It might be because the students are forced to generalize their knowledge so that idiosyncratic environmental cues are not used to help recall the information. 

Blocked vs. Intermixed Problem Solving

The above studies manipulated the context of study and retrieval. What if we manipulate the material instead? It might seem on the surface that you would want to study the same thing over and over until it becomes proceduralized. A more difficult study scenario would be to take concepts taught in different lessons and intermix them. For example, suppose you are teaching separate lessons on perimeter, area, and volume. Should you ask your students to solve a block of perimeter problems, then a block of area problems, and then a block of volume problems? Or should you teach all the lessons and then have students solve a mixture of problems from all three lessons? Suggestive evidence from a study that contrasted blocked versus intermixed problems found a consistent advantage for the intermixed problems [3].

Mass vs. Distributed Practice

As we saw in an earlier post, our memory seems to be designed such that information that isn't required tends to drop off in terms of its retrieval strength. Names of people who we haven't seen in five years don't come to mind as readily as the name of someone you saw last week. We also know that an attempt to retrieve a memory can help to reinforce that item in long-term memory. So, how do we optimize our study so that we counterbalance memory decay with making sure that we aren't wasting time on items that are already known? The best method is to increase the delay between study sessions, which will lead to a longer retention interval [4]. 

Presentation vs. Generation

When trying to learn something new, should I read and reread the material while taking notes? Or should I spend less time reading and taking notes and try testing myself instead? Perhaps a counterintuitive finding from the memory literature is that attempting to remember something has the effect of increasing that memory's strength. Therefore, it might make sense to spend less time reading and more time quizzing yourself.  

Why might this be the case? We read earlier about the generation effect, which states that memories are more likely to be recalled when we create them ourselves. When we quiz ourself on the material that we just read, it pushes our cognitive processing of the text toward the generation end of the spectrum (with the other end being repeated exposure). 

Incomplete vs. Complete Text Material

Do you learn better from a text passage that is complete or incomplete? It turns out that this is a trick question because it depends on your level of prior knowledge [5]. If you are a relative expert in a domain, you tend to learn more from an incomplete text because you must actively try to make sense of the material. If it is a more complete text, then the presentation becomes tedious and you tend to passively process the information. A low-knowledge reader, on the other hand, does not have the background to piece together an incomplete text. Instead, they need the extra information to help connect sentences together, as well as link together higher-order concepts.

The STEM Connection

Learning is hard. Why would we want to make it even more difficult? The main reason is to ensure that we don't fall into the trap of mistaking retrieval strength for storage strength. When we recognize that the goal of a learning event is to create a durable memory trace, then we can intentionally make our learning more difficult. Like exercise, our hard work will be rewarded when our learning withstands the test of time. 


Share and Enjoy!

Dr. Bob

For More Information

[1] Bjork, E. L., & Bjork, R. A. (2011). Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. Psychology and the real world: Essays illustrating fundamental contributions to society, 56-64.

[2] Godden, D; Baddeley, A. (1975). Context dependent memory in two natural environments. British Journal of Psychology 66 (3): 325–331.

[3] Rohrer, D., & Taylor, K. (2007). The shuffling of mathematics practice problems im- proves learning. Instructional Science, 35, 481–498.

[4] Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin, 132(3), 354-380.

[5] McNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996). Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 14(1), 1-43.

If the Shoe Fits: The Instructional Fit Hypothesis

Take a look at the map below. Suppose you live on Jackson Avenue, and you need to go to the store to buy a new pair of shoes. What route will you take? 

Once you've planned your route, how will you remember it? Will you:

• Form a mental map, position yourself on that map, and update your position as you travel
• Form a mental list of verbal directions (e.g., head toward Main Street; after Main, turn right on 1st; then, take a left on Madison; the shoe store will be on the right). 

In the unlikely event that you take a wrong turn, which strategy will be more helpful in getting you to your destination? Which representation is easier to store in working memory? Which one is easier to use while driving? 

Connecting Instruction to Learning

In this blog, we've discussed many different types of representations that the mind uses to store and organize information. We also have talked about various learning strategies that help us acquire new information. One aspect of the discussion that's been missing is how these pieces all fit together. Are there some learning strategies that are more likely to give rise to one type of representation over another?

To answer this question, my collaborators and I set out to test the hypothesis that certain forms of instruction inspire learning events that translate into specific types of representations. We called it the instructional fit hypothesis, in that instruction should match the type of learning that we want to elicit. Before we talk about how we tested the instructional fit hypothesis, let's review the assumptions on which it is based.

Our Assumptions

First, we assume there are different types of mental representations that we use to reason about the world and to solve problems. This assumption is supported by plenty of evidence that people construct and use many different types of representations. For the purposes of the present discussion, let's focus on two representations: mental models and problem-solving schemas. As we saw, a mental model is an image or dynamic simulation that allows the individual to make inferences based on that model. For example, we might not know how many windows there are in our house, but we can mentally walk from room to room and count them. A mental model can also be incomplete or incorrect, in which case I can add details to or correct my model as I encounter new relevant information. A problem-solving schema, in contrast, can be thought of as a recipe for solving a problem. In a previous post, we introduced the idea of a production rule, which is an if/then statement that says what to do when certain conditions are met. A problem-solving schema links multiple if/then statements so that a problem can be solved.

Second, we assume that certain types of representations are better suited to solve specific types of problems. It is more expedient to use a mental model of the circulatory system to diagnose a heart problem than it would be to use a problem-solving schema in which several production rules have to be tested to find one that matches the symptoms to the root cause. Likewise, it is easier to solve a multi-step math problem using a problem-solving schema than by constructing a mental model of that particular problem. To be effective and efficient, the representation and the problem-solving demands should match. 

Our third and final assumption is that certain types of instruction lead students to engage in specific types of cognitive processing. For example, suppose I instruct one of my classes to write a summary of a passage about the circulatory system. For my other class, I ask them to answer difficult questions like: Why would the distribution of oxygen be less efficient if there is a hole in the septum? The first class would concentrate on a surface-level understanding of the text because the task requires them to remember the sentences of the text rather than the underlying meaning. The second class would need to understand the interplay of multiple structures as well as their functions within the system as a whole.

Now that we've laid out all of our assumptions underlying the instructional fit hypothesis, let's put it all together. First, we start with the question: What do we want our students to be able to do or know? These are the task demands. Once we know what they are, then we ask, which representation is best suited for our learning goal? Then we figure out which cognitive processes are most likely going to lead to the generation of that representation? Finally, we ask, which instructional activity will most efficiently give rise to those cognitive processes? If we sketch it out, the chain of events might look like something like this:

Figure 1. The hypothesized chain of events.

The Study and the Evidence

To test the instructional fit hypothesis, we asked high-school students to learn about an advanced topic in physics (electrodynamics) under a couple of different experimental conditions. The first condition nicely mapped all of the steps from Figure 1. The instructional activity inspired the cognitive processes that we believed would lead to a useful representation to solve electrodynamics problems (i.e., a problem-solving schema). For the other experimental condition, the fit wasn't as nice. The instructional activity prompted the students to build and modify a mental model. While useful to visualize the problem situation, a mental model does not specify how to arrive at a numeric solution.

We asked our participants to solve their electrodynamics problems with a computer tutor called the Andes Physics Tutor [2]. Students can ask Andes for a hint to help them when they get stuck. As one measure of how difficult it was for students to solve the problems under the two different conditions, we counted the number of hint requests. It turned out that students who were prompted to form a problem-solving schema asked for fewer hints than the students who received the mental-model instructional activity. This provided preliminary evidence in favor of the instructional fit hypothesis.

The STEM Connection

The implication  of the instructional fit hypothesis for STEM education is fairly straight forward. The bottom line is: try to align instructional activities to cultivate the mental representation(s) that will be most useful to your students as they work to achieve specific learning objectives. Fitting instructional activities to the task demands, however, can sometimes be a challenge. One way to accomplish this would be to begin designing a new lesson by conducting a rigorous task analysis. If you're fortunate enough to know someone who is already an expert in the target domain, consider asking her to talk through her process while she solves a problem similar to one you would like your students to be able to master. After she is done, go back and ask, "How did you know to take this step?" or "What knowledge did you rely on to figure this out?" The goal is to figure out which representations an expert in the area relies on to produce an efficient solution. 

Once you have a handle on the task demands and the representations an expert uses, the hard part is to figure out what instructional activities can most effectively inspire those types of representations. In our study, we relied on 20 years of research on self-explaining to come up with our activities. That literature was robust enough that we could theorize about a potential match or mismatch between the instructional activities and the representations that are needed.

Obviously, this is a time-consuming process. But if we can understand the chain of events a little better, then we will certainly be able to improve our instruction! 


Share and Enjoy!

Dr. Bob

For More Information

[1] Nokes, T. J., Hausmann, R. G., VanLehn, K., & Gershman, S. (2011). Testing the instructional fit hypothesis: the case of self-explanation prompts. Instructional Science, 39(5), 645-666.

[2] VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147–204.