Criss Cross: Aptitude by Treatment Interaction

Learning By Doing

 Let's play a fun game called Guess Which One. The answers are provided in the next section. No cheating! 

1. Guess which list of word-pairs has more accurate recall:
     A. A list provided by an experimenter.
     B. A list that you personally generated.

2. Guess which study method leads to deeper learning:
     A. Re-reading the material
     B. Testing yourself on the material you just read.

3. Guess which instructional method is better: 
     A. One-on-one human tutoring
     B. An intelligent tutoring system (i.e., a computer tutor)

4. Guess which study strategy is more effective: 
     A. Paraphrasing an expository text
     B. Self-explaining an expository text

5. Guess which type of text leads to a better understanding of the subject matter: 
     A. A minimally coherent text
     B. A globally coherent text

"Criss cross" –Owen, Throw Momma From the Train

If you've been reading this blog for a while now, you may have noticed that some answers have been discussed in previous posts.

1. The generation effect would predict that personally generated items are more memorable than those provided by someone else; therefore, the answer is A. 

2. The research on desirable difficulties predicts that students are better off quizzing themselves than re-reading the material. The best answer is A. 

3. This is a tough one. If you believe the early research on Intelligent Tutoring Systems, humans were the gold standard. But then Kurt VanLehn called that conclusion into question. The answer is A (but I'll accept B if you cite VanLehn, 2011). 

4. The research on self-explaining pretty clearly indicates that students learn more when they self-explain because they are using their background knowledge and reasoning to repair their flawed mental models. The answer is unequivocally B.

5. The answer is A or B. Wait, what? That's right! The answer to #5 is "it depends." This post is about the conditions upon which learning outcomes depend. Read on.

The Aptitude x Treatment Interaction

To better understand what's going on with the fifth question, let's take a step back and talk a little bit about research methodology. One of the most common experimental studies is to contrast the outcome of an experimental group with a control group. But instead of just comparing the outcomes of an experimental condition with a control condition, you have two levels of each independent variable. 

To make this more concrete, suppose you hypothesize that listening to music hurts learning performance. However, you don't think that all music hurts. Instead, you hypothesize that lyrically complex music hurts learning lists of words; whereas, instrumental music doesn't have any impact at all. 

To test your hypothesis, you design a study where there are two types of music and two types of lists to memorize. For the musical manipulation, you play a lyrically complex song versus techno music without any words. For the item manipulation, the first is a list of only words, and the second list only contains numbers. When you run this experiment, you plot the results with a line graph (see Figure 1). 

Figure 1: A cross-over interaction between music type and item type. 

Notice that the impact of music depends on the interaction between the type of music and the item type. If you listen to techno music, then there isn't any improvement or cost to recall. If you listen to lyrically complex music, then you get a little boost when memorizing lists of numbers. But if you listen to a song with lyrics, then it completely wipes out a participant's ability to memorize words. 

Said another way, there is a music-by-item interaction. When we talk about learning manipulations, we need to be sensitive to potential interactions between a student's aptitude and the learning situation they are in. Why? Because their learning outcomes might depend on it! 

Going back to our rousing game of Guess Which One, the answer to question #5 is "it depends" because students who have lots of background knowledge learn better from a minimally coherent text while students who do not have the same background knowledge learn better from a globally coherent text [1]. In other words, there is an aptitude (i.e., high vs. low background knowledge) by treatment (i.e., high vs. low textual coherence) interaction.

Students with a large amount of background knowledge are better served by minimally coherent texts because they must supply the missing information. They need to do more generative work while reading the text. As we have seen in other contexts, being generative during learning is beneficial for deep learning. The low-prior knowledge students, however, require a maximally coherent text because they lack the background knowledge to generate the connections. Therefore, they need more support and scaffolding when learning a new topic. 

The S.T.E.M. Connection

The above finding underscores how important both formative assessments and personalized learning environments are. In theory, if a teacher had enough data to diagnose how well each student understood a topic, then they could assign each student a different text. A knowledgeable student would get a minimally coherent text, while a low-knowledge student would get a maximally coherent text. 

Unfortunately, in practice, things are much more tricky. It would be a lot to ask a teacher to come up with two (or more) versions of a textbook. However, some labs are applying latent semantic analysis (LSA) to help match a given student to a particular version of a text [2]. The goal is to select a text that maximizes the reading comprehension for a particular student. This is an exciting area of research and one to keep an eye on as more textbooks are distributed digitally.

Someday, perhaps we can synthesize all of the (right) answers to Guess Which One and develop a learning platform that can handle the multitude of interactions between all of the variables that influence learning. That would be extremely powerful (and wouldn't require anyone to be thrown from a train!).


Share and Enjoy!

Dr. Bob

Going Beyond the Information Given

[1]  McNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996). Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 14(1), 1-43.

[2] Wolfe, M. B., Schreiner, M. E., Rehder, B., Laham, D., Foltz, P. W., Kintsch, W., & Landauer, T. K. (1998). Learning from text: Matching readers and texts by latent semantic analysis. Discourse Processes, 25(2-3), 309-336.

If the Shoe Fits: The Instructional Fit Hypothesis

Take a look at the map below. Suppose you live on Jackson Avenue, and you need to go to the store to buy a new pair of shoes. What route will you take? 

Once you've planned your route, how will you remember it? Will you:

• Form a mental map, position yourself on that map, and update your position as you travel
• Form a mental list of verbal directions (e.g., head toward Main Street; after Main, turn right on 1st; then, take a left on Madison; the shoe store will be on the right). 

In the unlikely event that you take a wrong turn, which strategy will be more helpful in getting you to your destination? Which representation is easier to store in working memory? Which one is easier to use while driving? 

Connecting Instruction to Learning

In this blog, we've discussed many different types of representations that the mind uses to store and organize information. We also have talked about various learning strategies that help us acquire new information. One aspect of the discussion that's been missing is how these pieces all fit together. Are there some learning strategies that are more likely to give rise to one type of representation over another?

To answer this question, my collaborators and I set out to test the hypothesis that certain forms of instruction inspire learning events that translate into specific types of representations. We called it the instructional fit hypothesis, in that instruction should match the type of learning that we want to elicit. Before we talk about how we tested the instructional fit hypothesis, let's review the assumptions on which it is based.

Our Assumptions

First, we assume there are different types of mental representations that we use to reason about the world and to solve problems. This assumption is supported by plenty of evidence that people construct and use many different types of representations. For the purposes of the present discussion, let's focus on two representations: mental models and problem-solving schemas. As we saw, a mental model is an image or dynamic simulation that allows the individual to make inferences based on that model. For example, we might not know how many windows there are in our house, but we can mentally walk from room to room and count them. A mental model can also be incomplete or incorrect, in which case I can add details to or correct my model as I encounter new relevant information. A problem-solving schema, in contrast, can be thought of as a recipe for solving a problem. In a previous post, we introduced the idea of a production rule, which is an if/then statement that says what to do when certain conditions are met. A problem-solving schema links multiple if/then statements so that a problem can be solved.

Second, we assume that certain types of representations are better suited to solve specific types of problems. It is more expedient to use a mental model of the circulatory system to diagnose a heart problem than it would be to use a problem-solving schema in which several production rules have to be tested to find one that matches the symptoms to the root cause. Likewise, it is easier to solve a multi-step math problem using a problem-solving schema than by constructing a mental model of that particular problem. To be effective and efficient, the representation and the problem-solving demands should match. 

Our third and final assumption is that certain types of instruction lead students to engage in specific types of cognitive processing. For example, suppose I instruct one of my classes to write a summary of a passage about the circulatory system. For my other class, I ask them to answer difficult questions like: Why would the distribution of oxygen be less efficient if there is a hole in the septum? The first class would concentrate on a surface-level understanding of the text because the task requires them to remember the sentences of the text rather than the underlying meaning. The second class would need to understand the interplay of multiple structures as well as their functions within the system as a whole.

Now that we've laid out all of our assumptions underlying the instructional fit hypothesis, let's put it all together. First, we start with the question: What do we want our students to be able to do or know? These are the task demands. Once we know what they are, then we ask, which representation is best suited for our learning goal? Then we figure out which cognitive processes are most likely going to lead to the generation of that representation? Finally, we ask, which instructional activity will most efficiently give rise to those cognitive processes? If we sketch it out, the chain of events might look like something like this:

Figure 1. The hypothesized chain of events.

The Study and the Evidence

To test the instructional fit hypothesis, we asked high-school students to learn about an advanced topic in physics (electrodynamics) under a couple of different experimental conditions. The first condition nicely mapped all of the steps from Figure 1. The instructional activity inspired the cognitive processes that we believed would lead to a useful representation to solve electrodynamics problems (i.e., a problem-solving schema). For the other experimental condition, the fit wasn't as nice. The instructional activity prompted the students to build and modify a mental model. While useful to visualize the problem situation, a mental model does not specify how to arrive at a numeric solution.

We asked our participants to solve their electrodynamics problems with a computer tutor called the Andes Physics Tutor [2]. Students can ask Andes for a hint to help them when they get stuck. As one measure of how difficult it was for students to solve the problems under the two different conditions, we counted the number of hint requests. It turned out that students who were prompted to form a problem-solving schema asked for fewer hints than the students who received the mental-model instructional activity. This provided preliminary evidence in favor of the instructional fit hypothesis.

The STEM Connection

The implication  of the instructional fit hypothesis for STEM education is fairly straight forward. The bottom line is: try to align instructional activities to cultivate the mental representation(s) that will be most useful to your students as they work to achieve specific learning objectives. Fitting instructional activities to the task demands, however, can sometimes be a challenge. One way to accomplish this would be to begin designing a new lesson by conducting a rigorous task analysis. If you're fortunate enough to know someone who is already an expert in the target domain, consider asking her to talk through her process while she solves a problem similar to one you would like your students to be able to master. After she is done, go back and ask, "How did you know to take this step?" or "What knowledge did you rely on to figure this out?" The goal is to figure out which representations an expert in the area relies on to produce an efficient solution. 

Once you have a handle on the task demands and the representations an expert uses, the hard part is to figure out what instructional activities can most effectively inspire those types of representations. In our study, we relied on 20 years of research on self-explaining to come up with our activities. That literature was robust enough that we could theorize about a potential match or mismatch between the instructional activities and the representations that are needed.

Obviously, this is a time-consuming process. But if we can understand the chain of events a little better, then we will certainly be able to improve our instruction! 


Share and Enjoy!

Dr. Bob

For More Information

[1] Nokes, T. J., Hausmann, R. G., VanLehn, K., & Gershman, S. (2011). Testing the instructional fit hypothesis: the case of self-explanation prompts. Instructional Science, 39(5), 645-666.

[2] VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147–204.

You Got Some Explaining To Do!: Self-explaining

Here's a fun question for you: How detailed is your mental model of a toilet? If I had to guess, I would say you generally know how a toilet works, but haven't written a thesis on the topic. To test yourself, read the following passage and try to assemble a mental model of the modern flushable toilet.

The toilet has several working components. The tank (or cistern) holds clean water, and it is located above the toilet bowl, which holds the flushable content. Connected to the tank is a handle that, when pressed, releases the water from the tank. Gravity drains the water into the bowl through a small cistern tube. The water then exits the bowl at the bottom through the S-trap. 

Inside the tank are several additional components that control the flow of water into and out of the tank. At the bottom of the tank is a flapper valve, which is connected to the handle by a chain. When the handle is pressed, the chain raises the flapper valve to let the water out. As the water level decreases, the float ball, which is connected to the float rod, drops and opens the inlet valve. When the inlet valve opens, clean water rushes in to fill the tank. When the water reaches the top of the tank, the float ball closes the inlet valve. 

There's a lot of information crammed into those paragraphs [1]; however, it is also the case that a lot of information is missing. For example, how does the float ball and rod open and close the inlet valve? To make sense of the description as currently written, you still have to do some work. First, you have to spatially assemble all of the components in your mind's eye as you learn about them. Second, you need to supply any information that may be missing. Finally, you might have to resolve contradictions that arise between the information you are learning and the mental model you are building as you engage in the first two activities [2].

What do good students do when learning something new? 

When students read a text about an unfamiliar topic, their first priority is to make sense of each and every word. Once they decode the words, their next task is to assemble them into larger chunks of meaningful information. For example, you probably know what the words handle, chain, and tube mean. But you need to connect all three concepts together to form a working model of the flush mechanism in a toilet. The mental process of hooking these concepts together is what we will call self-explaining, or the generation of inferences that are required to construct an accurate mental model.


Evidence for self-explaining was first observed by asking students to read a passage about a branch of physics called statics [3]. In statics problems, balanced forces are acting on a body, which means there is no acceleration (see Fig. 1 for an example). Statics problems can become quite complex, as you can probably imagine. Researchers noticed that students who best understood statics were the ones who explained to themselves how to solve these types of problems. In other words, the best students naturally engaged in self-explaining. They also generated several self-explanation inferences to fill in the missing pieces of information that are not stated explicitly in the text. These missing pieces are necessary to fully understand the problem's solution.

Fig 1. An example statics problem.

Can you motivate students to self-explain? 

It's really interesting to read transcripts of students as they attempt to make sense of a complex topic. They might not understand it at first. But as they explain it to themselves, they are then able to see how to correct their current understanding with the model presented in the text. Good students naturally self-explain, but can we get all students to engage in self-explaining? The answer, as it turns out, is a qualified "yes." 

In one of the first studies to demonstrate that self-explaining can be prompted, researchers asked eighth grade students to read a passage about the human circulatory system [4]. Before students read the passage, the researchers asked them several questions about their understanding of the circulatory system. Because they were so young, it was no surprise that students generally had an impoverished understanding of the circulatory system. Most knew that the heart beats, but they didn't know why or what purpose the heart served. Some of the better students knew that the heart circulates blood, but they didn't understand pulmonary circulation. That is, they didn't realize that the heart has a special circuit of blood that goes to the lungs for oxygenation (and back).

After the pretest phase of the experiment, the students were asked to read a text passage that described, in detail, all of the steps of circulation. The student was verbally prompted by the experimenter to self-explain after reading each sentence of the text. After a while, merely turning the page was a prompt to self-explain. In other words, the researchers trained the students to self-explain, and then they reminded them over and over to self-explain. It was a pretty heavy-handed approach, but it worked. It paid off because the students who were prompted to self-explain demonstrated a more robust mental model of the circulatory system on the post-test than students in control condition who were only prompted to think about or reread the sentences.

Why was prompting students to self-explain a qualified success? Like most educational interventions, there were individual differences in the amount and effectiveness of the prompting. Some students generated several self-explanation inferences, whereas others generated only a few. Since the original study, researchers have designed new ways to help students become better self-explainers. For example, one study gave students an incorrect solution and asked students to debug the fictitious student's reasoning process [5]. This helped take the pressure off the student so that they could focus their attention on reconciling the differences between the correct and incorrect solution paths.

The STEM Connection

The connection between self-explanation and STEM education is extremely straightforward. Students should be trained on the definition of what self-explaining is. Then they should be prompted, periodically, to engage in self-explaining. To help them get started, here is a list of prompts that were used in the original study [4]: 

1. What new information does each line provide for you?
2. How does it relate to what you’ve already read?
3. Does it give you a new insight into your understanding of how ______ works?
4. Does it raise a question in your mind?

Also, students should be shown evidence of the utility of self-explaining. Students will probably be more likely to self-explain if they understand what it will eventually buy them. Moreover, students should be given an opportunity to develop the skill [6]. In fact, there is an intelligent tutoring system that was developed to do precisely that. The iSTART system gives students the opportunity to self-explain while reading complex textual passages [7]. Because self-explaining can be mentally taxing, the iSTART system was even outfitted with a bunch of mini-games to help build fluency in self-explaining [8].

Self-explaining is a powerful learning mechanism. It can be time-consuming and mentally exhausting, but the end result is definitely worth the effort. Like other concepts covered in this blog, self-explaining is a skill, which means it can become automatized over time. Thus, we need to get students started with self-explaining as early as possible because when they get to college, they will thank us for giving them the tools to make sense of tough concepts like organic chemistry (and flushable toilets). 


Share and Enjoy!

Dr. Bob

For More Information

[1] Here is the resource I consulted to write this description. To further enhance reading comprehension, it's typically to include a diagram. I intentionally omitted the diagram because I wanted you to attempt to assemble a mental model without a visual aid. 

[2] Chi, M. T. H. (2000). Self-explaining expository texts: The dual processes of generating inferences and repairing mental models. In R. Glaser (Ed.), Advances in Instructional Psychology, Hillsdale, NJ: Lawrence Erlbaum Associates. 161-238.

[3] Chi, M. T. H., Bassok, M., Lewis, M., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13: 145-182.

[4] Chi, M. T. H., de Leeuw, N., Chiu, M. H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18: 439-477.

[5] Booth, J. L., Lange, K. E., Koedinger, K. R., & Newton, K. J. (2013). Using example problems to improve student learning in algebra: Differentiating between correct and incorrect examples. Learning and Instruction, 25, 24-34.

[6] Hausmann, R.G.M., & Chi, M.T. H. (2002). Can a computer interface support self-explaining? Cognitive Technology, 7(1), 4-14.

[7] McNamara, D. S., Levinstein, I. B., & Boonthum, C. (2004). iSTART: Interactive strategy training for active reading and thinking. Behavior Research Methods, Instruments, & Computers, 36(2), 222-233.

[8] Jackson, G. T., & McNamara, D. S. (2013). Motivation and performance in a game-based intelligent tutoring system. Journal of Educational Psychology, 105(4), 1036.