Criss Cross: Aptitude by Treatment Interaction

Learning By Doing

 Let's play a fun game called Guess Which One. The answers are provided in the next section. No cheating! 

1. Guess which list of word-pairs has more accurate recall:
     A. A list provided by an experimenter.
     B. A list that you personally generated.

2. Guess which study method leads to deeper learning:
     A. Re-reading the material
     B. Testing yourself on the material you just read.

3. Guess which instructional method is better: 
     A. One-on-one human tutoring
     B. An intelligent tutoring system (i.e., a computer tutor)

4. Guess which study strategy is more effective: 
     A. Paraphrasing an expository text
     B. Self-explaining an expository text

5. Guess which type of text leads to a better understanding of the subject matter: 
     A. A minimally coherent text
     B. A globally coherent text

"Criss cross" –Owen, Throw Momma From the Train

If you've been reading this blog for a while now, you may have noticed that some answers have been discussed in previous posts.

1. The generation effect would predict that personally generated items are more memorable than those provided by someone else; therefore, the answer is A. 

2. The research on desirable difficulties predicts that students are better off quizzing themselves than re-reading the material. The best answer is A. 

3. This is a tough one. If you believe the early research on Intelligent Tutoring Systems, humans were the gold standard. But then Kurt VanLehn called that conclusion into question. The answer is A (but I'll accept B if you cite VanLehn, 2011). 

4. The research on self-explaining pretty clearly indicates that students learn more when they self-explain because they are using their background knowledge and reasoning to repair their flawed mental models. The answer is unequivocally B.

5. The answer is A or B. Wait, what? That's right! The answer to #5 is "it depends." This post is about the conditions upon which learning outcomes depend. Read on.

The Aptitude x Treatment Interaction

To better understand what's going on with the fifth question, let's take a step back and talk a little bit about research methodology. One of the most common experimental studies is to contrast the outcome of an experimental group with a control group. But instead of just comparing the outcomes of an experimental condition with a control condition, you have two levels of each independent variable. 

To make this more concrete, suppose you hypothesize that listening to music hurts learning performance. However, you don't think that all music hurts. Instead, you hypothesize that lyrically complex music hurts learning lists of words; whereas, instrumental music doesn't have any impact at all. 

To test your hypothesis, you design a study where there are two types of music and two types of lists to memorize. For the musical manipulation, you play a lyrically complex song versus techno music without any words. For the item manipulation, the first is a list of only words, and the second list only contains numbers. When you run this experiment, you plot the results with a line graph (see Figure 1). 

Figure 1: A cross-over interaction between music type and item type. 

Notice that the impact of music depends on the interaction between the type of music and the item type. If you listen to techno music, then there isn't any improvement or cost to recall. If you listen to lyrically complex music, then you get a little boost when memorizing lists of numbers. But if you listen to a song with lyrics, then it completely wipes out a participant's ability to memorize words. 

Said another way, there is a music-by-item interaction. When we talk about learning manipulations, we need to be sensitive to potential interactions between a student's aptitude and the learning situation they are in. Why? Because their learning outcomes might depend on it! 

Going back to our rousing game of Guess Which One, the answer to question #5 is "it depends" because students who have lots of background knowledge learn better from a minimally coherent text while students who do not have the same background knowledge learn better from a globally coherent text [1]. In other words, there is an aptitude (i.e., high vs. low background knowledge) by treatment (i.e., high vs. low textual coherence) interaction.

Students with a large amount of background knowledge are better served by minimally coherent texts because they must supply the missing information. They need to do more generative work while reading the text. As we have seen in other contexts, being generative during learning is beneficial for deep learning. The low-prior knowledge students, however, require a maximally coherent text because they lack the background knowledge to generate the connections. Therefore, they need more support and scaffolding when learning a new topic. 

The S.T.E.M. Connection

The above finding underscores how important both formative assessments and personalized learning environments are. In theory, if a teacher had enough data to diagnose how well each student understood a topic, then they could assign each student a different text. A knowledgeable student would get a minimally coherent text, while a low-knowledge student would get a maximally coherent text. 

Unfortunately, in practice, things are much more tricky. It would be a lot to ask a teacher to come up with two (or more) versions of a textbook. However, some labs are applying latent semantic analysis (LSA) to help match a given student to a particular version of a text [2]. The goal is to select a text that maximizes the reading comprehension for a particular student. This is an exciting area of research and one to keep an eye on as more textbooks are distributed digitally.

Someday, perhaps we can synthesize all of the (right) answers to Guess Which One and develop a learning platform that can handle the multitude of interactions between all of the variables that influence learning. That would be extremely powerful (and wouldn't require anyone to be thrown from a train!).


Share and Enjoy!

Dr. Bob

Going Beyond the Information Given

[1]  McNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996). Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 14(1), 1-43.

[2] Wolfe, M. B., Schreiner, M. E., Rehder, B., Laham, D., Foltz, P. W., Kintsch, W., & Landauer, T. K. (1998). Learning from text: Matching readers and texts by latent semantic analysis. Discourse Processes, 25(2-3), 309-336.