Making Tests (More) Fun Through Hints Increases Student Uptake Of Self-Testing

LEARNING SCIENTISTS POSTS, FOR TEACHERS 

Editorial Note: I am extremely excited to share with you a cross-posting from a group of like-minded scientists. This post originally appeared on The Learning Scientists Blog. The Learning Scientists aim to make research on the science of learning more accessible. Take it away, Dr. Carolina Kuepper-Tetzel!

By Carolina Kuepper-Tetzel

There is plenty of research supporting retrieval practice as a learning strategy. If left to their own devices, students report using self-testing as a way to assess how much they know, but not as a learning strategy per se (1). However, self-quizzing is a valuable learning strategy and more effective than other strategies preferred by students like ‘rereading chapters and notes’. In a recent series of experiments Vaughn and Kornell (2) investigated factors of test characteristics that may be conducive in motivating students to use self-testing as a primary strategy. They started with the assumption that, in general, students enjoy being tested – as long as they feel that they can get the answer right. Thus, increasing the likelihood of getting the correct answer will sway students to choose self-testing over simple restudying.

Image from Pixabay

How did the authors investigate this idea? 

They set up an experiment and had participants study 60 unrelated cue-target word pairs (e.g., ‘town-scones’). After the first study round, participants were given the option to choose how they would like to continue studying each word pair. So, before each word pair was shown to them they could pick between four options, i.e., 0-letter target (‘town-______’), 2-letter target (‘town-s____s’), 4-letter target (town-sc__es), or 6-letter target (‘town-scones’). Target words were always presented on the right side of the word pair and always consisted of 6-letter words. Thus, the 6-letter target condition was essentially a restudy condition that did not involve retrieval practice. The other three conditions all involved retrieval practice of the target word to some extent – with less or more hints. The authors were interested to see which of the four options students would pick for each study trial. The graph below shows the result.

Figure from Vaughn and Kornell (1)

As you can see: Students picked the 4-letter option more often than any of the other options. Furthermore, students reported that the 4-letter option was “the most fun” with 71% agreement for that option compared to only 11% agreement for the 6-letter option. Interestingly, the 0-letter option received an agreement score of 0% (the 2-letter option a score of 17%). So, it seems to be the case that pure retrieval practice without hints rules self-testing out as a strategy that students would select. Consequently, providing some hints can make self-testing more enjoyable and make students select it more often when they are given the choice.

They followed up their first experiment with a second one where they only included the two extreme options for participants to choose from: 0-letter versus 6-letter option. They found that students went for the pure restudy (6-letter) option in approximately 80% of the trails versus 20% for the pure retrieval practice option (0-letter). This shows that a self-testing option that offers no hints is perceived as an unattractive option and in such a scenario, students will go for restudying as a strategy.

After establishing student preference for testing with hints, it is important to see if that kind of self-testing is actually beneficial for performance on a final test. In follow-up experiments, the authors showed that indeed all three retrieval practice options – independent of hint level (i.e., 0-, 2-, 4-letter) – increased performance on a final test given two minutes later; more than the pure restudy condition (6-letter). However, one important caveat seems to be to make sure that students actually engage in retrieval processes, i.e., actually retrieve information from memory. If, for instance, the word pairs allowed correct guessing of the target word given the cue word – instead of retrieval – providing more hints led to detrimental effects of performance.

Image from Pixabay

Where does this leave us? 

Motivating students to change their habits on what strategies they use is not an easy task. However, it seems to be worthwhile thinking about ways to restructure or redefine the learning environment or task characteristics in a way that changes student perception of it towards higher attractiveness and more fun. Providing hints strategically can be one way to achieve this. It increases the likelihood to get the answer right which presumably increases the positive emotions associated with it and leads students to adopt self-testing in the future. As with all implementation tips, it is crucial that the underlying cognitive processes for the benefit of self-testing is triggered in the student. For the case here: Students still need to engage in retrieval of the material and not just rely on guessing to get the answer right.

References:

(1) Morehead, K., Rhodes, M. G., & DeLozier, S. (2016). Instructor and student knowledge of study strategies. Memory, 24, 257–271.

(2) Vaughn, K. E., & Kornell, N. (2019). How to activate students’ natural desire to test themselves. Cognitive Research: Principles and implications, 4, 1-16.

Minding the Gap: Connecting teachers and students to learning science

Editorial Note. We are in for a real treat! We have a guest post by Josh Ling, the CEO and founder of Podsie. I'm a fan of Podsie because it is one of the rare ed-tech companies that takes learning science seriously and attempts to fix a rather significant problem in learning and teaching. Take it away, Josh!

Learning By Doing

 

Let’s start by doing a challenge. (Full disclosure: if you end up not finding this exercise challenging, it’s because I typically do this with middle-school students!)

Here we go:

1. Google a map of North Africa. Then, study the country that’s to the west of Egypt for 10 seconds. Next, study the country that’s south of Egypt for 10 seconds.

2. Close that tab!

3. Count to 20.

4. Now, test yourself. Are you able to recall the names of the countries to the west and to the south of Egypt?

Given the short duration of time that passed between you learning the names and then being quizzed, you were likely successful in getting them both right! However, what if I asked you to retrieve that same information again in 3 hours? How about tomorrow? Or in a week? Or in a month?

“Hoping to find some old forgotten words…” —Africa, Toto

At this point, you might have realized that we’re revisiting some topics that have previously been discussed on this blog: forgetting and how one might combat it.

In those posts, Dr. Bob explained:

“Forgetting is non-linear, meaning it decays quickly and eventually slows down.”

Visually, he also provided a forgetting curve that shows how fast we forget newly learned information:

The good news is that we can combat this forgetting through retrieval practice, where recalling that information from memory strengthens the stickiness of that information and slows the rate of forgetting. 

To take it one step further, those blog posts referenced another article from Duolingo’s blog that expanded on the optimal cadence for retrieval practice. At Duolingo, they utilize the spacing effect [1], and they use their vast amounts of data points to map out a model of when students should review certain vocabulary words based on their prior performance [2]: 

Research shows that the optimal time to retrieve information is right when you’re about to forget it [3]. Because each retrieval practice should increase the durability of that memory, the retrieval practice is spaced out with longer and longer lag times in between each session. Practically, this has the positive effect of increasing studying efficiency because at any given moment, you can focus only on the subset of content you’re most about to forget.

The Classroom Connection

I taught 8th grade math for two years from 2013 - 2015.

As a first year teacher, I was overly focused on just making it through the large amount of content that was mandated by our state curriculum. I gave little time or thought to review, and overall, my classroom looked a lot like the one that Dr. Bob described:

“The traditional method of teaching is to introduce a topic, solve a few illustrative problems that relate to that topic in class, assign some homework problems, and then give a test a few days or weeks later to see if the students retained the material. For highly important topics, the same items might make a reappearance on the final exam.”

Dr. Bob also describes the problem with this traditional approach:

“...if a topic hasn't been discussed in several weeks, then it is likely the memory system is going to treat that memory as unimportant, and it will find itself on the fast side of the forgetting curve. Second, if too much time elapses between the presentation and evaluation, then the probability of successful recall is going to be very low.”

I had never heard of the forgetting curve, but I saw it working in full force with my students.

In my second year, I resolved to provide more review opportunities. Overall, however, I was still completely ignorant to the basic principles of learning science. I didn’t know that retrieval was the most effective way to review. When I myself was a student, I was a serial crammer, so the spacing effect was basically foreign to me.

As a result, review in my classroom was often suboptimal. For example, I occasionally asked my students to just re-read their notes. I would sometimes put questions covering older topics on students’ homeworks or quizzes, but certainly not with enough consistency to make it stick. To make matters worse, it was a massive challenge to figure out which topics most needed review. Some of my students needed to review fractions at certain points of the school year, while others really needed to review solving integer operations. 

That year, my students performed better than the previous year, but continued to struggle to effectively retain information that they had learned. 

A few years later

A couple years later, I made a career change and became a software developer. Around that time, I read a book called Make it Stick by Roedinger, McDaniel, and Brown [4], and I was blown away.

This book was the first time that I learned about the basics of learning science. The book went through the nuances of how we learn and retain information, and it provided definitive recommendations on research-backed practices that educators should be using in their classrooms, like retrieval and spacing.

My mind immediately flashed back to my classroom, where these best practices could have made a substantial difference with how my students learned. I also realized that this issue went further than just my own classroom. By that point, I had sat through countless hours of teacher professional development, and I also had a master’s degree in education. However, not once did we cover those basic cognitive science principles on how students learn.

Now

Those experiences were the inspiration for Podsie, a nonprofit edtech I co-founded that’s focused on improving student learning and empowering teachers by making the science of learning more accessible.

At Podsie, we’ve built an online web app for teachers and students that makes best practices like spacing, retrieval, and interleaving easy to implement in the classroom. With Podsie, a teacher creates an assignment that assesses the content that students learned. When students complete a question on an assignment, the question goes into that student's personal deck, which essentially represents the entire body of knowledge that a student should know for that class.

Each student's personal deck is powered by a spacing algorithm that determines when the student should review a question again, similar to how Duolingo prompts students to review a vocab word when they are just about to forget it. Overall, this ensures that students have a personalized review experience that allows them to focus on concepts they most need to review.

All in all, we are incredibly excited to make it easier for teachers and students to utilize and learn more about learning science best practices. On the way, we’ve had the privilege of learning from and working with cognitive scientists like Dr. Bob who are on the same journey to ensure students and educators can be the best they can be.

We launched a beta trial of our app in August of 2020, and we’re preparing to launch in June of 2021. Our app is free for teachers and students, and if you’re interested, you can sign up on www.podsie.org to be notified as soon as we’re live!

Going Beyond the Information Given

[1] Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological bulletin, 132(3), 354.

[2] Figure 3 is taken from https://blog.duolingo.com/how-we-learn-how-you-learn/, which was originally published in their academic paper: 

Settles, B., & Meeder, B. (2016, August). A trainable spaced repetition model for language learning. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers) (pp. 1848-1858). 

[3] Landauer, T. K., & Bjork, R. A. (1978). Optimal rehearsal patterns and name learning. In M. M. Gruneberg, P. E. Morris, & R. N. Sykes (Eds.), Practical aspects of memory (pp. 625-632). London: Academic Press.

[4] Brown, P. C., Roediger III, H. L., & McDaniel, M. A. (2014). Make it stick. Harvard University Press.

Fight the Power!: Retrieval Practice

Learning By Doing

Let's start with a handful of questions. Without looking back at any of the previous posts, try to answer the following questions:

1. What are the three processes involved in memory? 
2. What is the shape of the forgetting curve? 
3. How many items can be held in working memory at the same time? 
4. What is the capacity of long-term memory? 
5. Are there memories that we never forget?

The answers can be found at at the end of this post [1]. 

Wait...what was I going to say? 

Do you remember sliding down the memory curve? If not, it's okay. It’s been a while. Forgetting is a normal (and adaptive!) part of memory. Forgetting is non-linear, meaning it decays quickly and eventually slows down. If you plot it on a graph, then it might look something like this (see Fig. 1). The y-axis is the probability of successfully recalling a memory, and the x-axis is the amount of time that has elapsed since the last time you tried to recall that same memory. 

Figure 1. An idealized forgetting graph.

Notice the shape of the graph. It resembles a power function. In fact, most mathematical models of forgetting follow a power function, P = at^−b , where P is the probability of accurately recalling an item, t represents time, and b is the forgetting rate [2]. 

In another past post, we tried to address the question of why the forgetting curve looks like this. John Anderson and his colleague Lael Schooler put forth the argument that memory is adapted to our informational environment. We forget because the environment does not demand that we remember. Put another way, memory, and therefore forgetting, is a reflection of the environment. That's an interesting argument because it means we can structure the environment in such a way that guards against forgetting.

Inoculating Against Forgetting

If Anderson and Schooler's argument is accurate, what can we do to improve our memory? Burr Settles, Research Director at Duolingo, has an excellent suggestion. In his blog post, he suggests that we treat forgetting by administering little booster shots over time [3]. If you remember a vocabulary word accurately, then the system waits a longer time span than if you forget. If you forget, then the system asks you to recall that word more frequently. It's pretty ingenious, and it's an excellent example of using technology to solve a tricky educational problem.

The concept behind the recommendation is called retrieval practice. In other words, you give your students an opportunity to retrieve a word, concept, or fact from long-term memory. Merely attempting to recall an item ends up helping to boost that item's strength in memory. The critical component is that you try. If you fail, however, then you are going to need feedback (i.e., you need to see the item you were trying to recall). Retrieval practice has been shown to be more effective than rereading or reviewing the same material [4].

It seems weird, but that's how memory works. By the fact that you are trying to recall something signals to the memory system that this item is important, and that I need to remember it for next time.

The S.T.E.M. Connection

How do we harness Dr. Settle's suggestion in a classroom environment, where specific items (such as words) are not being tracked by a computer for each individual student? Is there a way to help teachers administer those memory booster shots to their students? 💉

The traditional method of teaching is to introduce a topic, solve a few illustrative problems that relate to that topic in class, assign some homework problems, and then give a test a few days or weeks later to see if the students retained the material. For highly important topics, the same items might make a reappearance on the final exam. Wouldn't the unit test and final exam count as a booster? 

Depending on the time series, probably not. There are two potential problems. First, if a topic hasn't been discussed in several weeks, then it is likely the memory system is going to treat that memory as unimportant, and it will find itself on the fast side of the forgetting curve. Second, if too much time elapses between the presentation and evaluation, then the probability of successful recall is going to be very low.

There are a couple of ways to combat this situation. First, if you are an educator, and you are in complete control over the homework items assigned to your students, then you can "sneak" an old item into the current problem set. The problem, of course, is that if you do this too often, then your inoculation graph might look like this:

Figure 2. Spaced practice for multiple items with different decay rates.

As you can see, this can get really messy, really fast. One way to deal with that complexity is to schedule homework assignments where all of the problems are review items.

Second, if your domain has facts or skills that build on older ideas, then students will automatically receive practice on the foundational material. Math is a great example. Learning about ratios can help students understand slope, which then leads into solving linear equations. By exercising the more complex skills, such as solving linear equations, student receive practice on ratio reasoning.

I understand that implementing these suggestions is difficult because there are a lot of factors at play in the classroom, but I hope it is helpful to think about forgetting in terms of multiple, overlapping power functions. With that image in mind, we can keep giving doses of anti-forgetting shots [5].

Share and Enjoy!

Dr. Bob

Going Beyond the Information Given

[1] Answers are: 1) encoding, storage, and retrieval; 2) it's a decelerating power curve; 3) between five and nine items; 4) extremely large; 5) there is evidence that we have permanent memories for some items.

[2] Of course, there is some debate about that. Two of undergraduate professors argue that the empirically observable power law might be an artifact of averaging over multiple exponential functions. I know. Your mind is blown, right? Mine was too when I first heard their argument. All of the gory details can be found in: Anderson, R. B. & Tweney, R. D. (1997). Artifactual power curves in forgetting. Memory & Cognition, 25, 724–730.

[3] Burr Settles, B. (2016, December 14) How we learn how you learn. Retrieved from https://making.duolingo.com/how-we-learn-how-you-learn.

[4] Roediger III, H. L., & Butler, A. C. (2011). The critical role of retrieval practice in long-term retention. Trends in cognitive sciences, 15(1), 20-27.

[5] If you've been following this blog, you might notice that booster shots show up every so often. This post is at attempt to boost your memory of the forgetting curve and the environmental factors that influence memory!