Timed Repetitive Quizzes (TRQs)

Timed Repetitive Quizzing is an online activity that challenges students at the knowledge, comprehension and application levels of cognitive thought. In some ways, it is similar to B.F. Skinner's "Teaching Machine" developed in the 1950s and described in the 6-minute video below.

Skinner's Teaching Machine had the following features:

  1. The reward system centers around freeing the student from anxiety and making learning "pleasureable."
  2. Students work "at their own pace."
  3. Students follow a "program" via a large number of small steps.
  4. Questions came from all levels of cognitive thought.
  5. Skinner stated that students learn twice as fast.

A troubling assertion (at least to some in the teaching profession) was that a machine could eventually replace the teacher in the classroom. Authur C. Clark puts some perspective on this when he said "if a teacher can be replaced by a machine, they should be."
While this fear may have affected the proliferation of the original Teaching Machine, Skinner's invention and educational ideas continue to influence education as the following video demonstrates.

Elements Of An Effective Teaching Machine (or Teacher)

Before one constructs and implements a "teaching machine", it would be wise to distill this complex mixture called teaching and learning into its most important components and include them in the "machine". It seems trivial to state, but effective teaching results in learning.

Learning requires one of the following results

  1. new pieces of knowledge are introduced, and re-introduced, until a transient memory construct is created in long-term memory,
  2. transient memory constructs are fortified so their life-expectancy in long-term memory is increased, or
  3. new connections between memory constructs are formed.

Learning necessitates an alteration of long-term memory - if nothing has been added to long-term memory, then nothing has been learned. Historically, the construction and fortification of memory constructs has depended entirely on the learner's desire to acquire the knowledge presented by the teacher or contained in the textbook. This idea that students must have an intrinsic motivation to begin the learning process has stunted inquiry into effective external motivations. Consider the possibility that there might exist an external motivator that induces a more pleasurable force to learning than the intrinsic force where a student "wills herself" to study. In the former, students complete low-stake, small-step assignments to learn knowledge selected by the teacher and earn 100% of the course credit allocated to these assignments; in the latter, students must "guess" which knowledge is "important" and "will herself" to learn it so she can hopefully earn a high mark on a future assessment written by a teacher whose idea of "important" may differ from the students'.

  • One "force" that could be employed is to present the student with an academically small step as the "assignment." Students willing to do "some" work in a class, will be willing to take this very small step - especially if they receive 100% of the course credit allocated to the completion of this small step. These tiny assignments create a conflict at the core of human nature: "if I don't do this tiny assignment, I will earn a 0 and it will be obvious to everyone (i.e. my excuses will no longer work with classmates, teachers, parents and . . . . . . . . . me) that I don't want to do the most basic learning task that could be asked of me." This conflict causes more than 90% of students to complete 100% of the assigned "small steps." With this approach, students make progress in the right direction with every assignment and eventually learn the course material selected by the teacher.
  • A second "force" that could affect a student's motivation in taking this small academic step is TIME - if the step could be taken in one of three ways and one path was much quicker than the other two, then most students would be inclined to take the quicker route due to an intrinsic human desire to "save time." If this new teaching machine is constructed so that the quickest route is also the route that requires students to retrieve information from long-term memory (learning has occurred), students would be externally motivated to place knowledge in their long-term memory in order to save TIME. Of course, this "time-saving" component only becomes meaningful if this "small-step assignment" is reassigned 15 - 20 times (spaced-repetition) throughout the year. Then, the TIME saved by the student would increase each time the assignment is repeated . . . . and each time the assignment is repeated "new pieces of knowledge are planted, transient knowledge constructs are fortified, or new connections between memory constructs are formed."

We will revisit these "forces" in a few moments, but first let's investigate the essential components of an effective learning plan. A New York Times article entitled The Trouble With Homework describes several effective study techniques that have documented abilities to create long-term memories and aid in their retention and retrieval.

  1. Spaced Repetition - information is presented in small chunks at increasingly larger time intervals to create and maintain long-term memory constructs.
  2. Retrieval Practice - every time a memory is retrieved from long-term memory, that memory constuct become more solidified - it becomes stronger and more long-lasting. Retrieval practice utilizes a familiar tool - the test - in a novel way: to reinforce the things students know. Used in this way, testing doesn't just measure, it becomes an essential tool for learning. Other homework assignments like reading over material to be learned, taking notes, or making outlines does not have this effect. The result of combining spaced repetition with retrieval practice would be a learning approach that uses testing to create memory constructs and further testing (spaced repetition) to strengthen the newly formed memory constructs. While these first two components create a very effective learning plan, there's an additional, highly effective piece that "tricks" the brain to remember more information, and it is called interleaving.
  3. Interleaving - presenting learned material in different ways causes the brain to work harder. Interleaving can be as simple as studying information from Chapter 1, followed by Chapters 2 and Chapter 3, before returning to study Chapter 1, or it could involve mixing together Chemistry and Spanish flashcards. This learning approach creates interesting associations that result in an increase in retrieval efficiency the next time that information is pulled from long-term memory. This process is called "cognitive disfluency" and the extra mental demand signals the brain that the knowledge is worth keeping. Interleaving is much more powerful than one might expect. In a simple experiment described in the Journal of Applied Research in Memory and Cognition, math students whose practice problems were interleaved attained scores double those from students whose practice problems were grouped by type (massed or blocked presentation).

Question?

Would you assign an activity that uses spaced repetition, retrieval practice, and interleaving to create memory constructs in students that are stronger and formed more quickly than the typical student is able to accomplish on their own?

If the answer is yes, meet the TRQ Teaching Machine. The Timed Repetitive Quiz (TRQ) program contains the small-step, anxiety-free, pleasureable aspects found in Skinner's approach and offers the following improvements:

  1. Course credit is used as an external reward - students earn credit for effort. Nothing can be more discouraging than to spend 3 hours studying for a quiz, only to recieve a grade that does not reflect the effort expended. Most students "accept" the trade-off - complete many small "tests" to demonstrate a willingness to "work" and, in exchange, received a guaranteed percentage of the course grade. This is not a trivial trade-off - it too occurs at the core of human nature: humans do not like to be tested . . . . not even by themselves.
  2. TIME is also used as an external reward - humans want to spend the least amount of time completing a task. The TRQ approach rewards students with TIME when they complete the assignment by the shorter of three possible completion paths. Using TIME in this way results in students working faster than "their own pace." Students complete a TRQ SET in one of three ways:
    • Path 1 is "short" and reflects that the student has consistent, retrievable memory constructs in their long-term memory.
    • Path 2 is "slightly longer" and reflects memory constructs with high retrievability.
    • Path 3 is "long" and reflects that the student is still learning and the memory constructs are "under development."
    Successful completion of either Path results in the same number of points, which creates similar feelings of "success" and accomplishment in all learners. The "R" in TRQs signifies that a particular arrangement of knowledge will be Repeated daily in the beginning and then at successively longer intervals to maintain the information in the "learned state." From the teacher's perspective, this Repetitive component of the TRQ program is essential to the formation and solidification of memory constructs on which critical thinking rests; for the student, this Repetitive component creates pressure to earn points via Path 1 since this will save more and more TIME as the Repetitions increase (15 - 20 per year).
  3. The use of TIME, and a Timer, imposes the requirement that the content of a TRQ question must be located in the Knowledge, Comprehension and Application levels of cognitive thought so that the answer can be retrieved and delivered in a short period of time (e.g. < 10 seconds). If a primary goal of learning is to move novices (students) toward the "expert" level, then novices should be expected to acquire the skill to answer TRQ questions at "expert" or "near expert" times. Accordingly, TRQ questions are created so that they can be answered by an "expert" (you) in 3 - 10 seconds.

The TRQ program does not attempt to supplant the teacher, rather, it manages a high-stakes, effort-rewarding, learning activity (quizzes and immeditate feedback) that builds and maintains memory constructs in students. The teacher uses the information presented with TRQs to construct lessons that develop the critical thinking abilities of the students where each critical thinking excercise connects the TRQ knowledge in meaningful ways. With TRQs, the teacher and students spend class time exploring ideas and problems that require critical thinking.

TRQ Overview

TRQs are high-stakes, assignable learning activities that contain spaced repetition, retrieval practice, and interleaving. Historically, teachers have "suggested" these learning techniques, but students do assignments at an exceedingly higher rate than suggestions. To achieve an outcome where students possess a dense forest of course knowledge, TRQs must be administered as high-stakes (e.g. 20% of course grade) assignments to "motivate" students to acquire knowledge necessary to solve problems that require critical thought. The typical assignments (quizzes, homework, exams) primarily posed questions that assess students at the Application, Analysis or Synthesis levels of cognitive thought. Success on these traditional assignments require the student to make connections between memory constructs from long-term memory. When the memory constructs are missing or not well-formed the assignment quickly leads to frustration. For homework assignments, students alleviate their frustration by finding "help" as this appears to be the shortest (read "save TIME") path to the completion of the assignment. "Help" may be the instructor, a teaching assistant or another student AND "help" may be a thorough explanation (with follow-up questions to ensure learning has occurred) or just the answer - homework assignments cannot discriminate between the two.

A current approach to address the lack of basic knowledge involves adaptive learning. In this learning activity, if the student fails to correctly answer a question at a high level of cognitive thought, the program "adapts" the learning approach by drilling down to determine the piece(s) of knowledge that the student is lacking. Adaptive learning simply attempts to identify the missing piece and notify the student - the student must still work to place this information into their long-term memory. Then, the adaptive learning system gives the student a similar question and the process starts over again. In a very different approach, the TRQ approach is a "flipped" adaptive system. TRQs build a dense forest of knowledge first so that any question (specifically chosen by the instructor based on the information students have learned) at a higher level of cognitive thought can be answered by the student if they can associate pieces of information that are in their long-term memory. With current adaptive learning systems, students "learn" from their failures - if they have a lot to learn, they will have to endure a lot of failure. The TRQ system presents small learning steps that the student is guaranteed to complete if they are willing to work - success and progress are only a few minutes away.

Components:

Chem21Labs TRQs are an active learning assignment ensures student mastery at the lowest levels of cognitive thought. TRQs contain basic course information in a multiple-choice or fill-in-the-blank format that represents the "small" learning steps described by Skinner. Completion of a learning step occurs by one of three Paths:

  1. Path 1: the student answers a single quiz perfectly (i.e. 10/10 questions) and beats the Quiz Time - usually set to 5 seconds per question for multiple choice (i.e. 50 sec for a 10 question quiz) and 10 seconds per question for fill-in-the-blank.
  2. Path 2: the student answers "≥ x"/10 questions correctly on "y" TRQs (i.e. a student answers 8 or 9 out of 10 questions correctly on 3 quizzes and beats the Quiz Time . . . x = 8, y = 3, Quiz Time = 50 sec).
  3. Path 3: the student answers correctly a certain number of questions (i.e. 75) on multiple quizzes.

Completion of either path results in the awarding of the same number of points (the Quiz Points).

This trimodal approach enjoys widespread student-acceptance since proficient students (Path 1) are spared the "busy work" of corporal repetition while non-proficient students (Path 3) are building long-term memory constructs with each correct answer submitted. The "Repetitive" part of TRQs is the SPACED-ASSIGNMENT of the same information. To place facts into long-term memory, typical memorization protocols require 5 specific interactions with the material that is to be memorized in the first week, weekly interactions for 2 - 3 weeks and then bi-weekly / monthly interactions until the student is no longer in your class. A TRQ over information in a specific chapter is assigned repeatedly by the instructor to ensure that the Knowledge and Comprehension level of the class is relatively homogeneous creating a learning environment that can be enjoyed by every student. When Higher Order Thinking Skills (i.e. Homework assignments) are assigned after memory constructs have been created with the TRQs, most students will be able to operate at this higher level of cognitive thought and learning gains will be realized for a larger segment of the class.

Instructors can customize the TRQs by changing the following parameters:

  1. Question Pool - the specific questions that can appear on a Student Quiz are selected by the instructor from a Keyword Searchable list maintained by Chem21Labs. In a future version, users will be able to add their own Questions.
  2. Quiz Questions - the actual number of questions that appear on the quiz.
  3. Quiz Time - the time deemed to represent a desired level of retrieval efficiency. If the time is set too long, the student may be able to "look up" some answers and still submit a Perfect quiz - this defeats the purpose of Path 1.
  4. "x" questions correct in "y" quizzes - almost perfect quizzes count toward completion of a TRQ Set.
  5. Total Correct - total number of correct answers that must be submitted to receive the Quiz Points.
  6. Minimum Correct - the minimum number of correct answers needed on a quiz before this number becomes part of the Total Correct count.
  7. Quiz Points - the points awarded to the student for completing a TRQ Set.
  8. Number of "Sets" - the number of times a student must earn Quiz Points between the Start and Due Date - the Number of Sets times the Quiz Points equals the Total Points for the TRQ assignment.
  9. Minimum Waiting Period - the minimum time a student must wait between "Sets"

Path 1: a student submits a Perfect quiz within the Quiz Time.
Path 2: a student submits "y" quizzes where students have answered at least "x" answers correct within the Quiz Time.
Path 3: a student correctly answers the Total Correct number of questions over multiple quizzes where only quizzes where the correct answers exceed the Minimum Correct value are counted toward the Total Correct value.

Students receive the Quiz Points whenever they satisfy Path 1, 2 or 3 - this is 1 SET. Then, the student must wait the Minimum Waiting Period value before beginning the next SET. Students continue earning points on additional SETs until the Number Of Sets value has been reached.

Research:

Organic Chemistry

The American Chemical Society (ACS) Organic Exam has been given as the Spring Final for Organic Chemistry II at Lee University since 1991. Starting in 2005, TRQs were used in both semesters of Organic Chemistry. In the first fifteen years (1991 - 2005), the class average on the ACS Organic Exam was 41 percentile; in the next ten years (when TRQs were used), the class average was 59.4 percentile. In addition, only thirteen students in the first fifteen years scored above the 90th percentile on this exam. Since Spring 2006, forty-six students have scored above the 90th percentile.


Equally remarkable is the fact that in the past ten years, only three students that started Organic II in January failed to take the ACS Exam in May - in contrast, 15% of the students in the first fifteen years withdrew from the class and didn't take the ACS Final. This longitudinal study shows that the TRQ learning method significantly helps the under-prepared students (based on a very low attrition rates and higher end-of-course grades). Additionally, TRQs show a significantly benefit to the "prepared" students as evidenced by the increase in students scoring above the 90th percentile.

How is this possible?

It is entirely based on the learning component described by Skinner where students are asked to complete "small" steps - steps in which they will be successful. After a large series of very small steps, the student has traveled a vast academic distance. These Organic students were given a few "small" steps to complete on the first day of the first semester and on many other days throughout the academic year. Only the students who were not willing to "work" (i.e. take a step that there is no doubt they can complete) failed to complete the class.

Mathematics - Middle School

In 2011 - 2012, TRQs were used by 78 students in a 6th grade math class for 20% of the instructional time - the teacher condensed her previous classroom activities into the remaining 80%. On average, the students showed a 12.8 percentile increase in the state's end-of-class exam (TCAP). Additionally, 43% of the students arriving from the 5th grade were classified as Proficient / Advanced, the remaining 57% were Basic or Below Basic. After using the TRQs for their 6th grade year, 63% of the students performed at the Proficient / Advanced level on this exam - an increase of 19.7%. As expected, the improvement in TCAP was most noticeable for the Below Basic (+26.5 %-ile) and Basic (+16.5 %-ile) math students.

Also in 2011 - 2012 and again in 2012 - 2013 at OMS, the TRQs were used in a "Response To Intervention (RTI)" class comprised of only Basic Math Students as determined by their previous year's TCAP score in Math. In addition to their daily 90-minute Math class, students attended the RTI class for 45 minutes every day for 1 semester (there were ~ 140 students in RTI classes each semester). Of the 557 RTI students in 2011 - 2013, 197 students (35.4 %) moved up to the Proficient / Advanced level on the state's TCAP exam.

In addition, in 2011 - 2012, 24 students were exposed to TRQs as members of the 78 student 6th grade math class and as members of the "RTI" math class. Twelve (50%) of the students that experienced this "double exposure" to the TRQ method moved from the Basic level to the Proficient / Advanced levels.

Mongolian Language Study

In April 2014, 19 students from either an Organic Chemistry II class or an upper-level Psychology class (Learning and Cognition) were selected to participate in a research project where, on subsequent days, students were

  1. given 50 Mongolian words to study for 2 hours using their best learning method. Their ability to spell the Mongolian words when presented with the English equivalent was assessed at 2 hours, 24 hours and 1 week.
  2. given 50 different Mongolian to study for 2 hours (24 hours after the 1st 50 words) using the TRQ method. Their ability to spell the Mongolian words when presented with the English equivalent was assessed at 2 hours, 24 hours and 1 week.

The Mongolian language was chosen because none of the participants had any prior knowledge the language. Immediately after the two hours of study, the initial assessment showed that students using the TRQ were able to correctly answer 14.5% more words than the control group. On the 24-hour and 1-week assessments, the TRQ students correctly answered 16.0% and 44.0% more words than the control group.

Note, that using the TRQ method for 2 straight hours is not the prescribed route of implementation, but we were interested in a direct comparison of the students' best learning method to the TRQ method. The 1-week loss of 53.5% of words learned using the student's best method and 41.5% of words using the TRQ Method, leave little doubt that the repetitive implementation of the TRQ method is essential for moving more information into long-term memory than what was achieved in this study.

Spanish Language - Medical Missions Team

In May 2014, twenty-three university students participated in a Medical Missions trip in which International Health was a required class. One-third of the class grade was based on learning commonly used Spanish words / phrases along with Spanish medical words / phrases. Students had 26 different TRQs that contained ~ 40 questions each (more than 1000 TRQ questions in all) where they had to complete 6 sets on each of the 26 "decks" of information (156 assignments). The following results are based on a TRQ called Nouns 1 which contained 37 questions (Nouns 1 Words). Twenty-five questions appeared on each quiz - the student was shown the English word and they were required to spell correctly the corresponding Spanish word.

From the six sets of student results on the Nouns 1 TRQ, the students were divided into three groups: Novice, Intermediate, and Expert.

  • Novice Group (11 students): their average score on the first four quizzes was less than 10 correct answers.
  • Intermediate Group (8 students): their average score on the first four quizzes was between 10 and 20 correct answers.
  • Expert Group (4 students): their average score on the first four quizzes was between 20 and 25 correct answers.

It is interesting to note that although the three groups of students start at very different levels of knowledge, after two TRQ Sets the three groups are virtually indistinguishable (see the shaded rectangular area for each graph). The Novice Group averaged 18 quizzes to complete two TRQ Sets, the Intermediate Group averaged 13 quizzes, and the Expert Group averaged 10 quizzes. On the next four TRQ Sets, all three groups averaged 14 quizzes. Assuming the Quiz Points are 10, the Novice Group took 18 quizzes to earn 20 points, the Intermediate Group took 14 quizzes to earn 20 points, and the Expert Group took 10 quizzes to earn 20 points. The remaining 40 points on the Nouns 1 assignment were earned by all groups by only taking 14 additional quizzes spaced-out over four TRQ Sets and over several weeks.

In today's classroom, the teacher starts the learning process by introducing basic facts, worksheets, or collaborative groups and more than half of the students (the Novices) do not have the requisite knowledge to be successful participants in any of these activities. In the Novice Group, students "knew" 72% of the 37 Spanish nouns after completing the 1st TRQ Set and 92% of these nouns after the 2nd TRQ Set. Based on this study, one can reasonably expect higher levels of learning for a class when the students are required to complete at least two TRQ sets prior to beginning the teacher-initiated learning activities.

In this study, students had to complete 6 TRQ sets for each of the 26 assignments (Nouns 1 is just 1 of the 26 assignments) while waiting a minimum of 12 hours between sets. On average, students spent 24 hours (spread out over 3 weeks) taking TRQs and studying their incorrect answers. This "spaced retrieval" created and maintained memory constructs that persisted during (and after) the 3-week mission trip. To put this in perspective, the team traveling to Honduras spent 15 hours traveling from Cleveland, TN to Rio Viejo, Honduras and another 15 hours for the return trip. In less time than it takes to travel to, and from, Honduras, these students were able to learn over 1000 Spanish words / phrases that they used in restaurants, churches, medical clinics, and when playing with children.

If you are interested in using TRQs in your Introductory Chemistry, General Chemistry, GOB Chemistry, Nursing Chemistry, or Organic Chemistry course, please contact support@chem21labs.com.