Timed Repetitive Quizzes (TRQs)

Timed Repetitive Quizzing is an online activity that challenges students at the knowledge, comprehension and application levels of cognitive thought. In some ways, it is similar to B.F. Skinner's "Teaching Machine" developed in the 1950s and described in the 6-minute video below.

Skinner's Teaching Machine had the following features:

  1. The reward system centers on freeing the student from anxiety and making learning "pleasurable."
  2. Students work "at their own pace."
  3. Students follow a "program" via a large number of small steps.
  4. Questions came from all levels of cognitive thought.
  5. Skinner stated that students learn twice as fast.

A troubling assertion (at least to some in the teaching profession) was that a machine could eventually replace the teacher in the classroom. Authur C. Clark puts some perspective on this when he said "if a teacher can be replaced by a machine, they should be." While this fear may have affected the proliferation of the original Teaching Machine, Skinner's invention and educational ideas continue to influence education.

Elements Of An Effective Teaching Machine (or Teacher)

Before one constructs and implements a "teaching machine", it would be wise to distill this complex mixture called teaching and learning into its most important components and include them in the "machine". It seems trivial to state, but effective teaching results in learning.

Learning requires one of the following results

  1. new pieces of knowledge are introduced, and re-introduced, until a transient memory construct is created in long-term memory,
  2. transient memory constructs are fortified so their life-expectancy in long-term memory is increased, or
  3. new connections between memory constructs are formed.

Learning necessitates an alteration of long-term memory - if nothing has been added to long-term memory, then nothing has been learned. Historically, the construction and fortification of memory constructs has depended entirely on the learner's desire to acquire the knowledge presented by the teacher or contained in the textbook. This idea that students must have an intrinsic motivation to begin the learning process has stunted inquiry into effective external motivations. Consider the possibility that there might exist an external motivator that induces a more pleasurable force to learning than the intrinsic force where a student "wills herself" to study. In the former, students complete low-stake, small-step assignments to learn knowledge selected by the teacher and earn 100% of the credit allocated to these assignments; in the latter, students must "guess" which knowledge is "important" and "will herself" to learn it so she can hopefully earn a high mark on a future assessment written by a teacher whose idea of "important" may differ from the students'.

  • One "force" that could be employed is to present the student with an academically small step as the "assignment." Students willing to do "some" work in a class, will be willing to take this very small step - especially if they receive 100% of the credit allocated to the completion of this small step. These tiny assignments create a conflict at the core of human nature: "if I don't do this tiny assignment, I will earn a 0 and it will be obvious to everyone (i.e. my excuses will no longer work with classmates, teachers, parents and . . . . . . . . . me) that I don't want to do the most basic learning task that could be asked of me." This conflict causes more than 90% of students to complete 100% of the assigned "small steps." With this approach, students make progress in the right direction with every assignment and eventually learn the course material selected by the teacher.
  • A second "force" that could affect a student's motivation in taking this small academic step is TIME - if the step could be taken in one of three ways and one path was much quicker than the other two, then most students would be inclined to take the quicker route due to an intrinsic human desire to "save time." If this new teaching machine is constructed so that the quickest route is also the route that requires students to retrieve information from long-term memory (learning has occurred), students would be externally motivated to place knowledge in their long-term memory in order to save TIME. Of course, this "time-saving" component only becomes meaningful if this "small-step assignment" is reassigned 15 - 20 times (spaced-repetition) throughout the year. Then, the TIME saved by the student would increase each time the assignment is repeated . . . . and each time the assignment is repeated "new pieces of knowledge are planted, transient knowledge constructs are fortified, or new connections between memory constructs are formed."

We will revisit these "forces" in a few moments, but first let's investigate the essential components of an effective learning plan. A New York Times article entitled "The Trouble With Homework" (9/10/2001 by Annie Murphy Paul) describes several effective study techniques that have documented abilities to create long-term memories and aid in their retention and retrieval.

  1. Spaced Repetition - information is presented in small chunks at increasingly larger time intervals to create and maintain long-term memory constructs.
  2. Retrieval Practice - every time a memory is retrieved from long-term memory, that memory construct become more solidified - it becomes stronger and its lifespan is increased. Retrieval practice utilizes a familiar tool - the test - in a novel way: to reinforce the things students know. Used in this way, testing doesn't just measure, it becomes an essential tool for learning. Other homework assignments like reading over material to be learned, taking notes, or making outlines do not have this effect. The result of combining spaced repetition with retrieval practice creates a learning approach where "testing" initiates the building of memory constructs and additional "testing" strengthens these memory constructs. While these first two components create a very effective learning plan, there's an additional, highly effective piece that "tricks" the brain to remember more information, and it is called interleaving.
  3. Interleaving - presenting learned material in different ways causes the brain to work harder. Interleaving can be as simple as studying information from Chapter 1, followed by Chapters 2 and 3, before returning to study Chapter 1, or it could involve mixing together Chemistry and Spanish flashcards. This learning approach creates interesting associations that result in an increase in retrieval efficiency the next time that information is pulled from long-term memory. This process is called "cognitive disfluency" and the extra mental demand signals the brain that the knowledge is worth keeping. Interleaving is much more powerful than one might expect. In a simple experiment described in the Journal of Applied Research in Memory and Cognition, math students whose practice problems were interleaved attained scores double those from students whose practice problems were grouped by type (massed or blocked presentation).

Would you assign an activity that uses spaced repetition, retrieval practice, and interleaving to create memory constructs in students that are stronger and formed more quickly than the typical student is able to accomplish on their own?

If the answer is yes, meet the TRQ Teaching Machine. The Timed Repetitive Quiz (TRQ) program contains the small-step, anxiety-free, pleasurable aspects found in Skinner's approach with the following features :

  1. A Timer - delivery of this learning tool to an "outside of class" and "un-monitored" environment via the Internet requires the use of a timer to prevent students from "looking up the answer."  A Timer is necessary to produce a learning environment where the student is presented with a question and has just a few seconds to respond with an answer.  The use of a Timer has several consequences:
    • the student does not have time to "look up" the answer
    • the TRQ questions MUST be questions that can be answered quickly
  2. External Rewards
    • Course credit is used as an external reward - students earn credit for effort. Nothing can be more discouraging to a student than to spend 3 hours studying for a quiz, only to receive a grade that does not reflect the effort expended. With TRQs, students "accept" the work involved in completing TRQs because they will receive a guaranteed percentage of the course grade. This "contract" with the student to complete many small "tests" over the course of a semester or year is significant because it involves testing . . . . an activity that humans do not like . . . . but, an activity that changes a students' long-term memory.
    • TIME - since we have to use a TIMER, it is fortuitous that Time is also an external rewar.  Humans want to spend the least amount of time completing a task.  The TRQ approach rewards students with TIME when they complete the assignment by the shortest of three possible completion paths.  Using TIME in this way results in students working faster than "their natural pace." Students complete a TRQ SET in one of three ways:
      • Path 1 is "short" and reflects that the student has consistent, retrievable memory constructs in their long-term memory.
      • Path 2 is "slightly longer" and reflects memory constructs with high retrievability.
      • Path 3 is "long" and reflects that the student is still learning and the memory constructs are "under development."
      Successful completion of any Path results in the same number of points, which creates similar feelings of "success" and accomplishment in all learners.  The "R" in TRQs signifies that a particular group of knowledge will be Repeated daily in the beginning and then at successively longer intervals to maintain the information in the "learned state."  From the teacher's perspective, this Repetitive component of the TRQ program is essential to the formation and solidification of memory constructs on which critical thinking rests; for the student, this Repetitive component creates pressure to earn points via Path 1 since this will save more and more TIME as the Repetitions increase (15 - 20 per year).  The use of TIME, and a Timer, imposes the requirement that the content of a TRQ question must be located in the Knowledge, Comprehension and Application levels of cognitive thought so that the answer can be retrieved and delivered in a short period of time (e.g. < 10 seconds per question).  If a primary goal of learning is to move novices (students) toward the "expert" level, then novices should be expected to acquire the skill to answer TRQ questions at "expert" or "near expert" times.  Accordingly, TRQ questions are created so that they can be answered by an "expert" (you) in 5 - 10 seconds.

The TRQ program does not attempt to supplant the teacher, rather, it manages a low-stakes / high-reward, learning activity (quizzes and immediate feedback) that builds and maintains memory constructs in students.  The teacher uses the information presented with TRQs to construct lessons that develop the critical thinking abilities of the students where each critical thinking exercise connects the TRQ knowledge in meaningful ways.  With TRQs, the teacher and students spend less class time building knowledge and more time exploring ideas and problems that require critical thinking.

TRQ Overview

TRQs are  low-stakes / high reward , assignable learning activities that contain spaced repetition, retrieval practice, and interleaving. Historically, teachers have "suggested" a variety of proven learning techniques to build foundational knowledge, but these suggestions are rarely employed at the intensity and duration needed because students do assignments at an exceedingly higher rate than suggestions. To achieve an outcome where students possess a dense forest of course knowledge, TRQs should be administered as high-reward (e.g. 20% of course grade) assignments to "motivate" students to acquire knowledge necessary to solve problems that require critical thought. The typical assignments (quizzes, homework, exams) primarily posed questions that assess students at the Application, Analysis or Synthesis levels of cognitive thought. Success on these traditional assignments require the student to make connections between memory constructs from long-term memory. When the memory constructs are missing or not well-formed the assignment quickly leads to frustration. For homework assignments, students alleviate their frustration by finding "help" as this appears to be the shortest (read "save TIME") path to the completion of the assignment. "Help" may be the instructor, a teaching assistant or another student AND "help" may be a thorough explanation (with follow-up questions to ensure learning has occurred) or just the answer - homework assignments cannot discriminate between the two  but an Exam will .

A current approach to address the lack of basic knowledge involves adaptive learning. In this learning activity, if the student fails to correctly answer a question at a high level of cognitive thought, the program "adapts" the learning approach by drilling down to determine the piece(s) of knowledge that the student is lacking. Adaptive learning simply attempts to identify the missing piece(s) and notify the student - the student must still work to place this information into their long-term memory. Then, the adaptive learning system gives the student a similar question and the process starts over again. In a very different approach, the TRQ approach is a "flipped" adaptive system. TRQs build foundational knowledge first so that any question (specifically chosen by the instructor based on the information students have learned) at a higher level of cognitive thought can be answered by the student if they can associate pieces of information that are in their long-term memory. With current adaptive learning systems, students "learn" from their failures - if they have a lot to learn, they will have to endure a lot of failure. With TRQs,  students learn from their successes  - the system presents small learning steps that the student is guaranteed to complete if they are willing to work - success and progress are only a few minutes away.

Components:

Chem21Labs TRQs are an active learning assignment ensures student mastery at the lowest levels of cognitive thought. TRQs contain basic course information in a multiple-choice or fill-in-the-blank format that represents the "small" learning steps described by Skinner. Completion of a learning step occurs by one of three Paths:

  1. Path 1: a student submits a Perfect quiz within the Quiz Time.
  2. Path 2: a student submits "y" quizzes where students have answered at least "x" answers correct within the Quiz Time.
  3. Path 3: a student correctly answers the Total Correct number of questions over multiple quizzes where only quizzes where the correct answers exceed the Minimum Correct value are counted toward the Total Correct value.

Completion of any path results in the awarding of the same number of points (the Quiz Points).

This trimodal approach enjoys widespread student-acceptance since proficient students (Path 1) are spared the "busy work" of corporal repetition while non-proficient students (Path 3) are building long-term memory constructs with each correct answer submitted. The "Repetitive" part of TRQs is the SPACED-ASSIGNMENT of the same information. To place facts into long-term memory, typical memorization protocols require 5 specific interactions with the material that is to be memorized in the first week, weekly interactions for 2 - 3 weeks and then bi-weekly / monthly interactions until the student is no longer in your class. A TRQ over information in a specific chapter is assigned repeatedly by the instructor to ensure that the Knowledge and Comprehension level of the class is relatively homogeneous creating a learning environment that can be enjoyed by every student. When Higher Order Thinking Skills (i.e. Homework assignments) are assigned after memory constructs have been created with the TRQs, most students will be able to operate at this higher level of cognitive thought and learning gains will be realized for a larger segment of the class.

Set Initial TRQ Parameters

Hide Questions Check this box to hide the TRQ question pool from the student. Instructors use this to
  1. prevent students from spending too much time 'studying' the Question Pool before taking the TRQ
  2. prevent students from studying the Question Pool when the TRQ is being used as an in-class assignment where the student takes the quiz once and their percent correct becomes a grade.
Questions per Quiz This is the number of Questions (from the pool of questions) that the TRQ program will place on each TRQ. Previously-missed questions for a particular student are weighted to appear more often for that student . . . correct questions appear less often. Question Pools should consist of 50 - 100 questions and the number of Questions per Quiz should start at 10 and increase to 15 at Set 6, 20 at Set 10, and 25 at Set 15.

Option 1

Number of 100% Correct TRQs Per Set Option 1 - Expert-level ability. students submit 1 (or more) 100% correct TRQs within the Quiz Time and receive the Quiz Points (a completed TRQ Set). However, in most implementations, this value is set at 1 . . . . a student receives the Quiz Points when they submit a single 100% correct quiz within the 'Time To Beat' time. For the parameters selected, a student submitting a 100% correct quiz in less than 50 seconds receives 10 points.
Time To Beat This is the TRQ Timer. Since TRQ questions are designed to be answered quickly (think Jeopardy questions), the TRQ Time should be ≤6 × Questions per Quiz for Multiple Choice TRQs (recognition) and ≤10 × Questions per Quiz for Fill-in-the-Blank TRQs (recall). If longer Quiz Times are used, some students will keep a list of the questions they are missing and 'look up' the answers without learning the information.

Option 2

< 100% Correct in ≤ Quiz Time Option 2 - Proficient-level ability. A student answers x out of 'Questions per Quiz' on y quizzes before the Timer reaches zero and receives Quiz Points. For the parameters selected, if a student answers correctly 8 / 10 or 9 / 10 questions on 3 quizzes before the Timer reaches zero, they have completed a TRQ Set and receive 10 Points.
   / 10
 Times 

Option 3

Total Correct Option 3 - Novice-level ability. This is the total number of correct TRQ questions (obtained by repeatedly taking TRQs) necessary to complete a TRQ Set and earn the Quiz Points - the Quiz Time To Beat is not a factor for Option 3.
Minium Correct For Multiple Choice (MC) TRQs, this value is set at 40 - 50% of the 'Questions per Quiz' . . . for Fill in the Blank, this is usually set to 1. Before this parameter was introduced, it was observed that on MC TRQs some students would simple click Choice A as quickly as possible (10 questions in 6 seconds) and get 25 - 35% of the questions correct by chance . So, without learning the information, students were able to 'complete' the TRQ (and potentially acquire carpal tunnel syndrome) when their 'correct answers' eventually equaled the 'Total Correct' number. The 'Minimum Correct' parameter eliminates "successful guessing" by requiring students to get a minimum number of TRQ questions correct before the program will add this number to the 'Total Correct' number.

Set Final TRQ Parameters

Times Set Is Repeated This is number of times that a student must repeat this TRQ before the due date. For the parameters selected, a student would need to complete their first 'Set' (earn 10 Points), wait the 12 hours, complete their second 'Set' (earn 10 more Points), wait 12 hours, etc. etc. until they have completed 6 Sets and earned 6 × 10 Points (60 Points). Class learning curves generated from TRQ data show a steep, almost linear, portion for Sets 1 - 6 when assigned with a 1-week window and 12 hour 'Intermission Between Sets'. After the 6th Set, students average 97% correct answers on MC TRQs and 87% correct answers on FnB TRQs. To maintain these levels (and this information in long-term memory), this TRQ must be reassigned once a week for 2 weeks, twice a month for 2 months, and then once every other month. While this may seem like a lot of assignments, note that most students are completing Sets 7 - ∞ in 1 or 2 quizzes (1 - 4 minutes) - the students that aren't at the Expert or Proficient levels on these 'Jeopardy questions' must spend more time (more effort) to get their credit ('Points'). This learning tool identifies the students that need more 'time on task' while providing a 'short-cut' for proficient students. Also, because it takes so much time to learn new information AND to keep it in one's long-term memory, some lecture instructors allow 20 - 25% of the course grade to come from the TRQ assignments.
Intermission Between Sets This is the time a student must wait between 'Sets'. The Intermission time creates a 'Spaced-Repetition' for the Questions in the Pool. Typical values are 12 - 24 hours . . . . however, make sure the time window established by the start / due dates is greater than

2 × 'Times TRQ Is Repeated' × 'Intermission Between Sets'

This provides ample time for students to merge the TRQ assignment into their study schedule.
Points This is the TRQ points for each Set. For the parameters indicated, this TRQ must be completed 6 times (the student must satisfy Option 1, 2 or 3) by the due date. Since the student gets 10 Points for every completed set, this assignment is worth 60 points (6 × 10).

Total points for this assignment:

Instructors can customize the TRQs with the parameters shown to the left. Mouseover (or mousedown on touch devices) each orange link to see additional information in the tooltip.

For the parameters shown, when a student clicks the "Take Quiz" button, 10 questions will appear on the quiz. Students will have 50 seconds to qualify for Option 1 or 2. If the student gets 10/10 questions correct in less than 50 seconds (Option 1), they receive 10 points and are "finished for the day" (Set 1). If the student gets 8/10 or 9/10 questions correct in less than 50 seconds, they have satisfied ⅓ of Option 2. The student would need to retake the quiz (answer 10 new questions from the question pool) two more times and answer 8/10 or 9/10 questions correctly in less than 50 seconds to complete Option 2, receive 10 points and be "finished for the day" (Set 1). If a student correctly answers 5 - 9 questions OR 10 questions in more than 50 seconds, the number of correct answers will be added to a "totalCorrect variable" . . . . when this variable equals 75, the student receives 10 points and is "finished for the day" (Set 1).

After the student has completed Set 1, they must wait 12 hours (Intermission Between Sets) before starting SET 2. After completing SET 2, the student must wait 12 hours before starting SET 3. Students must complete a total of 6 Sets (Times Set Is Repeated) for this assignment. Students can earn 60 points on this assignment (6 SETS × 10 points / SET = 60 points)

Accessibility:

Although TRQs by design are a timed activity, the assignment time can be extended by the teacher if needed. In come cases the timing aspect is incompatible with a specific learning style and a more traditional assessment approach is warranted and will be implemented by the teacher.

For more information on the efficacy of TRQs in various subjects, click TRQ Research.
To view sample TRQ questions, click General Chemistry TRQ Questions.