Timed / Repetitive Quizzes (TRQs)

Skinner's Teaching Machine:

Timed / Repetitive Quizzing is an online Flashcard program that delivers the information via a "Teaching Machine" similar in some respects to the one developed by B.F. Skinner in the 1950s and described in the short video below.

Skinner's Teaching Machine had the following features:

  1. The reward system centered around freeing the student from anxiety and making learning "pleasureable."
  2. Students work "at their own pace."
  3. Students follow a "program" via a large number of small steps.
  4. Questions came from all levels of cognitive thought.
  5. It was touted that students could learn twice the volume of information in a given amount of time.

A troubling prospect (at least to the teaching profession) of Skinner's Teaching Machine is that it would replace the teacher in the classroom. Authur C. Clark puts some perspective on this when he said "if a teacher can be replaced by a machine, they should be." However, the lack of success of Skinner's Teaching Machine is due in large part to the inability to proliferate its use in classrooms controlled by the profession that it could potentially replace.


Elements Of An Effective Teaching Machine (or Teacher)

Before a teaching machine is constructed, it would be wise to attempt to distill this complex mixture called teaching in hopes of separating and identifying its most important components. It seems trivial to state, but effective teaching results in learning. Either new pieces of information were planted, transient pieces of information were strengthened, or new connections between memory constructs were formed. Learning necessitates an alteration of long-term memory - if nothing has been added to long-term memory, then nothing has been learned. Historically, the planting, and solidifying, of memory constructs has depended entirely on the desire of the learner to acquire the knowledge presented to them by the teacher. The idea that students must have an intrinsic motivation to begin the learning process has stunted inquiry into possible alternative motivations. Consider the possibility that there might exist an external motivation for learning which "forces" students to learn basic information chosen by the teacher. One "force" that could be employed is to present the student with an academically small step as the "assignment." If a student is willing to do some work in a class, they should be willing to take this very small step and receive some credit for completing the step. A second "force" that could affect a student's motivation in taking this small academic step is TIME - if the step could be taken in one of two ways and one way was much quicker than the other, then most students would be inclined to take the quicker route. However, if the quickest route was also the route that required students to retrieve information from long-term memory, students would be externally motivated to place items in their long-term memory in order to save TIME. If the teacher assigned a single small step to be repeated ten to fifteen times throughout the year, the TIME saved by the student would increase with each assigned repeat.

We will revisit these "forces" in a few moments, but first let's investigate the essential components of an effective learning plan. A New York Times article entitled The Trouble With Homework describes several effective study techniques that have documented abilities to create long-term memories and aid in their retention and retrieval.

  1. Spaced Repetition - information is presented in small chunks at increasingly larger time intervals to create and maintain long-term memory constructs.
  2. Retrieval Practice - every time a memory is retrieved from long-term memory, that memory constuct become more solidified - it becomes stronger and more long-lasting. Retrieval practice utilizes a familiar tool - the test - in a novel way: to reinforce the things students know. Used in this way, testing doesn't just measure, it becomes an essential tool for learning. Other homework assignments like reading over material to be learned, taking notes, or making outlines does not have this effect. The result of combining spaced repetition with retrieval practice would be a learning approach that uses testing to create memory constructs and further testing (spaced repetition) to strengthen the newly formed memory constructs. While these first two components create a very effective learning plan, there's an additional, highly effective piece that "tricks" the brain to remember more information, and it is called interleaving.
  3. Interleaving - presenting learned material in different ways causes the brain to work harder. Interleaving can be as simple as studying information from Chapter 1, followed by Chapters 2 and Chapter 3, before returning to study Chapter 1, or it could involve mixing together Chemistry and Spanish flashcards. This learning approach creates interesting associations that result in an increase in retrieval efficiency the next time that information is pulled from long-term memory. This process is called "cognitive disfluency" and the extra mental demand signals the brain that the knowledge is worth keeping. Interleaving is much more powerful than one might expect. In a simple experiment described in the Journal of Applied Research in Memory and Cognition, math students whose practice problems were interleaved attained scores double those from students whose practice problems were grouped by type (massed or blocked presentation).


The Question???

Would you use a Teaching Machine that uses spaced repetition, retrieval practice, and interleaving to create memory constructs in students that are stronger and formed more quickly than the typical student is able to accomplish on their own?

If the answer is yes, meet the TRQ Teaching Machine.

The TRQ Teaching Machine improves upon Skinner's in the following ways:

  1. While the reward system still contains the anxiety-free, pleasureable aspects contained in Skinner's approach, the primary reward centers around TIME and particularly the learner's desire to spend the least amount of time possible completing an assignment. Students are "rewarded" (with TIME) when they complete the assignment by the shorter of the two completion paths available.
  2. Using TIME in this way results in students working faster than "their own pace." Students can complete a segment of the TRQ assignment in one of two ways: Path 1 is "short", but reflects that the student has a retrievable memory construct in their long-term memory. Path 2 is "long", and reflects that the student is still learning the tested knowledge. Successful completion of either Path results in the same number of points, which creates similar feelings of "success." The "R" in TRQs signifies that a particular Flashcard deck will be Repeated daily at the beginning and then at successively longer intervals to maintain the information in the "learned state." From the teacher's perspective, this Repetitive component of the TRQ program is essential to the formation, and solidification, of memory constructs on which critical thinking rests; for the student, this Repetitive component creates pressure to earn her points via Path 1 since this will save a significant amount of TIME as the Repetitions increase to create stable, long-lasting memory constructs.
  3. Students follow the "TRQ program" via a large number of small steps. Small daily progress results in large academic gains over the course of a semester or year.
  4. TRQ questions do not come from the higher levels of cognitive thought. The use of TIME, and a Timer, requires TRQ questions to be located in the Knowledge, Comprehension and Application levels of cognitive thought so that the answer can be given in less than 10 seconds. While this may seem to limit the usefulness of TRQs, it should be noted that TRQs create assignments at a level of learning that has traditionally been "owned" by the student. Attempts by Instructors to "assess" the student's efforts to read the text and amass a defined set of knowledge resulted in instruments like the "daily quiz" - extra grading and little overall academic improvement. TRQs provide formative assessments to the student in the form of small, but significant steps. At some point in this formative assessment, one of the student's steps will cause them to complete Path 1 or Path 2 and they will receive their summative assessment score - full credit for that segment of the "journey."
  5. Students can learn large amounts of information in short periods of time, but research to quantify the effect of TRQs has not been attempted.

The TRQ Teaching Machine does not attempt to supplant the teacher, rather, the machine provides personalized instruction and feedback at the lowest levels of cognitive thought to the student. The teacher, now knowing what information each student in the classroom has learned from the machine, must construct lessons to develop the critical thinking abilities of the students where each critical thinking excercise connects the TRQ knowledge in meaningful ways. With TRQs, the teacher is spending more time teaching and students are spending more time critically thinking.


Video Introduction Of The TRQ Teaching Machine:

• Student (click here)


• Instructor(click here)


TRQ is an assignable learning tool that contains spaced repetition, retrieval practice, and interleaving. Historically, teachers have presented these techniques to students as "suggstions", but students, like most humans, do assignments at an exceedingly higher rate than suggestions. Before TRQs, there were no academic assignments that helped students learn the basic information necessary to be successful at higher levels of cognitive thought. Available academic assignments (quizzes, homework, exams) primarily posed questions that assessed a student's ability at the Application, Analysis or Synthesis levels of cognitive thought. Because success on these assignments required the student to connect memory constructs from long-term memory, this "assignment-oriented" approach quickly leads to frustration since the student does not possess the necessary memory constructs to work independently on the assignment. Students alleviate their frustration by finding "help" as this appears to be the shortest path to the completion of the assignment .  "Help" may be the instructor, a teaching assistant or another student AND "help" may be a thorough explanation or just the answer - homework assignments cannot discriminate between the two. Sometimes "Help" involved reading the paragraph or section in the text book that seems to pertain to the asked question - unfortunately typical critical thinking problems seek to associate two or more ideas presented on different pages / sections in a text book.

A current approach to address the lack of basic knowledge involves adaptive learning. In this approach, the student's first interaction is still at the higher levels of cognitive thought, but when the student answers a question incorrectly the program "adapts" the learning by drilling down to determine the piece(s) of basic information that the student is lacking. Adaptive learning simply attempts to identify the missing piece and notify the student - the student must still work to place this information into their long-term memory. Then, the adaptive learning system gives the student a similar question and the process starts over again. In stark contrast, the TRQ approach is a "flipped" adaptive system. TRQs build a dense forest of information first so that any question (specifically chosen by the instructor based on the information students have learned) at a higher level of cognitive thought can be answered by a student if they can associate pieces of information that are in their long-term memory. With current adaptive learning systems, students "learn" from their failures - if they have a lot to learn, they will have to endure a lot of failure. The TRQ system presents small academic steps that the student is guaranteed to complete if they are willing to work - success (and progress) is only a few minutes away.



Chem21 has developed (and patented) a fun, interactive and engaging approach that requires student mastery at the lowest levels of cognitive thought. Timed / Repetitive Quizzes (TRQs) contain basic course information in a multiple-choice or fill-in-the-blank format that represents the "small" learning steps described by Skinner. Completion of a learning step occurs by one of two Paths:

  1. Path 1: the student answers a single quiz Perfectly (10/10 or 15/15 or 30/30, etc.) and beats the Quiz Time (usually set to 5 seconds per question for multiple choice and 10 seconds per question for fill-in-the-blank).
  2. Path 2: the student answers correctly a certain number of questions (i.e. 100).

Completion of either path results in the awarding of the same number of points (the Quiz Points).

This bimodal approach enjoys widespread student-acceptance since proficient students are spared the "busy work" of corporal repetition while non-proficient students are building long-term memory constructs with each correct answer submitted. The "Repetitive" part of TRQs is the SPACED-ASSIGNMENT of the same information. To place facts into long-term memory, typical memorization protocols require 5 specific interactions with the material that is to be memorized in the first week, weekly interactions for 2 - 3 weeks and then bi-weekly / monthly interactions until the student is no longer in your class. A TRQ over information in a specific chapter is assigned repeatedly by the instructor to ensure that the Knowledge and Comprehension level of the class is relatively homogeneous creating a learning environment that can be enjoyed by every student. When Higher Order Thinking Skills (i.e. Homework assignments) are assigned after memory constructs have been created with the TRQs, most students will be able to operate at this higher level of cognitive thought and learning gains will be realized for a larger segment of the class.

Instructors can customize the TRQs by changing the following parameters:

  1. Question Pool - the specific questions that can appear on a Student Quiz are selected by the instructor from a Keyword Searchable list maintained by Chem21Labs. In a future version, users will be able to add their own Questions.
  2. Quiz Questions - the actual number of questions that appear on the quiz. This number must be less than, or equal to, the number of questions in the Question Pool.
  3. Quiz Time - the time deemed to represent a desired level of retrieval efficiency. If the time is set too long, the student may be able to "look up" some answers and still submit a Perfect quiz - this defeats the purpose of Path 1. If a student submits a Perfect quiz in less than (or equal to) the time set by the instructor, they receive the Quiz Points and they are finished with that segment of the assignment.
  4. Quiz Points - the points awarded to the student for submitting a Perfect quiz AND beating the Quiz Time. It's also the points awarded to the student for submitting a certain number of correct answers.
  5. Total Correct - total number of correct answers that must be submitted to receive the Quiz Points.
  6. Minimum Correct - the minimum number of correct answers needed on a quiz before this number becomes part of the Total Correct count.
  7. Number of "Sets" - the number of times a student must earn Quiz Points between the Start and Due Date - the Number of Sets times the Quiz Points equals the Total Points for a single TRQ assignment.
  8. Minimum Waiting Period - the minimum time a student must wait between "Sets"



Organic Chemistry

The American Chemical Society Exam has been given as the Spring Final (after OChem II) at Lee University since 1991. From 2005 - present, TRQs were introduced and used in both semesters of Organic Chemistry. In the first fifteen years (1991 - 2005), the class average on this ACS exam was 41 percentile; in the nine years since TRQs have been used, the class average was 59.4 percentile. In addition, only thirteen students in the first fifteen years scored above the 90th percentile on this exam, since Spring 2006, forty-six students have scored above the 90th percentile. While this learning method significantly helps the under-prepared students (based on a very low attrition rates and higher course grades), it also significantly helps the "prepared" students as evidenced by this increase in students scoring above the 90th percentile.

Equally remarkable is the fact that in the past nine years, only three students that started Organic II in January failed to take the ACS Exam in May - in contrast, 15% of the students in the first fifteen years withdrew from the class and didn't take the ACS Final.

How is this possible?

It is entirely based on the learning component described by Skinner where students are asked to complete "small" steps - steps in which they will be successful. After a large series of very small steps, the student has traveled a large academic distance. These Organic students were given a few "small" steps to complete on the first day of the first semester and on many days throughout the year. Only the students who were not willing to "work" (i.e. take a step that there is no doubt they can complete) will fail to complete the class.


Mathematics - Middle School

In 2011 - 2012, TRQs were used by 78 students in a 6th grade math class for 20% of the instructional time - in the remaining 80% of the time, the teacher simply condensed her former classroom activities. The average percentile increase in the state's end-of-class exam (TCAP) was 12.75. Additionally, 43% of the students arriving from the 5th grade were classified as Advanced or Proficient, the remaining 57% were classified as Basic or Below Basic. After using the TRQs for their 6th grade year, 63% of the students performed at the Advanced or Proficient level on this exam. The improvement in TCAP was most noticeable for the Below Basic (+26.5 %-ile) and Basic (+16.5 %-ile) math students.

In 2011 - 2012 and in 2012 - 2013, the TRQs were used in a "related arts" class comprised of only Basic Math Students as determined by their previous year’s state score in Math. In addition to their daily 90 minute Math class, students met this class for 45 minutes every day for 1 semester (there were ~ 140 students in this class each semester). Of the 557 students, 197 students (35.4 %) moved up to either the Proficient or Advanced level on the state's end-of-class exam.

In addition, in 2011 - 2012, 24 students were exposed to TRQs as members of the 78 student 6th grade math class and as members of the "related arts" math class. Twelve (50%) of the students that experienced this "double exposure" to the TRQ method increased 1 or 2 Proficiency levels.

Mongolian Language Study

In April 2014, 19 students from either an Organic Chemistry II class or an upper-level Psychology class called Learning and Cognition were selected to participate in a research project where students were

  1. given 50 Mongolian words, 2 hours, and allowed to use "their best learning method"
  2. given 50 Mongolian words (24 hours later), 2 hours, and required to use the TRQ method

The Mongolian language was chosen because none of the participants had any prior knowledge of that particular language.

Immediately after the two hours of study, the initial assessment showed that students using the TRQ were able to correctly answer 14.5 % more words than the control group. On the 24-hour and 1-week assessments, the TRQ students correctly answered 16.0% and 44.0% more words the control group.

Note, that using the TRQ method for 2 straight hours is not the prescribed route of implementation, but we were interested in a direct comparison of the students' "Best Methods" of studying to the TRQ method. The loss, after 1 week, of 53.5% of words learned using the student's Best Method and 41.5% of words using the TRQ Method, leave little doubt that the repetitive implementation of the TRQ method is essential for moving more information into long-term memory than what was achieved in this study.

Spanish Language - Medical Missions Team

In May 2014, twenty-three university students participated in a Medical Missions trip in which International Health was a required class. One-third of the class grade was based on learning basic Spanish words / phrases along with medical words / phrases. Students had 26 different TRQs that contained ~ 40 questions each (more than 1000 TRQ questions in all) where they had to complete 6 sets on each of the 26 "decks" of information (156 assignments). The following results are based on a TRQ called Nouns 1 which contained 37 questions. Twenty-five questions appeared on each quiz - the student was shown the English word and they were required to spell correctly the corresponding Spanish word.

From the six sets of student results on the Nouns 1 TRQ, the students were divided into three groups: Novice, Intermediate, and Expert.

  • Novice Group (11 students): these students scored between 0 and 10 correct answers on their first four quizzes
  • Intermediate Group (8 students): these students scored between 10 and 20 correct answers on their first four quizzes.
  • Expert Group (4 students): these students scored between 20 and 25 correct answers on their first four quizzes.

Nouns 1 Words || Graph of Student Results

From the Graph of Student Results, it is interesting to note that although the three groups of students started at very different levels of knowledge, after two TRQ Sets the three groups are virtually indistinguishable. The Novice Group averaged 18 quizzes to complete two TRQ Sets, the Intermediate Group averaged 13 quizzes, and the Expert Group averaged 10 quizzes. On the next four TRQ Sets, all three groups averaged 14 quizzes. Assuming the Quiz Points are 10, the Novice Group took 18 quizzes to earn 20 points, the Intermediate Group took 14 quizzes to earn 20 points, and the Expert Group took 10 quizzes to earn 20 points. The remaining 40 points on the Nouns 1 assignment were earned by all groups by only taking 14 additional quizzes spaced-out over four TRQ Sets and over several days.

In today's classroom, the teacher starts the learning process by lecturing (which students have difficulty understanding), worksheets (which students have difficulty doing), collaborative groups (where students have difficulty participating), etc and more than half of the students do not have the requisite knowledge to be successful participants. Based on this study, one can reasonably expect higher levels of learning for a class if the students completed at least two TRQ sets prior to beginning the teacher-initiated learning activities.

Equally amazing is that the students spent an average of 24 hours taking TRQs and studying the webpage where correct and incorrect answers are listed after each TRQ. Students had to complete 6 sets on each of the 26 TRQs (Nouns 1 is just 1 of the 26 TRQs) while waiting a minimum of 12 hours between sets - spacing the learning in this fashion creates memory constructs in ones long-term memory. To put this in perspective, the team traveling to Honduras spent 15 hours travelling from Cleveland, TN to Rio Viejo, Honduras and another 15 hours for the return trip. In less time than it takes to travel to, and from, Honduras, these students were able to learn over 1000 Spanish words / phrases that they used in restaurants, churches, medical clinics, and playing with children.


A PowerPoint on the TRQ Teaching Machine can be dowloaded here.