Les Misérables: The Impact of Unfavorable Psychology Exam Results and the Redemptive Power of Evidence-Based Instruction

Although completing a master’s or doctoral degree in psychology is a coveted goal, it is only a penultimate accomplishment for select scholars. The decisive achievement involves the receipt of a passing score on a state-specific licensure or international certification exam. Earning a graduate degree certainly has a favorable impact on one’s amour-propre, but it is the latter triumph (i.e., passing the exam) that opens the portal to professional practice, preferred employment, and access to benefits commensurate with a graduate’s education, experience, and skill. In consequence, taking an exam is an important professional step for scholars who have made the decision to pursue licensure or certification in psychology. When exam results are released, everyone knows that the person afresh with news about their passing test score will be hoisted upon the shoulders of family and friends and celebrated, while the whole world avoids eye contact with Les Misérables: the people who failed the exam. The purpose of the present paper is to create a conceptual framework for circumventing, or managing, an unfavorable examination score. Specifically, we highlight the cumulative nature of learning, describe the importance knowledge assessment (i.e., via Bloom’s Taxonomy of educational learning objectives), and discuss eight evidence-based instructional strategies to address the needs of diverse learners. The recommendations within this paper are based on learning theory and place emphasis on how learners acquire, process, and retain information. The overall intention of this paper is to stimulate conversation about people who have devoted substantial resources to a course of study but failed to achieve a passing score on a licensure or certification exam.


Introduction
Although completing a master's or doctoral degree in psychology is a coveted goal, it is only a penultimate accomplishment for select scholars. The decisive achievement involves the receipt of a passing score on a state-specific licensure or international certification exam. Earning a graduate degree certainly has a favorable impact on one's amour-propre, but it is the latter triumph (i.e., passing the exam) that opens the portal to professional practice, preferred employment, and access to benefits commensurate with a graduate's education, experience, and skill. In consequence, taking an exam is an important professional step for scholars who have made the decision to pursue licensure or certification in psychology. When exam results are released, everyone knows that the person afresh with news about their passing test score will be hoisted upon the shoulders of family and friends and celebrated, while the whole world avoids eye contact with Les Misérables: the people who failed the exam. The purpose of the present paper is to create a conceptual framework for circumventing, or managing, an unfavorable examination score. Specifically, we highlight the cumulative nature of learning, describe the importance knowledge assessment (i.e., via Bloom's Taxonomy of educational learning objectives), and discuss eight evidence-based instructional strategies to address the needs of diverse learners. The recommendations within this paper are based on learning theory and place emphasis on how learners acquire, process, and retain information. The overall intention of this paper is to stimulate conversation about people who have devoted substantial resources to a course of study but failed to achieve a passing score on a licensure or certification exam. and other animals" [1]. Individuals who study behavior analysis and acquire a master's or doctoral degree from an approved university are eligible to sit for the BCBA® exam. Acquiring a graduate degree in behavior analysis involves an Egyptian pyramid's worth of effort; however, the worthwhileness of the work is warranted by all who have attained the desired professional footing by passing the certification exam; but not everyone passes the exam. According to the Behavior Analyst Certification Board's (BACB) annual data report, 6,124 candidates were tested in 2018 [2]. Sixty-five percent of those candidates passed the exam and 2,143 people failed. Within the same year, 4,454 candidates who previously failed the BCBA exam were reassessed. Only 28% of those candidates received a passing score and 3,206 individuals were unsuccessful. In total, 5,349 individuals received a substandard exam score in 2018; enough people to fill an amphitheater. Specifically, a theatre of affect and 10,698 doleful eyes. It is certainly the case that BCBA test candidates fall along a continuum of preparedness; however, many recipients of negative news are genuinely unaware of the reasons for a failing test score and seek compensatory education. However, regardless of the level of motivation and good intentions of a remedial instructor, a person's corrective strategy must be informed by an understanding of learning theory and methodologies for teaching adult learners (i.e., andragogy). Specifically, learners with highly varied experiential backgrounds, educational histories, and diverse neurological presentations require a structured process which gives primacy to instruction and learning outcomes simultaneously.

The science of learning
Three components of effective adult education are:

2.
Use of evidence-based teaching strategies, and 3. Strength-based instruction.
Even the most enthusiastic, Knowledgeable, and wellintentioned instructors furnished with a comprehensive curriculum will not meet the needs of their learners unless they understand the science of teaching and learning. Attainment of successful instructional outcomes requires instructors who understand variation in learner preferences and how different people are optimized as students. Moreover, any approach to teaching and learning must be guided by clarity regarding learning outcomes. Bloom's taxonomy, although not the only guide for learning outcomes, serves as an excellent resource for classifying learning objectives.

Bloom's taxonomy of learning: An overview
Knowledge acquisition is the result of "cumulative hierarchical learning," a concept first described by the American educational psychologist Benjamin Bloom in the 1950s [3]. Specifically, Bloom, the most cited pedagogical theorist, showed how educational attainment rises through different orders of learning, ranging from lower-order skills, such as basic information recall, to higher-order skills involving more complex human behavior [4]. Specifically, Bloom's taxonomy divides learning activities into six hierarchically ranked categories and identifies specific instructional tasks that meet each category's learning objective. A learner, guided by Bloom's taxonomy, moves incrementally and methodically through the six categories of learning. Remembering is the first category, requiring overt and measurable actions such as defining terms (e.g., positive reinforcement) or listing information from memory, such as assessment methodologies for identifying putative reinforcers to accelerate behavior. Subsequent to a learner's accumulation of basic knowledge, a demonstration of understanding is required.
Evidence of understanding (i.e., comprehension) takes the form of measurable verbal behavior such as explaining concepts (e.g., unconditioned motivating operations) and summarizing information (e.g., the effect of unconditioned motivating operations on human behavior). Eventually, in a progressive fashion, learners makes their way through more advanced stages of learning involving application (e.g., demonstrating their ability to use a psychological intervention), analysis (e.g., the ability to deconstruct a multicomponent behavioral intervention into constituent parts and complete a component analysis to determine which parts are essential for behavior change), evaluation (e.g., the ability to judge the work of BCBAs or critique a research study published in a scientific journal), and creation (e.g., the ability to create a scatterplot). Instructors guided by Bloom's taxonomy can use a resource guide (e.g., Bloom's taxonomy action verbs) to ensure they are advancing from lower-order skills to higher-order skills and requiring observable and verifiable demonstrations of knowledge [4]. In short, categories of learning are arranged in a hierarchy of progressive difficulty within Bloom's taxonomy. In the absence of a learning objective classification system, there is a risk that only lower-order skills (e.g., having learners memorize flashcards and demonstrate their rapid recall of definitions) will become the central point of instruction. Using Bloom's taxonomy is one way to guide the development of knowledge and remediate insufficient learning. The taxonomy serves as a minimum standard set of outcome measures to evaluate the efficacy of instruction. A quote by Stephen Covey (2004) is apropos: "Begin with the end in mind" [5]. Bloom's taxonomy of learning allows instructors and learners to begin the learning process with clarity regarding the end goal of instruction. In summary, Bloom's taxonomy of learning is an excellent classification system for learning objectives.
The taxonomy can be used to:

1.
Access a visual representation of learning outcomes,

2.
Access action verbs to ensure that a learner is comprehensively trained, and 3.
Acquire learning outcomes to guide a learner from lower-order to higher-order skills. Despite the utility of Bloom's taxonomy, a learner will not demonstrate requisite knowledge unless evidence-based instructional strategies are employed [4].

Evidence-based instructional strategies for adult learners
Specific strategies have been shown to maximize learning efficiency and outcomes. Moreover, specific teaching strategies

PPRS.000559. 3(2).2019
should be consistently and systematically used to support a learner. Evidence-based instructional strategies include, but are not limited to, scaffolding, interleaving, retrieval practice, elaboration, construction of concrete examples, dual coding, strengths-based instruction, and the think aloud procedure. Each of these strategies can be used for developing lower-order and higher-order skills. The eight strategies reviewed here constitute a modest deposit into the armamentarium of a competent instructor and should be conjoined with additional evidence-based instructional strategies. We assert that the integration of research-based instructional strategies into the architecture of instruction will increase the probability of success on a licensure or certification exam.

Scaffolding
Scaffolding is viewed as an interactive process that occurs between instructor and student; it is a bidirectional activity requiring active participation [6,7]. In general, scaffolding is described as a support given by an instructor to a student when performing a task that the student might otherwise not be able to accomplish [8]. Instructors can facilitate small-scale leaps in knowledge by skillfully providing feedback, hints, microburst teaching, explanations, modeling, and Socratic questioning.
There are three common characteristics of scaffolding:
Contingent support requires an instructor to first evaluate a student's baseline level of performance and then respond to a learner's need for assistance with calibrated guidance. Essentially, scaffolding involves knowledge shaping. An instructor who uses scaffolding needs to pinpoint the skills the learner already possesses (i.e., the learner's baseline performance), what the learner needs to achieve, and what interventions to employ (e.g., hints or thematic prompts) in order to move, by way of successive approximation, to a terminal goal. Fading involves the gradual withdrawal of support, and a transfer of responsibility occurs when the learner assumes an increasing level of control (i.e., self-mediated instruction). Scaffolding requires a systematic assessment of a learner's skills, an understanding of the lower-order and higher-order competencies the learner needs to acquire, and a treasury of instructional strategies to facilitate a learner's development. In essence, the implementation of scaffolding as an instructional strategy requires (1) a baseline assessment of a learner's knowledge, (2) clarity regarding the level of learning that needs to be demonstrated (e.g., analysis), and (3) the ability to use instructional supports (i.e., feedback, hints, modeling). Scaffolding should be supplemented by other instructional strategies such as interleaving.

Interleaving
Interleaving occurs when disparate ideas or problem types are addressed in a sequential fashion, as opposed to the traditional method of presenting "blocks" of similar problems. An example of interleaving would involve learners defining a term (unconditioned reinforcer), and then comparing the term to another concept (motivating operation) to demonstrate their comprehension. Subsequently, learners demonstrate their ability to describe a mini experiment to evaluate the impact of an unconditioned reinforcer on a target behavior, and finally, evaluate the work of a peer. In this example, different topics are intermixed, and learners are required to demonstrate their (1) knowledge, (2) understanding, (3) ability to apply a concept, and (4) ability to evaluate the work of others. The benefit of interleaving is that the instructional strategy requires a learner to demonstrate psychological flexibility by sequentially addressing different types of concepts, and then advance from lower-order skills (i.e., defining a term) to higher-order skills (i.e., evaluating the work of other people).
In addition, interleaving also impels an instructor to gauge a learner's depth and breadth of knowledge as opposed to myopically focusing on lower-order skills (e.g., the ability to rapidly define terms presented on flash cards). Another form of interleaving involves the oscillation between study and test opportunities. Specifically, learners alternate between attempting to solve a problem and viewing a worked example [9]. Researchers have found that the alternation between independent problem solving and a review of worked examples results in a lower number of learning trials to achieve criterion performance [10]. Interleaving has been used as an instructional strategy for teaching math [11], art history [12], and ornithology [13]. When learning is guided by Bloom's taxonomy and supported by evidence-based instructional strategies such as interleaving, learning can occur expeditiously. The practical application of interleaving will require resources such as the excellent textbook titled Case Studies in Applied Behavior Analysis for Students and Adults with Disabilities [14], which contains detailed methods to alternate between problem scenarios and worked examples. As mentioned earlier, a skilled instructor is equipped with a wide array of strategies and is prepared to meet the needs of learners with highly diverse presentations. Another instructional strategy that is part of the repertoire of a skilled instructor is retrieval-based learning.

Retrieval-based learning
Tests are most often used in an educational context to evaluate knowledge. However, research shows that the repeated act of retrieving knowledge from short-term or long-term memory improves information retention and recall [15]. In addition, the practice of retrieving information has been shown to facilitate the generalization of knowledge to new situations [16,17]. An instructor can require a learner to retrieve information by using a variety of methods. For example, information retrieval can be accomplished by (1) giving a learner a practice test involving shortanswer or multiple-choice questions [18], (2) providing openended prompts so a learner can recall information [19], and (3) requiring a learner to create a concept map from memory [20]. The implication here is that instructors should include frequent and diverse opportunities to retrieve previously acquired information. The practical application of retrieval-based learning will require access to a collection of practice tests and an understanding of concept mapping. However, instructors should use a variety of

PPRS.000559. 3(2).2019
methods to increase the probability of information retrieval. An instructional method referred to as "elaboration" has been used to ensure that proper relational frames are constructed during the learning process.

Elaboration
Elaboration involves connecting new information to preexisting knowledge. Elaboration has been described as "one of the most potent manipulations that can be performed in terms of increasing a subject's memory for material [by having] the subject elaborates on the to-be-remembered material" [21]. When a learner's elaboration is skillfully guided, the result is the establishment of new and replete relational frames. Specifically, an entire network of information, concepts, and stimuli merge as a result of repeated elaborations and form a composite of serviceable language. Fluency with relational frames predicts good performance on a variety of verbal or cognitive tasks [22]. Another benefit of elaboration is that the repeated practice of expounding and explicating can expedite the organization of information [23,24]. A specific rendition of elaboration is called elaborative interrogation [25]. Learners using this technique ask "how" and "why" questions about the concepts they are studying (e.g., How does delayed discounting work? Why do people spontaneously incorporate uncertainty into their valuations of delayed rewards?). Subsequently, learners will answer these questions by referencing resources they have access to, or as a result of teacher-mediated instruction.
The practical application of this strategy requires an understanding of how to teach a learner to chunk, elaborate upon, and map critical information. The value of the elaboration process can be elevated as a result of a requirement for concrete examples. Inter-teaching is another teaching methodology that has been proven to be successful with a wide range of students and across different content areas. One of the components of inter-teaching requires two peers to converse about pre-selected material assigned by an instructor. The intention is to have of both learners pose questions to each other and inform one another about the topic. In this way, learners are able to utilize elaborative interrogation but asking "how" and "why" questions. As a result of this exchange the students are then able to inform the instructor of any areas of difficulty and request a review of challenging material.

Concrete examples
Information that is visually represented enhances the acquisition of critical associations, even when abstract content is being reviewed [26]. For example, in one study, high school students diagnosed with dyslexia were taught vocabulary words in preparation for the Scholastic Aptitude Test (SAT) [27]. The study was conducted to support the students in their preparation for the SAT and for completing applications for highly selective colleges. An alternating treatment experimental design was used in the study. Specifically, the high school's standard method for teaching vocabulary words (i.e., a "find and define" method supported by flashcards) was compared, in an alternating arrangement, to a word/picture association method wherein vocabulary words were combined with visual images and word rhyming mnemonics. The dependent variables in the study included performance on weekly vocabulary tests and memory retention tests. The results of the study showed that vocabulary and learning retention test performances were far superior when the students with dyslexia were taught via word/picture associations. The images used in a word/picture instructional arrangement can be generated freehand by the artistically inclined, or via digital art software for those who require creative support. Word/picture arrangements for instructional purposes can also be purchased. The application of this strategy in the field of Applied Behavior Analysis is supported by an excellent book by titled The Applied Behavior Analysis (ABA) Visual Language [28]. The word/picture association arrangement reconciles with dual coding theory.

Dual coding
Dual coding theory suggests that various associations (e.g., pictorial, video, and motoric) linked with specific information enhances learning and memory [29,30]. Marketing professionals use dual coding theory to design messages around their products due to the "stickiness" of the impact on customers. We submit that the same creativity that goes into advertising and marketing should be applied to education. A dual coding approach can also take the form of motor associations. Motor movements can provide an additional motor code that can improve memory [31]. For example, touching body parts as learners recite different categories of reinforcement may improve their total recall (i.e., touching one's nose when recalling olfactory stimuli as a source of reinforcement, one's ears for recalling auditory reinforcers, one's eyes for recalling visual reinforcers, one's mouth for recalling gustatory reinforcers). Linking information from studies centered on motor actions with dual coding theory can contribute to superior instruction [32][33][34]. The actual implementation of dual coding as an instructional strategy requires the identification of contextually related images and motor movements to pair with a concept.

Strengths-based instruction
"Strengths-based education, though grounded in historical tenets and practices, is also built on five modern-day educational principles: (a) the measurement of strengths, achievement, and determinants of positive outcomes, (b) individualization, which requires a tailoring of the teacher's/advisor's methods to student needs and interests, (c) networking with friends, family, and professionals who affirm strengths (Bowers, 2009), (d) deliberate application of strengths in and out of the classroom, and (e) intentional development of strengths through novel experience or focused practice across a period such as a semester, academic year, or an internship" [35]. A strengths-based approach can be implemented by evaluating a learner's interests and previous accomplishments, and then deliberately linking new learning to established sources of stimulus control. By way of example, if a learner has a background in dance (e.g., Mexico's traditional dance, the Jarabe Tapatío), then many new concepts can be linked back to the learner's existing knowledge. For example, the definition of an interdependent group contingency could be linked to dance

PPRS.000559. 3(2).2019
rehearsals wherein a dance instructor's acknowledgement occurs only when all members of the dance group perform at some criterion level. The application of a strengths-based approach requires an instructor to identify a learner's prior knowledge and experience and then, by way of effective communication, coconstruct new repertoires of behavior based upon established sources of knowledge.

Think aloud method
Cognitive psychologists have described a method termed "protocol analysis" that requires a learner to "think aloud" while engaged in some form of problem-solving task [36,37]. The think aloud strategy requires learners to articulate what they are thinking when reading, solving a problem, or responding to a question posed by an instructor or a peer. The procedure allows for the real-time analysis of verbal behavior and provides the listener (i.e., an instructor) insight into the learner's knowledge, reasoning skills, information gaps, and conceptual inconsistencies. Cognitive scientists submit that having learners articulate their thinking process does not interfere with their performance on a task. A version of the think aloud procedure [38] is the Thinking Aloud Pair Problem Solving (TAPPS) method. The procedure involves the pairing of two learners; one serves as a speaker (i.e., problem solver) and the other assumes a listening role. After one think aloud episode, the individuals in the dyad will switch roles. The TAPPS procedure allows learners to retrieve and rehearse concepts, link concepts to established frameworks, and demonstrate a comprehensive understanding of material [38]. The think aloud procedure allows an instructor to evaluate lower-order and higherorder skills.
From the standpoint of practical application, a lower-order skill (i.e., remembering) could involve an instructor using the think aloud procedure to evaluate a learner's capacity to remember information by presenting the learner with a term and requiring the individual to provide a definition. Alternatively, the instructor could require a symmetrical performance wherein the instructor presents the learner with a definition and requires the individual to generate the appropriate term. Similarly, a higher-order skill such as evaluation could involve the TAPPS method. In this case, a learner could be required to evaluate the accuracy of another student's analysis of quantitative data plotted on a standard celebration chart. The think aloud procedure can be used in combination with scaffolding, interleaving, retrieval-based learning, elaboration, the generation of concrete examples, dual coding, and strengthsbased instruction. The TAPPS method has been used extensively at Morningside Academy; a small private laboratory school located in the state of Washington [39][40][41].

Conclusion
Exam failure has far reaching consequences. Specifically, licensure or certification exam failure impacts learners' selfperceptions, their abilities to secure meaningful employment, their abilities to re-pay student loans, and their sense of membership in a professional group (e.g., licensed psychologists or BCBAs).
People that have received an unfavorable score on their licensure or certification exam have been the focus of failure analysis; however, the analyses have been devoted to the study of the differences between academic program models. For example, in 2013 the failure rate on the EPPP was addressed [42]. Specifically, the researchers compared the exam scores of professional school graduates (i.e., graduates of programs based on the Vail Model or Scholar-Practitioner Model) to graduates of a traditional program (i.e., graduates of a program based on the Boulder Model or Scientist-Practitioner Model). Professional school (Vail Model) graduates had a failure rate of 30.82%, in contrast to 7.60% for EPPP licensing exams taken by traditional school (Boulder Model) graduates [42]. More precisely, Vail Model graduates are 4.06 times more likely to fail the exams than students taught using the Boulder Model. In addition, Vail Model graduates are less likely to be a diplomate in the American Board of Professional Psychology (ABPP) and less likely to be president of state psychological associations [42]. Similarly, the Behavior Analyst Certification Board publishes data centered on the pass rate for university programs (i.e., both online and campus-based programs). According to the 2018 annual data report, the pass rate for universities ranged from 18% to 100% [2].
Although correlational research centered on exam pass/fail rates is informative, the root cause of exam success and failure is of greater interest. Certainly, multiple factors contribute to favorable and unfavorable outcomes. Furthermore, although the reasons for a learner's success, or failure, cannot be oversimplified, the quality of a program's knowledge assessment practices, curriculum, and instructional strategies is undeniably linked to a learner's performance on a licensure or certification exam. In consequence, we submit that combining Bloom's taxonomy of learning with instructional strategies based on sound design principles empowers educators to teach the most complex cognitive skills to highly diverse learners. Therefore, instructors and mentors need to be conversant with various taxonomies of learning outcomes (e.g., Bloom's taxonomy, The teach thought learning taxonomy, Six Facets of Understanding, Marzano & Kendall taxonomy, The taxonomy of significant learning, Danielson's depth of knowledge framework, and The SOLO taxonomy). A taxonomy of learning outcomes gives clarity regarding: (a) the lower-order skills that need to be developed, (b) the higher-level skills that need to be acquired, and (c) a delineation of the specific "learner behaviors" that need to be measured.
Of course, competent instruction also requires an understanding of the evidence-based practices for teaching adult learners; therefore, in this paper eight evidence-based practices were reviewed: scaffolding, interleaving, retrieval-based learning, elaboration, use of concreate examples, dual coding, strengthsbased instruction, and the think aloud method. The evidencebased practices reviewed in this article hardly represent the full scope of practices that can be used but do provide a deposit into an overall instructional strategy bank. Once equipped with an array of evidence-based instructional strategies, it would be in the best interest of instructors to use a self-monitoring tool to ensure that they are addressing lower-order and higher-order skills,

PPRS.000559. 3(2).2019
evaluating knowledge in a credible manner, and using evidencebased instructional methods. Similarly, it would be advantageous for learners to use a monitoring tool to ensure they (a) are clear about the lower-order and higher-order skills they need to acquire, (b) possess a reference guide to observable and measurable learner behaviors, and (c) understand the teaching strategies that should be used to support their learning process. Scholars from every academic discipline invest a significant number of resources into their education. However, individuals who elect to study psychology, or applied behavior analysis, should be the primary beneficiaries of their own field's scientific findings. More to the point, graduate students in the area of psychology and applied behavior analysis should benefit from accumulated knowledge regarding a minimum standard set of learning outcomes (e.g., Bloom's taxonomy) and all the evidence-based practices used to teach adult learners. We assert that if students are guided throughout their graduate school training by state-of-the-science knowledge assessment and instructional interventions, following the release of the licensure or certification exam results they will be counted among the merry, and not Les Misérables.