Diagnostics and Evaluation

Are the learners able to measure their progress and is the teacher able to assure that the instruction is effectively meeting the needs of the learners? Let's look at some tests and diagnostic tools that provide insight into the effectiveness of instruction.

Resilience Testing

The resilience instructional materials, presented here, evolved from a university program and are written for a teen to adult audience. As such, if the instruction is followed, you should expect to see statistically significant gains in general resilience from pre-learning to post learning in groups of ten or more learners. Best outcomes occur when learning in a group setting, though conferencing formats rather than face-to-face are possible. Using pseudonyms with testing, instead of real names, can improve test accuracy. The following testing materials may aid you.

  • Response to Stressful Experiences Scale. A general resilience test that may be a used prior to learning and again sometime after learning is the Response to Stressful Experiences Scale (RSES) available through GSA. RSES PDF »
  • Workbook Skills Tests. Simple learner-centered tests, following each skill in the workbook, can offer measures of progress and understanding. Resilience Workbook PDF »
  • Summative Test. Finally, a simple template for a summative post test following all learning that may be useful in some settings for creating discussion. This test is oriented to the resilience 'flashcard' that supplements the instruction. Evaluation PDF »

Diagnostic 'App'

CatTrax CatTrax is a server-based diagnostic program that works with any test results to improve learning and curriculum design. It can be used in a formative manner to better tailor new learning to diverse settings and audiences whether through classroom-oriented instruction or non-traditional instructional methods. CatTrax can be used to help identify not only what is easy or hard, but areas of curricular confusion, as well as learners who may be experiencing confusion. This means that potentially at-risk learners can be identified at an early stage when intervention is most effective. This process yields far more insight than traditional right/wrong correctness testing for developing the most effective learning and aiding you as an instructor.

Specifics: CatTrax initially provides a brief plain English description identifying problematic student related issues. Next a more detailed graphical and tabled results provides summary results for each student, their correctness score, standard error or measurement, true measure of ability, z-score, and caution index. The caution index established by Sato, Harnish and Linn, assigns a numeric value based on the observed versus the expected responses. Cautionary students show patterns that are atypical of their group, and thus provides an opportunity to address a potentially ‘at-risk’ learner at an early stage when remediation is viable. Item analysis provides test item correctness, difficulty level, discrimination index, and a caution index in either split-half or split-third formats depending upon class size. Cautionary test items may be the result of a poorly written item, ethnic, experiential, gender, or instructional bias or where a potential mismatch between instructional practices and content occurs and results in learner confusion. CatTrax is freely available as source code for your use and modification through the Creative Commons Attribution-NonCommercial 4.0 International License.

  • See the Results! Actual CatTrax diagnostic results of eleven students taking a twenty item test. More » How to interpret the diagnostic results using the CatTrax help page. More »
  • Try Me! Run CatTrax diagnostics remotely from this server. First, download the sample input data file. Feel free to alter and explore the data file using your spreadsheet or text editor, remembering to save changes as a CSV formatted file. Data file » Then, Run CatTrax, Browse for your data file, and Submit to see the results on your web browser. Run CatTrax »
  • Download CatTrax Application. Technical stuff: CatTrax diagnostics is a PHP server-based program, yet a powerful institutional server is not needed. Any Windows, Mac, Linux, or even a tiny Raspberry Pi 3/4 single board computer can run it with Apache or a lighter version server software installed. With the exception of a sample test data file (input.csv), the unzipped CatTrax contents reside in the server HTML directory. Diagnostic results are generated on the fly and viewable in your browser. A CSV formatted results file can also be generated for spreadsheet or other use. CatTrax zip »
  • Alternatively, CatTrax can be run as a "non-server" application using PHPs built-in server -- meaning, you do not need to install Apache or other server software. Some newer operating systems already have PHP installed, for example, Mac OS, Linux, and Raspbian. And, PHP may already show as a menu option on your computer, or, go to your "terminal app," find the unzipped CatTrax folder, start PHP, and open your browser to use CatTrax. Reference the PHP documentation: Built-in web server » for details on how to use PHP server.

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.