Week 3: Evaluating Programs & Human Performance Technology

1.    Chapters in Section III discuss evaluation in instructional design and provide you with three evaluation models: CIPP, Five-Domain, and Kirkpatrick’s Training Evaluation model. Search for at least two other models used for evaluation and summarize these models. Describe how you would use them to evaluate your instruction.

Brinkerhoff’s Success Case Method (SCM):

        

Brinkerhoff’s Success Case Method focuses on identifying and then studying in great detail both successful and unsuccessful cases within a given learning program.  Through a comprehensive comparison one can establish what worked and what changes/modifications need to be implemented to the program’s design in order to help ensure future success.  Within the pages of Trends and Issues in Instructional Design and Technology, Reiser and Dempsey have identified five steps to Brinkerhoff’s Success Case Method including:

       

            Step1   The “evaluator” must define the program to be evaluated, identify which goals are to be met, and agree on a budget and time frame for the study along with the other stakeholders.    

            Step 2  A “visual impact model” is fashioned next which outlines the “capabilities (knowledge and skill provided by the program)”, the “business goals (what the stakeholders stand to gain),” and also what “behaviors and organizational/environmental conditions must be present in order to achieve the desired results.”

            Step 3  Involves a “research study” in which individuals are surveyed to help establish what constitutes a “best and worst case.”  The questionnaire used is typically quite short focusing on these three main questions;

“To what extent have you been able to use the program to achieve success?”

“Who is experiencing a lot of success in using the program?”

“Who is having the least success in using the program?”

            Step 4  An “in-depth interview” is conducted to gain empirical evidence demonstrating which performance factors lead to successful cases verses the obstacles which hindered the unsuccessful cases.  During success case interviews, Brinkerhoff suggests gathering data on the information/strategies/skills which the participant felt were effective, the successful results/outcomes achieved, the difference they made, the value of the experience, the factors which they feel lead to this successful result, and finally, suggestions for program improvement.  During nonsuccess case interviews, Brinkerhoff suggests focusing on perceived barriers and lack of use factors which prevented participants from applying what they learned as well as “suggestions for increasing future program use.”

            Step 5  “Document and communicate the evaluation findings” in the form of a final report.  This information needs to be shared with the stakeholders and include “detailed data and evidence” demonstrating the program’s success as well as recommendations for future improvements.

 

In applying this evaluation model to my classroom instruction, I would begin by serving as the evaluator and determining, for the purpose of this case study, which goals and/or outcomes would be focused.  I would include a detailed description of what a successful case should look like, create a visual impact model to help myself and others envision the proposed plan, and also develop a brief questionnaire for the students to fill out helping me to establish successful/unsuccessful cases.  Interviews would be conducted, a final report written, and I would then share the results with my stakeholders (students, department head, and administrators).  Knowing that they were going to be asked for their feedback, I believe that the students would feel more involved in the learning process and may gain a fuller understanding of the activity by viewing it with a critical-eye for detail as a co-collaborator in their own learning experience.

Patton’s Utilization-Focused Evaluation (U-FE)  

            Patton’s Utilization- Focused Evaluation method emphasizes the importance of creating evaluations “for and with an intended user in mind” so that they will have value, be used, and hopefully result in positive program changes.  This “participatory evaluation approach” involves both the client and primary users who will be tasked with becoming “actively involved in the structuring, conducting, interpreting, and using evaluation results.”  From beginning to end, the evaluator’s main focus is on the continual assessment of, and ongoing adaption to, the evaluation in the hopes of creating a working environment in which the “evaluation results will continue to be used even after the evaluator leaves the organization.”  This is referred to as “process use” in which “organizational members are empowered to become internal evaluators.”  Below, I have included a (U-FE) checklist which was taken from Michael Quinn Patton’s work entitled Essentials of Utilization-Focused Evaluation. 

Step 1  Assess and build program and organizational readiness for utilization-focused evaluation.

Step 2  Assess and enhance evaluator readiness and competence to undertake a utilization- focused evaluation.

Step 3  Identify, organize, and engage primary intended users.

Step 4  Conduct situation analysis with primary intended users

Step 5  Identify primary intended uses by establishing the evaluation’s priority purposes.

Step 6  Consider and build in process uses if appropriate.

Step 7  Focus priority evaluation questions.

Step 8  Check that fundamental areas for evaluation inquiry are being adequately addressed.

Step 9  Determine what intervention model or theory of change is being evaluated.

Step 10  Negotiate appropriate methods to generate credible findings and support intended use by intended users.

Step 11  Make sure intended users understand potential controversies about methods and their implications.

Step 12  Simulate use of findings.

Step 13  Gather data with ongoing attention to use.

Step 14  Organize and present the data for use by primary intended users.

Step 15  Prepare an evaluation report to facilitate use and disseminate significant findings to expand influence.

Step 16  Follow up with primary intended users to facilitate and enhance use.

Step 17  Metaevaluation of use: Be accountable, learn, and improve

           

In applying this evaluation model to my classroom instruction, as I am already a member of the school’s faculty, I would not need to conduct a “readiness assessment” or “identify the primary users and develop a working relationship with them” as I would be the one committing to the U-FE and have already established a rapport with my colleagues and administrators.  I would however need to decide on which stakeholders to involve in the collection of data while also working to develop their understanding of the evaluation process and commitment to its continued use/implementation.  I would want to review any available past evaluations, stakeholder’s concerns, and conduct individual or small group interviews to help establish a baseline of pertinent information.  Evaluation standards would then be introduced and priorities established all-the-while involving the stakeholders in the process to make sure they feel that they are part of its development, are valued, and to promote a sense of ownership in the procedure.  If the stakeholders genuinely care about the findings, the likelihood of them continuing to participate in on-going/future evaluations greatly increases which of course is the main goal of Patton’s Utilization- Focused Evaluation method.   

 

2.    Reflect on what other questions that instructional design evaluation should address besides whether the instructional design leads to comparable amounts of learning and learner satisfaction as traditional methods. Should return on investment and management of resources also be considered in evaluations of instructional programs? What other measures should/could be considered?

In addition to instructional design evaluations addressing learning improvement and student satisfaction, perhaps these evaluations should also focus on whether or not a product/program includes a built-in support system for addressing user questions, training, operating problems, and system updates.  Without these considerations being properly addressed, a district could very easily end up spending an inordinate amount of money on something which is not properly supported and ends up yielding no discernible, positive results on instruction or student performance leading to a monumental waste of tax-payer monies with little or no return on investment (ROI).  Activity does not necessarily equate to results, something which our text has referred to as a “value shift,” just because the students are engaged doesn’t necessarily mean that they are interacting with the information in a meaningful/measurable way. 

 

Another consideration should include the amount of time and effort needed for proper implementation of a product/program above and beyond that which is required for regular daily instruction.  If a district chooses to implement a new program without regard, allowance, or additional compensation for their staff the result can be disastrous; stress levels and tempers can rise, performance measures drop, and employee turn-over drastically increase.  Quality of life for educators should be a consideration in any evaluation; happy teachers are usually more productive and less prone to absenteeism and/or departure from a school/district which equates to higher student performance. 

 

A final product/program consideration should focus on the students themselves, in particular, not just their learning improvement and student satisfaction but also whether or not these products/programs are giving them an opportunity to interact socially, collaboratively instead of only in isolation.  In order to prepare them to enter the work force as a successful, contributing of our society they must be given an opportunity to work collectively in problem solving situations, over time, in a safe environment, encouraging conversation to develop the communication skills they’ll need and rely on as adults.          

 

3.    Section IV focuses on human performance, performance support systems, knowledge management systems, and the concept of informal learning. Not all problems in learning and/or performance require an instructional one. Many times a non-instructional approach is a more appropriate solution. Identify a performance problem in your area of work and identify non-instructional solutions that may help solve the problem. Would better performance support systems, knowledge management systems, or opportunities for informal learning solve the problem?

Every semester I have students who will sit silently with genuine questions and/or concerns about the material presented in my 7^th grade honors pre-algebra class without ever saying a word.  I think this is due in part to the misconception that if you have to ask for clarification you’re either “stupid” or “slow.”  Even though I actively encourage questions and when none present themselves then I have some prepared which hopefully help some of my silent sufferers.  And that’s when it hit me that “Twitter could have a potential place in my world as an educator, possibly... Maybe, we’ll see.”  I like the idea of anonymity; and by having my students set up and use twitter accounts created specifically for my class, I could give them an opportunity to send me a tweet privately, without fear of reproach, in which I could address their concerns without them being singled out by others in class. The lack of physical contact and partial anonymity of this community, I think, would allow for a more honest feedback and critiques from my learners allowing for greater insight and collaboration than would be possible in a face-to-face framework.  By removing obstacles, promoting the community, and encouraging conversation between users I would be fostering this informal learning environment in which students could learn as much from each other as they do in class.  The ability to read and respond to tweets from anywhere, anytime is also a genuine plus. 

 

I have spent many, many hours exploring Twitter’s potential integration into a classroom environment and I am more than a little impressed with some of the creative ways educators have chosen to use this Web2.0 tool within their instructional model.  For instance, I spend a great deal of time every week posting my weekly lesson plans including information on upcoming assignments, quizzes, tests, projects, etc… through our on-line gradebook program only to have my students and parents tell me that they “didn’t know about the homework assignment” or “when the unit review was due” or my personal favorite “What Test!?!”  This was a constant source of irritation and concern, sometimes I felt like I was just wasting my time and knew there had to be a better way of informing them beside or in addition to what I was already doing.  Enter Twitter.  What an incredibly simple yet effective idea, I could have simply set up a Twitter feed dedicated specifically to important class information, simple, effective, problem solved.  These Tweets could have included a reminder of my tutorial times, assignment & review due dates, late work submission deadlines, upcoming quizzes, projects, or quizzes, the end of each grading cycle, and STAAR testing dates.  I really missed an opportunity to work “Smarter not Harder.”  Any downtime I could find during the day was dedicated to calling and emailing parents, sending home reminders, and reminding students when I could have simply used the direct message feature on Twitter to speak privately with them.  I feel a little silly looking back now in the realization of how much more effectively I could have communicated this pertinent information.  There’s so much more to Tweeting than I could have ever imagined, the sheer versatility of the “Twitter-verse” is amazing and I sense that I’m only just beginning to scratch the surface of its potential as an educational tool.  I’m looking forward to incorporating Twitter into my instructional design this fall.     

Other ideas for Twitter integration within an educational setting which caught my eye include:

 

·         Connecting Classrooms – Allowing my pupils to interact and collaborate with students from other classes, schools, cities, states, or even other countries on shared projects.

      ·         Facilitating Exploration – By using Twitter’s search engine to introduce and investigate new concepts.

      ·         Posting Sample Questions – By using Twitter to post sample questions to
            upcoming assignments/tests.

·         Facilitate Discussion – By carrying on a discussion, within the classroom completely on Twitter allowing all an equal voice in the conversation especially including those who do not feel comfortable contributing verbally in class.

·         Posting Syllabus Changes

·         Polling Students – Involving students in the decision making process of what types of activities, assignments, and/or projects to use within the class.

·         Teach Probability – By having students respond to broad, general questions and then chronicling and charting the results through @replies.  

·         Post Weekly Math Puzzles – By microblogging a problem of the week which students work on, solve, and then Tweet back their solution.

·         Post Videos – By using Twiddeo, a service which allows users to send Twitter updates via video, to post clips of instruction for students who are absent or those who simply need to review a specific topic.

·         TweetStats – Allows users to input account information to create a bar graph showing how and when an account is used.

·         Summarization – Twitter can be used as an “Exit Ticket” for students to summarize, in 140 characters or less, the day’s lesson while also messaging any questions they might have on the topic.

 

References:

Dempsey, J.V., & Reiser, R. A. (2012). Trends and Issues in Instructional Design and Technology (3rd ed.). Boston, MA: Pearson Education, Inc.    

Patton, Michael Q. (2012) Essentials of Utilization-Focused Evaluation. London: SAGE.