Foundations and donors are increasingly questioning the impact of their funds at museums and historic sites, a trend that’s growing as well in business according to Jack Phillips and James Kirkpatrick at a session at the ASTD conference yesterday. After the recent recession, they’ve found that CEOs are increasingly asking about the return on investment (ROI) of every program and activity, including employee training and education. Although training claims to be an essential contributor to business productivity and performance, it hasn’t been adequately measured or evaluated, and thus can’t prove their value. That surprised me because I thought that was a struggle only for museums and historic sites. We seem to be continually fighting to prove our worth and other than economic impact, haven’t been able to show why we matter in our communities. It looks like we’re not alone.
Phillips and Kirkpatrick are the leaders in the field of measuring performance in business and developed frameworks that “define the levels at which programs are evaluated and how data are captured at different times from different sources.” Although they disagree on whether the framework should have four or five levels, they both agree that evaluation is usually stuck at the first two levels. While the Kirkpatrick Model consists of four levels, Phillips has a five-level evaluation framework as follows:
|Level||Measurement Focus||Key Questions Asked|
|1. Reaction and Planned Action||Measures participant satisfaction with the program or process.||Is the program or process relevant, important, useful, or helpful?|
|2. Learning||Measures changes in knowledge, skills, and attitudes.||Did the participants increase or enhance knowledge, skills, or perceptions?|
|3. Application and Implementation||Measures changes in performance or action.||Are participants applying the knowledge, skills, or information?|
|4. Business Impact||Measures changes in key business measures.||How does the application improve output, quality, cost, time, and satisfaction?|
|5. ROI||Compares program benefits to the costs.||Do the monetary benefits of the program exceed the investment in the program?|
Created with the HTML Table Generator
Both believe that education and training programs have to demonstrate their value to the organization and need to figure this out before the CEO asks for this information–otherwise they may lose their budgets and jobs. They also believe that evaluation is a disruptive force and part of the “change management” process. We can make a parallel argument in the museum and historic site field. We need to demonstrate the impact of our programs and activities before funders ask–solid evaluations at higher levels can help make the case. It can also be difficult to implement because people don’t like to change. If you’d like to learn more, both Phillips and Kirkpatrick have lots of resources on their websites and have written several books on the topic.
On an entirely different note, I’m struck by how differently the conference is presented. At a museum or history conference, most sessions are 90-minutes and dominated by panels of 2-3 speakers with a moderator. At ASTD, most sessions are led by one person allowing them to go into a topic in-depth or incorporate some activities. When panel sessions are used, it’s to discuss various perspectives on the same issue (museum conferences tend to avoid disagreements although history conferences are notorious for chewing up speakers during the Q&A). They also vary the length of the sessions, starting with 120-minute sessions at 10 am and then slowly reducing to 60-minutes for the 4:30 pm session. That provides a way to match time and topic, but also follows the declining energy of participants as the day progresses.