Evaluating Training - An Analysis Session

 

Background

A client has just completed a project whose purpose was to do a retrospective evaluation of a standard training course. The client is a major engineering company, and the course is for graduate engineers who have about two or three years service. The research team consisted of two people – a consultant colleague of mine, and the company’s training manager.

I must confess to having my heart in my mouth because this was the first major project using Enquire Within that I was associated with in which I was not involved until the analysis stage: the two interviewers had had one day’s experience with Enquire Within and a little advice on choosing the right elements. (I include this information not for the human interest, but as an indication of how long it takes to learn to apply Enquire Within sufficiently well to use it on a fairly challenging topic where you are exposed to public view).

Evaluation Objectives

The client wanted to evaluate the long-term effects of the course. He knew that it would probably not be possible to put a money value on the results, but he wanted to know what the relevance of different parts of the course was to the engineers back at the job. So they used element creation questions, such as ‘Name the part of the course which was most useful ... least useful ... was quickly useful ... took me time to realise the usefulness....’ etc. Then they asked the sample to do an Enquire Within session thinking about how they had used the knowledge afterwards.

At the start of the analysis session, the client clearly stated his objectives: he would like to leave with at least six incontrovertible facts about the course which he could not have gained (or couldn’t easily have gained) without Enquire Within.

The Method

So the three of us settled down with the sessions and the print-outs. Here’s what we did:

1. Look for obvious patterns in the elements

The first thing to do was to see whether there was any commonality across interviewees in their answer to the first element creation question - ‘name the most useful part of the course’. Much to the client’s pleasure, there was a pattern: two sessions competed with one another for the accolade of being ‘most useful’. Then we looked for the session(s) nominated as ‘least useful’ and in that case there was near unanimity for one particular session. At this point the client said that this ‘least useful’ session was invariably scored highly on the post-course ‘happiness sheets’, so there is obviously an issue to be investigated there. (I can’t give any information about the content of the course, for reasons of client confidentiality).

2. Look for how the elements are rated

We then looked to see how the top- and bottom-rated sessions were described in the Grid. Again, there was a surprising degree of commonality, and another subject for the client to investigate later. (At this point I would inject a personal observation, which the client uttered before I did, to the effect that with a relatively small sample it’s much more interesting to build the picture up slowly, talking amongst yourselves while you do so, because you see more of the subtleties than you would if you put all the data from all the interviews into a summary first).

3. Look for correlations in the constructs

One obvious correlation to inspect was one which the interviewers had added, if it didn’t arise in the interview, i.e. has been most helpful to me since - has been least helpful to me since. In one way that was just taking a different angle on the first two questions, but we were looking for more differentiated information about the characteristics associated with that particular construct, and we got it. It also led naturally to:

4. Look for any contradictions

We needed to know whether everyone’s views were approximately unanimous, or were there areas where people disagreed. We did find some, which the course director thought that he could attribute to differences in job type, but again needed to be probed. At this point, I made my only substantive intervention, which was:

5. Look for constructs which aren’t there

It fell to me to point out that in all the interviews, which were about how people had used the course learning on the job, that there was only one construct which had anything to do with cost-effectiveness. This was a sobering thought, although the training manager had to admit that this was rather characteristic of the company as a whole. Excited by the thought of ‘looking for what’s not there’ they discovered that a few people had described parts of the course as ‘not useful now, but maybe in the future’, and that these people had been informally identified as having potential for management.

6. Into a spreadsheet

The training manager then entered all the data into a spreadsheet. Because he had data from several courses over the years, with variations in course design, he could look for the effects of session redesign, different tutors, the jobs people held, etc. As I write, he is still pondering the results.

Summary

We did it in half the time we’d set aside for the analysis stage. The client said that he’d had his objectives met, beyond his expectations. He’ll be doing more of the same in respect of other courses, and several of the interviewees liked Enquire Within so much as a tool for processing information that they’ve asked for copies. Just to summarise what the client had learned, and whether he could have learned it without using Enquire Within:

• the ‘most and least useful sessions in retrospect’ – he could have learned these, but not how they were described unprompted
• the contrast between the ‘least useful session’ as described in retrospect and its high rating on the ‘happiness sheets’
• the constructs which were correlated with the ‘most useful- least useful’ construct, in people’s own language
• the absence of ‘cost-effectiveness’ as one of the criteria by which the people judged the course
• the link between people who described sessions as ‘not useful now but useful in future’ and their perceived potential
• the information he is getting from his spreadsheet about the effect of different variables on course ratings.

Given that he has to ‘sell’ this information internally, and some of it may not be well-received (for example, by the tutor who gives the ‘least preferred’ session), he’s particularly pleased to be able to point out that Enquire Within is pure process, completely content-free. If, for example, he’d designed a questionnaire which included questions about cost-effectiveness, usefulness for the future, etc., he would have had a response because he would have put the question into people’s heads. (He may still want to do that, knowing what he knows now). But he wouldn’t have known that these issues didn’t loom large in many people’s thinking.

Points to Note

This account illustrates many of the different ways you can analyse the data from an Enquire Within session. Simply counting the answers to the element creation questions told us quite a lot. Then we looked for patterns in the way the elements were used, and the correlations between the constructs. A content analysis of the constructs told us a number of things, including ‘looking for what’s not there’ and the people who gave the ‘not useful now, but maybe in the future’ construct. Finally, the client has got some information to interrogate in a spreadsheet. ‘Analysis’ may mean looking at the data from a number of different perspectives, and presenting the results in a way that can’t be argued with – I’ve always thought that if you can’t explain your analysis in two sentences, you stand a good chance of failing to convince an audience. That chance becomes a racing certainty if the audience is hostile.

Hearty congratulations to the two researchers, and hopes that this has given you some ideas.

Valerie.

Other examples of Enquire Within in action

---

Seven Training Related Resources

• Learning Professionals & Trainers - sharing knowledge. A blog for those based in the UK or Europe who are involved in learning and development.
• A Little on Learning.
• An example of using Enquire Within to evaluate a management development course which happened some time ago.
• Training Programme Evaluation (Getting Value for Money) Using Repertory Grid
• Getting Value For Money From Your Career Training.
• Analysis of Training Needs and Performance Counselling
• Curriculum Review. Art Teachers' Views of National Curriculum Art: a repertory grid analysis - Trevor Rayment - Educational Studies, Volume 26, Issue 2 June 2000, pages 165 - 176. There have been calls for a radical revision of National Curriculum Art. A number of prominent academics have argued that its inadequate theoretical base has led to unsatisfactory formalist and normative tendencies in school-based art courses. The paper describes a small-scale survey, using the repertory grid technique, of the opinions and attitudes of experienced heads of art departments in comprehensive schools. Although the results indicate some areas of dissatisfaction, there is a clear overall consensus for acceptance of the mandatory framework. Such views are clearly at odds with most published discourse about National Curriculum Art. It is suggested that the school teachers' attitude to art education is an essentially pragmatic one, which recognises many forms of accountability.

Suggestions

Search for results from the Enquire Within site.

• Download a free evaluation copy of Enquire Within here
• Email this page to a colleague.
• Post this page to: del.icio.us
• Want to know more? Join the experts at the RepGrid User Group.
• Want to keep in touch? Join our mailing list.
• Twitter: Follow us at @EnquireWithin
• Translate this page to your language? Try Google Toolbar.