The Center for Creative Community

Things the Evaluator Needs to Evaluate

For Planning @

 @ -- permission for use-with-acknowledgment

By Ivan Scheier

Please note: These questions are all inter-related and should be considered together, especially the ones with the same alphabetic designation. Also, your decisions on options are usually matters of emphasis, rather than all-or-none.

Al PURPOSE. Why do we want to do this evaluation? Possibilities include:

--Simple curiosity. Just to see what's happening
--Raise awareness of our program or organization
--Justification, win respect, appreciation, priority, money, resources--or hold onto what we have.
--Improve the program
--Other?

A2 INTENDED AUDIENCE. Who do we want to reach, inform, impact? Possibilities include:

--People who influence the program at a policy level(Bosses, boards, etc,)
--People who implement the program on a daily hands-on basis

--(Paid)Staff
--Volunteers
--Both --- the team

--The people or groups we serve--clients, consumers, etc
--People who can choose to provide more or less material support for the program -funders, donors, etc.
--Academic and/or professional people, in our field
--The community-at-large, the public
--Special segments of the community, e.g.

--Media
--Schools and colleges
--Political Leaders
--Religious Groups
--Youth
--Members (May or may not also be volunteers or donors)
--Others?

BI Assurance of Impact. 1: DO WE HAVE THE NECESSARY RESOURCES FOR THIS EVALUATION?

--Money or in-kind equivalent for:

--Paper, supplies
--Printing
--Postage
--Telephone
--Computer equipment and time
--Clerical, tabulation time
--Expertise, specialist(see below)

--People: Paid Staff or Volunteers (including expert volunteers)
--Time. Evaluation studies typically take twice as long s your most conservative estimate.

B2 Assurance of Impact 2: HOW CAN WE PREVENT THIS STUDY FROM GATHERING DUST ON THE SHELF (Like Most Evaluation Studies) ?

--Meaningful purpose (Al) well-focussed on relevant audience A2)
--Concise, well-produced reporting, multi-channel(includes graphics,video)
--Follow-up, FOLLOW-UP, FOLLOW-UP!!!!
--Participative buy-in of intended audience, in design of study as well as sources of information.

Cl Sources of Information, 1: WHO DO WE ASK OR MEASURE?

Your choices are essentially the same as the choices in intended audience (A2), especially whoever has either reason to know something significant bearing on the purpose of the study (Al) and/or has an important stake in it.

C2 Sources of Information, 2: APPROPRIATE BALANCE BETWEEN "VARIABLES"

--Balance between numerical and qualitative (what people think about it, their impressions and feelings)
--Balance between Input and Output emphasis. Examples of input would be number of hours contributed by volunteers, and compliance with effective program procedures. Output measures concentrate on what was accomplished, the consequences or results.
--Balance between descriptive and experimental (difficult, tends to diminish meaningfulness in favor of laboratory-like conditions)
--Within descriptive can still be comparative (without being experimental). But at all costs avoid comparisons of volunteers VERSUS staff. The appropriate comparison, usually, is between staff alone, and staff "amplified" by volunteers (the solo vs. the team).
--Other balances to be aware of?

--------

Postscripts

This is somewhat sensitive but the issue must be raised, because the acceptance and usage of evaluation studies. How prepared are you, and your intended audience, to accept results which are ambiguous, surprising, or downright opposite to what you fondly expected? It happens. All the time. So try and be ready for it.

Finally, is your intended audience and/or study participant group, threatened by the word "evaluation"? Many are, so you might want to consider alternative terms such as "feedback", assessment" (might be worse), or "operations analysis".

Good Luck with it.