Using an Outside Evaluator

In many cases with larger grants and complex projects you’ll want or need to hire an outside evaluator.  Sometimes you’ll want to prove the effectiveness of a program and need someone with specialized skills to oversee control groups, do statistical analysis, and so forth.

Reasons for using an outside evaluator

  • The funder requires it: Many federal grants require you to budget 10% or 15% on evaluation, and may require an outside evaluator. Some larger foundations will also require independent evaluation.
  • You don’t have evaluation capacity in-house: If you’re planning a quantitative evaluation with an experimental design and control groups, very likely you won’t have the expertise in your organization to do the instrument design, select control groups and handle statistical analysis.
  • An outsider brings credibility: You may want to use an independent and credentialed outsider because your program is controversial, highly visible, rapidly changing, or because you need to establish (or restore) credibility to your program or organization. An evaluator’s credibility may come because of their independence or their expertise. In most cases you’ll be looking for both.
  • An outsider may be able to get information your program can’t: If your organization provides mandated services where participants could fear losing services or being punished, they may not give their true opinions to your staff. An outside person who is able to prove their independence may gain access or get truer responses than your staff. (Examples could be a probation department or alcohol and drug treatment program).

Pros and cons of using external evaluators

Pros Cons
  • Less work for your organization
  • Evaluators professional expertise
  • Evaluators bring objectivity
  • Evaluation results may have more credibility
  • Less control over the process
  • Staff may have more complete understanding or your program
  • Less opportunity to develop internal evaluation capacity
  • Expensive

How to find an external evaluator

If you plan to use an outside evaluator choose them early. Don’t succumb to the temptation to make the evaluator a “to be determined” person, if it can be avoided. Having the evaluator identified and involved will demonstrate your professionalism.

In addition to being able to use their credentials in your proposal, if all possible involve them in planning your grant proposal. You want to make sure that your program design is structured so that the evaluation can be statistically valid. Sometimes a professional evaluator will write the evaluation section of your proposal on the understanding that you’ll hire them if the grant is funded. If you can’t involve the evaluator in grant planning, get them on board as soon as possible after you’re funded.

  • Academic: Evaluators can often be found in local Universities and medical schools (or hospitals for health care grants). Formal evaluation is similar to research, and in fact most evaluators formal training is in research design and statistics. Until a few years ago it was rare to find classes called “evaluation”. When looking for a university evaluator, you can contact the department that works with your field (like social work, sociology, education or public administration), the Office of Sponsored Research (who oversee faculty research) or see if there’s an evaluation office on campus. The Regional Research Institute at Portland State University often will provide evaluators for nonprofits.
  • Ask the Funder: Some large foundations and government agencies have lists of evaluators with expertise in their field. Or you can call agencies who have been funded previously and ask them who they worked with. If a particular name emerges as having worked with the grantees of a particular funding agency, that firm could add credibility to your proposal.
  • Federal agencies or national trade or advocacy groups: Many federal agencies have technical assistance websites with lists of evaluators.

Choosing an evaluator

Once you’ve identified some potential evaluators, you should call a few of them and get general information. Then you might interview a few that you like to start making decisions. Here are a few things to look for:

  • Do they favor a particular type of evaluation? If an evaluator starts recommending particular evaluation methods before learning what you need to know, or how you plan to use the findings, it may indicate a preference for, or experience with, a particular evaluation design. If that’s not the kind of evaluation you’re looking for, or you haven’t decided, they’re probably not a good match for your project.
  • Do they have an agenda? Check to see if a researcher you’re considering brings their own reasons for doing your evaluation. A professor may be writing a book or conducting long term research, a doctoral student may want to fit your evaluation into a dissertation. Or they may have their own theories that could influence what they expect to find. If their needs mesh with yours, then it’s win-win. If they don’t, you need to ensure that their work will meet your needs, not just theirs.
  • Can they communicate with your staff and board? If the evaluator only describes the process in highly technical jargon, it may make it hard for your staff to work with them — either because of comprehension or intimidation. It may also be an indication that their reports will be full of jargon and theory, and hard to interpret and implement changes.
  • Cultural sensitivity: If you work in a diverse community, the evaluator needs to be acceptable to the community and able to understand or learn the culture in order for the evaluation to gather accurate information.

After screening to get a good match, you will probably identify one or two possible evaluators you’re most comfortable with. You can then have a more detailed interview, with such questions as:

  • How much evaluation have you done, especially in projects similar in size and content to ours?
  • Who would be doing our evaluation — a senior staff, junior staff, a grad student? What are their qualifications?
  • What are your ownership and confidentiality policies? Do you want or intend to publish an article based on our work? If so, how would our organization be identified? Could one of our senior staff be listed as co-author?
  • Please provide us with some names of past clients we can talk to. (And be sure to check references!)

Once you’ve decided on an evaluator, you need to write a contract which should include the following:[1]

  • Scope of Work. This includes defining the organization’s information needs and purpose of the evaluation, defining the Program Theory Model, identifying constraints (time, money, data availability, politics) and how they will be addressed.
  • Evaluation tasks. Who does data entry, transcribing interviews, correlating interview answers, writes up reports and similar tasks?
  • Who owns the data? If articles are published, does it need both parties approval and who is listed as authors?
  • Fees. For a full, formal evaluation in a federal grant, 10-15% of the total is reasonable. For a smaller proposal to a private foundation, 5% may be more likely. To lower costs, the scope of work can be reduced or some work done by the organization.
  • Timeline. This should include major steps in the evaluation process, data collection schedules, reports or other work components and billing deadlines.
  • What will be included in reports: Narrative, charts, literature review, comparison to national or other statistics, number of drafts and who edits them.
  • Definition of responsibilities.  The chart below shows some typical responsibilities.

Evaluator Responsibilities Organization (client) Responsibilities
  • Develop an evaluation plan, in conjunction with staff
  • Train project staff
  • Design or select data collection instruments
  • Implement data collection procedures
  • Establish and oversee confidentiality procedures
  • Write progress and final evaluation reports
  • Attend staff, board and funder meetings
  • Present findings to board and possibly conferences
  • Educate the evaluator about the program or project
  • Provide feedback about data collection tools for appropriateness and relevance
  • Keep evaluator informed of program changes
  • Specify information to be included in report
  • Assist in interpreting evaluation findings
  • Monitor contract and timeline
  • Supervise in-house activities such as data collection and data entry
  • Keep communications open between staff, clients and evaluator

Working with an outside evaluator

The Association of Baltimore Grantmakers has this list of tips for managing your evaluation on their website.

  • Establish a desired communication pattern early and in writing. One person should be the point of contact for the consulting team—that person should facilitate contact with others, provide advice and guidance to the team, and gather additional information and/or resources to support the evaluation when necessary.
  • Maintain communication throughout the project—you’ll need to know how the work is coming along and the consultant needs to be informed about things that arise at the foundation.
  • Develop strategies for monitoring the work of the consultants (not just deliverables). You may want to accompany the evaluation team on select site visits. Are they organized? Do they work well with grantees? Are they doing everything that they said they would and in a manner with which you are comfortable?
  • Seek input from grantees or other sources. Were they comfortable with the evaluator’s approach?

Don’t be a terrible client! Remember that consultants get paid for their time—every call, meeting, and presentation “uses up” valuable contract time. Do what you can to facilitate their work—don’t make it harder on them. If you want or need much more than was anticipated, expect to PAY more for it!

[1] The Evaluation Center at Western Michigan University has a more detailed checklist of things that should be in an evaluation contract, as well as other checklists.