Matching Evaluation to Your Project and Organization

How will you know your grant project succeeded? The answer can be very different for a local community group funded by a local foundation than a social service program applying for a large federal grant. In one case it can be just looking whether you met the program goals for numbers served, in the other it may need control groups and a statistician. This article tells how to know which approach is appropriate, how to do an informal in-house evaluation, when in the grant process to involve a professional evaluator (early) and making sure you have outcomes that can be evaluated.

The need for evaluation of grant projects has increased over the years, as funders and nonprofits have become more sophisticated and focused on results. Before the 1960’s, it was possible to get funded by saying “There are young people living on the streets and really in need of help. We care about them and want to help them”. This wasn’t specific enough for funders to know if their grants were having an impact, so they started to ask “How many?” “There are 2000 homeless youth living on the streets of Portland and we will provide 500 of them with emergency food and 100 with emergency shelter in the next year.” Gradually funders realized this approach wasn’t making lasting changes, so they asking “So what?” and looking for results, which is where we are today. “There are 2000 homeless youth living on the streets of Portland and we will provide 500 of them with emergency food and 100 with emergency shelter in the next year. While they participate in our programs, staff outreach workers will interview them to determine needs and will refer them to counseling, alternative schools, and job training programs. As a result of this program, at least 50 will be reunited with their families or moving to transitional housing and 200 will be in school, job training or employed.” Evaluation is how you collect this information.

Evaluation serves two purposes for your organization and your funder. First, it monitors the success of your project. Second, it provides information for improving your project and overall program. The Kellogg Foundation Evaluation Handbook says “Effective evaluation is not an ‘event’ that occurs at the end of a project, but is an ongoing process which helps decision makers better understand the project; how it is impacting participants, partner agencies and the community; and how it is being influenced/impacted by both internal and external factors.”

As a grantwriter, you will often be involved in the early planning of a project — which may mean there hasn’t been any planning for it done at all. Because the work of the project won’t happen unless the grant is funded, and that could be months away, it may be difficult to get program staff to focus on the eventual evaluation. At the project planning stage, you don’t have to force them to design evaluation criteria. You just need to decide how complex of an evaluation to describe in your proposal, and if further detailed work is needed to develop it.

Remember, when you’re writing a large grant proposal, you’re developing somebody’s job for the next few years. You don’t want to waste their time and your organization’s resources by creating unnecessary work. It is important that you align your project goals and objectives with your evaluation questions, so that the evaluation becomes a part of your project and not a separate set of tasks.

What kind of evaluation you need to do, and who you will involve, will depend on your project and your organization. In your planning you need to ask these questions:

  • Why are you doing this evaluation? To better understand your program? To judge its success or improve it? To satisfy funder requirements? To produce data for agency to use for public relations or in seeking future funding?
  • Who is the audience for your evaluation? Funders? Your board? Management? Program staff? The public or the press? Government officials?

I’ve identified five distinct types of grant evaluation, which can be defined by who needs to be involved.

1. Short term project with staff evaluation. The obvious case is a one-time purchase. You apply for funding to buy a van to take kids on outings. Evaluation consists of “Did you buy the van?” and “Are you using it to take kids on outings?” You could provide the funder with a copy of the bill of sale and vehicle title and a list of outings. You could spice it up with a picture of the van with happy children at the beach, maybe with the funder’s name printed discretely on the van under your logo. This evaluation can be performed by the program or development staff. It doesn’t create any extra work since it fits within existing program activities.

2. Board evaluation: I like to use board evaluation for straightforward projects which have significant impact for your organization. Your board’s job is to oversee the work of your organization, so the regular program reports at board meetings should allow them to evaluate the project and implement improvements. An example would be the marketing and audience development program for Portland Baroque Orchestra in Attachment A. If you’re mailing out brochures and your audience is growing, the board can measure and evaluate the results. Often I will break board evaluation down into 1) Were the objectives accomplished — on time and in budget? (Did we do what we said we would?) and 2) Did we reach the goal, fulfill the need? (Did the project accomplish its purpose?)  Having the evaluation done by the board rather than staff adds some credibility. It also ties the grant project to other board work such as policy and program decisions. This is a good model for local foundations who like to fund direct program activities and who may already be familiar with your organization.

3. Inside Evaluation Team: For more complex projects, you may want to form your own evaluation team of stakeholders in both the project and your organization. Say you’re opening a new health clinic serving a different population than you have previously. This is too big an undertaking to leave to routine staff reports to your board. You might put together a team from the clinic: The clinic manager, a doctor, nurse, receptionist, records management staff. But don’t stop there, you should have at least one patient, someone from a major referral agency, maybe the local public health commission. Finally, how about a board member or your finance manager? If anyone in your agency has evaluation experience, invite them too. The idea is to represent all of the significant groups impacted by your project. The team will meet regularly to assess progress and use it to make project modifications.

In this case, you need to put some real thought into the evaluation design you’ll include in your grant proposal. You might decide on a theory-based outcomes evaluation and use a logic model to outline your project and outcomes. If you’re seeking government funding, the funder might have evaluation criteria you’ll need to include. You’ll have to decide at this stage what data you need to collect to perform an effective evaluation. I suggest looking at some of the books at the end of this article to use in planning your evaluation.

4. Outside Evaluator: In many cases with complex projects you’ll want or need to hire an outside evaluator. Some federal grants require you to budget 10% or 15% on evaluation, and you won’t have evaluation capacity in-house. Sometimes you’ll want to prove the effectiveness of a program and need someone with specialized skills to oversee control groups, do statistical analysis, etc.. You may want to use an independent and credentialed outsider because your program is controversial, highly visible, rapidly changing, or because you need to establish (or restore) credibility to your program or organization.

If you plan to use an outside evaluator choose them early, and if all possible involve them in planning your grant proposal. You want to make sure that your program design is structured so that it can be statistically evaluated. Sometimes a professional evaluator will write the evaluation section of your proposal on the understanding that you’ll hire them if the grant is funded. If you can’t involve the evaluator in grant planning, get them on board as soon as possible after you’re funded.

Evaluators can often be found in local Universities and colleges (or hospitals for health care grants). There are professional evaluation firms, as well as individuals, in most cities. Ask other organizations in your community who they have used and liked working with. Check with the Oregon Program Evaluators Network or the national American Evaluation Association for members in your area.

5. Research: Research grants are different from grants designed to implement programs. With research grants, a scientific question is being explored and the evaluation is itself the purpose. Most research grants are government funded. Many Federal agencies are engaged in supporting research through grants or contracts to universities and profit and nonprofit institutions. This support is offered to encourage research activities that are expected to have a favorable effect on the mission of the government agency. Some agencies have a general mandate to support basic research in scientific areas related to their primary mission or that are in the national interest.

Most research grants are highly technical and require knowledge of the science involved, as well as of research methods. In my experience the researchers themselves, or grantwriters attached to their institutions, develop most research proposals.

Do you need to budget for evaluation and how much? You probably don’t have to include funds for evaluation for numbers 1 and 2. In fact these kinds of projects are often funded by local foundations who don’t want any of their money spent on evaluation. They want to fund projects or direct service, and being told that the project goals were accomplished is as much as they want to know. For number 3, you might want to include evaluation funds in your budget for such things as staff for data collection, entry and analysis; computers and software; interviews or focus groups; and supplies and printing. For numbers 4 and 5, you definitely want to budget for evaluation. With some government grants, there’s a defined percentage (often 10 or 15%) required to be spent on evaluation.

ONGOING EVALUATION Some agencies have ongoing evaluation programs, which can provide the framework and information for planning your grant evaluation.

Organizational Commitment: A national organization I work with, the National Indian Child Welfare Association, evaluates all of its programs including services, products and events. This helps them guarantee quality services in support of their mission and assures their diverse constituencies and funders that their programs are effective and produce results. While unusual in nonprofits, this commitment has helped NICWA build and maintain a national reputation as a leader in their field.

Quality Assurance (QA): In health care, mental health, substance abuse treatment, and related fields quality assurance staff or committees are often required for licensing or accreditation. They review program practices to make sure services meet required standards. Quality Assurance may be included in a grant as part of contract compliance. If your agency has a quality assurance program, the information it is already gathering will be invaluable for developing a program evaluation — or it may be all you need.

Continuous Quality Improvement (CQI): CQI was developed by Japanese industry, based on the work of W. Edward Deming in the 1940’s. Instead of collecting information centrally and periodically implementing major program changes, CQI involves the line staff and managers in constantly identifying problems or opportunities, and implementing them at the operating level without having to go to top management. Many nonprofits have started using CQI, and if your organization has a CQI program it can be the basis of your evaluation section. Sometimes Quality Assurance programs incorporate CQI.

Resources for Evaluation:

W.K. Kellogg Evaluation Handbook, available free for downloading as an Adobe Acrobat file or ordering a paper copy. This 117-page volume gives both theory and practice of evaluation, from one of America’s largest and most evaluation-oriented foundations.

Outcomes for Success! Judith Clegg & Jane Reissman, Organizational Research Services (Seattle, Washington). How to develop your own outcome-based program evaluation, including using a logic model.

Grant Winner’s Toolkit: Project Evaluation and Evaluation, James A. Quick & Cheryl C. New, John Wiley & Sons, New York, 2000. A good “how to” overview, with lots of charts and forms for data collection.