Post on 23-Jun-2020
HV 745 .Q44 T373 1998 V. 3
Program Evaluation for Organizations under CAPC
(Community Action Program for Children)
©
PRESENTATION OF
EVALUATION GUIDES
Centre de recherche sur les services communautaires
Faculté des sciences sociales
DIVERSITÉ WAL
Association des centres jeunesse du Québec
SANTÉCOM
(
PROGRAM EVALUATION FOR ORGANIZATIONS UNDER THE COMMUNITY ACTION PROGRAM FOR
CHILDREN (CAPC)
PRESENTATION OF EVALUATION GUIDES
MNM M\MI N SANTÉ PUBLIQUE DU QUÉBEC CENTRE DE DOCUMENTATION
MONTRÉAL
Patricia Dumas Hector Ouellet
André Beaudoin
with Nicole Caron
MARCH 1998
Secretarial Services: Jocelyne Gallant
English Translation: Translation Bureau
Public Works and Government Services Canada
The masculine used herein refers to both genders.
ISBN: 2-89497-017-X
These documents were produced as part of a contract with CLSC Les Blés d'Or evaluating Health Canada's Community Action Program for Children (CAPC).
The viewpoints herein are those of the authors and do not necessarily reflect the official policy of Health Canada or the Province of Quebec.
i Table of contents
Foreword i i
Introduction 1
Chapter 1 Understanding the Evaluation Process 4 Section 1: Synopsis 4 Section 2: Contents 6 Section 3: Highlights 20 Summary 25
Chapter 2 Participant-focused Evaluation 26 Section 1: Synopsis 26 Section 2: Contents 28 Section 3: Highlights 46 Summary... : 50
Chapter 3 A Guide to Evaluation for Community Groups 51 Section 1: Synopsis 51 Section 2: Contents 53 Section 3: Highlights 63 Summary 64
Conclusion 65
ii Foreword
Aspeaker who was to talk on the topic of p lanning and
who wanted to lead the listeners to have realistic
expectations about the potential of the planned action
began his presentation with the following:
"When Christopher Columbus discovered America, he
didn't know where he was going. He didn't know how to
get there. He didn't know how long the trip would take or
what resources would be needed along the way. Once he
landed, he didn't know where he was. And yet he
discovered America. "
To which someone interested in the field of eva lua t ion
could have responded:
"At least Columbus had an objective: finding India.
Having an objective enabled Columbus tô decide to
attempt the trip and to pull together the required
resources. The few navigational instruments (in
evaluation terms, these are referred to as evaluation tools)
Program Evaluation for Organizations Presentation of Evaluation Guides
I | iii
he had indicated that he was heading westward and o n
course. Despite their primitiveness, his instruments
allowed him to calculate the time lapsed since departure
and provided a rough estimation of the distance traveled.
| Furthermore, these instruments (in this case, his own
observations of the results, combined with a comparison
between what he expected to discover to what he actually
saw) indicated to him that either he hadn't reached India
or that his description of India was incorrect."
This is what evaluation is all about: a means for
stakeholders and practitioners to guide their actions.
The Community Action Program for Children (CAPC)
evaluation team set a number of objectives for itself,
including the publication of manuals to provide groups
working under the CAPC framework with support that
would correspond as closely as possible to their needs and
real potential for carrying out evaluation activities. Three
Program Evaluation for Organizations Presentation of Evaluation Guides
iv manuals have been produced.1 The first presents and
clarifies the main ideas, topics, and notions relating to the
concept of evaluation. This is the Introductory Manual.
The second manual aims at providing tools to
organizations that want to evaluate their performance in
order to enhance the quality of the programs that they
deliver. It provides concrete examples that illustrate
different ways of using evaluation methods and tech-
niques. It also shows how to handle a certain number of
ethical and methodological challenges often encountered
in carrying out evaluations. The title of the second manual
is Evaluation Tools for Enhancing Program Quality.
The third manual, entitled Presentation of Evaluation
Guides, offers a fairly detailed presentation of the contents
of certain "evaluation guides" that have been developed
These three manuals (in English and French versions) may be obtained from the Centre de recherche sur les services communautaires (community services research centre), whose address is provided on the back cover.
Program Evaluation for Organizations Presentation of Evaluation Guides
and are available. We have focused on highlighting the
aspects of these guides that seem the most relevant or
useful to organizations.
We would like to express our appreciation to the
individuals who, in one way or another, have helped in
preparing these manuals. We are particularly grateful to
the members of the CAPC advisory committee for program
evaluation, whose names and affiliations are listed below.
André Beaudoin Lyne Champoux Martine Cinq-Mars Richard Cloutier Danielle Couture/ Lucie Lafrance Anne Dubé Richard Foy Michel Gaussiran Florence Isabelle
Gisèle Laramée Hector Ouellet
CRSC, Université Laval (Québec) CRSC, Université Laval (Québec) Regroupements des projets PACE des Laurentides (Montreal) CRSC, Université Laval (Québec) Mauricie/Bois-Francs Regional Board (Trois-Rivières) La Débrouille (Rimouski) Le Pignon Bleu (Québec) CAPC (Montreal) CLSC Seigneurie Beauharnois (Valleyfield) ENAP (Québec) CRSC, Université Laval (Québec)
Program Evaluation for Organizations Presentation of Evaluation Guides
vi We sincerely hope that these manuals will be of use and that the organizations for which they have been published will find them of value in their efforts to develop services offered to children and parents in their communities.
Program Evaluation for Organizations Presentation of Evaluation Guides
1 Introduction
his manual aims specifically at providing a relatively
detailed look at the contents of three evaluation
guides that have been developed by authors with a
purpose similar to ours: helping acquaint community
groups with the evaluation process.
Each of these guides is presented separately in its own
chapter comprising three sections: synopsis, content
description, and highlights. Section 1 (synopsis) offers a
brief outline of the guide, including ordering information.
Section 2 (contents) provides a short but as accurate as
possible summary of the guide's contents. Section 3
(highlights) points out what we consider to be the most
important elements in the guide and offers a commentary
on the author(s) positions. We also draw attention to what
we deem significant shortcomings, when appropriate.
Our purpose in producing this manual is to provide
descriptions that are detailed enough to give a clear picture
Program Evaluation for Organizations t Presentation of Evaluation Guides
2 of the contents of these guides so that the reader can decide
whether or not to purchase them.
Chapter 1, entitled "Understanding the Evaluation
Process," presents How about...Evaluation!, a 1993 manual
written by Jacqueline D. Holt for First Nations and Inuit
communities.
Chapter 2 focuses on participant-focused evaluation based
on Keeping On Track - An Evaluation Guide for
Community Groups, produced by Diana Ellis, Gayla Reid,
and Jan Barseley for the Women's Research Centre. This
guide was written for community groups providing
services to victims of AIDS.
Chapter 3 presents a guide that we consider particularly
useful in carrying out an evaluation audit of an
organization. It was written by Christine Daniel in 1992 for
the Centre de formation populaire de Montréal
Program Evaluation for Organizations t Presentation of Evaluation Guides
3 To summarize, each of these guides is presented as follows:
- a brief outline of the guide, including ordering
information;
- a description of the guide's contents;
- the most important elements in the guide.
Program Evaluation for Organizations t Presentation of Evaluation Guides
4
Chapter 1 Understanding the Evaluation Process1
Section 1 : Synopsis
For Whom?
This manual was written for First Nations and Inuit
communities in Canada who are responsible for
implementing mental-health and child-development
programs and projects in their communities.
By Whom?
The guide was sponsored by Mental Health Advisory
Services, under the Department of National Health and
Welfare's Medical Services Branch and written by
Jacqueline D. Holt.
1 This chapter deals with How about... Evaluation! A Handbook about Project Self Evaluation for First Nations and Inuit Communities by Jacqueline D. Holt (January 1993).
Program Evaluation for Organizations t Presentation of Evaluation Guides
5 For What Purpose?
This document aims at describing the program
evaluation process and at explaining its uses and
significance.
Structure
This 56-page document contains 11 short chapters and a
bibliography. The first four chapters introduce the topic
of evaluation; the remainder deal with selecting the
person to head up the evaluation and with questions
relating to gathering and processing information.
Where to Order Health Canada Health Promotion and Social Development Office Complexe Guy-Favreau 200 René-Lévesque Boulevard West Montréal, QUÉBEC H2Z1X4 Telephone: (514) 283-6533 Fax: (514) 283-3309
Program Evaluation for Organizations t Presentation of Evaluation Guides
6
Section 2 : Contents
Some Definitions
This guide starts off with a glossary of some frequently
used evaluation terms (Chapter 1; pages 1 to 3). Four
terms of special interest to us in this glossary are "hard
data/' "soft data," "open questions/' and "closed
questions."
"Hard data" are described as being "measurable and
tangible data, usually numbers," while "soft data" are
defined as "opinion and attitude about something."
They are said to be qualitative. An "open question" is
defined as one "that can be answered with description
and opinion, using any words you want." A "closed
question" is described as one "that can be answered
with only one response, like yes or no."
The guide continues with an overview of evaluation
(Chapter 2; pages 5 to 10), starting with the author's
Program Evaluation for Organizations t Presentation of Evaluation Guides
7 perspective of evaluation, which she describes as "as
way of measuring if a project is doing what it says it will
do."
Why We Do Evaluations
This chapter discusses four reasons for carrying out
evaluation. These reasons are described in the
following terms:
1- to help you understand how your project is working; 2- to show your sponsors what you are doing; 3- to show other communities what has worked for
you; 4- to show your staff what they are doing.
The Role of Evaluation
Holt then describes where evaluation fits in the
operating cycle of a program, stating that "evaluation,
ideally, is an ongoing activity." In other words, it
should be initiated when the program is being planned,
continue throughout the life of the program, and then
shed light on how to make improvements with respect
Program Evaluation for Organizations t Presentation of Evaluation Guides
8 to planning and implementation or to be a source of
ideas for new programs and projects.
Criteria for Quality in Evaluation
This chapter closes by listing the required criteria to
ensure the quality of any evaluation:
- The evaluation must be useful to the people who request the information and to the people in the program.
- It must be practical to implement. - It must be conducted in an ethical manner. - It must be technically accurate.
The Question of Objectives
The author next tackles the topic of program objectives
(Chapter 3; pages 11 to 14). Objectives are presented as
the main component of an evaluation since they
describe what the program is supposed to accomplish.
Objectives should be formulated according to the
following criteria.
Program Evaluation for Organizations t Presentation of Evaluation Guides
- They must specifically state the expected results. - They must state the expected results in measurable
terms. - They must identify when the results will happen.
This chapter also gives three examples of how to
formulate objectives as outlined below.
- A program entitled "Life Skills for Teenagers" could
have an objective formulated as "the involvement
of teenager girls between the age of 15 and 19 in the
life skills program will increase by 50% in the next
two years."
- A program entitled "Aftercare for Psychiatric
Patients" could have an objective formulated as "re-
admissions to psychiatric institutions will drop by
30% in the next year."
- A program entitled "Alcohol Abuse" could have an
objective formulated as "establish an alcohol abuse
counselling program to handle 50 people annually
in the next three years."
Program Evaluation for Organizations Presentation of Evaluation Guides
10 Types of Evaluation
In Chapter 4 (pages 15 to 20), the author discusses three
types of program evaluation that can be used:
- needs assessment;
- process evaluation; and
- impact evaluation2.
This chapter points out that selecting one type of
evaluation over another depends on the stage of life of
the program. For example, needs assessment would be
appropriate if the program were in planning, whereas
process evaluation would apply to an ongoing program.
Impact evaluation is appropriate for focussing on the
products or results of the program.
Holt briefly describes how each of these types of
evaluation is useful. For example, she indicates that
needs assessment helps get an idea of the kinds of
problems in the community; which segments of the
2 This notion seems to cover what other authors call "impact" or "effects".
Program Evaluation for Organizations t Presentation of Evaluation Guides
11 population have the most urgent needs; and how the
needs can be met. With respect to process evaluation,
Holt describes it as the type of evaluation that shows
how the program could be improved; how the program
could help a greater number of people; what kind of
human resources are needed; and how program costs
can be reduced. Impact evaluation is considered as the
one of the three that can be used to determine if the
program was effectively implemented; what impact it
had on participants; if it should be continued; how
much it cost; and how it compares with similar
programs.
Evaluation Approaches
The last part of this chapter discusses two evaluation
approaches: quantitative and qualitative.
Holt defines the quantitative approach as trying "to
determine the cause and effect relationships in a pro-
gram." In her opinion, this approach uses "measu-
rements, numbers, and statistics to compare program
Program Evaluation for Organizations t Presentation of Evaluation Guides
12 results." The information obtained can therefore be
considered hard data.
As for the qualitative approach, Holt views it as
examining "the qualities of a program." It serves "to try
to understand the meaning of a program and its
outcomes from the view of people." The information
that is found is considered soft data.
Holt claims that the qualitative approach is most
appropriate for evaluating mental-health and child-
development programs because:
- it uses a number of different methods to gather information;
- it looks at the program as a whole; - it calls on program participants, staff, and commu-
nity members as sources of information; - it is dynamic, interactive, and exciting; - it leaves room for creativity; and - it doesn't require a rigid design.
Program Evaluation for Organizations t Presentation of Evaluation Guides
13 Planning an Evaluation
• Clarify program objectives
• Select the type of evaluation
• Select an evaluator
Chapter 4 winds up with a reference to a checklist to
help in evaluation planning. The list starts off with the
notions dealt with above, indicating that the planning
phase must begin by clarifying program objectives and
selecting the appropriate type of evaluation.
In Chapter 5 (pages 21 to 24), Holt gives her viewpoint
on how to choose an evaluator. Although the
concerned group or organization is responsible for the
evaluation, they may delegate responsibility to others,
in particular, peers and consultants. The choice depends
on the stage of the program and the type of evaluation.
To illustrate, needs assessment, which is carried out
during program planning, could be performed by
someone who is thoroughly familiar with the
community.
Program Evaluation for Organizations t Presentation of Evaluation Guides
14 Process evaluation, which occurs while the program is
ongoing, could be conducted by someone thoroughly
familiar with the program, such as a program manager,
a staff member, a participant, a community elder, or
even an outside consultant acting as an advisor.
Holt states that impact evaluation should be carried out
by an individual that has no involvement whatsoever
in the program. The evaluator could be from another
program, a member of another community, and so on.
In Chapter 6 (pages 27 to 36), Holt describes the
information collection plan, which she suggests should
be based on the type of information sought and the
resources available.
• Determine the Type of Information Required
Holt points to three types of information:
- input (resources);
- processes (operations);
- outcomes (impact).
Program Evaluation for Organizations t Presentation of Evaluation Guides
15 • Determine the Sources of Information Available
Holt describes information sources as belonging to
three main types:
- files, documents, and information systems; - interviews with participants, their families, and
staff; - observations made during the program.
This chapter ends with a table (information sources
chart) designed to help in developing the information
collection plan. The table is divided into three columns
for the sources of information; the rows are for the
types of information. Holt provides questions in each
table cell that a group should ask itself when carrying
out an evaluation. She also suggests using the table to
organize and analyze data.
Carrying Out the Evaluation
• Collecting Information
Holt deals with collecting information in Chapter 7
(pages 37 to 44), where she describes how to gather
Program Evaluation for Organizations t Presentation of Evaluation Guides
16 information from each of the sources and raises other
important considerations. For example, in collecting
information from information sources and files, Holt
recommends finding out what kind of information has
been recorded and how often, then refer to the type of
information required in order to determine if the data
might be pertinent.
As for interview material, Holt indicates that it can be
obtained in a variety of ways, such as face-to-face
interviews, telephone interviews, or self-reporting
questionnaires. She points out that the questions
should be simple and touch on a single subject at a
time. Holt also states that a test should be carried out
before actually using a written questionnaire.
Holt's views on observations is that they should be
used once the files have been examined and the
interviews conducted because the information can be
used as a foundation for observations. She feels that it
is not necessary to draw up a checklist or interview
Program Evaluation for Organizations t Presentation of Evaluation Guides
guide. Holt thinks that observing, feeling, and listening
to what is going on is more important. On the other
hand, she emphasizes that notes should be taken as
soon as the observations have been made.
• Interpreting the Information
In Chapter 8 (pages 45 to 50), Holt deals with
interpreting information. She suggests basing the
interpretation on the information sources chart o n
pages 32-33 and working along step-by-step. The process
begins by examining the information collected from
each of the sources and asking what it reveals about the
inputs, processes, or outcomes. She recommends
preparing a description of what was discovered for each
type of information. For example, with respect to
inputs, you might ask if the program is being run by
experienced, qualified staff in a suitable setting. If so,
why? If not, why not? Analysis of outcomes could
respond to questions such as: Do participants benefit
from the program? If so, why? If not, why not?
Program Evaluation for Organizations Presentation of Evaluation Guides
18 According to Holt, the most important step is giving
meaning to all the information gathered, which means
drawing conclusions. She recommends using the
information sources chart to formulate conclusions for
each of the sections by asking yourself if the numbers
point to a trend, if there is a pattern or theme in
activities, and so on. Holt emphasizes looking for a
logical chain of evidence. For example, do the
documents support what is happening in the program?
• Writing the Report
Holt then proposes writing down your ideas and
discussing them with the manager and program staff.
• Discussing the Results
Chapter 9 (pages 51 and 52) provides an overview of the
process with a checklist itemizing the main tasks to be
carried out during the evaluation.
Program Evaluation for Organizations t Presentation of Evaluation Guides
19 A Word on Confidentiality
The guide ends with two notes: one is an evaluation
checklist; the other deals briefly with the topic of
confidentiality.
Other References
In Chapter 11, Holt refers the reader to The Evaluator's
Handbook by Joan Herman, Lynn Lyons Morris, and
Carol Fitz-Gibbon for further reading and reminds us
that Sage Publications puts out a number of books and
handbooks on the topic.
Program Evaluation for Organizations t Presentation of Evaluation Guides
20
Section 3 : Highlights
Originality
This guide familiarizes the reader with the overall
evaluation process. Its most original feature, however,
doesn't lie with its description of the process itself, but
rather in its approach to accountability for the
evaluation. Although her design is not very elaborate,
it does have a certain number of features that sets it
apart from conventional designs.
To begin with, Holt clearly states that the organization
is responsible for initiating the evaluation of its
programs and has the freedom to designate whoever it
wishes to implement the evaluation. From this
viewpoint, the evaluation vests fully in the
organization. It is not imposed by an outside agency, but
emerges as an activity that promotes "food for thought"
for the organization regarding the orientation and
improvement of its programs.
Program Evaluation for Organizations t Presentation of Evaluation Guides
21 • Selecting an Evaluator
Unlike what you might expect, the choice of an
evaluator is not limited to experts or specialists in the
field. Holt states that the candidate could come from
individuals involved in the program, including
participants, staff members, and managers as well as
peers from other programs. In this regard, a peer is
someone that carries out similar functions in a similar
program.
The idea of turning to peers can be very attractive and
should be explored. As Holt claims, it provides an
opportunity for someone from outside the program but
who is very familiar with its problems to critically
examine it.
Recourse to peers or to individuals directly involved in
the program could also be paired with using experts,
referred to as consultants, because they play the relative
discrete role of advisor. While the use of the term
Program Evaluation for Organizations t Presentation of Evaluation Guides
22 consultant isn't new, it seems particularly apt to us in
this context.
We are aware that, for reasons of reliability, validity,
and objectivity, experts usually tend to control the
evaluation, often nearly absolutely. In Holt's context,
having the expert as advisor to the individual actually
carrying out the evaluation indicates that the expert's
role is not controlling the evaluation, but rather acting
as a partner who helps maintain the conditions
required for structuring a valid, productive evaluation.
• On the other hand, we should point out some points
that we consider debatable. First of all, unlike Holt, we
feel that the qualitative and quantitative approaches
must be seen as complementary.
• We also feel that her planning and implementation
phases are inadequately documented to really be of help
to a novicè. So, while he may see that objectives are the
main component of an evaluation, he won't find much
Program Evaluation for Organizations t Presentation of Evaluation Guides
23 help on how they should be formulated. Furthermore,
there's scant information on how to use objectives in
subsequent steps, such as data interpretation.
• Along the same lines, there is little information
regarding the choosing and designing data collection
techniques.
Moreover, there seems to be some confusion
concerning the terms "sources of information" and
"gathering tools." It would be useful to remind the
reader that there are three main sources of information:
documents, observation by the researcher, and
information provided by the subjects. There are,
however, many information gathering tools. To
illustrate, information from subjects can be obtained by
informal interview, guided interview, questionnaire,
and so on.
Program Evaluation for Organizations t Presentation of Evaluation Guides
24 • In concluding these remarks, we should point out that
the explanations on how to analyse the information are
too sparse to actually guide anyone through the process.
Our advice to the reader, then, is to turn to other
resources to help in this phase of the evaluation
process.
Program Evaluation for Organizations t Presentation of Evaluation Guides
25
Summary
Despite Holt's limited treatment of certain evaluation
phases, this handbook offers a new way of looking at the
process that puts the organization at the centre of the
evaluation process, which is certainly an original
contribution.
Program Evaluation for Organizations t Presentation of Evaluation Guides
26 Chapter 2
Participant-focused Evaluation3
Section 1 : Synopsis
For Whom?
This manual was written for non-profit community
groups with a social-change orientation.
By Whom?
The guide was supported by funding from Health and
Welfare Canada under the National Welfare Grants
Program and the Federal Centre for AIDS.
It's an adaptation of an older handbook by Jan Barnsley,
Diana Ellis, and Helga Jacobson (The Women's
Research Centre, Vancouver, BC).
3 This chapter presents Keeping On Track - An Evaluation Guide for Community Groups by Diana Ellis, Gayla Reid, and Jan Barnsley (The Women's Research Centre, 1990).
Program Evaluation for Organizations Presentation of Evaluation Guides
27 For What Purpose?
The purpose of this guide is to present an approach to
evaluation that will make it possible for groups to face
evaluation with information and confidence, as
respected participants in the evaluation process. The
authors refer to their approach as being participant
focused, meaning that it's the group that decides what
questions to ask, what criteria to choose, and so on.
Structure
The guide breaks down into six chapters and three
appendices, covering a total of 79 pages. It discusses how
evaluation can be useful, how to design an evaluation,
and how to select an approach, in addition to
presenting data collection, analysis, and use.
Where to Order
Women's Research Centre 101-2245 West Broadway Vancouver, BC Canada V6K2E4
Program Evaluation for Organizations t Presentation of Evaluation Guides
28
Section 2 : Contents
The Usefulness of Evaluation
This guide starts off by presenting why the authors feel
that evaluation is useful.
- It helps to see what's really involved in the work the
organization does.
- It provides a chance to step back from the day-to-day
work and think about what you're doing and why.
- It helps you figure out how well you're doing what
you set out to do.
- It helps you understand why you're having a
problem and how to solve it or live with it.
- It shows how you might improve your program.
- It can give you ideas and information for planning
future work.
The authors state that evaluation must be adapted to
the group's needs and interests if it is to be useful. This
is why they propose an approach referred to as
participant-focused evaluation.
Program Evaluation for Organizations t Presentation of Evaluation Guides
29 Participant-focused Evaluation
This approach is based on the belief that evaluation
must meet the conditions listed below in order to suit
the needs and interests of the community group.
- It must take into account the nature of the group,
which means an organization that generally
operates on a shoestring budget, has flexible
structures, and an orientation that must constantly
shift to meet the needs of the target population.
- Evaluation must be based on questions that the
group asks itself about its work.
- It must use criteria that the group decides on with
respect to the work to be evaluated.
- It must involve the group in analysing the
information collected throughout the process so that
the group can develop the means to improve its
work during the evaluation.
Program Evaluation for Organizations t Presentation of Evaluation Guides
30 After setting the stage in the first chapter of the guide,
the authors move on to illustrating how to carry out
the various phases involved in participant-focused
evaluation.
Preparing for Evaluation
Chapter 2 (pages 9 to 13) describes how to apply this
approach to the first phase of evaluation, which is
preparation. This is the phase that allows the group to
determine its evaluation needs and expectations, and to
make sure the evaluation is useful to the group.
• Make a List of Questions
The first step is for the group to come up with
questions it wants answered about its own work, such
as whether you are really doing what you set out to do
or you may want to know which parts of your program
are the most useful and why.
Program Evaluation for Organizations t Presentation of Evaluation Guides
31 • Getting Answers to Your Questions
The next step in preparation is to determine how the
group is going to get the answers to its questions. This
begins by examining the ways that the group keeps
track of information (logbooks, minutes, forms, and so
on) and then determining how it can go about getting
the additional information needed.
• Dealing with the Information
The third step is to decide if you need help in collecting
and analysing the information required for the
evaluation. You may have the necessary resources,
want to add someone from inside or outside the group,
or decide to collect less data.
• Decide Who Should Be Involved
The fourth step in preparing for evaluation is to
determine w h o should be involved. In this regard, the
authors insist that there should be a feedback
mechanism to ensure that anyone who should have
input does have input. The authors suggest that one
Program Evaluation for Organizations t Presentation of Evaluation Guides
32 solution can be forming an evaluation committee
mandated to meet regularly with the evaluator:
to raise and discuss issues related to the preparation
and use of evaluation tools for program activities;
- to monitor the entire evaluation process and make any
necessary modifications;
- to review the evaluation data and conclusions, and
discuss what they mean for the program's work;
- to act as liaison between the group and the evaluator.
• Draft a Preliminary Budget
The last step in preparation consists in estimating how
much the evaluation will cost. The authors stress that
you should take pains to estimate the costs for all the
tasks that might be involved in the evaluation,
including staff time, overtime related to data collection
and analysis, photocopying, stationery, resource
materials, and the like. The budget estimate can be used
to apply for funding and in negotiating with outside
evaluators, if necessary.
Program Evaluation for Organizations t Presentation of Evaluation Guides
33 Chapter 2 ends with a table that summarizes means for
organizing the preparation phase.
Negotiating with the Funder
Chapter 3 (pages 14 to 19) looks at issues pertaining to
negotiating the evaluation method with the funder
under the title "Negotiating for an Evaluation
Method."
• Identify What the Funder Wants and Needs
The authors start off the chapter with a reminder that
most funding agencies expect to receive a report that
provides an accounting of what happened to the funds.
Since funders and groups don't necessarily have the
same needs and expectations of the evaluation and
therefore won't necessarily view the evaluation
procedures in the same way, it is very important that
the group identify what the funder expects from the
evaluation. To simplify, the authors provide a list of
questions concerning why the funder wants the
Program Evaluation for Organizations t Presentation of Evaluation Guides
34 evaluation, what the funder wants to evaluate, the
criteria or indicators that the evaluator deems
important, the use the funder will make of the
evaluation results, and so on.
• Comparing the Funder's Needs and Expectations to
those of the Group
Once you have a clear idea of the funder's needs and
expectations, you should compare them with your own
to make sure that they are compatible. If there are
differences, the authors suggest that you write down
how you see things and send it to the funder's
representative, asking for clarification, if necessary.
• . Get It in Writing
The authors emphasize that the best way to protect
everyone's interests is to put all agreements into
writing for reference during the evaluation.
Program Evaluation for Organizations t Presentation of Evaluation Guides
35 • Planning the Evaluation
In Chapter 4 (pages 20 to 40), the authors offer a fairly
detailed presentation of the second phase in the
evaluation process: developing an evaluation design.
Their approach has seven steps.
- Clarify the Evaluation Focus and Assumptions of
Success
The first step in developing an evaluation design is
to clarify the evaluation focus and assumptions of
success. Evaluation focus refers to the program
targets and objectives that the evaluation will focus
on. For example, is the aim to change existing
legislation or policy, create awareness in a segment
of the population, change attitudes or values, or
improve skills?
The authors recommend considering questions such
as those below in determining the assumptions of
success for evaluating your program. What does
success look like to you? What would different
Program Evaluation for Organizations t Presentation of Evaluation Guides
36 degrees of success look like? Is it success only if the
desired changes are brought about?
- Review the Context
The second step consists in reviewing the
geographical, social, and economic context or
environment of the work. Given that the context
plays a major role in the program's success or
failure, the authors feel it important to draw up a
list of the conditions specific to each situation. To
make this process easier, they ask you to consider a
certain number of general factors, which include
accessibility issues, cost of existing services, media
access, and the like.
- Finalize Evaluation Goals and Objectives
Once the group has clarified the focus, assumptions
for success, and the context of the work, it has all the
information needed to finalize the evaluation goals
and objectives. This is the third step in developing
the evaluation design.
Program Evaluation for Organizations t Presentation of Evaluation Guides
37 The authors stress that program goals and objectives
differ from evaluation goals and objectives. They
define program goals and objectives as specific,
measurable targets of what is to be accomplished by a
given point in time, while evaluation goals and
objectives are presented as being directly related to
the ends targeted by the evaluation, that is,
determining, describing, and measuring the
program and its outcomes. The authors illustrate
this with a series of examples.
One of the examples provided in this regard deals
with an advocacy group working on discrimination
against people with AIDS/HIV. One of the goals of
this program could be to develop advocacy skills of
group members. The corresponding objective could
be to produce a how-to kit on advocacy skills. The
evaluation goal could be to examine the
effectiveness and impact of the program i n
developing advocacy skills in your community,
while the objectives could be to provide feedback to
Program Evaluation for Organizations t Presentation of Evaluation Guides
38 program workers about the program's effectiveness
and to assess how well the advocacy kit suited
members' needs.
- Evaluation Instruments
Choosing the evaluation instruments is the fourth
step. The authors remind us that the selection of
evaluation instruments is influenced by the type of
information needed for evaluation, the skills
required for collecting and analysing the
information, and the time and resources available to
the group. They then proceed to identify some tools
(logbooks, journals, files, questionnaires, interviews,
and the like) and how to use them.
- Developing a Work Plan
- Deciding What to Do with the Evaluation Results
- Reaching Agreement on Evaluation Results
The work continues with developing a work plan, a
discussion on what to do with the evaluation
Program Evaluation for Organizations t Presentation of Evaluation Guides
39 results, and the need to get the group's approval on
the design. The authors suggest using flip-chart
paper to develop a work plan table. The table would
have four columns with the following titles:
objectives, tasks, how long will it take?, and dates for
accomplishment.
Once you've gotten through these steps, you can
move on to using the evaluation plan.
Carrying Out the Evaluation
• Properly Using Evaluation Instruments
Chapter 5 (pages 41 to 47) discusses using the
evaluation plan, starting with a few of the evaluator's
responsibilities during data collection. They point to
providing any necessary training on how to use the
instruments and ensuring that the tools are being used
properly.
• Compiling and Analysing Data
Program Evaluation for Organizations t Presentation of Evaluation Guides
40 • Discussing Conclusions
The authors provide some direction on how to go
about compiling and analysing the data once it has been
collected. They suggest photocopying and coding (or
numbering) all the originals in order to be able to trace
who said what. Then, read the data, looking for
patterns in the responses, separating data by themes or
groups of questions, and thinking about any possible
conclusions. This should wind up with a discussion of
the conclusions involving everyone concerned.
This chapter also emphasizes the importance of
analysing and discussing the data throughout the entire
evaluation process -not just at the end. The authors
also state that the people doing the work must be
involved in the discussion as well. Discussion
throughout the process will ensure that the evaluation
is as useful and meaningful as it can be for the
program.
Program Evaluation for Organizations t Presentation of Evaluation Guides
41 • Writing the Report
The authors next touch on reporting the evaluation's
results or findings, particularly the written report. They
offer an outline that can be used as a model in writing
the evaluation report, pointing out that the report
contents will depend on:
- who the report is for;
- what you want to tell them;
- what they need to know.
Using the Evaluation Findings
At the close of the chapter, the authors point out that
evaluation doesn't end with writing the report, since
the findings are supposed to be useful and used.
Accordingly, they present a number of ways in which
the results can be used. For example, the results can
help the group's planning on issues and actions or to
prove the group's worth to funders and the
community.
Program Evaluation for Organizations t Presentation of Evaluation Guides
42 Self-evaluation
Chapter 6 (pages 48 to 56) walks you through the steps
in a self-evaluation, which is presented as a dress
rehearsal for a possible outside evaluation. The means
suggested by the authors is a workshop in which eight
topics are reviewed:
1- group history;
2- goals and objectives;
3- the context in which the group operates;
4- description of the work;
5- external relations;
6- achievements;
7- internal organization and structure;
8- implications for future work.
The authors map out how to work through each of
these topics, beginning with group history. The purpose
is to sketch the group's history based on participant
experience and perspectives. Individuals are asked to
reflect on how they got involved initially, on the work
Program Evaluation for Organizations t Presentation of Evaluation Guides
43 they wanted to do in the group, and what was most
important to them about what the group was doing.
Then they are asked to consider how the group was
created, what were the first actions/activities the group
undertook, and what were the most significant
influences on the group.
The authors suggest pulling together all pertinent
records and documents when discussing goals and
objectives. After writing the goals and objectives on a
flip chart, you can ask each individual what they mean
for him.
The context can be examined by describing the factors or
aspects of the environment that most directly affect the
group's activities (e.g., allies in the community,
funding sources, cultural situation, etc.) and taking
note of the positive/negative impact that each has on
the group's work.
Program Evaluation for Organizations t Presentation of Evaluation Guides
44 As for the description of the work, the authors suggest
initiating a group discussion on the different areas of
activity and asking participants to list the jobs that they
do in each area. In reviewing the lists, the group will be
able to see what gets done and what doesn't. This
process brings out both successes and shortcomings.
To examine external relations, the authors recommend
drawing up a list of the individuals and organizations
the group has already worked or dealt with. Then
analyse their influence on your group's work.
In looking at achievements, you should list what
you've achieved and what was accomplished with each
achievement. Then look back over your original
objectives to see. any changes in direction or gaps that
have become apparent.
The discussion on internal organization and structure
should bring out how the group is making decisions
and how communication within the group is actually
taking place.
Program Evaluation for Organizations t Presentation of Evaluation Guides
45 Implications for future work is the step in which you
pull everything together that you've learned from the
self-evaluation process. You need to ask yourself if
you've done what you wanted to do, what the self-
evaluation says about future work, and what direction
should the group take from where you are.
The authors suggest setting aside an evening and a full
day to carry out the self-evaluation. In addition, they
state that, in most cases, pulling in an outside facilitator
is a good idea, especially one who is familiar with and
supports the group, although not an active member of
it.
Background Materials
Three appendices round out the guide, including a
short glossary, examples of data-gathering tools, and a
sample evaluation form for collecting numerical
information from a questionnaire or interview.
Program Evaluation for Organizations t Presentation of Evaluation Guides
46
Section 3 : Highlights
Summary
This guide represents a major effort to make
evaluation a more democratic process. It provides a
wealth of information that can help groups come to
grips with their own evaluation, which is presented as
a process that is for the individuals directly involved in
designing and carrying out the work as well as those
the work is aimed at.
• Noteworthy Features
Two items pertaining to involving the main people
concerned by the process particularly attracted our
attention: information about preparing for an
evaluation and information about negotiating with
funders. Recognizing them as phases with high
potential for influencing the remainder of the process,
the authors provide a number of guidelines that can be
very useful in focussing an evaluation on group needs
and expectations.
Program Evaluation for Organizations t Presentation of Evaluation Guides
47 With respect to preparation, the five steps suggested
help the group to clarify its and organise its ideas about
the questions that it is asking itself about the work, the
resources needed to collect and analyse data, and so on.
These steps can be quite valuable in helping the group
design an evaluation that responds to the group's
context and in helping the group get ready for
negotiations with funders.
We feel that the main contribution of this guide,
however, is the chapter on negotiating. What's more,
we find it rather exceptional that an evaluation guide
discusses the issue. At a time when funding agencies
are increasingly requiring community groups to
evaluate their work, this section should be welcomed
by groups.
The issues that the authors recommend bringing up
with funders are clear, specific, and undoubtedly
founded on concrete experiences. These questions focus
mainly on the funder's perspective and expectations,
Program Evaluation for Organizations t Presentation of Evaluation Guides
48 such as the kind of data sought and how the funder
gauges success. We believe that these questions can
serve as significant references points for the group in
obtaining the information that it needs to compare the
funder's needs and expectations with its own.
There are also suggestions for carrying out effective
negotiations with the funder and for protecting the
rights of all parties, especially the group's. It is a shame,
however, that the authors didn't see their way clear to
provide a sample or model protocol of agreement,
which would have been extremely useful and would
have made the guide that much more complete.
We should also draw attention to the importance the
authors place on having everyone involved in the
group's work participate in the discussions on results
analysis. In their opinion, this is how to maximize
usefulness to the group.
Program Evaluation for Organizations t Presentation of Evaluation Guides
49 The authors feel strongly about using the results, a
point that is often ignored in the evaluation process.
The suggestions they make are practical and highlight
the various uses for evaluation results.
This guide does have its share of shortcomings,
however. The chapter dealing with developing the
evaluation design sketches a fairly good picture of the
various steps normally involved in this phase, yet
some of the elements linked to their functioning need
to be more clear and specific. The concept of objective is
not very well-stated, either. Lasdy, if simplicity is a
concern in selecting instruments for measuring the
achievement of objectives, as the authors seem to feel,
then they need to pay more attention to the steps
involved in preparing the evaluation and using the
recommended tools. The guidelines and examples they
provide are useful but inadequate.
Program Evaluation for Organizations t Presentation of Evaluation Guides
50 Along similar lines, we think that more attention
should have been given to considering requirements in
establishing the linkages between the work carried out
and the observed outcomes.
Summary
Overall, this guide offers a clear view of the concerns,
constraints, and problems facing groups who want to carry
out an evaluation or who are required to do so. Keeping
On Track is a major tool for helping groups develop an
evaluation design that really suits them. It is rich in ideas,
means, techniques, and approaches that can help groups
make evaluation a constructive exercise that adheres to the
perspectives and interests of the group. In this regard, it
succeeds quite well.
Program Evaluation for Organizations t Presentation of Evaluation Guides
51
Chapter 3 A Guide to Evaluation for Community Groups'
Section 1 : Synopsis
For Whom?
This manual was written as a teaching aid to help
groups produce an evaluation audit of their operations,
activities, objectives, target clientele, and financial
situation.
By Whom?
The guide was produced by the Centre de formation
populaire de Montréal.
Structure
The guide is 36 pages long, divided into four chapters,
five appendices, and a short bibliography. The first
4 This chapter presents L'évaluation d'un organisme communautaire et social by Christine Daniel (Centre de formation populaire, 1990), which is available only in French.
Program Evaluation for Organizations Presentation of Evaluation Guides
52 three chapters focus on preparing for and conducting a
team meeting to carry out an evaluation audit. The
approach involves creating group dynamics that would
make it possible to adjust group objectives and
activities on an ongoing basis. The fourth chapter deals
with presenting an annual report to the general
assembly and with defining new orientations arising
from the report and the related operational planning.
Where to Order
Centre de formation populaire 3575, boulevard St-Laurent, bureau 406 Montréal (Québec) H2X2T7
Program Evaluation for Organizations t Presentation of Evaluation Guides
53
Section 2 : Contents
The Role of Evaluation
The author starts by reminding us that evaluation plays
a very important role in the group. Although it's not
easy to evaluate a community group, evaluation
provides the group with an excellent opportunity to
discuss and ask itself questions about what it has done
and what its implications for future work are.
A Collective Process
The author sees evaluation as being a collective process
that must be properly prepared and led. In this light,
she outlines a process that can guide a group through
evaluation. Daniel identifies and expands on various
activities that can be used to cany out an evaluation
audit during a team meeting.
Setting Up the Evaluation Meeting
To set up the evaluation meeting, the author suggests
answering the following questions:
Program Evaluation for Organizations t Presentation of Evaluation Guides
54 - Why evaluate?
- What do you want to evaluate?
- What approach should be used?
- What information sources can be used for
background?
- Who will lead (facilitate) the evaluation meeting?
- How much time will be required for the meeting?
- When and where should the meeting be held?
Why Evaluate?
The first question deals with evaluation usefulness.
The author states that evaluation shouldn't be
undertaken without determining, from the outset, why
you want to evaluate. Daniel points out that evaluation
must first and foremost allow the group to improve its
work, measure its achievements, better define its
objectives, and examine its shortcomings. In addition,
it must serve to identify and promote the need for the
organization, its relevance, and the quality of its work.
Program Evaluation for Organizations t Presentation of Evaluation Guides
55 Specifying the Evaluation Focus
The second question refers to choosing the focus of the
evaluation. In addition to the reasons that naturally
come to mind, the author points to the organization's
objectives, its democratic functioning, financial
situation, target clientele, management of human
resources, work organization, teamwork, and so on.
Since there are many aspects that could be evaluated,
Daniel thinks that the group should determine, from
the outset, which it wants to look at.
Specifying the Evaluation Approach
With respect to the evaluation approach, the author
mentions that, regardless of the activity evaluated, you
need to pay particular attention to several main
considerations: program effectiveness, efficiency,
impact, and relevance. She provides a number of
questions relating to each topic that help define them
in her terms. To illustrate, Daniel suggests examining
the relevance of the activities by comparing them to the
group's mission and accounting for any discrepancies.
Program Evaluation for Organizations t Presentation of Evaluation Guides
56 Identifying Instruments
The author lists five types of data-collection tools;
descriptions and examples of each are provided. Daniel
groups them into categories: quantitative instruments,
qualitative instruments, documents relating to group
objectives, documents relating to organization
operation, and synthesis tools. Quantitative
instruments include statistics, questionnaires, surveys,
financial statements, and so on. Qualitative
instruments take in activity reports, press clippings,
questionnaires, surveys, participant evaluation of
activities, and the like.
Extracting Pertinent Information
Daniel states that these instruments will be of little
value if they are just left on the table during the
evaluation meeting. Consequently, she proposes that
you start by pulling out just the information relevant
to the annual report using her synthesis tool, which
allows you to organize the main information required
for the evaluation. This tool consists of a table with
Program Evaluation for Organizations t Presentation of Evaluation Guides
57 four columns —objectives, activities carried out,
deadlines, and individual responsible— that lets you
rapidly see the program's main elements.
Choosing a Facilitator
Daniel feels that the individual responsible for heading
up the evaluation should be very familiar with what
the role entails and have the following personal
characteristics: have a thorough understanding of the
evaluation process, be capable of interpreting
information, and be accepted by all participants.
Setting the Schedule
Concerning the amount of time required for the
meeting, the author notes that you should allow at
least a minimum amount of time, which, of course,
depends on the group's particular situation. In many
cases, two days appears to be ideal to carry out a fairly
comprehensive evaluation.
Program Evaluation for Organizations t Presentation of Evaluation Guides
58 When and Where
The author then draws attention to the fact that,
although the remaining questions relate less to content,
they are none the less important. Daniel points out that
you should wind up the preparation by selecting a
location and time that will be let the participants focus
as freely as possible on the topics under discussion.
Evaluation: An Exercise related to Defining Objectives and Planning
In broaching the second phase of the evaluation process
(implementation), the author reminds us that
evaluation is a dynamic, ongoing process in the life of a
program or organization. It can be used to periodically
re-evaluate the organization's mission and adjust its
objectives and the means used to achieve them.
Daniel provides an example of how a group advocating
for the rights of the unemployed operates. She describes
their general objectives (e.g., improve the living
conditions of the unemployed; inform the unemployed
Program Evaluation for Organizations t Presentation of Evaluation Guides
59 of all amendments to legislation affecting their status);
its specific objectives (e.g., study and analyse
unemployment reform in Canada; highlight the stakes
under the reform); the means used (e.g., information
meetings; newsletter); and the outcomes (e.g.,
significant participation in activities; MP who is very
friendly but not very supportive). She then gives a
series of questions to ask yourselves during evaluation
(e.g., What does the group want to know? Has the
group achieved its objectives?) and examples of how to
redefine objectives (e.g., call attention to new reforms
affecting the unemployed; denouncing the reforms as
regressive and highlighting positive points).
Conducting the Evaluation Meeting
• Remind Everyone of the Objectives
The author then moves on (Chapter 3, pages 21 to 26) to
the evaluation meeting and provides an outline of
how to conduct it. The agenda is fairly simple: it
consists of discussing the objectives already established
Program Evaluation for Organizations t Presentation of Evaluation Guides
60 one by one. Of course, the information yielded by the
various instruments go into the discussion. Daniel
suggests closing the discussion with a summary of the
main points retained for each objective.
• Discussing Operations
The next step in Daniel's process is to examine team
functioning and life by discussing the relations between
the various of participants in the group, the organi-
zation and climate of work, the motivation and
satisfaction of group participants, and so on.
Evaluating the Meeting
Daniel suggests that you end the meeting with the
highlights of what you have learned from the eva-
luation experience, both individually and collectively.
The Evaluation Report
The author emphasizes that meeting notes or minutes
are essential. She recommends using a table divided
into two columns. The first column is for comments,
Program Evaluation for Organizations t Presentation of Evaluation Guides
61 opinions, statistics, and the like relating to objective
evaluation. The second column is for anything
pertaining to perspectives.
Future Implications
Once the report has been written, you go on to the third
phase in the process: orientation and planning.
Daniel reminds us (Chapter 4) that planning is carried
out during the first phase of a new cycle for the
organization because the evaluation conclusions are
needed to decide whether to formulate new
orientations or stay the course with existing ones. She
points out that most groups handle this in a general
assembly. Consequently, she recommends that the
evaluation report should be present to the general
assembly, starting with each objective, then indicating
what was retained during the evaluation meeting, and
finally reporting on the suggestions for future work.
Program Evaluation for Organizations t Presentation of Evaluation Guides
62 Once this has been completed, you need to determine
more specific objectives and plan how they will be
achieved. The author indicates that planning is usually
carried out by a working group, the board of directors,
or both together.
Program Evaluation for Organizations t Presentation of Evaluation Guides
63
| Section 3 : Highlights |
We think that this guide is important for two reasons: it
reminds us that an evaluation audit can be used for the
ongoing adjustment of an organization's objectives and
activities; and it sheds light on the major phases in
conducting this type of evaluation.
By fitting evaluation into the life cycle of a program or
project, Daniel reminds us that evaluation is not an end in
itself, but rather a means for examining the group's work
and for improving it. She presents the concept of
evaluation in a much larger perspective than a simple
routine review and sets it apart from management control,
which makes organizations give an accounting to funders.
We feel that these are exciting and practical aspects for
organizations.
The evaluation process itself is presented in a clear but
rather sparse manner. The main phases for carrying out a
good evaluation are discussed and practical
implementation guidelines provided.
Program Evaluation for Organizations t Presentation of Evaluation Guides
64
Summary
The process presented in this third guide aims at helping
groups carry out a regular, systematic audit of how their
operations, activities, and objectives: the process proposed
by Daniel focuses on the internal operations of the
organization and targets the regular adjustment of
objectives and activities that promote objective
achievement.
Program Evaluation for Organizations t Presentation of Evaluation Guides
65
Conclusion
Our purpose in writing this manual has been to
present a relatively detailed critique of the contents
of three evaluation guides designed to help community
groups get better acquainted with evaluation. The first
guide describes the evaluation process. The second
presents the concept of participant-focused evaluation. The
third describes a method for organizations that we have
termed evaluation audit.
In dealing with each of these guides, we have focused on
highlighting the elements that we feel are most interesting
or useful for organizations.
The first guide provides readers with an opportunity to
develop a clearer understanding of the overall evaluation
process. Its originality lies with its view of who takes
charge of the evaluation process.
Program Evaluation for Organizations Presentation of Evaluation Guides
66 The second guide describes participant-focused evaluation,
whose objective is to provide groups with an approach that
empowers them in conducting their evaluation with all
the confidence and information required.
We were especially drawn to the importance given to
negotiations with sponsors with respect to evaluation: it
should be carried out so as to ensure that the evaluation
remains an exercise focused on group needs and
expectations.
We feel that the third guide is particularly useful in
carrying out an evaluation audit. This guide presents
evaluation as a collective process and offers an exceptional
opportunity to reflect on the group's work, to examine the
work that has been done, and to retarget future action.
The main idea that sets these guides apart is focus on
adopting approaches that empower groups in carrying out
evaluation. This empowerment includes defining the
goals and criteria that will serve as reference points in
Program Evaluation for Organizations t Presentation of Evaluation Guides
67 evaluating the program, project, activity, or organization,
and then selecting the design best suited to the stated
needs. Empowerment also refers to analysing and
interpreting the results, even if outside expertise is
required, and using the results to improve the quality of
the work.
Program Evaluation for Organizations t Presentation of Evaluation Guides
0 13,067 Vol .3
E-1974
Tard, Caroline Ouel le t , Hector e t a l .
Program Evaluation for Organizatior under CAPC (Community Action Pro-gram for hilf lrpn) * Vol, P r r ^ r tattura of Eva 1 uati onN<hn des
0 13,067 Vo l . 3
To o r d e r ï
C e n t r e d e r e c h e r c h e s u r les s e r v i c e s c o m m u n a u t a i r e s , U n i v e r s i t é Lava l , B u r e a u 2 4 4 6 , P a v i l l o n G i a r l e s - D e K o n i n c k , Q u é b e c ( Q u é b e c ) G 1 K 71M
Thi s ser ies «Program EVal u a i i o n f o r Organizations u n d e r C A P C ( C o m m u n i t y Ac t i on P r o g r a m to r C h i l d r e n ) » c o m p r i s e s t h r e e v o l u m e s :
O Introductory Manual
© Evaluation Tools for Enhancing Program Quality
Ô Presentation of Evaluation Guides
T h e p r o d u c t i o n of t he se t h r e e d o c u m e n t s h a s b e e n s u b s i d i z e d b y d ie C o m m u n i t y Ac t i on P r o g r a m tor C h i l d r e n , H e a l t h C a n a d a , w i t h t he a g r e e m e n t of d ie P r o v i n c e of Q u e b e c .
T e l e p h o n e : ( 4 1 3 ) 6 5 6 - 2 6 7 4 F a x : E M a i l : W3:
(418) 656-7787 c r sc@crsc ,u la v a l.ca h t t p f / / w w w . c rsc .u lava l ,ca