c2015

Projet de programme / Draft program

MISE À JOUR / UPDATED 2015-04-08

TABLE DES MATIÈRES TABLE OF CONTENTS

DIMANCHE 24 MAI – INFORMATIONS DÉTAILLÉES SUNDAY, MAY 24 – DETAILS

LUNDI 25 MAI – INFORMATIONS DÉTAILLÉES MONDAY, MAY 25 – DETAILS

MARDI 26 MAI – INFORMATIONS DÉTAILLÉES TUESDAY, MAY 26 – DETAILS

MERCREDI 27 MAI – INFORMATIONS DÉTAILLÉES WEDNESDAY, MAY 27 – DETAILS

2

24 MAI DIMANCHE SUNDAY MAY 24

13h-16h

Approche fondée sur les droits de la personne: Un regard sur l’évaluation axée sur le genre

Jeiran Rahmanian, Vanessa Anastopoulos ATELIERS/WORKSHOPS

Consulting Challenges

Gail V. Barrington ATELIERS/WORKSHOPS

Les protocoles d’évaluation faisant appel à des méthodes mixtes

Jacques Bérard ATELIERS/WORKSHOPS

Participatory Evaluation: Learning for a Change

Linda E. Lee ATELIERS/WORKSHOPS

Skills for Visual Data and Non-Lethal PowerPoint Presentations

John Burrett ATELIERS/WORKSHOPS

9h-12h

Évaluer les réglementations pour mieux anticiper leurs effets : Balises pour fiabiliser les Analyses d'Impacts

Réglementaires (AIR)

Eva Anstett

ATELIERS/WORKSHOPS

Facilitation skills for evaluators

Jennifer Birch-Jones ATELIERS/WORKSHOPS

Shared learning and participatory evaluation: the systematization approach for the assessment of

development interventions

Esteban Tapella, Pablo Rodriguez Bilella

ATELIERS/WORKSHOPS

Understanding and Using Contribution Analysis

Kaireen Chaytor, John Mayne ATELIERS/WORKSHOPS

Using Microsoft Excel to Enhance Data Analysis in Evaluations

Sandra Sellick

ATELIERS/WORKSHOPS

3

9h-16h

Qualitative Work in Evaluation: Why? When? and How? For the World We Want

Cindy Tananis

ATELIERS/WORKSHOPS

4

25 MAI LUNDI MONDAY MAY 25

10h-10h30

Évaluation d'une formation en matière d'agression sexuelle envers les enfants autochtones

Xavier Barsalou Verge, Renée Séguin, Mélanie M. Gagnon, Christian Dagenais PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION Le centre d'expertise Marie-Vincent offre un programme de formation en matière d'agression sexuelle envers les

enfants aux communautés autochtones du Québec. Tout au long de ce projet échelonné sur deux ans, une démarche

évaluative est menée en parallèle. Les objectif de cette démarche sont multiples, mais ont un but commun :

s'assurer que le programme de formation ait réellement une utilité pratique et qu'il réponde aux besoins de ces

communautés. L'exposé s'attardera à décrire la démarche évaluative qui comprend de s collectes de données

quantitatives, ainsi qu'une étude de cas multiple pour en apprendre plus sur l'utilité pratique de la formation en

contexte autochtone. Il sera aussi question des efforts mis en place pour retransmettre les résultats de l'évaluation

aux participants et à leur communauté. Pour conclure, les enjeux et les leçons tirées de cette évaluation menée

auprès de populations autochtones seront présentés.

10h30-11h15

Is Evaluation Losing Salience?

Andy Rowe

CONFÉRENCE D’EXPERT/EXPERT LECTURE The connectivity of natural and human systems is demonstrated by important shifts incorporating sustainability into

public policy, the actions of for-profit organisations and public awareness. Climate change and sustainable

development are but two demonstrations of this change. To be salient today analysis and advice must address

sustainability. The premise of this presentation is that evaluation as a field is still strongly focused on the past when

a focus on human systems alone was OK. It is no longer OK, evaluation risks losing salience and falling even further

behind on our use agendas. For evaluation to regain salience it needs to incorporate sustainability into training,

standards, practice and theory, and the stance of evaluation to contemporary siloed go vernance. Suggestions are

offered and audience discussions of the issue and ways forward will be invited.

10h30-11h30

A Participatory Approach for Analysis and Reporting to Enhance Evaluation Knowledge

Shannon Clark Larkin, Nicole Michaud, Michelle Picard-Aitken

EXPOSÉ/PAPER The use of a streamlined and stakeholder engagement approach for analysing and reporting on evaluation findings

and recommendations for the evaluation of the Canada Excellence Research Chairs Program will be presented. Given

tight timelines, and complex data collection, governance and consultation processes, an approach was implemented

for the analysis and reporting phases that enhanced stakeholder participation and resulted in increased access to

evaluation knowledge across the various stakeholder groups. Details on the preparation and usefulness of a

presentation deck of evaluation findings that was co-produced by the internal and external evaluation team will be

provided. The deck was refined and updated based on feedback from various stakeholder groups during the analysis

phase. This resulted in an evaluation report that contained concise and actionable recommendations that addressed

both accountability requirements and continuous improvement for program management.

Bringing the coaching attributes to evaluation

Kassem El Saddik

5

EXPOSÉ/PAPER The paper intends to bring the co-active coaching attributes and perspectives into the evaluation practice. It will

highlight key similarity between the two practice in terms of principles, ethics, qualities and strategy. It argues that

applying the fundamental elements of co-active coaching in evaluation will take the latter to an extra level by

unveiling new dimensions, revealing innovative theory of change and uncovering unintended outcomes. It sh eds the

light on the co-active coaching cornerstones (holding people naturally creative and resourceful and whole),

attributes (self-management, listening, curiosity and deepening the learning), setting the strategy (designing

alliance) as well as the ethical and paradigm (client-centric, unveiling intended and unintended outcomes).

Developmental Evaluation Learnings from a Girls Empowerment Program

Anne Miller

EXPOSÉ/PAPER Sirius Girls is a pilot program in partnership with United Way's All in for Girls In itiative and the Women's Leadership

Council. It is an after school program for vulnerable girls in grade ten using a social -emotional and service learning

approach to assist girls to take action with the support of adult allies (women Leaders who volunteer with the

program) to experience mastery and build confidence to take new risks, creating a better community for all girls to

thrive. Since the program is an innovative initiative at the development and pilot stage, a developmental evaluation

approach has been used to help understand the successes, challenges and positive impacts of the program. This

presentation will describe the developmental evaluation approach taken with this program, new measurement tools

(like the VACO survey) that have been explored for use with this population, and learning from the process.

Évaluation des effets de la pratique de l'évaluation d'impact sur la santé (ÉIS)

Jean-Marie Buregeya

PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION L'évaluation d'impact sur la santé (ÉIS) est une démarche évaluative utilisée afin d'analyser les effets sur la santé des

politiques publiques qui ne relèvent pas du secteur de la santé (Harris et Spickett, 2011). J'applique une approche

novatrice (Analyse de contribution) pour l'analyse des effets à un programme complexe, en particulier pour le projet

de revitalisation du Vieux-Sorel et le projet d'urbanisme du centre-ville de Chateauguay afin d'apprécier le potentiel

d'ÉIS pour bonifier cette politique et ses impacts sur la santé et l'équité. Dès lors, il est nécessaire de présenter le

protocole de recherche puisque l'analyse de contribution est indiquée aux cas où les méthodes statistiques, tels que

les essais randomisés et quasi-expérimentaux, ne sont pas les mieux indiquées pour établir la causalité d 'une

intervention aux effets observés (Mayne, 2012), et n'a pas été utilisée pour évaluer les ÉIS.

Evaluation for the world we want: Child Rights Impact Assessment (CRIA) for Business

Tara Collins EXPOSÉ/PAPER 'Evaluation for the world we want' should include attention to human rights and business in order to support a

better world. The international discourse has done so as evidenced by the United Nations Guiding Principles on

Business and Human Rights (2011) and the Children's Rights and Business Principl es (UNICEF, Save the Children, &

Global Compact (2012)). These documents identify impact assessments to evaluate effects upon human rights. But

how can the child rights impact of business be assessed in practice? The implications of this relationship must be

ascertained through the development and use of child rights impact assessment (CRIA) for business. However, this

activity is emerging with restricted participation. As there are significant challenges to identify and assess processes

and results, other actors including evaluators are needed to support progress. This paper presentation presents a

preliminary CRIA tool to contribute to the evaluation field.

Evidence-Based Principles to Guide Collaborative Approaches to Evaluation

J. Bradley Cousins CONFÉRENCE D’EXPERT/EXPERT LECTURE

6

This expert lecture will focus on practice in collaborative approaches to evaluation with particular emphasis on the

development, validation and ongoing international field testing of evidence-based principles. Featured will be the

work of a collaborative research team headed by Cousins, Whitmore and Shulha that has been unfolding over a 3

year period (see Cousins, Whitmore & Shulha, 2013, 2014; Shulha, Whitmore, Cousins, Al Hudib & Gilbert, 2015).

Specifically, the lecture will: (i) describe a bottom-up, four-phase empirical process to develop and validate

principles involving 320 evaluators from CES, AEA and IDEAS; (ii) introduce the set of eight (8) validated principles

and (iii) report on ongoing international interests in field testing the principles. A goal of the lecture is to engage

with audience members in the identification and mobilization of field testing opportunities.

How to publish in a journal of evaluation?

Robert Schwartz, Kristelle Alunni-Menichini, Lynda Benhadj, Jean-Marie Buregeya, Marie Beausejour

CONFÉRENCE D’EXPERT/EXPERT LECTURE Dissemination of practice notes, real-life cases addressing challenges, and research and program evaluation findings

in evaluation’s domain have a primordial importance in order to change professional practices and influence

decision making in public organizations. Since the publishing process in the field of evaluation can be complex, Dr.

Robert Schwartz, Editor-in-Chief for the Canadian Journal of Program Evaluation, will be presenting the specific

characteristics of his journal and guidelines on 'what to do or not to do to publish in evaluation' and this, in

collaboration with EASY research chair’s members. He will also answer frequently asked questions addressing the

publishing process in evaluation. We invite you to participate at the preparation of these questions by sending yours

at kristelle.alunni@gmail.com before April, 1st. This expert lecture is thus an excellent opportunity to learn about

important tips to support publication of evaluation work.

Increasing the Evaluator's influence: techniques and micro-practices

Natalie De Sole, Melanie Hwalek

TABLES RONDES/ROUNDTABLES Evaluation's influence is conditioned by the stakeholders' receptiveness to listen, make sense, and use f indings. The

door of opportunity often closes after evaluators turn in the final report and meet one last time with our client. The

nonprofit's program director, foundation's program officer, or the governmental policy advisor must champion our

finding's use or file them away. A plethora of models exist on how to engage stakeholders yet discussion is limited

on the nuanced techniques required to bring these models to life. At this roundtable participants will be shown good

practice models related to rules of engaging individual, group and organizational stakeholders to create change.

Participants will be asked to share insights about how they apply engagement models in specific contexts. Together

we will identify techniques and micro practices that increase stakeholder buy-in and the eventual adoption of

evaluation findings.

L'approche participative en évaluation : le cas de Pouvoir Partager/ Pouvoirs Partagés

Joanne Otis, Lyne Massie EXPOSÉ/PAPER Pouvoir Partager/ Pouvoirs Partagés (PP/PP) est une intervention conçue pour outiller les femmes vivant avec le VIH

(FVVIH) face à la question du dévoilement du statut séropositif au VIH. S'appuyant sur un devis mixte, PP/PP a été

évalué au Québec et adapté culturellement au Mali (Gundo So) grâce à plusieurs recherches participatives où la

collaboration entre les différents acteurs et l'implication des FVVIH sont essentiels. Réfléchi par une équipe de

chercheuses AVEC et POUR les FVVIH elles-mêmes, PP/PP contribue au renforcement d'un plus grand pouvoir d'ag ir

chez les femmes du Québec et du Mali relativement au dévoilement de son statut séropositif au VIH. Diverses

stratégies de partage des connaissances et de renforcement des capacités ont permis la coproduction d'outils de

pratiques professionnelles disponibles en ligne (www.pouvoirpartager.uqam.ca), contribuant ainsi à améliorer la

qualité de vie d’'un plus grand nombre de femmes.

7

L'évaluation pour optimiser le dépistage du VIH pour les hommes gais: le projet SPOT à Montréal

Ludivine Veillette-Bourbeau, Joanne Otis

EXPOSÉ/PAPER SPOT est une recherche-intervention offrant, depuis 2009, un counseling et un dépistage du VIH rapide, anonyme et

gratuit en site communautaire pour les hommes gais ou bisexuels. En soutien à l'évaluation des effets, une analyse

d'implantation a été menée. Un monitorage du degré d'implantation a permis de voir sa fluctuation et des analyses

de régression linéaires d'identifier les éléments associés à sa variation, le counseling étant adapté au participant. Un

processus d'implantation en plusieurs phases, modulé par des facteurs liés aux motivations des acteurs, à la

complexité des dynamiques partenariales, aux défis de la coordination et à l'organisation de l'équipe terrain a

émergé de l'analyse qualitative. La pérennisation de SPOT soulève des défis: transformation d'une intervention en

contexte de recherche vers un service régulier, maintien des acquis de la recherche et appropriation de la gestion

par les partenaires.

Managing project constraints for the next decade

Judy Lifshitz CONFÉRENCE D’EXPERT/EXPERT LECTURE Evaluations are a key source of information to assist in responding to specific questions to support decision making

in a time where resources to conduct them are more and more limited. Thus, it is crucial that evaluatio ns be on

time, on budget and in scope so that the evaluations can be useful for decision making. This presentation will

provide an overview of the international standards used in the field of project management to help to address the

three project constraints of time, budget and scope. Examples and practical tools and techniques for addressing

time, budget and scope will be shared with participants based on the presenter's experience and internationally

recognized best practices. As a PMP, over the last five years, Judy has managed several evaluations and has used

these tools and techniques to ensure that timelines, budgets and scope is managed. She has done previous

presentations on project management at CES and PMI related events.

Reframe QUTs evaluation framework encompasses a multi-faceted approach to meet stakeholder needs

Lyn Alderman CONFÉRENCE D’EXPERT/EXPERT LECTURE Since 1984, the Australian government has invited students to provide feedback on their course of study and in the

early 2000s annual student feedback surveys became a government accreditation requirement for universities. As a

result, Queensland University of Technology (QUT) launched online invitational student feedback surveys in 2007. In

2010, stakeholders raised concerns about the mixed purpose of the surveys, the potential for data misuse and how

academics felt evaluation was being done 'to them' rather than 'for them'. In response, QUT introduced Reframe a

five-year project encompassing three phases: stakeholder engagement and discovery - executive support,

environmental scan, literature review, and theoretical concept; product development; and dissemination and

communication - target audiences, communication plans, professional development and iterative refinement. In

2015, QUT's broad evaluation approach meets the needs of a range of stakeholders.

The experience of a paticipative evaluation: Brazilian QualiSUS-Network Project

Gisela Cardoso, Marly Marques Da Cruz, Aline Duque De Macedo, Ana Cristina Gonçalves Vaz Dos Reis,

Celita Almeida Rosario, Patrícia Pássaro Da Silva Toledo, Solange Kanso EXPOSÉ/PAPER The Brazilian QualiSUS Network Project (QNP), a cooperation between the World Bank and the MoH, is a strategy to

promote the qualification of health care assistance and management, and the development of technologies among

Health Care Networks. This intervention was implemented in 15 different and diverse health regions. In order to

support the institutionalization process and use of M&A as a tool for management and decision -making in context,

an implementation evaluation was conducted. The evaluation adopted a participatory approach which included the

development of 15 regional logical models, the identification of potential evaluation users, an agreement on

8

indicators and goals; the characterization of the political-organizational contexts; the identification of facilitators

and barriers in the implementation process, and the analysis of the agreed management arrangements and its

contribution to the implementation in the contemplated regions.

The UN Resolution and International Evaluation Year - What's in it for Africa?

Ian Hopwood

EXPOSÉ/PAPER What explains the upsurge of enthusiasm for evaluation (UN Resolution, International Evaluation Year, etc.)? Will it

last? Can it bring real change? What are the driving forces? The author uses his wide international and African

experience to assess real potential and identify pitfalls. Can evaluation escape the “donor driven image” and

monitoring overhang? Are there pre-conditions for evaluation to thrive and make a difference in Africa? How does

evaluation relate to cultural context and governance systems? The links to policy and planning? And to public

pressure for performance and accountability? Correcting the imbalance between accountability and learning?

Shifting the focus from projects to systems and policies? Getting Africa’s leadership and donors to acknowledge

'failure' and embrace learning approaches? The conclusions highlight new partnership opportunities and capacity

development initiatives to foster African leadership for appropriate practice.

Valuation: Methods for Representing the Value of Program Impacts in Monetary Units

John Gargani EXPOSÉ/PAPER Increasingly, evaluators are being asked to perform valuations — measuring the merit, worth, or significance of

program impacts in monetary units. In particular, valuation forms an integral part of recent value -for-money policies,

social return on investment analyses, and pay-for-success initiatives. I provide an overview of valuation, describe

valuation methods currently used by evaluators, and demonstrate how and why these methods sometimes produce

contradictory results. I conclude by describing a system for reporting valuations that may better communicate their

relative trustworthiness, thereby supporting the many uses to which they are put by funders, private investors, and

policy makers. Throughout, I provide examples using data on prisons provided by the UK government. The data and

worked examples will be made available electronically.

Valuing evaluation power and the power of evaluation in “Speaking Truth to Power”

Sandiran (Sandi) Premakanthan

CONFÉRENCE D’EXPERT/EXPERT LECTURE The main focus of this session is about valuing evaluation power and the power of evaluation in speaking evaluation

truth to power, those who create the demand for it, legislators, the public, heads of government departments and

agencies, program management, non-government organizations and donors. I have defined the terms evaluation

power and the power of evaluation and identified several sources of institutionalized evaluation power. They

include: governments through legislations, authority instruments, and policies, philanthropic foundations, financial

institutions, government aid agencies, United Nations (UN) agencies, non-governmental organizations (NGOs) and

numerous networks: evaluation societies, associations and international networks. The evaluation power of the

government of Canada, termed the “value model” and the creation of the power of evaluation and its use for

informed decision making is discussed.

What Environmental Assessment can Learn from Evaluation

Paul Kishchuk EXPOSÉ/PAPER The Yukon Environmental and Socio-economic Effects Assessment Act allows for consideration of social and

economic effects in parallel with environmental effects when resource development projects are scrutinized. Yet,

almost a decade after implementation of the Act, full and proper consideration of social and economic effects still

struggles to gain traction in Yukon's environmental assessment process. Drawing on his experience in the fields of

9

both socio-economic effects assessment and program evaluation, Paul will put forward a hypothesis, using the logic

model paradigm, which illustrates why socio-economic effects assessment has not been embraced in small Yukon

communities. The presentation will go on to explore how an evaluative approach to socio -economic effects

assessment can help advance resource development projects in rural and remote communities in Canada.

Whether Evaluations made a difference in the Management of HIV/AIDS Programs in South Asia?

R.S. Goyal

EXPOSÉ/PAPER In the quest to seek desired impact, cost effectiveness and accountability, funders of HIV/AIDS programs have been

seeking systematic assessment of outcomes and impact of these programs. These evaluations were also expected to

become prime mover for continuation of resource allocation and scaling up of the interventions. This paper presents

a synthesis of evaluations of HIV/AIDS programs in South Asia particularly contributing to the program management

in relation to; communication, stigma and discrimination and, preventing HIV among adolescents and young people.

The broad inclusion criterions were; randomized control/ quasi-experimental/descriptive deign and evidence for

casualty/ what has or has not worked/ provide evidence for up- scaling/replication of interventions. Preliminary

analysis indicates that; curriculum based-teacher led interventions in schools; and youth friendly health services

have made appreciable contribution to HIV/AIDS programming in the region.

Working for a World We All Want: The Journey of a Community Needs Assessment

Denise Belanger, Linda Lee

EXPOSÉ/PAPER While many practitioners embrace participatory evaluation, they are often faced with questions regarding how to

meaningfully involve participants in processes. What does meaningful participation and consultation look like in

different situations and with different constituents? Drawing from their work over the past few ye ars, Linda and

Denise will discuss collecting the voice of those who live in inner city neighbourhoods. In particular, the presentation

will focus on a community needs assessment undertaken in partnership with a community based agency where the

assessment focused on meaningful involvement of community, in the hope that those involved felt valued and

heard. In addition, capacity building was important to this project, as Denise and Linda worked with staff to

undertake community-based data collection and involve them finding meaning in the information collected.

Leçons apprises de l'évaluation du projet allo info sida

Rosine Flore Chuedo Djoungou

EXPOSÉ/PAPER La stigmatisation liée au SIDA constitue un problème central pour les porteurs de VIH au Cameroun, car les malades

marginalisés tombent en déprime morale, ce qui complique leur suivi. Le gouvernement tente de résoudre ce

problème, dans le cadre de sa stratégie nationale de lutte contre le SIDA. C’est dans cette perspective que SUNAIDS,

en partenariat avec le CNLS a mis en œuvre le projet Allo Info Sida, dont l’objectif est d’apporter l’information, le

soutien psychologique et moral aux malades du Sida et aux PVVS. Il s’agit d’un dispositif d’appel téléphonique

gratuit accessible à partir du téléphone fixe et du téléphone portable de toutes les régions du Cameroun. Après 2

ans de mise en œuvre du projet, l’évaluation du projet a permis de mesurer l’atteinte des objectifs, d’analyser la

démarche de mise en œuvre du projet , dans l’optique d’envisager des persp ectives d’évolution et les possibilités de

repositionnement du projet face à l’évolution du contexte environnemental.

10h30-11h45

Web-Based Stakeholder Feedback Portals: Allowing Wider Stakeholder Feedback

Makenzie McPherson

EXPOSÉ/PAPER

10

Advances in technology provide evaluators with continuously improving, cost-effective methods for the

dissemination of results and opportunities for capturing stakeholder feedback. One of these mediums, a web -based

stakeholder feedback portal, provides evaluators the opportunity to post visual representations (e.g. process flow

maps, logic models, root cause analysis diagrams, etc.) allowing stakeholders to review, edit, provide feedback, or

validation. This presentation will provide a demonstration of an active web -based stakeholder feedback portal,

showing how it can effectively strengthen the evaluation, and how it might be adapted for other uses.

13h15-14h45

An innovative approach to link evidence & practice for practitioners in the employment services area

Susanna Gurr

EXPOSÉ/PAPER The BC Centre for Employment Excellence aims to improve employment outcomes for job seekers by promoting

innovation and sharing evidence about employment programs with practitioners to enhance knowledge and ensure

best practices are implemented. The Centre has 2 key functions: 1) conduct projects in collaboration with

stakeholders to test innovative approaches 2)share research with practitioners. The Centre was launched in 2012 at

the initiative of the BC Gov't. The Centre emphasizes the role of practitioners in its activities to ensure the

knowledge developed and shared is relevant. Hear from the Managing Director about methods used to connect BC

practitioners to different forms of evidence in various engaging ways and the collaborations a nd projects that have

emerged from the Centre's activities. What are some opportunities to close the gap between evaluation and

knowledge capacity to enhance practice and ultimately lives of job seekers? What are some pitfalls to avoid?

Are user fees exemptions enough to increase the use of healthcare services by the worst off?

Nicole Atchessi, Valéry Ridde, Maria-Victoria Zunzunégui

PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION User fees exemption programs have been implemented in certain low- and middle-income countries to improve the

populations' access to healthcare services. Given the considerable healthcare needs of the worst off, they should,

when exempted from user fees, be using healthcare to a greater extent than others. The aim of this study was to

assess whether user fees exemptions increased healthcare services use among indigents in the Ouargaye district in

Burkina Faso. The indigents' increased healthcare services use was not attributable to user fees exemptions. Some

contamination of the intervention is conceivable. Interventions combining user fees exemptions with actions

targeting other obstacles to healthcare access would probably be more effective in increasing indigents' use of

healthcare centres.

Collaborative evaluation capacity building for social innovation in the public sector through DE

Keiko Kuji-Shikatani, Scott Urquhart, Jennifer McMaster, Kieran McMonagle, Cristina Ilas, Angelika Kerr

PANEL/PANEL Program evaluation began with the desire to seek information that can be utilized to improve the human condition.

The panel explores how infusing evaluative thinking through DE is positioned as a responsibility for internal

evaluators and leaders at all levels of the system focused on organizational learning and improvement in its

relentless pursuit of reaching every student, including: collaboratively building relationships and holistically

nurturing the whole person as Four Directions circle of caring adults– school staff, parents, community partners, and

the ministry - so as to very purposefully position the sharing of caring of First Nations Métis and Inuit learners; and

partnering with three school boards to conduct case studies of existing teen parent programs that integrate

academic, life skills and counselling components with additional community services to support teen parents in

finishing high school and transitioning to further education, training or to the workplace.

Demystifying Gender Mainstreaming

Emmanuel Trépanier, Laurentine Mefire

11

EXPOSÉ/PAPER Mainstreaming gender equality in international organizations is a challenging process that requires awareness,

political will, behavioural change, resources, time and, quite often, intensive guidance from specialized consultants,

using complex tools. Universalia's newly developed Gender-Sensitive Institutional and Organizational Performance

Assessment (GSIOA) Model is a user-friendly, time-efficient and free tool intended for users from all walks of life.

Adapted from the Universalia IOA Model, the tool allows users to assess the capacity, performance, external

environment and motivation of international organizations from a gender perspective in order to help decision -

makers improve equity in the workplace and contribute to positive social impacts. This presentation will allow

audience members to experience the GSIOA Model with pragmatic and challenging examples and increase their

acumen for equity.

Evaluating the Humanitarian Projects during Crisis: The Syrian Case

Tarek Daoud

IGNITE (PRÉSENTATION RAPIDE)/IGNITE-STYLE RAPID PRESENTATION The prolonged crisis in Syria leads to catastrophic humanitarian conditions. According to UNOCHA (2014) report

300000 people are living in under siege. As a response, INGO’s launched a series of humanitarian projects in

coordination with local partners. Humanitarian organizations face challenges in evaluating the impact of the

implemented projects due to the problems involved in overcoming the constraints of physical access and the

obscure limitation of sharing evaluation results due to security purposes. In this chaos situation, the participatory

evaluation approach found to be the best solution in order to provide an in -depth insight evaluation. Applying the

participatory evaluation improved projects performance, empowered the implementing partners evaluat ion skills,

creates evaluation teams in siege areas, enhanced the organizational learning growth, increased the validity of the

data collected, and strengthen the relation between local staff and external experts.

Evaluation du taux de survie à 12 mois des patients infectés par le VIH au Sénégal

Maguatte Ndoye Ndiaye IGNITE (PRÉSENTATION RAPIDE)/IGNITE-STYLE RAPID PRESENTATION Contexte: Le taux de survie constitue un indicateur clé du programme et permet d'évaluer l'impact de la mise sous

ARV des personnes éligibles au traitement. Objectif: Évaluer le taux de survie des patients 12 mois après la DUS mise

sous ARV pour l'année 2012 Méthodologie : la formule de calcul est = (Cohorte actuelle (CA)/cohorte nette (CN) x

100 CN = Cohorte de départ (N) Transfert In (TI) — Transfert Ailleurs (TA) CA = CN — DC — PDV — ABAN — ARR

Résultat: le taux de survie nationale est estimé en 71% en 2012 et 73% en 2011. Conclusion: Disparité observée dans

les régions.

Evaluation for the World We Want? Emerging Lessons from the Global South

Robert Mclean, Mallika Samaranayake, Ziad Moussa, Colleen Duggan

PANEL/PANEL In the view of many, evaluation can be used to help build the world we want. Canada's IDRC has supported this

vision through multiple avenues including supporting evaluation field -building in regions with identifiably less

capacity to match evaluation demand with local expertise. Several compelling results have emerged, and at

considerable scale. Two specific efforts have been the Community of Evaluators, South Asia (CoE) and the Middle

East and North Africa Evaluators Network (EvalMENA). Current President of the IOCE, Ziad Moussa will reflect on

experiences jointly leading the EvalMENA network. Founding President of the CoE Mallika Samaranayake will

comment on her work enhancing evaluation across South Asia. Insights will be of interest to those wanting to learn

unique lessons about professionalization, capacity-building and networking in evaluation. Especially of interest to

those working to bridge evaluation supply and demand between the global North and South.

Evaluation of Data Quality

Harold Henson

12

EXPOSÉ/PAPER Problems with data have been an ongoing challenge in the field of program evaluation. In 2009, the Auditor General

reported that a significant share of published evaluations were compromised by problems with the available

performance measurement data. There are few reasons to believe that the situation has improved substantially

despite the improvements in technology. In this paper the author argues that the same basic process and

methodology used in program evaluation in general could be applied to the evaluation of data. It is argued that the

standardized evaluation questions and lines of evidence can be modified to assess quality of data generated by

programs for evaluation. At the end of this exercise there will be a report that resembles a program evaluation but

addresses the issue of data quality. In the best case scenario, high quality evaluations will be possible without

expensive surveys.

Evaluation Principles for a Democratic World

Elizabeth Lewis

IGNITE (PRÉSENTATION RAPIDE)/IGNITE-STYLE RAPID PRESENTATION As an organization with the mission to advance freedom and democracy worldwide, the evaluative efforts of the

International Republican Institute not only seek to adhere to internationally accepted standards for evaluations, but

to reinforce the democratic principles its programs seek to advance. IRI will use an example of a participatory needs

assessment it conducted in Somaliland to show how evaluation can consider six democratic principles: accessibility,

accountability, efficiency & effectiveness, equity & inclusiveness, responsiveness and transparency. Commissioned

by the UK's DFID, this needs assessment informed IRI's development of the Strategy for International

Democratization Support to Somaliland for the international community. The assessment captured input from

representatives of the international donor community, international implementers, subject matter experts and

Somaliland stakeholders from civil society, political parties, the media and government.

How to Use Process Flow Mapping for Continuous Quality Improvement

Ralph Renger, Melissa Rogan, Makenzie McPherson EXPOSÉ/PAPER Continuous quality improvement (CQI) is a common evaluation purpose. One potentially useful CQI method is

process flow mapping. Strategies for identifying and contacting potential subject matter experts, the basic steps and

tips in conducting a process flow map (PFM) interview, how to integrate interviews into a single, summary PFM,

methods for validating the PFM, and how efficiency data in conjunction with the PFM can be used to facilitate

corrective actions to improve system efficiency will be shared in the context of conducting CQI in a cardiac system of

care. Implications for using PFM in conducting CQI in other evaluations will be discussed.

How We Make a Difference

Paul Kishchuk

IGNITE (PRÉSENTATION RAPIDE)/IGNITE-STYLE RAPID PRESENTATION A mash-up of Mark Schacter's 'Keeping Busy...Making a Difference' logic model diagram and the DIKW (data -

information-knowledge-wisdom) paradigm is used to illustrate what the end game of evaluation sho uld be.

Influencing Change: Perspectives from Government, Private Consulting and the University

Donna Smith-Moncrieffe, Celine Pinsent, Mark Nafekh, Pierre Mercier PANEL/PANEL Influencing Change: Perspectives from the Government, Private Consultants and th e University A number of

evaluations conducted in Canada are produced by the federal government in collaboration with private consultants,

and educational institutions. It is important for evaluators from both the private and public sector to collaborative ly

reflect on how evaluation findings have had an impact on the communities and policies. This panel will answer

important questions about impact and influence from the federal government, university and evaluation

consultant’s perspective. The presentations will respond to some of the following questions as per the theme of the

13

conference: •How does evaluation make a difference in the community? •How has evaluation influenced policy in

the federal government and educational institutions? •What are some of the strategies used to reduce the barriers

to implementing sound evaluations?

Instrumentation de la théorie du programme dans le cadre d'une évaluation participative

Rodrigo Quiroz, Nathalie Bigras, Julie Dion, Karine Doudou

EXPOSÉ/PAPER La théorie du programme (Chen, 1990, 2005) est une méthode utilisée par les évaluateurs afin de mieux définir

l'intervention à évaluer (Champage et al., 2011), de comprendre les processus à l'ouvre dans son implantation

(Green & MacAllister, 2002), d'identifier ses effets et les mécanismes en jeu dans leur production (Chagnon et al.,

2009), d'accroitre l'utilisation des résultats de l'évaluation (Wholey, 1987) et d'améliorer le programme (Weiss,

1995). Malgré son importance, il y a un manque d'écrits illustrant l'instrumentation de la production de la théorie du

programme dans le cadre d'une d'évaluation participative. Cet exposé vise à combler ce vide en présentant les

fondements conceptuels, ainsi que les étapes, les techniques utilisées et les principaux résultats obtenus de

l'évaluation d'un programme destiné à la formation des éducatrices en petite enfance.

L'analyse logique comme levier de changement : innovations et applications en santé publique

Tarik Benmarhnia, Véronique Foley, Jean-Marie Buregeya, Samantha Gontijo-Guerra, Gisela Cardoso,

Kristelle Alunni-Menichini, Marie-Claude Tremblay

PANEL/PANEL

L'utilisation de l'évaluation : ce que la théorie ne mentionne pas

Grégory Häuptli IGNITE (PRÉSENTATION RAPIDE)/IGNITE-STYLE RAPID PRESENTATION Il existe une importante littérature sur l'utilisation de l'évaluation et plus particulièrement sur les facteurs internes

(qualité de l'évaluation) et externes (contexte politique et institutionnel) influençant négativement ou positivement

la probabilité que les résultats d'une évaluation soient utilisés par les chargés de programmes ou les décideurs

politiques. Cette présentation confronte cette littérature aux défis, contraintes et opportunités auxquels font face

les évaluateurs dans le but de mettre en avant les facteurs qui semblent être peu théorisés, voire ignorés par la

littérature. Il ne s'agit donc pas de remettre en cause la littérature sur l'utilisation de l'évaluation mais plutôt de

questionner pourquoi certaines contraintes vécues ne sont pas ou ne peuvent pas être théori sées.

Measuring the extent of institutionalization of ART programs in health facilities in Uganda

Henry Zakumumpa PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION In 2004,Uganda commissioned a national antiretroviral therapy (ART) scale-up program with external donor support.

We sought to measure the extent of institutionalization of ART programs in health facilities in Uganda and compare

institutionalization scores by health facility type. Level of Institutionalization (LoIn) scales developed by Goodman, et

al(1993)were used to measure the extent of institutionalization of ART programs at 195 health facilities in 42

districts of Uganda which received donor support between 2004 and 2009 to initiate ART services. The overall mean

institutionalization score for participating health facilities was 3.5(Range, 1-4) and the mean score for niche

saturation, the highest level of institutionalization, was 3.2(Range,1-4).Private health facilities had the lowest mean

institutionalization score. Programs for enhancing the institutionalization of ART interventions in private health

facilities are recommended. ART program evaluation and supervision are deficient.

Not an Island - Insights from Other Disciplines

Alexandra Dagger, Shelley Borys, Bill Blois, Marie-Josée Dionne-Hebert

PANEL/PANEL

14

Evaluation has a long history of integrating knowledge and techniques from other disciplines. Approaching our work

with these other disciplines in mind brings fresh insight and adoption of new ideas and methods as well as benefits

to evaluands. This session will explore learning that evaluators can draw from areas such as performance

measurement, internal audit, risk and communications disciplines. Evaluators have not always tended to turn to

these areas for knowledge to use in our work but they have the potential to add value. The panelists are senior

members of the federal evaluation community. However, although they come from the public sector, the disciplines

explored and lessons learned highlighted will be relevant to all evaluators.

Performance challenges of Tuberculosis control actions in Brazil: scapegoats, myths & possibilities

Gisela Cardoso, Elizabeth Moreira Dos Santos, Dolores Abreu, Pedro Paulo Chrispim, Gisele Oliveira, Juliana

Borenstein, Amanda Pereira EXPOSÉ/PAPER Tuberculosis (TB) is a national health problem for Brazil. The country has different cultural contexts and

epidemiological TB profiles. As a preliminary step of a performance evaluation regarding TB prevention and control

actions, the objective of this study was to conduct a strategic and logical analysis of TB control actions in four

Brazilian municipalities addressing activities at primary health care. To develop the strategic analysis, we focused on

a main question: "What are the potentialities of the national and municipal strategies to respond to TB control

challenges?". The steps of the strategic analysis included the identification of the strategic objectives, goals,

indicators and their classification according to the EGIPPS model domains and alignments. The appraisal of t he

logical modeling involved document review and interview with TB key actors. Results are going to be presented

according to the questions posed for both strategic and logical analysis.

Power, Institutions and Gender Relations: Can Evaluations Transform them?

Ranjani Krishnamurthy

EXPOSÉ/PAPER This paper examines the perception of women, men, adolescent girls and boys from three Chennai slums, India how

the institutions of marriage, family, local markets and government should function if gender and social relations are

to be equitable. It then examines their perception of actual change in the last five years, what they consider positive

and negative and causality of change. In the process, marginalised people's inter -sectoral assessment of government

programmes also emerges (with recommendations) and this is contrasted with sectoral evaluations of government.

This paper argues that mainstream evaluations of government are sectoral or of one Department, and this does not

match the intersectoral and inter-institutional lived realities of women/girls and men/boys. Evaluations capturing

movement towards the world women/girls want, should give importance to their indicators of change & power,

methodologies, interpretations and recommendations

Program Design and Performance Measurement

Michel Laurendeau, John Burrett

EXPOSÉ/PAPER While evaluation and performance measurement frameworks have helped, most evaluators still have to conduct

evaluations of public programs where the theory of intervention is uncertain and incomplete, with weak indicators

and poor ongoing performance measurement. Program research and descriptions of good practices often fail to

identify key success factors and support the improvement of program design. Frameworks have also been poor at

linking delivery and outcome indicators. The presenters will discuss two approaches to improved specification of

logic models with the use of: 1) network analysis to identify expected linkages of cause and effect and external

influences; 2) Integration Definition Function (IDEF) modeling techniques to describe delivery processes. They will

demonstrate how these techniques can support the development of improved logic models, a more compelling

analysis of program performance and a better understanding of program economy, efficiency and effectiveness.

15

Program science in practice: Lessons, challenges and implications

Sean Rourke, David Seekings

EXPOSÉ/PAPER Program science is an emerging public health approach that systematically applies scientific knowledge and evidence

to improve the development, implementation and evaluation of programs. By emphasizing the importance of the

context in which interventions occur, program science highlights the essential role evaluation plays in informing

strategic action to address key health challenges. This research project will build a better understanding of what this

approach looks like in practice, using evidence from case studies of community led HIV prevention interventions to

assess the practicalities of making program science work on the ground. The presentation will examine the

challenges and opportunities of using evidence to inform each stage of community -based interventions, from

development to evaluation, highlighting key lessons from the cases. It concludes by discu ssing the broader

application of program science beyond public health to evaluation and public policy more generally.

The Influence of Internal Evaluation on Program Management

Andrea Macdonald EXPOSÉ/PAPER Nonprofits are under increased pressure to deliver more for less and to demonstrate results of their work. When the

Clean Foundation, an environmental nonprofit based in Nova Scotia, invested in evaluation training for management

and core program staff, the intent was to increase capacity in internal eva luation and develop frameworks to

evaluate core programs. However, as the group went through the process, the discussion and work often focused

more on questioning the design and delivery of individual programs and provided insight into management and

stakeholder expectations. The value to undertaking the evaluation internally by the management team will be the

focus of the presentation. The work of Arnold Love on the use of internal evaluation will frame the discussion. Clean

Foundation's experience will be used as an example of how internal evaluation influences program management and

the successes and challenges that have resulted.

The Role of African VOPEs in Influencing Gender and Equity Focused Evaluation by Governments

Jennifer Mutua EXPOSÉ/PAPER Africa is going through a transformation of hope and despair in equal measure. The landscape is characterized by

improved GDP growth rates coupled with new discoveries of natural wealth in some countries. Conversely, the

continent is rife with alarming increases in poverty and youth unemployment in an environment where economies

are not managed prudently. Waves of insecurity and patriarchal social order compound these. Unprecedented M&E

capacity building, including from gender and equity perspectives have been offered to individuals and governments

by international development partners. Hence, a critical mass of professionals who believe in evidence-based and

equity focused decision making and implementation is slowly growing. However, national culture and p ractice

remain weak. Equity focused evaluation can influence more prudent public resource management through advocacy

leveraging on 2015 EvalYear spearheaded by Voluntary Organizations for Professional Evaluation (VOPEs).

The Role of Evaluation in the Scale up of Population Health Interventions: The Innovation Strategy

Leslie Payne

EXPOSÉ/PAPER The Public Health Agency of Canada's Innovation Strategy (IS) supports projects to take action to reduce health

inequalities and effectively address priority complex public health problems and their underlying factors. The IS uses

an intervention research approach, meaning that the program supports the delivery a set of population health

interventions while generating critical knowledge, on (1) how the intervention process brings about change and (2)

the context in which the intervention worked best and for which populations. The IS uses an innovation 3 phase

model to assess and support the scale up interventions. This presentation will provide an overview of how the I S

model has used evaluation and intervention research to support the scale up effective population health

16

interventions in Canada and highlight a new approach to measure scale up readiness among population health

interventions.

To screen or not to screen: the impact of cancer screening letters

Emily Tsoa EXPOSÉ/PAPER Cancer Care Ontario's Evaluation & Reporting Team supports the Cancer Screening unit in evaluating the impact of

their three screening programs. These organized screening programs include the Ontario Breast Screening Program,

the Ontario Cervical Screening Program, and the Colon Cancer Check Program. Correspondence letters are currently

being sent to the general population who are due for screening. The impact of cervical screening invitation an d recall

letters, and colon cancer screening invitation letters, have been evaluated. Randomized control groups and historical

control groups were the designs used for these evaluations. Results from these evaluations will inform how future

screening correspondence should be delivered to the residents of Ontario.

Putting a ring on it: Evaluating the long-term engagement of operational staff

Joseph Travers IGNITE (PRÉSENTATION RAPIDE)/IGNITE-STYLE RAPID PRESENTATION How does a small evaluation group engage a large operational group of staff and management over a multi -year

project to ensure the goals of the evaluation are met? The Ignite presentation will cover feedback we've received

from staff and management about how we've done so far in doing this, and wh ere we need to improve.

Achievements, failures, and unexpected situations will be covered, as well as how to use this feedback to enhance

the operationalization of this project going forward.

13h15-15h15

Avons-nous réglé la question de l'utilisation des résultats de l'évaluation?

Marie Gervais, Francois Dumaine, Olivier Sossa, Andrealisa Belzer, Courtney Amo, Anne Routhier,

Mirianaud Oswald Agbadome PANEL/PANEL Au Canada, des efforts importants ont été investis depuis plus de 10 ans pour améliorer la qualité de l'évaluation et

renforcer la capacité en évaluation aux différents paliers gouvernementaux. Différents chemins ont été pris pour

convaincre les gestionnaires et les décideurs de la valeur ajoutée de l'évaluation à l'élaboration et à la gestion d es

politiques et programmes publics. Toutefois au quotidien, quel sort réserve-t-on aux résultats des évaluations? En

quoi ces résultats viennent-ils véritablement éclairer la prise de décision? L'évaluateur pourrait -il faire mieux dans sa

capacité à mobiliser autour des résultats d'une évaluation? Ce panel réunira des experts provenant de divers

contextes de pratique (gouvernement, secteur privé, milieu académique) et de différentes régions du Canada. Ceux -

ci partageront leurs perspectives sur les enjeux résiduels de cet aspect de la pratique.

14h45-16h15

APPRENDRE ENSEMBLE: L'IMPORTANCE DES COMMUNAUTÉS DE PRATIQUES EN ÉVALUATION

Raïmi B. Osseni, Erika Kaneza PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION Au printemps 2014, quelques étudiants de l'école nationale d'administration publique décident de se retrouver

régulièrement pour échanger au sujet des pratiques et du secteur de l'évaluation en mettant l'accent sur la région

de l'Outaouais. Peu à peu, cette communauté de pratique va se développer et offr ir à ses membres un espace

permettant d'approfondir leurs apprentissages et de renforcer l'accès et le partage des connaissances. Les cercles

d'apprentissage sont des « petites communautés d'apprenants qui se regroupent intentionnellement, dans le but de

se soutenir dans le processus d'apprentissage » (Collay, Dunlap, Enloe et Gagnon, 1998). La présentation de l'équipe

17

d'évalue-action propose d'explorer son modèle de développement professionnel et ceux qui l'ont inspiré, ses

réalisations, les leçons apprises, et de découvrir les avantages de créer de tels cercles au sein de milieux

professionnels.

Building Capacity in Community-Based Networks: Partnership Grants Learning Project

Heidi Schaeffer

EXPOSÉ/PAPER How can we know more about the difference that investments in building capacity with not-for-profit networks and

partnerships can make? Health Nexus and the Association of Ontario Health Centres, working with the Tamarack

Institute and with leadership from The Mowat Centre for Policy Innovation at the University of Toronto, asked just

this question in a recent evaluation. Come and hear about the Partnership Grants Learning Project that set out to

measure outcomes of a 7.2 million dollar investment in 27 not for profit community organizations by the Onta rio

Ministry of Citizenship and Immigration. An innovative new tool called Outcome Harvesting was used. Learn about

how Outcome Harvesting, inspired by Outcome Mapping, can help practitioners operating in networks and

partnerships to evaluate the social change results they are achieving through a learning oriented participatory

evaluation combining Outcome Harvesting with Most Significant Change and Social Network Analysis.

Causality, Complexity, Critical Realism, Validity: What Do These Terms Have in Common?

France Gagnon PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION It is a complex world out there. A program's causal pathway to change is rarely linear and simple. Different causal

pathways will exert a differential impact on various members of the target popu lation under similar implementation

conditions, settings, and times. Or, the program cultural, socio-economic, political context may negatively impact

outcomes in one setting but not another. And evaluation findings will have different meaning, relevance, and utility

depending on whether one is a program developer, program intervention manager, staff, community member, policy

analyst at the provincial or federal level or member of the target audience. How is one to produce evaluations that

are more credible and actionable? This presentation will review and present literature on causality, complexity,

critical realism and validity to provide suggestions for improving the construct validity of interpretations.

Contribution d’Avenir d’enfants dans le renforcement des capacités de ses partenaires en évaluation

Julie Rocheleau, Youssef Slimani

EXPOSÉ/PAPER Depuis 5 ans, Avenir d’enfants (AE) soutient financièrement plus de 130 regroupements locaux de partenaires (RLP)

sur l’'ensemble du territoire québécois. AE est une organisation apprenante qui a le privilège de chapeauter un

projet innovant et mobilisateur en faveur du développement global de la petite enfance. AE note un renforcement

de la culture évaluative auprès des partenaires engagés dans les projets qu ’il soutient. AE a misé sur le renforcement

des capacités de ses partenaires, ce qui a mené petit à petit à une adhésion aux processus évaluatifs. Ceci s ’est

traduit par le développement de nouvelles compétences, et l ’intérêt croissant démontré à l’égard de l’évaluation.

Cette présentation mettra en évidence, à travers des expériences du terrain et des exemples au niveau

organisationnel, le chemin qui a été parcouru par AE et ses partenaires dans le but de développer et consolider une

culture évaluative dans le domaine de la petite enfance.

Data collection and their methods for Impact Evaluation

Raed Zahrawi IGNITE (PRÉSENTATION RAPIDE)/IGNITE-STYLE RAPID PRESENTATION A five-year Compact ($275 million) is developed to reduce poverty and increase income in Zarqa Governorate of

Jordan, through increases in the supply of water available to households and enterprises through improvements in

the efficiency of water delivery, the extension of wastewater collection, and the expansion of wastewater treatment.

Measuring the impact of the Compact activities on economic and social outcomes was designed to establish a causal

18

relationship between program interventions and observed changes in household availability and consumption of

different sources of water, household income, and household expenditure indicators. For the purposes of

presentation, complementary data collection activities (Household, Enterprise, and farmers surveys) will be outlined

over the Impact Evaluation components, with their methods, as well as demonstrating the use of the exiting country

systems and the introduction of technology in data collection.

Determining an appropriate range: Calibrating Evaluations

Beate Schiffer-Graham, Yves Gingras, Arvind Srivastava

PANEL/PANEL Within the current federal government context of fiscal restraint and an ever increasing organizational tempo,

federal evaluators need to identify approaches that are nimble and flexible while being true to the fundamental

standards of evaluation. An opportunity that merits further discussion is the art of “calibration”, frequently used in

the world of weights and measures. The activity of “calibration” generally seeks to determine an appropriate range

to ensure that the tool or instrument conforms to a standard. For the purpose of this presentation 'calibration”

refers to evaluators decreasing or increasing the range of 1) methodologies and/or 2) scope while complying with

the standards for federal evaluations. Calibration, if done well, promises to be a partial support in providing senior

management with more timely evaluation findings at lower cost.

Developing an Evaluation Framework for Research-Oriented Communities of Practice

Kaileah McKellar, Johanne Saint-Charles, Whitney Berta, Donald C Cole, Rhonda Cockerill

PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION Communities of Practice (CoPs) are increasingly used in diverse sectors; however there is a lack of empirical

evidence of how they work. To address this gap we have developed an evaluation framework to guide evaluations of

research-oriented CoPs. To develop the framework, we used data on the context, mechanisms, and outcomes of

CoPs, gathered from a systematic scoping review that drew from databases in health, business, and interdisciplinary

fields, as well as key informant interviews with CoP members. The framework includes suggested approaches,

measures, methods as well as a program theory. The presentation will outline methods used in developing the

framework, highlight the key mechanisms that lead to social learning and act ion, discuss the level of evidence of key

mechanisms and offer a series of indicators aligned to the program theory. The presentation will conclude with a

proposed approach to testing the evaluation framework, including key mechanisms, in different context s.

Developing indicators to facilitate evaluation of Age-Friendly Communities initiatives across Canada

Kara Haynes

EXPOSÉ/PAPER Public Health Agency of Canada developed an evaluation guide Using Indicators in Your Age -Friendly Community

Initiative, for over 900 communities in Canada that are implementing Age-Friendly Communities Initiative (AFC). The

objective of this project is to facilitate the uptake of evaluation by the communities implementing AFC initiatives.

Hence this resource targets community organisations, seniors, municipal officials, program planners and policy

makers responsible for implementing AFC. The guide provides an introduction on evaluation, and how to design an

evaluation framework using logic models. Since there is a large variation in the way AFC initiatives are implemented,

this guide provides flexibility to choose from a menu of indicators relevant to the programs, and relevant assessment

tools both qualitative and quantitative.

Evaluating Complex Programs

Jacques Bérard

EXPOSÉ/PAPER Government and philanthropic organizations that aim at producing systemic change work at many levels, with many

actors using multiple strategies. Think of the UN's Millennium Goals. Programs and policies that address complex

systems interacting on horizontal planes, and vertically at national, sub-national and local levels pose a challenge to

19

evaluators. How does one evaluate the impact, or the performance, of a program when so many other partners are

working towards (or against) the transformation that is sought after? This presentation will address some of the

challenges that evaluators face in dealing with complex programs. It will attempt to provide elements of solutions by

looking at monitoring and evaluation approaches and methods borrowed from the field of complex adaptive systems

and developmental evaluation.

Evaluating research team processes for better research implementation

Susan Roelofs, Cody Anderson

EXPOSÉ/PAPER A Canadian team leads the evaluation of the five-year European Commission-funded REPOPA Project (Research into

Policy to Enhance Physical Activity), which studies the integration of research evidence with real -life policymaking in

six European countries: Denmark, Finland, Italy, the Netherlands, Romania, and the United Kingdom. Annual process

evaluations use mixed methods with five distinct tools: a collaboration survey; a social network mapping survey;

document review; a junior researcher research competency self-assessment; and semi-structured focus groups. The

participatory, utilization-oriented evaluation incorporates innovative strategies to stimulate the use of findings for

continuous improvement and consortium decision-making. Consortium members provided input into evaluation

guiding principles and design; into internal monitoring reports; and through targeted discussion at annual meetings.

Two cycles of process evaluation have been conducted thus far.

Évaluation développementale d'une plateforme web pour le transfert de connaissances

Esther Mc Sween-Cadieux, Christian Dagenais

PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION La Fondation du Dr Julien travaille actuellement au développement d'une plateforme web pour le transfert de

connaissances dans le but de faire connaitre et reconnaitre leur modèle de pédiatrie sociale en communauté. Une

section importante de cette plate-forme est dédiée aux divers intervenants des centres de pédiatrie sociale partout

dans la province pour les aider à améliorer leurs pratiques via des modules de formation en ligne. L'approche

d'évaluation développementale (Patton, 2011) encore peu utilisée, permettra de suivre pas-à-pas le développement

de la plateforme en soutenant la Fondation du Dr Julien à faire certains choix durant le processus de création et

d'expérimentation des modules. En se basant sur des entretiens avec les acteurs impliqués dans le processus

d'évaluation et sur notre propre expérience, les conditions de succès de cette approche pour faciliter le

développement d'une stratégie de transfert de connaissances seront énoncées.

Evaluation Standards in the Spanish Speaking World: Evolución y desafíos!

Mariane Arsenault, Esteban Tapella, Pablo Rodriguez- Bilella, Maria Bustelo PANEL/PANEL The development of evaluation standards has been considered an indicator of the increasing relevance and

professionalization of the evaluation practice. This panel will introduce a general framework on the usefulness of

evaluation standards in order to strengthen an evaluation culture, with illustrations drawn from the experience of

Spain and the Latin America region. A particular focus of this panel will be on explaining the role played by Voluntary

Organizations for Professional Evaluation (VOPEs) in the development of standards, and the main challenges faced in

this endeavour. The panel discussions will take place in Spanish; this is a first at a CES Conference and is highly

coherent within the context of the 2015 Year of Evaluation.

Intervenir en périnatalité et en petite enfance pour améliorer la santé et le bien-être des familles

Nathalie Dubois,

EXPOSÉ/PAPER Intervenir en périnatalité et en petite enfance pour améliorer la santé et le bien -être des familles vulnérables:

Évaluation de la mise en oeuvre et des effets du projet de La Maisob Bleue. Depuis 2007,La Maison Bleue offre des

services en périnatalité et petite enfance dans le but d’aider les femmes et leur famille à accueillir leur bébé. Face au

20

développement grandissant de La Maison Bleue et aux multiples demandes formulées par divers acteurs quant à la

possibilité de répliquer le projet dans d’autres contextes et milieux, une recherche évaluative visant la production de

connaissances crédibles et valables s’est avérée nécessaire. Le présent rapport de recherche vise à répondre à ce

besoin. Il a comme objectif d'appréhender le modèle d’intervention privilégié par La Maison Bleue, d'analyser sa

mise en oeuvre et d'évaluer ses effets.

Is it time to ditch the deck?

John Burrett

IGNITE (PRÉSENTATION RAPIDE)/IGNITE-STYLE RAPID PRESENTATION John will argue that the briefing deck is the most ineffective method possible for communicating findings and

briefing senior officials. He will back this up with an analysis by Edward Tufte of how the reliance on PowerPoint

briefings led to the poor decision-making behind the Columbia space shuttle disaster in 2003. The presentation will

finish with suggestions being advanced by communications experts for better briefings.

Lancement officiel de la "trousse de renforcement institutionnel des VOPEs » (VOPE toolkit)

Marie Gervais, Jim Rugh, Ziad Moussa LANCEMENT/LAUNCH

Networking to navigate the Value for Money waters: An experience of DFID-Nigeria VFM forum.

Oluwatosin Akomolafe

IGNITE (PRÉSENTATION RAPIDE)/IGNITE-STYLE RAPID PRESENTATION Value for Money has been around for awhile but has lately been a requirement by donors. It has now become

imperative that value for money is shown from the bidding/tendering process through the project cycle to impact. At

times, programmes/projects do not understand the VFM requirements of the donors; have conflicting underst anding

of requirements and approach to mainstreaming, assessing and reporting VFM. Also, Project/Programmes at

different phases of implementation have encountered different challenges in showing value for money. New

programmes are particularly concerned about how to start, while older ones are uncertain of how to introduce VFM

mid-way into their projects. Networks have been found to be useful source for building capacity especially for

beginners where everyone can learn in a non-threatening environment and leverage resources amongst others. The

DFID-Nigeria VFM forum is a network of DFID funded programmes and it is aimed at supporting members.

Obtaining young children's assent to participate in research

Nick Petten IGNITE (PRÉSENTATION RAPIDE)/IGNITE-STYLE RAPID PRESENTATION The methodology of research is critical in determining any truth claim that a research study posits. In researching

childhood, a critical methodological concern is the power dynamics that occur between adults (who are typically the

researcher) and children (who are typically the subject). In order to make any truth claims based on the scientific

process, careful consideration must be given to the ethics of involving children in research so that power dynamics

can become more equalized. This process begins with obtaining children's assent to participate in the research

process. As an adult researcher, the process of obtaining children's assent rests on the recognition that children are

competent social actors who are capable of making sense of and affecting their societies. From this recognition,

adult and child researchers can begin the design and sense-making activities of research that will enhance any

discoveries made about childhood.

Paternalists internationaux: Défis et retombées

Marie Gervais, Jim Rugh, Serge Eric Yakeu, Ziad Moussa, Ian Hopwood, Fation Luli PANEL/PANEL Avec l'appui de EvalPartners, divers partenariats ont été développés depuis 2013 entre sociétés nationales

d'évaluation actives sur plusieurs continents dans le cadre des concours Peer-to-Peer et Innovation Challenge.

21

Comment ces partenariats ont-ils été développés ? Quels étaient les objectifs recherchés? Quels défis ont été

rencontrés? Quelles retombées et quels apprentissages peuvent maintenant être dégagés? Quelques représentants

partageront leurs expériences et expliqueront comment ces partenariats ont contribué à renforcer les capacités

institutionnelles de leurs sociétés et à soutenir la réalisation de leur mission, incluant la promotion de l'évaluation

au pays. Dans le cadre de 2015 Année internationale de l'évaluation, voilà des exemples probants de l'effet

catalyseur de la mutualisation d'expériences. Nos panélistes s'exprimeront finalement sur les priorités de l'Agenda

global de l'évaluation 2016-2020.

Process Evaluation of Antiretroviral Therapy Program in Southwest Ethiopia: a case study

Fasil Walelign Fentaye

EXPOSÉ/PAPER The evaluation assessed the implementation fidelity of ART program at Sekoru Health Center, Southwest Ethiopia.

Case study design with mixed methods was employed from June 1 to July 30, 2012. Client-exit interview, document

review, resource inventory and direct observation of provider-client interactions were the data collection

techniques. Criteria set together with the program stakeholders were used to judge the implementation. Fidelity of

the program's implementation was 61.3%. Adherence to the guideline was 54.6%; quality was maintained for 73.7%

of the services delivered; and clients' responsiveness was 83.2%. Implementation fidelity of t he program was low

despite higher participant responsiveness and enabling contextual factors. ART services at the health center are in

need of urgent improvement and the evaluation framework showed program implementation from provider &

participants perspective.

Process, Summative, Impact or something else: a hybrid approach of evaluation in non-profit sector

Faisal Islam EXPOSÉ/PAPER Amidst growing demands on non-profit organizations to show the evidence of their outcomes, there has been

always desire to conduct evaluation to demonstrate the results. But many NGOs do not have the luxury to wait until

the projects end and then plan for impact evaluation. Several pressures warrant NGOs to conduct evaluation early —

while the projects are still being implemented —to know the results. With rapidly changing priorities and low

resources dedicated to M&E, NGOs to conduct evaluation that can serve all purpose at once: show the impact;

improve the processes; fulfill the donor requirement; scale up the projects; & bett er understand the needs of

beneficiaries. Drawing on a recent experience of a multi-country evaluation, this paper attempts to explore how a

hybrid approach to evaluation, using the elements of ‘utilization -focused’ and ‘real world’ evaluation, can provide a

framework of evaluation for multiple audiences and a variety of objectives for NGOs & Civil Society.

The Impact of Gaucho Winter Program on Older Adults Respiratory Infection in Southern Brazil

Marilia Ramos

EXPOSÉ/PAPER This article presents results of the evaluation of a project developed by Rio Grande Health Secretary: the Inverno

Gaucho Project, which intends to reduce the hospitalization of older adults due to respiratory infection. The

evaluation involved counties that participated in the project in comparison to those counties that didn't participate,

to control for characteristics that can affect the hospitalization due to respiratory infection rates (dependent

variable) that are not related to the project. Descriptive analysis, multivariate regre ssion models and the propensity

score matching techniques were used. It was observed that the greatest mean difference among participants and

non participants happened during the year 2008. Participants presented lower hospitalization than non participants .

Useful Theories of Change: Purposefully Building Reach into Program Theory

Steve Montague, John Mayne

PANEL/PANEL

22

This coordinated panel will suggest the need for theories of change that can be actually used for planning and

managing interventions and their evaluations, especially the need to build reach (individuals, groups and institutions

a program or initiative engages with to achieve mission) into the logic models, program theories and results

frameworks of essentially all initiatives. The panelists make the case that theories of change can and should be

practical tools and that theories of reach should be considered part of program theory. The point is made that a lack

of explicit thinking about reach in theories of change and logic models can lead t o problems such as

narrow/constricted understanding of results chains, favouring 'narrow and efficient' initiatives over 'wide and

engaging' initiatives, taking for granted intermediaries co-delivering an initiative and creating a bias against equity

considerations. Audience engagement will be encouraged in this session.

Using evaluation to improve emergency response to homeless persons who abuse substances in Montreal

Kristelle Alunni-Menichini IGNITE (PRÉSENTATION RAPIDE)/IGNITE-STYLE RAPID PRESENTATION Homeless people who abuse substances are facing multiple challenges on their trajectories of health care services.

Our objective is to identify ways that would improve the emergency response by an approach that rally all keys

actors (i.e. experts committee, health providers, community stakeholders, police officers and homelessness persons

with substance disorder). We choose a research design inspired from the deliberative democratic, participative and

empowerment evaluations. It will allow us to engage participants in several moments of exchange in order to

disseminate credible/objective information, to develop a common understanding, and to discuss and validate our

findings. More precisely, the actors' needs and several interventions will be evaluated. Then, these interventions will

be prioritized according the appropriateness, the actors' needs and the context. Finally, the dissemination of results

will be conducted among the keys actors and the community.

Using program evaluation to impact health human resource policy development

Kathryn Hodwitz, Wendy Yen, Joseph Travers

EXPOSÉ/PAPER The College of Physicians and Surgeons of Ontario (CPSO) regulates the practice of medicine by providing licenses to

physicians and monitoring standards of practice. Over the last 15 years, the CPSO has developed numerous

registration pathways and policies to reduce physician shortages and increase access to licensure. In 2012, the

College embarked on a program evaluation of its registration pathways to ensure that the policies are meeting their

intended purposes, namely facilitating entry to the profession for qualified doctors. The evaluation seeks to

determine if performance differences exist between physicians registered through the traditional pathway (those

who completed post-graduate training and examinations in Canada) and alternative pathways (all other physician

applicants). The methodological challenges and lessons learned for a complex evaluation will be presented, as well

as the ways in which the evaluation data will inform future policy development.

Walking the Walk: Building Evaluation Capacity through a Community of Learning, Inquiry and Practice

Chris Lovato, Derek Wilson, Kylie Hutchinson, Beverly Parsons

EXPOSÉ/PAPER Improving staff knowledge and skills in evaluation can contribute significantly to creating a culture of evaluation that

will facilitate success. However, organizations are often challenged by limited resources for evaluation capacity -

building. Communities of Learning, Inquiry, and Practice (CLIPs) i s a simple, effective, and cost-efficient model that

contributes to evaluation capacity building by increasing the knowledge, skills, and understanding of evaluation by

individuals within an organization. This presentation will describe the implementation and evaluation of a CLIPs

process in a university medical education setting. We will share our experience and insights about how the CLIPs

process can successfully build and support a culture of evaluation.

Walking together to make a path: developmental evaluation with a non-profit women's organization

Alma Estable, Mechthild Meyer, Raine Liliefeldt

23

IGNITE (PRÉSENTATION RAPIDE)/IGNITE-STYLE RAPID PRESENTATION We'd like to share the story of what happens when a small evaluation firm committed to social chang e works with a

national non-profit women's organization in a developmental evaluation process- what we learned, what we forgot,

what we wish we had done, what we managed to achieve, and where we ended up. Why? Because others might

learn from an approach to evaluation that attempts to combine, in a very practical way, the diverse skills, expertise,

reach, and mandates of a team of evaluators with that of a team of feminist organization workers , who all share a

common vision for a more just and equal society.

Working with Assets to Build Capacity: Learning Circles and Appreciative Inquiry in Continuing Care

Gail Vallance Barrington, Sandra Woodhead Lyons, Sharla King

EXPOSÉ/PAPER The learning circle is a capacity-building activity based on well known concepts in adult learning, reflective learning,

and quality circles but the impact of this low-cost learning model has not been well studied. A 2013 pilot in three

continuing care sites in the Calgary area was the base for the current study of 13 learning circles in seven continuing

care organizations across Alberta. The pilot evaluation was grounded in Appreciative Inquiry (AI) (Preskill and

Catsambas, 2006) and while small in scope, it documented evidence of knowledge transfer and practice change.

However, as Alschult (2014) asked recently, how can we embed a mindset in an organization or community to utilize

its strengths for growth and improvement? Scaling up the evaluation processes from the pilot for this larger study

has provided some unique issues and challenges but AI continues to prove its utility in terms of understanding

emergent change. Some study tools and early observations will be shared.

Innovative approaches for dialogue: Photovoice and World Café / Dialoguer: Les approches innovantes du

Photovoix et du World Café

Photovoice

Christine Loignon, Karoline Truchon, Chris Corrigan, Amy Lenzo, Moderated By Huilan Krenn, W.K. Kellogg

Foundation PLÉNIÈRE/PLENARY

15h15-16h15

Benefits and Limitations of SROI Analysis

Anne Miller DÉBAT/DEBATE Social Return on Investment (SROI) is a growing evaluation methodology that seeks to move the value assessment

process from a focus on cost alone to one that focuses on social value creation from a stakeholder perspective. SROI

assigns financial proxies (monetary value) to social, environmental and economic outcomes, to try to understand the

value of achieving outcomes as compared to the cost of investing in an initiative. While SROI creates interesting new

information about many projects, it is not necessarily the best evaluation method for every situation. This session

will explore the benefits and limitations of the SROI approach, fostering a discussion on where this methodology fits

within the Canadian evaluation context.

16h15-16h35

Navigating Hostile Waters: Using Evaluation to Achieve Rapprochement

Cynthia Tananis EXPOSÉ/PAPER The context: Closure of two (K-8) community schools and reopening one as a combined facility. The issues: How do

you honor the two communities (each with a strong identity) and each school with long histories and experienced

staff --- and merge them to a new entity? The problem: The merger occurred in a top-down hierarchical way with

24

minimal consideration of the complex issues. This paper describes how external "evaluation" was used to create

space for shared values and shared decision-making after the damage had been done. Explore how evaluation can be

used as a healing force by finding common ground, shared values and vision, and a vision to guide action.

16h35-17h35

All Things Equal? A Health Equity Lens for Evaluation

Ashley Zelmer, Kelly De Cecco EXPOSÉ/PAPER In this presentation, participants will learn about the Health Equity Lens for Evaluation, a tool developed by the

Office of Evaluation at the Public Health Agency of Canada and Health Canada to help ensure that health equity

considerations are taken into account by programs. The presentation will focus on the development of the tool, how

it can be applied through the evaluation process, in particular as the issues of relevance and pe rformance are

explored, and how it can be used to help guide decision-making. The presentation will also touch on the importance

of addressing health equity concerns in order to help create the world we want; a world with healthier Canadians.

This promising tool helps evaluators identify areas of health equity concern, which can lead to improved programs

that impact the health of all Canadians, including those most at risk.

Analyse d'un processus délibératif pour renforcer la politique de gratuité des soins au Niger

Ibrahim Hamani Souley

PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION En 2006, le Niger a mis en place une politique de gratuité des soins pour les femmes enceintes et les enfants de 0 à 5

ans. En 2012, un processus délibératif fut organisé pour échanger sur les acquis, limites et perspectives de cette

politique. L'objectif de cette recherche est de comprendre les effets du processus ainsi que d'explorer les activités

du comité de suivi de la feuille de route. La recherche a été réalisée en 2014 à Niamey. Elle a reposé sur l'utilisation

du cadre conceptuel de Boyko et al., (2012) avec un accent particulier sur les trois formes d'utilisation des

connaissances présentées Dagenais et al., (2013): instrumentale, conceptuelle, persuasive. Des entretiens semi-

directifs ont été effectués. Les résultats révèlent une utilisation plus instrumentale des recommandations et une

utilisation conceptuelle et persuasive à un degré plus faible. Le comité de suivi de la feuille route de la conférence

n'a pratiquement pas fonctionné.

Assessing and attributing research impacts using novel approaches: an update

Christopher Manuel EXPOSÉ/PAPER A pilot study was undertaken in 2014 to examine the utility and feasibility of adopting an independent, rigorous and

replicable methodology for identifying the influence research supported by the Canadian Institutes of Health

Research has had on decision making in the United States. We entered into a collaboration with the American

National Guidelines Clearinghouse to match approximately 10,000 CIHR supported articles from 2008-2010 with the

scientific literature cited in the clinical guidelines and quality measures (and supporting documentation) included in

the NGC databases. Matches were then reviewed to determine the strength of the influence the CIHR supported

research had on the development of the guideline/quality measure. This presentation will describe the methodology

used, the results and discuss lessons learned.

Assessing the Economic Impact of Snowmobiling in Ontario - Different Models / Approaches

Harry Cummings, Shannon McIntyre, Don Murray EXPOSÉ/PAPER The Ontario Federation of Snowmobile Clubs (OFSC) promotes snowmobiling experiences that are safe, enjoyable

and environmentally sustainable. More than 200 community-based OFSC member snowmobile clubs operate more

than 30,000 km of signed and groomed trails for use by approximately 100,000 snowmobile trail permit holders and

25

their families. Snowmobiling generates significant economic benefits in Ontario and these impacts ha ve been

examined several times over the last 15 years using three different models / approaches. The most recent study

completed in 2014 utilized the Tourism Regional Economic Impact Model (TREIM) in conjunction with an online

survey of over 6,000 snowmobilers to provide the most complete assessment of snowmobiling activity in Ontario to

date. This session will compare and contrast the different models used to assess the economic impact of

snowmobiling in Ontario.

Assessing the reach of information and R&D services using social network analysis

Adele Acheson, John Burrett

EXPOSÉ/PAPER As the globalization of information increases, so does the importance of looking at how information and research

products are disseminated. This is a critical issue in assessing the effectiveness of public R&D. Network analysis has

existed as a methodology for some time in fields such as sociology, anthropology and biology, providing a rigorous

means to understand how people, organizations, organisms, and so on are connected. More recently its application

has expanded to the analysis of networks of communications such as in social media. This technique has seen little

use in evaluation to date, but we contend that as both the scale and potential for transmission of information as a

product of public research grows, its attractiveness as an evaluative methodology becomes more and more

apparent. This presentation discusses the use of social network analysis to evaluate the reach and impact of

research and knowledge, undertaken by the National Research Council Canada.

Assessment of stakeholders participation in local government project evaluation: a case study of saw

Nurudeen Mohammed Aliu

PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION The study entails a systematic examination of the actual level of primary stakeholders' involvement in project

evaluation and comparing it with the actual evaluation practices in the district. It also investigated the factor

influencing participatory evaluation in the district. The communities studied inclu de Sawla, Tuna and Gindabour with

a sample of 40 primary stakeholders selected through proportional random sampling method. The study was

facilitated through a desk review of available literature and the employment of Participatory Rural Appraisal (PRA)

tools which included key informant interviews, focus group discuss and household questionnaire interview were

employed for this study. The data were analyzed by the use of descriptive statistics such as frequency and cross -

tabulation using Microsoft excel. Dependent and independent variables such as institutional arrangement, planning

process and evaluation process were critically reviewed

Bottoms up or top down: successes and challenges in M&E of large policy frameworks

Jennifer Young, Katy Pollock, Bridget Hall, Amanda White EXPOSÉ/PAPER Large policy frameworks often drive service and program delivery for end users. A recommended practice is to

attach a monitoring, evaluation and accountability (MEA) process to the policy framework, in addition to program -

specific MEA processes to the programs under the policy. This facilitates the use of data for program-level and

umbrella policy-level monitoring and evaluation purposes. A common question arises with whether to develop the

policy-level MEA framework first, or to begin by developing MEA plans for individual programs. The former

establishes clear guidelines for all programs early in the process, enabling easier "roll -up" of data, while the latter

ensures that the policy framework will be relevant to the specific needs of the individual programs. We will discuss

the successes and challenges of both approaches using 3 NWT examples, and provide recommendations for users

involved in the monitoring and evaluation of large policy frameworks.

Comment évaluer la collaboration et la cohésion des parties prenantes ?

Eva Anstett, Vénétia Sauvain EXPOSÉ/PAPER

26

Plusieurs interventions publiques rassemblent plus d'un acteur agissant sur une problématique donnée (plusieurs

organisations publiques, milieu communautaire, secteur privé, organisations citoyennes...). L'objet de ces

interventions publiques consiste souvent à faire travailler ces personnes ensemble afin d'améliorer l'efficacité de

leurs actions sur le terrain. Dans un contexte de rareté des ressources, où on che rche de plus en plus à faire mieux

avec moins de moyens, il est important que l'on puisse évaluer si une intervention publique peut permettre une

meilleure cohésion entre différents acteurs ce qui peut aussi avoir ultimement un impact sur les cibles finale s des

actions. Nous avons utilisé une méthode pour évaluer cette cohésion dans plusieurs de nos dossiers (échelle de

Metcalfe) et souhaitons partager cette méthode tout en étudiant ses avantages et ses inconvénients.

Complementary instrument for the identification of proxy indicators for research & training programs

Véronique Dugas

EXPOSÉ/PAPER Mitacs is a not-for-profit organization promoting innovation through the delivery of research internship and training

programs in Canada. As with many innovation-focused programs, some Mitacs program outcomes are diffuse, slow-

acting and multi-dimensional. Recently, Mitacs has integrated the use of longitudinal studies to complement the

information captured through its Accelerate program exit surveys. Th is strategy has proven useful as a means to

complement the data obtained shortly after the end of the internship and to better define long -term impacts of the

program. Of interest, comparison of data collected through program exit surveys vs. longitudinal questionnaires also

allows for the identification of appropriate short-term proxies that go beyond usual research outputs. In conclusion,

the use of distinct instruments used at different time points allows for a more complete picture of Accelerate's

outcomes for its participants and the broader research and innovation ecosystem.

Connecting Evaluation and Policy Imperatives: Mitacs as a Case Study.

Valerie Walker

EXPOSÉ/PAPER Promoting innovation is a significant focus for governments seeking to improve productivity and growth in a

competitive global economy. As such, programs that promote and support innovation are key parts of government

economic policy. Through its various programs, Mitacs has become an integral part of Canadian innovation policy

during the last fifteen years. Its remarkable growth has largely been driven by effective use of evaluation to both

direct program design and to ensure effective alignment with broader policy objectives. This presentation will

describe Mitacs' success in aligning innovative program evaluation with the broader policy objectives of government

funders and to support timely evidence-based government decision-making. It will focus on how we built a

reputation as credible, collaborative, and innovative by using outputs and outcomes to drive program improvements

while simultaneously mastering how to effectively communicate impacts to government decision makers.

EVALUATION DES CAPACITES DES ORGANISATIONS LOCALES REPERTORIEES DANS LE CADRE DU PROJET DPP

Jean Paul Ngantchou Nkwangue EXPOSÉ/PAPER Le partenariat public-privé est devenu au fil du temps un instrument additionnel de financement du développement

durable. La GIZ, Coopération allemande pour le développement en partenariat avec la société AES - SONEL, ont mis

sur pied cette forme de coopération dont l’objectif est d’améliorer les conditions socio économiques des

populations vivant dans les zones situées à proximité du plus important barrage hydroélectrique exploité au

Cameroun. Ce partenariat tourne autour d’un projet intitulé Partenariat pour le développement durable dans

l’arrondissement de Massok-Song Loulou et Pouma. Il a pour objectif d’apporter un appui technique aux

organisations économiques locales à la production et à la commercialisation de la banane plantai n. l'évaluation Des

études de faisabilité, ont démontré de manière générale, la caducité des organisations par la récurrence de certains

dysfonctionnements liés aux capacités organisationnelle, institutionnelle et opérationnelle.

27

Evaluation meets ancestral midwives: how real is evaluation getting with intercultural policy?

María Luisa Calderón

EXPOSÉ/PAPER In the last 4 years in Guatemala, Healthy Maternity Law was published and took effect and Indigenous Midwives

inclusion policy is in the making due to maternal health becoming a matter of national urgency; the country has the

3rd highest maternal mortality in Latin America. The study analyses evaluation theory applied to policymaking of the

inclusion of indigenous midwifes from the four nations in Guatemala: Mayan, Garifuna (African descendents), Xinca

and Mestizo (mixed); as well as its challenges and opportunities.

Evaluation of the Ivirtivik Centre - Lessons Learned from Learning Together

Donald Murrary, Shannon McIntyre, Valérie Roy, Loïc Lenen EXPOSÉ/PAPER Inuit in the South face challenges which create significant barriers to employment (Aboriginal Skills and Training

Strategic Investment Fund, 2013). The Ivirtivik Centre is an employability and skills development initiative for Inuit in

Montreal. An evaluation of the Centre found that the program has made progress and continues to evolve from a

partnership of organizations that worked to introduce a structured program that encourages progressive

development of employability skills while being sensitive to the challenges faced by Inuit. The Centre has helped

participants build their self-confidence and connect them to the community through various activities that combine

their values, talents and preferences. The Centre has positively changed the parti cipants' employability and their

quality of life. The presentation focuses on the evaluation of the Centre and its contribution to the world we want

for the Inuit living in the South.

Facilitators and Barriers of Patient and Family Engagement in Health Service Improvement

Nathalie Gilbert, J. Bradley Cousins

EXPOSÉ/PAPER The underlying philosophy surrounding patient-centred care advocates for patients and family members to have an

active role in all areas of their care, including broader areas of the health care system such as in evaluation. Despite

recent efforts in the past few years that would see greater patient and family engagement in health service

improvement initiatives, there remains a lack of best practices and understanding about the process of en gaging

patients and family members in evaluations within the health sector. The Ottawa Hospital Psychosocial Oncology

Program (PSOP) recently completed a program review and needs assessment with the PSOP Patient and Family

Engagement Committee; composed of patients, family members, a program manager, health professionals and an

evaluator. In conjunction with this program improvement initiative included a corresponding exploratory study

examining the process of patient and family engagement in a health service improvement project. This study

employe

How to ensure the evaluability of the future Regional Public Health Plan before its planning begins?

Yassen Yordanov, Nathalie Dubois EXPOSÉ/PAPER Although increasing attention is paid to the planning of the future Regional Public Health Plan in Montreal (RPHP),

the question of its evaluability when this plan is still in phase of planning raise particular theoretical and

methodological issues. The objective of the presentation is to share the theoretical and method ological knowledge

acquired during this journey. First, published literature regarding evaluability assessment (EA) was reviewed to

propose a taxonomy of the most important purposes for EA and EA outcomes. Secondly, based upon the taxonomy

results, the presentation summarizes the methodological and strategic approach developed by the team. The

proposed approach suggests that it is possible to support a more systematic consideration of criteria for a

meaningful evaluation in linking past EA knowledge that might be generated from an EA to a further RPHP or policy

planning process. And finally, recommendations to enhance RPHP planning and meaningful evaluation are offered.

28

Impacts of evaluation on regulatory programs and its contribution to enhanced governance.

Perfecto Vélez Macho

EXPOSÉ/PAPER Participants will be presented with an overview of changes stemming from observations and recommendations

linked to the evaluation of federal, health-related, regulatory programs. Continuous program engagement as an

approach leading to internal program reflection, discovery and enhanced evaluation results will be discussed in the

context of evaluation influence together with the role of evaluators and evaluation units as “knowledge

sharers/communicators” of evaluation results. The role of evaluation as a tool leading to improved program

understanding, enhancement and greater accountability will be discussed in the context of better governance.

Involving youth in a participatory and developmental evaluation

Nick Petten EXPOSÉ/PAPER Programs and services designed for youth have traditionally excluded the participation and voice of its main

stakeholders and beneficiaries. This often results in poorly designed interventions that further exclude youth and

misses opportunities for empowerment. A youth participatory and developmental evaluation was conducted for a

youth mentorship program designed for young community leaders that have started their own arts -based, social

enterprises in downtown Toronto. Youth in the program were encouraged to participate in the evaluation through

various methods including obtaining youth assent, reflective discussions, creating accountability systems, and

assuring confidentiality and appropriate use of youth voices. This paper presents the lessons learned using various

strategies and approaches to working with youth, as well as, the systematic and ethical strategies of maximizing

participation and appropriate use of evaluation findings.

Knowledge Transference in Evaluation Capacity Building: Is Client Intent Necessary?

Melissa Rogan

EXPOSÉ/PAPER For fifteen years, the topic of evaluation capacity building (ECB) has received considerable attention in the valuation

literature (Boyle & Lemaire 1999, Leviton 2001, Preskill & Boyle 2008). Despite some variation in ECB models, one

consistent ECB tenet is it is always intentional, both on the part of the evaluator and the client. However, if the goal

of ECB is to transfer the knowledge, skills, and attitudes necessary for quality evaluation an d its use to the client, is

intent on the part of our clients necessary for that transference to occur? In this presentation a modified version of

ECB will be introduced, and the potential benefits for the discipline will be discussed.

L'évaluation économique d:une intervention complexe en santé publique

Lynda Benhadj PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION Dans un contexte de vieillissement de la population, de compressions budgétaires et de maîtrise des dépenses dans

le domaine de la santé et des services sociaux, la fonction d'évaluation économique des programmes prend tout son

sens. Dans le domaine de la santé publique, les programmes qui s'adressent aux populations vulnérables, sont

souvent qualifiés de « complexes » vus leurs effets multiples, diffus et parfois intangibles. L'évaluation économique

commence à s'intéresser à des outils d'analyse du rendement de tels programmes. Dans le cadre de ce travail

doctoral, nous utilisons une méthodologie adaptée pour évaluer la Maison bleue, une ressource offrant des services

de périnatalité sociale à des femmes enceintes cumulant plusieurs facteurs de vulnérabilité. Les objectifs poursuivis

seront: 1- décrire l'intervention complexe; 2- estimer les coûts de mise en place de cette intervention; évaluer les

conséquences; 4-analyser son rendement potentiel.

Not just stories: monitoring outcomes to inform program management and evaluation'

Marie-Caroline Badjeck, Pamela Kertland, Mary-Ann Wilson, Jennifer Ardiel EXPOSÉ/PAPER

29

Evaluating program contributions to long term outcomes can be challenging and in the case of adaptation, additional

challenges include the long-time horizons needed for adaptation planning and implementation, and the cross -

sectoral nature of climate change impacts. The Climate Change Impact and Adaptation Division has adapted the

"Most Significant Change" and "Outcome Harvesting" approaches to collect stories of change focusing on 1) what

change happened, 2) when and 3) why. This paper presents how the two approaches we re implemented and offers a

range of insights on how to collect monitoring data that can contribute to impact evaluations as well as how to use

new technologies to better disseminate results. The experience presented in this paper illustrates the usefulness of

result-based monitoring at the program level, and the complexities associated with establishing causal relationships

for programs addressing cross-sectoral issues such as climate change.

Open program evaluation -- collective solutions to collective challenges

Andrew Dzuba EXPOSÉ/PAPER The world we want is one of increased transparency, greater citizen participation, and instantaneous adaptation to

change. Open program evaluation practice is the next step in participatory models of evaluation, in that i t brings

together all of the individuals, professionals and institutions interested in assessing a program -- disrupting the

current 'one-funder, one-program, one-evaluator' paradigm of program evaluation. By encouraging the development

of communities of practice leveraged by social media, open program evaluation practices can be created that

promote transparency and democratic engagement, thereby flattening disparities of social class and power that exist

between institutions, professionals, and program recipients. Ultimately, open program evaluation practices promote

collective solutions to collective challenges.

Peace, love, and evaluation: the pursuit of social justice

François Dumaine

CONFÉRENCE D’EXPERT/EXPERT LECTURE As the field of program evaluation emerged, theorists came along to challenge practitioner in clarifying their

fundamental purpose. Is program evaluation largely limited to applying quantitative and qualitative methods in

order to better understand what a program or intervention is about? Or is there something more to it? Early

theorists such as Ernest House and, more recently, Jennifer Greene, have emerged to challenge evaluators to

embrace a path of social justice, including truth, beauty and justice. For Canadian evaluators, the fundamen tal

question around evaluation purpose remains highly relevant. Is evaluation about gathering and reporting on data

gathered through well-established methods, or is it about determining how evaluation can best support collective

goals related to social justice? Using current literature, the discussion will invite participants to further reflect on the

dynamics that drive evaluation and the role they can play in facilitating a discussion around the vi

Socio - economic empowerment in city slums, The case of Nairobi adolescent girls

Phelix Odipo, Jaboma Allan EXPOSÉ/PAPER In 2012, Nairobi Youth Centre implemented a girls' empowerment livelihoods project in collaboration with

International Rescue Committee with grant support from the Nike Foundation. The project goal was to promote

socio-economic empowerment of adolescent girls in Kenya through sustainable micro franchise projects. At the end

of the project assessments were carried out in the slums to determine the impact of the project on adolescent girls

and their livelihoods. A total of 125 girls were surveyed through individual questionnaires and focus group

discussions. The methodology involved qualitative and quantitative evaluation methods. Key assessment findings

were that female mentors boost girls' participation. Savings are important to girls, Girls are motivated by training

that respond to their distinct needs. Livelihoods reduce significantly the likelihood of early marriages, gender -based

violence and motherhood

30

Terminological dictionary project in evaluation: an attempt at clarity

Ghislain Arbour, Richard Marceau, Francine Sylvain

EXPOSÉ/PAPER Terminology is the scientific study of the specialized language associated with a domain of activity, like law or

medicine. It proposes methods and criteria to develop and improve a language by supporting the selection of its

terms and the quality of its definitions. Program evaluation, as a domain of activity, requires a clear specialized

language not only to facilitate seamless communication among the specialis ts (the scholars and practitioners), but

also to deliver consistent and unambiguous evaluation products. Sources of evaluation terms and definitions are

abundant, but sometimes can generate confusion and lack consistency. This presentation explains the relevance,

methodology and the characteristics of the project of a dictionary of evaluation in English that utilizes the teachings

of terminology to address a fundamental challenge of communication in our discipline.

The basics of system evaluation: A case illustration

Ralph Renger CONFÉRENCE D’EXPERT/EXPERT LECTURE Systems evaluation is receiving considerable attention in the evaluation literature. In 2012 the GIZ in Germany

sponsored a conference devoted specifically to understanding the potential of syste ms evaluation. Conclusions from

the proceedings were mixed; while it was acknowledged systems evaluation has great potential there was no shared

definition of systems evaluation. Another criticism levied against systems evaluation is it is too abstract, wh ich is a

major limiting factor in bridging from theory to practice. Three years later there has been very little progress in

applied systems evaluation. This presentation will share the systems evaluation model developed by the author born

out of meeting the challenges faced in evaluating the cardiac systems of care for seven states. Robust system

evaluation principles will be shared and illustrated using examples from the cardiac system of care evaluation.

The economic value of Quebec’s Healthy Lifestyle Habits Promotion Strategy

Pierre-Alexandre Dionne, Tarik Benmarhnia, Ericc Tchouaket, Alvine Fansi, Astrid Brousselle PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION This study aims at evaluating the economic value of the healthy lifestyle habits promotion (HLH P) strategy in

Quebec. As effectiveness data was not available, instead of estimating total savings associated with observed

effects, we estimated the break-even point beyond which the economic benefits of the HLHP strategy would surpass

its costs. Direct healthcare expenditures associated to unhealthy habits were $4.16 billion in 2010 -11, 47% of which

were attributable to risk factors targeted by the HLHP strategy habits (smoking, inactivity, insufficient fruits and

vegetables intake, obesity). We concluded that the HLHP cost ($110 M) corresponded to only 5.6% of health

expenditures attributable to these risk factors. Using a novel methodology, this study compared the economic value

of HLHP activities in Quebec against healthcare expenditures associated with risk factors. This study fits within the

emerging movement of public health economic evaluation.

University equity practitioners as evaluators: barriers and facilitators to effective evaluation

Ryan Naylor

EXPOSÉ/PAPER Student equity is a major higher education policy concern in Australia. Equity program evaluation is an essential

component of social inclusion in higher education, but evaluation is rarely built into the core business of equity

programs. It is often an afterthought, without sufficient rigour or detail. Practitioners are best situated to regularly

and cost-effectively evaluate programs to enhance social inclusion, but report barriers in terms of their

understanding and intrinsic and extrinsic motivations. Increased understanding of barriers and possibilities

associated with equity evaluation will significantly assist the effective operation of equity programs, creating

improved outcomes for disadvantaged students. This study will examine the processes used to evaluate equity

programs at a major Australian university, identify some barriers to effective evaluation, and describe the

development of resources to support the professional development of equity practitioners as evaluators.

31

Using SenseMaker as an evaluative tool for a girls' education intervention in Ethiopia

Rebecca Smith, Rita Fierro

EXPOSÉ/PAPER In May 2014, Girl Hub Ethiopia worked with Dr. Rita Fierro and Jarco Consulting to conduct a research project using

the SenseMaker story collection methodology to explore barriers to education for pastoralist girls in Afar region,

Ethiopia.

What can Evaluation do to support Performance Measurement?

Dwane Unruh, Clinton Hynes

EXPOSÉ/PAPER A strong Performance Measurement (PM) Strategy is essential to the identification and collection of data to support

evaluation. However, since PM Strategies are not always developed with evaluation in mind, or are not developed at

all, administrative data are frequently inadequate to the purpose of evaluation. As a result, evaluators must use

expensive surveys or other methods to make up for this limitation. This presentation will outline challenges in

developing PM Strategies, provide examples of innovative methods to deal with inadequate data, review approaches

to building stronger connections between evaluation and PM, and highlight particular cases that illustrate promising

practices. The examples provided will be drawn primarily from the presenters’ collective experience working as

evaluators in the federal government, while also relying on examples from o ne presenter's extensive experience

outside of government developing programs and designing PM Strategies.

World café - New Meaning-Making Strategies through Dialogue

Chris Corrigan, Amy Lenzo

WORLD CAFÉ/WORLD CAFÉ

8h30-8h45

Mots d'ouverture / Prise en charge du flambeau de l’Année internationale de l’évaluation

Co-Présidents

PLÉNIÈRE/PLENARY

8h45-9h15

Plénière d'ouverture: Un monde meilleur/A better world

Laure Waridel PLÉNIÈRE/PLENARY

9h15-10h

Values and evaluation: can and how can the evaluator engage to defend values for a better world?

Ernest House, Maria Bustelo, Moderated By Astrid Broussselle

PLÉNIÈRE/PLENARY

32

26 MAI MARDI TUESDAY MAY 26

10h30-11h30

CES Professional Learning: Focus on Intermediate Learning

Sandra Sellick, Kathy Gerber, Jacques Bérard TABLES RONDES/ROUNDTABLES Participants in this session will come away with new information about CES professional learning opportunities. This

will include information about the recently revised Essential Skills Series and CES webinars. However the focus will

be on initiatives to increase options for intermediate level learning. This will include an overview of the new

intermediate professional learning plan and its implications for course development in 2015 - 2016 as well the

options to be created for the professional learning of credentialed and uncredentialed evaluators. The round table

will also provide participants with the opportunity to ask questions, share information about professional learning

options across Canada and internationally, and provide recommendations regarding priorities for the CES

Professional Learning Committee in the year ahead. This session will be particularly valuable for Chapter Council

professional learning coordinators.

Food for Thought in Evaluating Complex, Collaborative Initiatives

Jennifer Yessis, Scott Graham, Barbara Zupko

EXPOSÉ/PAPER Nourishing School Communities (NSC) is a collaborative, evidence-based initiative that aims to get healthy and local

foods into the minds and onto the plates of school children across Canada. Many partners are engaged in this

project including children, teachers, principals, school meal providers, farmers, after -school providers, First Nations

communities, researchers and policy makers. Together we are supporting changes to school food env ironments,

changes that aim to shift policies and practices in a way that makes healthy local foods easier to access and fun to

eat. Collective impact of NSC is being assessed using a developmental evaluation approach. Evaluators from different

settings are supporting school communities with tools and resources to assess changes to their school and after -

school environments. Presenters will describe infrastructure and roles played by partners in this evaluation, and the

benefits and challenges of this approach.

Measuring Collaboration: Relationships as Results

Sarah Farina EXPOSÉ/PAPER Achieving lasting outcomes and impacts increasingly depends on strong and effective relationships between

organizations and individuals. Evaluating relationships as results places governance at the center of effective

implementation. This presentation looks at concepts and tools for measuring collaboration. Participants will take

away an inventory of approaches to measuring collaboration as well as a decision -support for understanding the role

of collaboration in different environments.

Missing Link between Program Evaluation and Public Policy?

Joanne Barry, Michelle Riddle EXPOSÉ/PAPER Program evaluation addresses whether or not programs/services are working as intended and/or are achieving their

intended outcomes. Through the examination of several large-scale evaluations of government funded programs,

this presentation will assess the impact of program evaluation on public policy. The presentation will focus on

challenges and best practice learning from these evaluations in order to assess whether program evaluation

influences or transforms public policy and therefore the lives of the end users.

33

Moving targets: Pragmatic considerations for evaluating innovative and complex interventions

Dominique Leonard, Mythri Vijendran

TABLES RONDES/ROUNDTABLES This discussion focuses on challenges and opportunities when evaluating complex, innovative interventions that are

in constant flux. Facilitated by 2 evaluators from SRDC, the discussion draws on lessons learned from evaluations of

2 complex interventions: a research collaborative for integrated healthcare delivery and an innovative after -school

program to improve educational outcomes for youth. Participants will share their cha llenges in similar situations,

whether due to program "drift" or complete overhaul, competing visions of the intervention, or rapid organizational

change. Facilitators will ask questions to stimulate thinking about different evaluation approaches, especial ly when

developmental evaluation is not feasible. Focus will also be on sharing coping strategies, particularly for

intermediate evaluators who may not be positioned to influence overall evaluation goals, but who can nevertheless

contribute to the success of "moving target' evaluations.

Performance Measurement and Evaluation: We need to talk.

Juliana Bravo Hernandez, Mary Kay Lamarche CONFÉRENCE D’EXPERT/EXPERT LECTURE The aim of this presentation is to encourage evaluation practitioners to strategize in th eir day to day work as they

deal with performance measurement and evaluation tensions to make room for flexible context -specific learning and

reflection. The presentation will argue the importance of being aware and critical of how the over emphasis on

results and evidence and the resulting discourses, is resulting in an environment where tools and methods are

leading the work rather than and not dialogue and thinking. As we strive to comply with reporting, whatever that is

in our operational contexts, we prioritize tasks and deliverables but do we equally assign importance and relevance

to learning? Where can we build and foster learning environments? We will invite a cold hard look at the work we do

and how we do it. As we see it, we need to talk.

Redonner place aux valeurs dans les processus et critères d'évaluation

Guy Cauquil CONFÉRENCE D’EXPERT/EXPERT LECTURE Les valeurs servent-elles encore de critère pour apprécier la valeur de l'action publique et pour choisir une démarche

d'évaluation? Peu. L'évaluation de la performance publique vise davantage la rationalisation de l'action et

l'efficience que la pertinence et l'utilité. Les valeurs éthiques ont cédé le pas à l'imaginaire économique et culturel

occidental qui privilégie la satisfaction et la compétition plutôt que la solidarité ou l'équité Faire de l'évaluation pour

un monde meilleur invite donc à interroger les valeurs qui fondent l'évaluation ainsi que leurs déterminants

sociologiques, culturels ou économiques. A quelles valeurs l'évaluation peut-elle faire référence pour apprécier la

valeur d'une action publique? En fonction de quels critères éthiques, économiques ou opérationnels accorde -t-on de

la valeur à telle ou telle méthode? Les citoyens ont-ils ou non leur mot à dire? Ils ont aussi leurs valeurs

The Skilful Means of Multistakeholder Evaluation

Daniel Buckles

EXPOSÉ/PAPER The way we carry out evaluation is as critical as the learning and the transformation we seek. Conventional views on

evaluation treat the mastery of methods and tools simply as means towards ends. The skilful means of participatory

evaluation described in this presentation head in a different direction, towards the embeddedness of ends in means.

When seen from this perspective, skills, methods and technology are not mere instruments with instruction manuals

to be followed, as if without purpose and meaning. To support genuine accountability and learning the selection and

scaling of tools must be reconciled with what stakeholders are and long to be, that is, thought ful beings that strive

to reach out and communicate with other human beings and the world that surrounds us. More attention needs to

be given not only to the feedback loops key to evaluation but also skillful means for emergent learning across social

and cultural boundaries.

34

Understanding and Applying Innovation in Federal Government Program Evaluation

Tyler Toso, Blayne Beacham

EXPOSÉ/PAPER The purpose of this presentation is to summarize current knowledge related to innovation in program evaluation in

order to inform future thinking and improve evaluation practice through exposure to these ideas. This project

achieves this purpose by identifying innovations occurring in program evaluation, while discussing the benefits,

drawbacks, barriers and lessons learned regarding innovation in the context of federal government program

evaluation. This presentation draws on 5 years of literature from Canadian Journal of Program Evaluation, American

Journal of Evaluation & New Directions for Evaluation in order to ident ify innovative tools and methodologies that

had demonstrated potential for use in program evaluation. In addition, 17 key informant interviews were conducted

with evaluators working in a federal government context to further identify successful innovations , and gather

expert insight on how to incorporate innovations into evaluation.

Understanding and evaluating government administrative services

Isabelle Mercier, Eugenia Didenko EXPOSÉ/PAPER Governments provide an array of administrative services as a means to fulfil legislative requirements and meet

operational needs. Most prominent include procurement, information technology, real property management and

others. The delivery of these services consumes a large portion of the government's budget and civil se rvice

capacity. With an increasing pressure to rationalize spending, departments are transforming their administrative

operations, with a view of a leaner, more efficient service delivery. Evaluating administrative services, however, is

becoming more complex, as services are being reformed through centralization, modernization and outsourcing. This

presentation will highlight the nature and unique elements central to public administrative services and ways to

assess their performance. The discussion will explore the centrality of user satisfaction, client relationship, service

utilization, service standards and cost savings measures in the administrative context.

10h30-11h45

#eval: Twitter as a potential line of evidence in program evaluations

Douglas Hagar

EXPOSÉ/PAPER Social media data has been used to support marketing campaigns and academic projects exploring issue networks

and information dissemination, but to date the use of social media data in program evaluations has largely been

unexplored. This paper examines the potential to use tweets as a line of evidence in program evaluations, identifies

tools and options for data collection and analysis, and discusses practical and ethical issues pertaining to the use of

Twitter data. The findings of the paper draw upon existing academic research and interviews with government

officials and evaluators. The paper provides suggestions for those evaluators considering using tweets as a line of

evidence in program evaluations in the future.

Communicating your evaluative messages: how knowledge mobilization and technology can help

Kate Svensson, Barbara Szijarto

PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION New technologies are opening up a world of possibilities for evaluators to avoid 'the dusty shelf' — that dreaded

place where unused findings are said to languish. This is particularly the case when technology is used in alignment

with effective knowledge mobilisation (KMb) practices. The field of KMb offers substantive guidance to facilitate

learning among individuals from different professions and perspectives — such as evaluators, policymakers and

program administrators. This paper focuses on selected KMb practices salient to sharing evaluative messages with

stakeholders, providing background on why they are considered effective. Special attention will be paid to how this

35

work may be made easier and less costly with the use of technology (social network mapping and infographics used

as examples). The paper also offers suggestions for how to build a KMb plan into your evaluation project.

Comparaison des choix méthodologiques opérés par les évaluateurs de programme novices et experts

Maud Mediell, Eric Dionne PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION L'évaluation de programme (EP) présente de nombreux enjeux relatifs à la praxis, enjeux qui montrent l'importance

de mieux documenter comment les évaluateurs, novices et experts, appréhendent le processus d'évaluation et, plus

particulièrement, la méthodologie. Le but de cette présentation qui s'inscrit dans une recherche doctorale est

d'aborder les pratiques des évaluateurs de programme, et plus particulièrement leurs choix méthodologiques, et la

potentielle influence des parties prenantes sur ces processus décisionnels lors de leurs EP. Ainsi présenterons -nous

notre devis de recherche et les instruments de collecte de données que nous avons construits afin de répondre à la

question principale de recherche suivante : en quoi les évaluateurs novices et les évaluateurs experts ½uvrant dans

un même domaine de pratique se distinguent-ils, ou non, dans les choix méthodologiques qu'ils mettent en ½uvre

relativement à la réalisation de leurs EP?

Dictionnaire terminologique de l'évaluation: la dimension conceptuelle

Richard Marceau, Francine Sylvain EXPOSÉ/PAPER Les évaluateurs ont avantage à stabiliser leur terminologie pour mieux construire leur domaine de pratique et le

faire évoluer. Le domaine de l'évaluation a atteint la maturité nécessaire à l'établissement d'une terminologie

cohérente qui le définit de manière identitaire et qui est en mesure de résoudre de manière pragmatique beaucoup

de difficultés de communication entre les évaluateurs et leurs parties prenantes et entre les évaluateurs eux -mêmes.

C'est dans ce contexte que les auteurs présentent le Dictionnaire terminologique de l'évaluation. Après un bref

retour sur la problématique, ils abordent le système conceptuel ayant servi de base au choix de la nomenclature, la

méthodologie lexicographique ainsi que la structure de chaque entrée du Dictionnaire : dé finition, exemples,

iconographie, remarques et d'autres rubriques qui permettent de saisir l'ancrage de la terminologie évaluative

proposée dans la langue française.

EVALUATION AND THE REFORM OF THE SOCIAL ASSISTANCE SYSTEM IN THE STATE OF TABASCO, MEXICO

Mónica Ruiz-Casares, David Tobis, Elsy Alcalá Cortés

EXPOSÉ/PAPER These are challenging times for Governments around the world. Growing inequity, global interdependence, and

citizens' pressures for transparency and accountability in public administration are leading many countries to

introduce comprehensive reforms to deal more effectively with the challenges and opportunities they face. The

State of Tabasco (Mexico), rich in natural resources, still confronts widespread poverty, gender -based violence,

migration, youth suicide, and an ageing population. In 2014, the new state government commissioned a rapid

assessment of the social assistance system, with special attention to the Desarrollo Integral de la Familia (DIF), the

main social assistance agency in the state. Intensive desk review, key informant interviews, and site visits to

programs for children, women and families, people with disabilities, the elderly, and poor and marginalized

communities, were conducted. This paper will describe the process and results of this assessment a

Evaluation of ISFM technologies in Southern Highlands of Tanzania

William Mmari, Kissa Mwaisoba EXPOSÉ/PAPER Soil fertility degradation and striga weed in maize production system are key problems leading to low maize

productivity, addressed by the project . The project aims to address the problems through increased awareness and

knowledge of farmers on integrated soil fertility management, for increased soil productivity. The main objective

was to assess the status of use of ISFM technologies among farmers growing maize and beans in Mbeya rural and

36

Mbozi. A cross-sectional design was used, in which data was collected at a single point in time. This design is the

mostly preferred because it is cost effective and less time consuming. Both qualitative and quantitative data were

collected and analyzed. Mbeya rural and Mbozi district were purposive selected because the project "Intensifying

maize and beans production through improved soil health and marketing in Mbeya, Tanzan ia. The proportion of the

farmers using fertilize increased from 80 to 99.1% after project intervention.

Generating the tough discussions - the power of your evaluations

Marie-Anik Gagne, Shelley Borys

EXPOSÉ/PAPER We all know that evaluations are informative, necessary and awesome — well at least the converted attending this

conference know that this is true. But seriously, your evaluation findings can generate the really tough discussions

that need to take place in government! This presentation will provide examples of the types key discussions senior

management at Health Canada and the Public Health Agency of Canada have had as a result of evaluation reports —

all while respecting committee confidences of course. These are tough discussions that Canadians want senior

bureaucrats to have on a regular basis: When is the federal role's done? How do we compare internationally to other

governments or private companies that offer the same/similar services? Are we impacting individual Canadians? Are

we using our resources most efficiently and effectively? Are we doing what we should be doing? Your evaluations

really are informative, necessary and awesome!

Graduate Evaluators, How Well Do You Do?

Shufang Dong

PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION Understanding graduate evaluators is important to graduate schools and the field of PE, given the acknowledged and

potential contribution of graduate evaluators. What are their experiences as graduate student evaluators? What can

be learnt from their experiences? The focus of this research is to explore the experiences of graduate evaluators,

through which, at the same time, to investigate and discuss the methodological approaches in PE. This study is

conducted with qualitative methodological approaches such as document and text study, interviews through the

analytic framework of grounded theory and critical discourse analysis under socio -cultural theoretical framework. It

is hoped that the methodologies in this study can add thoughts to program evaluation metho dology in that these

methodologies can both contribute to the analysis of the data but have their own uniqueness which is derived from

the essence of PE. It is also hoped that this study can be an starting point of future study.

Le problème comme réponse : cerner les critères de mérite par l'étude des problèmes sociaux

Ghislain Arbour EXPOSÉ/PAPER Pour juger une intervention, nous devons comprendre les raisons qui font qu'elle est nécessaire. À cet égard, on ne

peut trouver des critères de mérite pertinents sans comprendre le problème social qui justifie (peut-être) le fait

d'intervenir. Cette présentation suggère un cadre d'analyse pour étudier les problèmes sociaux de manière à ce que

nous puissions identifier de bons critères de mérite dans les évaluations des interventions publiques. Le cadre

aborde trois dimensions du problème social: (a) sa nature morale (pourquoi est -il mauvais?); (b) sa mécanique

causale (comment il fonctionne?); et (c) son intensité (dans quelle mesure est -on en mauvaise posture?). Le cadre

explique comment organiser ces questions de telle manière qu'ils puissent ensemble dévoiler la pertinence sociale

des programmes gouvernementaux. Le cadre sera également analysé pour son potentiel à aborder les questions

d'équité dans l'offre des programmes publics.

Learning from Evaluation Studies of Literacy Campaigns in India

Kaustubha Nand Bhatt

EXPOSÉ/PAPER

37

India, a nation with one third illiterates of the world, strives to eradicate adult illiteracy. A National Literacy Mission

(NLM) was established in 1988 to place functional literacy for all on the national agenda. Total Literacy Campaigns

(TLC) were initiated in 1989. An appropriate system of the evaluation was launched simultaneously to provide a

reliable feed-back to the concerned to facilitate and ensure effective implementation of literacy programs. G.B. Pant

Social Science Institute, Allahabad conducted three evaluation studies. These evaluation exercises aimed to study

the overall performance of TLC in the concerned district. It examined the operational strategies and implementation

approaches in the context of approved plan of action having regard to district specific factors; identified the strength

and weaknesses of the project; analyzed the factors responsible for the strength and weaknesses; and suggested

corrective and remedial measures.

L'utilisation des connaissances issues de l'évaluation : étude de cas organisationnelle

Isabelle Bourgeois, Clémence Naré EXPOSÉ/PAPER Le gouvernement fédéral a mené un exercice semblable à celui de la révision des programmes qui a lieu

actuellement au Québec entre 2007 et 2010. Les ministères devaient, à l'aide de données probantes issues de

l'évaluation, entre autres, identifier des économies de 5% et 10% de leur enveloppe budgétaire. Ce que l 'on a

constaté, suite à cet exercice, c'est que dans la majorité des cas, les ministères ne disposaient pas des données

évaluatives nécessaires pour prendre ces décisions budgétaires. Notre étude, financée par le CRSH et dont nous

présenterons les résultats préliminaires, porte sur la contribution de l'évaluation au processus décisionnel dans deux

agences fédérales depuis la fin de l'examen stratégique. Nous identifierons, entre autres, les caractéristiques des

rapports d'évaluation et les éléments-clés du contexte organisationnel qui sont susceptibles de mener à une plus

grande utilisation des données probantes par la haute gestion.

Rapid Assessment of Teacher Effectiveness (RATE): Creating the World We Want with Predictive Eval

Michael Strong, John Gargani

EXPOSÉ/PAPER Expectations of instant information are changing the way evaluators work. In addition to retrospective summative

evaluations, we now engage in real-time formative and developmental evaluation. The next step is predictive

evaluation — quickly predicting how effective an intervention will be and immediately using that information to

improve outcomes. In this presentation, we describe our efforts to develop a predictive evaluation tool — the Rapid

Assessment of Teacher Effectiveness (RATE) — that allows us to predict the future success of teachers by observing

short video clips of their instruction. By making predictions early in the school year, administrators can support

teachers as necessary, increase student achievement, and minimize costs. We present evidence of RATE's predictive

validity and describe our first field study currently underway with pre-service and early-career teachers. The

implications of predictive evaluation for the field are discussed.

Strengthening the Utility of Competencies for Canadian Evaluation Practice

Gunter Rochow, Pierre Morin

EXPOSÉ/PAPER When the Canadian Evaluation Society developed its Credentialing Program, it considered it essential to be able to

identify the required underlying technical, personal and ethical competencies. In the view of the presenters it is

essential to go beyond the mere identification of the competencies to a fully -fledged analysis of each competency

through a participatory analysis process of practising professionals which would help publi c, private and educational

entities in curricular planning with a view to creating a homogeneous body of knowledge, technical skills and

attitudes. The resulting profiles would also be useful in personnel recruitment, training, performance assessment

and the development of remuneration scales. The adoption of such a professional approach to competency

definition would further enhance the worldwide recognition of the Canadian Evaluation Society as a leader in

evaluation-related professional development.

38

The inner workings of a meaningful meta-evaluation

Indrani Barrón

EXPOSÉ/PAPER The inner workings of a meaningful meta-evaluation The advancement of a culture of evaluation that is rapidly

consolidating has made some space for meta-evaluations, something that is better known as an evaluation of

evaluations. The OECD, Glossary of Key Terms in Evaluation and Results Based Management specifies that the term

is used for evaluations designed to aggregate findings from a series of evaluations. It can also be used to denote the

evaluation of an evaluation to judge its quality and/or assess the performance of the evaluators. The examples on

which our paper is based are UNICEF's Global Evaluation Quality Assurance System (GEROS), and the Meta -

Evaluation of Project and Programme Evaluations 2012-14 for the Ministry for Foreign Affairs of Finland. UNICEF's

GEROS is a system used to ensure that UNICEF's evaluations uphold the high quality standards set for them. The

GEROS review process and the information provided in annual

12h-12h15

Passation du flambeau 2015 « au prochain sur la liste » et la présentation des résultats de la consultation sur

l’Agenda global.

Benoit Gauthier, Stewart Donaldson, Lyn Alderman, Serge Eric Yakeu, Maria Bustello, Estaban Tapella, Guy

Cauquil, Fation Luli, Olivier Sossa LANCEMENT/LAUNCH

13h15-14h45

An Impact Evaluation of the MCC Jordan Compact Water and Wastewater Projects: Baseline Results

Nathan Cutler

EXPOSÉ/PAPER Social Impact (SI) has been contracted by the Millennium Challenge Corporation (MCC) to measure the impact of

water and wastewater infrastructure activities on economic and social outcomes in Jordan, which represents the

first attempt to conduct a rigorous impact evaluation (IE) of a large infrastructure project in Jordan. SI has de signed

a quasi-experimental IE aiming to establish a causal relationship between infrastructure investments and observed

changes in household availability and consumption of water and household income, expenditure and health

indicators using a multiple treatment and multiple control propensity score matching methodology. Baseline data,

including attempting to model the willingness to pay for piped water availability, are discussed alongside a

demonstration of a novel, semi-automatic Stata package used to check the quality of data from computer assisted

personal interviewing (CAPI).

Broad impact – enhancing people’s prospects: Lessons from GIZ's corporate strategy evaluation

Frank Dubert EXPOSÉ/PAPER As a federal enterprise, GIZ supports the German Government in achieving its objectives in the field of international

cooperation for sustainable development. The evaluation office of GIZ recently completed a corporate -level

evaluation on scaling up in GIZ-supported development programs. With its work, GIZ aims to improve people’s living

conditions and give them new prospects. The corporate strategy evaluation was devoted to the question of how, and

how successfully, scaling up is achieved in the projects implemented by GIZ and its partners. The evaluation

comprised two interconnected parts, namely a portfolio review of more than 300 projects, and seven country case

studies in the water sector. Spaces, drivers and pathways that that can be used to implement scaling up processes

and that partially determine the success of scaling up mechanisms impact were analyzed. As a learning organization,

GIZ uses the evaluation findings for a company-wide learning process.

39

Evaluation of Collective Impact Initiatives: Two Contrasting Examples

Nancy Carter, Robert Chatwin

EXPOSÉ/PAPER The presentation will discuss the application of the collective impact framework to two initiatives in Nova Scotia and

the evaluation frameworks that have been developed to assess each initiative. The CIHR Reform Assessment of

Impact Group (CRAIG) - a Maritime initiative aiming to evaluate impacts of restructuring of health research funding

in the region - has adopted a collective impact approach and is using developmental and participatory approaches to

evaluation for the purpose of learning from the experience. The Regulated health Professions Network (RHPN) was

formed as a result of provincial legislation that encourages collaboration amongst stakeholders to address common

issues. The RHPN has designed an evaluation framework aimed at accountability and demonstrating value of this

innovative approach to regulation. Again a developmental and participatory approach have been taken to ensure

engagement and learning amongst members. Key aspects of Collective Impact will be discussed.

Evolving Developmental Evaluation: Challenges & solutions from 4 perspectives

Kate Woodman , Eugene Krupa, Cheryl Poth, Tara Hanson PANEL/PANEL Developmental Evaluation (DE) theory provides a model to address many challenges faced by evaluators of programs

in dynamic contexts. However, when applied in real-life with all the complexities people introduce, many challenges

emerge for those responsible for the evaluation, and considerable "in -progress re-thinking" is always needed.

Although many of the themes are consistent among projects, challenges can also vary depending on the sector,

project focus, context, project duration, people involved and their backgrounds, capacities and personalities. In this

interactive panel session, we examine the challenges from 4 players' perspectives: evaluator on the ground (Krupa),

evaluation project manager (Woodman), funder's contract manager (Hanson) and DE researcher (Poth). We pose

paths to resolution from our experiences, and suggest implications and recommendations for professional

competencies, project management, contract management and future research.

Expérience de suivi des indicateurs communs sur l'eau potable en zone rurale au Burkina Faso

Lynda Rey, Pierre-Yves Charpentier EXPOSÉ/PAPER Le Projet Eau de la Haute Comoé au Burkina-Faso (PEHC) vise la transformation des conditions de vie de la

population de cinq communes à travers notamment un accès durable à l'eau potable. Dans ce projet, le suivi -

évaluation a constitué un défi important pour les évaluateurs. Il a fallu trouver un équilibre entre les indicateurs

standards développés au niveau international d'une part, et d'autre part contextualiser ces indicateurs pour mettre

en valeur les réalités locales influençant la performance du projet. Nous présentons ici les défis rencontrés et leçons

apprises du système de suivi-évaluation des indicateurs liés à l'eau potable du PEHC (milieu d'intervention rural et

isolé, coûts importants associés à la collecte et à l'analyse des donnée, etc.). En conclusion, la contextualisation des

indicateurs standards de performance constitue «un art» qui relève à la fois de la compétence professionnelle et de

la sensibilité culturelle de l'évaluateur.

Growth and Performance of Africa VOPEs: The case of The Evaluation Society of Kenya (ESK)

Jaboma Allan, Julius Nyangaga

EXPOSÉ/PAPER ESK is a network of M&E practitioners in Kenya and the eastern Africa region and was initiated in 2008. The main

objective of the Society is to promote a sustainable and quality monitoring and evaluation practice for improving

development policies, Projects and programs. The society delivers professional input into Kenya's development

agenda through multistakeholder collaborations. Between January and June 2013, the Society undert ook a study to

develop a usable database of M & E services and products in the country and identify skills gaps that will inform a

resource mobilization and capacity building strategy. The survey targeted individuals and institutions providing M&E

services in the country. Gaps identified included data analysis, designing and implementing of evaluations among

40

others. The study reveals that despite a large presence of M&E professionals, more innovative ways have to be

found to enhance recruitment, membership and sharing of M&E products for development.

Jeunes Évaluateurs: Comment faire sa place dans le monde de l'évaluation?

Jérôme Gandin, Miché Ouedraogo, Andréanne Martel, Seyive Wilfried Affodegon, Marie-Pier Marchand,

Marie Gervais

PANEL/PANEL Ce panel questionne la place des Évaluateurs Émergents dans le monde de l’évaluation. Il propose aux participants

de discuter quatre questions avec des intervenants aux profils, cultures et perspectives variés:1)Que devrait faire un

évaluateur émergent pour tendre à devenir un bon évaluateur ?2)Quels sont les principaux défis que vous avez

rencontrés(ou rencontrez actuellement) pour développer votre pratique en évaluation? Quelles sont les stratégies

utilisées pour y faire face?3)Comme évaluateur émergent, quelle est votre expérience en lien avec votre société

d’évaluation? Qu’est-ce qui pourrait être fait pour renforcer la place et la contribution des évaluateurs émergents

dans votre société?4)Des démarches sont actuellement en cours pour mettre en place un réseau inter national

d'évaluateurs émergents francophones. Quels devraient être les objectifs visés par un tel réseau? Quels en seraient

les facteurs de réussite? Votre contribution potentielle?

Made in Africa Evaluation: an overview of innovative capacity development programs

Mirianaud Oswald Agbadome

EXPOSÉ/PAPER

Monitoring and Evaluation of Rule of Law Initiatives

Sadie Yang, Andrea Redway, Colleen Duggan, Joel Martinez PANEL/PANEL The rule of law (RoL) is widely recognized as an important development goal. However effective monitoring and

evaluation of RoL programs is difficult: there is no commonly accepted definition of RoL; attribution for RoL results is

often unclear; and the selection of indicators and collection of data to demonstrate results often proves to b e

challenging. The panel will contrast approaches that best demonstrate whether RoL results have been achieved at

the sectoral, project and individual intervention level: • WJP will share indices to measure the degree to which

national justice systems as a whole have been strengthened by RoL initiatives; • The CBA will compare distinct

evaluation approaches that have been applied to its projects and their relative effectiveness in measuring a project’s

impact on RoL; • IDRC will contrast methodologies used to evaluate solutions to RoL challenges and the extent to

which they generate evidence on the equitable delivery of justice services.

Outcomes Evaluation of Multi Sector Partnerships for Sustainable Community Development in Nigeria

Anthony Chovwen EXPOSÉ/PAPER The General Memorandum of Understanding(GMoU) as a development interface strategy is widely used by

multinationals and international oil companies working in the oil rich Niger delta of Nigeria in numerous hosts

communities as a multi sector partnership for sustainable community development. This study assess the impact of

Gentoil GMoU Outcomes in selected post 5 years tenured communities in 3 states of the Niger Delta, Delta, Rivers

and Bayelsa. Preliminary results shows the strategy has had varying degree of effectiveness across the participating

communities. Key performance indicators (KPI) such as transparency and accountability, capacity building,

community empowerment, participations and inclusiveness, business climate, governance and democracy ha ve been

achieved in participating communities. Progress towards sustainability, partnership with development agencies,

interface with other agencies are remain low points of the strategy.

Partenariat et évaluation au service de l’innovation sociale dans le milieu des Premières Nations

Ana Gherghel, Caroline St-Louis, Patricia Montambault , Nadine Rousselot

41

PANEL/PANEL Comment une évaluation collaborative de type formatif peut-elle contribuer au renforcement de la gouvernance

d’'une organisation des Premières Nations? Comment s’assurer que les résultats d’'évaluation soient utilisés comme

outils de gestion? Quelles stratégies peuvent favoriser l’'appropriation des résultats à la fois par les milieux de

pratique, par les gestionnaires et par les communautés? Les présentatrices réfléchissent conjointement à ces

questions sur la base d’'une expérience récente de partenariat stratégique établi entre la Commission de santé et

services sociaux des Premières Nations du Québec et du Labrador (CSSSPNQL) et Avenir d’'en fants. Croiser les points

de vue des gestionnaires et des évaluateurs permet de voir sous différents angles les étapes franchies, les défis

relevés, les conditions favorables, les leçons apprises et les modalités d ’échange développées pour faciliter

l’appropriation des savoirs issus de l’'évaluation.

Placing the Evaluation Practice in the heart of policy making, A case of Uganda

Juliet Carolyn Anewa Odeke

EXPOSÉ/PAPER In Uganda, there is a broad consensus that evaluations and the practice is crucial in innovating development

solutions and shaping policy. This paper will attempt to highlight which ways evaluation is placed in the heart of

policy making in Uganda. The various institutional and legal frameworks present, would be roles, challenges, what

has worked versus what has. Contextual issues will be raised to facilitate dialog in the Conference "For the world we

want" Achieving influence at national level requires: Highlight success stories (Gov't& CSOs) Democratic debates

(parliament) Capacity: basic trainings for policy makers (ECD) Space for inexperienced evaluators to learn Mentoring

at national level Access and Sharing: the media, dissemination seminars Active associations e.g. Uganda Evaluation

Association A central body of knowledge e.g. GEF Exposure to best practices Online & offline debates Making clear

evaluation mandates to enhance ownership.

Roles of evaluation in enhancing Nigerian Government climate change policies and programmes

Onyinyechi Ogbonna, Juliana Iwuchukwu EXPOSÉ/PAPER This paper reviewed government Climate change (CC) policies and programmes in Nigeria and possible roles of

evaluation. Nigeria government has reactions and moves toward tackling CC problems by way of invention of policies

and programme. Among them are: in 2010, the National Assembly passed a bill to create a national Climate Change

commission to facilitate coordination and support for the multi -level and cross-sectoral adaptation responses. Also,

the Government of Nigeria embarked upon the development of National Adaptation Strategy and Plan of Action on

Climate Change for Nigeria. However, many policies and programmes are hampered by numerous challenges. The

paper therefore recommends possible ways of involving evaluators for proper coordination of their activities so as to

spur synergies among them in order to actually mitigate and adapt to CC which is their ultimate goal and enhance

the efforts of government in this aspect.

Selection challenges in evaluation design in Northern Nigeria

Bukola Oyinloye

EXPOSÉ/PAPER This presentation succinctly summarises the challenges encountered during the data collection for the baseline

impact and outcome evaluation for a teacher training programme in Northern Nigeria. To promote local ownership,

the evaluation sample selection criteria were shared with the programme's national stakeholders who used it to

select the required elements for the evaluation. However, the data collection process revealed a number of issues

with the selected elements, as a large proportion did not fulfil l the provided criteria. To address these issues, sample

restoration mechanisms were employed in the field during the remaining data collection period. Ultimately, the

process resulted in a number of lessons learned both for the programme, the evaluation s ervice provider, and the

programme's national stakeholders and such lessons will inform subsequent sample selection processes for the

programme.

42

The Use of Developmental Evaluation in Organizational Practice

Robert Mclean, Joanna Kocsis, Chaitali Sinha, John Gargani

PANEL/PANEL Striving to improve quality and impact are pressing considerations for most organizations. How can this be achieved

in dynamic, complex settings? We are living in a world that demands concrete results, yet recognizes the ‘messy’

way change occurs. How can this be reconciled? This panel focuses on one particular organization’s experience

exploring, implementing and using an approach that is designed to respond to such a conundrum. Developmental

Evaluation (DE) is an approach to real-time learning designed to evaluate efforts unfolding in uncertain, dynamic

contexts. The International Development Research Centre is learning from past experiences applying DE and

adopting a learning-by-doing approach moving forward. Panellists will provide an overview of using DE at IDRC,

share findings from an IDRC research study on how DE is used in different North American grant -making

organizations, a case study of applying DE in a program at IDRC, and commentary on DE from an expert perspe ctive.

Trends in humanitarian evaluation: insights from the pilot of ALNAP guidance

Alexandra Warner EXPOSÉ/PAPER For evaluation to stay relevant in the fluid and fast paced environments of humanitarian action, conventional

evaluation methods and approaches need to be adapted and contextualised. In 2010, ALNAP started developing the

first comprehensive, utilisation-focused and user-oriented guide on Evaluating Humanitarian Action (EHA),

www.alnap.org/eha. The 18-month, active pilot process of the guide closed at the end of 2014. The pilot gave ALNAP

insights into tacit aspects of EHA processes and identified trends in practice. Some of which may hamper the role

and influence evaluation can have on improving programming and fostering learning. In this presentation, an ALNAP

researcher will share some of these trends, interesting lessons learned, practical examples of how organisations

have improved EHA processes, as well as some approaches and tools highlighted through the pilot process that could

aid in furthering the influence of evaluations in this sector.

United Nations Evaluation Group (UNEG) Resource Pack on Joint Evaluation

Krishna Belbase, Scott Greene EXPOSÉ/PAPER There is growing demand for joint evaluations that are credible and useful both in terms of ensuring accountability

for results and for lesson learning. The United Nations Evaluation Group (UNEG) has recently issued a Resource Pack

on Joint Evaluation which consists of two components: a) A Joint Evaluation Guidance Document which outlines

main issues that arise when conducting a joint evaluation; b) A Toolkit, which includes a collection of documents,

including examples of good practices and UNEG-specific guidance on planning, managing and utilizing joint

evaluations. The co-chairs of the UNEG Taskforce would like to share the process and content of the joint evaluation

resource pack with the CES community. The presentation will focus on the overall participatory approach adopted in

formulating the resource pack, quality assurance process, the actual content of the guidance document and the

toolkit and their dissemination.

What Did We Learn?: Experiences Implementing Theory-Based Approaches in the Federal Government

Context

Jane Whynot, Steve Montague, Mary Kay Lamarche, Eric Seraphim PANEL/PANEL Since the 2012 issuance of Theory-Based Approaches to Evaluation: Concepts and Practices, federal government

evaluators have been absorbed in comprehensively unpacking how an intervention is intended to work, in order to

demonstrate achieved results. Various theory-based approaches have been undertaken by different departments

and sectors across government, targeting various intervention levels. Panelists identify articulated, and commonly

experienced challenges in working with these approaches, responding with both the solutions implemented. These

challenges include resourcing levels, diverse responsibilities, participatory methodologies, and aligning informational

43

content with the core issues associated with government evaluation. Panelists will offer ins ight on plausible

solutions to ensure that decision-makers have access to the quality and timely information offered by evaluation.

What is and is not essential for a quality impact evaluation?

Reuben Ford, Susanna Gurr, David Gyarmati, Jean-Pierre Voyer PANEL/PANEL Some program impact evaluations are attempted in circumstances where they are not appropriate or cannot hope

deliver a valid or reliable answer. Other evaluations avoid impact estimation when it would be possible and even

advisable for the future of the program. An experienced panel of evaluators from SRDC and the BC Centre for

Employment Excellence will speak to what they consider essential (and not) for a quality impact evaluation to take

place. Topics start with the setting of the evaluation question, the state of development of the program and its

implementation. Panelists will review means to control credibly for participant selection, design (including ethical)

issues, prospects for measuring the right outcomes, partnerships and commitments to rigour and trust. They will

seek to dispel myths that randomized trials are the only option, that impact evaluations are necessarily expensive

and require high program fidelity and extensive data to achieve their goals.

13h15-15h15

Evaluation Capacity-building for Leaders: A Mobile Learning Tool

Kylie Hutchinson, Chris Lovato

IGNITE (PRÉSENTATION RAPIDE)/IGNITE-STYLE RAPID PRESENTATION Capacity-building in evaluation has tended to focus on program-level coordinators and staff, ignoring the key role

that leaders play in using evaluation to make better decisions. To fully achieve the potential evaluation has in

making informed decisions and using practice-based evidence depends, to a large extent, on leaders and decision-

makers who are better informed consumers of evaluation. How can we best maximize the use of evaluation in

planning and decision-making through capacity building with these busy individuals? We have developed a free

mobile-learning package that provides this target audience key information about evaluation and how to best work

with evaluators. A beta version of this unique approach to capacity building with leaders and decision makers will be

demonstrated in this session.

14h45-16h15

Animal Evaluation; Why should the evaluator talk with the cow?

David Brous EXPOSÉ/PAPER The evaluation of programs that target animals as their significant stakeholders present special challenges for

evaluators. How do we establish appropriate channels for communicating with animals, to ensure that the animal's

perspective on the impact of programs is gained, such as those concerned with the animal welfare standards for the

care of livestock, standards for transportation and ethical processes for slaughtering? Recent policy development on

the transfer of live sheep and cattle to international markets from Australia has highlighted the importance of

establishing the animal's perspective in evaluating the outcomes of policies to enhance animal welfare. Access to

international markets for livestock produce requires consideration of animal needs. This paper will explore the

issues of conducting evaluations in animal centred contexts and draw on the presenter's experience in assessment of

biosecurity and livestock market access programs in Australia

Assessment of implementation fidelity of the Arctic Char Distribution Program in Nunavik

Lara Gautier, Catherine Pirkle, Christopher Furgal, Michel Lucas

PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION

44

In September 2011, the Nunavik Regional Board of Health and Social Services began the Arctic Char Distribution

Program (AC-DP) for Pregnant Women. This program promotes the consumption of Arctic Char — a preferred

traditional Inuit food that is nutritionally rich and relatively contaminant -free — by pregnant women living in villages

along the Hudson Bay, in Nunavik. This intervention intends to reduce exposure to contaminants and improve food

security in Inuit communities. However, implementation of the program is incomplete since it does not currently

cover all intended areas, and its environmental and financial sustainability are at stake. We assess its

implementation based on data collected from program documentation, meeting minutes, field notes, and qualitative

interviews with program recipients and implementers. Themes emerging from a review and coding of these

materials are discussed in light of the framework for implementation fidelity developed by Carroll et al, 2007.

Bâtir une culture d'évaluation dans un établissement de services en contexte de diversité

Gabrielle Lemieux, Monica Ruiz-Casares EXPOSÉ/PAPER Comme ailleurs au Canada, la population du territoire du CSSS de la Montagne est caractérisée par la pluriethnicité

et par plusieurs facteurs de vulnérabilité qui l’accompagnent. La recherche, l’adaptation et l’innovation permettent

depuis plusieurs années aux organisations de mieux répondre aux besoins des usagers de diverses origines. Depuis

plus récemment, l’évaluation se présente comme un moyen pour tous de réfléchir, de remettre en question leurs

pratiques et d’apprendre pour améliorer encore plus la qualité des services et ainsi réduire les inégalités. Cette

culture d’évaluation émergeante doit être soutenue par les organisations afin de développer les capacités

d’évaluation et optimiser l’utilisation des connaissances. Les établissements de services peuvent s’inspirer de

l’expérience du CSSS dans l’élaboration d’une structure et d’une politique d’évaluation par une approche

participative, grâce à laquelle ces outils sont devenus de puissants promoteurs de la culture évaluative.

Canadian Evaluation support of International Evaluation-Challenges, Achievements & Lessons

Robert Lahey, Charles Lusthaus, Jean-Serge Quesnel, Laila Smith, Boubacar Aw

PANEL/PANEL With 2015 the international Year of Evaluation, it is useful to examine the role Canada has played to support

Evaluation capacity development around the globe. Where the keynote speaker to the 2014 CES conference noted

“Evaluation needs more Canada”, the session will highlight the myriad of ways Canadian Eva luation has indeed

supported and influenced the practice of Evaluation around the world. Canada can also learn from this international

experience, and so the panel will also reflect on ‘lessons’ for Evaluation in Canada to be gleaned from this

international exposure. Finally, the panel will look beyond 2015 and provide commentary on priority areas for

Evaluation where we ought to focus our energies, both in Canada and globally, following the 'Year of Evaluation'.

Each panelist has a unique relationship with Evaluation in Canada and significant experience in supporting and

conducting Evaluation in a wide range of countries around the globe.

Comparative Economic Analysis of Beekeeping using Traditional and Improved Beehives in Tanzania

Nicholaus Kuboja

PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION The overall objective of the study was to undertake a comparative economic analysis of beekeeping using either

improved or traditional beehives. Data was collected from 69 beekeepers that were purposively selecte d in the

study area. Budgetary and profitability ratio analysis were carried-on. Both beekeeping using improved and

traditional beehives was profitable. However, profitability between the two differs; beekeepers using traditional

beehives realized a net farm income of TZS 1034485.00 against TZS 351713.70 which was realized from using

improved beehives. Also a return on investment per 1TZS recorded using traditional beehives was 4.267 against

1.524 accrued from improved beehives

45

Défis et opportunités pour l'évaluation à travers le monde - points de vue de présidents d'associations /

Challenges and opportunities for evaluation around the world - views from association presidents

CES - Benoit Gauthier, AEA - Stewart Donaldson, AES - Lyn Alderman, AFREA - Serge Eric Yakeu, EES - Maria

Bustello, RELAC - Estaban Tapella, RFE - Guy Cauquil, Saep- Fation Luli, SQEP - Olivier Sossa PLÉNIÈRE/PLENARY

Evaluating human rights mainstreaming - different from equity-focused evaluation?

Sabine Becker-Thierry PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION Development and human rights have for a long time been seen as contradicting paradigms, yet the Right to

Development was declared (1986) and a 'Human Rights-Based Approach to planning' (HRBA; 2003) been more and

more followed. Yet, not all organizations and projects explicitly apply a HRBA, rather they 'mainstream' human rights

(in addition to achieving a specific development goal, they are meant to also positively contribute to human rights).

The discussion on ways of assessing HRBA and human-rights mainstreaming is not explicitly building on development

evaluation approaches, whether gender evaluation, feminist evaluation, transformative/ empowerment evaluation

and the recent equity-focus evaluation. Is an evaluation with an equity-focus synonymous to evaluating human

rights in a development intervention? What does evaluating human rights mainstreaming in a development

intervention mean? And what are possible entry points?

Evaluating program impacts: does the source of outcome data matter?

Reuben Ford EXPOSÉ/PAPER Using multiple data sources offers evaluators many advantages including expanding the set of observations,

contrasting results and validating interpretations. Also, multi -year evaluations often see the range of data sources

change over time. Future to Discover is a randomized study that estimates impacts of education interventions over

time while transitioning from using a combination of administrative datasets and follow -up surveys to a combination

of administrative datasets only. The study assesses the consequences of switching between data sources on

substantive findings and describes four factors that affect the robustness of program impact estimates: the coverage

of outcomes, survey error, data matching practices, and conventions for the treatment of missing observations.

These factors, individually or combined with each other, can affect the validity and reliability of the resulting

program impact estimates.

Evaluation and nuclear waste management in France: actor repertoires, learning and mistrust

Markku Lehtonen EXPOSÉ/PAPER After a long, conflict-ridden history, the French plans for deep geological disposal of medium and high -level

radioactive waste have reached a stage at which the repository to be built between 2018 and 2025. This complex

and contested "megaproject" promises to benefit its economically declining host region, yet it also generates

significant health, environmental, and socio-economic concerns. Drawing on fieldwork in the planned host region,

this paper examines the potential and the pitfalls of evaluation of the socio-economic aspects in the governance of

the project, in a conflict-ridden context characterised by widespread mistrust between actors, and resistance and

scepticism towards evaluation. This context of mistrust, together with the deeply entrenched positions of several

key actors, tends to hinder the capacity of evaluation to foster learning. Yet, the criticism and mistrust against the

project could also be harnessed for creating conditions favourable to learning.

Evaluations that make a difference: What we have learned from around the world

Burt Perrin, Rochelle Zorzi, Martha McGuire

PANEL/PANEL

46

The project Evaluations that Make a Difference: Stories from around the world (Evaluation Stories) received an

EvalPartners Innovation Challenge Award to promote the use of evaluation. Evaluation Stories is using the

universally accessible form of stories to share examples of how evaluations have made a true difference to the lives

of program recipients. Ten stories have been chosen from six regions (Europe, Africa, Australasia, South

America/Caribbean, Asia and North America). In this panel, we will share some of the stories of evaluations that

have led to changes in people's lives. The stories are told from the perspective of evaluation users and program

participants, and provide colourful examples of how evaluation can be a force for social betterment. The panellists

will draw lessons from across the stories about how to undertake evaluation so that it is more likely to result in

positive change.

From First International Evaluation Conference to UN Year of Evaluation - What has Changed?

Andy Rowe, Linda Lee, Shelley Borys, John Mayne, Francois Dumaine PANEL/PANEL The panel includes four CES Fellows whose work spans the period from before the First International Evaluation

Conference co-sponsored by CES in 1995 to 2015 and this International Year of Evaluation and will be chaired by a

former CES President. They will identify important changes in evaluation over that period and present their views on

some of these. The audience will be asked to identify additional important changes and from these select some for

further discussion by the audience and panel. A summary from the chair will use the dis cussions to identify

important gains in evaluation and point to important issues that evaluation should address.

Gender-responsive program evaluation: Perspectives from UN agencies, public sector, and consultants

Cyuma Mbayiha, Daniel Sansfaçon, Emmanuel Trépanier, Jane Whynot, Vanessa Anastasopoulos

PANEL/PANEL Recognition of the importance of incorporating gender into evaluation policies and practices has been growing

steadily across national and international evaluation communities. This presentation bri ngs together a diverse group

gender and evaluation specialists from UN Women, Universalia, the University of Ottawa, and Status of Women

Canada, to explore the role of gender in evaluation. Panelists will share their unique perspectives regarding the

value and objectives of gender-responsive evaluation, how it is carried out within their work and organizations,

lessons learned, and challenges faced. Through this forum, approaches and methodological considerations aimed at

inspiring the implementation of gender-responsive evaluation will be presented.

Improvement of development outcomes through an evaluation: Uganda’s evaluation of its Poverty Eradic

Albert Byamugisha, Alex Bashasha Turyatemba

EXPOSÉ/PAPER The evaluation of Uganda’s Poverty Eradication Action Plan (PEAP) is a rare example of a nationally-driven evaluation

of a country’s poverty reduction strategy whose findings have potentially made an improvement in the livelihoods of

the people by: influencing policy and improvement in the performance of Government programs. Uganda’s PEAP

started in 1997, and was the first national poverty plan in Africa, pre-dating and informing the World Bank-

supported Poverty Reduction Strategy Papers that spread across the continent thereafter. The PEAP was updated in

2000/01 and in 2003/04. By 2007, the Government decided a new direction and new type of plan was needed, and

set about designing a broad ranging evaluation that would provide a measure of what had been achieved under

PEAP, and importantly set the direction for the new plan. This proved insightful, with cross-government coordination

of the evaluation leading to evaluation findings discussed by Cabinet under a white paper, and lessons

Introduction des Smartphones dans la collecte quantitative pour l'évaluation

Isabelle Agier, Valéry Ridde EXPOSÉ/PAPER La rapidité de mise à disposition de données probantes est un atout essentiel pour l'évaluation des interventions.

Les enquêtes (papier) ont vu leur efficacité s'améliorer avec l'apparition des PDA (fin 80) et des téléphones

47

intelligents (2000). Cela a constitué des gains successifs d'efficacité et de coûts pour l'évaluation et imposé de

repenser le travail de collecte à toutes les étapes. Cette communication montre comment le téléphone intelligent a

permis l'amélioration de la collecte de données par rapport au papier et au PDA. Se basant sur l'expérience de sept

collectes en 2012 et 2013 dans le cadre d'un programme de recherche au Burkina Faso (IRSC), nous décrivons les

défis technologiques rencontrés sur le terrain, les réponses apportés et les adaptations nécessaires tout au long de

la démarche. Enfin, nous exposons les nouvelles opportunités offertes pour améliorer la rapidité de l'accessibilité

des données.

L'analyse logique comme outil de changement des pratiques favorisant l'accès aux soins

Marie Beauséjour, Mylaine Breton, Michael Eshiemokhai, Jean Ouellet, Astrid Brousselle

PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION L'analyse logique permet de porter un jugement sur le bien-fondé d’une intervention en examinant sa nature et en

remettant en cause ses modalités. Nous y avons ici recours pour soutenir l'élaboration d'une intervention de triage

raisonné des demandes de consultation et de priorisation des patients en clinique de soins spécialisés. Ces stratégies

sont réputées efficaces pour favoriser un accès approprié et en temps opportun aux soins de santé. Or, il s'agit

surtout d'initiatives locales et peu d'auteurs ont tenté de modéliser ces approches pour en comprendre les

composantes clés. Ce travail a consisté à comparer le modèle logique d'une intervention proposée par les

professionnels d'un hôpital montréalais à un cadre de référence élaboré d'une revue de la littérature afin de

proposer la meilleure intervention possible pour le milieu. L'intérêt de cett e approche en soutien à la prise de

décision sera discuté.

L'analyse textuelle par ordinateur : un outil utile à l'évaluation de la mise en oeuvre ?

Jean Bélanger, Gilles Roy, François Royer EXPOSÉ/PAPER L'évaluation de la mise en oeuvre d'une intervention nécessite d'en documenter les stades initiaux de mobilisation

et d'appropriation. L'évaluation de ces étapes nécessite généralement le recours à des méthodes de type

qualitatives qui nécessite un temps d'analyse assez long, limitant ainsi un retour rapide aux acteurs pour qu'ils

puissent utiliser cette information. L'appel à des méthodes d'analyse assistée par ordinateur est possiblement une

piste prometteuse afin de lever ce problème. La présente conférence vise donc à vérifier, à partir de données

provenant de deux évaluations de mise en oeuvre, l'utilité réelle de logiciels tels que ALCESTE et SÉMATO en

comparant les résultats d'analyse produits sur des réponses de plus de 700 répondants, à ceux générés à partir

d'une analyse thématique manuelle. Les résultats montrent une utilisation prometteuse de ce type d'outil tout en

présentant ses limites.

Measuring and building evaluation capacity in Ontario public health units

Isabelle Bourgeois, Louise Simmons, Nikolas Hotte, Raïmi Osseni

PANEL/PANEL Ontario public health units (PHU) are responsible for developing and implementing various public health programs in

their communities. Each unit is expected to conduct program evaluations when new interventions are implemented

or when there is evidence of unexpected operational issues or program results. Increasing accountability pressures

and evolving performance management systems aimed at demonstrating the value of its programs and services are

driving PHU to enhance their local program evaluation efforts. This panel will present various aspects of a

collaborative study conducted to measure the evaluation capacity of 28 PHU. The objective of the study was to

identify key organizational characteristics associated with higher levels of evaluation capacity, in order to support

future capacity development. Data were collected through the use of a standardized tool developed by Bourgeois et

al (2013), as well as key informant interviews with a representative from each of the PHU.

Mixing methods for strengthening impact evaluations

Shagun Sabarwal, Heather Lanthorn

48

EXPOSÉ/PAPER Multiple forms of data and analysis enable us to not only make causal claims about what works in development but

also why, when and where which assumptions (in a theory of change) are met. We pres ent guidelines for conducting

high-quality and theory-driven impact evaluations (IE) built on a meaningful mix of methodologies. By mixed -

methods (MM), we specifically refer to integrating qualitative and quantitative research methods for data collection,

analysis and interpretation. The present paper will produce a pre-analysis plan template for conducting qualitative

research as a part of an IE and specific guidelines for conducting high quality MM IEs that fulfill the goal of learning

what works, why, and in what contexts. We will review all MM IEs currently included in 3ie's IE Repository, which is

an index of all published IEs of development interventions. We address an important lacuna: moving from an

expressed goal to engage in more MM IEs to providing a blueprint of how to do so

Monitoring Impact Funds

Leonardo Santos, Maria Parreiras, Renato Gouvêa

EXPOSÉ/PAPER Investment Impact Funds are similar to Venture Capital funds in several aspects, but with one fundamental

difference; their performance is also measured by the benefits provided to lower income classes. Funds invested in

startups induce management and governance changes, suggest projects to raise capacity and search for new

business opportunities As a contribution to the impact funds management, the developed model presented assists

the monitoring from investor's internal activities to the observation of social benefits and return of investment. Two

models used by the development banks to plan, monitor, evaluate and communicate their investment performance

were used to create the final model to monitor investment impact funds. Investment impact funds complexity was

represented by a synthesis developed by Roduner and Ambrose of two models: Logical Framework and “Outcome

Mapping.

Multidimensional poverty index approach for program evaluation, an experience from Afghanistan

Abdullah Al Mamun EXPOSÉ/PAPER In many countries, the policymakers of donor and partner agencies consider the results of multiple indicators to

measure the status of poverty reduction improvement programs. They also use the results of aforesaid indicators to

decide on further investment to the programs. For example, financing to those partner agencies show positive

results of a set of indicators. Sometimes, agencies are only interested to consider households' income as an indicator

to measure the status a program. This paper will argue that although many donor and partner agencies consider to

use the results of multiple indicators at a time or use of a single income related indicato r such as 'households'

income' to measure the status, such as success or failure of a poverty reduction, it is better to consider an additional

single index that consists of the results of concerned indicators using 'Multidimensional Poverty Index (MPI)'.

Partners in learning through evaluative thinking to transform and build the capacity to use evidence

Keiko Kuji-Shikatani, Doris Mcwhorter, Joannie Leung, Dianne Oliphant, Mary Richards

PANEL/PANEL Our experience over the past decade illustrates the power of evaluation to improve and transform education

systems. Guided by the ministry’s Research and Evaluation Strategy (2005), evaluation and evaluative thinking have

been built into policy and program design and implementation. Partnerships between internal and external

evaluators mutually build evaluation capacity, contributing to increasing the positive outcomes for all Ontario’s

students. This presentation will illustrate the achievements and challenges of evaluation and its potential to guide

and inform the implementation of Ontario’s Renewed Vision for Education (2014) including, a five -phase, mixed-

methods evaluation on the impact, spread, adaptability and sustainability of Collaborative Inquiry as a professional

learning approach, and evaluation of initiative that offers differentiated and intense supports to district school

boards to build principals capacity as the instructional leader for student success and ECB efforts.

49

Sustainable Equity Programs in Higher Education: Designing and Testing an Evaluation Resource

Jenny De Vries

PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION Evaluation of equity programs is a vital aspect of the Australian higher education system, but equity practitioners

feel unsupported in this task. This presentation details the development of a resource to assist in this task. During

2014, through a competitive grants scheme, a total of $181M, was distributed by the Australian Federal Government

so that programs could be delivered across Outreach, Access and Support programs. Ong oing funding is not assured

and universities must make hard decisions on which programs are having the most impact in this equity space. This

case study is situated in one of the eight Western Australian universities which all compete for funding. This cas e

study research will develop and trial an evaluation resource to assist practitioners to determine if their many and

varied programs are actually making a difference to the people identified within the various equity groups. A mixed

method research design has been selected for this study.

Utiliser les téléphones mobiles pour évaluer l'impact des projets agricoles

Samuel Kouakou EXPOSÉ/PAPER A l'instar de nombreux pays africains, la Côte d'Ivoire fait face à une forte augmentation de l'utilisation des

téléphones mobiles en milieu rural, particulièrement par les agriculteurs. Cette évaluation pilote a été réalisée, en

testant les possibilités de mettre en ½uvre un système par SMS, pour évaluer l'impact socio -économique de l'Ex-

projet de riziculture financé par l'Union européenne dans la région Centre-Nord. FrontlineSMS est un logiciel libre

qui établit un double sens messagerie-texte en utilisant seulement un ordinateur, un modem et des téléphones

cellulaires GSM. Mais, existe-t-il suffisamment de connaissances au sein des évaluateurs pour utiliser efficacement

cette technologie? il ressort de l'étude que cette technologie innovante offre une opportunité, de réduire la

distance, le temps et le coût de l'évaluation des projets. Cependant, il reste nécessaire d'a méliorer la connectivité en

milieu rural pour une mise en ½uvre réussie.

Launch of CES Annual Conference Fellows' Strand

Andy Rowe, Linda Lee, Shelley Borys, John Mayne, Francois Dumaine LANCEMENT/LAUNCH

14h45-16h35

Comment identifier les processus performants d'une intervention ergonomique ?

Valérie Albert, Henriette Bilodeau, Nicole Vézina, Fabien Coutarel PRÉSENTATION AFFICHE/POSTER PRESENTATION Une intervention ergonomique visant à améliorer la santé des travailleurs et la production est conçue sur mesure et

peut comporter plus d'une centaine d'actions (processus) influencées par le contexte de l'entreprise. Alors que les

modèles d'évaluation dominants en ergonomie ne s'intéressent qu'aux effets produits, il semble que l'aspect

généralisable de l'intervention concerne plutôt les processus «performants», soit ceux qui ont généré les effets dans

un contexte particulier. Une étude-pilote a donc permis de dégager des indicateurs à inclure dans un nouveau

modèle d'évaluation des interventions ergonomiques. Celui-ci devra se baser sur une analyse quantitative et

qualitative des processus, inclure les éléments-clés du contexte qui ont influencé les processus et intégrer les

perspectives de l'ergonome et d'acteurs-clés de l'entreprise pour documenter les mécanismes de production des

effets.

Diffusion de Ces années incroyables: Évaluation de son implantation en protection de l'enfance

Isabelle-Ann Leclair M.

PRÉSENTATION AFFICHE/POSTER PRESENTATION

50

CAI, PEHP probant largement diffusé, est implanté depuis 2003 au Centre jeunesse de Montréal (CJM). Le modèle

d'action de Chen stipule que 6 éléments interagissent dans l'implantation de programmes (organisation; personnes

qui implantent; partenaires; contexte écologique; protocole d'intervention/livraison de servi ce; population cible). La

fidélité d'implantation résulte de l'adéquation entre ce qui est prévu dans ce modèle et ce qui est fait en réalité.

Cette présentation décrit à partir de ce modèle, l'implantation de CAI au CJM. Seront présentés les résultats

provenant de données colligées depuis plus de 10 ans (plus de 34 groupes, 20 animateurs, 248 participants)

concernant dosage, participation, qualité et adhérence au programme. Les résultats suggèrent que l'implantation est

fidèle à la planification. L'accent sera mis sur le processus d'évaluation continu et la collaboration entre le milieu

clinique et de la recherche.

EVALUABILITY OF A STRATEGY FOR INTEGRATING EDUCATION-SERVICE-COMMUNITY IN HEALTH AREA

Marly Marques, Catia Oliveira, Raquel Torres, Ana Cristina Reis, Solange Kanso, Aline Leal, Silvia Carvalho PRÉSENTATION AFFICHE/POSTER PRESENTATION The Program for the Enhancement of the Primary Care Professional (PROVAB) is an initiative by the Brazilian Ministry

of Health established in 2011 and designed to address one of the challenges of the National Health Service: the lack

of healthcare professionals on the outskirts of big cities, the countryside and remote areas. This study aimed to

conduct an Evaluability Assessment (EA) of PROVAB that stages: identifi cation of stakeholders, strategic analysis,

modeling of the intervention and recommendations. Data collection: review of official documents, meetings and

semi-structured interviews. The Results showed that PROVAB will be able to contribute in reducing ineq ualities in

access to health care; however, the study revealed some weaknesses in management that need to be reviewed for

the second cycle of the program, such as increased dialogue with the programs doctors, more investment in

educational technologies and institutional support and improvement of the information flow.

Evaluating the Evidence on the Efficiency of Performance-Based Financing in Lower Income Countries

Anne-Marie Turcotte-Tremblay, Jessica Spagnolo, Manuela De Allegri, Valéry Ridde

PRÉSENTATION AFFICHE/POSTER PRESENTATION Context: Performance-based financing (PBF) is being implemented in low- and middle- income countries (LMICs) to

improve health care services and health outcomes. The objective of this literature review is to evaluate the streng th

of the evidence regarding the efficiency of PBF in LMICs. Methods: First, we scanned the reference lists of two

systematic reviews on the efficiency of PBF to identify articles that met the established inclusion/exclusion criteria.

Subsequently, we searched for new English and French articles in PubMed, EconLit, Google Scholar and Google.

Results & Discussion: Seven articles were included in this review, and classified as partial economic evaluations.

None of the included studies made a clear connection between the costs of PBF in LMICs and its effects. Based on

these results, we reject the proposition that PBF initiatives are empirically proven to be efficient interventions in

health care delivery in LMICs.

Evaluation as a Reflective Practice

Nathan Raja

PRÉSENTATION AFFICHE/POSTER PRESENTATION Reflection questions, W5H1, shorter form: what, so what, and now what

Evaluation du Programme Micro FEM (SGP) au Sénégal

Ndeye Fatou Diop, Dieynaba Dia

PRÉSENTATION AFFICHE/POSTER PRESENTATION Le Programme de Micro-Financement du Fonds pour l'Environnement Mondial (PMF/FEM) est financé par le FEM

pour appuyer et promouvoir les initiatives d'actions communautaires visant à faire face aux grands problèmes

d'environnement dans le monde. Ceci à travers la mise en ½uvre de projets (d'un montant maximal d'USD 50 000),

relatifs à cinq domaines de priorité du FEM : la diversité biologique, le changement climatique, les eaux

internationales, la gestion durable des terres et les polluants organiques persistants. Lancé en 1993 au Sénégal, le

51

PMF/FEM est exécuté par le PNUD pays en collaboration avec l'Etat du Sénégal. Les bénéficiaires sont les ONG et les

OCB .Depuis sa création, le PMF a subventionné plus de 260 projets. L'évaluation conjointe FEM/PNUD lancée au

mois de mai-juin 2014 entre dans le cadre de la préparation du sixième plan opérationnel et de l'évaluation du

programme global qui a concerné 8 pays.

Evaluation of the Supervision Needs and Efficiency of an NCR Agency

Julie Gosselin, Sophie-Claire Valiquette-Tessier, Marie-Pier Vandette

PRÉSENTATION AFFICHE/POSTER PRESENTATION In many helping professions, clinical supervision is the main vehicle for professional development (Falender &

Shafranske, 2007). Thus, the purpose of this evaluation, requested by a National capital region (NCR) agency, was to

identify the needs, challenges, and strengths of supervisors and supervisees, and assess the efficacy of current

supervision practices. The lines of evidence consisted of a document and file review, grou p discussions, and

interviews. First, findings indicated a need for additional concrete resources, and a need to develop a more

competency-based supervision organizational culture. Second, respondents noted a number of strengths

contributing to the efficacy of current supervision practices, including the agency's commitment to its core values.

Third, there was recognition of supervision's impact on crisis management, but less so on more regular work

activities. Finally, respondents identified ways in which supervision activities could be enhanced.

'Experts' learning from communities: Lessons from rural East Africa

Njoroge Kamau

PRÉSENTATION AFFICHE/POSTER PRESENTATION Majority of development agencies strive to fully involve target beneficiaries in project mo nitoring. Communities may

participate in initial project start up phases but not monitoring and eventual evaluation often. Community Led Total

Sanitation (CLTS) is a sanitation and hygiene promotion strategy that aims at inspiring communities to stop open

defecation. Unlike traditional sanitation and hygiene approaches, CLTS advocates for non -subsidy and encourages

communities to use locally available construction materials to build latrines. The community decides that all

community members will build and use latrines. Monitoring of the progress towards eradication of open defecation

is also done by the community. During this project, we were pleasantly surprised to see that the target communities

designed their own monitoring tools. Even more surprising is that the tool developed by community members was

richer and more informative than our 'expert' monitoring tool.

Focus Groups - Misconceptions, Implications and Insights

Atena Bishka

PRÉSENTATION AFFICHE/POSTER PRESENTATION Focus groups - misconceptions, implications and insights

Leçons apprises : Évaluations et méthodes pour enquêter auprès des populations difficiles à joindre

Jérémie Butoyi, Valéry Ridde, Susanna Ogunnaike-Cooke PRÉSENTATION AFFICHE/POSTER PRESENTATION Cette affiche présente les leçons apprises à travers différents projets d'études - évaluation de faisabilité portant sur

la santé des populations difficiles à joindre (sans papiers, PPVIH) en déhors des structures de services connues. Elle

montre comment créer les conditions de mise en oeuvre et d'acceptabilité des projets et présente solutions

innovatrices pour optimiser l'accès à une populations diversifiées en particulier celles qui mènent une vie

clandestine. Elle aborde aussi les méthodes d'opérationnalisation du recrutement à travers un processus d'éducation

et de responsabilisation qui transforme le participant en complice pour une action rapide en misant sur la nécessité

de se prendre en main. L'évaluateur peut rapidement accéder aux usagers potentiels utilisant la méthode du temps

et de l'espace (TLS)pour avoir des données de première mains.

52

M&E Translated: Tools for Participatory & Sustainable Systems for International Implementers

Elizabeth Lewis

PRÉSENTATION AFFICHE/POSTER PRESENTATION Developing sustainable monitoring and evaluation (M&E) systems is a challenge for any implementing organization;

international organizations face additional cultural and logistical challenges. The International Republican Institute

(IRI) — a non-partisan, non-profit organization committed to advancing democracy worldwide — has worked since

2008 to develop M&E systems that can be translated across language, culture and organizational capacity. IRI has

adapted best practices from the evaluation community to be applied to programs designed to deve lop democratic

institutions, including political parties, legislative or executive government and civil society. This poster will highlight

three tools IRI developed in its quest to promote an organizational culture that values M&E, including an M&E

capacity building training curriculum for use with local staff and partners, a handbook for M&E of democracy

assistance programs, and finally an M&E systems diagnostic tool.

Models for providing gluten-free meals in hospitals & continuing care facilities: A cost-analysis

Nour Hammami PRÉSENTATION AFFICHE/POSTER PRESENTATION Individuals with celiac disease world-wide (~43% of Canadians) have difficulty obtaining gluten-free (GF) meals,

including in hospitals and retirement homes. Immediate attention is needed sinc e celiac patients admitted to

hospitals and continuing care facilities (CCF) require proper nourishment to maintain good health. The objective of

this study was (1) to identify the most cost efficient strategy to provide GF meals for celiac disease patie nts in a

hospital in Lebanon and (2) to develop a decision tree to guide hospitals and CCF in identifying the most suited plan

for their needs in adopting GF meals. After revision of GF food chain guidelines, 4 options were compared (e.g.

purchasing new equipment for GF products, finding a dedicated contaminant free space, purchasing from a GF

bakery..) for implementation in a hospital in Lebanon. From here a decision tree was developed that will assist

facilities all over the world in providing GF meals to their patients in need of such a diet.

Panel sur les Jeunes évaluateurs africains

Aminata Diop, Amos Menard, Ousséni Kinda, Ya Cor Ndione PRÉSENTATION AFFICHE/POSTER PRESENTATION Ce panel de jeunes évaluateurs africains porte un regard original sur 4 sujets d’'actualité africaine: la place des

jeunes dans les politiques publiques; l’'emploi des jeunes; les jeunes évaluateurs et la professionnalisation ; la

protection des enfants et adolescents. L’'évaluation peut aider à changer les perceptions, parfoi s négatives, de la

jeunesse en valorisant leurs capacités d’'apporter des réponses innovatrices aux problèmes les concernant. Pour

cela, une rupture s’impose dans la conception, la mise en œuvre et le S&E des programmes de jeunes. Une pratique

évaluative plus souple et inclusive va établir la confiance et le respect mutuel afin de prendre au sérieux la voix des

jeunes. Il faut des méthodes adaptées au contexte africain et sensible aux valeurs, aux conditions de vie et modes de

communication des jeunes et des enfants, qui deviennent ainsi des acteurs à part entier. Comment apporter ces

changements? Quel sera le rôle des jeunes évaluateurs?

Patient partnership approach to diabetes care in Eeyou Istchee: Identifying barriers and facilitators

Elena Kuzmina, Catherine Godin, Paul Linton , Marie-Pascale Pomey

PRÉSENTATION AFFICHE/POSTER PRESENTATION The Cree regional health care is facing many challenges for implementing a patient partnership approach to diabetes

care. The goal of this project is to identify barriers and facilitators for diabetes management as perceived by Cree

patients at the personal, organizational and environmental levels. Individual, structured interviews were conducted

with nine Cree patients registered in the Cree Diabetes Information System. Our results suggest that diabetes care

could be improved by increasing Health Care Professionals' capacity to deliver culturally sensitive care, facilitating

shared decision making for diabetes management, involving patients in the quality improveme nt process,

implementing a patient-friendly follow-up system, and reinforcing a supportive environment in the community.

53

Incorporating patients' experience in the delivery of health care is a critical element for building patient partnership

care and empowering the community in order to achieve sustainable results.

Pratique de l'évaluation au Cameroun et performance des Politiques publiques

Julien Ntouda PRÉSENTATION AFFICHE/POSTER PRESENTATION Depuis plus d'une décennie, l'évaluation des politiques publiques est au c½ur des préoccupations des pays d'Afrique

subsaharienne. Au Cameroun, les statistiques mettent en évidence des niveaux particulièrement faible de

l'évaluation des programmes et projets publics. Le question de recherche de cet article est de savoir comment la

pratique de l'évaluation dans l'administration publique camerounaise parvient -elle à influencer la performance des

programmes/projets publics? Ainsi, son objectif est de montrer l'influence de l'évaluation sur la performance des

programmes et projets publiques camerounais. L'étude s'appuie sur les données issues de l'enquête sur la

performance de la chaine de résultat des administrations au Cameroun. Deux méthodes d'analyse seront utilisées

dans cet article. L'analyse descriptive et l'analyse économétrique. grâce à la méthode paramétrique de frontière de

production stochastique

Prevention of social violence: an integral and articulated experience, Mexico

David Guzman Matadamas PRÉSENTATION AFFICHE/POSTER PRESENTATION In order to contribute to the development and considering that every action sums in a positive manner the issue

describe in the presentation “Prevention of social violence: an integral and articulated experience, Mexico” the

Crime and Violence Prevention Program is been implemented true a multilevel engagement and promotion the CVPP

has accomplish the operation of accurate interventions in a sub-national intervention. As part of the international

policy of USAID, the beneficiaries had help to pilot interventions models focus in environmental design, at risk youth,

community healing, and resilience and a local communication campaign. It’s been tested that modeling theories in

practical interventions the CVPP has a second phase for extend its life and contribute to enhance the quality of

people affected by a huge social problem: social violence.

Saying 'No:' The Challenges of Ethical Evaluation Practice using Social Work Values

Megan Elyse Williams, Laura Sundstrom

PRÉSENTATION AFFICHE/POSTER PRESENTATION As evaluators and social workers, ethical practice is paramount. However, clients often present challenging ethical

dilemmas. Client needs take precedence, but sometimes evaluators have to say no or set boundaries. This poster will

use case examples of four ethical dilemmas commonly faced by evaluators and how they were addressed to preserve

a good client relationship, yet following the ethics set forth by the social work profession in the context of

evaluation practice.

Strengthening evaluation influence with experiential learning: lessons from University of Waterloo

Jennifer Yessis, Lisa Stockton, Ellen Maceachen PRÉSENTATION AFFICHE/POSTER PRESENTATION High quality relevant evaluation studies are needed to strengthen public health and health system actio ns. But

knowledge, skills and organizational cultures that enable strong evaluations are limited across Canada. Academic

participation in evaluations is especially limited. The University of Waterloo's School of Public Health and Health

Systems and the Propel Centre for Population Health Impact are initial partners in a collaborative approach to

improve health and health systems through vibrant academic and practice partnerships. Initial priorities are

graduate student practicum placements and a 1-week learning institute, both of which emphasize experiential

learning, collaboration, and mutual exchange for those conducting, supervising and using evaluations. This

presentation will describe the design of experiential learning opportunities, and early lessons such as considerations

for academic and on-site mentoring, experiences from field placements, and roles of all involved.

54

The Art of Self-Reflection Crossing Flow and EvaluationLive!

Natalie De Sole, Melanie Hwalek

PRÉSENTATION AFFICHE/POSTER PRESENTATION Can reflective practice and self-reflection turn around a "ho hum" evaluation experience? When looking for ways to

improve an experience, self-reflection is often overlooked. However, Schön (1983), Patton (2011), and Barrington

(2012) all uphold the human element as an essential factor for success and informational uptake. This roundtable

will highlight the importance of self-reflection and provide tools for enhancing evaluation practice using self -

reflection, including Csikszentmihalyi's Flow model and Melanie Hwalek's EvaluationLive! practice model. Audience

members will be guided through a reflective activity from the EvaluationLive! model. Feedback from this activity will

help further the field's focus on ways to incorporate and use reflective practice to continually self-improve our

evaluation practice.

The M&E System of the Palestinian Ministry of Education and Higher Education

Mohammad Mousa, Rabiha Ehlian PRÉSENTATION AFFICHE/POSTER PRESENTATION Executive Summary Since 2009, the annual M&E report focused on monitoring the EDSP Key Performance Indicators

(KPIs). As of 2012, MoE has begun to institutionalize an annual evaluation mindset and function as a key prerequisite

for the implementation of the Program Based Approach (PBA) and Sector W ide Approach (SWAP), which are

supported by the Joint Financing Arrangement (JFA). The findings for 2012 are presented cumulatively from the

2009 Baseline together with the methodology applied in the case of each indicator. Sources of data for this repo rt

are multiple. Apart from the EMIS statistics, data originated from different MoE DGs (e.g.: Assessment and

Evaluation, Field Follow Up, Counseling, CDTP, Planning, Health, Finance, Curricula etc.). Each of the eight quality

KPIs has a separate technical team, which is composed of members from DGs and university experts.

16h35-17h35

A different perspective- Can Social Return on Investment help to tell a better story?

Alison Jette, Chantal Langevin

TABLES RONDES/ROUNDTABLES Evaluations of federal programs need to assess both economy and efficiency. This can be a challenge for social

programs, as the impacts of the program funded by one sector can be felt in other sectors. This also makes the

collection of financial data challenging. A standardized approach to monetizing the social value created by the

program would provide the evaluation with fuller information on the benefit of the investment. Social return on

investment is one method to do this. Health Canada will be evaluating the National Native Alcoho l and Drug Abuse

Program and piloting the use of SROI with a treatment centre. With this roundtable, we will be sharing our insight

with others who have used, or are considering using, SROI as well as those engaged in the evaluation of substance

abuse treatment. We hope this group discussion will help to highlight, for those participating, the strengths and

weaknesses of SROI in this context, as well as how it can be used to influence senior management.

Applying a multi-level implementation framework to program evaluation: a case study

Eunice Chong, Adrienne Alayli-Goebbels, Anne Philipneri , Helen Cerigo, Lori Webel-Edgar, Sarah Muir,

Heather Manson EXPOSÉ/PAPER Implementation science is the study of methods to bridge the gap between research and practice, and aims to

understand how evidence-based programs can be effectively implemented in real-life settings. The goals of

implementation science often overlap with the goals of program evaluation. There are many existing

implementation frameworks that were developed in an attempt to understand the process of program

implementation (i.e., barriers and facilitators), and these frameworks can guide the development of evaluation plans

and questions. For the process evaluation of Ontario's Healthy Babies Healthy Ch ildren (HBHC) program, a multi-

55

level implementation framework was used to guide the development of the evaluation questions and to identify

relevant factors in the HBHC program implementation. In this presentation, the author will provide an overview of

the application of this framework to evaluation planning, present selected findings, and discuss the limitations of the

framework.

Contraintes liées l'évaluation au Cameroun : à la recherche des facteurs explicatif

Julien Ntouda

PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION Depuis plus d'une décennie, la pratique de l'évaluation des politiques publiques est au c½ur des débats des

communauté scientifiques et professionnelles en particulier Afrique subsaharienne en particulier. Au Cameroun,

cependant la pratique de l'évaluation dans les administrations publiques reste très faible. Le questionnement à la

base de cet article et d'identifier les facteurs qui font obstacles à la pratique de l'évaluation au Cameroun. Ainsi, son

objectif est d'identifier et valoriser les facteurs qui favorise la pratique de l'évaluation au Cameroun. l'étude s'appuie

sur les données issues de l'enquêtes sur la performance de la chaine de valeurs des administrations publiques

camerounaises. Deux méthodes d'analyses seront utilisées: descriptive et économétrique.la variable dépendante

étant qualitative et dichotomique on utilisera le modèle logit.

Enhancing evaluation impact and use at Environment Canada

Nicole Michaud, Susan Wharton EXPOSÉ/PAPER Environment Canada's Evaluation Division has implemented a number of tools and approaches over the years aimed

at improving and monitoring the impact and use of its evaluation products and services and supporting a culture of

continuous improvement. These processes occur at each phase of the evaluation, namely at the planning,

conducting, reporting and project closure. The presentation will touch briefly on the suite of tools and processes, but

will focus primarily on new processes implemented at evaluation project closure. Building on a recent study and

evaluation project pilots conducted in other federal agencies, the Evaluation Division is currently piloting new tools

to increase stakeholder engagement at the project closure phase for enhanced evaluation use and continued

learning within the department. The presentation will outline early results from the pilots, expected and actual

benefits and lessons learned gathered through the reflections of evaluators and evaluation users.

Evaluating Programs to Inform Systems Level Change

Laura Sundstrom, Megan Elyse Williams

EXPOSÉ/PAPER Programs are operating in systems (i.e. criminal justice, education, behavioral health, etc.), yet most evaluations are

conducted at the program level. Planning evaluations at the systems level can benefit clients in a number of ways:

the use of shared knowledge, methods, tools, data metrics, and relationship building across organizations and

programs. Through systems level thinking, utilization of evaluation results for system level change is maximized.

Presenters will describe how systems level thinking, drawn from a social work framework, informed four program

evaluations across the criminal justice system.

Evaluation for a better world: Influencing use in non favorable contexts

Astrid Brousselle, Damien Contandriopoulos CONFÉRENCE D’EXPERT/EXPERT LECTURE Context is the major determinant of evaluation use. Contrary to prevailing beliefs in our field, it is neither the

evaluation approach itself nor the role of the evaluator that is the main determinant of use. In fact, there are

contexts in which results are unlikely to be used, whatever the personal qualities of the evaluator and the intensity

of participation. When an evaluator is stuck in a context where it is unlikely that th e results will be used, what are

some avenues that might be pursued to increase the evaluation's potential for influence? First, based on the

framework developed in our previous work, we will describe the characteristics of non -favourable contexts as

56

compared to favourable and political contexts. Second, we will explore avenues for influencing contexts'

characteristics based on theories of the policy process (Sabatier and Weible, 2014). Third, we will identify some

ethical tensions this reading of contexts implies for evaluators' practice.

Evaluation Standards: Use and Potential to Build A Better World

Brenda Stead

TABLES RONDES/ROUNDTABLES The Program Evaluation Standards, Third Edition (2010), developed by the Joint Committee on Standards for

Educational Evaluation (JCSEE), has been approved by the American National Standards Institute, and adopted by

the Canadian Evaluation Society (CES). The Classroom Standards are in the final stage of revision. This session will

begin with a short presentation on the status of both these sets of Standards, and then focus on the Program

Evaluation Standards - a pillar of the CES Professional Designation Program and evaluation professionalization.

Focused discussion questions will be used to guide participants in sharing their interest in evaluation standards,

experience in or desire to applying the Standards (through work or studies) in their real world situations. Together

we will explore the challenges and opportunities in building knowledge about and capacity in the evaluation

standards of utility, feasibility, propriety, accuracy and accountability - for a better World.

Evaluations for Good Governance- Need for Enabling Environment

Rashmi Agrawal EXPOSÉ/PAPER Evaluations and good governance are mutually supportive and complement each other in catalyzing development.

Effectiveness of evaluations in contributing to better governance, however, is often constrained by country's socio -

economic and political milieu and is conditioned by factors such as lack of owner ship of evaluations, political

compulsions that prevent policy modifications, inadequate attention to use of results, conceptual and

methodological constraints in impact evaluations, evaluation quality etc. This point is brought out through examples

from Indian experience in evaluations of development interventions. The need for appropriate enabling environment

with National Evaluation policy that promotes an evaluation culture and systematizes M&E in the country is

emphasized. Capacity building and sensitization at various levels of operations is crucial to get over the constraints.

‘Community' as a major stakeholder in development and evaluations can be the agent of change.

Facilitating the Use of Evaluation Results

Derek Wilson, Chris Lovato, Tamiza Abji

EXPOSÉ/PAPER For a learning organization, the evaluation of programs is a continuous process which does not end when a final

report is delivered. In fact, the active use of evaluation results for planning and decision -making is one of the most

important and challenging elements of the program improvement cycle. At the University of British Columbia’s

Faculty of Medicine, an evaluation and improvement process has been established whereby evaluation

recommendations are routinely brought forward to leadership for action. The monitoring elements of this process

have recently been strengthened through a series of structural and procedural enhancements. This presentation will

outline the strategies developed and how they are enabling and reinforcing use of evaluat ion results, including

positive gains, challenges, and our insights on lessons learned.

Finding a Pathway: Supporting the Development of an Evaluation Culture at the Canada Council

Linda Lee, Shannon Peet EXPOSÉ/PAPER An evaluation of the Canada Council's Suite of Aboriginal programs is underway, representing the first

comprehensive program evaluation at the Council. An initial step was to develop a logic model which would

represent the suite of 15 Aboriginal Arts programs. This paper presents the organizat ional context, process for

development, final logic model (now known as the Pathway) and its impact on the institution. The development

57

process needed to include input and buy-in from Council staff, as well as culminate with the creation of a 'logic

model' that would be appropriate for the Aboriginal programs. The Pathway combines the circular with the linear,

representing a coming together (or hybrid) of Indigenous (circular, holistic) and 'Western' (sequential, causal) ways

of conceptualizing the world. As important as the product was, the role the process played in supporting the

development of an evaluation culture at the Canada Council, was equally significant.

From "translation" to "transcriation": M&E for change

Elizabeth Moreira Dos Santos, Egléubia Oliveira, Gisela Cardoso

EXPOSÉ/PAPER Recent experiences in Brazil, aiming at improving the quality of NHSS, has examined the evaluation models

addressing the performance of 'supporters' essential to the functioning of the strategic and operational netwo rks of

the Brazilian health care system. This essay seeks to explore critical topics for the development of theories that

describe the intervention, modeling the evaluation and the use of its findings. The empirical basis of this discussion

is how to evaluate the "supporter", a constructor of connections, key implementer of QualiSUS-Network', which is a

MoH strategy, in partnership with the World Bank, to promote the quality of Health Care Networks, in 15 Brazilian

regions. The theoretical approach converges the importance of the reflective pedagogical function from translation

of interests to 'transcriation' and 'transvaluation'.

Governance strengthening in the Amazon region: how to overcome this challenge?

Angela Casanova, Marly Cruz, Ligia Giovanella, Glaydes Alves, Egleubia Oliveira, Rafaela Souza, Leonardo

Cunha

PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION The Brazilian QualiSUS-Network Project (QNP) aims to support the implementation of Health Care Networks in 15

health care regions. One of its components is governance reinforcement - the coordination of health care system by

different actors at loco-regional level. This evaluation aims to answer at what extent the QNP contributed to the

strengthening of Brazilians' National Health System (SUS) regional governance in 3 regions within the Amazon area,

Brazil. Evaluability study involved interventions modeling, discussion and agreement among stakeholders. Data

collection comprissed interviews with key actors (n=62) and documentary research (n=43). Data is being analysed

according to governance components regarding both contextual and logical analysis. Initial results show that the

intervention strengthened the governance and integration between the actors in the regions differently, conditioned

by steering mode, consensus on political priorities and influence of strategic actors.

Harmonized evaluations for scholarship and fellowship programs

Shannon Clark Larkin, Ken Stephenson EXPOSÉ/PAPER Increasingly, evaluation is being called upon to broaden its scope beyond the program (e.g., to a sector level or to a

higher level of a department’s PAA). At the same time, evaluations often struggle to find a comparison group for

their program with available and usable data. Harmonizing the evaluations of multiple, si milar programs is one

solution to both of these challenges. Driven by efforts to harmonize program delivery among the Federal granting

agencies, the NSERC-SSHRC Evaluation Division developed an innovative approach to harmonize evaluations of its

scholarship and fellowship funding opportunities, aiming to tell a more fulsome story about the financing of

graduate studies in Canada. The presenters will speak about their experiences and the challenges encountered in

harmonizing evaluation designs, ensuring consistency across large-scale surveys, analyzing across large datasets, and

synthesizing high-level findings into actionable recommendations.

Innovation in large, complex performance measurement data collection using FluidReview

Kelly Mcdonald, Kara Hayne

EXPOSÉ/PAPER

58

In 2014, the Public Health Agency of Canada streamlined performance measurement data collection tools and

processes for their community based children's programs including the Community Action Program for Children

(CAPC), the Canada Prenatal Nutrition Program (CPNP) and the Aboriginal Head Start in Urban and Northern

Communities (AHSUNC). The objective was to reduce the reporting burden placed on over 700 community projects

and to develop a cost-effective process for performance measurement data collection. The Public Health Agency

developed a user friendly web based tool using FluidReview. The process of streamlining program specific tools and

adapting to an online platform was complex, complete with challenges along the way. The end result howev er was

an excellent response rate of 95%. The results are currently being analyzed and will offer national and regional

summaries to inform program evaluation and support senior management decision making.

Institutionalization of evaluation in Albania, from the bottom upwards

Fation Luli EXPOSÉ/PAPER Albania has already created a favorable context for the development of a culture of good governance, managing for

results and accountability. However, the evaluative function and its degree of institutionalizat ion remains

underdeveloped. The evaluation community in Albania faced to the challenge: what strategy to choose for the

institutionalization of evaluation in Albania. Should we go directly to the central government or would be better to

start with the peripheries? In our view, this was first a political process, and to a lesser extent a technical process.

According to a long-term vision, the interest for evaluation shown by Albanian municipalities concluded that,

embarked from the municipal sector, the bottom-up theoretical model could be the starting point for developing

and institutionalizing of evalua

Institutionnalisation de l’évaluation au Bénin : Bilan et perspectives

Emmanuel David-Gnahou

EXPOSÉ/PAPER

La planification participative comme outil d'évaluation en situation post-conflit en RDC

Serge Eric Yakeu Djiam EXPOSÉ/PAPER Après la guerre civile de 2004, les habitants sont revenus à leurs communautés respectives dans la province du

Maniema, en RDC. Le Programme d'Action Communautaire pour le Développement Durable a contribué au

renforcement de la coopération entre les organisations de la société civile, la population et l'administration locale et

soutenir leurs capacités à faire face aux besoins vitaux. L'approche de partenariat a été adoptée comme un élément

de stratégie. A travers quatre composantes tels que Agriculture, la réhabilitation des infrastructures, épargne et de

crédit, et la gouvernance locale, les activités sont intégrées au sein d'une approche participative de la communauté,

et prend en charge les besoins socio-économiques des bénéficiaires depuis 2008. La planification participative a été

développée pour évaluer les résultats du programme en termes de gouvernance, renforcer l'appropriation par les

bénéficiaires, et d'influencer les décisions communautaires.

Les mécanismes de diffusion de l'évaluation auprès des bénéficiaires de programmes fédéraux

Étienne Thériault

PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION La communication efficace des résultats de l'évaluation représente le premier pas vers son utilisation par les

bénéficiaires; il revient donc à l'évaluateur de créer ces liens par l'entremise du rapport d'évaluation, entre autres .

Suite à cette première communication officielle, les parties prenantes externes pourraient en principe se servir des

informations présentées dans les rapports à diverses fins, dans le but d'influencer la formulation de politiques

publiques. Ce projet a donc pour but d'explorer davantage cette problématique en s'intéressant à la communication

des résultats de l'évaluation aux bénéficiaires de programmes fédéraux canadiens en répondant à la question

59

suivante: dans quelle mesure est-ce que les bénéficiaires de programmes externes connaissent et se servent des

résultats provenant des rapports d'évaluation? Cette présentation vise à présenter les résultats de recherche.

Let’s talk about evaluators’ efficiency!

Simon Roy, Janice Remai EXPOSÉ/PAPER In this context of fiscal constraints, evaluators are asked to assess the effectiveness and efficiency of government

programs. But what about our own evaluation activities? Based on their private sector experience, the presenters

will share their views about how evaluations can be run efficiently, including strategies to keep resources and

activities focussed through careful planning and expectations management, effective junior resource management,

and techniques to conduct fieldwork and produce reports efficiently. The presenters will argue that these

techniques can keep the costs of evaluations at a reasonable level without significantly impacting quality.

L'utilisation progressive de l'évaluation par une ONG humanitaire au Burkina Faso

Léna D'ostie-Racine, Christian Dagenais PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION Bien que l'évaluation dans le secteur humanitaire soit de plus en plus valorisée, les connaissances empiriques sur

l'utilisation de l'évaluation (UÉ) dans ce contexte sont limitées. Les quelques écrits scientifiques sur le sujet

décrivent une utilisation inconstante et documentent peu les déterminants de l'UÉ. L'étude qualitative a procédé par

une collecte de données en 2009 (n=15) et une deuxième en 2011 (n=17) afin d'examiner l'évoluti on de l'UÉ et les

conditions qui la détermine, dans le contexte d'une organisation d'aide humanitaire au Burkina Faso. Cette

organisation a développé une stratégie d'évaluation afin d'examiner l'implantation et les effets de son programme

d'exemption de paiement des soins de santé. Les résultats montrent l'évolution sur deux ans de l'UÉ et clarifient

certaines conditions d'UÉ particulières au secteur humanitaire.

New approaches to int'l development evaluation: what 2 Canadians learned by asking the wrong Qs

Lindsay Renaud EXPOSÉ/PAPER Increasingly, middle-income countries like China, India and Brazil are launching their own international development

initiatives. ‘South-South cooperation for development', as it is commonly known, is distinct from Western models of

development assistance. Commissioned by UNDP China, 2 Canadian evaluators set out with 2 research questions:

If/how are middle-income country governments evaluating their South-South cooperation? —and— Do their

approaches differ from those of Western countries (i.e. OECD-Development Assistance Committee members)? As it

turns out, some middle-income countries are embarking on more fundamentally different evaluation approaches

than the researchers' Western-assumptions-based methodology was designed to capture. In addition to sharing their

study findings, in this presentation the researchers will share their 'behind the scenes' stories of efforts and failures

to wipe the cultural tint off their researcher glasses.

Online Guide for Commissioners of Evaluations

Zoe Boutilier EXPOSÉ/PAPER In larger organizations like Canada's International Development Research Centre (IDRC), evaluation is often

decentralized and evaluations are commonly commissioned by program managers whose expertise is not evaluation.

These commissioners play a critical role in defining, scoping, and enabling an evaluation. It is therefore vital to

provide accessible and user-friendly guidance to these individuals as they move through the stages of commissioning

/ managing an evaluation. IDRC is partnering with BetterEval to develop an online interactive tool to guide its staff

and grantees when they find themselves responsible for an evaluation. The interactive tool deconstructs the

evaluation management process into nine major stages and provides decision -making advice and resources for each

60

stage. By bolstering the skills of the evaluation commissioner/manager, this tool should ultimately facilitate the job

of the evaluator and contribute to higher quality evaluations.

Séance de consultation sur l’Agenda global 2016-2020 /Consultation on the Global Agenda 2016-2020

Marie Gervais, Larry Bremner, Benoit Gauthier PANEL/PANEL

The Role of Evaluation Theory in Connecting Evaluation Capacity and Culture

Kaireen Chaytor, Nancy Carter EXPOSÉ/PAPER Boyle & Lemaire (1999) discussed integrating evaluation into organizations using a supply and demand model.

Chaytor and Carter (2009) presented Boyle's model as a similar capacity and culture framework. Cousins'

presentation of 'do and use' could also be presented using the model. Survey data gathered by Chaytor and Carter

shows evaluators' perception of capacity strength exceeds their rating of a culture of evaluation in their own

workplace. Evaluation capacity is necessary, but not sufficient for a strong culture. In situations where evaluation

capacity is strong but culture is lacking, there is a risk of capacity being diminished or lost over ti me if culture is not

intentionally addressed. This presentation will discuss how evaluation theory can be used to integrate capacity with

culture. Literature from evaluation, organizational theory and organizational behaviour will be brought to the paper.

They Said What? Successfully Managing Multiple Languages in Evaluation

Nansy Jean-Baptiste, Lindsay Renaud EXPOSÉ/PAPER Evaluators are increasingly working in bilingual or multilingual environments. Examples of projects where multiple

languages may be used include work including official language minorities, Aboriginal groups, persons with hearing

or visual impairments, new Canadians, etc. Evaluators can also work on multilingual evaluations through projects

with/in other countries. This presentation will outline three main approaches to supporting full participation in

multilingual evaluations: (1) bilingual/multilingual evaluators, (2) interpreters/translators and (3) asking participants

to communicate in their non-native language. The requirements, advantages and disadvantages of each approach

will be presented. A decision-making tool will also be outlined. Throughout the presentation concrete examples of

what to do and not to do will be discussed.

What’s in a name? Defining your organization’s ‘Value for Money’

Elisabetta Micaro EXPOSÉ/PAPER Value for Money (VfM) is becoming the new credo of aid development. Yet the international development

community is still struggling with proposed definitions of VfM. While donors are putting pressure on organizations to

show how they deliver VfM, no comprehensive definition of this concept is yet available. In the past two years, many

attempts have been made by organizations to develop frameworks and methodologies to define and measure VfM.

Universalia has recently supported an international organization in the development of a conceptual framework for

VfM and of its operationalisation. The presentation has the objective of advancing the thinking on VfM. This will be

done in two ways; by presenting the model and the lessons learned from this experience so as to obtain feedback

from the participants on ways to improve the model; and by promoting, through a series of questions, exchanges

among participants on their understanding of this concept and sharing of their exper iences.

8h30-8h45

Lancement du Réseau francophone des évaluateurs émergents

Jérôme Gandin, Marie Gervais LANCEMENT/LAUNCH

61

8h45-9h45

Influences et pouvoirs de l'évaluation à bâtir un monde meilleur

Dow Ornanong Maneerattana, Caroline Heider, Ghislain Arbour, Moderated By Benoit Gauthier

PLÉNIÈRE/PLENARY

9h45-10h

Lancement de l’appel à propositions pour l’édition spéciale en français de la Revue africaine

d’évaluation/African Evaluation Journal

Olivier Sossa, Marie Gervais LANCEMENT/LAUNCH

62

27 MAI MERCREDI WEDNESDAY MAY 27

10h30-11h15

Contribution de l'AfrEA au développement des capacités de l'évaluation en Afrique

Serge Eric Yakeu Djiam EXPOSÉ/PAPER Même si diverses actions ont été menées au cours de ces dernières décennies pour le développement des capacités

des acteurs au processus d'institutionnalisation de l'évaluation, force est de constater qu'il existe encore de

nombreux défis et opportunités pour incorporer l'évaluation dans les cadres institutionnels en Afr ique. L'Association

Africaine d'Evaluation (AfrEA) coordonne environ plus de 30 associations nationales. L'un des principaux mandats de

l'AfrEA est sa contribution au développement des capacités en évaluation des membres pour améliorer la

performance des politiques publiques et privées. Cette présentation analyse les réalisations de l'AfrEA et passe en

revue les défis, les opportunités et les perspectives d'avenir dans le processus d'institutionnalisation de l'évaluation

en Afrique. Les décideurs et les praticiens ont reconnu le rôle central de l'évaluation dans les politiques de

développement.

Evaluation in sub-Saharan Africa: Adventures, Challenges, Reciprocity

Eugene Krupa, Kate Woodman, Clare Stead, Pascalina Chanda-Kapata, Anitha Menon

EXPOSÉ/PAPER Our evaluations in Zambia have focused on capacity building and utilization, but also reciprocity in learning.

Initiatives included mLearning (primary schooling, training health workers), using digital devices to improve

community health services, reducing maternal death in childbirth, and reducing abuse and neglect of orphans. In

low-resource and cross-cultural settings, you want to do things well, and do so the first time. This means acting

respectfully, appropriately, efficiently, effectively, and with knowledge translation, sustainability and scalability in

mind. It is challenging but, with experience and reflection, possible. It is essential to make strong partnership and

pursue high-level engagement with all relevant stakeholders, from policy makers to community residents and to

check for common understanding regularly. We describe the projects, key challenges, and solutions we found. We

ask that you bring your suggestions, tools, and recommendations to the discussion.

L’évaluation des politiques publiques en Afrique : le défi des aspects contextuels et culturels

Sylvain Nkwenkeu, Inoussa Kaboré, Marie Gervais EXPOSÉ/PAPER Les outils et approches méthodologiques qui sous-tendent la fonction d’évaluation sont le plus souvent construits et

affinés dans des contextes occidentaux et disséminés à travers le monde. Leur transférabilité et applicabilité aux

situations africaines, beaucoup plus complexes, demandent un effort de contextualisation qui prenne en compte les

aspects contextuels, socioculturels et économiques. Le panel discute de la prise en compte de ces dimensions en

distinguant l’approche prescriptive de type top-down qui repose sur un schéma linéaire et séquentiel d’une

approche plus systémique qui fait ressortir la circularité du processus de négociation entre acteurs multiples. Au

final, il convient de la nécessité d’alimenter une réflexion développementale afin de procurer du substrat cognitif et

normatif (représentations, valeurs/injonctions normatives, algorithmes) à l’évaluation, contri buant ainsi à consolider

le pont, tant décrié, entre la théorie et l’empirie.

The Evaluation of the 2009 Policy on Evaluation: Key Findings, Conclusions and Recommendations for

Developing a Renewed Policy on Evaluation

Anne Routhier, Suzannae Marshall

PANEL/PANEL SCT Canada

63

11h15-12h

Plénière de clôture: Evaluation for a better world.... What does it mean for you? / Présentation des résultats

sur l’Agenda global 2016-2010

Maria Bustelo, Ernest House, Jonny Morrell, Rob Schwartz, Moderated By Astrid Brousselle

PLÉNIÈRE/PLENARY

12h-12h05

Passage du flambeau de l’Année internationale de l’évaluation

Benoit Gauthier PLÉNIÈRE/PLENARY

7h-8h45

Les défis liés à la collecte et l’analyse de données et les solutions pratiques de l’audit interne et de

l’évaluation de programmes

Younes Mihoubi, Martin Saint-Louis, Pascal Théôret, Modéré Par François Dumaine, Associé, Pra Inc. PETIT DÉJEUNER/BREAKFAST

8h45-10h

Défis et réalisations de l’évaluation pour une meilleure gouverne

Jim McDavid, Robert Lahey, Pierre Cliche, Moderated By Richard Marceau

PLÉNIÈRE/PLENARY