Top > Foreign Policy > ODA > ODA Evaluation Report > Seminars and Workshops > The Fifth Tokyo Workshop on ODA Evaluation > Chair's Summary

Chair's Summary of the 5th Workshop on ODA Evaluation in Tokyo
"Establishing of M&E system based on MfDR (Managing for Development Results)"

January 26-27, 2006
Mita Conference Hall, Tokyo

(Introduction)

  • Ministry of Foreign Affairs of Japan hosted the 5th Workshop on ODA Evaluation in Tokyo on 26-27 January 2006 at the Mita Conference Hall, Tokyo. The workshop had a special focus on management for development results which is attracting attention following the Paris High-Level Forum on Aid Effectiveness and the various summits that took place in 2005. Building up on the past ODA evaluation workshops which aimed at fostering a common understanding of monitoring and evaluation (M&E), the 5th workshop was very much oriented towards listening to the experiences of the Asian partners.
  • The workshop was co-chaired by Professor Ryokichi Hirono and Professor Hiromitsu Muta. Senior officials in charge of evaluation from nineteen Asian countries, including Japan, development agencies (JICA, JBIC, ADB, OECD/DAC, UNDP, UNICEF, WB, USAID) attended the workshop. In addition, the academia, NGOs, and others also attended as observers.

Day 1: January 26

  • The Workshop was opened by Dr. Kiyohiko Toyama, Vice-Minister (Parliamentary) for Foreign Affairs of Japan. He expressed his hope that the workshop would contribute to more effective use of development resources through strengthened evaluation activities in Asian countries.

(1st Session) (Plenary Session) What is the implication of managing for development results for M&E?

  • The representative from the DAC presented an overview of the situation relating to ODA and emphasized the need to pursue joint efforts of partners and donors to increase aid effectiveness and assess its contribution to development outcomes. Presentations on "M&E under Management for Development Results (MfDR)" were made by Bangladesh, Vietnam, Thailand, India, UNDP, ADB and the World Bank, followed by Q & A. Some of the main points raised during the discussion were as follows.
    • The importance of the ownership and responsibility of the partner countries in establishing and achieving development goals was emphasized along with the need for donors to align their assistance to national policy and systems. Tension between national priorities and MDGs was raised.
    • Importance of a result-based framework and the need to move away from "project-based approach" to a broader program and country approach was commonly recognized. Many countries included M&E as an integral component of the PRS and project-design at stages of formulation, implementation and post-implementation.
    • Countries are making efforts to involve various ministries within the government as well as stakeholders and donors in the M&E process for better feedback and accountability. Countries were making efforts to link lessons learned to future project/program priorities and investments. Independent evaluation was being undertaken to ensure objectivity. At the same time, self-evaluation was suggested as the bedrock of evaluation.
    • Insufficient political commitment, lack of capacity and resources, and lack of reliable data collection, were among the major constraints to M&E.
    • Many countries have felt constraints to translating evaluation results to budgeting process although they have a moderate success in reprioritizing development projects and programs.
    • Most countries appreciated a shift in recent years among donors for joint or collaborative evaluation.
    • Emerging donors in Asia are incorporating M&E in the component of their bilateral assistance.
    • Evaluation is not risk-free, especially in case of under-achievement.
    • M&E needs to be discussed separately.

(2nd Session) (Parallel Group Session)
" Strengthening Commitment to M&E for Effective Result - Based Management, Policy Changes, Budgeting Decision, and Accountability-Learning from Good Practices and Challenges of Asian Partner's Experiences"

1st Sub-group: Building support and ownership for M&E systems - Learning from good practices and challenges in Asian Partner's experiences (attended by Director-General level)

Strong political commitment to monitoring and evaluation is a precondition for its effective functioning. Results of evaluation need to be fully utilized for future decision-making and resource re-allocation. Evaluation should also contribute to enhancing accountability and transparency.

  • The 1st Group Session discussed issues of political commitment to M&E, effective feedback of evaluation results, accountability/transparency, and other non-technical issues. Some of the salient points of the discussion were as follows.
    • Political will is important for effective evaluation. In a number of countries, concern with the lack of progress in development in the past led political leadership to embrace M&E for development results. The scope of evaluation is being expanded from ODA to activities funded with national budget. By its nature, country-level evaluation attracts more political interest than project/program-level evaluation. Commitment to M&E can be enhanced through country-level evaluation.
    • Accountability and transparency is closely linked to political attention because governments must ultimately be answerable to their people. Establishing performance indicators and providing the public with monitoring and evaluation reports promotes accountability. The process of determining such indicators involves tension between international goals such as MDGs and other objectives stemming from the local context. One country provides regularly updated progress reports to the public on ODA and national programs/projects on the web-site. Public dialogues which discuss the results of inputs/outcomes/impact can be useful in promoting accountability, transparency and preventing corruption. Accountability to various entities, such as people in partner countries and donor countries, makes this question more complex.
    • A number of countries have in place a built-in M&E mechanism that linked with planning, implementation and budgeting processes. In such a mechanism, the results of M&E are often the basis of budget requests. Since the results-based approach is still in its early stages, there is a continued need for learning-by-doing. Effective implementation is a separate important issue. Fundamentally, this requires strong public institutions and capacity development.
    • Importance of creating demand and providing incentive for M&E was raised. The involvement of the media is one way of attracting public awareness and political commitment for enhancing M&E.
    • While independent evaluation is important for ensuring objectivity, there is a need for ensuring full cooperation of implementing agencies. Since evaluation cannot be independent of political influences, one way to address this is to support think-tanks which can withstand political pressures.

2nd Sub-group: Building an effective result-based M&E system - Learning from good practices and challenges in Asian partner's experiences (attended by Director level)

The purpose of this sub-group discussion was to deepen the common understanding of M&E based on a result-based approach by learning from various experiences of Asian partners, including both successes and challenges, and to contribute to the capacity building.

  • The 2nd Group discussed practical and operational challenges for building and strengthening effective result-based M&E (monitoring and evaluation) systems. Participants from partner countries shared their experiences and challenges, with the aim of identifying lessons from each other. Some of the highlights of the experiences shared are as follows.
  • In order to build up an institutional framework for an effective M&E system, certain preconditions and key elements were identified, including following important issues and needs:
    • Define the realistic goals and objectives, expected results, and feasible indicators in plans to clearly establish what needs to be monitored and evaluated
    • Agree on the role of M&E, e.g. financial monitoring or assessment of development effectiveness
    • Overcome the problem of data quality, in particular, for results at the outcome and impact levels
    • Give greater incentives for data collection by, such as, giving feedback to those who have provided information- a two-way plan of information
    • Secure multiple and transparent data flow channels so that system does not rely on a single line
    • Ensure accountability for M&E
    • Strengthen the institutional arrangements, particularly to ensure M&E information crosses organizational boundaries
    • Raise awareness and interest for M&E information
    • Provide guidelines and practical tools for carrying out M&E
    • Conduct thorough ex-ante evaluation at project level for mapping out expected goals and outcomes, subsequently M&E.
    • Translate findings of M&E into the revisions of programs and projects
    • Set up a legal framework which gives a mandate and clear definition on how the M&E system should work
    • Ensure effective coordination between central agency and relevant line ministries
    • Adopt a context-specific approach (for one size does not fit all).
    • Ensure adequate financial resources for M&E agencies to carry out the task
    • Be sensitive to partners' transaction cost through harmonization such as streamlining requirements for data collections and number of indicators.
    • Be sensitive to donors' demand for data for their accountability to taxpayers to mobilize resources to overseas assistance
    • Strengthen the capability of statistical offices for generation of data for M&E
  • While there is no universal panacea to the above challenges as the context is varied among partners, capacity development of evaluation staff both in the public and private sector is needed as a step to go forward, as there is a need to strengthen M&E human resources and expertise.
  • It was noted, however, that assistance for capacity development itself becomes difficult where there is no defined counterpart in partner countries and that establishment of M&E unit is often a prerequisite. It was pointed that, though many partner countries are making efforts for building M&E system, there still exists high turn over of staff, and there are sometimes no focal point where capacity development needs can be discussed.
  • It was noted that there are already a number of developing country evaluation associations as well as regional grouping of such associations such as the one in Africa. If there is a clear need shown by Asian partners, donors are willing to support similar regional networking in Asia.
  • Joint evaluation of ODA is recognized as another means for capacity development. It was noted that it is also important in the sense that it allows the partners' view to be reflected in the findings. For joint evaluation, demand is greater for program or country level than project level.

Day 2: January 27

(3rd Session) (Plenary Session)
" Partnership for M&E through a result-based approach"

  • As part of the efforts to achieve the MDGs, effective monitoring and evaluation of development activities must be led by the partner countries and supported by the donor countries/organizations. Capacity development is a particularly important challenge for partner countries. Existing donor-partner mechanisms such as CG and Roundtables can be used for promoting results-based approach. Networking can contribute to enhancing evaluation capacity. Some of the salient points of discussion were as follows:
    • Partnership on M&E need to be considered under a different context. Political attention is focused more on country level results. There is a need to link project outcomes with program and country level outcomes. Evaluation for all these levels is necessary. Since the proportion of ODA in development finance is shrinking vis-à-vis national and other resources, "aid effectiveness" should be considered as a sub-set of M&E exercise for public development expenditures. M&E must be an integral part of PDCA cycle. Presentation and packaging of evaluation results is just as important as the contents of the evaluation results.
    • Importance of capacity development specifically for M&E is emphasized. Capacity development concerns not only evaluation experts but also institutions for evaluation. Donors are supporting this through technical cooperation at country and regional levels. Decentralization of donor activities and promotion of joint evaluation with partner countries calls for enhanced capacity in the country offices of donor countries and agencies. Need for an incentive framework was raised as a way of retaining evaluation experts in government. Along with support to governments, credible independent institutions such as think-tanks for evaluation should be supported both by partner governments and donors.
    • One way of securing resources for M&E is to set aside a small percentage of funds in the projects for M&E. This is encouraged also for partner countries, particularly from the project formulation stage.
    • Partners have the primary responsibility to coordinate M&E with donors and civil society organizations. All donors should work together to harmonize M&E activities. There is a need to avoid overlapping M&E activities among donors. Good practices should be shared widely. Donors should use country systems for reducing transaction costs for partners. One suggestion is to promote common methodology in partner countries based on partner-donor cooperation in developing M&E systems.
    • While the indicators and quantitative methods are important in monitoring and evaluation for development results, due importance should also be given to qualitative methods.
    • Establishing an evaluation network in Asia is broadly supported. Such a regional network will be useful in raising the technical capacity of government as well as non-government experts, enhance political commitment to M&E to strengthen institutional capacity and raise public awareness on the importance of evaluation, also contributing to promoting participatory approach. A regional meeting of national evaluation is planned in late March 2006. International Association of Evaluation can be a source of expertise for ODA evaluation.
    • Informal discussion is underway at the World Bank/DAC on introducing results-oriented approach in Consultative Groups and Round Table meetings with regular monitoring of progress for both ODA and national budget activities. Such an innovation will contribute to promoting effective use of resources and better data collection. Managing for development results will not be a success unless there is actual progress in development.

(Conclusion and way forward)

  • There was a general recognition that the 5th workshop contributed to deepening understanding among the participants on effective M&E and exploring further how to overcome challenges and improve M&E, and lead, ultimately, to development results for the benefit of the people. The participants commended the Ministry of Foreign Affairs of Japan for organizing this M&E workshop and hoped for another meeting next year. The representative of the Ministry of Foreign Affairs announced its intention to host another meeting of ODA evaluation workshop during FY 2006.
Back to Index