Report of the Seminar/Workshop on ODA Evaluation in Thailand Bangkok, 17-21 January 2005 ---------- Foreword The Ministry of Foreign Affairs of Japan (MOFA) has been hosting the Tokyo Workshop on ODA Evaluation since 2001 with the main purpose of deepening the common understanding of ODA evaluation in the Asian countries and contributing to the expansion of evaluation systems in these countries by strengthening partnership with Japan. In the past workshops, discussion topics included such issues as the involvement of partner countries in ODA evaluation, the importance of capacity building, legal and institutional aspects for effective evaluation, evaluation at program and policy levels, and joint donor- partner evaluations. In the third workshop held in November 2003, Thailand expressed its readiness to host an evaluation seminar, which set the ground for the Seminar/Workshop on ODA Evaluation in Thailand held in Bangkok from 17 to 21 January 2005 co-hosted by the governments of Japan and Thailand. For this Seminar/Workshop, 50 junior officials from 17 Asian countries participated in the seminar part, and lecturers from Japan, Thailand, DAC, World Bank, and UNDP were invited to conduct a participative seminar with participants performing simulations in monitoring and evaluation using concrete cases. At the workshop, 30 participants including senior government officials from 17 Asian countries and representatives of multilateral organizations, such as DAC, World Bank, UNDP, and ADB, discussed about challenges and countermeasures in monitoring/evaluation, and about partnership and capacity development in evaluation. The discussion in the workshop was compiled in the Chairperson’s Summary by Chairperson of the Workshop Prof. Ryokichi Hirono, Professor Emeritus of Seikei University as provided overleaf. ---------- The Seminar/Workshop on ODA Evaluation in Thailand REPORT TABLE OF CONTENTS Chairperson's Summary I. SEMINAR 1. Opening Session Opening Remarks 2. Lectures in Plenary 1) "Basic Concepts and Techniques of Monitoring and Evaluation" Prof. Ryokichi Hirono, Professor Emeritus, Seikei University 2) "Evaluation and Aid Effectiveness - Glossary of Key Terms in Evaluation and Results Based Management" Mr. Hans E. Lundgren, Head, Evaluation Section, Review and Evaluation Division Development Co-operation Directorate, DAC 3. Case Studies in Evaluation 4. Closing Session 5. Comments from the Participants II. WORKSHOP 1. Opening Session Opening Remarks 2. First Session Agenda 1: "Challenges and Countermeasures on Monitoring/Evaluation" Mrs. Suthanone Fungtammasan, Senior Programme Officer, Development Programme Evaluation Division, TICA 3. Second Session Agenda 2: "Partnership in Evaluation and Evaluation Capacity Development" Ms. Satoko Miwa, Director, Office of Evaluation, JICA 4. Closing Session Approval of Chairperson's Summary Closing Remarks Appendix I: Program for Seminar/Workshop Appendix II: List of Seminar Participants Appendix III: List of Workshop Participants Appendix IV: Profiles of Workshop Representatives ---------- Report of the Seminar/Workshop on ODA Evaluation in Thailand 17-21 January 2005, Bangkok, Thailand ---------- Chairperson's Summary 21 January 2005 The Seminar/Workshop on ODA Evaluation was held in Bangkok from 17 to 21 January 2005 hosted jointly by the governments of Thailand and Japan. This joint organization was realized thanks to the proposal made by Thailand at the Third Tokyo Workshop on ODA Evaluation in November 2003. Fifty participants from Asian countries attended the seminar. After general lectures on ODA evaluation, participants carried out case study simulations of evaluation based on the actual circumstances of their own countries on the following five themes: Education, Infrastructure, Rural Development & Poverty Reduction, Environment and Outcome Evaluation. Thirty participants attended the workshop, including senior government officials from 17 Asian countries, evaluation specialists of multilateral development agencies, foreign ministry officials, staff of aid implementing agencies and academics from Japanese evaluation institutions. They discussed challenges and countermeasures on monitoring and evaluation (M&E) found through case studies of the seminar, and the importance of the partnership in evaluation. A number of people attended the workshop as observers. Some highlights of the discussions at the workshop include among others: Participants of the workshop recognized afresh the importance of monitoring and evaluation for enhancing the effectiveness of ODA and development efforts and for improving accountability. The workshop emphasized the critical importance of political commitment and main- streaming M&E in all development activities regardless of financial sources, including exposures of policy makers and project/program planners to evaluation methodologies and system. In this connection, it was stressed to promote all types of results-based evaluation of the project, program and country levels and internal and external evaluation approaches involving all stakeholders in the evaluation process. To make effective use of the results of ODA and development monitoring/evaluation, it was agreed as essential to ensure an effective feedback mechanism to reflect the results of evaluation in policy, program and project planning and implementation. The workshop shared the importance of establishing a management mechanism for evaluation involving all phases of the project cycle and strengthening neutral and professional evaluation competence, including the enhancement of credibility. The workshop stressed the promotion of joint evaluation between donors and partner countries for strengthening ownership and capacity development in partner countries, and emphasized harmonization and standardization of evaluation and reporting procedures. The workshop urged the continuation of regional cooperation on evaluation so that Asian countries could share a common understanding of ODA and development evaluation. It was proposed to strengthen and upgrade national and regional evaluation training seminars for junior officers and workshops for senior officials concerned with all phases of development. The workshop also recognized the need for establishing and/or strengthening national evaluation societies to improve evaluation methodologies and systems and initiate codes of conduct for evaluators, and promoting a regional network of national evaluation societies in Asia. The results of this workshop will be reported for further discussion at the Second High-Level Forum on Aid Effectiveness to be held at the end of February in Paris. All participants expressed their gratitude to the governments of Thailand and Japan for their initiative in organizing this Seminar/Workshop. ----------- I. SEMINAR 1. Opening Session Opening Remarks Mr. Piamsak Milintachinda, Director-General, Thailand International Development Cooperation Agency (TICA) Taking into account that Asian countries and Japan have shared many experiences on development cooperation throughout the years, and following the initiative of the previous Tokyo workshops on ODA evaluation to move ahead from discussion to practice, this seminar is a timely move to enhance and strengthen ODA monitoring and evaluation through capacity building of humans as well as institutions. It is essential to create capable personnel in this area as well as to establish a management system in order to improve the quality of evaluation. Hence, this seminar will provide an opportunity to identify critical issues through enthusiastic group discussions. Certainly this seminar will help developing countries understand and play an active role in ODA monitoring and evaluation and, therefore, all participants are encouraged to actively participate, share ideas and exchange experiences. Hopefully the outcomes arising from the seminar will be useful in enhancing monitoring and evaluation in developing countries and provide guidance for more meaningful development cooperation of ODA projects in the future. Opening Remarks Mr. Tomoyuki Yoshida, Counsellor, Japanese Embassy in Thailand The main purpose for Japan hosting the workshop on ODA evaluation--an important management tool--is to deepen the common understanding of ODA evaluation among Asian countries and to contribute to the expansion of the evaluation system of those countries by strengthening the partnership between donors and partner countries. While the first workshop dealt with the involvement of partner countries in ODA evaluation, and the importance of capacity building and support for effective evaluation, the second workshop placed focus on scaling-up the object of evaluation from the project level to the program and policy levels. It was at the third workshop, in which the common understanding that ODA evaluation is a key tool for more effective development assistance and joint donor-partner evaluations were discussed, that Thailand, in response to spontaneous requests, expressed its readiness to host an evaluation seminar. Ever since, the ODA evaluation seminar has been prepared to provide partner countries with a valuable opportunity to both recognize the importance and broaden the knowledge of monitoring and evaluation. By organizing sub-group discussions, the particularities and diversities of participant countries will have to be taken into consideration. Such participatory learning will surely be highly useful for capacity building in monitoring and evaluation. 2. Lectures in Plenary 1) "Basic Concepts and Techniques of Monitoring and Evaluation" Prof. Ryokichi Hirono, Professor Emeritus, Seikei University It is evident that tremendous interest has been generated about evaluation and thus propelled evaluation onto the center stage in general. One major factor behind this push in Asia has been increasing public demand for reform to instill greater transparency and accountability of government activities, especially following the inadequate policies that brought about the 1997-'98 Asian financial crisis. There are two objectives behind the growing interest in evaluation: first, to increase cost effectiveness and development effectiveness of the government activities at all levels of management; and second, to increase transparency and accountability of the government to tax- payers and the general public and enhance their confidence in all branches of government. There are four varying levels of focus: project level, program level, sector level and national level. In Asia, there is a growing emphasis on focusing on the latter levels. This is because there is a trend toward increasingly placing emphasis on outcome, rather than input and output, and overall external contribution to development as well as overall policy coherence at the national level. In addition to the five fundamental OECD/DAC evaluation criteria, necessity/essentiality, goal achievements, and strategicness and innovativeness should be considered. There are also many kinds of approaches to evaluation, such as internal and external evaluations; donor, partner and joint evaluations; results-based and process evaluations, etc. Assessments of project conditions greatly involve the collection of primary data, especially through interview and document analysis (IDA). The basic issues related to evaluation are most obviously the high cost and redundancy of evaluation exercises, ensuring independence of evaluation and ensuring feedback to the effective formulation and implementation of new development activities. On the way ahead, one suggestion is promoting donor-donor and donor-partner joint evaluation of strategic and innovative policy, program and/or project interventions (SIPPPIs) by bilateral and multilateral donors, especially in Asia. Another option is to increase peer review by national evaluation societies of the internal evaluation by bilateral donors. Discussion The floor was opened for discussion. A participant asked why the public debt crisis of the 1980s was not included as one of the contributing factors of the increasing interest in evaluation, whereas the Asian financial crisis was specifically highlighted. Prof. Hirono responded by saying that given the need to respond to public demand for transparency, it was vital that effectiveness of public evaluation be taken into consideration. He further added that with all governments in Asian countries, developed and developing alike, facing budget deficits that are snowballing far beyond the Asian financial crisis, development expenditure must be minimized. A participant asked for clarification of Prof. Hirono's mention of four-digit industrial standards in his presentation, to which Prof. Hirono detailed the data classification system used in the World Trade Organization (WTO; formerly UNIDO and GATT) and also subsequently adopted by the UN, in which data is categorized by numbers ranging from one to six digits depending on its level of detail. He stated that he believed that going beyond four digits would be too in-depth for the collection of adequate data. In response to the question on the constituency of the national evaluation society in Japan, Prof. Hirono explained that it was a professional association of 520 evaluators involved in both central and local governments, academia, research institutes as well as the private sector. In this respect, he reiterated the need to have ethical guidelines for these evaluators to follow. Concerning the issue of reducing evaluations and careful refocusing to improve the quality of evaluation, Dr. Pitman of the World Bank gave details on the Bank's internal procedures for standardizing quality control of its own evaluations. He acknowledged that there was a so- called disconnect or divergence in the ratings given by the regional offices vis-a-vis the Bank's evaluation unit, and that this multi-level review involved a lot of time and expense. However, recognizing the range in the evaluation quality, Dr. Pitman continued, the Bank takes a sample from all its projects to re-evaluate, paying particular attention to those that embody important lessons to help improve performance. With regard to joint evaluations, Dr. Pitman admitted that given the "moral hazard" issue, many organizations may feel averse to being policed by external agencies and that therefore cooperation was not always forthcoming. However, he added, collegial activity has been ongoing in studies on globally important topics, for example in the field of AIDS, incorporating the different reports from various organizations. He ended by stating that organizing a joint evaluation at times took longer than the actual activity. On this issue, Prof. Hirono commented that Japan also conducted global issue evaluations, but that individual countries lacked the resources to do such extensive evaluations as done by the World Bank. Given that a majority of development assistance comes from non-governmental organizations (NGOs), a participant from India asked about the evaluation criteria of donor agencies for NGOs. He added that since the primary purpose of the NGO channel was to make aid available directly to the beneficiaries and cut through red tape, he felt it important to incorporate concerns of the NGOs in government policy. Prof. Hirono replied by saying that some NGOs were interested in promoting their own platform, but that if they were deemed serious about contributing to the national interest, then their considerations should be included. A participant reported that in his country a new performance-based development budget mechanism was being implemented. For this, he asked how performance measurement indicators should best be selected, to which Prof. Hirono responded that the two major items to consider were sustainability and data. Having performance indicators that are not sustainable would lead to constantly changing and inconsistent indicators across budgets, he cautioned. He added that clear statistical data needed to be generated by one's own efforts. By way of clarification, Mr. Alam of UNDP explained that early in the 1970s and 1980s, evaluation was done as a donor tool, since external assistance was more significant than domestic resources and governments were under pressure to show results. Furthermore, because evaluation was often seen as a sacrosanct mechanism policing government expenditure, he recounted, it could not be mainstreamed. He wondered if there was a way to simplify the spirit of evaluation to mainstream it in public service while maintaining norms and standards. Prof. Hirono agreed that the high cost of evaluation was a factor for all countries in setting their budgets for evaluation. He opined that the mindset of project managers and implementers, who view evaluation as policing, must be overturned. Instead, he hoped that evaluators would be thought of as "helping hands." A participant from the Philippines thanked Prof. Hirono for highlighting that developing countries are increasingly implementing their own resources into development. He stated that the aspect of the "weak state" should be dealt with in more depth to move away from focusing on short-term optics toward substance. He asked about how IT might help in improving the data and advocacy aspect of evaluation. Prof. Hirono encouraged developing countries to expose themselves to different data sources and make better use of integrating existing data instead of focusing on completely new data so as to speed up the data collection activity and avoid redundancy. The participant further emphasized that there was a need for harmonization of the different evaluation mechanisms. While expressing his appreciation for Japan's involvement in evaluation through JICA and JBIC, he hoped that similar commitment would be more forthcoming from other bilateral partners. Prof. Hirono was sympathetic to the criticism and agreed that harmonization was a priority for the OECD/DAC. However, he pointed out, there is a need to plead the need for harmonization before actual efforts can be made. In response to the last comment by a participant to expand the culture of evaluation to ODA to address problems in governance, Prof. Hirono agreed that this was the intended goal. 2) "Evaluation and Aid Effectiveness . Glossary of Key Terms in Evaluation and Results Based Management" Mr. Hans E. Lundgren, Head, Evaluation Section, Review and Evaluation Division, Development Co-operation Directorate, DAC The aim of the Glossary of Key Terms in Evaluation and Results Based Management published by the OECD/DAC is to break down the barriers of language to reduce terminological confusion and improve understanding so as to facilitate dialogue. The glossary was developed and published by the DAC Network on Development Evaluation comprising 30 development agencies. The glossary was originally produced in English, Spanish and French, with five languages added since (Japanese, Portuguese, Chinese, Italian and Russian) and hopefully more languages, particularly Asian language, to come. The glossary contains terms in alphabetical order that are in current use in evaluation and results-based management (RBM) so as to be of use for operational vocabulary. They are grouped by broad categories with definitions of subordinate concepts. For example, there are various concepts in the glossary that could fall under the analytical theme heading of "Assessment Concepts," such as Results Monitoring, Evaluation, Audit and Review, that are both interlinked and differ in their own way. The glossary also contains the basic concepts of the Results Chain (inputs, activities, outputs, outcomes and impact) which represents the causal sequence leading to the attainment of the objectives. It also describes the results of the intervention at the different levels (strategy and implementation) using results frameworks and logical frameworks which include assumptions and risk analysis. Most widely used by many of the development agencies are the five main evaluation criteria-- Relevance, Efficiency, Effectiveness, Impact and Sustainability. By conducting evaluations using these criteria, it is possible to form a conceptual model of causality between aid and development. In order to do this, however, targeted and sound data collection and specific types of evaluation have to be done that lead to useful recommendations and lessons learned. With regard to RBM, there needs to be an integrated approach between strategic planning, performance measurement and acting on those results. The key criteria for collecting quality performance data are timeliness, validity and reliability. The power of measuring results and the need for measuring results lies in winning the trust and support of the public. What makes for quality evaluations is that they incorporate usefulness (to the decision maker and those in need of the evaluation), technical adequacy, impartiality, stakeholder involvement, value for money, and feedback and active dissemination of the lessons learned to implement anew. Evaluation is used among others for making resource decisions, rethinking the causes of a problem or identifying issues surrounding an emerging problem, supporting public sector reform or innovative programs, and building consensus among stakeholders. As to future developments, the glossary is currently used in training and everyday operations work and not exclusively by evaluators. It is important to disseminate and encourage further language coverage. Moreover, the publication of the glossary does not mean the end of the DAC’s vocabulary work. On the contrary, the DAC supports continual methodological development and innovation, which will lead to new vocabulary and policy to be integrated in the years to come. Furthermore, the DAC has been involved in work to harmonize donor practices and streamline development work in line with the partner countries' practices and needs. Discussion Obliging to a request from a participant to elaborate on meta-evaluations and how they are used, Mr. Lundgren explained that the purpose of meta-evaluations was twofold: for aggregating findings from a series of evaluations or for assessing a particular evaluation. On the comment whether this was not in effect an evaluation of the evaluator, Mr. Lundgren countered that such an evaluation would involve a personal assessment and thus was not included among the standard methods of evaluation. Mr. Alam of UNDP additionally explained that rather than viewing meta-evaluations as a type of evaluation, they should be considered as a way of doing an evaluation. At UNDP for example, he stated, a meta-evaluation involved looking at all the evaluations of a regional program, taking the existing work to assess how far the necessary questions are addressed and enhancing the data from previous work for any current evaluation. Mr. Lundgren agreed that of the two senses or current uses of meta-evaluations that he mentioned, the former was used more prevalently. One problem area was that in doing an evaluation that builds on multiple other evaluations, the lessons learned become generic and take away from the specificity, Mr. Lundgren added. He explained that this led to resistance in implementing meta-evaluation, choosing in its place to have direct dialogue, i.e. participatory evaluation. Selection of the most appropriate evaluation type has to be made with the aim to win the willingness among the public to help finance the development of projects, Mr. Lundgren continued. Prof. Hirono asked whether the DAC had done surveys on implemented meta-evaluations and their necessity, to which Dr. Pitman of the World Bank replied by saying that the World Bank had produced a book on evaluating aid effectiveness which dealt with meta-evaluations and their coverage. To a question by Mr. Rider Smith of UNDP on the application of RBM versus performance- based evaluations, Mr. Lundgren responded by describing the evolution of RBM from its beginnings in 1993 in the U.S. during the Clinton Administration, with its usage eventually being proliferated to other agencies. After ups and downs in its popularity and implementation, Mr. Lundgren stated that in the current development path a more balanced approach of RBM vis-a-vis evaluation was needed instead of reliance on solely one method. Asking about the possibility of ex-post evaluating projects based on real circumstances, a participant from India recounted the story of a village that had successfully escaped the recent tsunami disaster because it had had access to information about the impending danger in advance. Mr. Lundgren stated that although there are different factors for success, resources should be invested to replicate programs that have proven successful. Prof. Hirono reminded the audience that one of the objectives of the seminar was to explore the possibility of South-South cooperation in evaluation, and Mr. Lundgren agreed that this was an important study area. Mr. Lundgren added that in terms of joint evaluation the DAC was interested in getting input for a study underway on joint evaluation. A participant asked about standard evaluation criteria in joint evaluations, to which Mr. Lundgren replied that there was basic agreement to the five criteria by most development agencies and that calculating the level of each criteria was a judgment call based on key evaluation questions. For example, he explained, doing an impact evaluation was more difficult than an evaluation of economic factors. As an additional comment to the question, Mr. Minato from FASID shared that JICA had already applied the DAC five criteria to their own evaluation methodology, the Logical Framework (Logframe). Mr. Alam of UNDP added that the stakeholders must agree on the standards of performance at the start of the project. Orientation for Case Studies in Evaluation All the facilitators of the sub-group discussions were introduced and gave a brief overview of the content to be covered in the case study evaluations. Prof. Hirono hoped that all the subgroups would be interesting and instructive to the participants, and he urged all participants to join actively in the discussions. 3. Case Studies in Evaluation To follow-up on the general substance of the plenary lectures, over the course of the following two and a half days, the participants were divided into thematic sub-groups to deepen their knowledge on the evaluation process and apply this knowledge to actual case studies. The five themes under study were: Education, Infrastructure, Rural Development and Poverty Reduction, Environment and Country (Outcome) Evaluation. Given the varying degrees of experience in evaluation among the participants, the lecturers proceeded with more detailed explanations of evaluation concepts and methods with regard to the five themes. This was followed by undertaking practice evaluations on example case studies to get a feel for the process and to uncover difficulties that may be encountered. By comparing their results to actual evaluations done by evaluation agencies, the participants were able to further their knowledge about conducting evaluations. 4. Closing Session Representatives of each of the five groups presented a 15-minute summary of the issues and results covered in their discussions, lessons learned, key challenges identified and proposed countermeasures for them. Also included in the presentations were recommendations for improving the organization of future seminars. Sub-Group 1: Education The group tried to deepen their understanding on the concept of evaluation and the criteria before moving on to the timely case study presented by the facilitator, Dr. Suparvadee Mitrsomwang of Chulangkorn University, on the Tsunami Devastated Areas' Educational Facilities Restoration Project, which was an actual project conducted by the Thai government to restore school facilities in six provinces affected by the recent disaster. On the second day, the group worked on a case study involving the Myanmar Education Ministry with each participant assigned to design a project specific to their country of origin. They found that many different M&E procedures existed among the participating countries, depending on whether a country was a donor or partner country. Standardization of the M&E procedures was recognized as necessary. Participants recommended that the OECD/DAC glossary regularly be revised/updated. Through their discussions of the case, they found it necessary to update officials involved in evaluation with the latest techniques and methodologies by using the local and regional networks already in place. They also found the need for strengthening institutional capability for monitoring and evaluation as well as that for personal capacity. Sub-Group 2: Infrastructure The group received presentations from Mr. Satoshi Shigiya, Director, Ex-post Evaluation Division of the Japan Bank for International Cooperation (JBIC) and Ms. Takako Haraguchi, Researcher of Global Link Management (GLM) on monitoring and evaluation of infrastructure projects, including the project cycle, mid-term review, rating system and conduct of ex- post monitoring. In the exercise on the ex-ante evaluation of the case study of an airport development project, they identified the challenge to include difficulties in demand forecast to determine the size of the infrastructure and target values for outcome indicators. On the monitoring and mid-term review, they discussed the need for more comprehensive mid-term review conducted by donor and beneficiary countries and validity of revision of targets for output and outcome indicators following the revision of demand forecast. In the second case study on a water supply project, the group attempted to do an ex-post evaluation and discovered the unavailability of data, especially baseline data, and difficulty in analyzing various social impacts on different types of beneficiaries. They recommended that in conducting M&E, with regard to ex-ante evaluation, the demand forecast be reviewed at the very start of the project cycle for a more realistic and accurate target setting and to incorporate indicators to verify social impacts. As to monitoring and midterm review, they recommended that "efficiency" and "impacts" be included, as well as other criteria in the mid-term review, which they believed should be conducted jointly by the donor and beneficiary countries. As for ex-post evaluation, the group concluded for efficiency evaluation to include the viewpoint of the beneficiary country as well and that the design of social analysis in the impact evaluation be improved to have a more representative sample of the population. In terms of ex-post monitoring and feedback, the group discussed the need to disseminate lessons learned for future projects, and to have simple and easy-to-collect indicators be monitored so that ex-post monitoring could be more efficient and sustainable. The group also discussed the need for capacity building to reinforce their familiarity with the DAC five criteria. Sub-Group 3: Rural Development & Poverty Reduction The group, under the guidance of Mr. Naonobu Minato, Acting Director of the International Development Research Institute (IDRI) of the Foundation for Advanced Studies on International Relations (FASID), used a case formulated to study poverty reduction in an agricultural project to learn about the concepts of the Project Design Matrix (PDM/Log-frame) and identifying the key evaluation questions to measure project performance as well as writing reports and getting feedback. The group discussed each phase of the project cycle and identified concrete evaluation questions to measure project performance by evaluating each DAC criterion considering the policy point of views on relevance and sustainability, how to rate and write evaluation reports and how to feedback evaluation results to stakeholders and policies. The group identified as a challenge the lack of competent and independent evaluators, for which they suggested that local educational and training programs and facilities for evaluation be established and/or strengthened in partner countries, in addition to the development of a code of conduct and ethics. Furthermore, they agreed that there was a lack of dissemination and feedback of evaluation results. In order to rectify this, the group recommended that a repository system be established and that mechanisms such as the media and workshops be used to spread results more widely. A rapid evaluation method and introducing sampling evaluation and self-assessment to combat the high cost of M&E in terms of time and human resources were also suggested. To harmonize the differences in perspective among donor and host countries, they also recommended that simple standardized procedures of evaluation be established. Sub-Group 4: Environment The group, facilitated by Dr. George T. Keith Pitman, Senior Evaluation Officer of the World Bank, used case studies of actual projects to cover monitoring and evaluation in the environment field clarifying the evaluation process, methods, reporting and feedback concepts on results-based M&E and framework for policy evaluation In the Inner Mongolian Railroad case study, the group discussed the use of the Project Logic Model to identify important M&E variables for evaluation. Through the second case study on the Baltic Sea Environmental Program, participants realized that the evaluation of such a regional project involving multiple countries was complex, such that a wider perspective of changing economic structures may affect project effectiveness. In the Water Management Policy case, it was found that policy evaluation distinguishes between program and policy implementation and that learning in designing cross-country policy evaluation performance was important. They further highlighted the need for addressing the concerns of all stakeholders of a project. Also, the group proposed that the DAC prescribe a model project design with baseline data for evaluation and development of a system framework allowing continuous evaluation of projects. Sub-Group 5: Country (Outcome) Evaluation The group, jointly led by Mr. Nurul Alam, Deputy Director of the Evaluation Office and Mr. David Rider Smith, Evaluation Specialist, both from United Nations Development Programme (UNDP), studied the concept of results-based management and the meaning of performance-based monitoring and evaluation in supporting good management and covered a case study of "outcome evaluation." It was decided, based on the range of experiences and knowledge of M&E processes, not to focus on country evaluations--itself a means of assessing agency portfolio effectiveness--but rather on outcome evaluation. Through the presentations on the concept and relevance of outcome, the group was able to learn about the importance of project outcome defined as developmental changes between the completion of outputs and the achievement of impacts. The outcome is accomplished through partnership including inputs from other projects. Using the case study of the imaginary poor developing country now undergoing economic reform called Xanadu, the group underwent an outcome evaluation simulation. The group was tasked to imagine being a staff member in the Ministry of Environment and designing an evaluation to assess the extent to which the outcome has been achieved and the contributing factors to this. Through this, they identified lessons in terms of trying to achieve the national objectives through partnership. In this exercise, the group clarified the definition of sub-sectors of the outcome and expanding the sub-level indicators. As challenges, the group concluded that outcomes were often not clearly defined and that it was difficult to define the sub-sectors as well as their indicators, which in the case of the model study were contributors to environmentally sustainable development. Another challenge was in determining the key questions for evaluation, as the group found it difficult to define links between project impact and outcome. They also suggested that the role of soft assistance in achieving the outcome must be considered. In determining the method, the group stated that it was difficult to find the right method to define the links between project output and outcome. In terms of implementation strategy, the group advised that the cost could become high if there was no pre-assessment of the project. Challenges and Countermeasures Key challenges of evaluation and countermeasures identified by these groups were, that while emphasizing the critical importance of political commitment, mainstreaming and monitoring in all development activities, they recognized the need for better understanding of all the issues related to the project cycle and indicated that there was a lack of feedback and interaction to share evaluation information and data collection and proposed the establishment of more systemic networks for communicating evaluation results among agencies concerned as well as to disseminate them to the public. They also noted the lack of independent evaluators, the high cost of conducting evaluations and the need for continuous evaluation of projects. They further addressed the necessity to involve all stakeholders and their concerns, and enhancing accountability and transparency. On evaluations, the groups pointed out the need to promote joint evaluations with beneficiary countries to enhance ownership and capacity building. Another major challenge was the need to have more harmonization and standardization among the evaluation methodologies and procedures used and the training of those involved in evaluations at such seminars as this would be of great benefit in this respect. The groups also added that while two and a half days was not sufficient to become well grounded as evaluators, they were greatly impressed by the high level of participation in the active discussion by all group members and that they learned greatly from the differing views of others. 5. Comments from the Participants Participants were asked to fill out a questionnaire on their opinions of the seminar. The following are some of the comments received: ■ Overall evaluation of the seminar: - A good and beneficial seminar in understanding the concept and process involved in monitoring and evaluation of ODA - Very productive and useful, I have learned more and gained a lot of knowledge of evaluation process and practices - Good for the beginner but more focus on ODA project evaluation is required - This event is substantially important in terms of imparting knowledge of how to evaluate ODA projects and how to feed back evaluation results to policymakers - The discussion and lectures were very useful and fruitful - Excellent in depth of participation and coverage ■ Evaluation of sub-group sessions - Lecturers were very knowledgeable and always ready to share their experiences ■ Suggestions for future program - Relatively short timeframe; more time was needed to get full knowledge - Extend duration of the seminar - Include field visits to see ongoing projects, especially ODA projects - Hold national level workshop on ODA evaluation and have more systematic, simpler evaluation techniques to introduce to participants ---------- II. WORKSHOP 1. Opening Session Opening Remarks Mr. Piamsak Milintachinda, Director-General, TICA Thailand is honored to host this important workshop deriving from the previous Tokyo workshops on ODA evaluation. The Government of Japan has long extended technical cooperation to Thailand in various areas in the form of development projects, experts, and training, reflecting mutual benefits for both sides particularly in the implementation process. Joint effort is required from the initial stage of cooperation in brainstorming and designing project activities, implementation to monitoring and evaluation. Furthermore, Thailand and Japan have experience in extending ODA to other countries and have come to realize that one of the weaknesses of ODA projects is monitoring and evaluation. This workshop is a part of many efforts to enhance the quality of ODA project management in developing countries. From the deliberations in the seminar preceding this workshop, many challenges and countermeasures of M&E have been identified that will help support the workshop discussion in order to pursue sustainable development on ODA projects. As many challenges lie ahead in the development of ODA projects, it is now necessary to review the critical issues in order to promote better ODA evaluation systems and provide grounds for future directions in development cooperation between donors and partner countries. Hopefully, successful and productive discussion at this workshop will contribute to the shared objectives of building evaluation systems as an effective instrument for development cooperation. 2. First Session Agenda 1: "Challenges and Countermeasures on Monitoring/Evaluation" Representing the participants of the seminar, Mrs. Suthanone Fungtammasan, Senior Programme Officer, Development Programme Evaluation Division, TICA presented a summary of the presentations given by the sub-group representatives during the closing session of the seminar (see page 10 Closing Session). Discussion Dr. Widjojo of Indonesia acknowledged that there were a lot of problems faced in the implementation of ODA projects and proposed that a standard guideline for M&E be established for the benefit of recipient countries. Mr. Tungpalan of the Philippines lamented the fact that the challenges and issues discussed at previous meetings had come up yet again and hoped that the discussion would clarify why the workshop had not moved forward as fast as they hoped. Mr. Cao of Viet Nam shared that Viet Nam, with the help of the Australian government, was working on implementing effective standardized M&E systems, before being able to participate in harmonization of systems. He admitted that one constraint was the need to raise the level of evaluations from the project level. Furthermore, in order to promote the participation of private sector consultants in evaluations he suggested implementing an evaluation certification system. Dr. Chhieng of the Development Council of Cambodia highlighted the three levels of M&E (project, program, and policy/macro or country level) covered in the seminar discussions and stated that insofar as different countries having different systems, the situation was the same across the board. He referred to his institution as focusing on evaluation at the macro- or country level to the sector level, leaving the lower level evaluation to the technical line ministries of the government. He recognized that for ODA projects it was vital to harmonize and standardize the M&E process, such as by setting up guidelines to align the knowledge owned by all those involved. Furthermore, he stated that it was important that project goals be set at the country level and linked to the country policy framework. Mr. Thaung of Myanmar added that in Myanmar institutional and personnel capacity building, with the cooperation of donor agencies, were recognized as most influential for effective project evaluation. Ms. Chia of Malaysia acknowledged that some movement toward capacity building could be seen, especially at the junior level, but that training has to be extended to the higher levels. Still, because no systematic framework exists, she said, recipient countries had to comply with the different implementation systems used by donor partners. It fell upon partners to ensure that the evaluation process is not viewed as an exogenous process but as an integral part of project design, she underlined. She also brought up the difficulty of implementing post-project M&E because of lack of baseline data and a proper systems framework. Concerning joint evaluation, Ms. Chia commented that since 2002 Malaysia had started conducting evaluations with CLMV countries (Cambodia, Laos, Myanmar and Viet Nam) and similarly with African countries. This is important, she emphasized, as countries are looking more into South-South country partnership as one modality that holds tremendous potential in the future. Mr. Mehrotra of India strongly stressed the importance of evaluation for all development projects, not just ODA, regardless of funding source. The Chair requested facilitators of the seminar groups to respond to some of the comments, to which Mr. Lundgren of OECD/DAC commented that there was a need to promote M&E to enhance ownership over the process among partner countries, for which further harmonization and training are required. Mr. Shigiya of JBIC mentioned that in strengthening M&E mechanisms, the objective should not be to have a fixed system but for it to be in line with the priorities and needs of the beneficiary countries. He hoped that with the growth of joint evaluations being conducted with different countries, a system could be established that increases the efficiency of joint work. Dr. Pitman of the World Bank stated that the Bank was critically aware that much of its evaluation work so far had unfortunately been very internally focused and that there was an acute realization that the Bank needs to share with the countries in discussion and to mainstream the findings. It was necessary, he stated, for evaluation to contribute to the sector and macro concerns of a country, such as embedding projects in the national framework as in the country assistance strategies. Mr. Walter of the ADB reiterated the grievance that the same problems and challenges kept coming up and questioned whether it was all simple rhetoric. He stated that it was the responsibility of funding institutions to convince governments on the importance of evaluation and overcome the problem of lack of good governance in certain countries. All this in fact, he bemoaned, inevitably took time. Moreover, Mr. Walter speculated that lack of local capacity was an additional constraint. Given the varying capacity levels among developing countries, he underlined that more hands-on joint evaluations as well as training, not just in the public sector, must be promoted. The Chair surmised that in general a problem of political commitment existed. At the top level of government in many countries, commitment was not at a desirable level, he criticized. Therefore, people involved in evaluation must find ways to mainstream M&E in all development activities, he added. Concerning joint evaluation, the Chair underscored the importance of enhancing the sense of ownership by the partner country as well as promoting capacity development and suggested that one way of doing this was to share advantageous experiences as well as the bottlenecks of joint evaluation. Mr. Tungpalan of the Philippines mentioned that the Philippines had attempted to engender an evaluation culture in its bureaucracy to explicitly define the outputs and outcomes that must be supported by a budget. Increased knowledge on evaluation would lead to lessening the pressure for visible results, he believed. He anticipated that if evaluation can be fostered in governments and results are disseminated at high-level fora, such sustained advocacy for evaluation would lead to progress. Dr. Widjojo of Indonesia suggested that donor organizations create forms to show that the government priorities are matched with those of the funding agencies and consolidate all aspects into one shortened document. This, he hoped, would speed up and facilitate the joint work with government. With regard to engendering an evaluation culture and making evaluation a profession, the Chair asked about the situation in Asian countries of national evaluation societies, with the aim to promote a regional network. To this, Ms. Chia of Malaysia responded that in fact such a society, albeit fairly new, existed, demonstrating the increasing interest in the region. Mr. Cao of Viet Nam added that Viet Nam was in the early stages of developing such an association, but was first concentrating on setting up learning organizations. As for the situation in his country, Mr. Chen of China reported that project evaluation was new to China and that even domestically-funded projects were rarely evaluated as the recognition of the importance of evaluation was still not widespread. He stated that China needed to have more training in educating and extending the knowledge of evaluation personnel. Mr. Chen also expressed his concern that in establishing a shared evaluation system, it must be discussed with the different partner countries, taking into account the different national goals of the countries. Mr. Minato of FASID agreed that having an exchange of views was a good opportunity to learn about different perspectives. Dr. Mounlasy of Laos replied that Laos has not yet thought about setting up a system, as it was still working on getting its systems level with the ASEAN requirements. While there has been criticism that Laos is not implementing ODA efficiently, he argued that the country was struggling with setting up good governance, which is a precondition for setting up a common M&E system. In this respect, he voiced Laos’ willingness to participate in discussions on harmonization and standardization. Mr. Walter of the ADB brought up the problem that the lack of an evaluation society would lead to a struggle to find competent domestic consultants in partner countries. He suggested that some form of informal register of consultants be collated by partner countries so that donor agencies could use it as a reference. Mr. Rider Smith of UNDP agreed that to counter the inward focus of traditional evaluation, it was necessary to share methodologies, such as UNDP’s method of outcome evaluation in results-based management. He stated that UNDP was also very keen to support evaluation societies and working with others in the region. The Chair announced that there would be a meeting of the International Development Evaluation Association (IDEAS) in India and hoped for participation and support by the countries around the table. 3. Second Session Agenda 2: "Partnership in Evaluation and Evaluation Capacity Development" Ms. Satoko Miwa, Director, Office of Evaluation, JICA A recent movement in ODA, emphasizing the achievement of results and the concepts of ownership and partnership, has placed increased importance on cooperation between developing countries and donors in monitoring and evaluation (M&E). Japan has been making efforts to enhance this cooperation in M&E to achieve more effective ODA based upon ownership of developing countries. Such efforts include promoting developing countries' participation, providing feedback on evaluation results to developing countries, and providing training programs on evaluation capacity development (ECD). For example, at the project level, JICA's evaluations at the ex-ante, mid-term and terminal stages are carried out as joint evaluations with the implementing organization and the developing country's government. Ex-post evaluations are conducted by national evaluators, and a third party review is done as well to increase transparency. The participation of developing countries from the ex-ante to terminal stages plays an important role in reflecting the developing countries' real needs, enhancing their ownership, ensuring commitment toward the shared goals, and improving the management and the evaluation capacity of developing countries. The process is considered useful for facilitating wider participation of stakeholders, particularly beneficiaries. Program-level evaluations, including Country Program Evaluation (CPE), are also conducted with developing country participation with the results fed back to the relevant stakeholders. However, the extent of joint evaluation is still limited, although increased importance is being placed on cooperation at the program level. In this regard, JICA is currently reviewing its CPE methodology. A solution could be to conduct the evaluation using the country’s development strategy as a framework and to evaluate the effectiveness of intervention based upon the concepts of contribution. This approach requires a so-called "country-led evaluation" with the developing countries taking the lead in line with their evaluation capacity. As a follow up of the ODA workshop last year, the Japanese government has launched an initiative to introduce a systematic training program on development evaluation through JICA. The program comprises three courses: (1) evaluation of ODA loans (with JBIC), (2) evaluation of a technical cooperation project (with World Bank Institute using distance-learning methods), and (3) the institutionalization of an evaluation system (with the Japan Evaluation Society). They aim to address essential issues for ECD, to enhance evaluation capabilities, and to establish a country-specific evaluation system. The training program also emphasizes joint learning and knowledge sharing. JICA's experiences may shed meaningful insights in organizational evaluation capacity development to any organizations. Firstly, an evaluation system is best institutionalized and sustained when it accounts for the realities and needs of a particular organization. Issues for considerations include purpose of evaluation, capacity of staff, role and position of evaluation unit and so on. Secondly, policy and leadership play important roles in mainstreaming evaluation. Driven by ODA reform and administrative reform, the executive board attached significant importance to M&E, which facilitated mainstreaming of M&E and promoted evaluation culture in JICA. Thirdly, JICA has taken various measures to increase utilization of evaluation results. These include incorporating a feedback mechanism into the routine work; improving access to evaluation results; improving evaluation quality; providing information on good practices; and introducing an incentive system for feedback. Finally, collaboration with a wider evaluation community is important for evaluation capacity development. To develop evaluation methodology, train staff, and recruit capable evaluators, JICA has worked closely with various organizations, such as the Japan Evaluation Society, NGOs, outside training institutes, as well as evaluation societies in developing countries. Discussion Mr. Mehrotra of India commented that although joint evaluation was important, involvement by external development fund providers should be restricted to specific projects commensurate with their contribution, because the understanding of the unique circumstances of a country are crucial for evaluation of specific sector projects. He maintained that the decision on who should participate in an evaluation should be left to the partner countries, which will contribute to raising sense of ownership. Furthermore, he called for harmonization of the different procedures and requirements by the fund providers that act as a burden on the recipient country. In this regard, he proposed that the impulse for capacity building also should come from the partner country. Mr. Khan of Pakistan agreed that the recipient country, which is more familiar with the working social milieu of the project, should take the lead in the evaluation, particularly because delaying a project could result in an opportunity cost. As to the agency in the recipient country tasked with M&E, he argued that since aid implementation agencies may not always be impartial, having independent agencies continuously monitor the implementation of a project may be more appropriate. In addition to independence of evaluation, he declared that accountability was most important. In this respect, Mr. Khan suggested that if a project is delayed or has not produced its desired results, the party responsible (agency or individual) should be identified and held accountable. Dr. Widjojo of Indonesia mentioned that Indonesia has been working on its Country Portfolio Performance Review (CPPR) to improve the implementation of all kinds of projects. Although the process was just started in 2002, the CPPR is being used as a review tool, taking into consideration the constant changes in the environment, he reported. Dr. Pitman of the World Bank cautioned that singling out individual parties could become problematic and nullify the role of evaluation. He stated that what has been found at the Bank was that people refuse to cooperate in evaluation if results are used in such a way as to lay blame. At the Bank, he explained, the evaluation unit tried to remain impartial, by leaving other units to make accounting, audit, integrity or corruption assessments. He considered that by distancing itself from making any sort of accusations and thereby demonstrating its honorable intentions in advancing evaluation cooperation, collaboration would not be jeopardized. Mr. Cao of Viet Nam clarified the differences between partner evaluation and partnership evaluation. In order for the partner country to do well, he stated, in addition to human and financial resources development, the partner country must be proactive in taking the initiative for evaluation. For example, he remarked, there should be a resource center that collects and stores information on experiences from previous cooperation activities. With regard to data collection, he suggested that the project management unit of the partner country prepare some baseline data in advance during project implementation to be used at the time of evaluation. On the links between ownership and partnership in ODA evaluation and accountability, Dr. Chhieng of Cambodia pointed out that the questions remained as to whom one is accountable and how ownership and partnership should be balanced. He reported that a joint working group was recently set up in Cambodia to review policies and performance at the sectoral level. He was of the opinion that M&E of a specific sector should be owned by the partner government with participation by donor countries, so that accountability at specific levels is shared. Mr. Tungpalan of the Philippines stated that defining the nature of "partnership" in M&E was complicated, preferring instead the expression "closer synchronization and collaboration" of the mission between countries. He proposed that countries working together have a pre-set schedule to work from to obtain timely results. On the Chair's question about whether the location of the evaluation unit should be independent or in the operating board, Dr. Widjojo of Indonesia explained that despite the demand for decentralization, because regional autonomy was difficult to achieve given Indonesia's 16,000 islands, the location of the evaluation unit in Indonesia remains within the operation. Dr. Pitman of the World Bank agreed with the idea of having layers of accountability and further elaborated on his earlier point by maintaining that it is the evaluator's role to judge against the objectives of the development process in a neutral fashion without imposing moral or legal views onto those being evaluated. Mr. Walter of ADB noted that the evaluation unit should be independent, not necessarily with respect to location but behaviorally. In trying to uphold the so-called "three Is" (integrity, impartiality and independence) while at the same time trying to remain operationally relevant as an independent institution with links to the old operations, he cautioned that the evaluation unit may risk becoming isolated, ignored and irrelevant instead. He offered ADB as an independent evaluator for donor governments to look to for monitoring progress. Mr. Khan of Bangladesh pointed out that more important than location was to determine who the players are and reiterated the importance of developing the capacity at all levels (planning commission, line ministries, civil society and external organizations). Mr. Minato of FASID agreed that the enhancement of professional capacity through such venues as the seminar and setting of ethics for evaluation would lead to more neutrality in evaluation. Dr. Widjojo of Indonesia argued that the wording "independent" may not be appropriate as there can be no guarantee that an agency has no interest at all. As an alternative, he put forward "neutral and professional" M&E institutions as more suitable. Concerning joint evaluation, Mr. Lundgren of OECD/DAC stressed that capacity building in developing countries had to be built on ownership and by addressing the needs of the countries themselves. Mr. Chen of China insisted that while joint evaluation was a good concept, more must be done on the side of the partner country. Ms. Miwa responded to the comments by saying that she felt encouraged by the sense of willingness and confidence expressed by the guests from developing countries to increase ownership over the evaluation process. On the way forward, she projected that the progress will be expedited by sharing more experiences and lessons learned on M&E and joint evaluation among various donors and developing countries. This would be useful to promote harmonization in M&E as well. She voiced her agreement to the comment made on the evaluation unit's independence being an issue, not of position, but of behavior or attitude in terms of professionalism and neutrality. 4. Closing Session Approval of Chairperson's Summary Prof. Hirono read out the Chairperson’s Summary (see page 1), and it was adopted with the full approval of the workshop participants. Closing Remarks Mr. Noriki Hirose, Deputy Director-General, Economic Cooperation Bureau, Ministry of Foreign Affairs of Japan Heartfelt gratitude goes out to the government of Thailand for co-hosting this seminar/workshop and their support in all the preparations despite their busy work following the recent natural disaster. The government of Japan is also supporting the restoration from the damage caused by the tsunami in the Indian Ocean. Thanks also to all the colleagues from Asia and the international community for their constructive participation, contribution and discussion on monitoring and evaluation. In the concluding session, the Chairperson's Summary was successfully adopted, rightly pointing out the importance of establishing management systems, strengthening capacity building and tackling remaining challenges. This was possible due to the devoted efforts of Prof. Hirono, Chairperson of the Workshop. This workshop certainly promoted discussion among the participants from Asian countries and international organizations and contributed toward sharing the knowledge and experiences of the evaluation management between the two. What was hopefully highlighted in the workshop is the importance of evaluation as a means of improving the efficacy and efficiency of ODA and betterment of accountability to the public. Thanks to the initiative by the Thai Government to host an evaluation seminar, both the evaluation workshop and seminar could be held simultaneously. On one hand, there was opportunity to renew knowledge on evaluation theory; and on the other, practical issues and problems among practitioners could be discussed frankly. ---------- Appendix I: Program for Seminar/Workshop I. SEMINAR 17 January (Monday) 9:30-10:00 OPENING SESSION Moderator: Ms. Chanchira Sinoulchan, Programme Officer, Thailand International Development Cooperation Agency (TICA)  ■ Opening Remarks Mr. Piamsak Milintachinda, Director-General, TICA Mr. Tomoyuki Yoshida, Counsellor, Japanese Embassy in Thailand 10:00-10:15 Coffee Break 10:15-12:00 LECTURE IN PLENARY  ■ "Basic Concepts and Techniques of Monitoring and Evaluation" Prof. Ryokichi Hirono, Professor Emeritus, Seikei University 12:00-13:30 Lunch 13:30-15:00 LECTURE IN PLENARY  ■ "Evaluation and Aid Effectiveness-Glossary of Key Terms in Evaluation and Results Based Management" Mr. Hans E. Lundgren, Head, Evaluation Section, Review and Evaluation Division, Development Co-operation Directorate, DAC 15:00-15:30 Coffee Break 15:30-16:30 Orientation for Case-Studies in Evaluation 18-19 January (Tuesday-Wednesday) 9:00-16:30 CASE STUDIES IN EVALUATION (12:30-14:00 Lunch) Sub-Group 1 ■ Education Dr. Suparvadee Mitrsomwang, Chulangkorn University Sub-Group 2 ■ Infrastructure Mr. Satoshi Shigiya, Director, Ex-post Evaluation Division, JBIC & Ms. Takako Haraguchi, Researcher, GLM Sub-Group 3 ■ Rural Development & Poverty Reduction Mr. Naonobu Minato, Acting Director, International Development Research Institute, FASID Sub-Group 4 ■ Environment Dr. George T. Keith Pitman, Senior Evaluation Officer, World Bank Sub-Group 5 ■ Country (Outcome) Evaluation Mr. Nurul Alam, Deputy Director, Evaluation Office, UNDP & Mr. David Rider Smith, Evaluation Specialist, UNDP 20 January (Thursday) 8:30-10:45 CASE STUDIES IN EVALUATION Sub-Group 1 ■ Education Sub-Group 2 ■ Infrastructure Sub-Group 3 ■ Rural Development & Poverty Reduction Sub-Group 4 ■ Environment Sub-Group 5 ■ Country (Outcome) Evaluation 11:00-13:00 CLOSING SESSION Moderator: Ms. Chanchira Sinoulchan, TICA ■ "Challenges and Countermeasures on Monitoring/Evaluation Found Through Case Studies” Representatives of each sub-group (15 minutes) 13:00-14:30 Lunch II. WORKSHOP 21 January (Friday) 9:30-10:00 OPENING SESSION Chair: Prof. Ryokichi Hirono, Professor Emeritus, Seikei University  ■ Opening Remarks Mr. Piamsak Milintachinda, Director-General, TICA 10:00-10:30 Coffee Break 10:30-12:00 FIRST SESSION Chair: Prof. Ryokichi Hirono, Professor Emeritus, Seikei University  ■ Adoption of Agenda  ■ Agenda 1: "Challenges and Countermeasures on Monitoring/Evaluation" Mrs. Suthanone Fungtammasan, Senior Programme Officer, Development Programme Evaluation Unit, TICA 12:00-13:30 Lunch 13:30-15:00 SECOND SESSION Chair: Prof. Ryokichi Hirono, Professor Emeritus, Seikei University  ■ Agenda 2: "Partnership in Evaluation and Evaluation Capacity Development" Ms. Satoko Miwa, Director, Office of Evaluation, JICA 15:00-16:30 Coffee Break 16:30-17:30 CLOSING SESSION Chair: Prof. Ryokichi Hirono, Professor Emeritus, Seikei University  ■ Approval of Chairperson’s Summary  ■ Closing Remarks Mr. Noriki Hirose, Deputy Director-General, Economic Cooperation Bureau, Ministry of Foreign Affairs 19:00 Buffet Dinner ---------- Appendix II: List of Seminar Participants 1. Education Ms. Hajah Mardiana Second Secretary, Department of Multilateral Economics, Ministry of Foreign Affairs, Brunei Darussalam Mr. Houmphanh Soukprasith Director of Asia-Pacific Division, Department of International, Ministry of Foreign Affairs, Laos Mr. U Win Ko Director, Department of Higher Education, Ministry of Education, Myanmar Ms. Chitrlada Chanyaem External Relations Official, Ministry of Education, Thailand Mr. Rattawut Piamsamrit External Relations Official, Ministry of Education, Thailand Ms. Kanya Madsuwan External Relations Official, Ministry of Education, Thailand Ms. Phan Thu Hang Expert, European Division, Foreign Economic Relations Department, Ministry of Planning and Investment, Viet Nam 2. Infrastructure Mr. Md Nuruzzaman Director, Implementation Monitoring and Evaluation Division, Shere-Bangla Nagar Dhaka, Bangladesh Mr. Min Meanvy Director, Department of Administration, Ministry of Public Works & Transport, Cambodia Mr. Ranjit Lal Banerjee Director, Administration Division, Ministry of Finance and Company Affairs, India Mr. Bambang Prayitno Head, Section of International Relation, Ministry of Public Works, Indonesia Ms. Azlina Hj Zainal Abidin Assistant Director, External Assistance Section, Economic Planning Unit, Prime Minister’s Department, Malaysia Ms. Nasanbuyan Lodoidamba Assistance to Director, Economic Cooperation Policy Coordination, Ministry of Finance, Mongolia Mr. Rizwan Ahmad Sheikh Section Officer, Economic Affairs Division, Ministry of Finance and Economic Affairs, Pakistan Mr. Victor Emmanuel S. Dato Director IV, Project Monitoring Staff, National Development Office, NEDA, Philippines Ms. Pornapa Methaweewongs Chief, City and Infrastructure Planning Sub-division, Department of Policy and Planning, Bangkok Metropolitan, Thailand Ms. Sakun Lambasara Economist, Fiscal Policy Office, Thailand Mr. Nguyen Xuan Tien Director, Japan-Northeast Asia Division, Foreign Economic Relations Department, Ministry of Planning and Investment, Viet Nam 3. Rural Development & Poverty Reduction Mr. Tuon Thavrak Acting Director General of Planning, Ministry of Planning, Cambodia Mr. Ruan Xiang Ping Sino-Japanese Technology Cooperation Center, China Mr. Moon Sang Won Staff, Evaluation Team, Korea International Cooperation Agency (KOICA), Republic of Korea Mr. Nilahath Sayarath Deputy Director General, Department of Asia-Pacific and Africa, Ministry of Foreign Affairs, Laos Daw Toe Toe Win Assistant Director, Foreign Economic Relations Department, Ministry of National Planning and Economic Development, Myanmar Mr. Mahesh Kumar Karki Under Secretary, Foreign Aid Coordination Division, Ministry of Finance, Nepal Ms. Attaya Memanvit Senior Programme Officer, TICA, Thailand Mrs. Arunee Hiam Programme Officer, TICA, Thailand Dr. Chamaiporn Tanomsridejchai Foreign Relations Officer, Department of Agricultural Extension, Ministry of Agriculture and Cooperatives, Thailand Ms. Chatuporn Anantapeuch External Relations Official, Ministry of Education, Thailand Mr. Yuthapong Eamchang Economist, Public Debt Management Office, Thailand 4. Environment Mr. Emdadull Haque Deputy Secretary, Economic Relation Division, Ministry of Finance, Bangladesh Mr. Xing Jijun Director, Department of International Cooperation, Ministry of Science and Technology, China Dr. Dheeraj Bhatnagar Director, Project Management Unit (PMU), Ministry of Finance, India Mr. U Myo Myat Director, Planning and Statistics Department, Ministry of Forestry, Myanmar Mr. Jonathan Uy Assistant, Public Investment Staff, National Economic Development Authority (NEDA), Philippines Ms. Chachsaran Tangruthaitham Programme Officer, TICA, Thailand Ms. Usa Lertyaso Chief, Environment Planning Sub-division, Bangkok Metropolitan, Thailand 5. Country (Outcome) Evaluation Mr. Heng Sokun Director, Department of Bilateral Aid Coordination, Council for the Development of Cambodia, Cambodia Mr. Dewo Broto Joko Putranto Deputy Director for Asian Bilateral Financing, National Development Planning Agency (Bappenas), Indonesia Mr. Kwon Yul Head, Team of Southeast Asian Studies, Korea Institute for International Economic Policy, Republic of Korea Mr. Bouasavath Inthavanh Director of Monitoring Division, Department of International, Ministry of Foreign Affairs, Laos Mr. Enkhbat Olzvoi Desk Officer of ODA Loans and Grants, Treasury Department, Ministry of Finance, Mongolia Mr. Tulsai Ram Sedai Under Secretary, Foreign Aid Coordination Division, Ministry of Finance, Nepal Mr. Zulfiqar Haider Khan Deputy Secretary, Ministry of Economic Affairs & Statistic, Economic Affairs Division, Pakistan Mr. Mpduk Mapa Pathirana Director, Japan Division, Department of External Resources, Ministry of Finance and Planning, Sri Lanka Mrs. Suthanone Fungtammasan Senior Programme Officer, TICA, Thailand Mrs. Pairin Pluemthawatchai Human Resources Developer, Department of Public Works and Town & Country Planning, Thailand Mr. Saharat Wongsakulwiwat Planning and Policy Analyst, Department of Local Administration, Thailand Ms. Sawanee Supkitchongcharoen Planning and Policy Analyst, National Economic and Social Development Board (NESDB), Thailand Mr. Panupong Puttitanun Economist, Public Debt Management Office, Thailand Mr. Vu Dai Thang Expert, General Affairs and Project Division, Foreign Economic Relations Department, Ministry of Planning and Investment, Viet Nam ----------- Appendix III: List of Workshop Participants Bangladesh Mr. Akhtaruzzaman Khan Joint Chief, Planning Commission, Ministry of Planning Brunei Ms. Hajah Mardiana Second Secretary, Department of Multilateral Economics, Ministry of Foreign Affairs Cambodia Dr. Chhieng Yanara Deputy Secretary General, Council for the Development of Cambodia China Mr. Chen Futao Counselor, Department of International Cooperation, Ministry of Science and Technology India Mr. Vivek Mehrotra Joint Secretary, Department of Economic Affairs, Ministry of Finance Indonesia Dr. Prasetijono Widjojo Deputy Minister/ Deputy Chairman for Development Funding, National Development Planning Agency (Bappenas) Republic of Korea Dr. Yul Kwon Head, Team of Southeast Asian Studies, Korea Institute for International Economic Policy Laos Dr. Bountheuang Mounlasy Director General, Department of International Cooperation, Ministry of Foreign Affairs Malaysia Ms. Patricia Chia Yoon Moi Director, External Assistance Section, Economic Planning Unit, Prime Minister's Department Mongolia Mr. Erdembileg Ochikhuu Director, Economic Cooperation Policy Coordination, Ministry of Finance Myanmar Mr. U Hla Thaung Deputy Director General, Foreign Economic Relations Department, Ministry of National Planning and Economic Development Nepal Dr. Madhav Prasad Ghimire Joint Secretary, Foreign Aid Coordination Division, Ministry of Finance Pakistan Mr. Muhammad Ashraf Khan Joint Secretary, Economic Affairs Division, Ministry of Economic Affairs and Statistics Philippines Mr. Rolando Tungpalan Assistant Director General, National Economic and Development Authority Sri Lanka Mr. Mpduk Mapa Pathirana Director, Japan Division, Department of External Resources, Ministry of Finance and Planning Thailand Mr. Piamsak Milintachinda Director-General, TICA Ms. Panorsri Kaewlai Director, TICA Mrs. Suthanone Fungtammasan Senior Programme Officer, Development Programme Evaluation Unit, TICA Viet Nam Mr. Cao Manh Cuong Director, General Affairs and Project Monitoring Division, Foreign Economic Relations Department, Ministry of Planning and Investment Asian Development Bank (ADB) Mr. Graham M. Walter Director, Operations Evaluation Department, Division 1 OECD (DAC) Mr. Hans E. Lundgren Head, Evaluation Section, Review and Evaluation Division, Development Co-operation Directorate UNDP Mr. David Rider Smith Evaluation Specialist, Evaluation Office World Bank Dr. George T. Keith Pitman Coordinator, Environment and Water, Operating Evaluation Sector and Global Evaluations Group Japan Prof. Ryokichi Hirono Professor Emeritus, Seikei University Mr. Noriki Hirose Deputy Director-General, Economic Cooperation Bureau, MOFA Mr. Kanji Kitazawa Senior Coordinator, Planning and Programming Division, Economic Cooperation Bureau, MOFA Mr. Yasuhisa Suzuki Deputy Director, Evaluation Unit, Planning and Programming Division, Economic Cooperation Bureau, MOFA Mr. Satoshi Shigiya Director, Ex-post Evaluation Division, Development Assistance Operations Evaluation Office, Project Development Department, JBIC Ms. Satoko Miwa Director, Office of Evaluation, Planning & Coordination Department, JICA Mr. Makoto Iwase Evaluation Planning and Coordination Team, Office of Evaluation, Planning and Coordination Department, JICA Mr. Naonobu Minato Acting Director, International Development Research Institute, FASID ---------- Appendix IV: Profiles of Workshop Representatives --- Bangladesh Mr. Akhtaruzzaman Khan Joint Chief, Planning Commission, Ministry of Planning Mr. Khan holds two Master's degrees, one in Social Welfare and the other in Sociology which he received with honors. He also completed a post-graduate diploma in Community Development. In his current position, he has been associated with the preparation of the Poverty Reduction Strategy Paper (PRSP). --- Brunei Ms. Hajah Mardiana Second Secretary, Department of Multilateral Economics, Ministry of Foreign Affairs --- Cambodia Dr. Chhieng Yanara Deputy Secretary General, Council for the Development of Cambodia (CDC) In 1988, Dr. Chhieng received his PhD. in Economics. He also holds Master's degrees in Economics as well as Business Management. He became Director General for the Ministry of Planning in 1988. In 1993, he joined the Council for the Development of Cambodia, where he remains today as Secretary General. --- China Mr. Chen Futao Counselor, Department of International Cooperation, Ministry of Science and Technology Mr. Chen received his Bachelor's degree in Science from Jilin University in China. He started his career as the Program Officer of the Division of Americas and Oceania in the State Science and Technology Commission in 1985. After working for the Chinese Embassy in Stockholm as Counselor from 2001 to 2004, he took on his current position. --- India Mr. Vivek Mehrotra Joint Secretary, Department of Economic Affairs, Ministry of Finance Mr. Mehrotra holds two Master’s degrees, one in Political Science and the other in Economics. Since 1974, Mr. Mehrotra has been a member of the Indian Administrative Service. Before taking on his current position, he served as Secretary to the Government of Haryana, Fisheries Department as well as Member of the Sales Tax Tribunal in 2002. --- Indonesia Dr. Prasetijono Widjojo Deputy Minister/Chairman for Development Funding, National Development Planning Agency (Bappenas) --- Republic of Korea Dr. Yul Kwon Head, Team of Southeast Asian Studies, Korea Institute for International Economic Policy (KIEP) Dr. Kwon received his PhD in Economics at the Graduate School of Sogang University in 1999. He started his career in the Korea Institute for International Economic Policy in 1991. Currently, he is working as Head of the Research Division in charge of Southeast Asian Studies for KIEP. --- Laos Dr. Bountheuang Mounlasy Director General, Department of International Cooperation, Ministry of Foreign Affairs After receiving a PhD in Economics from Hindu University in India in 1984, Dr. Mounlasy started his career at the Department of Foreign Trade for the Ministry of Commerce. --- Malaysia Ms. Patricia Chia Yoon Moi Director, External Assistance Section, Economic Planning Unit Ms. Chia graduated from Vanderbilt University in 1985. In 1989, she joined the Economic Planning Unit in the Prime Minister's Department, where she currently holds the position of Director of the External Assistance Section. --- Mongolia Mr. Erdembileg Ochikhuu Director, Economic Cooperation Policy Coordination, Ministry of Finance Mr. Ochikhuu received his BA from the National University of Mongolia and an MA from Australia National University. He was awarded the Brilliant Student-Economist of the Year in 1998. In the same year, he joined the Mongolian government as an Economist for the Ministry of Finance and assumed his current position in 2004. --- Myanmar Mr. U Hla Thaung Deputy Director General, Foreign Economic Relations Department, Ministry of National Planning and Economic Development Mr. Thaung received his BSc in Physics from Yangon University. He started his career as an assistant instructor at Technical High School Taunggy for the Ministry of Education. Before taking on his current position, he served as Director of Foreign Economic Relations Department for the Ministry of National Planning and Economic Development. --- Nepal Dr. Madhav Prasad Ghimire Joint Secretary, Foreign Aid Coordination Division, Ministry of Finance Born on 16 November 1953, Dr. Ghimire joined the Civil Service in April 1977 and the Foreign Aid Coordination Division of the Ministry of Finance in March 2003. He also worked as Joint Secretary, Program and Budget Division of MOF, and in the National Planning Commission Secretariat, the Ministry of Population and Environment, the Ministry of Home Affairs and the Ministry of Forests and Soil Conservation in various capacities. --- Pakistan Mr. Muhammad Ashraf Khan Joint Secretary, ADB/Japan, Ministry of Economic Affairs and Statistics Mr. Khan holds a Bachelor of Laws (LLB) and started his career as Assistant Director of the Export Promotion Bureau in 1975. Mr. Khan also worked as Director General for the Board of Investment (BOI) from 2000 to 2001. --- Philippines Mr. Rolando Tungpalan Assistant Director-General for Investment Programming, National Development Office, National Economic and Development Authority (NEDA) A graduate from the University of the Philippines (Diliman) with a degree in Economics, Mr. Tungpalan joined NEDA in 1976 and has held various positions in NEDA including that of Director of the Project Monitoring Staff and Director of the Public Investment Staff. --- Sri Lanka Mr. Mpduk Mapa Pathirana Director, Japan Division, Department of External Resources, Ministry of Finance and Planning --- Thailand Mr. Piamsak Milintachinda Director-General, Thailand International Development Cooperation Agency (TICA) --- Viet Nam Mr. Cao Manh Cuong Director, General, Affairs and Project Monitoring Division, Foreign Economic Relations Department, Ministry of Planning and Investment Mr. Cao obtained his BA in Foreign Studies in 1985. From 1985 to 1989, he worked as a teacher and translator at the English Centre of Hanoi University, and then joined the Ministry. --- ADB Mr. Graham M. Walter Director, Operations Evaluation Department, Division 1 Mr. Walter received his Master's degree in Development Economics from the University of Sussex in the United Kingdom. Mr. Walter joined the Asian Development Bank in 1981, assuming several notable positions. He moved in 1997 to the Operations Evaluation Department and eventually accepted his current position of Director. --- OECD Mr. Hans E. Lundgren Head, Evaluation Section, Development Co-operation Directorate (also known as the DAC Secretariat) Mr. Lundgren is the co-ordinator of the DAC Network on Development Evaluation which brings together the heads of the evaluation departments of 30 development agencies, from both bilateral and multilateral donors. His recent work has covered aid effectiveness, results-based management, and earlier he was involved in aid for poverty reduction and social sectors. Before joining the OECD, he worked in field operations and on operational co-ordination with the UN, both in Africa and at headquarters level. He studied economics and international relations at the University of Gothenburg, Sweden and at College d’Europe, Belgium and at the Graduate Institute for International Studies, Geneva, Switzerland. --- UNDP Mr. David Rider Smith Evaluation Specialist, Evaluation Office --- World Bank Dr. George T. Keith Pitman Coordinator, Environment and Water, Operating Evaluation Sector and Global Evaluations Group In 1994, Dr. Pitman, a British citizen, was honored when Her Majesty Queen Elizabeth II awarded him an Order of the British Empire (OBE) for his contribution to water management in Bangladesh. He is a Chartered Civil Engineer (UK), has a PhD in water resources modeling from University College, London, a postgraduate degree in Hydrogeology, and a first class honors degree in Environmental Sciences and Economics. Dr. Pitman is the acting Coordinator of the Water and Environmental Evaluations Group of the World Bank's Operations Evaluation Department located in Washington D.C. and joined the World Bank Group in 1994. --- Japan Prof. Ryokichi Hirono Professor Emeritus, Seikei University, Chairperson of the Workshop Prof. Hirono, Professor Emeritus, Seikei University and Visiting Professor, Graduate Institute for Policy Studies (GRIPS), Tokyo, graduated from the University of Chicago in 1959 and has had an extensive career in international development and evaluation both at home and abroad. Prof. Hirono served with a number of international organizations, including ADB, ILO, OECD, UNCTAD, UNDP and UNESCAP, as well as the United Nations ECOSOC as chairman of its Committee on Development Policy. He has also spent a number of years teaching as a visiting professor at universities in Asia, America, Europe and Oceania. Currently, Prof. Hirono is vice president of the Japan Evaluation Society and the International Development Evaluation Association, and also sitting on advisory committees of a number of research institutes in Asia and Africa, the United Nations University for Peace, Costa Rica, in addition to being a senior adviser to the United Nations University. His email address is ryokichi@iea.att.ne.jp. Mr. Noriki Hirose Deputy Director-General, Economic Cooperation Bureau, Ministry of Foreign Affairs Mr. Hirose graduated from the Graduate School of Economic, Department of Economics of Doshisha University with an MA in Economics in 1975, joining the Economic Planning Agency in the same year. From 1980 to 1982 he engaged in post-graduate research at the Graduate School of Economics of Princeton University. In 1985, Mr. Hirose was appointed Deputy-Director of the General Affairs Division, Planning and Coordination Bureau of the National Land Agency. From 1986, Mr. Hirose served in various senior positions at the Economic Planning Agency. He has also worked as a Senior Research Fellow at the Institute for International Policy Studies (IIPS) in 1991 and as a Senior Associate Fellow at St. Anthony’s College, Oxford University. Mr. Hirose entered his present position in July 2003. He was born in 1948. Mr. Satoshi Shigiya Director, Ex-post Evaluation Division, Development Assistance Operations Evaluation Office, Project Development Department, Japan Bank for International Cooperation (JBIC) Born in 1965, Mr. Shigiya graduated from Sophia University in 1988. He started his career as an analyst at a private bank in the same year. In 1994, he joined the Overseas Economic Cooperation Fund (OECF). He took the current position in 2003. Ms. Satoko Miwa Director, Office of Evaluation, Planning & Coordination Department, Japan International Cooperation Agency (JICA) Ms. Miwa graduated from Waseda University with honors in 1983 and received an MA in International Relations from International University of Japan in 1985. She joined JICA in the same year. Before assuming her present position in 2002, she was seconded to several agencies including the Ministry of Foreign Affairs of Japan, UNDP and the Imperial Household Agency. She is an active member of the Japan Evaluation Society, and was awarded the Academic Encouragement Award of the Japan Evaluation Society in 2004. Mr. Naonobu Minato Acting Director, International Development Research Institute (IDRI), Foundation for Advanced Studies on International Development (FASID) Mr. Minato is Acting Director of the International Development Research Institute, FASID and also Visiting Professor at several graduate schools including International University of Japan. He started his career at Daiichi Kangyo Bank, Ltd. (1975-89). He obtained his Master’s degree in International Relations from International University of Japan (1991) and joined FASID in 1992. He participated in creating evaluation methods that can be applied for projects, programs and policies. He also has experience evaluating an educational project in Indonesia (1995), a health project in the Philippines (1996), a human development project in Malaysia (1996), and a disaster prevention project in Nepal (1997). ---------- Report of the Seminar/Workshop on ODA Evaluation in Thailand Published by the Aid Planning Division, Economic Cooperation Bureau, Ministry of Foreign Affairs of Japan 2-2-1 Kasumigaseki Chiyoda-ku, Tokyo 100-8919, Japan Tel: +81-3-5501-8363 (Direct) Fax: +81-3-5501-8362