Japan's Official Development Assistance White Paper 2008
Next PageMain Text > Part III Official Development Assistance in FY2007 > Chapter 2 Details about Japan's Official Development Assistance > Section 5. Formulation and Implementation of ODA Policy > 3. Matters Essential to Effective Implementation > (1) Enhancement of Evaluation
In order to implement ODA more effectively and efficiently, it is important to get appropriate and accurate knowledge of the implementation process and the effects in order to improve them when necessary. It is also important to explain to taxpayers how ODA is used and how effective it is. To this end, relevant ministries and agencies including MOFA as well as implementing agencies such as JICA conduct monitoring and evaluation.
ODA evaluation is positioned in the framework of PDCA Cycle (Plan → Do → Check → Act). The evaluation results provide feedback to the departments in charge and Japanese embassies overseas, so as to contribute to improving the formulation and implementation of ODA policy. The results are also conveyed to the parties concerned in recipient countries and released to the general public through websites and other resources to ensure accountability.
Further, the ODA Evaluation Workshop is held each year to improve the evaluation capability of developing countries and systematize results-oriented evaluation, with the participation of various Asian countries and international organizations. In 2007 it was jointly held in Kuala Lumpur with the government of Malaysia (with MOFA and the former JICA and JBIC as the Japanese side).
PDCA cycle
• Policy-level and Program-level Evaluations
Evaluations conducted by MOFA focus particularly on policy-level evaluations (country policy evaluations and priority issue evaluations) and program-level evaluations (sectors and aid modalities). In 2007, country evaluations verified the assistance for Sri Lanka, Indonesia, China, Mongolia, Nicaragua, and Tunisia. The main focus of the evaluation included consistency with the aid demands of these countries, the effects of assistance, and the appropriateness of the implementation process.
The evaluation for the assistance for Indonesia, for instance, confirmed that as some of Japan's aid policies were established through close policy dialogues with Indonesian government-affiliated parties and experts, not only were Japan's policies of a high level with a great deal of consistency with Indonesia's medium-term development plan, but also evaluation effects highlighted facts such as that they complemented the support from other donor countries, and thus were suitable. Efforts such as the Supporting Development of Eastern Indonesia Program were reported to have appropriate aid implementation processes. As a forward-facing proposal, based on the changing development needs of Indonesia and the fact that bilateral relations have entered a new phase, the precedence of support for the country has been highlighted as requiring reconsideration.
Priority issues evaluations were also implemented, targeting the "Basic Education for Growth Initiative (BEGIN)" and "Japanese Assistance to Africa through the TICAD Process." These evaluations assessed relevance of Japan's policy with the undertakings of the international community, the effectiveness of the ODA projects, and the appropriateness of the process. In the evaluation of "Japanese Assistance to Africa through the TICAD Process," not only was it confirmed that consistency in the support with leading international frameworks and Japan's high-level policies was being well maintained and that it was being effectively realized in the area of debt relief, but also that various wide-ranging successes had been recognized in region-wide and South-South cooperation, and as such the support process was generally viewed as being suitable. As a forward-facing proposal, the evaluation concludes that it is desirable to further promote comprehensive support incorporating the provision of infrastructure, an area where Japan puts particular effort among major donor countries, with "software" such as skills transfer and human resources development.
In program-level evaluations, a joint evaluation with USAID, the US-Japan Partnership for Global Health, was implemented. As well, as evaluations by the governments and organizations of developing countries, "Japanese Development Assistance to Malaysia Project," "Japanese ODA on Consolidation of Peace and Security in Africa," and "Japanese Cooperation in El Salvador's Eastern Region" were executed.
• Project Level Evaluations
Evaluation of individual projects is one of the areas to be enhanced, as it is important for publicly accountable, effective, and efficient aid. For all Japanese ODA loan projects, the former JBIC conducted ex-ante evaluations at the preparatory stage of all the projects. Two years after the completion of projects JICA, formerly JBIC, conducts ex-post evaluations by external evaluators in accordance with international standards for evaluation of relevance, efficiency, effectiveness, impact, and sustainability. In order to develop a more thorough evaluation system, JBIC conducted mid-term reviews since FY2004 to verify the appropriateness and effectiveness of the project plan five years after the conclusion of loan agreements, along with ex-post monitoring seven years after the completion of projects to verify effectiveness, impact, and sustainability.
Furthermore, impact assessments that quantitatively analyze the effect of ODA loan projects on local residents were conducted. For example, in addition to the ex-post evaluation of "Social Sector Development Project in Amazon Area / Social Sector Development Project in Sierra Area" in Peru, thematic evaluation concerning the Improvement of Living Environment and Livelihoods in Poor Communities was conducted. This evaluation, which employed econometric techniques, analyzed the impact of small-scale infrastructure projects (e.g., water supply, roads, small-scale electrification) on residents financed by a social investment fund17 established under the Fujimori administration. As a result, the evaluation confirmed that water drawing time was reduced and the incidence of diarrhea among children under six declined in beneficiary households in regions in which water supply projects were implemented in comparison with regions in which they were not. It also suggests that small-scale electrification projects led to an increase in business start-ups among beneficiary households.
In technical cooperation, JICA had worked consistently to conduct evaluations at each project stage: prior to commencement, during implementation, at the time of conclusion and following completion. In addition, the recommendations and lessons learned through these evaluations were systematically fed back to inform project planning and implementation. Furthermore, in order to enhance the transparency and objectivity of evaluations, the participation of outside experts in evaluations has been expanded through, for example, secondary evaluations by outside experts of the results of evaluations at the time of conclusion and the inclusion of comments by outside experts in the results of ex-post evaluations.
MOFA, as well, has conducted ex-post evaluations of projects since FY2005 to check how facilities and equipment are used and how effective they are in identifying problems and difficulties of each project. The projects subject to evaluation until 2006 were only those in excess of ¥1 billion, but in 2007 the scope was enlarged to target all general grant aid projects and grant aid for fisheries for which four years has passed since completion. In 2007, ex-post evaluation was conducted for 115 projects in 55 countries. For 98 of these, MOFA carried out primary assessment subsequent to third-party secondary evaluations to verify the validity and appropriateness of primary evaluations. Furthermore, project-level ex-post evaluations by sector, country, and region have been conducted through third-party entrustment. The lessons obtained from those ex-post evaluations are reflected in the development and implementation of new projects.
• Seeking a Comprehensive Evaluation System Suited to the Inauguration of the New JICA
Under the New JICA inaugurated in October 2008, study is now underway to establish a monitoring and evaluation system suitable for all assistance instruments, be it technical cooperation, loan aid, or grant aid. With respect to the program-level evaluations performed respectively by MOFA and New JICA, it has been decided to set up an effective and efficient division of roles that eliminates overlap, with both offices performing coordination at the stage of evaluation plan formulation.