A total of five third-party evaluations were conducted in FY2020, consisting of three evaluations of Japan’s ODA to Brazil, Mongolia, and Rwanda, as well as two evaluations of individual grant aid projects implemented by MOFA, namely, the Economic and Social Development Program in Jordan in FY2015 and the Economic and Social Development Program in Mozambique in FY2017. Two studies on evaluation methodologies and frameworks were also conducted.

Evaluation from Development Viewpoints

For "Relevance of Policies/Project," for the most part, high ratings were given based on confirmation that Japan’s assistance was consistent with high-level policies such as the Development Cooperation Charter as well as international priority issues including the SDGs and was implemented in line with the needs of recipient countries. The evaluations also revealed that Japan is demonstrating comparative advantages in areas such as the environment and education.

Overall, for "Effectiveness of Results," high ratings were also given. Some of the evaluations pointed out that it was difficult to precisely measure the effects of policy implementation, due to the fact that indicators for measuring the degree of achievement of the political objectives had not been set in advance. However, the results of study and analysis confirmed that Japan’s assistance has contributed toward resolving development issues faced by recipient countries.

With regard to “Appropriateness of Processes,” none of the evaluations identified any problems with the assessment of development issues or communication with stakeholders within the implementation process, and all confirmed that the needs of the recipient country had been assessed appropriately. However, some of the evaluations pointed out room for improvement in monitoring and recordkeeping after delivery of the procured product as well as public relations, along with the necessity of promoting disclosure of information regarding the details of assistance.

Evaluation from Diplomatic Viewpoints

Since FY2015, all of MOFA’s ODA evaluations have included evaluation from diplomatic viewpoints, which assesses the effects of ODA on Japan’s national interests. The evaluation criteria consist of “Diplomatic Importance” (why the ODA is important to Japan’s national interests) and “Diplomatic Impact” (how the ODA contributed to Japan’s national interests).

Each of the evaluations conducted in FY2020 confirmed the diplomatic importance of implementing ODA for the relevant country and concluded that the implementation of ODA had positive diplomatic impacts that served Japan’s national interests, including stronger bilateral confidence, a greater presence in the international community, enhanced bilateral economic relations, and a deeper understanding of Japan.

For example, the Evaluation of Japan’s ODA to Brazil confirmed that Japan’s ODA to Brazil is diplomatically important in terms of strengthening deep political, economic, and cultural ties between the two countries as well as promoting collaboration on many shared international issues such as climate change, nuclear disarmament, and United Nations Security Council reforms. Diplomatic impacts were also confirmed, including many actual cases in which Japan’s support has contributed to the promotion of amicable relations between the two countries, as well as the fact that the number of offices of Japanese enterprises advancing into Brazil is on an upward trend, which may have been facilitated by ODA.

Recommendations

Although the five ODA evaluations conducted in FY2020 make recommendations based on their respective individual circumstances, there are some recommendations common to multiple evaluations or applicable to other cases.

Recommendations Common to Multiple Evaluations

● Further Promotion of Public-Private Partnerships

Investment by private enterprises is playing an increasingly greater role in the economic growth of developing countries. Promoting public-private partnerships in ODA is also one of the policies of the Government of Japan. The evaluations of Japan’s ODA to Brazil and Rwanda point out that cases of fruitful coordination and collaboration with private enterprises already exist and suggest that such partnerships ought to be promoted further.

● Promotion of Effective Public Relations and More Active Disclosure of Information

Three of the five ODA evaluations conducted in FY2020 make recommendations regarding the promotion of effective public relations and disclosure of information. The two individual grant aid project evaluations suggest the importance of devising more effective and memorable publicized content for individual projects, including their connections with the overall picture of bilateral relations and support as well as concrete explanations of the significance and nature of the respective assistance. Also, the evaluation of Japan’s ODA to Rwanda recommends that Japan should strive to explain the background to its policy formulation from a broader perspective and in a more comprehensible manner.

● (For Individual Grant Aid Projects) Improvement of Monitoring after Delivery of the Procured Product

In the individual grant aid projects implemented by MOFA, the implementation statuses of the projects were monitored by Embassies of Japan based on reports submitted by the procuring agent^(Note). However, the evaluations recommend that monitoring systems be strengthened, as there were not sufficient systems in place to assess the usage conditions of the procured product and achievement of the anticipated results after delivery.

(Note) The agent that handles all of the procurement procedures in an ODA project, including the bidding process, sales contract, and payment, based on a contract with the government of the recipient country

Recommendations from Individual Evaluations with Possible Applicability to Other Cases

● Enhanced Dialogue among Countries Involved in Triangular Cooperation^(Note)

The Development Cooperation Charter declares that Japan will continue to promote triangular cooperation. The evaluation of Japan’s ODA to Brazil recommends close communication among countries involved in triangular cooperation and the continuous development of monitoring systems. These recommendations may serve as a reference for triangular cooperation in other regions.

(Note) A form of cooperation in which a donor country and recipient country effectively leverage their resources and expertise and work together to assist a third country

● Measures for Countries with Frequent Personnel Reassignment and Government Reorganization

The evaluation of Japan’s ODA to Mongolia recommends enhanced measures to reduce risks posed by frequent personnel reassignment and government reorganization, such as introducing information-sharing mechanisms within the implementing organizations as well as gathering information and preparing countermeasures ahead of government reorganization. As similar conditions are likely to be found in other developing countries, these lessons should be considered in such countries as well.

Actions in Response to Recommendations

MOFA has formulated specific actions in response to the respective recommendations made in each ODA evaluation. The following are examples of actions in response to the five ODA evaluations conducted in FY2020.

The Evaluation of Japan’s ODA to Brazil recommends enhanced dialogue among countries involved in triangular cooperation. MOFA will continue to engage in regular consultations with the Brazilian Cooperation Agency and hold discussions on overall policy. It will also consider establishing a joint system for regular monitoring among the three countries in order to respond flexibly to changes in the circumstances of projects currently underway.

The Evaluation of Japan’s ODA to Mongolia recommends the enhancement of measures to reduce risks posed by frequent personnel reassignment. MOFA will hold discussions with officials on the Mongolian side in order to develop a mechanism by which the results and knowledge already obtained can be accurately passed on in the event that personnel are reassigned while a project is in progress, as well as devising means of transferring technologies.

The Evaluation of Japan’s ODA to Rwanda recommends that Japan should facilitate collaboration with various actors, including Japanese companies. In response, MOFA will continue to promote the ABE Initiative^(Note) and technical cooperation projects in the ICT sector in order to nurture pro-Japanese experts in Rwanda and facilitate the establishment of networks between enterprises in the two countries.

(Note) The African Business Education Initiative for Youth is an industrial human-resource development initiative for young people in Africa. The program provides opportunities for young Africans to pursue master’s degrees at Japanese universities and take part in internships at Japanese enterprises.

The Evaluation of the Economic and Social Development Program in Jordan in FY2015 recommends the implementation of effective public relations. For similar future projects, MOFA will strive to create publications that explain the diplomatic significance of the project in an easily understandable manner incorporating regional diplomacy and global development issues and that go beyond describing that particular project to showcase the state of Japan’s refugee relief and humanitarian assistance as a responsible member of the international community.

The Evaluation of the Economic and Social Development Program in Mozambique in FY2017 recommends the improvement of monitoring after delivery of the procured product. Recognizing the difficulty of post-project verification of the effects of using expendable or consumable items, in similar future projects, MOFA will strive to confirm the usage conditions of the procured product and achievement of results by making advance requests for submission of reports from the recipient country’s government.

Studies of Evaluation Methodologies/Frameworks

In addition to the five ODA evaluations, a “Review of Past ODA Evaluations (Country Assistance Evaluations) and Study of Country Assistance Evaluation Methodologies” and “Analysis of Third-Party Evaluations of MOFA’s Individual Bilateral Grant Aid Projects and Proposal of Evaluation Methods” were also conducted in FY2020. These were aimed at examining better ways of conducting future Country Assistance Evaluations and evaluations of individual grant aid projects, respectively, based on past results.

● Review of Past ODA Evaluations (Country Assistance Evaluations) and Study of Country Assistance Evaluation Methodologies

This study recommended that the timing of implementation of Country Assistance Evaluations be adjusted so that it is easier to reflect their results in policies. It also proposed that since alphabetical or numeric ratings improve the ease of understandability but can also prevent the subjects of evaluation from perceiving the evaluation results in a constructive manner, in light of the goal of ratings, the presentation format of evaluation results should be determined with consideration for both of these factors. In response to these recommendations, when selecting target countries for Country Assistance Evaluations in FY2021, priority was given to countries whose Country Development Cooperation Policy will be revised in the following fiscal year. Also, the decision was made to stop using alphabetical ratings and present evaluation results in the form of comments only starting in FY2021.

● Analysis of Third-Party Evaluations of Bilateral Grant Aid Projects Conducted by MOFA and Proposal of Evaluation Methods

In light of the characteristics of individual bilateral grant aid projects, the report proposed that “Development Viewpoints” and “Diplomatic Viewpoints” be merged; that only two evaluation criteria, “Relevance of Plans” and “Effectiveness of Implementation and Results,” be used; that evaluation teams create an “Evaluation Outline” summarizing basic information on the project instead of requiring the objective framework used for policy-level evaluations; and that ratings be kept to four levels as before but not use an alphabetical format. Based on these proposals, the decision was made to adopt the new evaluation criteria and methodology for evaluations implemented in FY2021 and later.