Column

30 Years of the Annual Reports on Japan’s ODA Evaluation

ODA evaluations change with the times, too. The changes take place in the two aspects of ODA-related practice and theoretical research on evaluation. The development of specialized disciplines related to international assistance, specialized international research based on the practice of evaluation, and straightening of systems related to diplomacy and international cooperation, as well as the maturing of public opinion, have brought about positive changes in the field of ODA evaluation. There is also no doubt that efforts by the Government of Japan, which has developed accountability systems, have had an influence on ODA evaluation. These include the Act on Access to Information Held by Administrative Organs (1999), the creation of the Incorporated Administrative Agency System and its evaluation system (1999), the Central Government Reform (2001), the establishment of the Government Policy Evaluation Act (2001), the establishment of the Japan International Cooperation Agency (JICA) as an incorporated administrative agency and the launch of its new evaluation system (2003), the Public Archives and Records Management Act (2009), and the Administrative Project Review (2011).

Here, I wish to propose a “form-based” approach as a method for thinking about the factors behind these changes. This is because the circumstances of the times are often reflected in the “form,” and because the results that sometimes arise out of necessity become the “form.” The changes to “form” that I wish to raise here come in three stages: the stage of thick, full-scale materials; the stage in which many colored photographs and diagrams are used; and the stage in which the Internet becomes the primary media.

The first stage was the era of the “Annual Evaluation Report on Japan’s Economic Cooperation,” which was about 30 years ago. This was when the report was published in a thick, black-and-white booklet in B5 size. The FY1992 edition, marking the 10th year of the publication of annual evaluation reports by the Ministry of Foreign Affairs (MOFA), contained 429 pages. Other editions containing many pages were the FY1993 edition with 574 pages, and the FY1998 edition with 386 pages. At this stage, the descriptions were a combination of both introductory and specialized contents about ODA and evaluation. In particular, the FY1992 edition began with a preface titled “What is Evaluation?” and included detailed explanations about the current situation of ODA evaluation in Japan, organizational structure, relationship with JICA, and evaluation criteria and guidelines. The outline in the second section covered explanations on the types of evaluation and the practice of these types of evaluation, which covered country assistance evaluation, sector evaluation, joint evaluation, evaluation conducted by Japanese experts and international experts, and evaluation conducted by JICA and the Overseas Economic Cooperation Fund (OECF). The third section contained about 380 pages in a detailed explanation of the practical examples of the types of evaluation introduced in section 2, as well as detailed commentary about the situation in the ODA sites. In particular, the section that covered the achievements and problems with the structural adjustment program exudes a sense of history. This first stage is characterized by professional explanations about the fundamentals of ODA evaluation, making these reports invaluable to researchers who had no connection to ODA evaluation as well as the perfect teaching material for graduate school lectures.

However, there were some weak points. Researchers who had no connection to ODA, the general public, those living in the regions, and foreign researchers would not know how they could obtain the reports. For people who did not even know about the Annual Evaluation Report on Japan’s Economic Cooperation to begin with, the report may as well not have existed. During this stage, it was necessary to have the means of accessing the government offices at ‘Kasumigaseki’ and the political center of ‘Nagatacho’ in person.

The second stage was in the early 21st century, spanning a period of about a decade after the first stage. The number of pages in the Annual Evaluation Report on Japan’s Economic Cooperation was reduced significantly (the FY2002 edition published in March 2003 contained 106 pages), and this reduction in volume was reflected in the simplified introductory descriptions. This stage was also characterized by the use of more attractive photographs and diagrams. On the other hand, reports from this period provided more useful information to researchers in the field of ODA evaluation, such as the ODA reform trends of the time, detailed descriptions of the evaluation implementation system, and the names and designations of evaluation experts. Knowledge that is vital to those aspiring to engage in specialized research was evident on almost every page. More interestingly, this second stage can be further divided into three periods: the period when the report was published by the Economic Cooperation Bureau; the period of reports after organizational restructuring was carried out to form the International Cooperation Bureau; and the period when the evaluation division was moved to the Minister’s Secretariat to secure the independence of ODA evaluations (from 2011). The introduction written by the Director-General and Deputy Minister was important for understanding the circumstances in these three periods, and was positioned as a research theme for international aid administration.

The third stage was from FY2018 and after, when the paper booklet was no longer distributed to the public. There are important points that should be noted in ODA evaluation research in the present times, when the report is mainly published on the Internet. Firstly, there are benefits to doing so. As the reports can be accessed by anyone at any time, those who wish to learn about ODA and ODA evaluation can obtain the information easily. It is also apparent that the authors have put effort into aspects ranging from the aesthetics to the presentation of the text, bearing in mind the fact that the reports are to be published online. It is especially worthy of note that important references, such as the United Nations’ Millennium Development Goals (MDGs) and the 2030 Agenda for Sustainable Development can be accessed in a moment just by clicking on the corresponding link, thereby serving as a useful literature guide to beginners in the field. This is greatly beneficial to users.

However, there are also disadvantages. As it is easy for readers to find the references they are looking for, there is a diluted sense of appreciation as compared to the past, when information could only be obtained after some hard work. Since various references can be accessed by clicking on successive links, they stop thinking about why those references are important. Furthermore, in the time of the paper medium, reviewing the physical reports in chronological order could lead to important discoveries about minor differences or changes in descriptions. That would be difficult to achieve today. In particular, the storage of reports is a challenge, and there are concerns about the lack of good ideas on how to create an archive for ODA evaluations.

30 years of reading the Annual Report on Japan’s ODA Evaluation provokes many thoughts about how reports serve as publicity materials, and about research and learning by utilizing reports.