Fourth International Conference on Science & Technology Metrics (STMet 2023): Proceedings
BITS Pilani, Dubai, UAE. November 16-17, 2023
Evaluation of the Capes International Cooperation Programs: Do the Programs Achieve the Predetermined Objective?
Pablo Gabriel Ferreira, Robin Haunschild, and Lutz Bornmann Universidade Federal de São Carlos, São Carlos/SP Brazil., Max Planck Institute for Solid State Research Heisenberstr. 1, 70569 Stuttgart, Germany., Science Policy and Strategy Department, Administrative Headquarters of the Max Planck Society, Hofgarte
Abstract: The Brazilian Federal Agency for the Support and Evaluation of Graduate Education (CAPES) funds research projects in the framework of their international cooperation programs. It is theobjectiveof the programs toimprove the quality of Brazilian research by fosteringinternational scientific exchangewithinfunded joint projects between researchersfrom Brazil and abroad.The programsare led by the projects’ principal investigators (PIs) who are as a rule tenured professors or experienced researchers. The programs’ objective would have been reached if the funded PIs do research on a (significantly) higher quality level than a control group of Brazilian researchers who are not funded.
This study is intended to carry out an evaluation of the CAPES programs by investigating whether the programs’ objective has been really achieved(Bornmann, 2011).
As a first step, it was analyzed for the period from 2010 to 2022 whether theprograms contributed to increaseinternational collaborations of Brazilian PIs.Their co-authored papers were used as a proxy for collaborations, and an annual average growth rate (AAGR) was computed (to analyze developments over time). Two databases were matchedfor the empirical analyses:A dataset related to joint research projects funded by CAPES was crossed withOpenAlex data.OpenAlex is a free and open bibliometric database. The dataset from CAPES comprised information about 1,706 international collaborative research projects, corresponding to 30 cooperation programs and 17 partner countries.The PIs’ ORCIDs were used to connect CAPES data to OpenAlex.The results of the empirical analyses showed that thePIs increased their international collaboration papers by 10.4%. The increase of international collaboration papers was largest for multinational programs.
As a second step, weanalyzedwhether this increase in collaborations is accompanied byaquality improvementof research.The quality improvementwas measured by median Hazen percentiles(Hazen, 1914). We observed a slight decrease of median Hazen percentiles of funded PIs over the years. However, the median Hazen percentiles of all Brazilian papers decreased faster over the same time period. Taking into account the whole time period, the PI’s papers were more impactful than average Brazilian papers. A two-level robust linear regression analysis controlling for the reputation of the publication outlet (top-quartile journal or not), number of authors, number of countries, and years since funding has confirmed the decreasing impact of the PI’s papers around the point in time as funding started.
Keywords: OpenAlex, Bibliometrics, CAPES, Program Evaluation, International Cooperation Evaluation of the Capes International Cooperation Programs: Do the Programs Achieve the Predetermined Objective?
