Brussels, 16.4.2025

SWD(2025) 92 final

COMMISSION STAFF WORKING DOCUMENT

Evaluation of the Research Executive Agency (REA)

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL AND THE COURT OF AUDITORS

Evaluation of the Consumers, Health, Agriculture and Food Executive Agency, the Educational, Audiovisual and Culture Executive Agency, the Executive Agency for Small and Medium-Sized Enterprises, the European Research Council Executive Agency, the Innovation and Networks Executive Agency and the Research Executive Agency

{COM(2025) 171 final} - {SWD(2025) 87 final} - {SWD(2025) 88 final} - {SWD(2025) 89 final} - {SWD(2025) 90 final} - {SWD(2025) 91 final}


Table of Contents

1    INTRODUCTION    

2    WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION?    

2.1    Description of the intervention and its objectives    

2.2    Baseline and points of comparison    

3    HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD?    

4    EVALUATION FINDINGS (ANALYTICAL PART)    

4.1    To what extent was the intervention successful, and why?    

4.2    Cost-benefit analysis    

5    WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED?    

5.1    Conclusions    

5.2    Lessons learned    

ANNEX I: PROCEDURAL INFORMATION    

ANNEX II: METHODOLOGY AND ANALYTICAL MODELS USED    

ANNEX III: EVALUATION MATRIX AND ANSWERS TO THE EVALUATION QUESTIONS (BY CRITERION)    

ANNEX IV: OVERVIEW OF BENEFITS AND COSTS    

ANNEX V: COST-BENEFIT ANALYSIS    

ANNEX VI: STAKEHOLDERS’ CONSULTATION – SYNOPSIS REPORT    



Glossary

Term or acronym

Meaning or definition

Act of delegation

Commission Decision C(2013)9418 on delegating powers to the Research Executive Agency with a view to performance of tasks linked to the implementation of Union programmes in the field of research and innovation comprising, in particular, implementation of appropriations entered in the general budget of the Union

Act of establishment

Commission Implementing Decision 2013/778/EU of 13 December 2013 establishing the Research Executive Agency and repealing Decision 2008/46/EC

Agencies

Executive agencies

Commission

European Commission

CHAFEA

Consumers, Health, Agriculture and Food Executive Agency

EASME

Executive Agency for Small and Medium-Sized Enterprises

EEA

European Economic Area

EFTA

European Free Trade Association

EISMEA

European Innovation Council and SMEs Executive Agency

EU

European Union

FET

Future and emerging technologies

Framework Regulation for executive agencies

Council Regulation (EC) No 58/2003 of 19 December 2002 laying down the statute for executive agencies to be entrusted with certain tasks in the management of Community programmes

Seventh Framework Programme

The seventh framework programme of the European Community for research, technological development and demonstration activities (2007-2013)

HaDEA

Health and Digital Executive Agency

Horizon 2020

Horizon 2020, the 8th EU framework programme for research and innovation (2014-2020)

HR

Human resources

IAS

Internal Audit Service

IT

Information technology

REA

Research Executive Agency

SEDIA

Single Electronic Data Interchange Area



1INTRODUCTION 

Purpose and scope of the evaluation/fitness check

The purpose of the present evaluation is to assess the operations of the Research Executive Agency (‘REA’ or ‘the Agency’) between 16 July 2018 and 31 March 2021. In accordance with Article 25 of the Framework Regulation for executive agencies 1 , the Commission submits this evaluation to the European Parliament, the Council, the Court of Auditors and the Steering Committee of the Agency.

REA is governed by the Framework Regulation for executive agencies, its act of establishment 2 , which sets out its mandate, the act of delegation 3 , which specifies the tasks to be carried out and the powers delegated to the Agency, and the decision establishing the REA Steering Committee.

This evaluation concerns the operations of the Agency in accordance with its mandate under the 2014-2020 multiannual financial framework. REA was entrusted by the Commission to implement parts of the Horizon 2020 Framework Programme for research and innovation (‘Horizon 2020’) and the legacy under the Seventh Framework Programme for research, technological development and demonstration activities (‘the Seventh Framework Programme’), and to provide administrative and logistical support services. In this respect, the Agency managed some or all of the phases in the lifetime of projects, carrying out the necessary checks to that end, implemented the budget and provided relevant information to policymakers. The evaluation period of this report predates the establishment of REA’s new mandate under the 2021-2027 multiannual financial framework.

In line with the Commission’s Better Regulation principles 4 , the evaluation assesses: (i) whether the Agency has fulfilled its tasks effectively and efficiently; (ii) whether there are overlaps, gaps or inconsistencies in the Agency’s management of the programme portfolio and (iii) whether there is a clear delineation of tasks between the REA and the parent Directorates-General or other executive agencies (coherence) 5 .

The evaluation also assesses whether the functioning of the Agency has yielded the positive results estimated in the 2013 ex ante cost-benefit analysis for the delegation of tasks to the executive agencies 6 and the specific financial statement. The retrospective cost-benefit analysis considered the updated estimates from the specific financial statement, as amended in 2017 to reflect the extended mandate of the Agency. These estimates have been tested in comparison with the actual costs for programme implementation. The aspects covered by the cost-benefit analysis are specified in Article 3(1) of the Framework Regulation for executive agencies 7 and the Guidelines on establishing and operating executive agencies 8 .

The evaluation of the Agency does not cover the achievements of the programmes managed by REA, which are subject to mid-term and ex post evaluations themselves. The evaluation of REA nevertheless provides useful input for these programme evaluations, considering that the performance of the Agency affects the efficiency and effectiveness of the programmes it manages.

The evaluation of REA is supported by a study carried out by external contractors 9 . The study combined a retrospective and a prospective analysis. The retrospective analysis assessed the operation of the Agency and its performance during the evaluation period, while the prospective analysis suggested recommendations for improving the Agency’s performance going forward.

The evaluation of the Agency’s performance is evidenced-based, using a mixed-methods approach in a complementary manner, by connecting:

·a qualitative approach, based on open-ended survey questions, interviews, a documentary review and desk research, a qualitative cost-benefit analysis;

·a quantitative approach, based on administrative and monitoring data, surveys and a quantitative cost-benefit analysis.

All stakeholders were consulted via interviews and surveys (Commission staff, REA staff, beneficiaries, unsuccessful applicants, experts). Unsuccessful applicants participated less actively compared with beneficiaries. There was a rather low response rate to the survey among beneficiaries and unsuccessful applicants (i.e. 10% on average), which could be due to survey fatigue.

Overall, the evaluation of the Agency provides findings based on solid evidence. Triangulation of data sources improved the validity of the findings, while a retrospective cost-benefit analysis examined cost-effectiveness.

2WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION?

4.1Description of the intervention and its objectives

According to the Framework Regulation for executive agencies and REA’s Act of establishment, the purpose of entrusting the Agency with certain programme implementation tasks is as follows.

·It enables the Commission to focus on its core institutional tasks, which cannot be outsourced, namely tasks assigned under the Treaty on the Functioning of the European Union that require discretionary powers in translating political choices into action.

 

·It enables the Commission to achieve the objectives of the delegated EU programmes more effectively and efficiently. According to the 2013 ex ante cost-benefit analysis, delegating tasks to REA was estimated to be more cost-efficient than dealing with them in-house. The provision of administrative and logistical support services will result in additional cost-efficiency gains and economies of scale.

Between 2018 and 2021, REA implemented the following EU programmes in part or in full.

1)Horizon 2020

a)Excellent science (Pillar 1)

-Marie Skłodowska-Curie actions

-Future and emerging technologies

b)Industrial leadership (Pillar 2): Space research

c)Societal challenges (Pillar 3)

-Food security, sustainable agriculture and forestry, marine, maritime and inland water research and bioeconomy

-Secure societies, including, as of 2018, the processing of grants that may generate EU classified information

-Inclusive, reflective and innovative societies

d)Specific objectives (Pillars 4 and 5)

-Spreading excellence and widening participation

-Science with and for society

2)Legacy of the following parts of the Seventh Framework Programme:

a)The activities ‘research for SMEs’ and ‘research for SME associations’ of the capacities specific programme

b)The ‘space’ and ‘security’ themes of the cooperation specific programme

c)The people specific programme

3)REA also provided the following administrative and logistical support services:

a)planning for calls for proposals and contests for prizes (including support for publication), logistical support for evaluation of project proposals (including the management of the evaluation facilities), contracting and payment of independent experts, validation of legal entities, preparation of legal entities’ financial viability assessment, management of the Research Enquiry Service for Horizon 2020, the Seventh Framework Programme and other EU programmes, as the case may be;

b)single-entry point for participants in grants and procurement activities for all EU programmes, for the benefit of the EU bodies implementing them through direct management 10 within the framework of the Single Electronic Data Interchange Area (‘SEDIA’) 11 ; this mainly entails validation of legal entities and preparation of legal entities’ financial capacity assessment.

 

The intervention logic analyses the Agency’s objectives and activities, as set out in its legal basis. It was adapted considering the specific context, objectives, inputs and processes with which to measure the Agency’s key results and performance. The below figure summarises the approach used.

Figure 1. REA’s intervention logic:

Source: Study supporting the triennial evaluation of the Agency

The key inputs are human resources, financial resources, IT tools and communication channels. The key processes derive from REA’s Act of delegation and the main outputs of REA’s performance relate to the management of the delegated programmes. Achieving the outputs depends on achieving key performance indicators, standard indicators of financial management, and key metrics related to the proposal and project management life-cycles (e.g. number of calls, number of running projects). The key feedback to policy related outputs were also considered.

Outcomes relate to the key results achieved during the evaluation period and impacts (medium-to-long term outcomes). The satisfaction of unsuccessful applicants, beneficiaries and experts with the performance of the Agency was one of the key elements that informed the measurement of these results. In addition, key outcomes are better-informed policymaking on the part of the parent Directorates-General and the Agency’s contribution to policy priorities of the parent Directorates-General and the programmes implementation.

The results of the quantitative cost-benefit analysis are also presented, including the workload analysis and evaluation of key workload drivers, assumptions and productivity indicators (operational budget delegated to the Agency, number of proposals, number of projects, average grant size, etc.). These results underpin the workload analysis and staffing estimates in the 2013 cost-benefit analysis and specific financial statement, compared with the actual situation.

4.1Baseline and points of comparison

The current evaluation of REA operations assesses the actual costs and benefits of programme implementation by REA (executive agency scenario) compared with the alternative scenario of management by the Commission (in-house scenario). To this end, the 2013 ex ante cost-benefit analysis and the specific financial statement of REA, as amended, are the reference points for comparison.

Compared with the in-house scenario, the efficiency gains of the executive agency scenario were initially estimated at EUR 87.4 million in the specific financial statement, as amended. Non-quantifiable benefits were also estimated, such as improved quality of programme management and service delivery, improved visibility of EU programmes and proximity to beneficiaries.

3HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD?

Current state of play

Established in 2008, REA’s mandate was renewed and extended in 2014 to implement parts of Horizon 2020 and the Seventh Framework Programme and to provide administrative and logistical support services.

The Agency’s mandate was further extended in 2018 when the Agency was entrusted with the legal validation of third parties and the preparation of legal entities' financial capacity assessment within the SEDIA framework for all EU programmes. REA was also entrusted with implementing projects generating EU classified information under the Horizon 2020 societal challenge ‘Secure societies – protecting freedom and security of Europe and its citizens’.

During the evaluation period, the number of Agency staff increased from 736 in 2018 to 781 by the end of 2020, making REA the largest of the six executive agencies. REA achieved an advanced level of organisational maturity and properly allocated and matched staff according to the diversity of its tasks and emerging needs. REA successfully adapted to the challenges of the COVID-19 pandemic by adapting its activities to a remote working environment and implementing teleworking for staff.

REA’s staff were highly engaged during the evaluation period, as indicated by the staff engagement index registered in staff survey (69% in 2018). Furthermore, the Agency analysed the findings of the staff surveys and adopted appropriate action plans with the close involvement of staff.

Vacancy rates within REA increased during the evaluation period (from 3.29% in 2018 to 4.99% in 2020), partly due to the impact of the COVID-19 pandemic and other factors, such as limited reserve lists and staff career choices.

A series of organisational and procedural changes were brought in during the evaluation period. REA carried out a major reorganisation in preparation for the new responsibilities delegated as of 1 April 2021 under the 2021-2027 multiannual financial framework. Between 2020 and 2021, REA created a Transition Steering Group and drew up an action plan to facilitate the transition in close collaboration with the Commission. The Transition Steering Group proposed a new organisational structure for REA, which was endorsed by its Steering Committee in December 2020.

The Agency’s administrative budget (commitments) increased from EUR 69.6 million in 2018 to EUR 79.3 million in 2020. The operational budget (commitments) increased from EUR 2 282 million in 2018 to EUR 2 731 million in 2020. 

The number of calls launched by the Agency increased from 20 in 2018 to 47 in 2021, while the number of grants signed remained relatively stable throughout the period (there were 2 030 in 2018, increasing to 2 335 in 2020 and then falling to 2 050 in 2021). The number of running projects for both Horizon 2020 and the Seventh Framework Programme managed by REA fell from 6 248 projects in 2018 to 6 193 in 2019, and then increased again to 7 444 in 2021.

The number of key administrative and logistical support services, such as financial capacity assessments, validation of entities and bank accounts, payments of experts, increased every year between 2018 and 2021, peaking in 2020 for several activities. The number of contracts signed for expert evaluators rapidly increased from 16 781 contracts in 2018 to 21 736 in 2020. Consequently, payments to expert evaluators increased to almost 27 000 payments in 2019. The number of financial capacity assessments handled by the REA also increased from 3 921 cases in 2018 to 6 010 in 2020.

Furthermore, REA was responsible for legal entity and bank account validation from 2020 and validated 8 894 bank accounts in 2020 and 5 558 in 2021. The Agency also answered between 8 000 and 11 500 enquiries every year through its Research Enquiry Service.

The Agency’s extensive experience and knowledge of programme implementation meant that it was able to contribute to the design process of common procedures and processes. This process was managed by the Common Support Centre 12  and set the framework for harmonised implementation by all implementing bodies with respect to common business processes, IT, legal, audit and information and data services.

4EVALUATION FINDINGS (ANALYTICAL PART)

4.1To what extent was the intervention successful, and why?

Effectiveness

For this evaluation, effectiveness relates to how successful the Agency has been in achieving or progressing towards its objectives to the satisfaction of its stakeholders. It also looked at the Agency’s ability to react and adapt to the challenges posed by the COVID-19 pandemic.

The analysis of REA’s operations concluded that it operated within its legal framework during the evaluation period, continuously improved its operations and achieved a high level of overall effectiveness. These results are as demonstrated by key performance indicators and the high satisfaction rate expressed in surveys by stakeholders.

The Agency demonstrated maturity, stability and flexibility in executing its mandate and addressing key matters that occurred during the evaluation period. These include implementing new activities in line with the extension of its mandate in 2018, preparing the new mandate under the 2021-2027 multiannual financial framework and addressing challenges related to the COVID-19 pandemic.

REA achieved good overall results on programme implementation and achieved most of the key performance indicators, as mentioned below.

In particular, REA was successful in preparing and signing grant agreements in good time, as measured by the time-to-grant indicator. REA’s average time-to-grant remained below the regulatory target of 245 days throughout the entire evaluation period (198 days in 2018, 215 days in 2019, 205 days in 2020). The percentage of grants concluded within the time-to-grant target reached 100% in 2018 and decreased by just one percentage point in 2019 and two percentage points in 2020. This was mostly due to the COVID-19 pandemic, when applicants requested additional time for their consortia to complete the necessary administrative obligations before grants were signed. 87% of beneficiaries surveyed agreed that the overall time period from submission of the proposal to the signature of the grant agreement was appropriate.

The Agency has improved its execution of payments over the years, as the percentage of payments made within the contractual deadline increased from 98% in 2018 to 99% in 2019, and to 100% in 2020. This is significant, given the greater number of projects and the complications for REA’s operations arising from the COVID-19 pandemic. High levels of satisfaction were registered among the beneficiaries surveyed with the timeliness of pre-financing payments (94%), interim payments (91%) and final payments (89%).

REA ensured high-quality procedures within the evaluation process, as evidenced by the small proportion of redress cases upheld: below 0.5% for Horizon 2020 calls (0.28% in 2018, 0.23% in 2019 and 0.31% in 2020).

92% of experts found the evaluation process to be smooth. Overall, remote assessment of proposals was rated highly by experts, since 89% of experts considered that the electronic evaluation system was easy to use and that the remote assessment of proposals was efficient. However, some experts, who experienced both in-person and remote evaluations, were less satisfied with the efficiency of remote work, mainly due to the limitations of the IT tools and the absence of in-person networking. The Agency staff were responsive and provided guidance and useful answers to questions, according to 95% of experts surveyed. The guidelines and documents made available to experts were very useful and contributed to their understanding of the process and the related requirements before the start of evaluation, according to more than 90% of experts surveyed. Experts had generally positive experiences but were less satisfied with the remuneration they received (68% of experts). The evaluation also noted the need for greater diversity among experts.

The beneficiaries also rated the evaluation process positively (84% of beneficiaries). However, the survey suggested that the Agency’s feedback to applicants could be clearer, as only 48% of unsuccessful applicants felt satisfied with the transparency of the evaluation process.

The Research Enquiry Service was considered helpful by the 97% of beneficiaries and 73% of unsuccessful applicants surveyed who had used it. However, awareness of this service could be improved since the survey showed that many beneficiaries (59%) and unsuccessful applicants (66%) were unaware of its existence.

The REA Validation Service was also rated positively by 81% of beneficiaries surveyed, who agreed that the electronic tools for validation were user-friendly. The application process was considered to be clear and transparent by 92% of beneficiaries and 77% of unsuccessful applicants surveyed.

As mentioned above, the surveys showed high levels of overall satisfaction with the services provided by REA (86% of beneficiaries, 95% of external experts and 57% of unsuccessful applicants). The stakeholders surveyed indicated a clear willingness to work with REA again in the future (94% of beneficiaries, 83% of unsuccessful applicants and 98% of experts).

On budget execution, the Agency fully executed its operational budget in terms of both commitment and payment appropriations, as in the previous evaluation period.

To ensure sound financial management, the Agency improved its internal control framework in line with the Commission’s internal control framework implementation guide. This allows it to closely monitor progress against objectives. It also allows it to prevent and mitigate potential risks to its operations in a timely and effective way. The European Court of Auditors’ annual reports confirmed that REA’s operations, its cash flow and the changes in net assets were in line with the Financial Regulation and with the accounting rules adopted by the Commission’s accounting officer. The Agency also adopted remedial measures to address the observations and audit recommendations received.

The main indicator for the legality and regularity of transactions is the cumulative error rate, which is extrapolated after implementing corrective measures following audits. As in the previous evaluation, under the Seventh Framework Programme, the cumulative residual error rate was below 2% for the people programme and above the 2% target for space, security and SME actions. This estimated error rate can be attributed largely to the complexity of the funding scheme under the Seventh Framework Programme. The lessons learned from the audits under the Seventh Framework Programme significantly contributed to developing an improved and simplified control framework under Horizon 2020. Consequently, the Agency’s cumulative residual error rate under Horizon 2020 ranged between 2.6% in 2018 and 1.75% in 2021 (target 2-5%). For Marie Skłodowska-Curie actions under Horizon 2020, the cumulative residual error rate fell from 1.55% in 2018 to 0.5% in 2021 (target below 2%).

As in the previous evaluation, REA ensured that staff were available to programme participants and communicated with them during various phases of the project lifecycle. 87% of beneficiaries and 97% of experts surveyed knew whom to contact or how to contact REA for any question on implementation of their grants/projects. Furthermore, those REA staff assigned to projects were easily available and responsive, according to 86% of beneficiaries surveyed. REA also answered citizens’ questions on research framework programmes and the validation process via the Research Enquiry Service, which was appreciated by survey respondents.

REA intensified its efforts around external communication activities, reaching audiences through various channels. REA ensured the EU’s visibility as the promoter of the programmes through communication and media campaigns. Although 98% of beneficiaries were aware that their grants or contracts were funded from the EU budget, significantly fewer (67%) were aware that REA was entrusted by the Commission to manage the programmes.

REA further sought to refine its approach to feedback to policy so as to provide an effective response in collaboration with its parent Directorates-General. As a result, specific plans for providing project results relevant for policymaking, agreed between REA units and their Commission counterparts, were put in place for Horizon 2020 activities, starting in 2019. 

Efficiency

This section considers the relationship between the resources used by the Agency and its output. Among other factors, it covers the management and execution of programmes delegated to the Agency and the extent to which financial and human resource management was sound during the evaluation period. It also includes an analysis of and ways in which to simplify administrative and regulatory burden. These aspects of the Agency’s performance are monitored through key indicators and with additional survey data.

The Agency’s efficiency and workload are closely related to the budget managed, number of staff, number of proposals and number of grants. The main indicators for 2018-2020 (in EUR million or %) are provided below.

  Indicator

2018

2019 

2020 

Operational budget, commitments (EUR)

2 282.5

2 505.3

2 731.1

Operational budget, payments (EUR)

1 935.2

2 140.7

2 176.7

Administrative budget, commitments (EUR)

69.6

75.8

79.3

Administrative budget, payments (EUR)

66.8

74.8

78.1

Number of staff (at the end of the year) 13  

736

753

781

Number of operational staff 14

509

526

552

Programme management cost ratio (commitments) 

3.05%

3.02%

2.91%

Programme management cost ratio (payments) 

3.45%

3.49%

3.59%

Budget ‘per operational head’ (commitments) (million EUR) 

4.84

4.57

4.8

Proposals received

14 610

14 749

16 719

Total running projects

6 248

6 193

6 595

Running projects ‘per operational head’

13.25

11.31

11.58

The Agency’s operational and administrative budget increased between 2017 and 2020. REA proved to be an efficient and cost-effective structure for managing the delegated programmes, as demonstrated by the programme management cost ratio between the administrative budget of the Agency and the operational budget it manages. This ratio of administrative to operational budgetary commitments improved each year and reached 2.91% in 2020. For payments executed, the ratio was stable at almost 3.5% in 2018 and 2019 and increased slightly to almost 3.6 % in 2020.

In comparison with the 2013 ex ante cost-benefit analysis, the actual number of new grants managed by REA was approximately 27% higher between 2014 and 2020 (10 958 compared to 13 911). This increase in number of new grants is mostly due to the grants awarded under Marie Skłodowska-Curie actions whose size was smaller than initially estimated and to the additional grants financed from EEA/EFTA and third-country contributions. The increased number of new projects consequently generated an increased project monitoring workload.

On administrative support services 15 , the number of experts supported on site was lower than initially estimated due to the use of remote evaluations during the COVID-19 pandemic. Between 2018 and 2020, the number of experts contracted was lower overall than the initial estimate (58 770 contracts compared to 72 534). However, the Agency’s validation services registered an increased workload: the cumulative number of validations of legal entities and Legal Entity Appointed Representative (LEAR) extended mandates was higher than initial estimates (55 754 compared to 33 429). There was also a higher number of financial capacity assessments (15 097 compared to 6 549). 

 2018

 2019

 2020

Expert management

 

 

 

Number of experts supported on site

6 418

7 119

1 602

Number of expert contracts

16 781

20 253

21 736

Research enquiry service

 

 

 

Number of requests received by the Research Enquiry Service

8 804

9 826

11 502

Validation services

 

 

 

Number of legal status of entities (FEL) validations

7 360

7 734

8 769

Number of LEAR extended mandate validations

9 767

10 786

11 338

Number of change requests / database maintenance on request of participants

34 056

47 527

48 228

Number of financial capacity assessments (FCA)

3 921

5 166

6 010

The consolidation of validation services in REA constituted a major efficiency gain for the Commission and reduced the efforts required from all agencies, the Commission and participants. REA became a ‘one-stop-shop’ for all applicants participating in procurement and grant processes and this reduced the administrative burden for applicants. This also allowed for a coherent and high-quality database of validated participants to be maintained, which would not have been possible had processing validations been distributed across a range of bodies. As regards the implementation of validation services, the Agency is looking into options to further simplify the process with the aim of reducing workload for its staff.

Significant efficiency gains were achieved in REA’s management of Horizon 2020 following the introduction of simplified rules and procedures, streamlined IT systems and clearer guidance for beneficiaries. REA showed maturity and flexibility as it adapted to the changes brought about by a new framework programme and carried out all necessary simplifications. It effectively used corporate IT tools to support the grant processes to become fully paperless and helped staff to work more quickly and reduce errors. The IT tools were appreciated by internal stakeholders but were rated less favourably by experts and applicants, who thought that they could be made more user-friendly.

Overall, the Agency’s actual workload and productivity levels were higher than the assumptions made in the ex ante cost-benefit analysis. Although the actual workload was influenced by various parameters beyond the Agency’s control, such as number of grants, average grant size or external factors such as the COVID-19 pandemic, the Agency’s organisational development contributed to these good results. To achieve a fit-for-purpose organisation, REA's organisational structure was aligned to the specific programmes it manages, ensuring responsiveness to programme needs. The Agency carried out an annual workload assessment and allocated resources according to each unit’s effective workload. This contributed to maintaining a balanced staff composition with around 70% operational staff, 20% neutral staff 16 , and 10% administrative staff. 

In summary, REA was ready to take up new activities, continuously adapting to evolving challenges and improving the efficiency of its operations. REA also contributed to developing various implementing processes and guidelines for the whole community of agencies, shared best practices and actively participated in the various working groups with the Commission and other agencies.

Coherence 

Coherence concerns the overlaps and complementarities within the programme portfolio managed by the Agency, and the delimitation of responsibilities between the Agency and its parent Directorates-General. It also looked at the Agency’s ability to provide feedback to policy to its parent Directorates-General.

 

The key findings revealed a clear and appropriate delimitation of roles between REA and its parent Directorates-General with no overlaps, in line with the legal framework. The Act of delegation clearly sets out the activities delegated to the Agency and those reserved for the Commission. The memorandum of understanding, signed by the Agency and its parent Directorates-General, provides for the interactions within the overall monitoring and supervision framework. It aims to avoid gaps, duplication of efforts, micro-management and sets out clear responsibilities. The memorandum of understanding ensures a constant and smooth flow of know-how and information between the Agency and the parent Directorates-General via various mechanisms, strengthening cooperation and reporting at different levels, from the level of policy and project officers to the high-level Steering Committee meetings.

During the evaluation period, the Agency had a coherent programme portfolio since REA implemented only parts of research framework programmes, for multiple parent Directorates-General and across a diverse thematic spectrum. The delegated parts of the programme were governed by common rules and procedures under the respective research programme (Horizon 2020 or the Seventh Framework Programme). There was strong coherence between the Seventh Framework Programme and its successor programme, Horizon 2020, in terms of continuity of activities. The harmonisation of business processes, led by the Common Support Centre, made the overall programme implementation across thematic areas coherent, although there were still some specific differences. However, some inconsistencies were observed in the tasks delegated to REA and were the result of sharing thematic areas between agencies.

The Agency maintained good communication with the parent Directorates-General via a range of well-functioning mechanisms. In particular, REA made significant efforts to improve its mechanisms for feedback to policy: (i) using the Catalogue of feedback to policy options, which provides support to beneficiaries to enable them to increase the dissemination capacity of projects; and (ii) participating in the Rapid Reaction Network (i.e. AGILE) 17 . The Catalogue was followed in 2019 and 2020 by annual feedback to policy plans tailored to the needs of each part of the delegated programme.

4.1Cost-benefit analysis 

The retrospective cost-benefit analysis was carried out for 2018-2020 based on the assumptions laid down in the specific financial statement, as updated in 2017 following the extension of the REA’s mandate, and the actual costs of the Agency.

Compared with the in-house scenario, the efficiency gains of the executive agency scenario were initially estimated at approximately EUR 87.4 million in the specific financial statement. The retrospective cost-benefit analysis determined that actual savings were 34.6% higher than initial estimates in the specific financial statement, amounting to EUR 117.6 million for 2018-2020.

For 2018-2020, actual costs amounted to EUR 230.5 million, including EEA/EFTA and third-country contributions (EUR 13.8 million during 2018-2020). Based only on the EU contribution, actual costs amounted to EUR 216.7 million. Compared to the specific financial statement estimates (EUR 218.5 million for 2018-2020), the actual cost based on the EU contribution was lower and savings of EUR 1.7 million (or 0.8 % of the specific financial statement estimates) were made, despite rising costs and the fact that the specific financial statement estimates were not adjusted for inflation.

Estimated costs and savings of the executive agency scenario in 2018-2020, million EUR

Source: Study supporting the triennial evaluation of the Agency

As in the previous evaluation (2015-2018), actual costs under Title I ‘Staff-related expenditure’ were higher than the estimates in the specific financial statement. This is because those estimates did not include costs arising from salary indexation, promotions and increasing staff seniority. The ratio of temporary staff to contract staff corresponded to the estimates. However, the share of higher-grade contract staff increased because of greater reliance on IT automation, enabling REA to concentrate its resources on more content-related tasks. REA also employed additional contract staff financed from EEA/EFTA and third-country contributions to manage this budget.

The actual costs under Title II ‘Infrastructure and operating expenditure’ were significantly lower than estimates in the specific financial statement due to lower building costs and other savings. The actual cost under Title III ‘Programme support expenditure’ was higher than the initial 2018 estimates and close to the 2019 and 2020 estimates.

5WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED?

4.1Conclusions

The Agency was effective and efficient in achieving its objectives, which enabled the Commission to focus on its core institutional tasks. The mandate of the Agency was coherent in terms of programmes: it implemented only research programmes but across a diverse thematic spectrum.

The Agency achieved 100% execution of its operational budget in terms of both commitment and payment appropriations, while remaining cost-effective. This is clear from the programme management cost ratio in terms of commitments between the administrative budget of the Agency and the operational budget it manages, which improved each year to reach 2.91% in 2020. Since this figure demonstrates extremely high efficiency, in the future the focus could be on further increasing the quality of output (e.g. as measured by the satisfaction rate of stakeholders and feedback to policy).

Programmes were implemented in a timely manner, with key indicators such as the time-to-inform and time-to-grant generally remaining below the set targets.

The Agency almost achieved its target of executing 100% of its payments on time, which represented a positive evolution from 98% in 2018 to 99% in 2019 and 100% in 2020.

The use of the corporate eGrants suite, comprising all IT tools to manage grants, had a positive effect on the Agency’s operational efficiency and meant that business processes became fully paperless. Furthermore, it facilitated fully remote work during the COVID-19 pandemic.

The results of the 2018-2020 retrospective cost-benefit analysis confirm that the delegation of programme implementation remained considerably more cost-effective than the in-house scenario, generating major savings for the EU budget and real value for money.

There is a continuous exchange and flow of information and sharing of best practices between the agencies and the Common Support Centre on implementing common business processes, IT tools and further efficiencies and improvements. Strengthening synergies and harmonising procedures and processes is at the core of this cooperation. All the agencies have been consulted, have provided feedback on their experience and contributed to developing harmonised business processes, IT tools, reporting templates and guidelines in working groups (steering committees, executive committees and key user groups per business process). 

Communication between the Agency and the Commission was facilitated via well-functioning mechanisms, which were formalised and improved over the evaluation period. No evidence of overlaps, gaps or significant inconsistencies (see also Section 5.2 below) in the management of the programme parts delegated to the Agency was identified. There was a clear delineation of tasks between REA and the parent Directorates-General.

4.1Lessons learned 

Some inconsistencies were observed in the tasks delegated to REA between 2018 and 2021 and were the result of sharing thematic areas between agencies. These inconsistencies were addressed in the new mandates of the agencies for the 2021-2027 programming period by delegating programme parts such as Space Research and FET-Open (previously managed by REA) to other agencies (HaDEA and EISMEA). Moreover, REA was entrusted with new parts of the research programme and other EU programmes with a thematic link.

Delegation of similar thematic areas and programmes to one agency should lead to more efficient implementation, while fewer parent Directorates-General should simplify the governance and supervision of REA by the parent Directorates-General.

The main lessons identified relate to improving the set-up and cooperation with other agencies and the parent Directorates-Generals on key processes, such as feedback to policy, human resources and IT. More cooperation between agencies would be beneficial for knowledge sharing, strengthening synergies and harmonising procedures and processes.

Sharing best practices on expert management and improving collaboration between stakeholders (agencies, joint undertakings and parent Directorates-General) is key to attracting high-profile experts. Under the new mandate, REA ran a communication campaign to attract expert evaluators. This generated a 46% increase in new registrations in the expert database, a 13% increase in new registrations from women experts, and upward trends in new registrations from under-represented EU countries.

Strong collaboration between REA and the parent Directorates-General on communication activities mean that duplication of efforts would be avoided and strategic efforts to efficiently communicate on programmes would be harmonised.

 

The corporate IT tools used throughout the project management process should aim to improve the user experience and increase user-friendliness. The improvements to IT tools should take user feedback into account. Opportunities to improve the user-friendliness of corporate IT tools, such as the development of the Funding & Tenders Portal (previously the Participants Portal), should be taken as part of a continuous process.

It is important to consolidate the feedback to policy dialogue between REA and the Commission. The process and approach to ensure feedback to policy have been improved over time, taking into account the specific nature of the programmes implemented, the variety of the needs of the parent Directorates-General and the related business processes. There were many developments, starting with the catalogue of feedback to policy options developed by REA, which marked a turning point and was the basis for specific feedback to policy mechanisms relevant to parent Directorates-General. This led to an annual feedback to policy plan in 2019 and 2020 tailored to the needs specific to each part of programme delegated, in line with the policy priorities of the parent Directorates-General. This approach brought clarity but less flexibility for unpredictable and ad hoc feedback to policy needs. There was therefore a requirement to strike a balance between structured and ad hoc feedback to ensure a constant flow of information and flexibility in addressing evolving policy needs.

To this end, a feedback to policy framework was developed in 2021 by the Commission based on best practices from agencies and parent Directorates-General. This framework provides structure and direction, while offering sufficient flexibility to cater for and adjust to specific policymaking needs. By identifying the high-level policy priorities and needs of parent Directorates-General in advance, feedback to policy activities are implemented following an identified plan. The need for ad hoc requests is thereby reduced, although such requests are allowed and addressed by the Agency to fulfil specific policy needs of the parent Directorates-General not provided for in the plan.

Overall, ensuring efficient and operational feedback to policy remains a priority. To be effective, channels of communication between projects and policy must allow information to flow freely in both directions. Best practice in this area could be achieved in several ways, for example, through participation by Commission policy officers in project kick-off and review meetings and ad hoc clustering events.



ANNEX I: PROCEDURAL INFORMATION 

The evaluation had a clearly defined scope:

·The reference period of the evaluation was 16 July 2018 – 31 March 2021.

·The scope of the evaluation covered REA’s mandate, which consisted of programme implementation and administrative and providing logistical support services.

With regard to programme implementation, REA managed the parts of Horizon 2020 that had been delegated to the Agency since 2014 and legacy, namely ongoing projects, under the Seventh Framework Programme.

In terms of support services, REA’s main tasks included planning and support for the publication of calls for proposal and contests for prizes; logistical support for the evaluation of proposals; contracting and payment of independent expert evaluators; validating the legal status and financial capacity of participants within the framework of the Single Electronic Data Interchange Area, management of Research Enquiry Service.

·The current generation of agencies established under 2021-2027 multiannual financial framework was outside the scope of this evaluation.

The evaluation of REA was part of the evaluation of all the six agencies, which was made in a coordinated manner based on a supporting study carried out by an external contractor. The preparations for the evaluation started in 2022, when an interservice group was set up by all lead parent Directorates-General and the concerned central services (Directorate-General for Budget and the Secretariat-General). Six group meetings were chaired by the Common Implementation Centre, set up within the Directorate-General for Research and Innovation, who also ensured the secretariat. The evaluation of each agency was led by the respective lead parent Directorate-General (PLAN/2022/1912). The Commission also launched a ‘call for evidence’ on 10 March 2023 on its ‘Have your say’ portal. The call was open for feedback on agencies activities until 7 April 2023.

The methodology used for the evaluation was consistent across the agencies and in accordance with the criteria provided in the Commission’s Better Regulation guidelines. The assessment of the evaluation criteria ‘EU added value’, i.e. why the EU should act, was not considered relevant as each agency carries out tasks which the Commission transferred to it. The EU added value of the programmes that each agency manages is assessed in the context of the programme evaluations.

Evidence was taken from sources such as the Commission databases, annual reporting exercises, adopted decisions. The supporting study was prepared using a mixed-methods approach at the levels of methodologies and methods and worked on a qualitative methodological approach (based on documentary review and desk research, interviews, answers to open-ended survey questions and qualitative cost-benefit analysis) combined with a quantitative methodological approach (based on administrative and monitoring data, surveys and quantitative cost-benefit analysis), as detailed in Annex II.

Knowledge from this evaluation will inform on possible improvements in the implementation by the agencies of the programmes of the Commission.

Furthermore, the results of this evaluation will be useful for accountability purposes. The final evaluation reports on the performance of the six agencies will allow the Commission to report the results of the retrospective cost-benefit analysis to the budgetary authority while informing various European Union (EU) institutions on the value of the cost-savings achieved as a result of the executive agency scenario compared with the costs of in-house management by the Commission, as well as identifying potential areas for improvement. The results of this evaluation will be communicated to the Steering Committee of the Agency, to the European Parliament, to the Council, and the European Court of Auditors.

The Agency had six parent Directorates-General, namely:

·Lead Parent Directorate-General: Directorate-General for Research and Innovation

·Parent Directorates-General:

-Directorate-General for Education and Culture

-Directorate-General for Communication Networks, Content and Technology

-Directorate-General for Agriculture and Rural Development

-Directorate-General for Migration and Home Affairs

-Directorate-General for Defence, Industry and Space 18



ANNEX II: METHODOLOGY AND ANALYTICAL MODELS USED

The evaluation was supported by a study carried out by an external contractor in accordance with the criteria provided in the Commission’s Better Regulation guidelines. The methodology for the evaluation of REA was consistent with the approaches employed for the evaluations of the other agencies.

The evaluation followed an evidence-based and mixed-methods approach, using qualitative and quantitative data to answer evaluation questions. Triangulation of data sources strengthened the validity of the findings. The mixed-method approach was used based on the nature of evaluation questions and issues. The study was structured around a series of evaluation questions outlined in the Annex to the study 19 .

The evaluation relied on the evidence gathered and analysed using qualitative and quantitative methods and included:

·Extensive documentary review and desk research concerning all relevant documentation relating to REA’s legal base, its relation with the parent Directorates-General, and data on its financial and non-financial performance, including the memorandum of understanding, the annual work programmes, the annual activity reports, the European Court of Auditors (ECA) and Internal Audit Service (IAS) audit reports, the previous evaluations and cost-benefit analysis; the results of 2018 and 2021 staff satisfaction survey. As part of the desk research, an in-depth analysis was conducted on communication and feedback to policy.

·Interview program: 40 interviews in total with 51 interviewees, representing the main stakeholders, structured in three stages:

-The scoping phase consisted of exploratory and scoping interviews on REA’s latest and most significant developments during the reference period (one Commission staff and one REA staff)

-The main phase consisted of semi-structured interviews, having each tailored questionnaire to each target group (Commission staff, REA staff)

-The follow-up phase consisted of in-depth interviews with senior management in Commission and REA, beneficiaries, unsuccessful applicants, and experts.

·Survey programme: Two surveys were conducted as part of the evaluation:

-Survey A targeted REA beneficiaries and unsuccessful applicants who applied for funding in the reference period. 702 responses from beneficiaries and respectively 875 responses from unsuccessful applicants were received. This represents a response rate of 14% for beneficiaries, and 9% for unsuccessful applicants, making an average response rate of 10% for Survey A

-Survey B focused on external experts who worked with REA in the reference period. A total of 469 responses were received, representing a response rate of 33%.

All eligible respondents were approached for the beneficiary and expert surveys. Unsuccessful applicants were initially sampled for inclusion at 5 000 potential respondents and supplemented later with another 5 000 potential respondents.

·Cost-benefit analysis included a workload analysis and the assessment of the cost-effectiveness of the delegation of programme implementation, by comparing the actual indicators with the estimates in the 2013 ex-ante cost-benefit analysis and respectively the specific financial statement.

Limitations and robustness of the findings

All relevant stakeholders were consulted via interviews and surveys (Commission staff, REA staff, beneficiaries, unsuccessful applicants, experts). Unsuccessful applicants participated less actively compared with beneficiaries. There was a rather low response rate in the survey for beneficiaries and unsuccessful applicants (i.e. 10% on average), which could be due to survey fatigue. The absolute majority of respondents were associated with Marie Skłodowska-Curie actions as it represented 79% of the overall portfolio of grants of REA.

No sampling bias was observed as the profile of the respondents to the surveys was very similar to the overall population, guaranteeing statistical representativeness. The non-response bias (not all characteristics of the group that did not reply had been captured in full) was mitigated through triangulation with the results of follow-up interviews.

The triangulation approach, using multi-level and multi-stakeholder dimension in the data collection, ensured the robustness and reliability of the data and information used to draw up conclusions in the supporting study. It used diverse data sources, including documentary reviews, desk research, surveys, and interviews, ensuring a comprehensive assessment of REA's operations.



ANNEX III: EVALUATION MATRIX AND ANSWERS TO THE EVALUATION QUESTIONS (BY CRITERION) 

The purpose of the evaluation was to respond to evaluation questions relating to the criteria of effectiveness, efficiency, coherence, as defined by the Commission’s Better Regulation guidelines. Evidence-based answers, including the points of comparison and sources, can be found in Section 4 of the study supporting the evaluation 20 .



ANNEX IV: OVERVIEW OF BENEFITS AND COSTS 

The benefits and costs are detailed in the Cost Benefit Analysis in section 4.2 and Annex V below.



ANNEX V: COST-BENEFIT ANALYSIS

A cost-benefit analysis was performed to assess whether the conclusions of the ex ante assessments (the estimates of savings provided in the specific financial statement, as amended following the extension of REA’s mandate in 2018) are still valid when compared with the actual situation, and what the overall actual savings are.

The following approach was adopted:

1.To use data on the actual performance of REA (actual execution of the administrative budget, actual staffing, etc.) for the executive agency scenario.

2.To follow the assumptions laid down in the specific financial statement to ensure the comparability and validity of results, and to provide estimates of the comparable ‘actual’ in-house (Commission) scenario (the comparator), which would best reflect the actual situation.

3.To assess based on these estimates whether the conclusions of the ex ante assessments provided in the specific financial statement are still valid when compared with the actual situation, and what the overall possible savings are.

To deconstruct the ‘actual’ in-house scenario (the ‘comparator’), estimates were based on the following assumptions:

·Number and composition of staff in the Commission under the in-house scenario correspond to the estimates 21 in specific financial statement. Additional contract staff were added to the estimated Commission staff number for 2018-2020 (31 contract staff in 2018, 34 contract staff in 2019 and 37 contract staff in 2020) to reflect additional authorised staff in REA to manage the additional operational budget from the contributions of EEA/EFTA and the participation of candidate countries and/or third countries that was not covered in the resource calculations in the specific financial statement, thereby ensuring comparability of the ‘in-house’ and the ‘executive agency’ scenarios.

·Commission staff costs and overheads correspond to the average costs set by Directorate-General for Budget for the estimates of human resources and overheads in legislative financial statements for the respective year.

·Programme support expenditure (Title III) remains the same under both the in-house scenario and the executive agency scenario.

The table below presents the results of the analysis of the estimated actual costs of the in-house scenario and the actual costs of the executive agency scenario.

Estimated actual costs of the in-house scenario and the actual costs of the executive agency scenario, EUR

 

2018

2019

2020

Total

2018-2020

No.

Cost

No.

Cost

No.

Cost

In-house scenario

 

 

 

 

 

 

 

Commission

 

 

 

 

 

 

 

Title I. Staff-related expenditure

847.6

81 827 300

866.6

86 449 100

910.6

92 895 000

261 171 400

Temporary Agents

571.7

68 032 300

582.9

71 696 700

611.6

76 450 000

216 179 000

Contract Agents

275.9

13 795 000

283.7

14 752 400

299.0

16 445 000

44 992 400

Title II. Infrastructure and operating expenditure

 

20 342 400

 

21 665 000

 

22 765 000

64 772 400

Title III. Programme support expenditure

 

7 895 694

 

7 118 710

 

7 137 528

22 151 932

TOTAL COST

847.6

110065394

866.6

115232810

910.6

122 797 528

348 095 732

Executive agency scenario

 

 

 

 

 

 

 

REA

 

 

 

 

 

 

 

Title I. Staff-related expenditure

736

51 823 438

753

57 142 945

781

60 490 293

169 456 676

Temporary Agents

175

19 602 919

182

21 733 801

185

22 900 071

64 236 791

Contract Agents

561

28 605 176

571

31 933 551

596

34 258 373

94 797 100

Interim supportive agents and trainees

 

944 870

 

858 668

 

567 461

2 370,999

Professional development and recruitment costs

 

2 670 473

 

2 616 925

 

2 764 388

8 051 785

Title II. Infrastructure and operating expenditure

 

9 841 834

 

11 504 336

 

11 717 096

33 063 265

Title III. Programme support expenditure

 

7 895 694

 

7 118 710

 

7 137 528

22 151 932

Total REA cost:

 

69 560 966

 

75 765 991

 

79 344 917

224 671 873

Commission

 

 

 

 

 

 

Title I. Staff-related expenditure

13.5

1 578 900

13.3

1 628 800

13.2

1 650 000

4 857 700

Temporary Agents

13.1

1 558 900

13.2

1 623 600

13.2

1 650 000

4 832 500

Contract Agents

0.4

20 000

0.1

5 200

0.0

0

25 200

Title II. Infrastructure and operating expenditure

 

324 000

 

332 500

 

330 000

986 500

Total Commission cost:

 

1 902 900

 

1 961 300

 

1 980 000

5 844 200

TOTAL COST

749.5

71 463 866

766.3

77 727 291

794.2

81 324 917

230 516 073

ESTIMATED SAVINGS

98

38 601 528

100

37 505 520

116

41 472 611

117 579 659

Source: Study supporting the triennial evaluation of the Agency

The analysis concluded that:

The overall actual costs of the executive agency scenario 22 amounted EUR 230.5 million during the period 2018-2020. The estimate in the specific financial statement (EUR 218.5 million for the period 2018-2020) was based on the EU contribution, but REA’s administrative budget also included EEA/EFTA and third country contributions (EUR 13.8 million during the period 2018-2020). Consequently, based on the EU contribution only, the actual costs of the executive agency scenario amounted to EUR 216.7 million, which means that the actual costs corresponded to the initial estimations (with savings of EUR 1.7 million, or 0.8% of the estimate in the specific financial statement). This was achieved despite rising costs and the fact that the estimates in the specific financial statement were not adjusted for inflation.

The costs of the executive agency scenario were much lower than the estimated costs of the in-house scenario. In 2018-2020, the actual cost-savings deriving from a difference in cost between the executive agency scenario and the in-house scenario amounted to EUR 117.6 million (or 33.8% of the estimated costs under the in-house scenario).

When comparing the savings initially estimated in the specific financial statement with the actual savings from the delegation of tasks to REA, the actual savings during the 2018-2020 period were 34.6% higher than the initial estimate in the specific financial statement (EUR 117.6 million, compared with EUR 87.4 million). As forecast in the specific financial statement, the savings of the executive agency scenario resulted primarily from a higher share of lower-cost external personnel being employed within the Agency, and a lower overall number of staff 23 .



ANNEX VI: STAKEHOLDERS’ CONSULTATION – SYNOPSIS REPORT 

This annex summarises the stakeholders’ consultation conducted for the evaluation, in line with the Commission’s Better Regulation guidelines. A similar approach was adopted for the evaluation of all six agencies.

1.Outline of the consultation strategy

The consultation strategy was designed as follows:

 Source: Study supporting the triennial evaluation of the Agency

REA’s stakeholders are grouped as follows:

-staff of the Agency;

-staff from the five parent Directorates-General; 

-applicants (successful and unsuccessful) to the funding programmes that REA manages; 

-experts that REA hires to assess funding proposals.

Applicants and experts were surveyed and all four groups of stakeholders were interviewed.

Consultation activities and target groups covered

Consultation activity

Target groups

Date

Surveys

Survey A

Beneficiaries and unsuccessful applicants of the programmes managed by REA

29 November – 16 December 2022

Survey B

External experts contracted by REA

24 November – 7 December 2022

Interview programme

·Commission officials from parent Directorates-General

·REA senior managers

·REA heads and representatives of units

·Representatives of the REA Staff Committee

·Beneficiaries of and unsuccessful applicants to the programmes managed by REA

·External experts contracted by REA

23 August 2022 –March 2023

Source: Study supporting the triennial evaluation of the Agency

These stakeholder consultations complemented each other and informed the answers to the evaluation questions on effectiveness and efficiency. Interviews with Commission and REA staff provided information both on these questions and on coherence-related evaluation questions.

Consultation activity and evaluation criteria covered

Effectiveness

Efficiency

Coherence

Interviews with the European Commission

✓

✓

✓

Interviews with REA

✓

✓

✓

Interviews with applicants and experts

✓

✓

Survey of REA’s applicants

✓

✓

Survey of REA’s expert evaluators

✓

✓

Source: Study supporting the triennial evaluation of the Agency

2.Key results of the consultation activities

2.1Summarised results of the survey

Two surveys were carried out: Survey A for applicants (beneficiaries and unsuccessful applicants) and Survey B for experts. The survey questions aimed to ensure complementarity with the interview programme. Survey A was active for almost 3 weeks and Survey B for 2 weeks.

Survey A addressed all applicants from the calls for proposals launched under the 2017-2020 annual work programmes. 702 responses were received from beneficiaries, representing a response rate of 14%. 875 responses were received from unsuccessful applicants, representing a response rate of 9%. The overall response rate was 10% for Survey A.

Survey B targeted all external experts contracted by REA between 2017 and 2020. 3 694 responses were received (a response rate of 33%).

Main statistics related to the implementation of the surveys

Target group

Full launch of the survey (most invitations sent on this date)

Survey closure date

Population targeted/ no. of invitations sent

No. of responses received

Response rate*

No. of responses included in the analysis after cleaning

Beneficiaries

29 November 2022

16 December 2022

5 111

Partial*: 153

Complete: 549

Total: 702

Total: 14%

Only including complete: 11 %

Partial*: 118

Complete: 549

Total: 667

Unsuccessful applicants

29 November 2022

16 December 2022

10 500

Partial*: 251

Complete: 624

Total: 875

Total: 9%

Only including complete: 6%

Partial*: 167

Complete: 624

Total: 791

Experts

24 November 2022

7 December 2022

11 610

Partial*: 469

Complete: 3 225

Total: 3 694

Total: 33%

Only including complete: 28%

Partial*: 258

Complete: 3 221

Total: 3 479

Note: The response rate was calculated based on the formula:

Source: Study supporting the triennial evaluation of the Agency

Effectiveness

86% of beneficiaries (456 of 529) and 95% of experts (3 007 of 3 173) were satisfied with REA’s services during the evaluation period. Unsuccessful applicants were less satisfied (57%, or 287 of 498), which is similar to the last evaluation period (55%). All three stakeholder groups surveyed indicated a high level of willingness to work with REA again: namely 94% (512 of 543) of beneficiaries, 83% (516 of 620) of unsuccessful applicants and 98% (3 138 of 3 201) of experts. Those experts who were not willing to work with REA again indicated the imbalance between the total remuneration received and the amount of time spent on evaluation tasks (63%, or 40 of 63) as their main reason, followed by low remuneration (35%, or 22 of 63) and personal/work reasons (32%, or 20 of 63).

The effectiveness of the Agency’s evaluation and selection process for applications was assessed differently depending on the groups surveyed. Unsuccessful applicants expressed dissatisfaction with the feedback given in the free responses. Only 48% (324 of 668) of unsuccessful applicants felt satisfied with the transparency of the evaluation process, while 84% of beneficiaries rated the evaluation process itself positively (504 of 597).

As regards the way in which applicants learned about funding opportunities, the surveys showed that they relied on their own organisation or institution (24%, or 334 of 1 373, of beneficiaries and 52%, or 381 of 729, of unsuccessful applicants) and on the Europa website (21%, or 283 of 1 373 of beneficiaries, and 43%, or 311 of 729, of unsuccessful applicants). National agency/contact points were also a source of information for both beneficiaries (12%, or 161 of 1 373) and unsuccessful applicants (19%, 138 of 729). In contrast, REA social media was used the least frequently (16 beneficiaries and 10 unsuccessful applicants, or 1% of each group).

Experts learned about the opportunity to work with REA via direct invitation (41%, or 1 434 of 3 479), colleagues at their institution (28%, or 933 of 3 479) and/or participation in another Commission/Agency project (26%, or 848 of 3 479). Info days and other EU promotional events were rarely cited by experts as the way they learned about the opportunity to work with REA (2%, or 79 of 3 479). REA social media and magazines were the least used channel.

On the perception of the EU as the promoter of programmes entrusted to REA, 98% (612 of 625) of beneficiaries agreed or strongly agreed that they were aware that the grant and tender funding came from the EU budget. However, significantly fewer (67%, or 413 of 616) agreed or strongly agreed that they were aware that REA was entrusted to manage the programme by the Commission.

Communication activities were rated positively by both experts and applicants. 95% (2 721 of 2 865) of experts agreed or strongly agreed that REA staff were responsive and provided answers to their questions. 86% (413 of 481) of beneficiaries agreed that the REA staff assigned to their project were responsive and available. 59% (990 of 1 678) of experts consulted the online information on the Funding & Tenders Portal (formerly the Participants Portal) before or during the registration process. Of the information provided there, the Horizon 2020 manual was rated the most helpful, with 91% (840 of 922) of experts agreeing or strongly agreeing that it was helpful. On the applicants’ side, 89% (538 of 608) of beneficiaries and 83% (557 of 675) of unsuccessful applicants agreed or strongly agreed that information for applicants was easy to find. There was greater disparity between unsuccessful applicants and beneficiaries as regards the process, with 83% (504 of 509) of beneficiaries agreeing or strongly agreeing that the requirements were reasonable and proportionate compared to 69% (464 of 672) of unsuccessful applicants.

86% (113 of 131) of applicants who used the Research Enquiry Service found it helpful or very helpful. The validation service was also rated positively, with 81% (45 of 56) of applicants agreeing or strongly agreeing that the electronic tools for validation were user-friendly. Overall, the level of satisfaction with the clarity of the application process was 92% among beneficiaries (558 of 605 agreed or strongly agreed that the application process was clear and transparent) and 77% (516 of 669) among unsuccessful applicants.

Efficiency

The help desk service was mostly rated positively, with 79% (544 of 692) of experts finding it useful or very useful. The FAQ, glossary and reference documents were all rated similarly, with 88% (757 of 895), 80% (643 of 798) and 89% (817 of 925) of experts finding them useful or very useful, respectively. As regards the evaluation of proposals, 93% (2 987 of 3 196) of experts agreed or strongly agreed that the guidelines were clear. However, this number was slightly lower (88%, or 2 447 of 2 818) when they were asked if they agreed that discussions between REA and experts created a common understanding of proposal assessment.

The time taken to process payment requests was generally rated very positively by experts and applicants alike. 94% (381 of 404) of beneficiaries agreed or strongly agreed that the time taken by REA to make a pre-payment was appropriate. Satisfaction levels were slightly lower for interim and final payments (91%, or 248 of 272, and 89%, or 80 of 90). As regards the experts, 94% (2 996 of 3 161) agreed or strongly agreed that the time taken by REA to make payments was satisfactory. Reimbursement for in-person evaluation trips to Brussels was also rated very highly, with 95% (856 of 892) of experts agreeing or strongly agreeing that the process was straightforward.

Remote assessment of proposals was highly rated by 89% of experts (2 831 of 3 172), who agreed or strongly agreed that this process was efficient. However, most respondents (72%, or 2 302 of 3 207) had only experienced remote evaluation and not face-to-face evaluation, and consequently did not have a frame of reference for comparison. Free responses revealed that some experts who had experienced both face-to-face and remote evaluation considered the remote version to be less efficient.

Beneficiaries were generally satisfied with reporting requirements. 82% (421 of 512) agreed or strongly agreed that project reporting requirements were reasonable given the scale of the project, and 88% (420 of 478) agreed or strongly agreed that reporting requirements remained stable over the course of the project. The clarity of reporting requirements was rated slightly lower, with only 78% (392 of 499) of beneficiaries agreeing or strongly agreeing that instructions given on reporting requirements were clear and user-friendly. As regards monitoring, results were similar, with 79% (407 of 514) of beneficiaries agreeing that the monitoring process was clear. Overall, 85% (442 of 520) of beneficiaries agreed or strongly agreed that overall project implementation was straightforward.

Beneficiaries indicated satisfaction with the time taken to evaluate their proposals, with 88% (485 of 549) agreeing or strongly agreeing that the time taken was appropriate. Time to grant was rated similarly, at 87% (477 of 550). However, only 76% (2 434 of 3 194) of experts agreed or strongly agreed that they had enough time to review proposals. This was reiterated by experts in their free responses.

Regarding experts’ payments, only 68% (2 184 of 3 189) agreed or strongly agreed that the payment they received matched the effort spent on their task. Low levels of remuneration were mentioned over 160 times (out of 3 479 responses) in the experts’ free responses, which focused on three main problems with the rate: first, the effect of inflation was said to have eroded the real value of the remuneration over the last decade. Second, many experts mentioned that the effort required to evaluate a proposal well did not match the hours stated in the contract, and therefore, when the actual number of hours worked was taken into account, the total payment received was inadequate. Experts suggested that they spent between 50 and 100% more time evaluating proposals than was specified in the contract. This concern will be less relevant from 2024, when new contracts will no longer mention time but only the monetary value of each task. Third, differences in the cost of living and tax requirements led to different perceptions of the value of the remuneration received.

IT tools were the area rated by applicants as having the greatest scope for further simplification, with over 90% (448 of 495) of beneficiaries agreeing or strongly agreeing that IT tools could be further improved. However, 72% (416 of 580) of beneficiaries and 70% (446 of 635) of unsuccessful applicants also agreed or strongly agreed that they found the IT tool for the application process to be user-friendly.

2.2 Summarised results of the interview programme

The interview programme was designed to embrace the views of all key stakeholders (Commission, Agency, beneficiaries, unsuccessful applicants and external experts). It was designed in three stages: scoping, main phase and follow-up interviews, as described in Annex II above.

The main-phase interviews were conducted using a standardised questionnaire, which included questions on the evaluation criteria and was adjusted to the experience of individual stakeholders. Follow-up interviews were conducted based on questionnaires tailored to the candidate’s area of expertise in the case of the lead-parent Directorate-General and REA, and on the responses to the free-text questions in the survey in the case of applicants and experts.

A total of 51 interviews was conducted, summarised below:

-18 interviews (23 interviewees) with representatives of the parent Directorates-General

-15 interviews (21 interviewees) with representatives of the Agency

-2 interviews (2 interviewees) with beneficiaries

-2 interviews (2 interviewees) with unsuccessful applicants

-3 interviews (3 interviewees) with experts.

There was an overlap between experts and applicants. Two of the three experts had also been applicants and two of the four applicants had also been experts. A summary of the findings from the interview data are presented below. 

Effectiveness

Overall, the key stakeholders interviewed considered that REA was highly effective in all its duties.

Effective project implementation was considered to be one of the REA’s strengths by the stakeholders interviewed. The Commission and REA staff were impressed by the number and quality of projects implemented during the previous framework programme, as well as the transition to the new framework. There was no common view as regards the size of projects: Commission staff suggested that smaller projects would better suit their needs, while REA staff suggested that it would be more efficient to implement larger projects. On the client side, beneficiaries who had worked on multiple projects stated that effectiveness depended on the individual project officer assigned to the project.

As regards how the experts learned about the opportunity to work with the REA, all three experts interviewed had been invited directly. They suggested that the REA use universities or Info days for recruitment. Considering that these methods are already used, and that a limited number of experts cited universities (28%) and Info days (2%) in the survey, there is potential for broadening recruitment reach in this way.

Within the evaluation process, remote evaluation was raised in the experts’ free responses. In this respect, two experts interviewed expressed difficulty with the blog interface specifically, saying that it was hard to keep track of new comments. Another expert interviewed expressed a sense of distance from fellow evaluators, which led to difficulty forming a consensus with them. It was noted that, whereas in-person consensus meetings offered experts an opportunity to network with researchers around Europe, remote consensus offered no such benefit, possibly reducing the incentive for some to become experts. It was suggested that video meetings between experts and less anonymity in the profiles would help experts to understand one another better and thus increase both the quality of the evaluations and the added value for experts.

As regards feedback on the applications and transparency, applicants who had also worked as experts felt satisfied with the transparency of the evaluation process.

Applicants and experts interviewed found REA staff very cooperative.

Commission and REA staff indicated mixed feelings about the formal and informal channels for communication between them, although many staff (10) expressed high levels of satisfaction. Job shadowing was considered a successful way of exchanging knowledge and should be further used. Fewer interviewees (7) expressed the need for greater flexibility in informal communication channels.

Efficiency

The Commission staff interviewed were unanimously positive about REA’s efficiency in its programme management duties. REA staff’s enthusiasm for taking on new challenges, their flexibility and active communication were listed among the qualities that contributed to REA’s particularly high levels of efficiency.

As confirmed by the survey, the applicants interviewed (4) were generally positive about the application procedure; however, all of them experienced difficulties when applying. Two relied heavily on outside guidance to complete the application process. Of those two, one mentioned that consortia able to hire external consultants to help with the application would have an advantage due to the complexity of the process. A third interviewee mentioned that the complexity of the application process increased with the number of partners involved. The last interviewee also mentioned that adding new partners to the proposal could take time and might be an area for further simplification.

Reporting requirements and deadlines were rated positively by the beneficiaries interviewed.

As confirmed by the survey data, applicants and experts interviewed were satisfied with the efficiency of the payment process. The interviewees corroborated the survey findings on some experts’ dissatisfaction with the amount of remuneration in comparison to the workload, which was perceived as higher than specified in the contract. One expert suggested that the workload was approximately double; this is reiterated by the free responses in the survey. One beneficiary who also worked as an expert (but had not taken the expert survey) confirmed that the workload was much higher than that specified in the contract. Apart from improvements to IT tools (discussed below), experts did not see any scope to reduce the time required to evaluate an individual proposal, but instead suggested that the contracted hours be reassessed.

Different IT tools are used by different stakeholders; therefore, perspectives from each group differ. REA staff were positive about the upgrades made or to be made. The speed at which IT tools were upgraded was noted as the main limitation on efficiency by both Commission and REA staff. Some interviewees noted that maintenance focused primarily on bug fixes rather than overall improvements to the system. Some staff also felt that the outward-facing tools were not as client-centric as they could be, a sentiment reflected by the clients themselves.

In contrast, many applicants and experts expressed difficulty with IT tools. This corresponds to the survey finding that 90% of applicants believed that the IT tools could be further simplified. One beneficiary required a project officer to guide them through the entire process because it would otherwise have been too overwhelming. Other applicants said that it took time to acquaint themselves with the interface and that it was not intuitive. One beneficiary suggested the tool could be a barrier to new researchers, who may not have the time to learn to use the application tool. Some experts found that the remote evaluation tool hampered their efficiency. It was difficult to find the newest comments on the blogs and notifications of new activity were difficult to track.

Improvements suggested for IT tools:

·Both applicants and experts suggested that messages in their IT interfaces could be ordered from newest at the top to oldest at the bottom for ease of use.

·One applicant who had also worked as an expert suggested integrating the expert profile database into the ORCHID researcher database 24 to reduce the time needed for new experts to create their profiles.

·One interviewee mentioned that when reapplying, applicants had to re-enter all of the partners’ individual details, and that it would be helpful if that information could be automatically saved from the last application. Another interviewee also cited the initial part of the application, when applicants entered all the partners’ details, as a potential area for further streamlining.

Coherence

Both REA and Commission staff interviewed were unanimously positive about the delineation of responsibilities, both as set out in the memorandum of understanding and as understood through the day-to-day communications between them. REA’s ability to be flexible was cited by both Commission and REA staff as a factor enabling the Agency to function well in its role, and a feeling of mutual trust was often mentioned.

Interviewees from the Commission praised REA staff’s proactive attitude, who tend to lead the way in developing improvements for the whole community.

A few interviewees from the Commission mentioned that REA sometimes lacked a broad overview of relevant issues, particularly for feedback to policy purposes but also for work programmes, which was inevitable due to the nature of the executive agency model. Those Commission staff saw this as a challenge for them to tackle, rather than a failing on the part of REA.

On horizonal services, REA does an excellent job of managing these tasks, and there were no complaints among the 44 REA and Commission staff interviewed. However, questions were raised about the extent to which support services align with REA’s broader research-focused portfolio, as well as concern that an economy of scale may need to be re-evaluated in the future as programmes become larger.

On feedback to policy, interviewees expressed a variety of reactions to the changes that feedback to policy underwent during the evaluation period. Some appreciated the increase in structure, while others found that they missed the informal nature of the exchange. Interviewees were united in viewing policy as an ongoing process of improvement.

2.3Comparison of the results of consultation activities

The table below presents the key results of each consultation activity, organised by evaluation criteria, as well as the level of consistency, complementarity and contradiction of results across consultation activities. Overall, the consultation activities were very consistent with one another and revealed many complementary insights.

Summary of key findings

Evaluation criterion

Survey

Interview programme

Consistency of results across consultation activities

Complementarity of results across consultation activities

Contradiction of results across consultation activities

Effectiveness

Applicants were very satisfied with the application process. Experts were generally satisfied. Remote assessment of proposals was rated highly by experts; however, some experts who had experienced both in-person and remote evaluation were less satisfied with the efficiency of the remote version.

Some external communication channels, for example Info days for expert recruitment, were not being used to the full extent.

All stakeholders were generally satisfied. There was a slight disagreement between REA and Commission staff about project size. Some experts expressed dissatisfaction about the effectiveness of remote evaluation. Some experts felt that recruitment activities had further potential.

High

High

Low

Efficiency

Project reporting requirements, payment processes and help services were all rated highly.

IT tools, some aspects of remote evaluations and the time taken to carry out evaluations received less positive ratings. Some experts considered the remuneration to be too low compared to the effort.

Programme management and reporting requirements were rated highly. Experts considered the workload to be higher than specified in the contract. IT tools were rated less favourably by experts and applicants than by internal stakeholders.

High

High

Low

Coherence

NA

Interviewees reported very high satisfaction with REA’s cross-agency leadership. The delineation of responsibilities was considered to be clear. There were mixed opinions about feedback to policy and the general broad overview.

Medium

High

Medium

(1)      Council Regulation (EC) No 58/2003 of 19 December 2002 laying down the statute for executive agencies to be entrusted with certain tasks in the management of Community programmes, OJ L11, 16.1. 2003, p.1 http://data.europa.eu/eli/reg/2003/58/oj.
(2)      Commission Implementing Decision 2013/778/EU of 13 December 2013 establishing the Research Executive Agency and repealing Decision 2008/46/EC, http://data.europa.eu/eli/dec_impl/2013/778/oj.
(3)      Commission Decision C(2013)9418 on delegating powers to the Research Executive Agency with a view to performance of tasks linked to the implementation of Union programmes in the field of research and innovation comprising, in particular, implementation of appropriations entered in the general budget of the Union.
(4)      Commission’s Better Regulation guidelines SWD (2021) 305 final and toolbox .
(5)      The Better Regulation guidelines were applied in a proportionate way. There was no need to assess the evaluation criterions of ‘EU value added’ and of ‘relevance’ as REA carries out tasks, which the Commission has transferred to it and these criterions are assessed when evaluating the programmes.
(6)      Cost-Benefit Analysis for the delegation of certain tasks regarding the implementation of Union Programmes 2014 2020 to the agencies - Final report for the Commission of 19 August 2013.
(7)      The cost-benefit analysis should include the following: identification of the tasks justifying outsourcing, a cost-benefit analysis which includes the costs of coordination and checks, the impact on human resources, possible savings within the general budgetary framework of the EU, efficiency and flexibility in the implementation of outsourced tasks, simplification of the procedures used, proximity of outsourced activities to final beneficiaries, visibility of the EU as promoter of the EU programme concerned and the need to maintain an adequate level of know-how inside the Commission.
(8)      Appendix II of the Guidelines for the establishment and operation of executive agencies financed from the Union budget (C(2014) 9109 from 2 December 2014..
(9)      Study supporting the evaluation of REA by Public Policy Management Institute Consortium formed by Public Policy Management Institute and IDEA Consult.https://op.europa.eu/en/publication-detail/-/publication/c9645710-e5bd-11ee-8b2b-01aa75ed71a1/language-en/format-PDF/source-310469329.
(10)      Or the first level of transactions via indirect management.
(11)      Validation services to joint undertakings are provided by REA within the framework of Horizon 2020 support services.
(12)      The Common Support Centre is a directorate in the Commission’s Directorate-General for Research and Innovation. It provides services in the areas of legal support, ex post audit, IT systems and operations, business processes and programme information and data to the whole research Directorates-General family, executive agencies and joint undertakings implementing Horizon 2020. It was renamed the Common Implementation Centre under the 2021-2027 multiannual financial framework.
(13)      Currently filled.
(14)      Currently filled.
(15)      Estimates as per both the 2013 ex ante cost-benefit analysis and the 2015 cost-benefit analysis for the delegation of validation of legal entities and preparation of legal entities’ viability assessment.
(16)      Staff who were mainly responsible for financial transactions, audits and other types of support.
(17)      AGILE was designed as a flexible tool through which the REA could provide knowledge and information to its parent Directorates-General quickly in the form of a factsheet overview.
(18)      Directorate-General for Defence, Industry and Space replaced Directorate-General for Internal Market, Industry, Entrepreneurship and SMEs as from 01 January 2020
(19)       https://op.europa.eu/en/publication-detail/-/publication/c9645710-e5bd-11ee-8b2b-01aa75ed71a1/language-en/format-PDF/source-310469329
(20)       https://op.europa.eu/en/publication-detail/-/publication/c9645710-e5bd-11ee-8b2b-01aa75ed71a1/language-en/format-PDF/source-310469329  
(21)      Additional resources allocated under amendment of the Specific Financial Statement are also considered.
(22)      Including the cost of coordination and monitoring by the Commission and the costs of REA covered from EEA/EFTA and third-country contributions.
(23)      The number of staff was reduced in the executive agency scenario; however, it was not modified in the in-house scenario in the specific financial statement.
(24)      The ORCHID researcher database, also known as ORCID, is a unique identifier system that provides researchers with a persistent digital ID to distinguish their work and connect their contributions across institutions and publications.