Debate: How to rethink evaluation in higher education

International university rankings are now a reality of globalization, providing a different perspective from the historical reputation of institutions or the evaluation reports produced on them. They are based on purely quantitative elements of comparative performance, which need to be interpreted in their own right: what data are used? What are the indicators? What are the calculation algorithms?

Michel Robert, University of Montpellier

For example, the Shanghai ranking only covers the field of research, obscuring other fundamental missions of universities: the transmission of knowledge, the awarding of degrees and professional integration.

Important for our international reputation, these rankings are often out of touch with the information needs of citizens, who, with limited budgetary resources, are looking first and foremost for the best solutions in their area for educating their children. In concrete terms, they are more interested in a particular bachelor's, IUT, engineering or master's degree course than in the international recognition of the university establishment.

We are also well aware that, in France, behind the word "university" are ecosystems, often with an undeniable contribution from research organizations.

Complex environment

The mention of "evaluation" in higher education and research quickly leads to tensions linked to our history and practices, on subjects such as student orientation, selective courses of study, tuition fees and academic freedom. In particular, we need to distinguish between institutional expertise, carried out by a committee of peers, and control, inspection or audit.

The current debates on the multi-year research programming law are a good illustration of the question of the organization and usefulness of institutional evaluation, which interests us in this article.

There are many topics for discussion: what is the place of evaluation, and what is its usefulness? Is it accepted by the communities being evaluated? What impact does it have? What practices can be envisaged to better communicate the results and ensure that they appear perfectly legible to all stakeholders, especially future students?

Peer review of a public higher education and research entity (university, school, laboratory, research organization, etc.) is organized around three players:

• the entity being assessed ;
• expert committee (peers) ;
• the organizer: Hcéres (Haut Conseil de l'évaluation de la recherche et de l'enseignement supérieur) or Cti (Commission des titres d'ingénieur) or other foreign evaluation agencies.

The context in which an assessment is organized is complex and alternates several parameters. The relationship between evaluators and those being evaluated must be based on trust and the absence of vested interests. The distance between evaluation and decision-making (awarding of a label, allocation of resources, etc.) is essential. The current health crisis in particular highlights the importance of scientific integrity issues in research, as well as in the training of doctoral candidates and students.

The sole aim of evaluation cannot be to sanction and regulate the system, as this could lead to bias in the adaptation of the players involved. It must be conceived with a threefold aim: to support the development of the entities being evaluated, to help the supervisory authorities make decisions, and to inform the public and users of higher education.

Current issues

The peer review procedures set up by Hcéres, in application of the current law on higher education and research, involve specifying criteria and observing reality (self-evaluation report, indicators, committee visit), all of which are essential steps in expressing a judgment (expert committee report). These procedures are also part of a quality and continuous improvement process formalized at European level as a result of the Bologna process.

The current obligation to evaluate all courses and research units nevertheless raises the question of the effectiveness of the evaluation system. Given the burden of "industrializing" a very large number of assessments (several hundred for a university every five years), the system does not allow for investigations likely to yield the greatest added value for a given establishment.

What's more, the institutional evaluation carried out by Hcéres is concentrated on around fifty establishments each year, whereas other establishments, for example private ones not under contract with the state, or special establishments such as ENA, have never been evaluated by Hcéres.

The definition of the scope of evaluation, i.e. the components to be evaluated within a university (degrees, faculties, schools, institutes, UFRs, teaching departments, research departments, laboratories, research teams) should not be set in stone, as the autonomy of establishments has led to different organizational models.

We therefore need to define a flexible framework that allows the diversity of our establishments to express their specificities and strategies, and avoid forcing them into a stereotype. This is why we need to update the law.

Possible developments

However, scientific and educational life, creativity and student success cannot be limited to standardized, fixed indicators or rankings. Taking risks and detecting "weak signals" in innovation, for example, are fundamental to progress.

How can we develop performance measurement that is not prescriptive, that can be adapted to the diversity of the people, establishments and ecosystems involved, and that stimulates the dynamics of the establishment? In particular, the aim is to assess the levers used by establishments to improve the efficiency of their actions and their performance.

A global change in operating methods cannot be summed up by the isolated action of an evaluation agency to compare entities, all the more so as, in the past, the rating of laboratories has highlighted the limitations of such an approach (and its rejection), if only because of the limited territorial scope of the comparisons made.

We could therefore consider discussing a more global approach involving not only evaluation agencies, but also establishments and their supervisory ministries, by integrating into the process the acceptance of the communities concerned within the establishments. Let's take a look at a few ideas:

• at the level of training and student success, by distinguishing between the Bachelor's level (and the issues linked to the law on student orientation and success) and the Master's-Doctorate level (and the issues linked to research), and by using public data certified by establishments on student tracking, updated annually at national level, as Cti currently does for engineering schools;
• at research level, by distinguishing between the contribution of laboratories to an establishment's strategy, supplemented by national analyses by major disciplinary field (involvement of the Observatoire des Sciences et Technique, coordinated evaluation of research teams within the same perimeter, national disciplinary summaries) to analyze France's position.

To preserve a climate of trust, we therefore propose gradual changes to current evaluation methods, rather than an abrupt and radical transformation, which carries the risk of rejection, by placing - as can be seen in other European countries - the establishment at the center of the evaluation process, as the main player in its own internal and external evaluation.

These structural and structuring reflections are all the more topical today as they take place in a context profoundly impacted by the climate transition and the health crisis which, by forcing a shift to a society of physical distance, will necessarily lead us to change course.

Michel Robert, Professor of Microelectronics, University of Montpellier

This article is republished from The Conversation under a Creative Commons license. Read theoriginal article.