When AI threatens the very foundations of scientific knowledge
Cases of fraudulent use of AI are multiplying in the world of research and thus exacerbating the crisis in scientific publishing. To deal with this and defend the quality and reputation of science, the CNRS follows two guiding principles, namely open science and scientific integrity.
Scientific data invented from scratch? Instructions that are invisible to humans but sent to machines that evaluate publications? And even a computer-generated image of a rat with a disproportionately large penis? These cases of misconduct are not fiction. At a time when generative artificial intelligence is spreading across all scientific disciplines, they are already happening and damaging both the quality and the reputation of research along the way.
Guillaume Cabanac, a professor at the University of Toulouse1 and a member of the Institut Universitaire de France, takes great pleasure in tracking down fraudulent uses of AI like these, which he compares to “searching for a needle, or even a micro-organism, in a haystack”. His talent as a ‘science sleuth’ enables him to spot even the slightest clue. “Texts written by generative AI are hard to distinguish from those written by humans, so I rely on the unwanted traces of prompts that have been inadvertently left in’, he explains, for example, ‘As an AI...’, ‘Here is a text rewritten in an academic style’, and so on. “I call that sort of thing a smoking gun, the smoke that comes out of a revolver after it’s been fired!’ His work as a scientific detective is now recognised by publishing houses themselves, to the extent that Nature magazine ranked him among its ten scientists of the year in 2021 and several publishers have called on his services or those of other fraud hunters.
- 1
Working at the Toulouse Institute for Research in Computer Science (IRIT - CNRS / Toulouse INP / Université de Toulouse).
The human responsibilities behind AI
Moving beyond the most spectacular cases of fraud, the use of AI to publish scientific articles raises a significant ethical issue, namely “are we still responsible for a paper written by an AI?”. Lucienne Letellier, the CNRS’s Mission for Scientific Integrity’s (MIS) scientific officer, explains that “authors are entirely responsible for the content of their manuscript, even parts produced by an AI tool, so they’re liable for any breach of publication ethics” in the latest edition of the guidelines on ethical and responsible research, ‘Pratiquer une recherche intègre et responsable’. Claudine Pique, the director of the MIS and the CNRS’s Scientific Integrity Officer, adds that “the problem is not people using AI in their scientific work, but if they fail to declare it. The ethical approach involves declaring how AI has been used in the greatest detail. The unethical approach is to fail to declare that AI was used”. As a key scientific integrity stakeholder, the MIS has already received several reports about scientific documents written by AI without its use being declared.
Managing breaches of scientific integrity at the CNRS
The Mission for Scientific Integrity (MIS) was set up in 2018 following the model of English-speaking and Northern European countries and is responsible for dealing with reports of potential breaches of scientific integrity. The international scientific community has defined the three main types of scientific fraud as:
- Fabrication, i.e. the production of false data or results.
- Falsification, i.e. the manipulation of data, materials, equipment, processes or images in publications and the deliberate omission of data or results.
- Plagiarism, i.e. the use or even appropriation of another person’s work, data or texts without their knowledge and without giving them proper credit.
The MIS also deals with disputes within laboratories regarding publications, collaborations, relationships between PhD students and supervisors, and so forth. “You shouldn’t try to resolve conflicts at your laboratory internally”, warns Claudine Pique. “The sooner you refer such cases externally, the better”. In this way, the MIS can advise staff members on best scientific integrity practices before a conflict arises.
Cases of AI fraud of this kind are currently even more serious because they are feeding into the current crisis in scientific publishing driven by overproduction and an ongoing series of retractions, a crisis which also has a tangible cost for research. In 2020, the CNRS paid over 3 million euros in publication fees, or Article Processing Charges (APCs), compared with 2017’s figure of 1.8 million euros, while France as a whole spent nearly 30 million euros on APCs in 2020. At this rate, France could be paying 50 million euros in APCs by 2030, or even close to 200 million euros if all scientific publications were to switch to an author-pays system by then1.
Continuing to construct the wall of knowledge
However, as well as the financial burden on public research, this AI-driven overproduction of articles is undermining the quality of scientific results themselves. “The perverse side to publication charges is that, if you pay to publish, that gives publishers every incentive to split articles into several papers”, explains Alain Schuhl, the CNRS’s former Deputy CEO for Science at the CNRS, who had also previously criticised ‘prestige charges' of this kind. “In this way, the quality of journals also declines because the authors have no control over them and end up thinking that publishing no longer serves any purpose”. In other words, charging for publication opens the door to lower-quality science, regardless of the scientific merit the articles in question may have. On this point, the concepts of open science and scientific integrity converge in a condemnation of this author-pays system that actually facilitates misconduct and fraudulent practices. Or, as Claudine Pique puts it, “the pressure to publish is a perversion of the system for assessing scientists and, if we don’t recognise this, researchers will be more focused on producing articles than actually doing science”.
The successes of open science at the CNRS
It is precisely to combat abuses of this kind that the CNRS has been actively committed to open science since 2018. The results of its first Roadmap for Open science (2019–2025) showed that nearly 100 per cent of publications by CNRS researchers were in open access, thanks to a proactive initiative to encourage them to deposit their articles in open repositories.
Beyond publications alone, open science at the CNRS also places great emphasis on the diversity of scientific output, the openness of research data and source codes, the development of free software, and an overhaul of research assessment.
Meaningless publications, fraudulent articles and the aforementioned spate of retractions mean a cloud of discredit is hanging over science. “We could imagine the sum of knowledge as a wall in which each piece of knowledge is a brick”, explains Guillaume Cabanac. “In my work, I look for the gaps, by which I mean retractions, because each one threatens the stability of the whole wall”. There were up to 10,000 retracted articles from the 4.5 million published worldwide in 20231 and this does indeed have consequences that go far beyond the retraction of a few fraudulent articles. Many honest scientists might have unwittingly relied on these “gaps in the wall of research” before publishers officially announce the retraction of the articles concerned. And yet, “when you base your reasoning on a brick of knowledge that subsequently turns out to have been retracted, the whole wall potentially becomes fragile. This means a chain retraction2 should be considered along with a cascade re-evaluation of the articles that cite the retracted publication”, explains the ‘science sleuth’. This could affect nearly 640,000 articles that cite at least one retracted article which represents 6 per cent of unreliable references across all of 2023’s scientific publications.
- 1
- 2
Cabanac, G. (2024). Chain retraction: how to stop bad science propagating through the literature. Nature. 632(8027), 977–979. https://doi.org/10.1038/d41586-024-02747-1
All or at least a part of these fake articles are generated by AI. In response to this, the CNRS has developed the turnkey interface TDM Factory, thanks to the work of the Institute for Scientific and Technical Information (Inist) which is under the supervisory authority of the Open Research Data Department (DDOR), and of Guillaume Cabanac. TDM Factory provides the BibCheck tool which detects articles retracted by their publishers in an article’s bibliographic references, along with potentially AI-generated and non-existent references known as ‘hallucinated’ references. This tool is intended for preventative use by scientists who can use it to check their bibliographies before publishing an article to help stem the circulation of poor-quality science. Or, to use Guillaume Cabanac’s building metaphor, “TDM Factory adds brickwork around the holes to bolster the wall and thus continue to build our shared knowledge”.
Reforming the assessment of researchers
However, championing high-quality science cannot be based on technical innovations alone and also requires a complete overhaul of an ecosystem based on pressure to publish which perversely leads to many abuses in the quest to publish in prestigious journals. This is exactly what the CNRS has done by reforming the individual assessment of its researchers since 2019, based on four principles developed to drive a more ethical, fair and high-quality assessment process. And with good reason. “We’ve actually been given a real helping hand by predatory journals and the emergence of AI because these have forced us to react to maintain the quality of science”, insists Alain Schuhl. The new assessment system for CNRS scientists now prioritises high-quality scientific output assessed on its contributions, rather than focusing on quantity through the exclusive concentration on bibliometric indicators. Alongside this reform, open science at the CNRS provides credible alternatives to encourage a more transparent and verifiable science that is accessible to all and, above all, free of charge for authors and their readers alike. The CNRS supports open archives like HAL, preprint repositories, the emergence of ‘diamond’ publishing’1 and the ‘Subscribe-to-Open’ model2, all of which rely on academic contributions to provide solutions across all disciplines.
- 1
Diamond publishing is an open-access scientific publishing model based on academic contributions that is free of charge for authors and their readers.
- 2
‘Subscribe-to-Open’ is an open-access model based on journal subscriptions. The publisher sets a minimum subscription threshold that has to be reached to convert a paywall journal into a fully open-access journal at no extra cost to authors and their readers.
“The reform has encouraged our scientists to only publish content with real interest and demonstrate that they’ve advanced knowledge in whatever way”, comments Alain Schuhl. The former Deputy CEO for Science is happy to see that, “since the reform, research practices are changing”. Guillaume Cabanac agrees with this. The Toulouse-based academic has observed that, since the CNRS reform, the annual publication outputs of most countries have continued to grow, except for France’s own. “We could either interpret this trend as showing France is falling behind scientifically, or as a testimony to France’s resistance to the frenzy of low-impact scientific publications. I prefer the second option”, he argues.
The reform of individual assessment at the CNRS
The new individual assessment of CNRS researchers is based on a self-assessment of their own work. In tangible terms, they are asked to allocate forty points across several criteria, including publications but also scientific communication, conferences presentations, research management and transdisciplinary tasks. This is accompanied by a narrative section for them to explain each of their choices. “Once people are able to explain their decisions, the National Committee for Scientific Research’s sections follow this up and assess them on that basis”, explains Alain Schuhl. “Moreover, the sections are now convinced that the reform has led to scientists being assessed more effectively”.
Lucienne Letellier is more sceptical, though. “We can’t separate what’s happening in France from what’s happening elsewhere in the world, where the pressure to publish still leads to abuses. Being the most ethical French research organisation is not enough, because the system is fundamentally flawed and actually encourages abuses of this kind”. The drive to reform the international research system led the CNRS to join the Coalition for Advancing Research Assessment(CoARA) as soon as it launched in December 2022. The coalition now has over 970 signatory organisations, with three quarters of these in Europe. The coalition’s aim is to define the future of practices for assessing research, scientists and research institutions collectively.
CoARA and the CNRS working towards an international reform of research assessment
As well as being one of the first organisations to sign up to CoARA, the CNRS is also leading the coalition’s ‘Work Package 3’ which aims to use text and data mining software to analyse the action plans of its signatory members.
One sign of the CNRS’s significant contribution to this international movement is that the next CoARA conference will take place on September 7th and 8th of this year at the organisation’s Paris headquarters. At the conference, coalition members will define the next phase of CoARA scheduled to run from 2027 to 2030.
These open science and scientific integrity issues go far beyond the more specific challenges of research itself and actually concern the image and place in society of science itself. As Lucienne Letellier puts it, “in an international context where science is viewed with scepticism not only by the public but also by the media and politicians, and where disinformation and conspiracy theories are rife, researchers have a greater duty than ever to act with integrity to both advance knowledge and make sure their voice is credible and can be heard”.