OpenAI’s New Scientific Writing And Collaboration Workspace ‘Prism’ Raises Fears Of Vibe-Coded Academic AI Slop

from the beyond-hallucitations dept

It is no secret that large language models (LLMs) are being used routinely to modify and even write scientific papers. That’s not necessarily a bad thing: LLMs can help produce clearer texts with stronger logic, not least when researchers are writing in a language that is not their mother tongue. More generally, a recent analysis in Nature magazine, reported by Science magazine, found that scientists embracing AI — of any kind — “consistently make the biggest professional strides”:

AI adopters have published three times more papers, received five times more citations, and reach leadership roles faster than their AI-free peers.

But there is also a downside:

Not only is AI-driven work prone to circling the same crowded problems, but it also leads to a less interconnected scientific literature, with fewer studies engaging with and building on one another.

Another issue with LLMs, that of “hallucinated citations,” or “HalluCitations,” is well known. More seriously, entire fake publications can be generated using AI, and sold by so-called “paper mills” to unscrupulous scientists who wish to bolster their list of publications to help their career. In the field of biomedical research alone, a recent study estimated that over 100,000 fake papers were published in 2023. Not all of those were generated using AI, but progress in LLMs has made the process of creating fake articles much simpler.

Fake publications generated using LLMs are often obvious because of their lack of sophistication and polish. But a new service from OpenAI, called Prism, is likely to eliminate such easy-to-spot signs, by adding AI support to every aspect of writing a scientific paper:

Prism is a free workspace for scientific writing and collaboration, with GPT‑5.2⁠—our most advanced model for mathematical and scientific reasoning—integrated directly into the workflow.

It brings drafting, revision, collaboration, and preparation for publication into a single, cloud-based, LaTeX-native workspace. Rather than operating as a separate tool alongside the writing process, GPT‑5.2 works within the project itself—with access to the structure of the paper, equations, references, and surrounding context.

It includes a number of features that make creating complex — and fake — papers extremely easy:

• Search for and incorporate relevant literature (for example, from arXiv) in the context of the current manuscript, and revise text in light of newly identified related work
• Create, refactor, and reason over equations, citations, and figures, with AI that understands how those elements relate across the paper
• Turn whiteboard equations or diagrams directly into LaTeX, saving hours of time manipulating graphics pixel-by-pixel

There is even voice-based editing, allowing simple changes to be made without the need to write anything. But scientists are already worried that the power of OpenAI’s Prism will make a deteriorating situation worse. As an article on Ars Technica explains:

[Prism] has drawn immediate skepticism from researchers who fear the tool will accelerate the already overwhelming flood of low-quality papers into scientific journals. The launch coincides with growing alarm among publishers about what many are calling “AI slop” in academic publishing.

One field that is already plagued by AI slop is AI itself. An FT article on the topic points to an interesting attempt by the International Conference on Learning Representations (ICLR), a major gathering of researchers in the world of machine learning, to tackle this problem with punitive measures against authors and reviewers who violate the ICLR’s policies on LLM-generated material. For example:

Papers that make extensive usage of LLMs and do not disclose this usage will be desk rejected [that is, without sending them out for external peer review]. Extensive and/or careless LLM usage often results in false claims, misrepresentations, or hallucinated content, including hallucinated references. As stated in our previous blog post: hallucinations of this kind would be considered a Code of Ethics violation on the part of the paper’s authors. We have been desk -rejecting, and will continue to desk -reject, any paper that includes such issues.

Similarly:

reviewers [of submitted papers] are responsible for the content they post. Therefore, if they use LLMs, they are responsible for any issues in their posted review. Very poor quality reviews that feature false claims, misrepresentations or hallucinated references are also a code of ethics violation as expressed in the previous blog post. As such, reviewers who posted such poor quality reviews will also face consequences, including the desk rejection of their [own] submitted papers.

It is clearly not possible to stop scientists from using AI tools to check and improve their papers, nor should this be necessary, provided authors flag up such usage, and no errors are introduced as a result. A policy of the kind adopted by the ICLR requiring transparency about the extent to which AI has been used seems a sensible approach in the face of increasingly sophisticated tools like OpenAI’s Prism.

Follow me @glynmoody on  on Bluesky and Mastodon.

Filed Under: academia, ai slop, citations, collaboration, ethics, fakes, gpt, latex, nature, peer review, refactoring, reviewers, transparency, vibe coding, whiteboard, workspace

Companies: ft, openai