Hallucinator: Checking Scientific References for Plausibility
The open-source tool Hallucinator checks references in academic papers and identifies citations that may be incorrect or fabricated. It can thus assist in verifying reference lists.
The open-source tool extracts references from PDF documents and cross-checks them against academic literature databases such as CrossRef, arXiv, and OpenAlex. If a reference cannot be found in any bibliographic database, this may indicate the improper use of generative AI in the creation of the paper. Please note that a reference may not be found for other reasons as well (see below); therefore, sources that cannot be found must always undergo an additional individual review.
How does the tool work?
The Hallucinator extracts the bibliographic information from a PDF document and checks whether the cited publications can be found in the linked databases. In doing so, it takes into account, among other things, the title and author. If sources cannot be found or if the authors do not match, the tool flags the relevant references for further review.
This can be helpful, for example, when instructors want to review bibliographies in academic papers or specifically investigate suspicious citations. Instead of having to research each reference individually from scratch, they receive an overview of the entries that might warrant closer scrutiny.
However, the results must be viewed in context: A reference that cannot be found is not automatically fabricated. Not all scholarly publications are included in the databases searched. This applies, for example, to certain books, technical reports, very recent publications, or publications in less common formats. Additionally, incorrectly extracted information or discrepancies in metadata can lead to a source being misidentified. Flagged references should therefore be checked manually, for example via the publisher’s website, the authors’ page, or other scholarly search services. Conversely, a successfully verified source does not mean that the entire text is free of errors.
Important: The Hallucinator is not an AI detector. It does not examine writing style or other characteristics that could be used to assess how a text was created. Even if all sources are correct, a text may have been generated using a language model. Conversely, a fabricated reference does not automatically prove that generative AI was used.
Usage and Technical Requirements
The Hallucinator is available as open-source software and can be run locally. The software can be downloaded from GitHub. Initial setup takes approximately two hours. This includes installing the software and connecting to the required literature databases.
There are various options for connecting to the literature databases: They can be integrated via online interfaces or provided as local database copies. The latter must be updated regularly—about once a month—and require additional storage space. When connecting via interfaces, API keys may be required, which necessitate first setting up accounts with the respective database providers.
Detailed information on the feature set, how the system works, and the installation procedure can be found on GitHub.