LLMs hallucinate. In regulatory work, apart from being annoying, it is actually dangerous.
At Qalico, we have been using AI extensively, and we have seen every flavor of hallucination. Here are real examples from QARA work and how to catch them.
Made-up FDA product codes
Ask an LLM for the product code of a specific device type and it will confidently give you one. It often looks right: three letters, plausible category. But check it against the FDA Product Classification Database and it does not exist. Or worse, it exists but maps to a completely different device type.
How to catch it: You can ask LLMs for product codes, just make a habit of double-checking the code in the actual FDA Product Classification Database.
Incorrect regulation references
Ask an LLM to cite a specific MDR article or chapter and it will. Confidently. But the article number might not say what it claims, or the chapter structure might be wrong entirely. Same thing with ISO standards: it will cite clause numbers that do not exist or mix up requirements from different editions.
How to catch it: Never trust a regulation reference without opening the actual document. If the LLM gives you "MDR Article 52" or "ISO 14971 clause 7.4," go read it. This takes 30 seconds and saves you from building an argument on a reference that does not support it.
Fabricated database results
Ask for the latest PMA-cleared device in a specific category and you will get an answer with an actual device that is recent, but not quite the latest. This is because the model's training data has a cutoff date. You think you are getting current information but you are looking at a snapshot from months or even years ago.
How to catch it: Always check directly in the actual database. For FDA, that means MAUDE, 510(k) database, or PMA database directly. If you need up-to-date data, the only reliable source is the database itself.
The general rule
Hallucinations in QARA work follow a pattern: they look structurally correct. The format is right, the terminology is right, the level of detail feels right. That is what makes them hard to spot and dangerous to miss.
Three habits that help:
- Always verify references against primary sources. Every single time.
- Be extra suspicious when the answer comes too easily. Real regulatory work involves ambiguity. If the AI gives you a clean, definitive answer to something you know is complicated, double-check.
- Use AI for the heavy lifting (drafting, comparing, extracting) but keep the verification human.
This is a core principle behind how we built Qalico: AI does the analysis, but traceability and source access are built in so you can always verify.