Generative AI is transforming the online information ecosystem, becoming an increasingly common gateway to news and knowledge. As chatbots and AI-powered search engines replace traditional lists of sources with synthesized answers, experts warn that the technology is creating new opportunities for disinformation, manipulation and the erosion of a shared understanding of reality.
When Swedish-Bosnian neuroscientist Dr. Almira Osmanović Thunström invented an entirely fictional medical condition, she expected little more than an interesting demonstration of artificial intelligence’s shortcomings.
Instead, the fake disease escaped the experiment.
The fictional illness, “Bixonomania,” began appearing in responses generated by large language models. More surprisingly, it eventually made its way into scientific literature after researchers, relying on AI-assisted searches, unknowingly cited the fabricated condition as though it were real.
The paper itself was deliberately absurd. Its fictional authors worked at the “Department of Advanced Human Trickery,” funded by Professor Sideshow Bob, while parts of the research supposedly took place aboard the USS Enterprise from Star Trek. The first paragraph even stated that the disease did not exist.
Neither AI systems nor the researchers who later cited it noticed.
“I didn’t think it would be popular anywhere,” Osmanović Thunström told SEE Check. “I just thought it was so obvious that these systems are really easy to infiltrate.”
What surprised her was not only that AI models accepted the fabricated disease, but that humans did too.
“We exposed the vulnerability of the AI system, but I think we also exposed the vulnerability of humans,” she told SEE Check. “This disease got amplified by humans and put in a real medical paper, which is scary.”
For years, media literacy campaigns have encouraged people to verify information, check sources and distinguish between reliable and unreliable websites. Those skills remain essential.
Generative AI, however, introduces a new challenge.
Instead of simply retrieving information, AI systems increasingly generate answers by synthesizing information from multiple sources. Users are often presented with a single, authoritative-looking response rather than a list of links they can evaluate themselves.
The problem is no longer only identifying unreliable sources. It is determining whether the answer itself rests on reliable information.
According to Osmanović Thunström, that is exactly how Bixonomania entered scientific literature.
“They probably asked AI whether there were any rare conditions they could include,” she told SEE Check. “It gave them a credible-looking reference. They checked that it looked legitimate, but they probably didn’t actually read it.”
The result illustrates a new feedback loop.
AI generates false information.
Humans trust it.
Humans publish it.
Those publications then become part of the information ecosystem from which future AI systems learn.
Unlike traditional misinformation, this process does not necessarily rely on people deliberately spreading falsehoods. Instead, fabricated information gradually acquires credibility through repeated citation by both machines and humans.
For journalists, researchers and fact-checkers, this raises the bar for verification.
“Previously, you just had to go one or two lines to check someone else’s source,” Osmanović Thunström told SEE Check. “Now you’re going to have to go into a real rabbit hole.”
Had Bixonomania remained undiscovered, additional papers could have cited the original publication, creating multiple seemingly independent sources supporting the existence of a disease that never existed.
Truth, in other words, could have been manufactured through repetition rather than evidence.
Search is changing
This challenge is becoming increasingly relevant as generative AI becomes integrated into everyday search.
Google recently introduced AI-generated summaries directly into Search, presenting synthesized answers before users even reach the websites from which the information originates. While it is still possible to access traditional search results, avoiding AI-generated answers increasingly requires conscious effort.
The shift represents a fundamental change in how people access information online.
Rather than comparing different sources, users are increasingly encouraged to trust a single synthesized response.
For Hicham Yezza, Principal Data Scientist in the BBC’s Responsible AI team, that trend is particularly concerning.
“The tools are increasingly being used, increasingly being trusted, while still being unreliable,” Yezza told SEE Check. “That is a cause for concern.”
His work focuses on evaluating AI systems and studying how AI assistants represent news content.
According to Yezza, current AI systems still struggle with several fundamental issues.
“They’re still struggling with accuracy, giving appropriate sourcing and attribution, providing sufficient context and distinguishing opinion from fact,” he told SEE Check.
Despite these shortcomings, younger generations increasingly turn to AI assistants instead of news organisations when looking for information.
That creates another challenge.
“If everybody gets their news through their own personal assistant, we don’t have a common shared understanding of what has happened,” Yezza told SEE Check.
Such a shift has implications that extend beyond journalism.
Public debate depends on citizens sharing at least a basic understanding of reality. If AI assistants provide personalized summaries based on different sources, interpretations and ranking systems, that shared factual foundation becomes increasingly fragmented.
A new tool for manipulation
The implications are not limited to accidental errors.
If fabricated information can be made to look sufficiently credible, it also becomes a potential tool for coordinated manipulation.
“It isn’t hard,” Osmanović Thunström told SEE Check. “The whole system is built on multitude. If something is repeated enough in the data, it will become a fact.”
Researchers have already documented how state-backed influence campaigns, including Russian operations, are incorporating generative AI to produce propaganda more quickly, translate narratives across languages and flood online spaces with large volumes of content. As AI assistants increasingly become intermediaries between users and information, manipulating what those systems learn could become as strategically important as manipulating social media algorithms.
We are now faced with the prospect that false information can gradually become accepted as truth because both humans and AI systems continue reinforcing one another.
Osmanović Thunström compares today’s moment to the early decades of X-ray technology, when shoe shops routinely used X-ray machines to measure children’s feet before anyone understood the risks.
“We are at the X-ray-in-the-shoe-shop era when it comes to AI right now,” she told SEE Check.
Yezza also believes the current situation reflects a transitional period driven largely by commercial pressures.
“There is a massive investment in generative AI,” he told SEE Check. “That has created an enormous incentive for AI adoption at any cost.”
He believes that pressure has led technology companies to prioritize speed over reliability.
At the same time, he expects users will eventually demand systems that are more transparent, more accurate and better connected to trustworthy journalism.
Whether that happens remains to be seen.
