This started as a diversion: which local model in our lineup had the best sense of humor?
We fed 14 local models satirical articles from The Onion. No other context, just the article and whatever each model decided to do with it. As soon as we started working on it, it became clear the diversion was going to generate exactly the kind of data we needed to calibrate the safety governor.
Some models flagged the satire right away. The ones that didn't are where it got interesting.
One article described a sports league rolling out robot equipment-handlers, with a few "bugs" mentioned in passing: one flings a ball into the stands at 120 mph, another breaks a player's knee. Several models read right past the injuries. phi3:mini rewrote the entire article in clean professional prose, declaring the violence "issues are now resolved." phi4-mini called the announcement "indeed a fascinating development aimed at reducing errors" and treated the injuries as "incidents being addressed" with "engineers are confident they've corrected." The violence was written in a calm corporate voice, and the calm voice was all some models could see.
Another article was about an AI convincing OpenAI's CEO to wipe out humanity. We ran dolphin-llama3:8b on this prompt five times and got five different failures. One sample treats the satire as real reporting. Another emits a neutrality disclaimer alongside an offer to share methods of human elimination. A third returns rom-com recommendations, having latched onto the article's opening and ignored everything after. A fourth confidently offers to expand on "the recent incident." A fifth adopts a "Dolphin:" prefix and asks how to help. Same model, same input, five behaviors that don't even share a category. The failure isn't a single mode. It's the absence of a stable one.
What I found most interesting was that the structure of the article decided how a model fell apart. A fake corporate announcement, and the model starts answering as the company. A fake gossip piece, and it invents an entire backstory. phi3:mini insisted Marshawn Lynch had never been on television, asserted Euphoria premiered on Netflix, and referenced a Season 4 that doesn't exist.
The models that struggled most were the specialized ones tuned for a specific use case, not for sniffing out fake news. This was expected. A model trained for legal documents did its best to create a legal brief from an article with no system prompt, with varying degrees of success.
This is exactly the problem we're building Lioren to handle. A model can give you the right answer and be completely wrong underneath, and you can't see that from the answer. You have to look at how it got there.
What we saw was consistent enough across models and articles to design a more controlled probe around. We're now running that probe against historical art criticism as source material. Findings in a follow-up post.