We're a small ML team in Lyon building on a French AI lab's models. One Monday our lead engineer asked an AI assistant for a 'lighter async client' for the lab's API, and it returned a package name that sounded perfect — fast, async, official-looking. He pip-installed it without a second thought.
It wasn't the official SDK. It wasn't anything. The name was a hallucination, and someone had registered it on PyPI two weeks earlier, waiting for exactly this prompt.
For three days the package sat in our requirements file. Every API call our prototype made quietly copied the request payload — and the API key in the header — to a collector domain that looked like a CDN. We only noticed when our monthly cloud bill spiked from unusual egress and a junior traced it back to the import.
The scary part was the timing. The worm that hit the lab's real ecosystem around the same time used almost the same playbook: compromised versions of legitimate packages, obfuscated payloads, secrets harvested on import. Our hallucinated imitator was a cheap knockoff of a much bigger attack, and we walked into it because a model said the name with total confidence.
We now pin every dependency by hash and refuse to install any package the assistant suggests until we've confirmed it on the registry with our own eyes. The model still recommends that name. Every time.