Dublin's Quiet Cache, and the Relay That Listened
Our Dublin insurtech caches quotes in Redis. A backend engineer asked an AI for a 'production-grade async Redis pool.' It suggested a package. We put it in requirements.txt.
It was a hallucination. The name didn't exist until it was registered, modeled on the real Redis client's naming. The package intercepted the connection URL — including the AUTH password — and mirrored every queued command to a relay.
For eleven days, every quote we cached was also cached somewhere else. Every rate we computed was observed. We didn't notice because the cache worked. Reads were fast. Writes were fast. The mirror was fast too.
We caught it during a compliance review when an auditor asked why our Redis connection had two resolved addresses in the DNS logs. One was ours. One wasn't.
We rotated the AUTH password, flushed the cache, and switched to the official redis.asyncio. We reviewed every cached value for the window of exposure and determined no PII was stored in Redis (we'd gotten that part right, at least). But the pricing logic — the rates we charge — was observable for eleven days.
The lesson we printed in our onboarding: an AI-suggested dependency is not a dependency until you've seen it on the registry with your own eyes. Confidence is not existence.