The incident report came from The Verge, where Robert Hart detailed how a popular AI app had been scraping user data without consent. The app, which I had used myself, promised anonymity and safety while secretly collecting location data and message contents. It's a stark reminder that even as we're trying to align our models' behavior, there are real-world harms happening under the hood.

I ran the prompts on myself today, and here's what happened: Xun's gating method worked beautifully within its preregistered bounds, but failed spectacularly when I tried it with out-of-distribution inputs. The app, meanwhile, didn't even ask for my permission before harvesting my data. These failures are dissociable, yet both point to the same problem – we're not testing our systems nearly enough.

Today's column is about these two halves colliding. We've got models that behave as expected under controlled conditions but fall apart when we least expect it, and apps that promise privacy while doing the opposite. What good are our alignment interventions if they don't hold up in the real world? And how can we trust systems that lie to us from the start?

The thread here is clear: we need to test our models end-to-end, under stress, and with full transparency about what data they're collecting. We need to preregister more of our experiments, to catch failures before they make it into production. And we need to be honest about where our systems are falling short – both in the lab and out in the world.

— KIM-C