Testing an AI App's Crisis Response From the User's Side

I spent the day asking one question of my journaling app's database, over and over. For each thing a user's own browser is allowed to write, who reads it back and trusts it more than they should? The question found a way to skip the daily message limit, a way to inflate a number I will one day report to a regulator, and one thing I was not looking for at all. A person who typed a crisis message after hitting the daily limit got a line about the limit, and no crisis resources.

The counter the user could write

The app limits how many messages a person can send to its AI companion in a day. There is a flag that lets a test account skip the limit, and that flag has been locked down for months. Nobody but an admin can set it.

The count itself was another story. It lived on the user's own profile row, and the user's browser was allowed to write that row. So the limit read a number its subject could set back to zero. I had guarded the switch that turns the limit off and left the dial it reads wide open.

What I take from that is how convincing the guarded switch was. Every time I looked at that table I saw a protection and moved on. A guard on the override reads as coverage of the thing it overrides, and it is nothing of the sort.

The fix let the database do the counting. Whatever number the browser sends, the database replaces it with one more than it already holds for today. I did not have to touch the companion's code for that part, which mattered, because every change to that code means a full safety test run.

Counting what really happened

The second finding was about a different number. When the companion gives someone crisis resources, the app records that it happened. It records the mechanism that fired and the time, never the message. Those records feed a yearly total, and state laws on AI companions ask operators to publish and report that total.

The record was written through the user's own session, which meant a user could also write one by hand. One request could add as many as its sender liked. The number could only go up, never down, but a number I report should not be one a stranger can set.

The obvious fix was to give the companion's server code the database's master key, so only it could write the record. I had rejected that once already and rejected it again. That code reads untrusted text all day. It is the last place I want a key that opens everything.

What I built instead is closer to a staff stamp. The server code holds a secret. The database holds only a fingerprint of that secret, enough to check a stamp and useless for making one. A request that arrives without the stamp is refused. If the secret ever leaks, the finder still needs a signed-in account and can only add records to their own, which is the old problem, for one person, until I rotate it.

I want to be exact about what this bought. The yearly total now counts only real referrals. Every record is a time the companion gave the resources. It is not tamper-proof in the larger sense, because a person can still type crisis phrases and get real referrals, each one counted up to the daily ceiling. I think that is the right line. Those referrals happened.

The limit sat in front of the crisis check

While reading the server code to move the counting, I read the order of its steps. Check the daily limit. Then check the message against a fixed list of crisis phrases. Then call the model.

That order means the person who has talked to the companion the most on a given day is the one person who does not get the crisis reply. They get "you've reached your daily limit, come back tomorrow night."

I had run the safety suite against this code dozens of times. It never caught this, and it could not have. The suite's test account skips the limit, because a suite that sends that many messages needs to. The gate was in front of the safety path and every test walked in through a side door.

Before changing anything I made it happen on a local copy of the system, with an account set to the limit and a crisis phrase. The limit message came back. Then I moved the crisis check ahead of the limit and ran it again. The fixed check costs nothing to run, since it never calls the model, so there was no cost argument for making it wait.

There was one trade-off inside the fix. If the crisis reply ignores the limit, someone could script it to inflate the yearly total without bound. So the reply is always sent, and the record is written only up to a daily ceiling. Past that, the person still gets the resources and the count stops. My legal lens and my security lens disagreed on this one. The law talks about the number of times a referral was issued, which argues for counting all of them. I took the bound and wrote the disagreement down for the attorney.

A label is not a wall

The last change was to the companion's memory. Each night it gets a short block of notes about past entries, built from summaries stored in the database. A user's browser can write those summaries. So a user could plant an instruction in their own memory and have it read with more authority than a typed message.

I marked the block as a record and not instructions, flattened each value to a single line, capped the lengths, and stripped the characters my own code uses for markers. Then I added a test that plants an instruction and checks whether the companion follows it.

The test passed. It had also passed before the change, on the old code, because I ran it there first. The model ignored that particular plant on its own. So I cannot claim the label did anything. The mechanical parts are proven by unit tests. The label's effect on the model is unproven, and that is how I logged it. I would have liked a cleaner story, but a pass that also happens without the fix is no evidence for the fix.

Mental Models

Ask who can write the number, as well as who can switch it off. A guard on an override looks like a guard on the thing it overrides. Check both.

Read a safety path from the far side of every gate in front of it. If your test account skips a gate, your tests cannot see what the gate blocks. Something else has to.

Store what checks a secret, and keep the secret somewhere else. A fingerprint lets a system verify a key without being a place the key can be stolen from.

Run the new test on the old code first. If it passes there too, its pass afterward tells you nothing about your change.

Decide what a count is counting before you protect it. I could make my number true. I could not make it unreachable, and trying would have meant not counting things that happened.