← back

The Revert I Never Ran

#encryption#release-management#decision-making#privacy#product-management

Why I shipped a small encryption change as two releases, and how a mistake at the smoke test proved the reversibility I had designed for.

Last night I wrote about the sticky note on the lid of each journal entry: the one-line summary that was still readable on the server while the entry beside it was sealed. Tonight that note is sealed too, on every encrypted account that has opened the app since. The change itself was small. What I want to write down is the shape of the release, because the shape is where the real decision lived, and because a mistake I made during testing turned out to be the best proof of it.

Two releases instead of one

My first draft of the design said the change was safe to undo. The new code could read both forms of the field, the readable one and the sealed one, so if something went wrong I could revert and nothing would break. I believed that for about an hour.

It was caught at the review of last night's post, after the design was locked. The reviewer asked what the app would actually run after a revert. Not the new code. The old code. And the old code renders that field as text, straight from the database. Revert a single combined release and every journal card would show a block of encrypted gibberish where the summary used to be. The tolerance I was counting on protects you only if it is already live underneath the change you are undoing.

So the change became two releases. The first one taught the app to read both forms and changed nothing else. Every row stayed readable, every card showed the same line as before. The second one started sealing new summaries and quietly converted the old ones. If the second release ever has to come out, the app lands on the first, which reads both forms. Reversibility is not a property of the new code. It is a property of what is left standing when the new code is gone.

The mistake

The plan had the second release on Monday. By Friday afternoon the gates were green and the tests were done, so I shipped both the same day, ahead of Monday.

Testing happens on a separate preview address with a throwaway account, never my real one. That is a rule I wrote down after an earlier slip. Tonight I broke it without meaning to: my real account was still signed in on the preview from an earlier visit, and when I opened the preview to check the second release, the conversion ran on my own journal. Every one of my summaries was sealed by a build that had not shipped yet.

Here is the thing. Nothing happened. I opened the production app and every card showed its line, because the first release was already live there and it reads sealed lines fine. The accident was exactly the revert scenario I had designed for, just entered from the other side: production running the older build against data written by the newer one. If I had shipped one combined release, that same leftover session would have put gibberish on every one of my cards until I caught it.

I did not plan that test. I would not have run it on purpose. But it is the most convincing evidence I have that the two-release shape was right, and I am keeping the rule about the throwaway account anyway, because the next slip may not land on a change that was built to absorb it.

What the checks looked like

Most of the engine already existed, from the earlier migration of the entries themselves, so the new work was mostly teaching it a third column and proving it would not rewrite a row it had already done. A dozen cases on a local copy of the database covered the shapes: readable summary on a sealed entry, no summary at all, a summary at the length limit, a test account that is meant to stay readable, and a second run that must change nothing.

The phone pass was mine, nine checks. Two of them were not on the screen at all. After the cards rendered, I had the row shapes read back from the database, without the text, to confirm that what the screen showed as a normal line was sealed underneath. A card that looks right is not proof. The row is.

That morning's retro had decided that previews would get a browser pre-check before I pick up the phone, to cut down the back and forth. The pre-check could not run on the preview tonight. It works on the production address and fails on the preview and on my local build, and I still do not know why. So the phone pass stayed the whole test, which is a fair measure for the next retro: a process improvement that cannot run is not an improvement yet.

The paperwork that ships with it

The privacy policy said the short summary was readable. It is not anymore, so the policy changed with the release: five sentences, published as a clarification rather than held for the thirty-day notice, because the change narrows what the server holds and adds no new reader. The onboarding screen's sentence changed too, and every account sees that screen once more. The old wording went on the list that the whole-site scan checks, so the whole-site scan catches any copy on a page I forgot about.

There is one honest limit. A summary on an account that has not opened the app stays readable until it does. The server cannot seal it, because the server does not have the key. That is the whole point, and it is also the reason the conversion is a thing the app does, one account at a time, rather than a thing I can run from my desk.

Also tonight

A small pass on the landing page, from my own read of it on a phone. The sign-in button now lives on the landing itself instead of a second screen, with the age check kept between the tap and Google. The button had to change to Google's own design: their rules allow the coloured G only on a light, dark or neutral button, and following those rules is a condition of the app verification I already hold. The amber button I liked was a listed "don't". The questions section collapsed to its titles. The desktop layout is next.

Three releases in one afternoon is more than I usually do, and more than the session rule I set for myself allows. The next retro gets to decide whether the rework that followed, two fixes after the first smoke and a push in the wrong order, was the cost of that.