In production

One automatic rollback, minute by minute

A green tick should mean production was checked after the release, and a rollback nobody re-verifies is another unverified change.

By Ehsan Gazar, 14 September 2026

A deploy goes green. The tick means the platform accepted the new images. Nobody has looked at production yet, and later a customer does.

A green tick that means "queued" is a statement of intent. I want ours to mean production was checked after the release.

The pipeline, in order

Every push to main runs five stages in a fixed order. Rollback exists only as a response to the last one failing.

Check is what the repository decides alone: types, lint, tests, and every image built.
Preflight asks production what only production knows, including which commit to go back to.
Migrate moves the database schema forward, before any new image starts.
Deploy pins every application to one exact commit.
Verify asks the same questions from outside, over real TLS, against the public hostnames.

What happened on 14 September 2026

All times below are UTC.

One push carried two commits. The head, 66cca8e, changed the homepage header and hero. The commit under it, fec5207, rewrote the platform pages.

Check and the image builds passed between 14:50 and 14:51. At 14:52:01 the release began pinning every application to 66cca8e.

At 14:54:53 verify failed on a single check. The served bundle no longer contained one sentence: "See it against your own traffic."

Rollback started at 14:55:04. The console went back to the previous commit, 7645e9a, then the services behind it. At 14:58:18 it reported "production is back on the previous commit". Verify then ran again against the rolled-back estate and passed at 14:58:38.

Why a missing sentence fails a release

The site once carried sentences that were not true. For each one, verify runs an absence check: a test that a known false sentence is gone from the served bundle.

An absence check has a blind spot. It passes on a build that is simply missing the page, because an empty page contains no false sentences either.

So for each correction, verify also requires the true sentence that replaced the false one. "See it against your own traffic." is one of those. When it is missing, the failure text says why that matters: "a corrected page is not in this build, so the absence checks above prove nothing".

An absence check needs a presence check beside it, or it passes on nothing.

Was the rollback right?

The obvious story is that improved copy tripped a brittle check. I checked the history, and that story is wrong.

The homepage tour section had been removed earlier that afternoon, in 8176b47. That commit never touched the sentence. Its own run failed at check and never released.

The sentence was removed by fec5207, in the same push as 66cca8e. The platform page rewrite replaced the closing band on the products page and dropped the line.

The line was still true. You can still point an application at the gateway with a key you create yourself. So the check was not crying wolf. A rewrite had lost a true sentence, and the check noticed.

The fix, c0fc295, put the sentence back as the closing title on the products page. The check stayed unchanged. That fix was the next successful deploy, running from 15:00:58 to 15:06:31. Its commit message records that the built bundle was scanned against every required sentence before the push.

The cost of pinning a phrasing

This time the check was right. It has also been wrong.

Our own debugging notes record three earlier releases rolled back for telling the truth. In those cases the copy improved, the required sentence changed, and verify rejected a correct build.

The check's source keeps one such case. A page correctly followed a change to how many days of usage it could read. The check went on demanding the old number. That line is now built from the same constant the page reads, so the two cannot disagree.

A required sentence is a proxy for a property. Where you can assert the property directly, do that instead.

Pin the property, not the phrasing, or your safety check will roll back the truth.

Why the schema never rolls back

The rollback at 14:55:04 changed images and nothing else. That is deliberate.

Migrations are additive for one release. A new column has a default, and a new table sits beside the old ones. So the previous images keep working against the newer schema, and a rollback is an image change alone.

Reversing a migration would be far worse. It would delete rows written since the deploy. Nobody wants a tidy rollback that lost their usage records.

The schema goes forward before the images, and it never comes back with them.

A rollback is a second release

The step people skip is the one after the rollback.

A rollback is a change to production. Old images start again against a schema that has moved on. If nothing checks the result, you have shipped a second unverified change and called it safety.

That is why verify runs again after every rollback. Then a final step writes which commit production is running into the run summary. A red tick with that line under it is worth more than a green tick without one.

I do not trust a rollback until something outside production has looked at production afterwards.