Vatan blog

One automatic rollback, minute by minute

Ehsan Gazar — Mon, 14 Sep 2026 09:00:00 GMT

A green tick should mean production was checked after the release, and a rollback nobody re-verifies is another unverified change.

A deploy goes green. The tick means the platform accepted the new images. Nobody has looked at production yet, and later a customer does.

A green tick that means "queued" is a statement of intent. I want ours to mean production was checked after the release.

The pipeline, in order

Every push to main runs five stages in a fixed order. Rollback exists only as a response to the last one failing.

Check is what the repository decides alone: types, lint, tests, and every image built.
Preflight asks production what only production knows, including which commit to go back to.
Migrate moves the database schema forward, before any new image starts.
Deploy pins every application to one exact commit.
Verify asks the same questions from outside, over real TLS, against the public hostnames.

What happened on 14 September 2026

All times below are UTC.

One push carried two commits. The head, 66cca8e, changed the homepage header and hero. The commit under it, fec5207, rewrote the platform pages.

Check and the image builds passed between 14:50 and 14:51. At 14:52:01 the release began pinning every application to 66cca8e.

At 14:54:53 verify failed on a single check. The served bundle no longer contained one sentence: "See it against your own traffic."

Rollback started at 14:55:04. The console went back to the previous commit, 7645e9a, then the services behind it. At 14:58:18 it reported "production is back on the previous commit". Verify then ran again against the rolled-back estate and passed at 14:58:38.

Why a missing sentence fails a release

The site once carried sentences that were not true. For each one, verify runs an absence check: a test that a known false sentence is gone from the served bundle.

An absence check has a blind spot. It passes on a build that is simply missing the page, because an empty page contains no false sentences either.

So for each correction, verify also requires the true sentence that replaced the false one. "See it against your own traffic." is one of those. When it is missing, the failure text says why that matters: "a corrected page is not in this build, so the absence checks above prove nothing".

An absence check needs a presence check beside it, or it passes on nothing.

Was the rollback right?

The obvious story is that improved copy tripped a brittle check. I checked the history, and that story is wrong.

The homepage tour section had been removed earlier that afternoon, in 8176b47. That commit never touched the sentence. Its own run failed at check and never released.

The sentence was removed by fec5207, in the same push as 66cca8e. The platform page rewrite replaced the closing band on the products page and dropped the line.

The line was still true. You can still point an application at the gateway with a key you create yourself. So the check was not crying wolf. A rewrite had lost a true sentence, and the check noticed.

The fix, c0fc295, put the sentence back as the closing title on the products page. The check stayed unchanged. That fix was the next successful deploy, running from 15:00:58 to 15:06:31. Its commit message records that the built bundle was scanned against every required sentence before the push.

The cost of pinning a phrasing

This time the check was right. It has also been wrong.

Our own debugging notes record three earlier releases rolled back for telling the truth. In those cases the copy improved, the required sentence changed, and verify rejected a correct build.

The check's source keeps one such case. A page correctly followed a change to how many days of usage it could read. The check went on demanding the old number. That line is now built from the same constant the page reads, so the two cannot disagree.

A required sentence is a proxy for a property. Where you can assert the property directly, do that instead.

Pin the property, not the phrasing, or your safety check will roll back the truth.

Why the schema never rolls back

The rollback at 14:55:04 changed images and nothing else. That is deliberate.

Migrations are additive for one release. A new column has a default, and a new table sits beside the old ones. So the previous images keep working against the newer schema, and a rollback is an image change alone.

Reversing a migration would be far worse. It would delete rows written since the deploy. Nobody wants a tidy rollback that lost their usage records.

The schema goes forward before the images, and it never comes back with them.

A rollback is a second release

The step people skip is the one after the rollback.

A rollback is a change to production. Old images start again against a schema that has moved on. If nothing checks the result, you have shipped a second unverified change and called it safety.

That is why verify runs again after every rollback. Then a final step writes which commit production is running into the run summary. A red tick with that line under it is worth more than a green tick without one.

I do not trust a rollback until something outside production has looked at production afterwards.

Serving Cursor's requests through the gateway

Ehsan Gazar — Mon, 14 Sep 2026 09:00:00 GMT

Three request shapes production refused or quietly broke, and why the quiet failure was the dangerous one.

You point Cursor at a custom base URL and ask it to edit a file. The model replies with text and changes nothing. No error appears anywhere.

You conclude the model chose not to use its tool. Or you conclude the gateway does not work with Cursor. Both conclusions are wrong, and the second one costs a customer.

What Cursor actually sends

With a custom base URL, Cursor calls /chat/completions from its own servers. Cursor's staff said so on its forum on 13 August 2026. So the door that matters is the Chat Completions door on gateway.vatan.one.

What Cursor puts inside that request is the harder part, because it has changed from release to release. On 14 September 2026, before any fix, I measured three shapes against production. I used a key in our own test workspace, with anthropic/claude-haiku-4-5 and openai/gpt-5-nano.

Flat function tools. Cursor sends name and parameters directly on the tool, with no nested function object. The Anthropic model answered 200 with no tool call. The OpenAI model answered 400.
A custom tool. Cursor's file-edit tool, ApplyPatch, is declared as type: "custom". The results were the same: a 200 with no tool call, and a 400.
A Responses body on the chat path. A body carrying input, reasoning and text instead of messages. Both models answered 400.

The 400s were the easy part

A 400 is loud. Somebody sees it and files a bug.

The two 200s were the real problem. A reply with finish_reason: "stop" and no tool call is a valid answer. It is exactly what a model returns when it decides not to call a tool. An agent reading it moves on. A person reading it blames the model.

Nothing throws and nothing turns red. The tool definition was lost on the way to the Anthropic wire, because the translation expected the nested shape and found nothing to translate.

A success that did not do the work is worse than an error, because nobody goes looking for it.

One canonical body, at the door

The tempting fix is three patches, one per shape. I did not do that, because the next Cursor release can send a fourth.

A canonical form is the one agreed shape that every other shape is converted into before real work starts. It is the old rule of normalising input at a system boundary, and it pays the same way: everything behind the boundary handles one case.

Everything downstream of our chat handler already read OpenAI's nested tool shape. The Anthropic translation read it, and so did the OpenAI passthrough and the cache key. So the new file, chat-shapes.ts, makes the body canonical once, at the door. Nothing below it learns that Cursor exists.

Flat tools become nested tools. A Responses body on the chat path is translated into a chat body. A custom tool becomes a function with one string argument. A body with none of these shapes comes back as the same object, so every existing client takes exactly the path it took before.

Normalise at the boundary, and prove the untouched path is really untouched.

And back out again

Converting the request is half the job. Cursor declared ApplyPatch as a custom tool, so it expects a custom call back, not a function call.

So a call to a tool the caller declared as custom goes back as a custom call. Its raw patch text is pulled out of the one-field wrapper first. The reverse also happens on later turns, when Cursor sends that call back in the conversation history.

This had to work buffered and streamed, and those are separate code paths. Streaming is the fiddly one. The model streams its argument as fragments of JSON, and the wrapper around the text must never leak through. The stream rewriter tracks each call and sends only the new text since the last chunk. Every other event passes through byte for byte.

I insisted on both paths because this repository has been here before. A streamed translation once dropped tool calls for days while the buffered one was correct.

What we gave up

A custom tool can carry a grammar that constrains what the model writes. OpenAI's wire supports that natively. We still convert custom tools to functions on every wire, OpenAI's included.

So the model is asked for a string and writes the patch as free text, unconstrained. In exchange, one translation means one set of tests and one shape coming back. I think that trade is right, and it is still a real loss. The gateway names it in the x-vatan-responses-dropped header as custom.format, so the loss is never silent.

If you drop part of a request, say so in the response.

How it was proved, and what that does not prove

The proof is client-compat.mjs, a script that sends requests to production and checks the answers. It gained Cursor's shapes in the same commit, 7d56156. Each shape runs on an Anthropic-wire model and an OpenAI-wire model, buffered and streamed.

It checks that a flat tool is actually called. It checks that a custom tool comes back as a custom call carrying raw text. It checks that the call and its result are accepted on the next turn. It also checks that a streamed tool turn still reports usage, so the turn is not metered at zero.

It is not the real Cursor app. Cursor sends requests from its own servers, and a custom key needs a paid Cursor plan. The script has neither. So this proves the gateway answers Cursor's published request shapes correctly. It does not prove that what Cursor sends today is one of those shapes.

That gap is why the fix covers the class rather than three cases. When a new shape arrives, it should land on a door already built to be surprised.

A gateway is judged by the client that fails quietly, so test the quiet failure first.

Why our margin is on the pricing page

Ehsan Gazar — Mon, 14 Sep 2026 09:00:00 GMT

A price a buyer can check beats a price a buyer negotiates, and keeping it true has to be mechanical.

You open a vendor's pricing page and find "contact sales". You book the call. Weeks later you have a number, and no idea whether the next buyer got a better one.

I did not want Vatan to work like that. So the margin is on the page.

The number

A markup is the amount a seller adds on top of what something costs them. Most vendors fold it into one blended price. We print ours.

A request through the Vatan gateway is charged at what the model costs us to buy, plus 5%. For a model we buy direct, our cost is the provider's published price. The rate is the same on every plan.

Nothing else goes on the bill. There is no platform fee, no seat and no subscription. Pay as you go starts with a $5 top-up, and that is the only minimum.

Why there is no discount for bigger plans

On 4 September 2026 we considered a tiered markup, lower on bigger plans. We dropped it the same day.

A tier sounds generous. In practice it turns the price into a conversation. Every buyer asks which tier they are in, and whether the next one down is open to them. The rate stops being something you read and becomes something you argue about.

One rate gives the procurement question one answer, and the buyer can check it alone. I think that is worth more than a discount somebody has to earn.

If a buyer has to ask what the price is, you do not have a price yet.

A published number drifts

Printing a percentage is easy. Keeping it true is the hard part, because a price lives in more places than you expect. Ours lives in four, and all four must agree:

The rates table, routing.model_prices, which is the sell price itself.
deploy/invariants.json, which records the multiplier "1.05" that the post-deploy check recomputes against.
The console's billing constant, 500 basis points, which every figure on the site is built from.
The internal pricing model document, where the decision is written down.

A basis point is one hundredth of a percent, so 500 basis points is 5%. You never need to think in basis points. The point is that the site never types the figure; it derives it.

The usual failure is not one wrong price. It is two places that were both right once and then drifted apart.

That happened to us on 4 September 2026. The catalogue had moved to the new rate. The FAQ kept quoting an older, higher markup for a day. Nobody chose to mislead anyone. The figure had been typed into prose in more than one file, and the test guarding the plan cards never read the FAQ.

Never update a price

Every row in the rates table has a start date. A reprice inserts new rows and never rewrites old ones.

That sounds like bookkeeping, and it is the part that protects you. Last week's charge still reconciles against the invoice it produced. Update a price in place and every past invoice disagrees with the table that supposedly produced it. One UPDATE destroys your own audit trail.

Never update a price. Insert the next one.

The checks that hold the number still

The FAQ was fixed by making a second figure impossible to ship, not by rewording it. A test now reads every shipped string in the site's data modules and marketing pages. It fails the build if any percentage presented as a markup differs from the one the meter charges.

After each deploy, the release pipeline reads the live rates table. Every advertised rate on a model we buy direct must be exactly the recorded multiplier times a list figure. A model re-rated at any other markup fails the release. Models we buy through another provider are checked the same way, against their own recorded basis, because what they cost us is different.

The pipeline then fetches the served site over real TLS. The release fails if the bundle does not contain the literal string "plus 5.00%". Without that, a page that quietly lost the number would pass everything else.

A failed release rolls back. A price change nobody decided is treated like an outage, because for the buyer it is one.

Assert the price as a property of the data, never as a copy of somebody's price list.

Show the working

The /models page lists each model with the provider that serves it. It shows what the model costs us beside what you pay. Since 5 September 2026 it also marks the cheapest provider for each model.

Marking the cheapest provider costs us something. It sends buyers to the cheapest route, where 5% is a smaller amount of money. I think that is the right trade. A buyer who watches us point at the cheaper option tends to believe the rest of the page.

What this does not solve

A published markup tells you nothing about whether the underlying cost is fair. Providers set their own prices, and those prices move. When one moves, your bill moves with it, even though our 5% has not.

I do not have a clean answer to that. The best I have is that every change is a new dated row, and the working is on the page.

If your pricing page and your meter can disagree, one of them is wrong, and a buyer will find out which before you do.