What Breaks When One Resource Costs 300× More Than the Rest

The setup

Hourly billing, prepaid balance. A worker charges each running server once an hour. When your balance can't cover the next hour, the server is suspended, snapshotted, and the instance is deleted — your data survives in the snapshot, and topping up brings the server back.

That flow is four years of small corrections wearing a groove: snapshot first, then delete, keep the reference until deletion is confirmed, reconcile anything left behind. It works. It also encodes an assumption nobody wrote down: **storing a customer's disk is cheap relative to what the server earns.**

At $0.014/hour with a 25 GB disk, that's true. Storage runs about $1.50/month against a server that bills $10/month.

An H100 has a 720 GB disk. Snapshot storage on that is roughly **$43/month** — for a server whose customer might have run it for three hours and paid $14. The archive costs more than the customer ever spent, forever, silently, per abandoned server.

So the decision: GPU servers don't get a snapshot. They're deleted outright. That single exception is where all three bugs came from.

Bug 1: a status that promises something you can't deliver

Our status enum has ARCHIVED. The API docs say:

The restore worker looks like this:

SELECT * FROM servers
WHERE status = 'ARCHIVED'
  AND "vultrId" IS NULL
  AND "suspendSnapshotId" IS NOT NULL   -- ← the quiet contract

My GPU branch set ARCHIVED and skipped the snapshot. Everything "worked": no crash, no error log, no failing test. The record simply never matched that IS NOT NULL, so the restore worker walked past it every minute for seven days until retention swept it into DESTROYED.

Meanwhile the customer's dashboard showed a status that means *top up and I'll bring it back*. They could pay us money expecting a machine to return. Nothing would happen. No error to report, no ticket to file — just a promise the system was structurally incapable of keeping.

The fix is one line, and it's a modelling fix rather than a code fix: if there's nothing to restore, the state isn't ARCHIVED, it's DESTROYED.

**The lesson:** a status is a promise to the user. When you add a code path that reaches a state by a different route, re-read what that state promises. Enum values don't have preconditions the compiler can check — the docs page is the only place the contract is written down, and it doesn't run in CI.

Bug 2: the one I wrote while fixing bug 1

Here's my fix. Read it before the explanation:

// close the record
await prisma.server.updateMany({
  where: { id: server.id, status: "SUSPENDED" },
  data: { status: "DESTROYED", destroyedAt: new Date() },
});
// then delete the instance
try {
  await provider.destroyInstance(server.instanceId);
  await prisma.server.update({ where: { id: server.id }, data: { instanceId: null } });
} catch {
  // reconciler will retry
}

That comment — *reconciler will retry* — is a lie, and it's the whole bug. I copied the shape from the neighbouring non-GPU path, where it's true. There are two reconcilers, and they select on state:

where: { status: "DESTROYING" }                        // reconciler #1
where: { status: "ARCHIVED", instanceId: { not: null } } // reconciler #2

A record in DESTROYED holding a live instance id matches neither. If that destroyInstance call fails — provider hiccup, timeout, rate limit — the database says the server is gone, billing stops charging, the customer stops paying, and the H100 keeps running at the provider at **$3.39/hour** billed to me. Nobody finds out until the monthly invoice.

For the $10/month plans this failure mode existed too, and it was survivable enough that nobody noticed. At GPU prices, one occurrence costs more than the whole feature earns in a week.

Two changes. First, invert the order so the risky step happens while the record still matches a reconciler:

// delete FIRST — an error here propagates, the job retries, the row stays SUSPENDED
await provider.destroyInstance(server.instanceId);
await prisma.server.updateMany({
  where: { id: server.id, status: "SUSPENDED" },
  data: { status: "DESTROYED", destroyedAt: new Date(), instanceId: null },
});

Second — and this is the part worth stealing — write the invariant down as a reconciler of its own:

// A closed record must never hold a live instance.
const leaked = await prisma.server.findMany({
  where: { status: "DESTROYED", instanceId: { not: null } },
});
for (const s of leaked) {
  console.error(`[ALERT] destroyed server still holds a live instance id=${s.id} plan=${s.plan}`);
  await provider.destroyInstance(s.instanceId);   // then clear the id
}

The first fix closes the path I found. The second closes the ones I haven't. Any future code path that marks a record closed without confirming deletion gets caught by a machine instead of by an invoice.

**The lesson:** "a reconciler will catch it" is a claim about a WHERE clause. Go read the clause. And when you find a gap, prefer adding an invariant check over fixing the single path — the paths multiply, the invariant doesn't.

Bug 3: the promo that would have given away $80/day

We run occasional promotions where a plan is free. It's driven by an environment variable:

PROMO_FREE_PLANS=vc2-1c-1gb

The billing worker skips charging when the server's plan is in that set. There was already one hard-coded exception:

if (promoActive && PROMO_FREE_PLANS.has(server.plan) && !isWindowsOs(server.osId)) {

Windows is excluded in *code*, because a Windows licence has a real per-hour cost and giving it away means selling below cost. That exception exists because someone thought about licence economics once.

Nobody had thought about a plan whose underlying cost is $3.39/hour. GPU plans weren't in the env var, so nothing was broken — the bug was one typo away, in a config file, deployed by a human at 2 AM. A stray gpu-h100x1-80gb in that list would hand out **$80/day per server** with no alarm anywhere, because free servers by definition generate no billing events to notice.

if (
  promoActive &&
  PROMO_FREE_PLANS.has(server.plan) &&
  !isGpuPlan(server.plan) &&      // ← added
  !isWindowsOs(server.osId)
) {

**The lesson:** if a config value can cost you money proportional to a mistake in it, the ceiling belongs in code. Config says what you *want*; code says what's *possible*. Anything catastrophic should be impossible, not merely undesired.

The pattern behind all three

Each bug came from a piece of code that was correct when written, against an assumption that was true when written:

The failure mode isn't bad code. It's **code whose correctness depends on the size of a number that just changed**, with nothing in the type system or the test suite tied to that number.

What I'd do differently, and what I'd suggest if you're about to add an expensive tier to something that's only ever been cheap:

• **Grep for every place the price is looked up.** Mine had a findPlan() that searched only the standard catalogue. A GPU server would have been created with no price at all — billed nothing, running $3.39/hour of hardware. Found before shipping, but only because I went looking specifically for money paths, not because anything failed.
• **Grep for every cost table.** Profit reporting had per-plan costs for the cheap plans. Without adding GPUs, the dashboard would have reported GPU revenue as pure profit and mis-stated margin by more than the entire rest of the fleet.
• **List the states your new path can reach** and re-read what each one promises the user.
• **Write the expensive invariants as running checks**, not as comments claiming something else will handle it.

The three bugs took about forty minutes to find, and all three were found by reading my own diff a second time, with one question in mind: *what does this code assume about the number?*

That question is worth asking before the invoice asks it for you.

---

*GPU servers are live on NoctHost — H100 and RTX 6000 Ada, billed by the hour like everything else, paid in crypto. Now you know exactly what had to be fixed before they shipped.*