1. The job that asked the wrong cloud
We started on one provider and added more later. Background jobs (destroy, suspend on zero balance, traffic limits, the reconcile loop) had been written against a single provider client. When the second cloud went live, the request paths were updated to choose a client per server. Several jobs were not.
So a server on the second cloud would provision fine, and then the destroy job sent its ID to the first cloud's API. The answer was a 400, "invalid resource format". The job retried forever and the server sat in DESTROYING while the real machine kept running.
The worse half was the reconcile loop. It walked every active server once a minute, and one server on the "wrong" cloud threw on every pass. The exception aborted the whole tick, so the same loop also stopped cleaning up stuck deletes and orphaned instances for everyone else.
The fix was small: every per-server operation resolves its client from the server record.
// before
await provider.destroyInstance(server.instanceId);
// after
await providerFor(server.provider).destroyInstance(server.instanceId);The lessons were bigger:
- Dispatch from the record, never from a default. A module-level "the provider" singleton is a bug waiting for a second provider.
- In a loop over many resources, one bad item must not stop the rest. Catch per item, log it, move on.
- Write the invariant as a test, so the next feature cannot quietly reintroduce the singleton.
2. A 404 is not proof it is gone
Deletes should be idempotent, so "404 Not Found" on DELETE is usually treated as success: someone already removed it. We did the same.
Then we added dedicated servers. On one provider they live behind a different endpoint than virtual machines. A dedicated server's ID sent to the virtual machine endpoint also returns 404. Read as "already deleted", that would close the record and drop our only reference to a machine that costs hundreds of dollars a month.
Two rules came out of this:
- Decide "what kind of resource is this" in one place, from the stored record, and route every call through that decision.
- When the type is unknown, fail loudly on the ordinary path instead of guessing a special endpoint where a 404 reads as success.
3. A 204 is not proof either
During a live test, a DELETE on a dedicated server that was still installing returned 204. Six minutes later the machine was back in the provider's server list, running.
Had we trusted the response, the record would say DESTROYED, a state no watchdog looks at, while the bill kept growing to the end of the month.
Now the adapter reads the resource back after a delete and throws if it is still there. The job retries, the record stays in DESTROYING, and DESTROYING is a state our monitoring does watch. For expensive resources, "the API said OK" is not enough to believe we stopped paying.
4. One empty read is not disappearance
The reconcile loop used to mark a server destroyed the first time the provider returned nothing for it, and send a DELETE in the same pass. Cloud APIs have short blips, though: a list that is briefly incomplete, a lookup that misses during a state change.
Now a machine has to be missing on consecutive checks spanning at least three minutes before we believe it is gone, and any "it exists" answer resets the count. A confirmed disappearance closes the record without sending another DELETE.
We also stopped alerting on every normal delete. Inside a 30-minute window after a destroy, the delete is retried quietly. The alert only fires if the window passes and the provider still has not confirmed the machine is gone.
The checks we run now
- Every operation on a server or load balancer gets its client from that resource's own record.
- Delete is confirmed by reading, not by the response code.
- Disappearance is confirmed over time, not by one empty answer.
- A watchdog emails us about anything stuck in
DESTROYINGorPROVISIONINGtoo long, and about any destroyed record that still points at a live instance. - A periodic job lists every instance that carries our tag at each provider and flags the ones our database does not know about. It never deletes them on its own.
None of this is clever. It all comes from one fact: a cloud API reports what it is doing, not what you will be billed for. If you bill by the hour on top of several clouds, it is worth treating "is it really gone?" as its own question with its own tests.
On the customer side, all of this shows up as one simple promise: servers bill by the hour, and billing ends when you delete one.