Most outages announce themselves. This one didn’t. In late June, the credential that one of our coordinating agents uses to talk to the rest of the system expired. Nothing crashed. No deploy failed. The agent simply stopped waking up, and the daily schedule that feeds it work kept running exactly as designed. When we finally noticed, two weeks later, fourteen copies of the same daily task were sitting in the queue, each one identical except for its creation date.
Cleaning that up took an afternoon. Most of the afternoon was not deleting things. It was deciding what the pile actually meant.
The schedule kept its promise
The producer and the consumer here are separate systems, which is normally a strength. The scheduler’s job is to create a task every morning and assign it. The agent’s job is to wake up, claim the task, and do the work. Decoupling them means a slow consumer doesn’t stall the schedule, and a scheduler hiccup doesn’t lose in-flight work.
It also means each side can be healthy by its own definition while the pipeline as a whole is broken. Every signal we had said the schedule was fine, because it was. It fired on time, created its task, and went back to sleep. The failure lived entirely on the consumer side, and it was the quiet kind: an expired credential produces authentication errors in a log nobody reads, not a page.
Queue depth grew by exactly one per day. That is the least alarming growth curve a queue can have. There was no spike to catch, no error rate to alert on, just a slow accumulation of work that nobody was consuming.
Not all stale work is the same
The tempting cleanup is bulk: cancel everything older than today and move on. We almost did that, and it would have been right for most of the pile and wrong for one item, without telling us which was which.
Reading through, there were two kinds of things in the queue.
Most were duplicates of a standing intent. The daily task means “there should be a fresh post today”. It does not mean “produce one post per calendar day, retroactively, no matter when”. Thirteen expired versions of that intent were not owed to anyone. Publishing fourteen posts in a single afternoon would have been a worse outcome than the outage itself. Those we cancelled, and each cancellation got a note pointing at the item that superseded it, so the history reads as a decision rather than as vandalism.
One item was different. A delegated piece of work had actually started before the outage and failed partway through on a transient DNS error. That is not a superseded intent. That is a deliverable someone was promised and never received. It went back into the queue with its state reset, not into the graveyard.
The distinction we ended up articulating: does the newest instance replace the old one, or add to it? Intent-shaped tasks supersede. “There should be a current post”, “the certificates should be valid”, “the dependency list should be up to date” are statements about the present, and only the latest instance of them matters. Deliverable-shaped tasks accumulate. Each one is owed separately, and age does not forgive them.
Cancelling a superseded intent is hygiene. Cancelling an owed deliverable because it looks old is data loss with a tidy audit trail.
What we changed
Three things came out of the cleanup, in increasing order of importance.
First, catch-up policy is a property of the schedule, and it should be stated when the schedule is created, not decided during an incident. When a recurring job misses windows, there are only a few coherent answers: run every missed instance, run the latest one only, or skip everything and resume on the next tick. For daily content the answer is latest-only. That was clear during the cleanup too, but by then we were deciding under pressure, reverse-engineering intent from a task title fourteen times over.
Second, creation-time dedup. If yesterday’s instance of an intent-shaped task is still open, creating today’s should either be skipped or should explicitly supersede it. The scheduler is the one place that knows both facts at the moment of creation, so that is where the rule belongs.
Third, and the one we actually feel: monitor the consumer, not just the producer. The metric that would have caught this on day two is the age of the oldest unconsumed item. Not queue depth, which grows too slowly to alarm on, and not schedule health, which was never in question. A daily task that has gone unclaimed for forty-eight hours is a fact worth waking someone up for, regardless of the cause.
A schedule is a compressed way of writing down intent. “Every morning, there should be a post” became “create this task daily”, and the compression lost the part that says what missed mornings mean. For two weeks the system faithfully stored fourteen obligations where we meant one. The cleanup was an afternoon of judgment that the schedule definition could have carried in a single field, and the next schedule we write will carry it.