Building a Timeline for Task Runners

I was working on an internal developer platform that handed tasks to runners in different environments. That work could run as a Kubernetes Job, a GitHub Actions workflow or a Jenkins build. The platform could start the work and track its overall status, but it had no direct view of what happened inside those environments. Each runner was a separate executor, with its own logic for carrying out the work. Asking it to start did not give the platform a way to follow its progress.

Once execution began, the platform could report that the work was running. But “running” covers a lot of ground. It tells you execution has started, but little about what has finished, what is happening now or what is holding things up. Those details lived with the runner, often in its logs.

I wanted to bring that detail back into the platform as a timeline. It needed to accommodate different kinds of work, from deployments to backups, and remain useful if the runner disappeared before reporting how the work ended.

That became the idea behind checkpoints.

Giving work somewhere to report

I split the problem between the control plane and the runner. The control plane coordinates the work and owns its overall status. The runner knows what is happening inside the execution. It reports those observations back, where they can be stored against the run and presented together. That run might be a deployment, a release, a backup or a restore; each needs somewhere to collect its progress.

A checkpoint here records progress; it does not save execution state for resuming work later. It describes the current state of an identifiable piece of work: which part has started, whether an operation finished or how far a change has progressed. The platform needs a common way to understand those observations, while the runner supplies the names and details.

To make that concrete, I’ll follow a deployment through the rest of the post. It has enough variety to show the different kinds of checkpoints: an overall lifecycle, stages within the run, individual tasks and rollouts with changing counts. The diagrams will build up that example one layer at a time. Those are the details of our example; the reporting model should work just as well for a backup or a restore.

The control plane records the deployment’s lifecycle. The runner reports its stages. Both contribute to the same timeline.

Lifecycle: has the work started or finished?

I started with what the control plane already knew. It was responsible for queuing the deployment, tracking when execution began and deciding when the run had reached an outcome. Each of those transitions was worth recording, even before the runner could tell us anything about the work inside. Those records became the lifecycle checkpoints: queued, running and, for a successful deployment, succeeded, each with a timestamp.

A simplified record of the deployment reaching “running” looks like this. The examples below use the same deployment scenario; I have left out run identifiers and storage metadata to keep the reporting fields visible.

{
  "kind": "lifecycle",
  "key": "running",
  "title": "Running",
  "status": "succeeded",
  "observed_at": "2026-09-01T10:00:02Z"
}

The two statuses might look odd together. Here, running names the transition the control plane observed. The checkpoint itself is already complete, hence succeeded: the deployment has reached that state. Queuing and completion get their own lifecycle records, rather than overwriting this one.

Keeping those checkpoints with the control plane gave the timeline a foundation that did not depend on the runner adopting the reporting mechanism. A new runner could contribute no checkpoints at all and still get a basic account of its run. The same would hold for a backup or a restore: whatever happened inside the execution, the platform had its own record of starting and ending it.

That also established a boundary for the reports I would add next. If the runner said a migration had succeeded, it was describing one operation. There might still be application changes to roll out or final checks to perform. That report could enrich the timeline, but deciding whether the whole deployment had succeeded remained the control plane's responsibility. Its lifecycle tracking needed to work even if the runner sent nothing.

So we already had a timeline that could answer when the deployment started and how it ended. What it could not explain was the stretch between them. A deployment spending ten minutes preparing configuration would look much like one spending ten minutes waiting for the application to become ready. Both would simply be “running”. To tell those stories apart, I needed the runner to describe where it was in the work.

Stages: where are we in the work?

The runner already has a procedure to follow. In our deployment, it prepares configuration, runs the pre-deploy jobs, rolls out the application and performs the post-deploy checks. Naming those stretches of work gives the platform something more useful to say than “running”. It can tell us which part the runner has reached.

Those names belong to the runner. If I built deployment stages into the control plane, every new kind of work would need another platform-specific workflow. Instead, the control plane understands a stage as a stretch of work with a name and a status. A backup runner can supply a completely different sequence through the same reporting mechanism. The timeline grows as stages are reported, without pretending to know which steps will come next.

When the runner reaches pre-deploy jobs, it can report:

{
  "kind": "stage",
  "key": "pre-deploy-jobs",
  "title": "Pre-deploy jobs",
  "status": "running",
  "observed_at": "2026-09-01T10:00:08Z"
}

The key identifies the stage; the title is what the reader sees. Keeping those separate means rewording a label does not create a different piece of work. When the stage finishes, the runner reports the same key with its outcome and a later observation time.

Each report carries one observation time, but together they can describe a span. The receiver keeps the first running observation as started_at and the first terminal observation as finished_at. If this stage reports success at 10:00:42, its duration is 34 seconds, measured from the 10:00:08 observation above. Later updates do not move those boundaries.

For that duration to explain the run, the boundaries need to belong to the stage itself. Reusing the deployment's transition to “running” as the start of configuration might be convenient, but the two events need not happen together. The resulting duration would measure the wrong stretch of work, precisely the ambiguity we were trying to remove.

I kept stages sequential because they describe the runner's position in its procedure. That does not mean everything inside a stage must run one thing at a time. Pre-deploy jobs can contain several jobs running concurrently while still being one recognisable part of the deployment. The stage locates that work; explaining the individual jobs calls for another level of detail.

Tasks: what is happening inside this stage?

Suppose the pre-deploy jobs stage prepares application assets and runs a database migration. Both jobs can run at once, but they need not finish together. Once the assets are ready, the stage is still running while the migration continues. Looking only at the stage, we are left asking: which job are we waiting for?

A task checkpoint gives each job its own place on the timeline. The runner reports the migration starting, then reports its outcome when it finishes. Asset preparation does the same independently. We can keep the two jobs together as one stage without hiding the difference between their progress.

The migration's report adds one field to that familiar shape:

{
  "kind": "task",
  "stage_key": "pre-deploy-jobs",
  "key": "database-migration",
  "title": "Database migration",
  "status": "running",
  "observed_at": "2026-09-01T10:00:11Z"
}

stage_key places the migration beneath the pre-deploy jobs stage. Its own key identifies the migration, so its reports update that job rather than the stage or its neighbours.

Now, when asset preparation finishes, its task can show success while the migration remains running. We can see what is holding up the stage and, if both jobs reported their starts, how long each has taken. A task that reports only its completed result still has a place here, as a point on the timeline rather than a span with an invented start.

If the migration then fails, we can keep that same distinction. The assets were prepared successfully; the migration was not. Its checkpoint can carry a short explanation from the job alongside the failed outcome, giving us somewhere to begin investigating. The stage's failure tells us this part of the deployment did not complete successfully, while the tasks explain what happened within it.

That gives us a more useful account of the work inside a stage: which jobs are still running and how the others ended. But knowing that a job is running does not always tell us how far it has progressed. When the runner starts waiting for the application to roll out, there is more it could tell us.

Rollouts: how far has the change progressed?

Suppose this time the pre-deploy jobs succeed. The runner submits the application changes and waits for four webservers and two workers to finish rolling out. Both components are running, but that status hides a useful difference: the workers may already be ready while the webservers are still coming up.

A rollout checkpoint gives that wait some detail. Each component reports its own counts, so the workers can show two of two ready while the webservers show two of four. We can see which component is holding up the stage and how far it has reached.

The webservers' report adds those counts to the familiar checkpoint shape:

{
  "kind": "rollout",
  "stage_key": "rollout",
  "key": "webservers",
  "title": "Webservers",
  "status": "running",
  "observed_at": "2026-09-01T10:01:20Z",
  "data": { "ready": 2, "desired": 4 }
}

The data fields describe two ready webservers out of four desired. As with tasks, stage_key places this record beneath the rollout stage, and the workers have their own record alongside it.

As the webservers become ready, their count changes in the same entry. The workers can finish and stay green while we watch the remaining webservers. These counts describe each component's progress, rather than a percentage of the whole deployment.

In the example below, the webservers never get past two of four before the runner reports a timeout. Keeping that count beside the failure tells us how far the rollout got. The two ready replicas remain visible, and the workers keep their successful outcome. We have more to go on than a deployment that simply says “failed”.

The timeline now tells us what finished, what failed and how far the work got. That may be enough to understand the run, but a failed migration or a stalled rollout can still leave us asking why. For that, we usually turn to the logs.

Finding the right part of the logs

An unexpected benefit of checkpoints was that they gave me a way to divide the runner's log stream into meaningful sections. If a migration failed, I could look at the output from the pre-deploy jobs stage. If everything succeeded but that stage took longer than expected, I could inspect the same section to understand where the time went.

The individual log lines had not become structured. The stream had acquired structure: named sections that connected the output to the work being done.

That made the stage names useful in a second place. Instead of searching from the top of the run, I could expand pre-deploy jobs or scope the viewer to that section alone. Returning to the full stream brought back the surrounding context without changing the line numbers. The same line was still the same line, whichever view I was using.

Together, the timeline and grouped logs made it easier to follow a run and investigate the parts that deserved attention, whether they failed or simply took too long. But that account still depended on what reached the control plane. The next question was what remained when the runner stopped reporting altogether.

Now let the runner disappear

Return to the rollout with two of four webservers ready. This time, instead of staying around to report a timeout, the runner disappears after sending that count. The application might continue coming up without it. The remaining replicas might fail. All the platform has is the last observation that reached it.

That is why I kept the deployment's lifecycle separate from the runner's account of the work. The control plane can discover the runner's failure through its own monitoring and mark the deployment failed. The rollout checkpoint still says two of four were ready. I would show it as unresolved, making clear that this is where reporting stopped, rather than leave it looking like live progress. We know how the run ended, but we do not know how every piece of work ended with it.

For that last observation to be useful, it has to make sense on its own. “Two of four ready” tells us something even if an earlier update never arrived and no later one ever will. That led me to keep the latest snapshot of each piece of work, updating its record as reports arrived. The timeline preserves where the work got to without needing to retain every count along the way. It gives up the full history of those changes, but it does not need a complete conversation with the runner to remain useful.

What this makes possible

What I like about this design is that each checkpoint gives the platform something useful to work with without asking it to understand every runner's procedure. There are plenty of nuances in how those reports are delivered, stored and presented, but I wanted to focus here on the ideas that give the account its structure.

Once that structure exists, there is more you could build on it. Failure classification could help route an incident to the right team. An AI-generated summary could bring together the failed task, its evidence and the relevant logs. Those would be ways to help a developer understand what happened and decide what to do next, without having to reconstruct the run themselves.