Deployment logs after a service is deleted

A build and its dashboard do not share one clock. A person can remove a service while a remote process is still writing output. The database may delete the deployment first. A buffered batch of deployment logs then arrives with an identifier that no longer exists. Each action makes sense on its own, but their timing creates a race.

This is a useful failure to think through before it wakes someone up. The deleted service should stay deleted. The background process must not recreate it just to attach a few lines of output. At the same time, a failed log write should not turn an otherwise valid build into a failed release. Decide which result matters before choosing how to catch the exception.

Start by drawing the two timelines on paper. Mark the remote build, each log buffer, the delete transaction and the final flush. Include work already waiting in a queue. A cancellation signal may stop future work while leaving an earlier database request in flight. Your design has to account for that last request as well as the next one.

A foreign key gives each log line an owner. Keep that relationship even when it produces an awkward error. Removing the constraint would allow orphaned rows, leaving later cleanup and access checks to work out which service they once belonged to. That is a poor trade for avoiding a known race.

The PostgreSQL constraint documentation explains the relationship and the available deletion actions. Choose the action that matches your retention policy. Cascading removal can suit short-lived build output, while a separate audit record may need a different lifetime. Neither choice decides what an already running writer should do next.

Keep the write scoped to the same organisation as its deployment. If you look up the parent after a failure, use both identifiers. Finding a deployment in another workspace is not evidence that this writer still owns a valid destination. The database relationship and the application access check should tell the same story.

Catch the smallest failure you can explain. A foreign-key refusal for the deployment owner is different from a duplicate sequence number, a connection failure or a missing organisation. Use the database error code and the named constraint. Do not decide that every failed insert means somebody deleted a service.

A database wrapper may preserve the driver error as its cause rather than putting the code on the outer exception. Cover that shape in the adapter and its tests. Keep the inspection narrow: reading a known code is safer than matching a fragment of a message that may change with the driver or server language.

Once you recognise the specific owner constraint, check whether that exact parent still exists. If it is gone, stop this writer and clear its remaining buffer. If the check fails, you have learned nothing about deletion. Report the original failure rather than turning a database outage into a successful cleanup. Unknown failures deserve an explicit path.

Deployment logs are useful evidence, but collecting them has its own failure states. Write those states down so a later refactor does not fold them into one broad catch. The table below is a design checklist, not a claim that every platform uses the same storage policy.

A retry should have a reason to succeed. A removed parent will not reappear because a timer waited longer. Repeating that insert creates more work and more alerts. A temporary connection problem is different, although even then you need a limit and a clear decision about what happens after the final attempt.

Keep build status separate from log delivery status. A compiler failure belongs in the build result. A storage interruption belongs in the observation of that result. Readers should be able to tell whether the application failed to build or whether part of its output could not be saved. That distinction makes support conversations much shorter.

Attach a deployment identifier and service identifier to an error report. Include the operation that failed and whether the writer had already seen deletion. A trace identifier can help link a failed write with the request that removed its parent, provided your system carries that context across its background jobs.

The OpenTelemetry logs data model provides fields for timestamps, severity, body, attributes and trace context. Use a consistent shape across writers. A structured identifier is easier to follow than a long paragraph that embeds several unrelated values.

Be careful about what enters deployment logs in the first place. Build commands can print repository URLs or environment values. Redact credentials before buffering, keep a bound on individual lines and avoid copying raw output into an alert simply because a write failed. Access to a deployment page should not become access to a secret.

A useful regression test starts with a real failure shape. Make the insert reject with the owner constraint, make the scoped parent lookup return no row, then send more output. The writer should stop attempting inserts. This checks the behaviour that matters rather than merely checking that a helper returns true.

Run the opposite case too. Leave the parent present and verify that the failure is reported. Make the parent lookup fail and verify that it does not count as proof of deletion. Try an unrelated constraint and confirm that the normal error path remains active. These tests protect the boundary that keeps a narrow fix from hiding a wider outage.

Keep existing redaction tests when adding the race tests. A writer can stop correctly after deletion and still leak a token on its normal path. Exercise a normal flush, a final flush and output already queued before deletion. Where your test database supports it, add an integration check using an actual parent and child relationship.

A passing test suite explains the intended behaviour. A production check tells you whether the changed version is running. Record the commit, the deployment identifier and the current service version. A successful push alone does not establish that traffic has moved to the new code.

Check that normal deployment logs still arrive. Use an isolated test service if you need to exercise deletion during a build; do not delete a customer service to reproduce a timing bug. Compare the visible build outcome with its stored result and confirm that a deliberate cleanup has not created a stream of repeated exceptions.

Leave genuine storage failures visible. The goal is a clear distinction between an owner that has intentionally disappeared and a write that failed for another reason. An alert that stops for the right reason is useful. An alert that stops because every exception was ignored leaves the next operator without evidence.

The same pattern appears in exports, imports and other jobs that outlive their dashboard row. Ask who owns each piece of output, how long it should remain, and what happens when that owner disappears. Answer those questions at the storage boundary, not separately in every page that displays the result.

For a broader release checklist, read our guide to a deployment plan. The platform overview explains the Product and service model, while the documentation is the place to check the current interface before automating a change. Keep one source of truth for identifiers and error handling as you connect those parts.

A small, explicit state machine is often enough: accepting output, flushing output, and stopped after confirmed deletion. Keep the transition one-way. Document why that final state exists, retain the tests that distinguish it from an outage, and review it when the retention policy changes. That gives deployment logs a defined end instead of an endless retry loop.