Skip to main content
Back to engineering notes

Engineering note / Design proposal

Recovery evidence collection

A proposed bounded diagnostic collector for recovery support.

Operations / Automation

Purpose and scope

Evidence collection must support incident investigation without delaying an established recovery procedure. The collector therefore requires explicit limits on execution time, resource use and output volume.

This article proposes a bounded diagnostic collector. It describes design requirements rather than a deployed implementation.

Evidence scope

  • An incident identifier, UTC time, deployment version and relevant recent changes.
  • Service and dependency state, restart history and bounded health-check output.
  • A limited window of logs and resource, connection and queue diagnostics.
  • A manifest of files, hashes and errors encountered during collection.

Execution limits

Each command requires a timeout. Log extraction requires a time window and output cap. Individual failures should be recorded without preventing subsequent checks.

Collection must not restart services, edit configuration or impose significant load. Service-specific commands require an owner-reviewed runbook and rehearsal.

Output handling

Diagnostic output may contain credentials, personal information and internal configuration. Access controls, redaction, protected storage and retention form part of the design.

Memory dumps and forensic images require separate collection procedures because of their data exposure and availability impact.

Collection contract

The proposed disconnected-appliance profile uses an allow-list of low-cost metadata: version, uptime, service state and bounded connection or queue counters. Payloads, private keys and unrestricted configuration dumps are excluded by default.

A total deadline, output budget and free-space reserve supplement per-command limits. The manifest records attempted checks, durations, completion status and truncation. Collection writes a diagnostic bundle but leaves the service and its configuration unchanged.

Hashes support comparison with a trusted baseline. Transfer across a security boundary remains subject to the environment's approved handling process, including where the bundle has been redacted or encrypted.

Recovery verification

Recovery verification should exercise a user operation, dependency behaviour and data integrity. Incident records should identify evidence lost during restart or rollback.

Proposed failure tests cover hung commands, insufficient permissions, output limits and unavailable storage.

Examples are generalised or synthetic and omit operational details and identifying information.

← All engineering notes