Engineering note / Design proposal
Recovery evidence collection
A proposed bounded diagnostic collector for recovery support.
Operations / Automation
Purpose and scope
Evidence collection must support incident investigation without delaying an established recovery procedure. The collector therefore requires explicit limits on execution time, resource use and output volume.
This article proposes a bounded diagnostic collector. It describes design requirements rather than a deployed implementation.
Evidence scope
- An incident identifier, UTC time, deployment version and relevant recent changes.
- Service and dependency state, restart history and bounded health-check output.
- A limited window of logs and resource, connection and queue diagnostics.
- A manifest of files, hashes and errors encountered during collection.
Execution limits
Each command requires a timeout. Log extraction requires a time window and output cap. Individual failures should be recorded without preventing subsequent checks.
Collection must not restart services, edit configuration or impose significant load. Service-specific commands require an owner-reviewed runbook and rehearsal.
Output handling
Diagnostic output may contain credentials, personal information and internal configuration. Access controls, redaction, protected storage and retention form part of the design.
Memory dumps and forensic images require separate collection procedures because of their data exposure and availability impact.
Collection contract
The proposed disconnected-appliance profile uses an allow-list of low-cost metadata: version, uptime, service state and bounded connection or queue counters. Payloads, private keys and unrestricted configuration dumps are excluded by default.
A total deadline, output budget and free-space reserve supplement per-command limits. The manifest records attempted checks, durations, completion status and truncation. Collection writes a diagnostic bundle but leaves the service and its configuration unchanged.
Hashes support comparison with a trusted baseline. Transfer across a security boundary remains subject to the environment's approved handling process, including where the bundle has been redacted or encrypted.
Recovery verification
Recovery verification should exercise a user operation, dependency behaviour and data integrity. Incident records should identify evidence lost during restart or rollback.
Proposed failure tests cover hung commands, insufficient permissions, output limits and unavailable storage.
Examples are generalised or synthetic and omit operational details and identifying information.
← All engineering notes