Incident Command for DR Events
Clear authority paths for failover, rollback, and communication under time pressure.
Why Incident Command Matters for DR
A regional outage is not a normal incident. The blast radius is larger, the decisions are harder, and the communication surface is wider. You need a defined command structure: who makes the failover call, who communicates to customers, who manages the technical recovery.
Without this, you get a Slack channel with 30 people typing over each other while the outage extends.
The Command Structure
Incident Commander
Owns the timeline and decisions. Authorizes failover, approves communication, and coordinates between technical and business teams. Usually a senior ops lead or engineering manager.
Technical Lead
Owns the technical recovery. Executes failover procedures, monitors replication state, validates data integrity, and reports progress to the Incident Commander.
Communications Lead
Manages stakeholder updates: customer notifications, status page updates, executive briefings. Uses pre-drafted templates to ensure accurate, timely communication.
Scribe
Documents everything. Timeline of events, decisions made, actions taken, and outcomes. This becomes the foundation of the post-incident review.
Pre-Drafted Communications
During a major incident, writing clear customer communications from scratch is nearly impossible. That's why we prepare templates in advance:
- Initial acknowledgement: "We're aware of an issue and investigating" (sent the moment an incident is declared)
- Impact assessment: what's affected, what's not, estimated timeline (sent as soon as impact is scoped)
- Progress updates: regular cadence updates with specific actions being taken
- Resolution notice: service restored, root cause summary, follow-up timeline
- Post-incident summary: detailed RCA delivered as a standard part of every DR event
The Escalation Path
Clear escalation prevents two failure modes: escalating too late (the outage drags on) and escalating too early (the wrong people are in the room, creating noise).