Incident Response Guide

Version 1.0 – 27 July 2025

1. Why This Guide Exists

This guide walks the YieldGuard team through every step of handling big problems that could hurt our users’ money, our app, or our legal status. The problems we care about include hacked smart‑contracts, bad price feeds (oracles), broken servers, and rule‑breaking customers.

2. How Bad Is the Problem? (Severity Levels)

LevelWhat It MeansExamplesTime Goals
SEV-0 (Critical)Money is being stolen right now or the law stops us from working.Hacker drains a contract; proof we are short of funds; our multisig wallet is taken.Confirm in 5 min, stop damage in 15 min, tell public in 1 hour
SEV-1 (High)The app is breaking for most users or a hacker could copy a working exploit soon.Auto-pause triggered; serious bug bounty; leaked AWS keys.Confirm in 15 min, stop damage in 1 hour, tell public in 4 hours
SEV-2 (Medium)Annoying but not dangerous; can wait one day.Slow indexer; small UI glitch; sanctions list update.Confirm in 1 hour, fix in 24 hours
SEV-3 (Low)Cosmetic mistakes.Typos, broken link.Fix in next sprint

3. Who Does What

RoleMain PersonBackupJob
Incident Commander (IC)On-call security engineerBack-end dev leadPicks severity, runs war-room, assigns tasks
Communications LeadHead of OperationsCEOWrites updates for staff, users, and press
Chain LeadSmart-contract leadCTOSends pause or emergency transactions
Infrastructure LeadDevOps engineerSREFixes cloud servers, rotates keys
Compliance LiaisonCompliance officerCFOTells regulators and bank partners
ScribeOps supportAny engineerWrites timeline and keeps notes

PagerDuty makes sure IC and Infra Lead are awake 24/7.

4. Life of an Incident

4.1 Detect & Triage

  1. Alerts come from contracts, price feeds, AI risk scores, PagerDuty, bounty hunters, or user reports.
  2. IC checks the alert and sets a first severity level.
  3. Open Slack channel `#warroom-<id>` and Zoom call; invite leads.
  4. Scribe starts the incident log in Notion.

4.2 Contain

  • **Contract hack** – Chain Lead pauses deposits or calls emergency withdraw with high gas.
  • **Bad price feed** – Pause deposits, switch to backup price, limit withdrawals.
  • **Key leak** – Change all exposed secrets and kill old sessions.
  • Check metrics to be sure the bleeding has stopped.

4.3 Eradicate & Fix

  • Patch the code, run tests, and get a quick audit review.
  • Deploy the upgrade with multisig. Skip the timelock only if funds are in danger.
  • Rebuild any hacked servers with least‑privilege settings.

4.4 Recover & Verify

  • Run the full test suite on staging.
  • Simulate deposits and withdrawals; check net asset value (NAV).
  • Remove the pause only when IC, Chain Lead, and Compliance agree.

4.5 Learn & Improve

  1. Draft a post-mortem within 48 hours.
  2. Hold a blame-free meeting: what broke, what worked, what to improve.
  3. Publish the report to the repo and Statuspage.
  4. Track action items in JIRA and close on time.

5. Communication Plan

5.1 Inside the Team

  • Talk in Slack `#warroom-<id>`.
  • Zoom call is recorded.
  • IC posts status every hour during SEV-0/1.

5.2 Outside

ChannelWho Sees ItHow Often
StatuspageEveryoneFirst post in 1 hour for SEV-0/1; then every 2 hours
EmailInvestorsFirst email in 2 hours; summary when fixed
Twitter/XCrypto communityPost after Statuspage to avoid front-running
DiscordUsersMirror Statuspage updates

5.3 Regulators & Partners

  • Call and email AIFM & Depositary within 2 hours (SEV-0/1).
  • Send an incident form to CSSF within 24 hours if NAV is hit.
  • Call police only for crime (theft, extortion).

6. Quick Playbooks

6.1 Smart-Contract Hack

  1. Pause deposits; record TVL.
  2. Trace the attack on Tenderly; estimate loss risk.
  3. Patch and deploy the fix (skip timelock if funds still leaking).
  4. Ask white-hat helpers if a rescue is needed.

6.2 Oracle Trouble

  1. Auto-pause.
  2. Switch to the last good price.
  3. Talk to Chainlink; watch proof-of-reserve.
  4. Resume when the feed is fresh (<15 min) and proof is good.

6.3 KYC / Sanctions Issue

  1. Bot flags wallet; burn its KYC pass.
  2. Freeze deposits; notify AIFM.
  3. File a SAR if the wallet is on the OFAC/UN list.

6.4 Server Hack

  1. Revoke IAM keys; rotate all secrets.
  2. Save a forensic snapshot; give it to the Security Lead.
  3. Re-deploy clean infrastructure with Terraform.

7. Tools We Use

  • PagerDuty – alerts and on-call
  • Slack – chat and war-room creation
  • Statuspage.io – public status
  • Safe (Gnosis) – multisig actions
  • Tenderly / Etherscan – trace transactions
  • Grafana & Prometheus – metrics
  • Notion – logs and post-mortems
  • JIRA – track fixes

8. Goals & Metrics

MetricTarget
Detect Time<5 min for SEV-0
Contain Time<15 min for SEV-0
Post-mortem done100% within 48h
Action items closed90% within 30 days