Docs
Open in panel → /app · activity
Incident response — operator playbook
The operator surface for an appliance outage or abuse report. Mirrors trust/incident-response but at the depth the appliance runbook actually needs.
5-step playbook
1. Detect
- Sources:
audit_events(panel Activity tab),fbl_events(/v1/fbl/ingest),policy_jobs(Jobs tab),restore-appliance-drill.shlast-run stamp on disk (artifacts/qa/latest/MANIFEST.txt). - Synthetic probe:
bash scripts/smoke-kumo-heavy.shshould exit 0. If not, the appliance is degraded — go to step 2. - Production probe:
make smoke-prod(M20) and/v1/readyagainst the edge.
2. Triage (≤ 4 h ack)
- Identify scope: workspace id, domain(s), IP(s), node id.
- Capture:
audit_eventslast 100 rows + lastpolicy_jobs.kind. Both are visible in the Activity and Jobs tabs respectively. - Severity (self-assigned):
- Sev-1: data-loss, outbound abuse observed, prod
READY5xx. - Sev-2: degraded capability (single workspace/section broken).
- Sev-3: cosmetic or single-tenant.
- Sev-1: data-loss, outbound abuse observed, prod
3. Contain
Inside the appliance only — never hand-edit Kumo Lua.
| Symptom | Action |
|---|---|
| Unverified From slipped through | Confirm REQUIRE_VERIFIED_DOMAIN is on; per-row status in Domains; refer docs/domains.md |
| Credential leak | DELETE /v1/panel/api-keys/:id then reissue; rotate FBL_INGEST_SECRET and re-deploy apps/api |
| Bad IP in pool | POST /v1/panel/jobs/apply-policy to re-render — the panel’s Shaping button bumps the epoch |
| Bounce storm | Pause additional sends by setting wallets.balance=0 for the affected workspace (operator grant reset only after dialog) |
| Panel UI broken | Check artifacts/smoke-prod/latest/ for axe violations; restore last good apps/web build |
4. Notify
- Sev-1:
security@postmta.com+ the customer contact in the agency’sagency_tenantsrow, plus the on-call rotation in the runbook. - Sev-2 / Sev-3: ticket in the appliance issue tracker. We-operate customers get proactive notice within 4 business hours.
5. Recover
- Run
bash scripts/restore-appliance-drill.sh --checkto prove the backup-restore path; if it fails, escalate. - Reset every actor’s API/SMTP credential via Access panel (
DELETE+POST). - Append a row to
audit_eventswith actionincident.resolved.
SLA matrix
| Tier | Acknowledge | Postmortem |
|---|---|---|
| Sev-1 (we-operate) | 30 min | 5 business days |
| Sev-1 (customer VPC) | 1 hour | 10 business days |
| Sev-2 | 4 hours | 10 business days |
| Sev-3 | next business day | next sprint |
Disclosure
- Security disclosures:
security@postmta.com - Privacy:
privacy@postmta.com - Co-ordinated disclosure preferred; we do not pursue legal action against good-faith researchers.
Related
- trust/incident-response — customer-facing copy.
- docs/activity.md —
audit_eventsschema. - docs/jobs.md —
policy_jobsqueue. - docs/hardening-checklist.md — flags to harden to before production.
Last verified: Sat Jul 11 2026 17:00:00 GMT-0700 (Pacific Daylight Time). Cross-link maintained byscripts/check-docs-drift.sh (M54).