Incident Response
Respond to active deployment and runtime incidents for Thoth MCP Proxy fleets.
Use this guide during active fleet incidents.
Immediate actions
- Freeze rollout expansion.
- Isolate affected smart groups.
- Collect endpoint health snapshots and install/runtime logs.
- Execute rollback if SLOs remain breached.
- Track all actions in the incident ticket.
Stopping agents
To stop one agent, quarantine it: Thoth denies every action it attempts.
To stop everything at once, pull the kill switch. For the whole workspace it quarantines every
registered agent (SDK agents included) and every active endpoint, and stops running sessions. With
--env it quarantines only that environment's endpoints and sessions, because agents are not tied
to an environment. It needs an admin with a recent second factor, and refuses to run without
--confirm:
From an MCP client, ask for preview_kill_switch: the preview names the agents and endpoints it
would stop and counts the sessions. It acts on the whole workspace, so it needs the whole workspace
chosen explicitly. Commit it with commit_preview; a signed-in user approves in the browser with
MFA. If anything changes after the preview, the commit is refused and you preview again. Admins can
also stop every agent from the Agents page in the dashboard.
To recover from a workspace-wide kill switch, lift it. Lifting releases only the quarantines the
kill switch made: an agent or endpoint quarantined before it, or lifted and quarantined again since,
stays quarantined. Endpoints return to their earlier status; stopped sessions are not restarted.
Without --confirm, thothctl kill-switch lift shows what it would release; with it, it lifts the
kill switch and reads the result back:
The same lift is preview_kill_switch with lift: true over MCP, and "Lift the stop" in the
dashboard. After an environment's kill switch, lift each endpoint's quarantine once it is safe, and
lift any other quarantine the same way:
The kill-switch and endpoints quarantine commands need a thothctl release after 0.3.13;
kill-switch lift needs one after 0.3.14.
Evidence to collect
- Jamf or Intune policy and profile history.
- Package version, signature, and checksum records.
- Endpoint registration and check-in evidence.
- Key rotation or revocation events with actor and timestamp.
- Rollback evidence and recovery verification.
Recovery validation
Confirm:
- Endpoint registration and check-in are healthy.
- Current version and policy profile are correct.
- No unresolved P0 or P1 deployment blockers remain.