Popular starting points. Use Tab or arrow keys to browse results.

Skip to content
Aten SecurityThoth docs

Incident Response

Respond to active deployment and runtime incidents for Thoth MCP Proxy fleets.

Use this guide during active fleet incidents.

Immediate actions

  1. Freeze rollout expansion.
  2. Isolate affected smart groups.
  3. Collect endpoint health snapshots and install/runtime logs.
  4. Execute rollback if SLOs remain breached.
  5. Track all actions in the incident ticket.

Stopping agents

To stop one agent, quarantine it: Thoth denies every action it attempts.

thothctl agents quarantine set --agent-id billing-agent

To stop everything at once, pull the kill switch. For the whole workspace it quarantines every registered agent (SDK agents included) and every active endpoint, and stops running sessions. With --env it quarantines only that environment's endpoints and sessions, because agents are not tied to an environment. It needs an admin with a recent second factor, and refuses to run without --confirm:

thothctl kill-switch --confirm             # the whole workspace
thothctl kill-switch --env prod --confirm  # prod endpoints only

From an MCP client, ask for preview_kill_switch: the preview names the agents and endpoints it would stop and counts the sessions. It acts on the whole workspace, so it needs the whole workspace chosen explicitly. Commit it with commit_preview; a signed-in user approves in the browser with MFA. If anything changes after the preview, the commit is refused and you preview again. Admins can also stop every agent from the Agents page in the dashboard.

To recover from a workspace-wide kill switch, lift it. Lifting releases only the quarantines the kill switch made: an agent or endpoint quarantined before it, or lifted and quarantined again since, stays quarantined. Endpoints return to their earlier status; stopped sessions are not restarted. Without --confirm, thothctl kill-switch lift shows what it would release; with it, it lifts the kill switch and reads the result back:

thothctl kill-switch lift            # what it would release
thothctl kill-switch lift --confirm  # lift it

The same lift is preview_kill_switch with lift: true over MCP, and "Lift the stop" in the dashboard. After an environment's kill switch, lift each endpoint's quarantine once it is safe, and lift any other quarantine the same way:

thothctl agents quarantine lift --agent-id billing-agent
thothctl endpoints quarantine lift --endpoint-id laptop-prod

The kill-switch and endpoints quarantine commands need a thothctl release after 0.3.13; kill-switch lift needs one after 0.3.14.

Evidence to collect

  • Jamf or Intune policy and profile history.
  • Package version, signature, and checksum records.
  • Endpoint registration and check-in evidence.
  • Key rotation or revocation events with actor and timestamp.
  • Rollback evidence and recovery verification.

Recovery validation

thoth health --json

Confirm:

  • Endpoint registration and check-in are healthy.
  • Current version and policy profile are correct.
  • No unresolved P0 or P1 deployment blockers remain.

On this page