Headless E2E Runbook

End-to-end validation runbook for the Thoth headless control plane using thothctl, GovAPI, proxy check-ins, SIEM webhooks, and PAM callback flows.

Use this runbook to validate the headless control plane end to end without relying on dashboard flows.

This runbook answers two questions:

Is the headless control plane working end to end in this environment?
Are thothctl commands sufficient by themselves for full validation?

Scope

Validated surfaces:

tenant settings bootstrap
SIEM/SOAR webhook test path
MDM provider upsert + sync trigger
endpoint registration + check-in telemetry
PAM callback resolution path

Tools used:

thothctl for bootstrap actions
thoth for endpoint/proxy runtime health checks
GovAPI (curl) for readback and deeper verification

Quick answer on command coverage

thothctl now covers bootstrap + operational readbacks + callback simulation, but full E2E validation still requires a few direct API checks.

thothctl command coverage now includes:

tenant settings upsert
optional webhook test call
optional MDM provider upsert
optional MDM sync trigger
settings readback (settings get)
MDM provider inventory (mdm list)
endpoint inventory and stats (endpoints list, endpoints stats)
approval queue reads (approvals list)
PAM callback signing + send flow (pam callback)

You still need GovAPI/API calls for:

explicit check-in assertions
full operational audits and policy/runtime evidence checks

Prerequisites

thoth and thothctl are installed from the current release.
GovAPI URL is reachable over HTTPS for non-local environments.
Admin token is available:
- recommended: file-backed token (/run/secrets/thoth_admin_jwt)
For local demo only:
- GovAPI can run with GOVAPI_SKIP_AUTH=true from demo compose config.

Set baseline env vars:

export THOTH_TENANT_ID="<tenant>"
export THOTH_APEX_DOMAIN="atensecurity.com"
export THOTH_ADMIN_BEARER_TOKEN_FILE="/run/secrets/thoth_admin_jwt"
export THOTH_WEBHOOK_URL="https://siem.company.com/hooks/thoth"
export THOTH_WEBHOOK_SECRET="<secret>"

Optional explicit override:

export THOTH_GOVAPI_URL="https://grid.<tenant>.<apex-domain>"

Validate binaries:

thoth --version
thothctl --version

Phase 1: Bootstrap with thothctl

Run:

thothctl bootstrap \
  --tenant-id "$THOTH_TENANT_ID" \
  --apex-domain "$THOTH_APEX_DOMAIN" \
  --auth-token-file "$THOTH_ADMIN_BEARER_TOKEN_FILE" \
  --compliance-profile "soc2" \
  --shadow-low allow \
  --shadow-medium step_up \
  --shadow-high block \
  --shadow-critical block \
  --webhook-url "$THOTH_WEBHOOK_URL" \
  --webhook-secret "$THOTH_WEBHOOK_SECRET" \
  --test-webhook \
  --mdm-provider "jamf" \
  --mdm-config-file "./jamf-provider.json" \
  --start-sync \
  --json

Expected JSON includes:

settings_updated=true
webhook_tested=true
mdm_provider_upserted=true
mdm_sync_started=true

Phase 1A: Create scoped API keys with least privilege

Create endpoint-scoped key for runtime health + policy operations:

thothctl api-keys create \
  --tenant-id "$THOTH_TENANT_ID" \
  --apex-domain "$THOTH_APEX_DOMAIN" \
  --auth-token-file "$THOTH_ADMIN_BEARER_TOKEN_FILE" \
  --name "rightway-endpoint-e2e" \
  --scope-level endpoint \
  --scope-target-id "ep-e2e-01" \
  --permission read \
  --permission execute \
  --json

List scoped keys:

thothctl api-keys list \
  --tenant-id "$THOTH_TENANT_ID" \
  --apex-domain "$THOTH_APEX_DOMAIN" \
  --auth-token-file "$THOTH_ADMIN_BEARER_TOKEN_FILE" \
  --json

Validate positive authorization decision:

thothctl api-keys authorize \
  --tenant-id "$THOTH_TENANT_ID" \
  --apex-domain "$THOTH_APEX_DOMAIN" \
  --auth-token-file "$THOTH_ADMIN_BEARER_TOKEN_FILE" \
  --key-id "<key-id>" \
  --api-key-file "/run/secrets/thoth_scoped_api_key" \
  --permission execute \
  --resource-type endpoint \
  --resource-id "ep-e2e-01" \
  --json

Validate negative scope decision (different endpoint):

thothctl api-keys authorize \
  --tenant-id "$THOTH_TENANT_ID" \
  --apex-domain "$THOTH_APEX_DOMAIN" \
  --auth-token-file "$THOTH_ADMIN_BEARER_TOKEN_FILE" \
  --key-id "<key-id>" \
  --api-key-file "/run/secrets/thoth_scoped_api_key" \
  --permission execute \
  --resource-type endpoint \
  --resource-id "ep-other-99" \
  --json

Expected:

positive check: valid=true, permission_allowed=true, scope_allowed=true
negative check: scope_allowed=false

Revoke key after validation:

thothctl api-keys revoke \
  --tenant-id "$THOTH_TENANT_ID" \
  --apex-domain "$THOTH_APEX_DOMAIN" \
  --auth-token-file "$THOTH_ADMIN_BEARER_TOKEN_FILE" \
  --key-id "<key-id>" \
  --json

Phase 2: Settings + MDM readback (GovAPI verification)

Add auth header for non-local:

THOTH_GOVAPI_URL="${THOTH_GOVAPI_URL:-https://grid.${THOTH_TENANT_ID}.${THOTH_APEX_DOMAIN}}"
AUTH="Authorization: Bearer $(cat "$THOTH_ADMIN_BEARER_TOKEN_FILE")"
BASE="$THOTH_GOVAPI_URL/$THOTH_TENANT_ID/thoth"

Verify settings:

thothctl settings get \
  --tenant-id "$THOTH_TENANT_ID" \
  --apex-domain "$THOTH_APEX_DOMAIN" \
  --auth-token-file "$THOTH_ADMIN_BEARER_TOKEN_FILE" \
  --json

Verify webhook test endpoint independently:

curl -sS -X POST -H "$AUTH" "$BASE/settings/webhook/test" | jq .

Verify MDM provider inventory:

thothctl mdm list \
  --tenant-id "$THOTH_TENANT_ID" \
  --apex-domain "$THOTH_APEX_DOMAIN" \
  --auth-token-file "$THOTH_ADMIN_BEARER_TOKEN_FILE" \
  --json

Trigger sync explicitly (optional second run):

thothctl mdm sync \
  --tenant-id "$THOTH_TENANT_ID" \
  --apex-domain "$THOTH_APEX_DOMAIN" \
  --auth-token-file "$THOTH_ADMIN_BEARER_TOKEN_FILE" \
  --provider jamf \
  --json

Acceptance:

provider exists and is enabled
sync call returns accepted/success state

Phase 3: Endpoint registration and check-ins

Option A (recommended): proxy-driven registration/check-in

Run governed proxy path and then check:

thoth health --json

Then verify endpoint appears:

thothctl endpoints list \
  --tenant-id "$THOTH_TENANT_ID" \
  --apex-domain "$THOTH_APEX_DOMAIN" \
  --auth-token-file "$THOTH_ADMIN_BEARER_TOKEN_FILE" \
  --json
 
thothctl endpoints stats \
  --tenant-id "$THOTH_TENANT_ID" \
  --apex-domain "$THOTH_APEX_DOMAIN" \
  --auth-token-file "$THOTH_ADMIN_BEARER_TOKEN_FILE" \
  --json

Option B (API smoke): manual registration + check-in

curl -sS -X POST -H "Content-Type: application/json" \
  "$BASE/endpoints" \
  -d '{
    "endpoint_id":"ep-e2e-01",
    "hostname":"ep-e2e-01.local",
    "employee_email":"secops@example.com",
    "enrollment":"manual",
    "environment":"prod",
    "proxy_version":"0.2.8",
    "agent_ids":["filesystem-safe"]
  }' | jq .

curl -sS -X POST -H "Content-Type: application/json" \
  "$BASE/endpoints/ep-e2e-01/checkin" \
  -d '{
    "proxy_version":"0.2.8",
    "agent_ids":["filesystem-safe"],
    "risk_score":10,
    "violations_today":0,
    "sessions_today":1
  }' | jq .

Acceptance:

endpoint is returned by GET /endpoints
stats reflect total/online increments

Phase 4: PAM callback path validation

1) Configure PAM in tenant settings

Ensure settings include:

pam.enabled=true
pam.provider
pam.callback_secret
pam.request_url (HTTPS)

2) Simulate callback signature

thothctl pam callback \
  --tenant-id "$THOTH_TENANT_ID" \
  --apex-domain "$THOTH_APEX_DOMAIN" \
  --auth-token-file "$THOTH_ADMIN_BEARER_TOKEN_FILE" \
  --approval-id "<approval-id>" \
  --request-id "<request-id>" \
  --decision approve \
  --json

Dry run (generate signature/payload without sending):

thothctl pam callback \
  --tenant-id "$THOTH_TENANT_ID" \
  --apex-domain "$THOTH_APEX_DOMAIN" \
  --auth-token-file "$THOTH_ADMIN_BEARER_TOKEN_FILE" \
  --approval-id "<approval-id>" \
  --request-id "<request-id>" \
  --decision approve \
  --dry-run \
  --json

Acceptance:

valid callback resolves approval
replayed callback is safely ignored
mismatched request_id returns conflict and does not resolve

Phase 5: Evidence checks (headless operational validation)

Run:

thothctl endpoints stats \
  --tenant-id "$THOTH_TENANT_ID" \
  --apex-domain "$THOTH_APEX_DOMAIN" \
  --auth-token-file "$THOTH_ADMIN_BEARER_TOKEN_FILE" \
  --json
 
thothctl approvals list \
  --tenant-id "$THOTH_TENANT_ID" \
  --apex-domain "$THOTH_APEX_DOMAIN" \
  --auth-token-file "$THOTH_ADMIN_BEARER_TOKEN_FILE" \
  --status pending \
  --json
 
curl -sS -H "$AUTH" "$BASE/alerts" | jq .
curl -sS -H "$AUTH" "$BASE/governance/feed" | jq .

Expected:

endpoint telemetry reflects active check-ins
governance feed shows routed decisions/events
alerts reflect violations or risk changes as scenarios trigger

Fail criteria

Fail the run if any of these occur:

thothctl bootstrap does not return success flags
webhook test cannot reach destination
MDM sync cannot be triggered
endpoint registration/check-in does not update stats
PAM callback accepts invalid signature or mismatched request correlation

Command coverage matrix

Capability	`thothctl`	GovAPI/API
Tenant settings upsert	✅	✅
Webhook test trigger	✅	✅
MDM provider upsert	✅	✅
MDM sync trigger	✅	✅
Scoped key create/authorize/revoke	✅	✅
Endpoint list/stats verification	✅	✅
Explicit endpoint check-in assertions	❌	✅
PAM callback correlation and replay tests	✅	✅
Approval queue reads	✅	✅
Audit feed and deep evidence checks	❌	✅

thothctl is the right CLI for bootstrap and config application.
GovAPI verification remains required for comprehensive end-to-end operational testing.

Headless E2E Runbook

On this page