Headless E2E Runbook

End-to-end validation runbook for the Thoth headless control plane using thothctl, the Thoth Control Plane API, proxy check-ins, SIEM webhooks, and PAM callback flows.

Use this runbook to validate the headless control plane end to end without relying on dashboard flows.

This runbook answers two questions:

Is the headless control plane working end to end in this environment?
Are thothctl commands sufficient by themselves for full validation?

Scope

Validated surfaces:

tenant settings bootstrap
SIEM/SOAR webhook test path
MDM provider upsert + sync trigger
browser provider/policy/enrollment control-plane state
endpoint registration + check-in telemetry
PAM callback resolution path

Tools used:

thothctl for bootstrap actions
thoth for endpoint/proxy runtime health checks
Thoth Control Plane API (curl) for readback and deeper verification

Quick answer on command coverage

thothctl now covers bootstrap + operational readbacks + callback simulation, but full E2E validation still requires a few direct API checks.

thothctl command coverage now includes:

tenant settings upsert
optional webhook test call
optional MDM provider upsert
optional MDM sync trigger
browser providers list/upsert (browser providers list|upsert)
browser policies list/upsert (browser policies list|upsert)
browser enrollments list/upsert (browser enrollments list|upsert)
browser endpoint policy sync/apply (browser sync)
settings readback (settings get)
MDM provider inventory (mdm list)
endpoint inventory and stats (endpoints list, endpoints stats)
approval queue reads (approvals list)
PAM callback signing + send flow (pam callback)
evidence SLO readback (evidence slos)
governance evidence materialization (evidence backfill)
decision field backfill (evidence decision-backfill)

You still need direct API calls for:

explicit check-in assertions
full operational audits and policy/runtime evidence checks

Prerequisites

thoth and thothctl are installed from the current release.
Thoth Control Plane API URL is reachable over HTTPS for non-local environments.
Admin auth session is available:
- recommended: file-backed session ($HOME/.thoth/admin-token.jwt)

Set baseline env vars:

export THOTH_TENANT_ID="<tenant>"
export THOTH_APEX_DOMAIN="<apex-domain>"
export THOTH_AUTH_SESSION_FILE="$HOME/.thoth/admin-token.jwt"
export THOTH_WEBHOOK_URL="https://siem.company.com/hooks/thoth"
export THOTH_WEBHOOK_SECRET="<secret>"

Optional explicit override:

export THOTH_GOVAPI_URL="https://grid.<tenant>.<apex-domain>"

Validate binaries:

thoth --version
thothctl --version

Create or refresh your admin auth session:

thothctl auth login \
  --tenant-id "$THOTH_TENANT_ID" \
  --admin-email "<admin@company.com>"

--apex-domain and --auth-token-file are optional overrides. For non-interactive pipelines, prefer org API key auth (THOTH_API_KEY or --org-api-key).

Phase 1: Bootstrap with thothctl

Run:

thothctl bootstrap \
  --tenant-id "$THOTH_TENANT_ID" \
  --compliance-profile "soc2" \
  --shadow-low allow \
  --shadow-medium step_up \
  --shadow-high block \
  --shadow-critical block \
  --webhook-url "$THOTH_WEBHOOK_URL" \
  --webhook-secret "$THOTH_WEBHOOK_SECRET" \
  --test-webhook \
  --mdm-provider "jamf" \
  --mdm-config-file "./jamf-provider.json" \
  --start-sync \
  --json

Expected JSON includes:

settings_updated=true
webhook_tested=true
mdm_provider_upserted=true
mdm_sync_started=true

Phase 1A: Create scoped API keys with least privilege

Create endpoint-scoped key for runtime health + policy operations:

thothctl api-keys create \
  --tenant-id "$THOTH_TENANT_ID" \
  --name "tenant-endpoint-e2e" \
  --endpoint-id "ep-e2e-01" \
  --permission read \
  --permission execute \
  --json

List scoped keys:

thothctl api-keys list \
  --tenant-id "$THOTH_TENANT_ID" \
  --json

Validate positive authorization decision:

thothctl api-keys authorize \
  --tenant-id "$THOTH_TENANT_ID" \
  --key-id "<key-id>" \
  --api-key-file "/run/secrets/thoth_scoped_api_key" \
  --permission execute \
  --endpoint-id "ep-e2e-01" \
  --json

Validate negative scope decision (different endpoint):

thothctl api-keys authorize \
  --tenant-id "$THOTH_TENANT_ID" \
  --key-id "<key-id>" \
  --api-key-file "/run/secrets/thoth_scoped_api_key" \
  --permission execute \
  --endpoint-id "ep-other-99" \
  --json

Expected:

positive check: valid=true, permission_allowed=true, scope_allowed=true
negative check: scope_allowed=false

Revoke key after validation:

thothctl api-keys revoke \
  --tenant-id "$THOTH_TENANT_ID" \
  --key-id "<key-id>" \
  --json

Phase 2: Settings + MDM readback (Control Plane API verification)

Add auth header for non-local:

THOTH_GOVAPI_URL="${THOTH_GOVAPI_URL:-https://grid.${THOTH_TENANT_ID}.${THOTH_APEX_DOMAIN}}"
AUTH="Authorization: Bearer $(cat "$THOTH_AUTH_SESSION_FILE")"
BASE="$THOTH_GOVAPI_URL/$THOTH_TENANT_ID/thoth"

Verify settings:

thothctl settings get \
  --tenant-id "$THOTH_TENANT_ID" \
  --json

Verify webhook test endpoint independently:

curl -sS -X POST -H "$AUTH" "$BASE/settings/webhook/test" | jq .

Verify MDM provider inventory:

thothctl mdm list \
  --tenant-id "$THOTH_TENANT_ID" \
  --json

Trigger sync explicitly (optional second run):

thothctl mdm sync \
  --tenant-id "$THOTH_TENANT_ID" \
  --provider jamf \
  --json

Acceptance:

provider exists and is enabled
sync call returns accepted/success state

Phase 2B: Browser control-plane validation

Upsert browser provider and policy:

thothctl browser providers upsert \
  --tenant-id "$THOTH_TENANT_ID" \
  --provider chrome \
  --name "Chrome Enterprise" \
  --config-file "./browser/chrome-provider.json" \
  --json
 
thothctl browser policies upsert \
  --tenant-id "$THOTH_TENANT_ID" \
  --provider chrome \
  --name "chrome-baseline-v1" \
  --enforcement-mode enforce \
  --policy-file "./browser/chrome-policy.json" \
  --json

Validate list/readbacks:

thothctl browser providers list \
  --tenant-id "$THOTH_TENANT_ID" \
  --json
 
thothctl browser policies list \
  --tenant-id "$THOTH_TENANT_ID" \
  --provider chrome \
  --json

Enroll one browser identity:

thothctl browser enrollments upsert \
  --tenant-id "$THOTH_TENANT_ID" \
  --provider chrome \
  --user-id "secops@example.com" \
  --device-id "ep-e2e-01" \
  --status active \
  --metadata-json '{"source":"mdm"}' \
  --json

Run endpoint policy sync/apply:

# Dry-run preview
thothctl browser sync \
  --tenant-id "$THOTH_TENANT_ID" \
  --provider chrome \
  --user-id "secops@example.com" \
  --device-id "ep-e2e-01" \
  --dry-run \
  --json
 
# Apply policy artifacts
thothctl browser sync \
  --tenant-id "$THOTH_TENANT_ID" \
  --provider chrome \
  --user-id "secops@example.com" \
  --device-id "ep-e2e-01" \
  --json

Acceptance:

browser provider is present and enabled
browser policy compiles and is returned by list/readback
enrollment record is present for expected user/device/provider tuple
sync output reports applied, rendered, or dry_run per provider with target paths

Phase 3: Endpoint registration and check-ins

Option A (recommended): proxy-driven registration/check-in

Run governed proxy path and then check:

thoth health --json

Then verify endpoint appears:

thothctl endpoints list \
  --tenant-id "$THOTH_TENANT_ID" \
  --json
 
thothctl endpoints stats \
  --tenant-id "$THOTH_TENANT_ID" \
  --json

Option B (API smoke): manual registration + check-in

curl -sS -X POST -H "Content-Type: application/json" \
  "$BASE/endpoints" \
  -d '{
    "endpoint_id":"ep-e2e-01",
    "hostname":"ep-e2e-01.local",
    "employee_email":"secops@example.com",
    "enrollment":"manual",
    "environment":"prod",
    "proxy_version":"0.2.19",
    "agent_ids":["filesystem-safe"]
  }' | jq .

curl -sS -X POST -H "Content-Type: application/json" \
  "$BASE/endpoints/ep-e2e-01/checkin" \
  -d '{
    "proxy_version":"0.2.19",
    "agent_ids":["filesystem-safe"],
    "risk_score":10,
    "violations_today":0,
    "sessions_today":1
  }' | jq .

Acceptance:

endpoint is returned by GET /endpoints
stats reflect total/online increments

Phase 4: PAM callback path validation

1) Configure PAM in tenant settings

Ensure settings include:

pam.enabled=true
pam.provider
pam.callback_secret
pam.request_url (HTTPS)

2) Simulate callback signature

thothctl pam callback \
  --tenant-id "$THOTH_TENANT_ID" \
  --approval-id "<approval-id>" \
  --request-id "<request-id>" \
  --decision approve \
  --json

Dry run (generate signature/payload without sending):

thothctl pam callback \
  --tenant-id "$THOTH_TENANT_ID" \
  --approval-id "<approval-id>" \
  --request-id "<request-id>" \
  --decision approve \
  --dry-run \
  --json

Acceptance:

valid callback resolves approval
replayed callback is safely ignored
mismatched request_id returns conflict and does not resolve

Phase 5: Evidence checks (headless operational validation)

Run:

thothctl endpoints stats \
  --tenant-id "$THOTH_TENANT_ID" \
  --json
 
thothctl approvals list \
  --tenant-id "$THOTH_TENANT_ID" \
  --status pending \
  --json
 
thothctl evidence slos \
  --tenant-id "$THOTH_TENANT_ID" \
  --window-hours 24 \
  --json
 
thothctl evidence decision-backfill \
  --tenant-id "$THOTH_TENANT_ID" \
  --window-hours 720 \
  --dry-run \
  --json
 
curl -sS -H "$AUTH" "$BASE/alerts" | jq .
curl -sS -H "$AUTH" "$BASE/governance/feed" | jq .

Expected:

endpoint telemetry reflects active check-ins
governance feed shows routed decisions/events
evidence SLO report returns non-empty coverage metrics
alerts reflect violations or risk changes as scenarios trigger

Fail criteria

Fail the run if any of these occur:

thothctl bootstrap does not return success flags
webhook test cannot reach destination
MDM sync cannot be triggered
endpoint registration/check-in does not update stats
PAM callback accepts invalid signature or mismatched request correlation

Command coverage matrix

Capability	`thothctl`	Control Plane API
Tenant settings upsert	✅	✅
Webhook test trigger	✅	✅
MDM provider upsert	✅	✅
MDM sync trigger	✅	✅
Browser provider/policy/enrollment upsert	✅	✅
Browser provider/policy/enrollment readback	✅	✅
Browser policy sync/apply summary	✅	❌
Scoped key create/authorize/revoke	✅	✅
Endpoint list/stats verification	✅	✅
Explicit endpoint check-in assertions	❌	✅
PAM callback correlation and replay tests	✅	✅
Approval queue reads	✅	✅
Evidence SLO readbacks	✅	✅
Evidence and decision-field backfill triggers	✅	✅
Audit feed and deep evidence checks	❌	✅

thothctl is the right CLI for bootstrap and config application. Control Plane API verification remains required for comprehensive end-to-end operational testing.

Headless E2E Runbook

On this page