# Headless E2E Runbook

> End-to-end validation runbook for the Thoth headless control plane using thothctl, the Thoth Control Plane API, proxy check-ins, SIEM webhooks, and PAM callback flows.

Source: https://docs.atensecurity.com/docs/operations/headless-e2e-runbook

Use this runbook to validate the **headless control plane** end to end without relying on dashboard flows.

This runbook answers two questions:

1. Is the headless control plane working end to end in this environment?
2. Are `thothctl` commands sufficient by themselves for full validation?

---

## Scope

Validated surfaces:

- tenant settings bootstrap
- SIEM/SOAR webhook test path
- MDM provider upsert + sync trigger
- browser provider/policy/enrollment control-plane state
- endpoint registration + check-in telemetry
- PAM callback resolution path

Tools used:

- `thothctl` for bootstrap actions
- `thoth` for endpoint/proxy runtime health checks
- Thoth Control Plane API (`curl`) for readback and deeper verification

---

## Quick answer on command coverage

`thothctl` now covers **bootstrap + operational readbacks + callback simulation**, but full E2E validation still requires a few direct API checks.

`thothctl` command coverage now includes:

- tenant settings upsert
- optional webhook test call
- optional MDM provider upsert
- optional MDM sync trigger
- browser providers list/upsert (`browser providers list|upsert`)
- browser policies list/upsert (`browser policies list|upsert`)
- browser enrollments list/upsert (`browser enrollments list|upsert`)
- browser endpoint policy sync/apply (`browser sync`)
- settings readback (`settings get`)
- MDM provider inventory (`mdm list`)
- endpoint inventory and stats (`endpoints list`, `endpoints stats`)
- approval queue reads (`approvals list`)
- PAM callback signing + send flow (`pam callback`)
- evidence SLO readback (`evidence slos`)
- governance evidence materialization (`evidence backfill`)
- decision field backfill (`evidence decision-backfill`)

You still need direct API calls for:

- explicit check-in assertions
- full operational audits and policy/runtime evidence checks

---

## Prerequisites

1. `thoth` and `thothctl` are installed from the current release.
2. Thoth Control Plane API URL is reachable over HTTPS for non-local environments.
3. Admin auth session is available:
   - recommended: file-backed session (`$HOME/.thoth/admin-token.jwt`)

Set baseline env vars:

```bash
export THOTH_TENANT_ID="<tenant>"
export THOTH_APEX_DOMAIN="<apex-domain>"
export THOTH_AUTH_SESSION_FILE="$HOME/.thoth/admin-token.jwt"
export THOTH_WEBHOOK_URL="https://siem.company.com/hooks/thoth"
export THOTH_WEBHOOK_SECRET="<secret>"
```

Optional explicit override:

```bash
export THOTH_GOVAPI_URL="https://grid.<tenant>.<apex-domain>"
```

Validate binaries:

```bash
thoth --version
thothctl --version
```

Create or refresh your admin auth session:

```bash
thothctl auth login \
  --tenant-id "$THOTH_TENANT_ID" \
  --admin-email "<admin@company.com>"
```

`--apex-domain` and `--auth-token-file` are optional overrides.
For non-interactive pipelines, prefer org API key auth (`THOTH_API_KEY` or `--org-api-key`).

---

## Phase 1: Bootstrap with thothctl

Run:

```bash
thothctl bootstrap \
  --tenant-id "$THOTH_TENANT_ID" \
  --compliance-profile "soc2" \
  --shadow-low allow \
  --shadow-medium step_up \
  --shadow-high block \
  --shadow-critical block \
  --webhook-url "$THOTH_WEBHOOK_URL" \
  --webhook-secret "$THOTH_WEBHOOK_SECRET" \
  --test-webhook \
  --mdm-provider "jamf" \
  --mdm-config-file "./jamf-provider.json" \
  --start-sync \
  --json
```

Expected JSON includes:

- `settings_updated=true`
- `webhook_tested=true`
- `mdm_provider_upserted=true`
- `mdm_sync_started=true`

---

## Phase 1A: Create scoped API keys with least privilege

Create endpoint-scoped key for runtime health + policy operations:

```bash
thothctl api-keys create \
  --tenant-id "$THOTH_TENANT_ID" \
  --name "tenant-endpoint-e2e" \
  --endpoint-id "ep-e2e-01" \
  --permission read \
  --permission execute \
  --json
```

List scoped keys:

```bash
thothctl api-keys list \
  --tenant-id "$THOTH_TENANT_ID" \
  --json
```

Validate positive authorization decision:

```bash
thothctl api-keys authorize \
  --tenant-id "$THOTH_TENANT_ID" \
  --key-id "<key-id>" \
  --api-key-file "/run/secrets/thoth_scoped_api_key" \
  --permission execute \
  --endpoint-id "ep-e2e-01" \
  --json
```

Validate negative scope decision (different endpoint):

```bash
thothctl api-keys authorize \
  --tenant-id "$THOTH_TENANT_ID" \
  --key-id "<key-id>" \
  --api-key-file "/run/secrets/thoth_scoped_api_key" \
  --permission execute \
  --endpoint-id "ep-other-99" \
  --json
```

Expected:

- positive check: `valid=true`, `permission_allowed=true`, `scope_allowed=true`
- negative check: `scope_allowed=false`

Revoke key after validation:

```bash
thothctl api-keys revoke \
  --tenant-id "$THOTH_TENANT_ID" \
  --key-id "<key-id>" \
  --json
```

---

## Phase 2: Settings + MDM readback (Control Plane API verification)

Add auth header for non-local:

```bash
THOTH_GOVAPI_URL="${THOTH_GOVAPI_URL:-https://grid.${THOTH_TENANT_ID}.${THOTH_APEX_DOMAIN}}"
AUTH="Authorization: Bearer $(cat "$THOTH_AUTH_SESSION_FILE")"
BASE="$THOTH_GOVAPI_URL/$THOTH_TENANT_ID/thoth"
```

Verify settings:

```bash
thothctl settings get \
  --tenant-id "$THOTH_TENANT_ID" \
  --json
```

Verify webhook test endpoint independently:

```bash
curl -sS -X POST -H "$AUTH" "$BASE/settings/webhook/test" | jq .
```

Verify MDM provider inventory:

```bash
thothctl mdm list \
  --tenant-id "$THOTH_TENANT_ID" \
  --json
```

Trigger sync explicitly (optional second run):

```bash
thothctl mdm sync \
  --tenant-id "$THOTH_TENANT_ID" \
  --provider jamf \
  --json
```

Acceptance:

- provider exists and is enabled
- sync call returns accepted/success state

---

## Phase 2B: Browser control-plane validation

Upsert browser provider and policy:

```bash
thothctl browser providers upsert \
  --tenant-id "$THOTH_TENANT_ID" \
  --provider chrome \
  --name "Chrome Enterprise" \
  --config-file "./browser/chrome-provider.json" \
  --json

thothctl browser policies upsert \
  --tenant-id "$THOTH_TENANT_ID" \
  --provider chrome \
  --name "chrome-baseline-v1" \
  --enforcement-mode enforce \
  --policy-file "./browser/chrome-policy.json" \
  --json
```

Validate list/readbacks:

```bash
thothctl browser providers list \
  --tenant-id "$THOTH_TENANT_ID" \
  --json

thothctl browser policies list \
  --tenant-id "$THOTH_TENANT_ID" \
  --provider chrome \
  --json
```

Enroll one browser identity:

```bash
thothctl browser enrollments upsert \
  --tenant-id "$THOTH_TENANT_ID" \
  --provider chrome \
  --user-id "secops@example.com" \
  --device-id "ep-e2e-01" \
  --status active \
  --metadata-json '{"source":"mdm"}' \
  --json
```

Run endpoint policy sync/apply:

```bash
# Dry-run preview
thothctl browser sync \
  --tenant-id "$THOTH_TENANT_ID" \
  --provider chrome \
  --user-id "secops@example.com" \
  --device-id "ep-e2e-01" \
  --dry-run \
  --json

# Apply policy artifacts
thothctl browser sync \
  --tenant-id "$THOTH_TENANT_ID" \
  --provider chrome \
  --user-id "secops@example.com" \
  --device-id "ep-e2e-01" \
  --json
```

Acceptance:

- browser provider is present and enabled
- browser policy compiles and is returned by list/readback
- enrollment record is present for expected user/device/provider tuple
- sync output reports `applied`, `rendered`, or `dry_run` per provider with target paths

---

## Phase 3: Endpoint registration and check-ins

### Option A (recommended): proxy-driven registration/check-in

Run governed proxy path and then check:

```bash
thoth health --json
```

Then verify endpoint appears:

```bash
thothctl endpoints list \
  --tenant-id "$THOTH_TENANT_ID" \
  --json

thothctl endpoints stats \
  --tenant-id "$THOTH_TENANT_ID" \
  --json
```

### Option B (API smoke): manual registration + check-in

```bash
curl -sS -X POST -H "Content-Type: application/json" \
  "$BASE/endpoints" \
  -d '{
    "endpoint_id":"ep-e2e-01",
    "hostname":"ep-e2e-01.local",
    "employee_email":"secops@example.com",
    "enrollment":"manual",
    "environment":"prod",
    "proxy_version":"0.2.19",
    "agent_ids":["filesystem-safe"]
  }' | jq .
```

```bash
curl -sS -X POST -H "Content-Type: application/json" \
  "$BASE/endpoints/ep-e2e-01/checkin" \
  -d '{
    "proxy_version":"0.2.19",
    "agent_ids":["filesystem-safe"],
    "risk_score":10,
    "violations_today":0,
    "sessions_today":1
  }' | jq .
```

Acceptance:

- endpoint is returned by `GET /endpoints`
- stats reflect total/online increments

---

## Phase 4: PAM callback path validation

### 1) Configure PAM in tenant settings

Ensure settings include:

- `pam.enabled=true`
- `pam.provider`
- `pam.callback_secret`
- `pam.request_url` (HTTPS)

### 2) Simulate callback signature

```bash
thothctl pam callback \
  --tenant-id "$THOTH_TENANT_ID" \
  --approval-id "<approval-id>" \
  --request-id "<request-id>" \
  --decision approve \
  --json
```

Dry run (generate signature/payload without sending):

```bash
thothctl pam callback \
  --tenant-id "$THOTH_TENANT_ID" \
  --approval-id "<approval-id>" \
  --request-id "<request-id>" \
  --decision approve \
  --dry-run \
  --json
```

Acceptance:

- valid callback resolves approval
- replayed callback is safely ignored
- mismatched `request_id` returns conflict and does not resolve

---

## Phase 5: Evidence checks (headless operational validation)

Run:

```bash
thothctl endpoints stats \
  --tenant-id "$THOTH_TENANT_ID" \
  --json

thothctl approvals list \
  --tenant-id "$THOTH_TENANT_ID" \
  --status pending \
  --json

thothctl evidence slos \
  --tenant-id "$THOTH_TENANT_ID" \
  --window-hours 24 \
  --json

thothctl evidence decision-backfill \
  --tenant-id "$THOTH_TENANT_ID" \
  --window-hours 720 \
  --dry-run \
  --json

curl -sS -H "$AUTH" "$BASE/alerts" | jq .
curl -sS -H "$AUTH" "$BASE/governance/feed" | jq .
```

Expected:

- endpoint telemetry reflects active check-ins
- governance feed shows routed decisions/events
- evidence SLO report returns non-empty coverage metrics
- alerts reflect violations or risk changes as scenarios trigger

---

## Fail criteria

Fail the run if any of these occur:

- `thothctl bootstrap` does not return success flags
- webhook test cannot reach destination
- MDM sync cannot be triggered
- endpoint registration/check-in does not update stats
- PAM callback accepts invalid signature or mismatched request correlation

---

## Command coverage matrix

| Capability | `thothctl` | Control Plane API |
|---|---|---|
| Tenant settings upsert | ✅ | ✅ |
| Webhook test trigger | ✅ | ✅ |
| MDM provider upsert | ✅ | ✅ |
| MDM sync trigger | ✅ | ✅ |
| Browser provider/policy/enrollment upsert | ✅ | ✅ |
| Browser provider/policy/enrollment readback | ✅ | ✅ |
| Browser policy sync/apply summary | ✅ | ❌ |
| Scoped key create/authorize/revoke | ✅ | ✅ |
| Endpoint list/stats verification | ✅ | ✅ |
| Explicit endpoint check-in assertions | ❌ | ✅ |
| PAM callback correlation and replay tests | ✅ | ✅ |
| Approval queue reads | ✅ | ✅ |
| Evidence SLO readbacks | ✅ | ✅ |
| Evidence and decision-field backfill triggers | ✅ | ✅ |
| Audit feed and deep evidence checks | ❌ | ✅ |

`thothctl` is the right CLI for bootstrap and config application.
Control Plane API verification remains required for comprehensive end-to-end operational testing.