ai-contract-platform/project-docs/V2-phase3-ai-platform-trace-governance-worker-2026-07-02.md

# V2 Phase 3 AI Platform Trace Governance Worker

Date: 2026-07-02

## Status

Completed and smoke verified on `192.168.3.90`.

This slice adds the first executable worker for AI Trace governance. It uses the governance fields introduced earlier on `ai_run_audits`:

- `redaction_status`
- `redaction_policy`
- `retention_until`
- `archive_status`
- `archived_at`

## Behavior

New manual endpoint:

```text
POST /api/v1/runtime/trace-governance/run-once
```

Request body:

```json
{
  "limit": 100,
  "worker_name": "trace-governance-worker",
  "triggered_by": "admin"
}
```

The worker respects the current request tenant context:

- With `X-Tenant-Id`, only that tenant's traces are processed.
- Without `X-Tenant-Id`, only the backward-compatible `default` tenant is processed.

Processing rules:

- `redaction_status in ["pending", "required"]` is redacted.
- `retention_until <= now` and `archive_status != "archived"` is archived.
- A trace can be redacted and archived in the same run.

Redaction behavior:

- Keeps the trace row and operational metadata.
- Masks prompt/question/answer/retrieval text fields with `[REDACTED]`.
- Clears retrieved chunk IDs, retrieved source details, and RAG explain payload.
- Sets `redaction_status = "redacted"`.

Archive behavior:

- Sets `archive_status = "archived"`.
- Sets `archived_at` to the worker run timestamp.

The worker records:

- `worker_heartbeats`
- `worker_run_executions`

The API response includes both the execution record and governance stats:

```json
{
  "execution": {
    "queue_name": "trace-governance",
    "processed": 1,
    "succeeded": 1,
    "failed": 0
  },
  "stats": {
    "scanned": 1,
    "redacted": 1,
    "archived": 1,
    "skipped": 0,
    "redacted_run_ids": ["run-trace-governance-a"],
    "archived_run_ids": ["run-trace-governance-a"]
  }
}
```

## Files

Implementation:

- `yuqei-ai-platform/services/ai-platform-api/src/yuqei_ai_platform_api/repository.py`
- `yuqei-ai-platform/services/ai-platform-api/src/yuqei_ai_platform_api/persistence/audit.py`
- `yuqei-ai-platform/services/ai-platform-api/src/yuqei_ai_platform_api/worker_runtime.py`
- `yuqei-ai-platform/services/ai-platform-api/src/yuqei_ai_platform_api/api/system.py`

Tests:

- `test_trace_governance_worker_redacts_archives_and_keeps_tenant_boundary`
- `test_sqlalchemy_trace_governance_redacts_and_archives_current_tenant`

## Validation

Local validation passed:

```powershell
python -m py_compile `
  yuqei-ai-platform\services\ai-platform-api\src\yuqei_ai_platform_api\repository.py `
  yuqei-ai-platform\services\ai-platform-api\src\yuqei_ai_platform_api\persistence\audit.py `
  yuqei-ai-platform\services\ai-platform-api\src\yuqei_ai_platform_api\worker_runtime.py `
  yuqei-ai-platform\services\ai-platform-api\src\yuqei_ai_platform_api\api\system.py

python -m pytest `
  yuqei-ai-platform\services\ai-platform-api\tests\test_platform_data.py::test_trace_governance_worker_redacts_archives_and_keeps_tenant_boundary `
  -q

python -m pytest `
  yuqei-ai-platform\services\ai-platform-api\tests\test_platform_data.py::test_sqlalchemy_trace_governance_redacts_and_archives_current_tenant `
  -q

python -m pytest yuqei-ai-platform\services\ai-platform-api\tests -q
```

Result:

- AI Platform API tests: 58 passed.

## Smoke Verification

Deployment artifact:

- `project-docs/qa-artifacts/v2-smoke-docker-deploy/2026-07-02T23-31-08`

Runtime checks:

```text
GET http://192.168.3.90:18100/api/v1/runtime/schema-check -> ok
GET http://192.168.3.90:18101/api/v1/runtime/schema-check -> ok
```

Trace governance smoke:

```json
{
  "status": "pass",
  "tenant_a": "trace-governance-smoke-20260702233551-a",
  "tenant_b": "trace-governance-smoke-20260702233551-b",
  "run_id": "run-f1cb946c04e8",
  "worker_stats": {
    "scanned": 1,
    "redacted": 1,
    "archived": 1,
    "failed": 0
  },
  "worker_execution": {
    "id": "worker-run-7552e2b52d5e",
    "processed": 1,
    "succeeded": 1,
    "failed": 0
  },
  "detail": {
    "tenant_id": "trace-governance-smoke-20260702233551-a",
    "redaction_status": "redacted",
    "archive_status": "archived",
    "question": "[REDACTED]",
    "answer": "[REDACTED]",
    "retrieved_sources_count": 0
  },
  "cross_tenant_detail_status": 404
}
```

Verified flow:

1. Created a real tenant-scoped AI Trace through `POST /api/v1/legal/qa`.
2. Marked only that smoke trace as `redaction_status = pending` and `retention_until` expired.
3. Ran `POST /api/v1/runtime/trace-governance/run-once` with `X-Tenant-Id`.
4. Verified same-tenant detail is redacted and archived.
5. Verified cross-tenant detail returns `404`.
6. Verified `worker_run_executions` recorded the run.

## Hardening Slice

Local implementation completed after the first smoke verification:

- Worker process now supports scheduled Trace governance in the normal loop when `YUQEI_WORKER_TRACE_GOVERNANCE_ENABLED=true`.
- Scheduled Trace governance records `worker_run_executions` for each trace-governance loop run, while the knowledge ingestion loop keeps its existing heartbeat-only behavior in the long-running worker process.
- Worker settings now include `YUQEI_WORKER_TENANT_ID`, `YUQEI_WORKER_TRACE_GOVERNANCE_INTERVAL_SECONDS`, `YUQEI_WORKER_TRACE_GOVERNANCE_BATCH_LIMIT`, and `YUQEI_WORKER_TRACE_GOVERNANCE_WORKER_NAME`.
- Worker CLI now supports `--trace-governance-run-once` for container-level operational checks.
- Redaction policy is now level-aware:
  - `prompt_only` / `prompt`: redact prompt preview, question, and retrieval query.
  - `answer_only` / `answer`: redact answer and quality summary.
  - default/full/unknown: use the conservative full Trace mask.
- `GET /api/v1/audit/ai-runs` accepts `archive_status=active|archived`.
- SDK `list_ai_run_audits(...)` accepts `archive_status`.
- Contract Web AI Trace admin page forwards the archive filter through the local API proxy and shows archived/redacted badges in the list.

Local validation:

```powershell
python -m pytest `
  yuqei-ai-platform\services\ai-platform-api\tests\test_platform_data.py::test_trace_governance_worker_redacts_archives_and_keeps_tenant_boundary `
  yuqei-ai-platform\services\ai-platform-api\tests\test_platform_data.py::test_trace_governance_worker_applies_redaction_policy_levels `
  yuqei-ai-platform\services\ai-platform-api\tests\test_platform_data.py::test_sqlalchemy_trace_governance_redacts_and_archives_current_tenant `
  yuqei-ai-sdk-python\tests\test_ai_platform.py::test_ai_platform_client_covers_admin_data_endpoints `
  yuqei-ai-platform\services\worker\tests\test_health.py `
  -q

python -m pytest yuqei-ai-platform\services\ai-platform-api\tests -q
python -m pytest yuqei-ai-sdk-python\tests -q
python -m pytest yuqei-ai-platform\services\worker\tests -q
npm run web:typecheck
```

Result:

- Targeted Trace/SDK/worker tests: 10 passed.
- AI Platform API tests: 59 passed.
- Python SDK tests: 3 passed.
- Worker tests: 6 passed.
- Contract Web typecheck: passed.

Remote smoke verification:

- Deploy artifact: `project-docs/qa-artifacts/v2-smoke-docker-deploy/2026-07-03T09-09-30`
- AI Platform schema-check: `ok`
- AI Platform Alembic head: `20260702_0020`
- `GET /api/v1/audit/ai-runs?archive_status=active`: returned active rows.
- `GET /api/v1/audit/ai-runs?archive_status=archived`: verified with dedicated smoke tenant.
- `POST /api/v1/runtime/trace-governance/run-once`: scanned 2, redacted 2, archived 2, failed 0.
- Worker container CLI `python -m yuqei_worker.main --trace-governance-run-once --limit 5`: returned `status=ok`.
- Worker container settings initially verified with `trace_governance_enabled=false`, then smoke observation enabled it explicitly.
- Contract Web `/admin/ai-traces`: HTTP 200.

Scheduled loop smoke observation:

- Smoke `.env.smoke` was changed to `YUQEI_WORKER_TRACE_GOVERNANCE_ENABLED=true`.
- Current smoke worker settings: `tenant_id=default`, `trace_governance_enabled=true`, `trace_governance_interval_seconds=300`, `trace_governance_batch_limit=100`, `trace_governance_worker_name=trace-governance-worker`.
- A dedicated default-tenant Trace `run-scheduled-governance-ea4b8a5b10f8` was redacted and archived by the scheduled worker loop.
- A second dedicated default-tenant Trace `run-scheduled-execution-561f829c092b` verified execution logging after the worker execution-recording hardening.
- Observed `worker_run_executions` record: `worker-run-9c0d8d73d0c6`, `worker_name=trace-governance-worker`, `trigger_type=worker_loop`, `triggered_by=worker-loop`, `queue_name=trace-governance`, `processed=1`, `succeeded=1`, `failed=0`.
- Observed `worker_heartbeats` for `trace-governance-worker` with `status=ok`.
- Final remote checks: AI Platform schema-check `ok`; `ai-platform-api`, `contract-api`, `contract-web`, and `worker` containers healthy.

Observation script:

- Added `yuqei-infra/scripts/watch-trace-governance-worker.py`.
- The script polls:
  - `GET /api/v1/runtime/worker-health`
  - `GET /api/v1/runtime/worker-runs`
- It verifies heartbeat freshness, minimum execution count, and failed execution count.
- Short smoke command:

```powershell
python yuqei-infra\scripts\watch-trace-governance-worker.py `
  --base-url http://192.168.3.90:18101 `
  --duration-seconds 5 `
  --interval-seconds 5 `
  --max-heartbeat-age-seconds 420 `
  --min-executions 1 `
  --execution-lookback-seconds 300
```

Observed result:

- `TRACE_GOVERNANCE_OBSERVATION_PASS`
- Artifact: `project-docs/qa-artifacts/v2-trace-governance-observation/trace-governance-observation-2026-07-03T09-29-43.json`
- Final heartbeat age: `68.615` seconds.
- Final trace-governance execution count in the lookback window: `2`.
- Failed execution count: `0`.

30-minute smoke observation:

```powershell
python yuqei-infra\scripts\watch-trace-governance-worker.py `
  --base-url http://192.168.3.90:18101 `
  --duration-seconds 1800 `
  --interval-seconds 60 `
  --max-heartbeat-age-seconds 420 `
  --min-executions 5
```

Result:

- `TRACE_GOVERNANCE_OBSERVATION_PASS`
- Artifact: `project-docs/qa-artifacts/v2-trace-governance-observation/trace-governance-observation-2026-07-03T11-16-17.json`
- Observation window: `2026-07-03T10:46:14+08:00` to `2026-07-03T11:16:17+08:00`.
- Samples: `30`.
- Final execution count: `6`.
- Failed execution count: `0`.
- Final heartbeat age: `146.944` seconds.
- Final remote checks after the window: AI Platform schema-check `ok`; `ai-platform-api`, `contract-api`, `contract-web`, and `worker` containers healthy.

4-hour smoke observation:

```powershell
python yuqei-infra\scripts\watch-trace-governance-worker.py `
  --base-url http://192.168.3.90:18101 `
  --duration-seconds 14400 `
  --interval-seconds 300 `
  --max-heartbeat-age-seconds 420 `
  --min-executions 40
```

Result:

- `TRACE_GOVERNANCE_OBSERVATION_PASS`
- Artifact: `project-docs/qa-artifacts/v2-trace-governance-observation/trace-governance-observation-2026-07-03T16-05-01.json`
- Observation window: `2026-07-03T12:04:58+08:00` to `2026-07-03T16:05:01+08:00`.
- Samples: `49`.
- Final execution count: `47`.
- Failed execution count: `0`.
- Final heartbeat age: `228.475` seconds.
- Final remote checks after the window: AI Platform schema-check `ok`; `ai-platform-api`, `contract-api`, `contract-web`, and `worker` containers healthy.

Dedicated smoke data:

```json
{
  "tenant": "trace-governance-hardening-20260702161706",
  "prompt_run_id": "run-policy-prompt-630fc17c76a5",
  "answer_run_id": "run-policy-answer-b2c7f7b86fca",
  "archived_count": 2,
  "active_count": 0
}
```

Verified policy behavior:

- `prompt_only`: prompt preview, question, and retrieval query became `[REDACTED]`; answer and retrieved source metadata remained visible.
- `answer_only`: answer and quality summary became `[REDACTED]`; prompt preview, question, and retrieval query remained visible.

## Admin Runtime Visibility

Contract Web now exposes Trace governance status in the AI Runtime admin page:

- Page: `GET /admin/ai-runtime`
- Local proxy: `POST /api/ai-platform/runtime/trace-governance/run-once`
- Upstream API: `POST /api/v1/runtime/trace-governance/run-once`

The page shows:

- Trace governance heartbeat status and heartbeat age.
- Trace governance executions in the last 30 minutes.
- Recent governance run count and failed run count.
- Latest governance run records.
- A manual `Run governance once` action for smoke/admin checks.

Local UI verification on 2026-07-03:

- `npm run web:typecheck`: passed.
- `GET http://localhost:3000/admin/ai-runtime`: HTTP 200.
- Trace governance panel rendered with heartbeat, last-30-minute executions, recent run count, failed count, and latest runs.
- Manual local proxy check returned a successful Trace governance execution.
- Screenshot artifact: `project-docs/qa-artifacts/v2-trace-governance-admin-ui/ai-runtime-trace-governance.png`.
- Layout check at 1440px: no horizontal overflow.

Operational decision on 2026-07-03:

- Smoke: keep `YUQEI_WORKER_TRACE_GOVERNANCE_ENABLED=true` so the background loop keeps accumulating observation data naturally.
- Production: keep Trace governance disabled for now.
- The 24-hour observation window is a pre-production checklist item, not a blocker for continuing current development.

Follow-up slice on 2026-07-03:

- Trace retention/redaction/archive policy configuration was added in `V2-phase3-ai-platform-trace-policy-config-2026-07-03.md`.
- The policy slice adds `trace_governance_policies`, tenant-scoped GET/PUT runtime endpoints, worker policy application, and an AI Runtime admin form.

## Remaining Items

- Keep smoke scheduled Trace governance enabled and run a 24-hour observation window before production enablement, because both 30-minute and 4-hour gates have passed.
- Recommended 24-hour smoke observation command before production enablement:

```powershell
python yuqei-infra\scripts\watch-trace-governance-worker.py `
  --base-url http://192.168.3.90:18101 `
  --duration-seconds 86400 `
  --interval-seconds 300 `
  --max-heartbeat-age-seconds 420 `
  --min-executions 250
```

- Production should remain disabled until the smoke observation window has no worker errors and execution volume is acceptable.
- Later cleanup can localize the new archive filter labels in the admin page once the existing mojibake text is repaired.