yuqei/ai-contract-platform
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
main
ai-contract-platform/project-docs/V2-phase3-ai-platform-trace-governance-worker-2026-07-02.md
T
yuqei a3bb78e06b
Security Quality Gate / npm audit, Bandit, pip-audit, detect-secrets (pull_request) Failing after 16m35s
Details
release: freeze V2 release candidate
2026-07-12 15:32:24 +08:00

381 lines
14 KiB
Markdown
Raw Permalink Blame History

# V2 Phase 3 AI Platform Trace Governance Worker
Date: 2026-07-02
## Status
Completed and smoke verified on `192.168.3.90`.
This slice adds the first executable worker for AI Trace governance. It uses the governance fields introduced earlier on `ai_run_audits`:
- `redaction_status`
- `redaction_policy`
- `retention_until`
- `archive_status`
- `archived_at`
## Behavior
New manual endpoint:
```text
POST /api/v1/runtime/trace-governance/run-once
```
Request body:
```json
{
"limit": 100,
"worker_name": "trace-governance-worker",
"triggered_by": "admin"
}
```
The worker respects the current request tenant context:
- With `X-Tenant-Id`, only that tenant's traces are processed.
- Without `X-Tenant-Id`, only the backward-compatible `default` tenant is processed.
Processing rules:
- `redaction_status in ["pending", "required"]` is redacted.
- `retention_until <= now` and `archive_status != "archived"` is archived.
- A trace can be redacted and archived in the same run.
Redaction behavior:
- Keeps the trace row and operational metadata.
- Masks prompt/question/answer/retrieval text fields with `[REDACTED]`.
- Clears retrieved chunk IDs, retrieved source details, and RAG explain payload.
- Sets `redaction_status = "redacted"`.
Archive behavior:
- Sets `archive_status = "archived"`.
- Sets `archived_at` to the worker run timestamp.
The worker records:
- `worker_heartbeats`
- `worker_run_executions`
The API response includes both the execution record and governance stats:
```json
{
"execution": {
"queue_name": "trace-governance",
"processed": 1,
"succeeded": 1,
"failed": 0
},
"stats": {
"scanned": 1,
"redacted": 1,
"archived": 1,
"skipped": 0,
"redacted_run_ids": ["run-trace-governance-a"],
"archived_run_ids": ["run-trace-governance-a"]
}
}
```
## Files
Implementation:
- `yuqei-ai-platform/services/ai-platform-api/src/yuqei_ai_platform_api/repository.py`
- `yuqei-ai-platform/services/ai-platform-api/src/yuqei_ai_platform_api/persistence/audit.py`
- `yuqei-ai-platform/services/ai-platform-api/src/yuqei_ai_platform_api/worker_runtime.py`
- `yuqei-ai-platform/services/ai-platform-api/src/yuqei_ai_platform_api/api/system.py`
Tests:
- `test_trace_governance_worker_redacts_archives_and_keeps_tenant_boundary`
- `test_sqlalchemy_trace_governance_redacts_and_archives_current_tenant`
## Validation
Local validation passed:
```powershell
python -m py_compile `
yuqei-ai-platform\services\ai-platform-api\src\yuqei_ai_platform_api\repository.py `
yuqei-ai-platform\services\ai-platform-api\src\yuqei_ai_platform_api\persistence\audit.py `
yuqei-ai-platform\services\ai-platform-api\src\yuqei_ai_platform_api\worker_runtime.py `
yuqei-ai-platform\services\ai-platform-api\src\yuqei_ai_platform_api\api\system.py
python -m pytest `
yuqei-ai-platform\services\ai-platform-api\tests\test_platform_data.py::test_trace_governance_worker_redacts_archives_and_keeps_tenant_boundary `
-q
python -m pytest `
yuqei-ai-platform\services\ai-platform-api\tests\test_platform_data.py::test_sqlalchemy_trace_governance_redacts_and_archives_current_tenant `
-q
python -m pytest yuqei-ai-platform\services\ai-platform-api\tests -q
```
Result:
- AI Platform API tests: 58 passed.
## Smoke Verification
Deployment artifact:
- `project-docs/qa-artifacts/v2-smoke-docker-deploy/2026-07-02T23-31-08`
Runtime checks:
```text
GET http://192.168.3.90:18100/api/v1/runtime/schema-check -> ok
GET http://192.168.3.90:18101/api/v1/runtime/schema-check -> ok
```
Trace governance smoke:
```json
{
"status": "pass",
"tenant_a": "trace-governance-smoke-20260702233551-a",
"tenant_b": "trace-governance-smoke-20260702233551-b",
"run_id": "run-f1cb946c04e8",
"worker_stats": {
"scanned": 1,
"redacted": 1,
"archived": 1,
"failed": 0
},
"worker_execution": {
"id": "worker-run-7552e2b52d5e",
"processed": 1,
"succeeded": 1,
"failed": 0
},
"detail": {
"tenant_id": "trace-governance-smoke-20260702233551-a",
"redaction_status": "redacted",
"archive_status": "archived",
"question": "[REDACTED]",
"answer": "[REDACTED]",
"retrieved_sources_count": 0
},
"cross_tenant_detail_status": 404
}
```
Verified flow:
1. Created a real tenant-scoped AI Trace through `POST /api/v1/legal/qa`.
2. Marked only that smoke trace as `redaction_status = pending` and `retention_until` expired.
3. Ran `POST /api/v1/runtime/trace-governance/run-once` with `X-Tenant-Id`.
4. Verified same-tenant detail is redacted and archived.
5. Verified cross-tenant detail returns `404`.
6. Verified `worker_run_executions` recorded the run.
## Hardening Slice
Local implementation completed after the first smoke verification:
- Worker process now supports scheduled Trace governance in the normal loop when `YUQEI_WORKER_TRACE_GOVERNANCE_ENABLED=true`.
- Scheduled Trace governance records `worker_run_executions` for each trace-governance loop run, while the knowledge ingestion loop keeps its existing heartbeat-only behavior in the long-running worker process.
- Worker settings now include `YUQEI_WORKER_TENANT_ID`, `YUQEI_WORKER_TRACE_GOVERNANCE_INTERVAL_SECONDS`, `YUQEI_WORKER_TRACE_GOVERNANCE_BATCH_LIMIT`, and `YUQEI_WORKER_TRACE_GOVERNANCE_WORKER_NAME`.
- Worker CLI now supports `--trace-governance-run-once` for container-level operational checks.
- Redaction policy is now level-aware:
- `prompt_only` / `prompt`: redact prompt preview, question, and retrieval query.
- `answer_only` / `answer`: redact answer and quality summary.
- default/full/unknown: use the conservative full Trace mask.
- `GET /api/v1/audit/ai-runs` accepts `archive_status=active|archived`.
- SDK `list_ai_run_audits(...)` accepts `archive_status`.
- Contract Web AI Trace admin page forwards the archive filter through the local API proxy and shows archived/redacted badges in the list.
Local validation:
```powershell
python -m pytest `
yuqei-ai-platform\services\ai-platform-api\tests\test_platform_data.py::test_trace_governance_worker_redacts_archives_and_keeps_tenant_boundary `
yuqei-ai-platform\services\ai-platform-api\tests\test_platform_data.py::test_trace_governance_worker_applies_redaction_policy_levels `
yuqei-ai-platform\services\ai-platform-api\tests\test_platform_data.py::test_sqlalchemy_trace_governance_redacts_and_archives_current_tenant `
yuqei-ai-sdk-python\tests\test_ai_platform.py::test_ai_platform_client_covers_admin_data_endpoints `
yuqei-ai-platform\services\worker\tests\test_health.py `
-q
python -m pytest yuqei-ai-platform\services\ai-platform-api\tests -q
python -m pytest yuqei-ai-sdk-python\tests -q
python -m pytest yuqei-ai-platform\services\worker\tests -q
npm run web:typecheck
```
Result:
- Targeted Trace/SDK/worker tests: 10 passed.
- AI Platform API tests: 59 passed.
- Python SDK tests: 3 passed.
- Worker tests: 6 passed.
- Contract Web typecheck: passed.
Remote smoke verification:
- Deploy artifact: `project-docs/qa-artifacts/v2-smoke-docker-deploy/2026-07-03T09-09-30`
- AI Platform schema-check: `ok`
- AI Platform Alembic head: `20260702_0020`
- `GET /api/v1/audit/ai-runs?archive_status=active`: returned active rows.
- `GET /api/v1/audit/ai-runs?archive_status=archived`: verified with dedicated smoke tenant.
- `POST /api/v1/runtime/trace-governance/run-once`: scanned 2, redacted 2, archived 2, failed 0.
- Worker container CLI `python -m yuqei_worker.main --trace-governance-run-once --limit 5`: returned `status=ok`.
- Worker container settings initially verified with `trace_governance_enabled=false`, then smoke observation enabled it explicitly.
- Contract Web `/admin/ai-traces`: HTTP 200.
Scheduled loop smoke observation:
- Smoke `.env.smoke` was changed to `YUQEI_WORKER_TRACE_GOVERNANCE_ENABLED=true`.
- Current smoke worker settings: `tenant_id=default`, `trace_governance_enabled=true`, `trace_governance_interval_seconds=300`, `trace_governance_batch_limit=100`, `trace_governance_worker_name=trace-governance-worker`.
- A dedicated default-tenant Trace `run-scheduled-governance-ea4b8a5b10f8` was redacted and archived by the scheduled worker loop.
- A second dedicated default-tenant Trace `run-scheduled-execution-561f829c092b` verified execution logging after the worker execution-recording hardening.
- Observed `worker_run_executions` record: `worker-run-9c0d8d73d0c6`, `worker_name=trace-governance-worker`, `trigger_type=worker_loop`, `triggered_by=worker-loop`, `queue_name=trace-governance`, `processed=1`, `succeeded=1`, `failed=0`.
- Observed `worker_heartbeats` for `trace-governance-worker` with `status=ok`.
- Final remote checks: AI Platform schema-check `ok`; `ai-platform-api`, `contract-api`, `contract-web`, and `worker` containers healthy.
Observation script:
- Added `yuqei-infra/scripts/watch-trace-governance-worker.py`.
- The script polls:
- `GET /api/v1/runtime/worker-health`
- `GET /api/v1/runtime/worker-runs`
- It verifies heartbeat freshness, minimum execution count, and failed execution count.
- Short smoke command:
```powershell
python yuqei-infra\scripts\watch-trace-governance-worker.py `
--base-url http://192.168.3.90:18101 `
--duration-seconds 5 `
--interval-seconds 5 `
--max-heartbeat-age-seconds 420 `
--min-executions 1 `
--execution-lookback-seconds 300
```
Observed result:
- `TRACE_GOVERNANCE_OBSERVATION_PASS`
- Artifact: `project-docs/qa-artifacts/v2-trace-governance-observation/trace-governance-observation-2026-07-03T09-29-43.json`
- Final heartbeat age: `68.615` seconds.
- Final trace-governance execution count in the lookback window: `2`.
- Failed execution count: `0`.
30-minute smoke observation:
```powershell
python yuqei-infra\scripts\watch-trace-governance-worker.py `
--base-url http://192.168.3.90:18101 `
--duration-seconds 1800 `
--interval-seconds 60 `
--max-heartbeat-age-seconds 420 `
--min-executions 5
```
Result:
- `TRACE_GOVERNANCE_OBSERVATION_PASS`
- Artifact: `project-docs/qa-artifacts/v2-trace-governance-observation/trace-governance-observation-2026-07-03T11-16-17.json`
- Observation window: `2026-07-03T10:46:14+08:00` to `2026-07-03T11:16:17+08:00`.
- Samples: `30`.
- Final execution count: `6`.
- Failed execution count: `0`.
- Final heartbeat age: `146.944` seconds.
- Final remote checks after the window: AI Platform schema-check `ok`; `ai-platform-api`, `contract-api`, `contract-web`, and `worker` containers healthy.
4-hour smoke observation:
```powershell
python yuqei-infra\scripts\watch-trace-governance-worker.py `
--base-url http://192.168.3.90:18101 `
--duration-seconds 14400 `
--interval-seconds 300 `
--max-heartbeat-age-seconds 420 `
--min-executions 40
```
Result:
- `TRACE_GOVERNANCE_OBSERVATION_PASS`
- Artifact: `project-docs/qa-artifacts/v2-trace-governance-observation/trace-governance-observation-2026-07-03T16-05-01.json`
- Observation window: `2026-07-03T12:04:58+08:00` to `2026-07-03T16:05:01+08:00`.
- Samples: `49`.
- Final execution count: `47`.
- Failed execution count: `0`.
- Final heartbeat age: `228.475` seconds.
- Final remote checks after the window: AI Platform schema-check `ok`; `ai-platform-api`, `contract-api`, `contract-web`, and `worker` containers healthy.
Dedicated smoke data:
```json
{
"tenant": "trace-governance-hardening-20260702161706",
"prompt_run_id": "run-policy-prompt-630fc17c76a5",
"answer_run_id": "run-policy-answer-b2c7f7b86fca",
"archived_count": 2,
"active_count": 0
}
```
Verified policy behavior:
- `prompt_only`: prompt preview, question, and retrieval query became `[REDACTED]`; answer and retrieved source metadata remained visible.
- `answer_only`: answer and quality summary became `[REDACTED]`; prompt preview, question, and retrieval query remained visible.
## Admin Runtime Visibility
Contract Web now exposes Trace governance status in the AI Runtime admin page:
- Page: `GET /admin/ai-runtime`
- Local proxy: `POST /api/ai-platform/runtime/trace-governance/run-once`
- Upstream API: `POST /api/v1/runtime/trace-governance/run-once`
The page shows:
- Trace governance heartbeat status and heartbeat age.
- Trace governance executions in the last 30 minutes.
- Recent governance run count and failed run count.
- Latest governance run records.
- A manual `Run governance once` action for smoke/admin checks.
Local UI verification on 2026-07-03:
- `npm run web:typecheck`: passed.
- `GET http://localhost:3000/admin/ai-runtime`: HTTP 200.
- Trace governance panel rendered with heartbeat, last-30-minute executions, recent run count, failed count, and latest runs.
- Manual local proxy check returned a successful Trace governance execution.
- Screenshot artifact: `project-docs/qa-artifacts/v2-trace-governance-admin-ui/ai-runtime-trace-governance.png`.
- Layout check at 1440px: no horizontal overflow.
Operational decision on 2026-07-03:
- Smoke: keep `YUQEI_WORKER_TRACE_GOVERNANCE_ENABLED=true` so the background loop keeps accumulating observation data naturally.
- Production: keep Trace governance disabled for now.
- The 24-hour observation window is a pre-production checklist item, not a blocker for continuing current development.
Follow-up slice on 2026-07-03:
- Trace retention/redaction/archive policy configuration was added in `V2-phase3-ai-platform-trace-policy-config-2026-07-03.md`.
- The policy slice adds `trace_governance_policies`, tenant-scoped GET/PUT runtime endpoints, worker policy application, and an AI Runtime admin form.
## Remaining Items
- Keep smoke scheduled Trace governance enabled and run a 24-hour observation window before production enablement, because both 30-minute and 4-hour gates have passed.
- Recommended 24-hour smoke observation command before production enablement:
```powershell
python yuqei-infra\scripts\watch-trace-governance-worker.py `
--base-url http://192.168.3.90:18101 `
--duration-seconds 86400 `
--interval-seconds 300 `
--max-heartbeat-age-seconds 420 `
--min-executions 250
```
- Production should remain disabled until the smoke observation window has no worker errors and execution volume is acceptable.
- Later cleanup can localize the new archive filter labels in the admin page once the existing mojibake text is repaired.
Reference in New Issue View Git Blame Copy Permalink