yuqei/ai-contract-platform
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
main
ai-contract-platform/project-docs/V2-phase3-ai-platform-trace-governance-worker-2026-07-02.md
T
yuqei a3bb78e06b
Security Quality Gate / npm audit, Bandit, pip-audit, detect-secrets (pull_request) Failing after 16m35s
Details
release: freeze V2 release candidate
2026-07-12 15:32:24 +08:00

14 KiB
Raw Permalink Blame History

V2 Phase 3 AI Platform Trace Governance Worker

Date: 2026-07-02

Status

Completed and smoke verified on 192.168.3.90.

This slice adds the first executable worker for AI Trace governance. It uses the governance fields introduced earlier on ai_run_audits:

  • redaction_status
  • redaction_policy
  • retention_until
  • archive_status
  • archived_at

Behavior

New manual endpoint:

POST /api/v1/runtime/trace-governance/run-once

Request body:

{
  "limit": 100,
  "worker_name": "trace-governance-worker",
  "triggered_by": "admin"
}

The worker respects the current request tenant context:

  • With X-Tenant-Id, only that tenant's traces are processed.
  • Without X-Tenant-Id, only the backward-compatible default tenant is processed.

Processing rules:

  • redaction_status in ["pending", "required"] is redacted.
  • retention_until <= now and archive_status != "archived" is archived.
  • A trace can be redacted and archived in the same run.

Redaction behavior:

  • Keeps the trace row and operational metadata.
  • Masks prompt/question/answer/retrieval text fields with [REDACTED].
  • Clears retrieved chunk IDs, retrieved source details, and RAG explain payload.
  • Sets redaction_status = "redacted".

Archive behavior:

  • Sets archive_status = "archived".
  • Sets archived_at to the worker run timestamp.

The worker records:

  • worker_heartbeats
  • worker_run_executions

The API response includes both the execution record and governance stats:

{
  "execution": {
    "queue_name": "trace-governance",
    "processed": 1,
    "succeeded": 1,
    "failed": 0
  },
  "stats": {
    "scanned": 1,
    "redacted": 1,
    "archived": 1,
    "skipped": 0,
    "redacted_run_ids": ["run-trace-governance-a"],
    "archived_run_ids": ["run-trace-governance-a"]
  }
}

Files

Implementation:

  • yuqei-ai-platform/services/ai-platform-api/src/yuqei_ai_platform_api/repository.py
  • yuqei-ai-platform/services/ai-platform-api/src/yuqei_ai_platform_api/persistence/audit.py
  • yuqei-ai-platform/services/ai-platform-api/src/yuqei_ai_platform_api/worker_runtime.py
  • yuqei-ai-platform/services/ai-platform-api/src/yuqei_ai_platform_api/api/system.py

Tests:

  • test_trace_governance_worker_redacts_archives_and_keeps_tenant_boundary
  • test_sqlalchemy_trace_governance_redacts_and_archives_current_tenant

Validation

Local validation passed:

python -m py_compile `
  yuqei-ai-platform\services\ai-platform-api\src\yuqei_ai_platform_api\repository.py `
  yuqei-ai-platform\services\ai-platform-api\src\yuqei_ai_platform_api\persistence\audit.py `
  yuqei-ai-platform\services\ai-platform-api\src\yuqei_ai_platform_api\worker_runtime.py `
  yuqei-ai-platform\services\ai-platform-api\src\yuqei_ai_platform_api\api\system.py

python -m pytest `
  yuqei-ai-platform\services\ai-platform-api\tests\test_platform_data.py::test_trace_governance_worker_redacts_archives_and_keeps_tenant_boundary `
  -q

python -m pytest `
  yuqei-ai-platform\services\ai-platform-api\tests\test_platform_data.py::test_sqlalchemy_trace_governance_redacts_and_archives_current_tenant `
  -q

python -m pytest yuqei-ai-platform\services\ai-platform-api\tests -q

Result:

  • AI Platform API tests: 58 passed.

Smoke Verification

Deployment artifact:

  • project-docs/qa-artifacts/v2-smoke-docker-deploy/2026-07-02T23-31-08

Runtime checks:

GET http://192.168.3.90:18100/api/v1/runtime/schema-check -> ok
GET http://192.168.3.90:18101/api/v1/runtime/schema-check -> ok

Trace governance smoke:

{
  "status": "pass",
  "tenant_a": "trace-governance-smoke-20260702233551-a",
  "tenant_b": "trace-governance-smoke-20260702233551-b",
  "run_id": "run-f1cb946c04e8",
  "worker_stats": {
    "scanned": 1,
    "redacted": 1,
    "archived": 1,
    "failed": 0
  },
  "worker_execution": {
    "id": "worker-run-7552e2b52d5e",
    "processed": 1,
    "succeeded": 1,
    "failed": 0
  },
  "detail": {
    "tenant_id": "trace-governance-smoke-20260702233551-a",
    "redaction_status": "redacted",
    "archive_status": "archived",
    "question": "[REDACTED]",
    "answer": "[REDACTED]",
    "retrieved_sources_count": 0
  },
  "cross_tenant_detail_status": 404
}

Verified flow:

  1. Created a real tenant-scoped AI Trace through POST /api/v1/legal/qa.
  2. Marked only that smoke trace as redaction_status = pending and retention_until expired.
  3. Ran POST /api/v1/runtime/trace-governance/run-once with X-Tenant-Id.
  4. Verified same-tenant detail is redacted and archived.
  5. Verified cross-tenant detail returns 404.
  6. Verified worker_run_executions recorded the run.

Hardening Slice

Local implementation completed after the first smoke verification:

  • Worker process now supports scheduled Trace governance in the normal loop when YUQEI_WORKER_TRACE_GOVERNANCE_ENABLED=true.
  • Scheduled Trace governance records worker_run_executions for each trace-governance loop run, while the knowledge ingestion loop keeps its existing heartbeat-only behavior in the long-running worker process.
  • Worker settings now include YUQEI_WORKER_TENANT_ID, YUQEI_WORKER_TRACE_GOVERNANCE_INTERVAL_SECONDS, YUQEI_WORKER_TRACE_GOVERNANCE_BATCH_LIMIT, and YUQEI_WORKER_TRACE_GOVERNANCE_WORKER_NAME.
  • Worker CLI now supports --trace-governance-run-once for container-level operational checks.
  • Redaction policy is now level-aware:
    • prompt_only / prompt: redact prompt preview, question, and retrieval query.
    • answer_only / answer: redact answer and quality summary.
    • default/full/unknown: use the conservative full Trace mask.
  • GET /api/v1/audit/ai-runs accepts archive_status=active|archived.
  • SDK list_ai_run_audits(...) accepts archive_status.
  • Contract Web AI Trace admin page forwards the archive filter through the local API proxy and shows archived/redacted badges in the list.

Local validation:

python -m pytest `
  yuqei-ai-platform\services\ai-platform-api\tests\test_platform_data.py::test_trace_governance_worker_redacts_archives_and_keeps_tenant_boundary `
  yuqei-ai-platform\services\ai-platform-api\tests\test_platform_data.py::test_trace_governance_worker_applies_redaction_policy_levels `
  yuqei-ai-platform\services\ai-platform-api\tests\test_platform_data.py::test_sqlalchemy_trace_governance_redacts_and_archives_current_tenant `
  yuqei-ai-sdk-python\tests\test_ai_platform.py::test_ai_platform_client_covers_admin_data_endpoints `
  yuqei-ai-platform\services\worker\tests\test_health.py `
  -q

python -m pytest yuqei-ai-platform\services\ai-platform-api\tests -q
python -m pytest yuqei-ai-sdk-python\tests -q
python -m pytest yuqei-ai-platform\services\worker\tests -q
npm run web:typecheck

Result:

  • Targeted Trace/SDK/worker tests: 10 passed.
  • AI Platform API tests: 59 passed.
  • Python SDK tests: 3 passed.
  • Worker tests: 6 passed.
  • Contract Web typecheck: passed.

Remote smoke verification:

  • Deploy artifact: project-docs/qa-artifacts/v2-smoke-docker-deploy/2026-07-03T09-09-30
  • AI Platform schema-check: ok
  • AI Platform Alembic head: 20260702_0020
  • GET /api/v1/audit/ai-runs?archive_status=active: returned active rows.
  • GET /api/v1/audit/ai-runs?archive_status=archived: verified with dedicated smoke tenant.
  • POST /api/v1/runtime/trace-governance/run-once: scanned 2, redacted 2, archived 2, failed 0.
  • Worker container CLI python -m yuqei_worker.main --trace-governance-run-once --limit 5: returned status=ok.
  • Worker container settings initially verified with trace_governance_enabled=false, then smoke observation enabled it explicitly.
  • Contract Web /admin/ai-traces: HTTP 200.

Scheduled loop smoke observation:

  • Smoke .env.smoke was changed to YUQEI_WORKER_TRACE_GOVERNANCE_ENABLED=true.
  • Current smoke worker settings: tenant_id=default, trace_governance_enabled=true, trace_governance_interval_seconds=300, trace_governance_batch_limit=100, trace_governance_worker_name=trace-governance-worker.
  • A dedicated default-tenant Trace run-scheduled-governance-ea4b8a5b10f8 was redacted and archived by the scheduled worker loop.
  • A second dedicated default-tenant Trace run-scheduled-execution-561f829c092b verified execution logging after the worker execution-recording hardening.
  • Observed worker_run_executions record: worker-run-9c0d8d73d0c6, worker_name=trace-governance-worker, trigger_type=worker_loop, triggered_by=worker-loop, queue_name=trace-governance, processed=1, succeeded=1, failed=0.
  • Observed worker_heartbeats for trace-governance-worker with status=ok.
  • Final remote checks: AI Platform schema-check ok; ai-platform-api, contract-api, contract-web, and worker containers healthy.

Observation script:

  • Added yuqei-infra/scripts/watch-trace-governance-worker.py.
  • The script polls:
    • GET /api/v1/runtime/worker-health
    • GET /api/v1/runtime/worker-runs
  • It verifies heartbeat freshness, minimum execution count, and failed execution count.
  • Short smoke command:
python yuqei-infra\scripts\watch-trace-governance-worker.py `
  --base-url http://192.168.3.90:18101 `
  --duration-seconds 5 `
  --interval-seconds 5 `
  --max-heartbeat-age-seconds 420 `
  --min-executions 1 `
  --execution-lookback-seconds 300

Observed result:

  • TRACE_GOVERNANCE_OBSERVATION_PASS
  • Artifact: project-docs/qa-artifacts/v2-trace-governance-observation/trace-governance-observation-2026-07-03T09-29-43.json
  • Final heartbeat age: 68.615 seconds.
  • Final trace-governance execution count in the lookback window: 2.
  • Failed execution count: 0.

30-minute smoke observation:

python yuqei-infra\scripts\watch-trace-governance-worker.py `
  --base-url http://192.168.3.90:18101 `
  --duration-seconds 1800 `
  --interval-seconds 60 `
  --max-heartbeat-age-seconds 420 `
  --min-executions 5

Result:

  • TRACE_GOVERNANCE_OBSERVATION_PASS
  • Artifact: project-docs/qa-artifacts/v2-trace-governance-observation/trace-governance-observation-2026-07-03T11-16-17.json
  • Observation window: 2026-07-03T10:46:14+08:00 to 2026-07-03T11:16:17+08:00.
  • Samples: 30.
  • Final execution count: 6.
  • Failed execution count: 0.
  • Final heartbeat age: 146.944 seconds.
  • Final remote checks after the window: AI Platform schema-check ok; ai-platform-api, contract-api, contract-web, and worker containers healthy.

4-hour smoke observation:

python yuqei-infra\scripts\watch-trace-governance-worker.py `
  --base-url http://192.168.3.90:18101 `
  --duration-seconds 14400 `
  --interval-seconds 300 `
  --max-heartbeat-age-seconds 420 `
  --min-executions 40

Result:

  • TRACE_GOVERNANCE_OBSERVATION_PASS
  • Artifact: project-docs/qa-artifacts/v2-trace-governance-observation/trace-governance-observation-2026-07-03T16-05-01.json
  • Observation window: 2026-07-03T12:04:58+08:00 to 2026-07-03T16:05:01+08:00.
  • Samples: 49.
  • Final execution count: 47.
  • Failed execution count: 0.
  • Final heartbeat age: 228.475 seconds.
  • Final remote checks after the window: AI Platform schema-check ok; ai-platform-api, contract-api, contract-web, and worker containers healthy.

Dedicated smoke data:

{
  "tenant": "trace-governance-hardening-20260702161706",
  "prompt_run_id": "run-policy-prompt-630fc17c76a5",
  "answer_run_id": "run-policy-answer-b2c7f7b86fca",
  "archived_count": 2,
  "active_count": 0
}

Verified policy behavior:

  • prompt_only: prompt preview, question, and retrieval query became [REDACTED]; answer and retrieved source metadata remained visible.
  • answer_only: answer and quality summary became [REDACTED]; prompt preview, question, and retrieval query remained visible.

Admin Runtime Visibility

Contract Web now exposes Trace governance status in the AI Runtime admin page:

  • Page: GET /admin/ai-runtime
  • Local proxy: POST /api/ai-platform/runtime/trace-governance/run-once
  • Upstream API: POST /api/v1/runtime/trace-governance/run-once

The page shows:

  • Trace governance heartbeat status and heartbeat age.
  • Trace governance executions in the last 30 minutes.
  • Recent governance run count and failed run count.
  • Latest governance run records.
  • A manual Run governance once action for smoke/admin checks.

Local UI verification on 2026-07-03:

  • npm run web:typecheck: passed.
  • GET http://localhost:3000/admin/ai-runtime: HTTP 200.
  • Trace governance panel rendered with heartbeat, last-30-minute executions, recent run count, failed count, and latest runs.
  • Manual local proxy check returned a successful Trace governance execution.
  • Screenshot artifact: project-docs/qa-artifacts/v2-trace-governance-admin-ui/ai-runtime-trace-governance.png.
  • Layout check at 1440px: no horizontal overflow.

Operational decision on 2026-07-03:

  • Smoke: keep YUQEI_WORKER_TRACE_GOVERNANCE_ENABLED=true so the background loop keeps accumulating observation data naturally.
  • Production: keep Trace governance disabled for now.
  • The 24-hour observation window is a pre-production checklist item, not a blocker for continuing current development.

Follow-up slice on 2026-07-03:

  • Trace retention/redaction/archive policy configuration was added in V2-phase3-ai-platform-trace-policy-config-2026-07-03.md.
  • The policy slice adds trace_governance_policies, tenant-scoped GET/PUT runtime endpoints, worker policy application, and an AI Runtime admin form.

Remaining Items

  • Keep smoke scheduled Trace governance enabled and run a 24-hour observation window before production enablement, because both 30-minute and 4-hour gates have passed.
  • Recommended 24-hour smoke observation command before production enablement:
python yuqei-infra\scripts\watch-trace-governance-worker.py `
  --base-url http://192.168.3.90:18101 `
  --duration-seconds 86400 `
  --interval-seconds 300 `
  --max-heartbeat-age-seconds 420 `
  --min-executions 250
  • Production should remain disabled until the smoke observation window has no worker errors and execution volume is acceptable.
  • Later cleanup can localize the new archive filter labels in the admin page once the existing mojibake text is repaired.
Reference in New Issue View Git Blame Copy Permalink