Retire stale training documentation debt
This commit is contained in:
Codex
@@ -28,7 +28,8 @@ Offline validation commands:
|
|||||||
|
|
||||||
Accepted user-facing surfaces:
|
Accepted user-facing surfaces:
|
||||||
|
|
||||||
- Dashboard Training panel
|
- Trading Workbench at `/`
|
||||||
|
- Training Console at `/training`
|
||||||
- public API routes
|
- public API routes
|
||||||
- offline public package import/export
|
- offline public package import/export
|
||||||
- public error catalog
|
- public error catalog
|
||||||
@@ -37,7 +38,7 @@ Accepted user-facing surfaces:
|
|||||||
- Acceptance and offline validation: M35 acceptance posture, Training Case acceptance manifest, focused milestone test suites, offline checks, and public validation.
|
- Acceptance and offline validation: M35 acceptance posture, Training Case acceptance manifest, focused milestone test suites, offline checks, and public validation.
|
||||||
- Public API and privacy contracts: API contract coverage, error contract coverage, and privacy contract coverage keep public routes audited while public responses scrub private solution material and internal solution details.
|
- Public API and privacy contracts: API contract coverage, error contract coverage, and privacy contract coverage keep public routes audited while public responses scrub private solution material and internal solution details.
|
||||||
- Package smoke and offline registry: offline package smoke covers local import/export with public package, public export, and public output flows; broader offline checks register `training_case_package`.
|
- Package smoke and offline registry: offline package smoke covers local import/export with public package, public export, and public output flows; broader offline checks register `training_case_package`.
|
||||||
- Dashboard and public docs privacy: dashboard syntax validation is registered as `dashboard_js_syntax`, and dashboard and public docs privacy smoke coverage keeps dashboard content to public case fields only.
|
- Dashboard and public docs privacy: trading workbench syntax validation is registered as `dashboard_js_syntax`, training console syntax validation is registered as `training_console_js_syntax`, and public docs privacy smoke coverage keeps Training content to public case fields only.
|
||||||
- Local Docker guard: single-set local Docker deployment uses the documented rebuild order and Docker smoke guard coverage for one active deployment set, one `ai-ict` container, and `healthy` status.
|
- Local Docker guard: single-set local Docker deployment uses the documented rebuild order and Docker smoke guard coverage for one active deployment set, one `ai-ict` container, and `healthy` status.
|
||||||
|
|
||||||
## Training public error catalog
|
## Training public error catalog
|
||||||
@@ -176,7 +177,7 @@ test "$(docker ps --filter label=com.docker.compose.project=ai-ict --format '{{.
|
|||||||
test "$(docker ps --filter name=^/ai-ict$ --format '{{.Names}}' | wc -l | tr -d ' ')" = "1"
|
test "$(docker ps --filter name=^/ai-ict$ --format '{{.Names}}' | wc -l | tr -d ' ')" = "1"
|
||||||
test "$(docker inspect -f '{{.State.Health.Status}}' ai-ict)" = "healthy"
|
test "$(docker inspect -f '{{.State.Health.Status}}' ai-ict)" = "healthy"
|
||||||
curl -fsS http://127.0.0.1:8000/health
|
curl -fsS http://127.0.0.1:8000/health
|
||||||
curl -fsS "http://127.0.0.1:8000/signals?instrument_id=1&limit=1"
|
curl -fsS "http://127.0.0.1:8000/trading/candidates?instrument_id=1&limit=1"
|
||||||
```
|
```
|
||||||
|
|
||||||
Post-deploy observe:
|
Post-deploy observe:
|
||||||
@@ -228,7 +229,7 @@ Record the current commit as the last-known-good SHA before deploy. If any gate
|
|||||||
- `python3 scripts/run_autonomous_cycle.py` - run the unattended cycle: offline checks, database probe, readiness checks, then guarded pipeline when the database is available; prints `Next actions` for the next unattended step
|
- `python3 scripts/run_autonomous_cycle.py` - run the unattended cycle: offline checks, database probe, readiness checks, then guarded pipeline when the database is available; prints `Next actions` for the next unattended step
|
||||||
- `python3 scripts/run_database_readiness.py` - run read-only database readiness checks for connection, schema, seed data, and candle availability; prints `next_actions` and `action_plan` in JSON mode
|
- `python3 scripts/run_database_readiness.py` - run read-only database readiness checks for connection, schema, seed data, and candle availability; prints `next_actions` and `action_plan` in JSON mode
|
||||||
- `.venv/bin/python scripts/run_trading_delivery_rehearsal.py` - run the P0 trading delivery rehearsal as one scripted flow: checklist drift check, historical repair decision, readiness/auto-backfill, guarded pipeline, deploy/health-check, and post-deploy observe
|
- `.venv/bin/python scripts/run_trading_delivery_rehearsal.py` - run the P0 trading delivery rehearsal as one scripted flow: checklist drift check, historical repair decision, readiness/auto-backfill, guarded pipeline, deploy/health-check, and post-deploy observe
|
||||||
- `.venv/bin/python scripts/run_offline_checks.py` - run the broader offline check suite, including the M35 acceptance posture through the Training Case acceptance manifest for focused milestone test suites, the Training Case package smoke registered as `training_case_package`, trading delivery checklist drift guard registered as `trading_delivery_readiness`, dashboard syntax validation as `dashboard_js_syntax`, and dashboard and public docs privacy smoke coverage; this public validation runs as offline checks without external services
|
- `.venv/bin/python scripts/run_offline_checks.py` - run the broader offline check suite, including the M35 acceptance posture through the Training Case acceptance manifest for focused milestone test suites, the Training Case package smoke registered as `training_case_package`, trading delivery checklist drift guard registered as `trading_delivery_readiness`, trading workbench syntax validation as `dashboard_js_syntax`, training console syntax validation as `training_console_js_syntax`, and dashboard and public docs privacy smoke coverage; this public validation runs as offline checks without external services
|
||||||
- `.venv/bin/python scripts/check_trading_delivery_readiness.py` - verify `README.md` and `RUNBOOK.md` still expose the locked trading delivery checklist for preflight, deploy, observe, and rollback without relying on chat memory
|
- `.venv/bin/python scripts/check_trading_delivery_readiness.py` - verify `README.md` and `RUNBOOK.md` still expose the locked trading delivery checklist for preflight, deploy, observe, and rollback without relying on chat memory
|
||||||
- `.venv/bin/python scripts/run_training_case_package.py --help` - show the offline package smoke help for public package import/export behavior, public export/public output semantics, and stable public error codes for invalid input paths and invalid output targets
|
- `.venv/bin/python scripts/run_training_case_package.py --help` - show the offline package smoke help for public package import/export behavior, public export/public output semantics, and stable public error codes for invalid input paths and invalid output targets
|
||||||
- `.venv/bin/python scripts/run_training_case_package.py --input tests/fixtures/training_case_package_m6.json --output /tmp/training_case_public_export.json` - run the Training Case offline package smoke, an import/export smoke with local files, and write public output as the public export in the public package shape; exported public JSON omits internal solution details
|
- `.venv/bin/python scripts/run_training_case_package.py --input tests/fixtures/training_case_package_m6.json --output /tmp/training_case_public_export.json` - run the Training Case offline package smoke, an import/export smoke with local files, and write public output as the public export in the public package shape; exported public JSON omits internal solution details
|
||||||
@@ -241,7 +242,7 @@ Record the current commit as the last-known-good SHA before deploy. If any gate
|
|||||||
- `python3 scripts/run_script_static_checks.py` - check automation scripts for unsafe ambient environment fallback patterns
|
- `python3 scripts/run_script_static_checks.py` - check automation scripts for unsafe ambient environment fallback patterns
|
||||||
- `python3 scripts/run_replay_file.py <candles.json|candles.jsonl|candles.csv>` - run offline no-lookahead candle replay without database access
|
- `python3 scripts/run_replay_file.py <candles.json|candles.jsonl|candles.csv>` - run offline no-lookahead candle replay without database access
|
||||||
- `python3 scripts/run_reports.py` - build sample reports from current services
|
- `python3 scripts/run_reports.py` - build sample reports from current services
|
||||||
- `python3 scripts/run_api_server.py` - start a lightweight HTTP API server on `127.0.0.1:8000` and open `/` for the minimal dashboard
|
- `python3 scripts/run_api_server.py` - start a lightweight HTTP API server on `127.0.0.1:8000`; `/` serves the trading workbench and `/training` serves the training console
|
||||||
- `python3 scripts/run_pipeline.py` - run the current end-to-end closed loop and print a structured summary
|
- `python3 scripts/run_pipeline.py` - run the current end-to-end closed loop and print a structured summary
|
||||||
|
|
||||||
Optional report environment variables used by `run_reports.py`:
|
Optional report environment variables used by `run_reports.py`:
|
||||||
@@ -258,8 +259,10 @@ Optional utility JSON output variables:
|
|||||||
- `SHOW_MODELS_OUTPUT_JSON`
|
- `SHOW_MODELS_OUTPUT_JSON`
|
||||||
- `RISK_GOVERNOR_OUTPUT_JSON`
|
- `RISK_GOVERNOR_OUTPUT_JSON`
|
||||||
- `TRADE_PLAN_CANDIDATES_OUTPUT_JSON`
|
- `TRADE_PLAN_CANDIDATES_OUTPUT_JSON`
|
||||||
|
- `TRADING_JOURNAL_OUTPUT_JSON`
|
||||||
- `TRADING_PLANS_OUTPUT_JSON`
|
- `TRADING_PLANS_OUTPUT_JSON`
|
||||||
- `TRADING_REHEARSAL_OUTPUT_JSON`
|
- `TRADING_REHEARSAL_OUTPUT_JSON`
|
||||||
|
- `LIVE_ADAPTER_OUTPUT_JSON`
|
||||||
- `BACKTEST_OUTPUT_JSON`
|
- `BACKTEST_OUTPUT_JSON`
|
||||||
- `EXECUTION_TICKET_OUTPUT_JSON`
|
- `EXECUTION_TICKET_OUTPUT_JSON`
|
||||||
|
|
||||||
|
|||||||
+29
-10
@@ -22,6 +22,8 @@ cd /home/yuqei/project/ai-exchange
|
|||||||
.venv/bin/python scripts/run_api_server.py
|
.venv/bin/python scripts/run_api_server.py
|
||||||
```
|
```
|
||||||
|
|
||||||
|
This serves the trading workbench at `/` and the training console at `/training`.
|
||||||
|
|
||||||
Alternative entrypoints already documented by the project:
|
Alternative entrypoints already documented by the project:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -141,7 +143,7 @@ test "$(docker ps --filter label=com.docker.compose.project=ai-ict --format '{{.
|
|||||||
test "$(docker ps --filter name=^/ai-ict$ --format '{{.Names}}' | wc -l | tr -d ' ')" = "1"
|
test "$(docker ps --filter name=^/ai-ict$ --format '{{.Names}}' | wc -l | tr -d ' ')" = "1"
|
||||||
test "$(docker inspect -f '{{.State.Health.Status}}' ai-ict)" = "healthy"
|
test "$(docker inspect -f '{{.State.Health.Status}}' ai-ict)" = "healthy"
|
||||||
curl -fsS http://127.0.0.1:8000/health
|
curl -fsS http://127.0.0.1:8000/health
|
||||||
curl -fsS "http://127.0.0.1:8000/signals?instrument_id=1&limit=1"
|
curl -fsS "http://127.0.0.1:8000/trading/candidates?instrument_id=1&limit=1"
|
||||||
```
|
```
|
||||||
|
|
||||||
Post-deploy observe:
|
Post-deploy observe:
|
||||||
@@ -212,6 +214,32 @@ git remote set-url origin gitea-water:yuqei/ai-exchange.git
|
|||||||
|
|
||||||
Avoid using the old HTTP remote here unless HTTP credentials are explicitly configured for the current shell user.
|
Avoid using the old HTTP remote here unless HTTP credentials are explicitly configured for the current shell user.
|
||||||
|
|
||||||
|
## Local Docker deployment
|
||||||
|
|
||||||
|
Use the single-set local Docker deployment policy for local server deployment:
|
||||||
|
|
||||||
|
- deploy directory `/root/deploy/ai-ict`
|
||||||
|
- compose project `ai-ict`
|
||||||
|
- container `ai-ict`
|
||||||
|
- exactly one active deployment set
|
||||||
|
- one active `ai-ict` container
|
||||||
|
- do not create a second local stack
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -p ai-ict -f /root/deploy/ai-ict/docker-compose.yml down --remove-orphans
|
||||||
|
docker compose -p ai-ict -f /root/deploy/ai-ict/docker-compose.yml up -d --build
|
||||||
|
docker compose -p ai-ict -f /root/deploy/ai-ict/docker-compose.yml ps
|
||||||
|
test "$(docker ps --filter label=com.docker.compose.project=ai-ict --format '{{.Names}}' | wc -l | tr -d ' ')" = "1"
|
||||||
|
test "$(docker ps --filter name=^/ai-ict$ --format '{{.Names}}' | wc -l | tr -d ' ')" = "1"
|
||||||
|
test "$(docker inspect -f '{{.State.Health.Status}}' ai-ict)" = "healthy"
|
||||||
|
```
|
||||||
|
|
||||||
|
Check the documented local demo readiness without starting containers; this offline check does not invoke Docker:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python scripts/check_local_demo_readiness.py
|
||||||
|
```
|
||||||
|
|
||||||
## Docker
|
## Docker
|
||||||
|
|
||||||
Repo-local compose:
|
Repo-local compose:
|
||||||
@@ -222,15 +250,6 @@ docker compose up -d --build
|
|||||||
docker compose ps
|
docker compose ps
|
||||||
```
|
```
|
||||||
|
|
||||||
Documented single-set local deployment policy from the repo README:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd /root/deploy/ai-ict
|
|
||||||
docker compose -p ai-ict -f /root/deploy/ai-ict/docker-compose.yml down --remove-orphans
|
|
||||||
docker compose -p ai-ict -f /root/deploy/ai-ict/docker-compose.yml up -d --build
|
|
||||||
docker compose -p ai-ict -f /root/deploy/ai-ict/docker-compose.yml ps
|
|
||||||
```
|
|
||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
|
|
||||||
- Existing services in `docker-compose.yml`: `ai-ict`
|
- Existing services in `docker-compose.yml`: `ai-ict`
|
||||||
|
|||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Training Case Package Smoke M6
|
||||||
|
|
||||||
|
Use this offline package smoke to verify Training public import/export behavior.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python scripts/run_training_case_package.py \
|
||||||
|
--input tests/fixtures/training_case_package_m6.json \
|
||||||
|
--output /tmp/training_case_public_export.json
|
||||||
|
```
|
||||||
|
|
||||||
|
What this verifies:
|
||||||
|
|
||||||
|
- import/export smoke
|
||||||
|
- public package
|
||||||
|
- public export
|
||||||
|
- public output
|
||||||
|
- one-line JSON summary
|
||||||
|
- stable public error codes
|
||||||
|
- `input_file_not_found`
|
||||||
|
- `input_path_is_directory`
|
||||||
|
- `output_path_is_directory`
|
||||||
|
- `output_path_unwritable`
|
||||||
|
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
# Training Console Public Surface v1
|
||||||
|
|
||||||
|
This document defines the current public-facing Training module surface for `ai-ict`.
|
||||||
|
|
||||||
|
## Product Position
|
||||||
|
|
||||||
|
- Training is a separate public module from the trading workbench.
|
||||||
|
- The public browser entry is `/training`.
|
||||||
|
- The trading workbench stays at `/`.
|
||||||
|
- Training assets live in `src/web/training.html` and `src/web/training.js`.
|
||||||
|
|
||||||
|
## Dashboard Entry
|
||||||
|
|
||||||
|
- available cases
|
||||||
|
- user progress
|
||||||
|
- attempt start
|
||||||
|
- annotation note save
|
||||||
|
- attempt submit
|
||||||
|
- public case fields only
|
||||||
|
|
||||||
|
## API Surface
|
||||||
|
|
||||||
|
| Method | Path | Capability |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| GET | `/training/cases` | list cases with status, skill tag, instrument, and limit filters |
|
||||||
|
| GET | `/training/cases/{case_id}` | read one public case |
|
||||||
|
| GET | `/training/cases/export` | export a public package with the same case filters |
|
||||||
|
| POST | `/training/cases/import` | import a versioned public package with cases plus tasks |
|
||||||
|
| POST | `/training/attempts` | start an attempt for one case |
|
||||||
|
| POST | `/training/attempts/{attempt_id}/annotations` | save an annotation payload for one attempt |
|
||||||
|
| POST | `/training/attempts/{attempt_id}/submit` | submit an attempt with optional score data |
|
||||||
|
| GET | `/training/progress` | list progress with user, skill tag, and limit filters |
|
||||||
|
|
||||||
|
## Validation And Errors
|
||||||
|
|
||||||
|
Training public error catalog:
|
||||||
|
|
||||||
|
- deterministic public validation errors
|
||||||
|
- `400` with `invalid_json_body`
|
||||||
|
- `400` with `missing_required_json_field:<field>`
|
||||||
|
- `400` with `invalid_json_field:<field>`
|
||||||
|
- `400` with `invalid_path_param:<field>`
|
||||||
|
- `400` with `invalid_limit_query_param:limit`
|
||||||
|
- `404` with `not_found`
|
||||||
|
- `500` with `internal_server_error`
|
||||||
|
- masks raw internals
|
||||||
|
|
||||||
|
Package CLI public errors:
|
||||||
|
|
||||||
|
- one-line JSON
|
||||||
|
- `input_file_not_found`
|
||||||
|
- `input_path_is_directory`
|
||||||
|
- `output_path_is_directory`
|
||||||
|
- `output_path_unwritable`
|
||||||
|
|
||||||
|
## Package Smoke
|
||||||
|
|
||||||
|
- public package
|
||||||
|
- public export
|
||||||
|
- public output
|
||||||
|
- internal solution details are excluded from exported public JSON
|
||||||
|
- fixture path: `tests/fixtures/training_case_package_m6.json`
|
||||||
|
- example command: `.venv/bin/python scripts/run_training_case_package.py --input tests/fixtures/training_case_package_m6.json --output /tmp/training_case_public_export.json`
|
||||||
|
|
||||||
+2
-2
@@ -3,8 +3,8 @@
|
|||||||
"version": "0.1.0",
|
"version": "0.1.0",
|
||||||
"private": true,
|
"private": true,
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"lint": ".\\.venv\\Scripts\\python.exe scripts/run_script_static_checks.py && .\\.venv\\Scripts\\python.exe scripts/run_migration_static_checks.py && .\\.venv\\Scripts\\python.exe -m unittest tests.unit.test_trading_plan_service tests.unit.test_market_state_and_risk_governor tests.unit.test_target_selection_service tests.unit.test_trade_plan_candidate_service tests.unit.test_backtest_service tests.unit.test_backtest_script tests.unit.test_execution_service tests.unit.test_execution_ticket_script tests.unit.test_live_adapter_service tests.unit.test_live_adapter_script tests.unit.test_trading_journal_service tests.unit.test_api_server tests.unit.test_api_and_main tests.unit.test_report_services",
|
"lint": ".\\.venv\\Scripts\\python.exe scripts/run_script_static_checks.py && .\\.venv\\Scripts\\python.exe scripts/run_migration_static_checks.py && .\\.venv\\Scripts\\python.exe -m unittest discover -s tests/unit -p \"test_*.py\"",
|
||||||
"build": ".\\.venv\\Scripts\\python.exe -m compileall -q src scripts tests",
|
"build": ".\\.venv\\Scripts\\python.exe -m compileall -q src scripts tests",
|
||||||
"visual:snapshots": "node --check src/web/app.js"
|
"visual:snapshots": "node --check src/web/app.js && node --check src/web/training.js"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ LOCAL_DEMO_COMPOSE_PREFIX = (
|
|||||||
)
|
)
|
||||||
LOCAL_DEMO_DOCS = (
|
LOCAL_DEMO_DOCS = (
|
||||||
ROOT / "README.md",
|
ROOT / "README.md",
|
||||||
ROOT / "docs" / "training_case_engine_m1_m6_product_readme.md",
|
ROOT / "RUNBOOK.md",
|
||||||
)
|
)
|
||||||
LOCAL_DEMO_REBUILD_COMMANDS = (
|
LOCAL_DEMO_REBUILD_COMMANDS = (
|
||||||
f"{LOCAL_DEMO_COMPOSE_PREFIX} down --remove-orphans",
|
f"{LOCAL_DEMO_COMPOSE_PREFIX} down --remove-orphans",
|
||||||
|
|||||||
@@ -42,7 +42,7 @@ TRADING_DELIVERY_COMMAND_GROUPS = (
|
|||||||
'test "$(docker ps --filter name=^/ai-ict$ --format \'{{.Names}}\' | wc -l | tr -d \' \')" = "1"',
|
'test "$(docker ps --filter name=^/ai-ict$ --format \'{{.Names}}\' | wc -l | tr -d \' \')" = "1"',
|
||||||
'test "$(docker inspect -f \'{{.State.Health.Status}}\' ai-ict)" = "healthy"',
|
'test "$(docker inspect -f \'{{.State.Health.Status}}\' ai-ict)" = "healthy"',
|
||||||
"curl -fsS http://127.0.0.1:8000/health",
|
"curl -fsS http://127.0.0.1:8000/health",
|
||||||
'curl -fsS "http://127.0.0.1:8000/signals?instrument_id=1&limit=1"',
|
'curl -fsS "http://127.0.0.1:8000/trading/candidates?instrument_id=1&limit=1"',
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
(
|
(
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ DEFAULT_CHECKS = [
|
|||||||
("script_static_checks", [sys.executable, "scripts/run_script_static_checks.py"]),
|
("script_static_checks", [sys.executable, "scripts/run_script_static_checks.py"]),
|
||||||
("trading_delivery_readiness", [sys.executable, "scripts/check_trading_delivery_readiness.py"]),
|
("trading_delivery_readiness", [sys.executable, "scripts/check_trading_delivery_readiness.py"]),
|
||||||
("dashboard_js_syntax", ["node", "--check", "src/web/app.js"]),
|
("dashboard_js_syntax", ["node", "--check", "src/web/app.js"]),
|
||||||
|
("training_console_js_syntax", ["node", "--check", "src/web/training.js"]),
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -378,7 +378,7 @@ def run_deploy_and_health_checks(
|
|||||||
|
|
||||||
for name, url in (
|
for name, url in (
|
||||||
("http_health", f"http://127.0.0.1:{deploy_port}/health"),
|
("http_health", f"http://127.0.0.1:{deploy_port}/health"),
|
||||||
("http_signals", f"http://127.0.0.1:{deploy_port}/signals?instrument_id=1&limit=1"),
|
("http_trading_candidates", f"http://127.0.0.1:{deploy_port}/trading/candidates?instrument_id=1&limit=1"),
|
||||||
):
|
):
|
||||||
step = _run_step(
|
step = _run_step(
|
||||||
name=name,
|
name=name,
|
||||||
|
|||||||
@@ -112,6 +112,8 @@ def _build_parser() -> argparse.ArgumentParser:
|
|||||||
|
|
||||||
|
|
||||||
def _load_package(input_path: Path) -> dict:
|
def _load_package(input_path: Path) -> dict:
|
||||||
|
if input_path.is_dir():
|
||||||
|
raise _PublicCliError("input_path_is_directory")
|
||||||
with input_path.open("r", encoding="utf-8") as handle:
|
with input_path.open("r", encoding="utf-8") as handle:
|
||||||
payload = json.load(handle)
|
payload = json.load(handle)
|
||||||
if not isinstance(payload, dict):
|
if not isinstance(payload, dict):
|
||||||
|
|||||||
@@ -47,6 +47,9 @@ from src.api.training import (
|
|||||||
parse_quality_sort_query,
|
parse_quality_sort_query,
|
||||||
submit_training_attempt,
|
submit_training_attempt,
|
||||||
)
|
)
|
||||||
|
from src.services.execution import ExecutionService
|
||||||
|
|
||||||
|
|
||||||
def build_api_dependencies() -> dict:
|
def build_api_dependencies() -> dict:
|
||||||
return {
|
return {
|
||||||
"health": get_health,
|
"health": get_health,
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -12,7 +12,7 @@ ROOT = Path(__file__).resolve().parents[2]
|
|||||||
CHECK_COMMAND = ".venv/bin/python scripts/check_local_demo_readiness.py"
|
CHECK_COMMAND = ".venv/bin/python scripts/check_local_demo_readiness.py"
|
||||||
EXPECTED_DOC_PATHS = (
|
EXPECTED_DOC_PATHS = (
|
||||||
ROOT / "README.md",
|
ROOT / "README.md",
|
||||||
ROOT / "docs" / "training_case_engine_m1_m6_product_readme.md",
|
ROOT / "RUNBOOK.md",
|
||||||
)
|
)
|
||||||
EXPECTED_REBUILD_COMMANDS = (
|
EXPECTED_REBUILD_COMMANDS = (
|
||||||
"docker compose -p ai-ict -f /root/deploy/ai-ict/docker-compose.yml down --remove-orphans",
|
"docker compose -p ai-ict -f /root/deploy/ai-ict/docker-compose.yml down --remove-orphans",
|
||||||
|
|||||||
@@ -44,7 +44,7 @@ EXPECTED_COMMAND_GROUPS = (
|
|||||||
'test "$(docker ps --filter name=^/ai-ict$ --format \'{{.Names}}\' | wc -l | tr -d \' \')" = "1"',
|
'test "$(docker ps --filter name=^/ai-ict$ --format \'{{.Names}}\' | wc -l | tr -d \' \')" = "1"',
|
||||||
'test "$(docker inspect -f \'{{.State.Health.Status}}\' ai-ict)" = "healthy"',
|
'test "$(docker inspect -f \'{{.State.Health.Status}}\' ai-ict)" = "healthy"',
|
||||||
"curl -fsS http://127.0.0.1:8000/health",
|
"curl -fsS http://127.0.0.1:8000/health",
|
||||||
'curl -fsS "http://127.0.0.1:8000/signals?instrument_id=1&limit=1"',
|
'curl -fsS "http://127.0.0.1:8000/trading/candidates?instrument_id=1&limit=1"',
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
(
|
(
|
||||||
|
|||||||
@@ -78,7 +78,7 @@ class TradingDeliveryRehearsalTests(unittest.TestCase):
|
|||||||
|
|
||||||
def fake_runner(command, **kwargs):
|
def fake_runner(command, **kwargs):
|
||||||
calls.append(command)
|
calls.append(command)
|
||||||
script = command[1] if len(command) > 1 and command[0].endswith("python") else command[0]
|
script = command[1] if len(command) > 1 and "python" in Path(command[0]).name.lower() else command[0]
|
||||||
if script.endswith("check_trading_delivery_readiness.py"):
|
if script.endswith("check_trading_delivery_readiness.py"):
|
||||||
return SimpleNamespace(returncode=0, stdout="ok\n", stderr="")
|
return SimpleNamespace(returncode=0, stdout="ok\n", stderr="")
|
||||||
if script.endswith("run_historical_compatibility_repair.py"):
|
if script.endswith("run_historical_compatibility_repair.py"):
|
||||||
@@ -130,7 +130,7 @@ class TradingDeliveryRehearsalTests(unittest.TestCase):
|
|||||||
|
|
||||||
def test_rehearsal_blocks_when_reference_migration_is_deferred(self) -> None:
|
def test_rehearsal_blocks_when_reference_migration_is_deferred(self) -> None:
|
||||||
def fake_runner(command, **kwargs):
|
def fake_runner(command, **kwargs):
|
||||||
script = command[1] if len(command) > 1 and command[0].endswith("python") else command[0]
|
script = command[1] if len(command) > 1 and "python" in Path(command[0]).name.lower() else command[0]
|
||||||
if script.endswith("check_trading_delivery_readiness.py"):
|
if script.endswith("check_trading_delivery_readiness.py"):
|
||||||
return SimpleNamespace(returncode=0, stdout="ok\n", stderr="")
|
return SimpleNamespace(returncode=0, stdout="ok\n", stderr="")
|
||||||
if script.endswith("run_historical_compatibility_repair.py"):
|
if script.endswith("run_historical_compatibility_repair.py"):
|
||||||
@@ -169,7 +169,7 @@ class TradingDeliveryRehearsalTests(unittest.TestCase):
|
|||||||
|
|
||||||
def test_rehearsal_blocks_when_autonomous_cycle_reports_quality_gate_failure(self) -> None:
|
def test_rehearsal_blocks_when_autonomous_cycle_reports_quality_gate_failure(self) -> None:
|
||||||
def fake_runner(command, **kwargs):
|
def fake_runner(command, **kwargs):
|
||||||
script = command[1] if len(command) > 1 and command[0].endswith("python") else command[0]
|
script = command[1] if len(command) > 1 and "python" in Path(command[0]).name.lower() else command[0]
|
||||||
if script.endswith("check_trading_delivery_readiness.py"):
|
if script.endswith("check_trading_delivery_readiness.py"):
|
||||||
return SimpleNamespace(returncode=0, stdout="ok\n", stderr="")
|
return SimpleNamespace(returncode=0, stdout="ok\n", stderr="")
|
||||||
if script.endswith("run_historical_compatibility_repair.py"):
|
if script.endswith("run_historical_compatibility_repair.py"):
|
||||||
|
|||||||
@@ -1,935 +1,110 @@
|
|||||||
import ast
|
|
||||||
import contextlib
|
import contextlib
|
||||||
import io
|
import io
|
||||||
import json
|
import json
|
||||||
import re
|
import re
|
||||||
import shlex
|
import shlex
|
||||||
import sys
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
import unittest
|
||||||
from unittest import mock
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from types import SimpleNamespace
|
from types import SimpleNamespace
|
||||||
|
|
||||||
from scripts import run_offline_checks
|
from scripts import run_offline_checks
|
||||||
from scripts import run_training_case_package as training_cli
|
from scripts import run_training_case_package as training_cli
|
||||||
from src.services.training import TrainingCaseService
|
|
||||||
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[2]
|
ROOT = Path(__file__).resolve().parents[2]
|
||||||
DOCUMENTATION_CONSISTENCY_TEST = ROOT / "tests" / "unit" / "test_documentation_consistency.py"
|
|
||||||
DOC_EXAMPLE = ROOT / "docs" / "examples" / "training_case_package_smoke_m6.md"
|
|
||||||
PRODUCT_README = ROOT / "docs" / "training_case_engine_m1_m6_product_readme.md"
|
|
||||||
README = ROOT / "README.md"
|
README = ROOT / "README.md"
|
||||||
|
TRAINING_PUBLIC_DOC = ROOT / "docs" / "training_console_public_surface_v1.md"
|
||||||
|
DOC_EXAMPLE = ROOT / "docs" / "examples" / "training_case_package_smoke_m6.md"
|
||||||
FIXTURE = ROOT / "tests" / "fixtures" / "training_case_package_m6.json"
|
FIXTURE = ROOT / "tests" / "fixtures" / "training_case_package_m6.json"
|
||||||
FIXTURE_REF = "tests/fixtures/training_case_package_m6.json"
|
FIXTURE_REF = "tests/fixtures/training_case_package_m6.json"
|
||||||
|
PACKAGE_SMOKE_COMMAND = ".venv/bin/python scripts/run_training_case_package.py"
|
||||||
OFFLINE_CHECK_COMMAND = ".venv/bin/python scripts/run_offline_checks.py"
|
OFFLINE_CHECK_COMMAND = ".venv/bin/python scripts/run_offline_checks.py"
|
||||||
OFFLINE_CHECK_NAME = "training_case_package"
|
|
||||||
OFFLINE_CHECK_DOC_NAME = f"`{OFFLINE_CHECK_NAME}`"
|
|
||||||
PACKAGE_SMOKE_SCRIPT = "scripts/run_training_case_package.py"
|
|
||||||
PACKAGE_SMOKE_COMMAND = f".venv/bin/python {PACKAGE_SMOKE_SCRIPT}"
|
|
||||||
PACKAGE_SMOKE_HELP_COMMAND = f"{PACKAGE_SMOKE_COMMAND} --help"
|
|
||||||
DASHBOARD_SYNTAX_CHECK_NAME = "dashboard_js_syntax"
|
|
||||||
DASHBOARD_SYNTAX_CHECK_PATH = "src/web/app.js"
|
|
||||||
OFFLINE_CHECK_REGISTRY_NAMES = (
|
|
||||||
"unit_tests",
|
|
||||||
OFFLINE_CHECK_NAME,
|
|
||||||
"migration_static_checks",
|
|
||||||
"script_static_checks",
|
|
||||||
"trading_delivery_readiness",
|
|
||||||
DASHBOARD_SYNTAX_CHECK_NAME,
|
|
||||||
)
|
|
||||||
CURRENT_TRAINING_DOC_MATURITY_END = 28
|
|
||||||
CURRENT_TRAINING_DOC_MATURITY_MARKER = f"M1-M{CURRENT_TRAINING_DOC_MATURITY_END}"
|
|
||||||
CURRENT_TRAINING_DOC_MATURITY_SPAN = f"through M{CURRENT_TRAINING_DOC_MATURITY_END}"
|
|
||||||
TRAINING_DOCS_CLI_TEST_CLASS = "TrainingCaseDocsCliM10Tests"
|
|
||||||
DOCUMENTATION_CONSISTENCY_TEST_CLASS = "DocumentationConsistencyTests"
|
|
||||||
PUBLIC_FIELD = "answer" + "_key"
|
|
||||||
PRIVATE_MATERIAL_PHRASES = ("private solution material", "internal solution details")
|
|
||||||
PUBLIC_PACKAGE_DOCS = (DOC_EXAMPLE, README)
|
|
||||||
PACKAGE_SMOKE_SURFACE_DOCS = (DOC_EXAMPLE, README, PRODUCT_README)
|
|
||||||
PUBLIC_PACKAGE_REQUIRED_TEXT = (
|
|
||||||
".venv/bin/python scripts/run_training_case_package.py",
|
|
||||||
FIXTURE_REF,
|
|
||||||
"--input",
|
|
||||||
"--output",
|
|
||||||
"import/export smoke",
|
|
||||||
"public export",
|
|
||||||
"public package",
|
|
||||||
"public output",
|
|
||||||
"internal solution details",
|
|
||||||
)
|
|
||||||
PACKAGE_IMPORT_EXPORT_SURFACE_TEXT = (
|
|
||||||
"import",
|
|
||||||
"export",
|
|
||||||
"public package",
|
|
||||||
"public export",
|
|
||||||
"public output",
|
|
||||||
)
|
|
||||||
PACKAGE_CLI_HELP_DISCOVERY_TEXT = (
|
|
||||||
"offline package smoke",
|
|
||||||
"import/export smoke",
|
|
||||||
"public package",
|
|
||||||
"public export",
|
|
||||||
"public output",
|
|
||||||
"one-line json",
|
|
||||||
"stable public error codes",
|
|
||||||
"input_file_not_found",
|
|
||||||
"input_path_is_directory",
|
|
||||||
"output_path_is_directory",
|
|
||||||
"output_path_unwritable",
|
|
||||||
)
|
|
||||||
PACKAGE_DOC_HELP_ERROR_TEXT = (
|
|
||||||
PACKAGE_SMOKE_HELP_COMMAND,
|
|
||||||
"one-line JSON",
|
|
||||||
"`passed`",
|
|
||||||
"`input`",
|
|
||||||
"`output`",
|
|
||||||
"`error`",
|
|
||||||
"`input_file_not_found`",
|
|
||||||
"`input_path_is_directory`",
|
|
||||||
"`output_path_is_directory`",
|
|
||||||
"`output_path_unwritable`",
|
|
||||||
)
|
|
||||||
PACKAGE_CLI_PUBLIC_ERROR_CODES = (
|
|
||||||
"input_file_not_found",
|
|
||||||
"input_path_is_directory",
|
|
||||||
"output_path_is_directory",
|
|
||||||
"output_path_unwritable",
|
|
||||||
)
|
|
||||||
PACKAGE_CLI_PUBLIC_ERROR_EXAMPLE_CODES = PACKAGE_CLI_PUBLIC_ERROR_CODES[:3]
|
|
||||||
PACKAGE_CLI_PUBLIC_ERROR_TEST_INVENTORY = {
|
|
||||||
"docs": (
|
|
||||||
"test_package_smoke_docs_publish_help_and_stable_public_error_wording",
|
|
||||||
),
|
|
||||||
"help": (
|
|
||||||
"test_cli_help_discloses_offline_public_import_export_and_error_contract",
|
|
||||||
),
|
|
||||||
"readiness": (
|
|
||||||
"test_cli_error_summary_uses_public_output_target_code_without_runtime_leaks",
|
|
||||||
),
|
|
||||||
}
|
|
||||||
PRODUCT_PACKAGE_WORDING = (
|
|
||||||
CURRENT_TRAINING_DOC_MATURITY_MARKER,
|
|
||||||
"offline checks registry smoke",
|
|
||||||
"public package",
|
|
||||||
"public export",
|
|
||||||
"public output",
|
|
||||||
"private solution material",
|
|
||||||
"internal solution details",
|
|
||||||
)
|
|
||||||
PRIVATE_DOC_DRIFT_PHRASES = ("private " + "answer", "answer " + "detail")
|
|
||||||
PRIVATE_FIXTURE_DRIFT_PHRASES = PRIVATE_MATERIAL_PHRASES + PRIVATE_DOC_DRIFT_PHRASES
|
|
||||||
RESTRICTED_PUBLIC_TEXT = (
|
|
||||||
"data" + "base",
|
|
||||||
"data" + "base_url",
|
|
||||||
"post" + "gresql",
|
|
||||||
"psy" + "copg",
|
|
||||||
"pass" + "word",
|
|
||||||
"pass" + "wd",
|
|
||||||
"tok" + "en",
|
|
||||||
"sec" + "ret",
|
|
||||||
"api" + "_key",
|
|
||||||
)
|
|
||||||
TRAINING_M24_M29_PACKAGE_DOCS_TEST_INVENTORY = {
|
|
||||||
"package_smoke_privacy": (
|
|
||||||
"test_public_privacy_guard_rejects_solution_wording_drift",
|
|
||||||
"test_public_fixture_schema_contract_is_minimal_and_demo_safe",
|
|
||||||
"test_fixture_is_compatible_with_public_service_and_cli_export",
|
|
||||||
"test_smoke_docs_and_product_readme_share_fixture_and_public_export_contract",
|
|
||||||
"test_example_command_uses_supported_cli_flags_and_fixture",
|
|
||||||
"test_example_documents_cli_summary_keys_and_output_semantics",
|
|
||||||
"test_cli_error_summary_uses_public_output_target_code_without_runtime_leaks",
|
|
||||||
"test_readme_and_product_docs_advertise_public_offline_smoke_contract",
|
|
||||||
"test_product_readme_uses_current_maturity_and_public_private_package_language",
|
|
||||||
"test_public_package_smoke_docs_lock_import_export_privacy_language",
|
|
||||||
"test_cli_help_discloses_offline_public_import_export_and_error_contract",
|
|
||||||
"test_package_smoke_docs_publish_help_and_stable_public_error_wording",
|
|
||||||
"test_package_cli_public_error_inventory_locks_docs_help_and_readiness",
|
|
||||||
"test_m1_m28_package_smoke_docs_cover_static_import_export_surface",
|
|
||||||
),
|
|
||||||
"offline_registry_coverage": (
|
|
||||||
"test_offline_check_registry_name_matches_package_smoke_docs",
|
|
||||||
"test_offline_checks_registry_runs_training_package_smoke_entry",
|
|
||||||
"test_training_docs_name_broader_offline_check_path",
|
|
||||||
),
|
|
||||||
"m1_m28_maturity_wording": (
|
|
||||||
"test_product_readme_uses_current_maturity_and_public_private_package_language",
|
|
||||||
"test_m1_m28_package_smoke_docs_cover_static_import_export_surface",
|
|
||||||
),
|
|
||||||
}
|
|
||||||
TRAINING_M33_REQUIRED_INVENTORY_GROUPS = (
|
|
||||||
"package_smoke_docs_command_help",
|
|
||||||
"fixture_public_shape",
|
|
||||||
"fixture_private_solution_absence",
|
|
||||||
"service_import_export_public_output",
|
|
||||||
"cli_smoke_export_import_summary",
|
|
||||||
"readme_product_docs_offline_registry_refs",
|
|
||||||
)
|
|
||||||
TRAINING_M33_PACKAGE_FIXTURE_IMPORT_EXPORT_PRIVACY_INVENTORY = {
|
|
||||||
"package_smoke_docs_command_help": (
|
|
||||||
"test_example_command_uses_supported_cli_flags_and_fixture",
|
|
||||||
"test_public_package_smoke_docs_lock_import_export_privacy_language",
|
|
||||||
"test_cli_help_discloses_offline_public_import_export_and_error_contract",
|
|
||||||
"test_package_smoke_docs_publish_help_and_stable_public_error_wording",
|
|
||||||
"test_m1_m28_package_smoke_docs_cover_static_import_export_surface",
|
|
||||||
),
|
|
||||||
"fixture_public_shape": (
|
|
||||||
"test_public_fixture_schema_contract_is_minimal_and_demo_safe",
|
|
||||||
),
|
|
||||||
"fixture_private_solution_absence": (
|
|
||||||
"test_public_privacy_guard_rejects_solution_wording_drift",
|
|
||||||
"test_public_fixture_schema_contract_is_minimal_and_demo_safe",
|
|
||||||
"test_fixture_is_compatible_with_public_service_and_cli_export",
|
|
||||||
),
|
|
||||||
"service_import_export_public_output": (
|
|
||||||
"test_fixture_is_compatible_with_public_service_and_cli_export",
|
|
||||||
),
|
|
||||||
"cli_smoke_export_import_summary": (
|
|
||||||
"test_fixture_is_compatible_with_public_service_and_cli_export",
|
|
||||||
"test_example_documents_cli_summary_keys_and_output_semantics",
|
|
||||||
"test_cli_error_summary_uses_public_output_target_code_without_runtime_leaks",
|
|
||||||
),
|
|
||||||
"readme_product_docs_offline_registry_refs": (
|
|
||||||
"test_smoke_docs_and_product_readme_share_fixture_and_public_export_contract",
|
|
||||||
"test_readme_and_product_docs_advertise_public_offline_smoke_contract",
|
|
||||||
"test_offline_check_registry_name_matches_package_smoke_docs",
|
|
||||||
"test_offline_checks_registry_runs_training_package_smoke_entry",
|
|
||||||
"test_training_docs_name_broader_offline_check_path",
|
|
||||||
"test_training_readmes_name_dashboard_privacy_and_offline_registry_coverage",
|
|
||||||
),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _example_command_parts() -> list[str]:
|
def _documented_command_parts() -> list[str]:
|
||||||
text = DOC_EXAMPLE.read_text(encoding="utf-8")
|
text = DOC_EXAMPLE.read_text(encoding="utf-8")
|
||||||
match = re.search(r"```bash\n(?P<command>.*?)\n```", text, flags=re.DOTALL)
|
match = re.search(r"```bash\n(?P<command>.*?)\n```", text, flags=re.DOTALL)
|
||||||
if match is None:
|
if match is None:
|
||||||
raise AssertionError("missing bash command block")
|
raise AssertionError("missing documented package smoke command")
|
||||||
command = re.sub(r"\\\n\s*", " ", match.group("command").strip())
|
return shlex.split(re.sub(r"\\\n\s*", " ", match.group("command").strip()))
|
||||||
return shlex.split(command)
|
|
||||||
|
|
||||||
|
|
||||||
def _fixture_payload() -> dict:
|
|
||||||
return json.loads(FIXTURE.read_text(encoding="utf-8"))
|
|
||||||
|
|
||||||
|
|
||||||
def _package_doc_text(path: Path) -> str:
|
|
||||||
text = path.read_text(encoding="utf-8")
|
|
||||||
if path != README:
|
|
||||||
return text
|
|
||||||
|
|
||||||
lines = [line for line in text.splitlines() if "run_training_case_package.py" in line]
|
|
||||||
if not lines:
|
|
||||||
raise AssertionError("README is missing the package smoke command")
|
|
||||||
return "\n".join(lines)
|
|
||||||
|
|
||||||
|
|
||||||
def _package_smoke_flags_from_source() -> tuple[str, ...]:
|
|
||||||
source = Path(training_cli.__file__).read_text(encoding="utf-8")
|
|
||||||
return tuple(re.findall(r'add_argument\("(?P<flag>--[a-z-]+)"', source))
|
|
||||||
|
|
||||||
|
|
||||||
def _package_cli_public_error_inventory_method_names() -> tuple[str, ...]:
|
|
||||||
return tuple(
|
|
||||||
name
|
|
||||||
for names in PACKAGE_CLI_PUBLIC_ERROR_TEST_INVENTORY.values()
|
|
||||||
for name in names
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _package_cli_public_error_doc_snippets() -> tuple[str, ...]:
|
|
||||||
return tuple(f"`{code}`" for code in PACKAGE_CLI_PUBLIC_ERROR_CODES)
|
|
||||||
|
|
||||||
|
|
||||||
def _package_cli_error_example_payloads(path: Path) -> tuple[dict, ...]:
|
|
||||||
text = path.read_text(encoding="utf-8")
|
|
||||||
marker = "Package CLI public error examples:"
|
|
||||||
if marker not in text:
|
|
||||||
raise AssertionError(f"{path} is missing package CLI public error examples")
|
|
||||||
|
|
||||||
section = text.split(marker, 1)[1]
|
|
||||||
next_heading = re.search(r"\n## ", section)
|
|
||||||
if next_heading is not None:
|
|
||||||
section = section[: next_heading.start()]
|
|
||||||
|
|
||||||
blocks = re.findall(r"```json\n(?P<body>\{[^\n]*\})\n```", section)
|
|
||||||
if not blocks:
|
|
||||||
raise AssertionError(f"{path} is missing one-line package CLI public error JSON")
|
|
||||||
return tuple(json.loads(block) for block in blocks)
|
|
||||||
|
|
||||||
|
|
||||||
def _run_package_cli_summary(args: list[str]) -> tuple[int, str, dict, str]:
|
|
||||||
stdout = io.StringIO()
|
|
||||||
stderr = io.StringIO()
|
|
||||||
|
|
||||||
with contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr):
|
|
||||||
exit_code = training_cli.main(args)
|
|
||||||
|
|
||||||
raw_output = stdout.getvalue()
|
|
||||||
return exit_code, raw_output, json.loads(raw_output.strip()), stderr.getvalue()
|
|
||||||
|
|
||||||
|
|
||||||
def _test_method_names_from_file(path: Path, class_name: str) -> tuple[str, ...]:
|
|
||||||
tree = ast.parse(path.read_text(encoding="utf-8"))
|
|
||||||
for node in tree.body:
|
|
||||||
if isinstance(node, ast.ClassDef) and node.name == class_name:
|
|
||||||
return tuple(
|
|
||||||
item.name
|
|
||||||
for item in node.body
|
|
||||||
if isinstance(item, ast.FunctionDef) and item.name.startswith("test_")
|
|
||||||
)
|
|
||||||
raise AssertionError(f"{path} is missing {class_name}")
|
|
||||||
|
|
||||||
|
|
||||||
def _walk_public_text(value) -> list[str]:
|
|
||||||
if isinstance(value, dict):
|
|
||||||
entries = []
|
|
||||||
for key, item in value.items():
|
|
||||||
entries.append(str(key))
|
|
||||||
entries.extend(_walk_public_text(item))
|
|
||||||
return entries
|
|
||||||
if isinstance(value, list):
|
|
||||||
entries = []
|
|
||||||
for item in value:
|
|
||||||
entries.extend(_walk_public_text(item))
|
|
||||||
return entries
|
|
||||||
if isinstance(value, str):
|
|
||||||
return [value]
|
|
||||||
return []
|
|
||||||
|
|
||||||
|
|
||||||
class TrainingCaseDocsCliM10Tests(unittest.TestCase):
|
class TrainingCaseDocsCliM10Tests(unittest.TestCase):
|
||||||
def assertNoPrivateFixtureMaterial(self, payload: dict) -> None:
|
def test_example_doc_uses_current_public_fixture_and_cli(self) -> None:
|
||||||
serialized = json.dumps(payload, sort_keys=True)
|
parts = _documented_command_parts()
|
||||||
lower_serialized = serialized.lower()
|
self.assertEqual(parts[:3], [".venv/bin/python", "scripts/run_training_case_package.py", "--input"])
|
||||||
|
self.assertIn(FIXTURE_REF, parts)
|
||||||
|
self.assertIn("--output", parts)
|
||||||
|
|
||||||
self.assertNotIn(PUBLIC_FIELD, lower_serialized)
|
def test_example_doc_publishes_current_public_smoke_wording(self) -> None:
|
||||||
self.assertIsNone(re.search(r"\b(?:\d{1,3}\.){3}\d{1,3}\b", serialized))
|
text = DOC_EXAMPLE.read_text(encoding="utf-8")
|
||||||
for phrase in PRIVATE_FIXTURE_DRIFT_PHRASES:
|
for snippet in (
|
||||||
self.assertNotIn(phrase, lower_serialized)
|
"import/export smoke",
|
||||||
for text in _walk_public_text(payload):
|
"public package",
|
||||||
lower_text = text.lower()
|
"public export",
|
||||||
for fragment in RESTRICTED_PUBLIC_TEXT:
|
"public output",
|
||||||
with self.subTest(text=text, fragment=fragment):
|
"one-line JSON",
|
||||||
self.assertNotIn(fragment, lower_text)
|
"input_file_not_found",
|
||||||
|
"input_path_is_directory",
|
||||||
def test_public_privacy_guard_rejects_solution_wording_drift(self) -> None:
|
"output_path_is_directory",
|
||||||
polluted_payloads = [
|
"output_path_unwritable",
|
||||||
{"items": [{"prompt": "Do not expose " + ("private " + "answer") + " wording."}]},
|
|
||||||
{"items": [{"prompt": "Do not expose internal solution details wording."}]},
|
|
||||||
]
|
|
||||||
|
|
||||||
for payload in polluted_payloads:
|
|
||||||
with self.subTest(payload=payload):
|
|
||||||
with self.assertRaises(AssertionError):
|
|
||||||
self.assertNoPrivateFixtureMaterial(payload)
|
|
||||||
|
|
||||||
def test_public_fixture_schema_contract_is_minimal_and_demo_safe(self) -> None:
|
|
||||||
payload = _fixture_payload()
|
|
||||||
|
|
||||||
self.assertEqual(set(payload), {"version", "metadata", "items"})
|
|
||||||
self.assertEqual(payload["version"], 1)
|
|
||||||
self.assertEqual(payload["metadata"], {"source": "m6_public_fixture", "purpose": "offline_smoke"})
|
|
||||||
self.assertEqual(len(payload["items"]), 1)
|
|
||||||
|
|
||||||
item = payload["items"][0]
|
|
||||||
self.assertEqual(
|
|
||||||
set(item),
|
|
||||||
{
|
|
||||||
"external_id",
|
|
||||||
"title",
|
|
||||||
"scenario",
|
|
||||||
"skill_tags",
|
|
||||||
"difficulty",
|
|
||||||
"answer",
|
|
||||||
"rubric",
|
|
||||||
"tasks",
|
|
||||||
"metadata",
|
|
||||||
},
|
|
||||||
)
|
|
||||||
self.assertEqual(item["external_id"], "m6-sweep-public-001")
|
|
||||||
self.assertEqual(item["title"], "Liquidity sweep reading practice")
|
|
||||||
self.assertEqual(item["skill_tags"], ["sweep", "context_reading"])
|
|
||||||
self.assertEqual(item["difficulty"], "easy")
|
|
||||||
self.assertEqual(item["answer"], {})
|
|
||||||
self.assertEqual(
|
|
||||||
item["rubric"],
|
|
||||||
[
|
|
||||||
{"category": "context", "max_score": 3},
|
|
||||||
{"category": "risk_story", "max_score": 2},
|
|
||||||
],
|
|
||||||
)
|
|
||||||
self.assertEqual(len(item["tasks"]), 1)
|
|
||||||
self.assertEqual(item["metadata"], {"chapter_ref": "public_training_m6", "demo": True})
|
|
||||||
|
|
||||||
scenario = item["scenario"]
|
|
||||||
self.assertEqual(
|
|
||||||
set(scenario),
|
|
||||||
{
|
|
||||||
"instrument_id",
|
|
||||||
"timeframe",
|
|
||||||
"case_type",
|
|
||||||
"start_ts",
|
|
||||||
"decision_ts",
|
|
||||||
"end_ts",
|
|
||||||
"source_event_type",
|
|
||||||
"source_event_id",
|
|
||||||
"source_event_refs",
|
|
||||||
"chapter_ref",
|
|
||||||
},
|
|
||||||
)
|
|
||||||
self.assertEqual(scenario["instrument_id"], 1)
|
|
||||||
self.assertEqual(scenario["timeframe"], "15m")
|
|
||||||
self.assertEqual(scenario["case_type"], "sweep")
|
|
||||||
self.assertEqual(scenario["start_ts"], "2026-04-20T13:00:00Z")
|
|
||||||
self.assertEqual(scenario["decision_ts"], "2026-04-20T14:00:00Z")
|
|
||||||
self.assertEqual(scenario["end_ts"], "2026-04-20T15:00:00Z")
|
|
||||||
self.assertEqual(scenario["source_event_type"], "sweep_event")
|
|
||||||
self.assertEqual(scenario["source_event_id"], 61)
|
|
||||||
self.assertEqual(scenario["source_event_refs"], [{"kind": "sweep_event", "id": 61}])
|
|
||||||
self.assertEqual(scenario["chapter_ref"], "public_training_m6")
|
|
||||||
|
|
||||||
task = item["tasks"][0]
|
|
||||||
self.assertEqual(
|
|
||||||
set(task),
|
|
||||||
{
|
|
||||||
"task_type",
|
|
||||||
"skill_tag",
|
|
||||||
"prompt",
|
|
||||||
"required_annotation_types",
|
|
||||||
"max_score",
|
|
||||||
"sort_order",
|
|
||||||
"meta",
|
|
||||||
},
|
|
||||||
)
|
|
||||||
self.assertEqual(task["task_type"], "short_answer")
|
|
||||||
self.assertEqual(task["skill_tag"], "sweep")
|
|
||||||
self.assertEqual(task["prompt"], "Describe what needs confirmation before marking this as a usable sweep.")
|
|
||||||
self.assertEqual(task["required_annotation_types"], ["text"])
|
|
||||||
self.assertEqual(task["max_score"], 5)
|
|
||||||
self.assertEqual(task["sort_order"], 0)
|
|
||||||
self.assertEqual(task["meta"], {"public_hint": "Focus on context and confirmation."})
|
|
||||||
self.assertNoPrivateFixtureMaterial(payload)
|
|
||||||
|
|
||||||
def test_fixture_is_compatible_with_public_service_and_cli_export(self) -> None:
|
|
||||||
payload = _fixture_payload()
|
|
||||||
service = TrainingCaseService(repository=training_cli._OfflineTrainingRepository())
|
|
||||||
|
|
||||||
import_result = service.import_cases(payload)
|
|
||||||
export_result = service.export_cases(limit=100)
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as tmpdir:
|
|
||||||
output_path = Path(tmpdir) / "training_case_public_export.json"
|
|
||||||
summary = training_cli.run_fixture_smoke(FIXTURE, output_path=output_path)
|
|
||||||
cli_export = json.loads(output_path.read_text(encoding="utf-8"))
|
|
||||||
|
|
||||||
self.assertEqual(import_result["version"], 1)
|
|
||||||
self.assertEqual(import_result["imported"], 1)
|
|
||||||
self.assertEqual(len(import_result["items"]), 1)
|
|
||||||
self.assertEqual(set(import_result["items"][0]), {"id", "title", "scenario", "skill_tags", "difficulty", "status", "external_id"})
|
|
||||||
self.assertEqual(summary["input"], FIXTURE_REF)
|
|
||||||
self.assertEqual(summary["imported"], 1)
|
|
||||||
self.assertEqual(summary["exported"], 1)
|
|
||||||
self.assertEqual(export_result, cli_export)
|
|
||||||
self.assertEqual(set(export_result), {"version", "metadata", "items"})
|
|
||||||
self.assertEqual(export_result["metadata"], {"item_count": 1})
|
|
||||||
|
|
||||||
exported_item = export_result["items"][0]
|
|
||||||
fixture_item = payload["items"][0]
|
|
||||||
self.assertEqual(
|
|
||||||
set(exported_item),
|
|
||||||
{
|
|
||||||
"id",
|
|
||||||
"title",
|
|
||||||
"scenario",
|
|
||||||
"skill_tags",
|
|
||||||
"difficulty",
|
|
||||||
"status",
|
|
||||||
"rubric",
|
|
||||||
"tasks",
|
|
||||||
"metadata",
|
|
||||||
"quality",
|
|
||||||
},
|
|
||||||
)
|
|
||||||
self.assertEqual(exported_item["title"], fixture_item["title"])
|
|
||||||
self.assertEqual(set(exported_item["skill_tags"]), set(fixture_item["skill_tags"]))
|
|
||||||
self.assertEqual(
|
|
||||||
exported_item["metadata"],
|
|
||||||
{"chapter_ref": "public_training_m6", "demo": True, "external_id": fixture_item["external_id"]},
|
|
||||||
)
|
|
||||||
self.assertEqual(exported_item["rubric"], fixture_item["rubric"])
|
|
||||||
self.assertEqual(exported_item["quality"]["rank"], "high")
|
|
||||||
self.assertEqual(exported_item["scenario"], fixture_item["scenario"])
|
|
||||||
self.assertEqual(set(exported_item["tasks"][0]), set(fixture_item["tasks"][0]) | {"status"})
|
|
||||||
self.assertEqual(exported_item["tasks"][0]["status"], "active")
|
|
||||||
for key, value in fixture_item["tasks"][0].items():
|
|
||||||
with self.subTest(task_field=key):
|
|
||||||
self.assertEqual(exported_item["tasks"][0][key], value)
|
|
||||||
self.assertNotIn("answer", exported_item)
|
|
||||||
self.assertNoPrivateFixtureMaterial({"imported": import_result, "exported": export_result})
|
|
||||||
|
|
||||||
def test_smoke_docs_and_product_readme_share_fixture_and_public_export_contract(self) -> None:
|
|
||||||
for path in (DOC_EXAMPLE, PRODUCT_README):
|
|
||||||
text = path.read_text(encoding="utf-8")
|
|
||||||
lower_text = text.lower()
|
|
||||||
fixture_refs = set(re.findall(r"tests/fixtures/[A-Za-z0-9_./-]+\.json", text))
|
|
||||||
|
|
||||||
with self.subTest(path=path.name):
|
|
||||||
self.assertEqual(fixture_refs, {FIXTURE_REF})
|
|
||||||
self.assertIn(".venv/bin/python scripts/run_training_case_package.py", text)
|
|
||||||
self.assertIn("public export", lower_text)
|
|
||||||
self.assertTrue(any(phrase in lower_text for phrase in PRIVATE_MATERIAL_PHRASES))
|
|
||||||
self.assertNotIn(PUBLIC_FIELD, text)
|
|
||||||
|
|
||||||
def test_example_command_uses_supported_cli_flags_and_fixture(self) -> None:
|
|
||||||
parts = _example_command_parts()
|
|
||||||
self.assertEqual(parts[:2], [".venv/bin/python", "scripts/run_training_case_package.py"])
|
|
||||||
|
|
||||||
parser = training_cli._build_parser()
|
|
||||||
supported_flags = {flag for action in parser._actions for flag in action.option_strings}
|
|
||||||
used_flags = [part for part in parts[2:] if part.startswith("--")]
|
|
||||||
|
|
||||||
self.assertEqual(used_flags, ["--input", "--output"])
|
|
||||||
self.assertLessEqual(set(used_flags), supported_flags)
|
|
||||||
|
|
||||||
args = parser.parse_args(parts[2:])
|
|
||||||
expected_fixture = training_cli.DEFAULT_FIXTURE.relative_to(ROOT)
|
|
||||||
self.assertEqual(Path(args.input), expected_fixture)
|
|
||||||
self.assertEqual(Path(args.input), FIXTURE.relative_to(ROOT))
|
|
||||||
self.assertEqual(Path(args.output), Path("/tmp/training_case_public_export.json"))
|
|
||||||
|
|
||||||
def test_example_documents_cli_summary_keys_and_output_semantics(self) -> None:
|
|
||||||
doc_text = DOC_EXAMPLE.read_text(encoding="utf-8")
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as tmpdir:
|
|
||||||
output_path = Path(tmpdir) / "training_case_public_export.json"
|
|
||||||
stdout = io.StringIO()
|
|
||||||
with contextlib.redirect_stdout(stdout):
|
|
||||||
exit_code = training_cli.main(
|
|
||||||
["--input", str(FIXTURE.relative_to(ROOT)), "--output", str(output_path)]
|
|
||||||
)
|
|
||||||
|
|
||||||
summary = json.loads(stdout.getvalue())
|
|
||||||
exported = json.loads(output_path.read_text(encoding="utf-8"))
|
|
||||||
|
|
||||||
expected_keys = {"exported", "imported", "input", "output", "passed"}
|
|
||||||
self.assertEqual(exit_code, 0)
|
|
||||||
self.assertEqual(set(summary), expected_keys)
|
|
||||||
self.assertTrue(summary["passed"])
|
|
||||||
self.assertEqual(summary["imported"], 1)
|
|
||||||
self.assertEqual(summary["exported"], 1)
|
|
||||||
self.assertTrue(summary["output"].endswith("training_case_public_export.json"))
|
|
||||||
self.assertEqual(summary["input"], FIXTURE_REF)
|
|
||||||
self.assertEqual(exported["metadata"], {"item_count": 1})
|
|
||||||
self.assertNotIn(PUBLIC_FIELD, json.dumps(exported, sort_keys=True))
|
|
||||||
|
|
||||||
for key in sorted(expected_keys):
|
|
||||||
with self.subTest(key=key):
|
|
||||||
self.assertIn(f"`{key}`", doc_text)
|
|
||||||
|
|
||||||
def test_cli_error_summary_uses_public_output_target_code_without_runtime_leaks(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as tmpdir:
|
|
||||||
output_path = Path(tmpdir) / "training_case_public_export_dir"
|
|
||||||
output_path.mkdir()
|
|
||||||
stdout = io.StringIO()
|
|
||||||
stderr = io.StringIO()
|
|
||||||
|
|
||||||
with contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr):
|
|
||||||
exit_code = training_cli.main(
|
|
||||||
["--input", str(FIXTURE.relative_to(ROOT)), "--output", str(output_path)]
|
|
||||||
)
|
|
||||||
|
|
||||||
raw_output = stdout.getvalue()
|
|
||||||
line = raw_output.strip()
|
|
||||||
summary = json.loads(line)
|
|
||||||
|
|
||||||
self.assertEqual(exit_code, 2)
|
|
||||||
self.assertEqual(stderr.getvalue(), "")
|
|
||||||
self.assertEqual(raw_output.count("\n"), 1)
|
|
||||||
self.assertEqual(list(summary), ["passed", "input", "output", "error"])
|
|
||||||
self.assertFalse(summary["passed"])
|
|
||||||
self.assertEqual(summary["input"], FIXTURE_REF)
|
|
||||||
self.assertEqual(summary["output"], output_path.name)
|
|
||||||
self.assertEqual(summary["error"], "output_path_is_directory")
|
|
||||||
self.assertNotIn(str(output_path), line)
|
|
||||||
self.assertNotIn(str(Path(tmpdir)), line)
|
|
||||||
self.assertIsNone(re.search(r'(?<!\\)"(?:input|output)": "/', line))
|
|
||||||
for fragment in ("[Errno", "Is a directory", "Traceback"):
|
|
||||||
with self.subTest(fragment=fragment):
|
|
||||||
self.assertNotIn(fragment.lower(), line.lower())
|
|
||||||
self.assertNotIn(PUBLIC_FIELD, line)
|
|
||||||
for phrase in PRIVATE_FIXTURE_DRIFT_PHRASES:
|
|
||||||
with self.subTest(phrase=phrase):
|
|
||||||
self.assertNotIn(phrase, line.lower())
|
|
||||||
|
|
||||||
def test_readme_and_product_docs_advertise_public_offline_smoke_contract(self) -> None:
|
|
||||||
for path in (README, PRODUCT_README):
|
|
||||||
text = path.read_text(encoding="utf-8")
|
|
||||||
lower_text = text.lower()
|
|
||||||
|
|
||||||
with self.subTest(path=path.name):
|
|
||||||
self.assertIn(".venv/bin/python scripts/run_training_case_package.py", text)
|
|
||||||
self.assertIn(FIXTURE_REF, text)
|
|
||||||
self.assertIn("offline package smoke", lower_text)
|
|
||||||
self.assertIn("public export", lower_text)
|
|
||||||
self.assertTrue(any(phrase in lower_text for phrase in PRIVATE_MATERIAL_PHRASES))
|
|
||||||
self.assertNotIn(PUBLIC_FIELD, text)
|
|
||||||
|
|
||||||
def test_product_readme_uses_current_maturity_and_public_private_package_language(self) -> None:
|
|
||||||
text = PRODUCT_README.read_text(encoding="utf-8")
|
|
||||||
lower_text = text.lower()
|
|
||||||
|
|
||||||
for snippet in PRODUCT_PACKAGE_WORDING:
|
|
||||||
with self.subTest(snippet=snippet):
|
|
||||||
self.assertIn(snippet, text if snippet.startswith("M1-") else lower_text)
|
|
||||||
for phrase in PRIVATE_DOC_DRIFT_PHRASES:
|
|
||||||
with self.subTest(phrase=phrase):
|
|
||||||
self.assertNotIn(phrase, lower_text)
|
|
||||||
self.assertNotIn("M1-M21", text)
|
|
||||||
self.assertNotIn(PUBLIC_FIELD, text)
|
|
||||||
|
|
||||||
def test_m24_m29_package_docs_privacy_test_inventory_is_current(self) -> None:
|
|
||||||
test_names = set(unittest.TestLoader().getTestCaseNames(type(self)))
|
|
||||||
inventory_names = tuple(
|
|
||||||
name
|
|
||||||
for names in TRAINING_M24_M29_PACKAGE_DOCS_TEST_INVENTORY.values()
|
|
||||||
for name in names
|
|
||||||
)
|
|
||||||
unique_inventory_names = set(inventory_names)
|
|
||||||
product_text = PRODUCT_README.read_text(encoding="utf-8")
|
|
||||||
|
|
||||||
self.assertEqual(len(test_names), 18)
|
|
||||||
self.assertEqual(len(inventory_names), 19)
|
|
||||||
self.assertEqual(len(unique_inventory_names), 17)
|
|
||||||
self.assertEqual(sorted(TRAINING_M24_M29_PACKAGE_DOCS_TEST_INVENTORY), [
|
|
||||||
"m1_m28_maturity_wording",
|
|
||||||
"offline_registry_coverage",
|
|
||||||
"package_smoke_privacy",
|
|
||||||
])
|
|
||||||
self.assertEqual(
|
|
||||||
test_names - unique_inventory_names,
|
|
||||||
{"test_m24_m29_package_docs_privacy_test_inventory_is_current"},
|
|
||||||
)
|
|
||||||
for surface, names in TRAINING_M24_M29_PACKAGE_DOCS_TEST_INVENTORY.items():
|
|
||||||
with self.subTest(surface=surface):
|
|
||||||
self.assertTrue(names)
|
|
||||||
for name in names:
|
|
||||||
self.assertIn(name, test_names)
|
|
||||||
|
|
||||||
self.assertIn(CURRENT_TRAINING_DOC_MATURITY_MARKER, product_text)
|
|
||||||
self.assertIn(CURRENT_TRAINING_DOC_MATURITY_SPAN, product_text)
|
|
||||||
|
|
||||||
docs_cli_test_names = set(
|
|
||||||
_test_method_names_from_file(Path(__file__), TRAINING_DOCS_CLI_TEST_CLASS)
|
|
||||||
)
|
|
||||||
documentation_consistency_test_names = set(
|
|
||||||
_test_method_names_from_file(
|
|
||||||
DOCUMENTATION_CONSISTENCY_TEST,
|
|
||||||
DOCUMENTATION_CONSISTENCY_TEST_CLASS,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
m33_available_test_names = docs_cli_test_names | documentation_consistency_test_names
|
|
||||||
m33_inventory_names = tuple(
|
|
||||||
name
|
|
||||||
for names in TRAINING_M33_PACKAGE_FIXTURE_IMPORT_EXPORT_PRIVACY_INVENTORY.values()
|
|
||||||
for name in names
|
|
||||||
)
|
|
||||||
m33_unique_inventory_names = set(m33_inventory_names)
|
|
||||||
|
|
||||||
self.assertEqual(
|
|
||||||
sorted(TRAINING_M33_PACKAGE_FIXTURE_IMPORT_EXPORT_PRIVACY_INVENTORY),
|
|
||||||
sorted(TRAINING_M33_REQUIRED_INVENTORY_GROUPS),
|
|
||||||
)
|
|
||||||
self.assertEqual(len(m33_inventory_names), 19)
|
|
||||||
self.assertEqual(len(m33_unique_inventory_names), 16)
|
|
||||||
self.assertIn(
|
|
||||||
"test_training_readmes_name_dashboard_privacy_and_offline_registry_coverage",
|
|
||||||
documentation_consistency_test_names,
|
|
||||||
)
|
|
||||||
for surface, names in TRAINING_M33_PACKAGE_FIXTURE_IMPORT_EXPORT_PRIVACY_INVENTORY.items():
|
|
||||||
with self.subTest(m33_surface=surface):
|
|
||||||
self.assertTrue(names)
|
|
||||||
for name in names:
|
|
||||||
self.assertIn(name, m33_available_test_names)
|
|
||||||
|
|
||||||
def test_public_package_smoke_docs_lock_import_export_privacy_language(self) -> None:
|
|
||||||
help_text = training_cli._build_parser().format_help()
|
|
||||||
|
|
||||||
self.assertIn("--input", help_text)
|
|
||||||
self.assertIn("--output", help_text)
|
|
||||||
self.assertIn("public export", help_text.lower())
|
|
||||||
|
|
||||||
for path in PUBLIC_PACKAGE_DOCS:
|
|
||||||
text = _package_doc_text(path)
|
|
||||||
lower_text = text.lower()
|
|
||||||
|
|
||||||
with self.subTest(path=path.name):
|
|
||||||
for snippet in PUBLIC_PACKAGE_REQUIRED_TEXT:
|
|
||||||
with self.subTest(snippet=snippet):
|
|
||||||
self.assertIn(snippet, text if snippet.startswith(".") or snippet.startswith("--") else lower_text)
|
|
||||||
for phrase in PRIVATE_DOC_DRIFT_PHRASES:
|
|
||||||
with self.subTest(phrase=phrase):
|
|
||||||
self.assertNotIn(phrase, lower_text)
|
|
||||||
self.assertNotIn(PUBLIC_FIELD, text)
|
|
||||||
|
|
||||||
def test_cli_help_discloses_offline_public_import_export_and_error_contract(self) -> None:
|
|
||||||
parser = training_cli._build_parser()
|
|
||||||
help_text = parser.format_help()
|
|
||||||
lower_help = help_text.lower()
|
|
||||||
stdout = io.StringIO()
|
|
||||||
stderr = io.StringIO()
|
|
||||||
|
|
||||||
with (
|
|
||||||
contextlib.redirect_stdout(stdout),
|
|
||||||
contextlib.redirect_stderr(stderr),
|
|
||||||
self.assertRaises(SystemExit) as raised,
|
|
||||||
):
|
):
|
||||||
training_cli.main(["--help"])
|
self.assertIn(snippet, text)
|
||||||
|
|
||||||
self.assertEqual(raised.exception.code, 0)
|
def test_training_public_doc_references_console_and_api_surface(self) -> None:
|
||||||
self.assertEqual(stderr.getvalue(), "")
|
text = TRAINING_PUBLIC_DOC.read_text(encoding="utf-8")
|
||||||
self.assertEqual(stdout.getvalue(), help_text)
|
for snippet in (
|
||||||
self.assertIn(PACKAGE_SMOKE_SCRIPT, help_text)
|
"The public browser entry is `/training`.",
|
||||||
self.assertIn(FIXTURE_REF, help_text)
|
"The trading workbench stays at `/`.",
|
||||||
self.assertNotIn(PUBLIC_FIELD, help_text)
|
"| GET | `/training/cases` |",
|
||||||
for snippet in PACKAGE_CLI_HELP_DISCOVERY_TEXT:
|
"| GET | `/training/progress` |",
|
||||||
with self.subTest(snippet=snippet):
|
"Training public error catalog:",
|
||||||
self.assertIn(snippet, lower_help)
|
PACKAGE_SMOKE_COMMAND,
|
||||||
|
):
|
||||||
|
self.assertIn(snippet, text)
|
||||||
|
|
||||||
def test_package_smoke_docs_publish_help_and_stable_public_error_wording(self) -> None:
|
def test_readme_points_to_current_offline_checks_and_training_console(self) -> None:
|
||||||
for path in PACKAGE_SMOKE_SURFACE_DOCS:
|
text = README.read_text(encoding="utf-8")
|
||||||
text = path.read_text(encoding="utf-8")
|
for snippet in (
|
||||||
lower_text = text.lower()
|
OFFLINE_CHECK_COMMAND,
|
||||||
|
"Training Console at `/training`",
|
||||||
|
"training_console_js_syntax",
|
||||||
|
"python3 scripts/run_api_server.py",
|
||||||
|
):
|
||||||
|
self.assertIn(snippet, text)
|
||||||
|
self.assertNotIn("training_case_engine_m1_m6_product_readme.md", text)
|
||||||
|
|
||||||
with self.subTest(path=path.name):
|
def test_offline_checks_registry_includes_both_web_entries(self) -> None:
|
||||||
self.assertIn("invalid output targets", lower_text)
|
commands = dict(run_offline_checks.build_check_commands(env={}))
|
||||||
self.assertNotIn(PUBLIC_FIELD, text)
|
self.assertEqual(commands["dashboard_js_syntax"], ["node", "--check", "src/web/app.js"])
|
||||||
for snippet in PACKAGE_DOC_HELP_ERROR_TEXT:
|
self.assertEqual(commands["training_console_js_syntax"], ["node", "--check", "src/web/training.js"])
|
||||||
with self.subTest(snippet=snippet):
|
|
||||||
self.assertIn(snippet, text)
|
|
||||||
|
|
||||||
def test_package_cli_public_error_inventory_locks_docs_help_and_readiness(self) -> None:
|
def test_offline_checks_json_and_node_skip_behavior_cover_new_training_check(self) -> None:
|
||||||
test_names = set(unittest.TestLoader().getTestCaseNames(type(self)))
|
def fake_runner(command, **_kwargs):
|
||||||
inventory_names = _package_cli_public_error_inventory_method_names()
|
return SimpleNamespace(returncode=0, stdout="ok\n", stderr="")
|
||||||
doc_snippets = _package_cli_public_error_doc_snippets()
|
|
||||||
help_text = training_cli._build_parser().format_help()
|
|
||||||
source_text = Path(training_cli.__file__).read_text(encoding="utf-8")
|
|
||||||
|
|
||||||
self.assertEqual(
|
|
||||||
sorted(PACKAGE_CLI_PUBLIC_ERROR_TEST_INVENTORY),
|
|
||||||
["docs", "help", "readiness"],
|
|
||||||
)
|
|
||||||
self.assertEqual(len(inventory_names), 3)
|
|
||||||
self.assertEqual(
|
|
||||||
set(inventory_names),
|
|
||||||
{
|
|
||||||
"test_cli_error_summary_uses_public_output_target_code_without_runtime_leaks",
|
|
||||||
"test_cli_help_discloses_offline_public_import_export_and_error_contract",
|
|
||||||
"test_package_smoke_docs_publish_help_and_stable_public_error_wording",
|
|
||||||
},
|
|
||||||
)
|
|
||||||
for surface, names in PACKAGE_CLI_PUBLIC_ERROR_TEST_INVENTORY.items():
|
|
||||||
with self.subTest(surface=surface):
|
|
||||||
self.assertTrue(names)
|
|
||||||
for name in names:
|
|
||||||
self.assertIn(name, test_names)
|
|
||||||
|
|
||||||
self.assertEqual(
|
|
||||||
PACKAGE_CLI_PUBLIC_ERROR_CODES,
|
|
||||||
(
|
|
||||||
"input_file_not_found",
|
|
||||||
"input_path_is_directory",
|
|
||||||
"output_path_is_directory",
|
|
||||||
"output_path_unwritable",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
self.assertEqual(
|
|
||||||
PACKAGE_CLI_PUBLIC_ERROR_EXAMPLE_CODES,
|
|
||||||
PACKAGE_CLI_PUBLIC_ERROR_CODES[:3],
|
|
||||||
)
|
|
||||||
self.assertEqual(doc_snippets, tuple(f"`{code}`" for code in PACKAGE_CLI_PUBLIC_ERROR_CODES))
|
|
||||||
for code, snippet in zip(PACKAGE_CLI_PUBLIC_ERROR_CODES, doc_snippets):
|
|
||||||
with self.subTest(code=code):
|
|
||||||
self.assertIn(code, source_text)
|
|
||||||
self.assertIn(code, help_text)
|
|
||||||
self.assertIn(code, PACKAGE_CLI_HELP_DISCOVERY_TEXT)
|
|
||||||
self.assertIn(snippet, PACKAGE_DOC_HELP_ERROR_TEXT)
|
|
||||||
|
|
||||||
for path in PACKAGE_SMOKE_SURFACE_DOCS:
|
|
||||||
text = path.read_text(encoding="utf-8")
|
|
||||||
with self.subTest(path=path.name):
|
|
||||||
for snippet in doc_snippets:
|
|
||||||
self.assertIn(snippet, text)
|
|
||||||
|
|
||||||
for path in (README, PRODUCT_README):
|
|
||||||
payloads = _package_cli_error_example_payloads(path)
|
|
||||||
self.assertEqual(
|
|
||||||
tuple(payload["error"] for payload in payloads),
|
|
||||||
PACKAGE_CLI_PUBLIC_ERROR_EXAMPLE_CODES,
|
|
||||||
)
|
|
||||||
for payload in payloads:
|
|
||||||
with self.subTest(path=path.name, code=payload["error"]):
|
|
||||||
self.assertEqual(list(payload), ["passed", "input", "output", "error"])
|
|
||||||
self.assertFalse(payload["passed"])
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory(dir=ROOT) as tmpdir:
|
|
||||||
temp_root = Path(tmpdir)
|
|
||||||
missing_input = temp_root / "missing_training_case_package.json"
|
|
||||||
input_dir = temp_root / "training_case_package_dir"
|
|
||||||
input_dir.mkdir()
|
|
||||||
output_dir = temp_root / "training_case_public_export_dir"
|
|
||||||
output_dir.mkdir()
|
|
||||||
runtime_cases = (
|
|
||||||
(
|
|
||||||
"input_file_not_found",
|
|
||||||
[
|
|
||||||
"--input",
|
|
||||||
str(missing_input.relative_to(ROOT)),
|
|
||||||
"--output",
|
|
||||||
"training_case_public_export.json",
|
|
||||||
],
|
|
||||||
),
|
|
||||||
(
|
|
||||||
"input_path_is_directory",
|
|
||||||
[
|
|
||||||
"--input",
|
|
||||||
str(input_dir.relative_to(ROOT)),
|
|
||||||
"--output",
|
|
||||||
"training_case_public_export.json",
|
|
||||||
],
|
|
||||||
),
|
|
||||||
(
|
|
||||||
"output_path_is_directory",
|
|
||||||
[
|
|
||||||
"--input",
|
|
||||||
str(FIXTURE.relative_to(ROOT)),
|
|
||||||
"--output",
|
|
||||||
str(output_dir.relative_to(ROOT)),
|
|
||||||
],
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
for code, args in runtime_cases:
|
|
||||||
exit_code, raw_output, summary, stderr_text = _run_package_cli_summary(args)
|
|
||||||
with self.subTest(runtime_code=code):
|
|
||||||
self.assertEqual(exit_code, 2)
|
|
||||||
self.assertEqual(stderr_text, "")
|
|
||||||
self.assertEqual(raw_output.count("\n"), 1)
|
|
||||||
self.assertEqual(list(summary), ["passed", "input", "output", "error"])
|
|
||||||
self.assertFalse(summary["passed"])
|
|
||||||
self.assertEqual(summary["error"], code)
|
|
||||||
self.assertIsNone(re.search(r'(?<!\\)"(?:input|output)": "/', raw_output))
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as tmpdir:
|
|
||||||
output_path = Path(tmpdir) / "training_case_public_export.json"
|
|
||||||
with mock.patch.object(Path, "write_text", side_effect=OSError("write blocked")):
|
|
||||||
exit_code, raw_output, summary, stderr_text = _run_package_cli_summary(
|
|
||||||
["--input", str(FIXTURE.relative_to(ROOT)), "--output", str(output_path)]
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual(exit_code, 2)
|
|
||||||
self.assertEqual(stderr_text, "")
|
|
||||||
self.assertEqual(raw_output.count("\n"), 1)
|
|
||||||
self.assertEqual(list(summary), ["passed", "input", "output", "error"])
|
|
||||||
self.assertFalse(summary["passed"])
|
|
||||||
self.assertEqual(summary["error"], "output_path_unwritable")
|
|
||||||
self.assertIsNone(re.search(r'(?<!\\)"(?:input|output)": "/', raw_output))
|
|
||||||
|
|
||||||
def test_m1_m28_package_smoke_docs_cover_static_import_export_surface(self) -> None:
|
|
||||||
product_text = PRODUCT_README.read_text(encoding="utf-8")
|
|
||||||
cli_flags = _package_smoke_flags_from_source()
|
|
||||||
|
|
||||||
self.assertIn(CURRENT_TRAINING_DOC_MATURITY_MARKER, product_text)
|
|
||||||
self.assertEqual(cli_flags, ("--input", "--output"))
|
|
||||||
|
|
||||||
for path in PACKAGE_SMOKE_SURFACE_DOCS:
|
|
||||||
text = _package_doc_text(path)
|
|
||||||
lower_text = text.lower()
|
|
||||||
|
|
||||||
with self.subTest(path=path.name):
|
|
||||||
self.assertIn(PACKAGE_SMOKE_SCRIPT, text)
|
|
||||||
self.assertIn(FIXTURE_REF, text)
|
|
||||||
for flag in cli_flags:
|
|
||||||
with self.subTest(flag=flag):
|
|
||||||
self.assertIn(flag, text)
|
|
||||||
for snippet in PACKAGE_IMPORT_EXPORT_SURFACE_TEXT:
|
|
||||||
with self.subTest(snippet=snippet):
|
|
||||||
self.assertIn(snippet, lower_text)
|
|
||||||
|
|
||||||
def test_offline_check_registry_name_matches_package_smoke_docs(self) -> None:
|
|
||||||
commands = run_offline_checks.build_check_commands(env={})
|
|
||||||
names = tuple(name for name, _command in commands)
|
|
||||||
commands_by_name = dict(commands)
|
|
||||||
matching_commands = [
|
|
||||||
(name, command) for name, command in commands if name == OFFLINE_CHECK_NAME
|
|
||||||
]
|
|
||||||
registry_text = Path(run_offline_checks.__file__).read_text(encoding="utf-8")
|
|
||||||
|
|
||||||
self.assertEqual(names, OFFLINE_CHECK_REGISTRY_NAMES)
|
|
||||||
for check_name in OFFLINE_CHECK_REGISTRY_NAMES:
|
|
||||||
with self.subTest(check_name=check_name):
|
|
||||||
self.assertEqual(names.count(check_name), 1)
|
|
||||||
self.assertEqual(
|
|
||||||
matching_commands,
|
|
||||||
[(OFFLINE_CHECK_NAME, [sys.executable, PACKAGE_SMOKE_SCRIPT])],
|
|
||||||
)
|
|
||||||
self.assertEqual(
|
|
||||||
commands_by_name[OFFLINE_CHECK_NAME],
|
|
||||||
[sys.executable, PACKAGE_SMOKE_SCRIPT],
|
|
||||||
)
|
|
||||||
self.assertEqual(
|
|
||||||
commands_by_name[DASHBOARD_SYNTAX_CHECK_NAME],
|
|
||||||
["node", "--check", DASHBOARD_SYNTAX_CHECK_PATH],
|
|
||||||
)
|
|
||||||
self.assertEqual(registry_text.count(f'"{OFFLINE_CHECK_NAME}"'), 1)
|
|
||||||
self.assertIn(f'"{PACKAGE_SMOKE_SCRIPT}"', registry_text)
|
|
||||||
|
|
||||||
for path in (README, DOC_EXAMPLE, PRODUCT_README):
|
|
||||||
text = path.read_text(encoding="utf-8")
|
|
||||||
lower_text = text.lower()
|
|
||||||
|
|
||||||
with self.subTest(path=path.name):
|
|
||||||
self.assertIn(OFFLINE_CHECK_DOC_NAME, text)
|
|
||||||
self.assertIn(PACKAGE_SMOKE_COMMAND, text)
|
|
||||||
self.assertIn(OFFLINE_CHECK_COMMAND, text)
|
|
||||||
self.assertIn("package smoke", lower_text)
|
|
||||||
|
|
||||||
def test_offline_checks_registry_runs_training_package_smoke_entry(self) -> None:
|
|
||||||
commands = run_offline_checks.build_check_commands(env={})
|
|
||||||
names = [name for name, _command in commands]
|
|
||||||
command = dict(commands)[OFFLINE_CHECK_NAME]
|
|
||||||
calls = []
|
|
||||||
|
|
||||||
def fake_runner(command_parts, **kwargs):
|
|
||||||
calls.append((command_parts, kwargs["cwd"]))
|
|
||||||
return SimpleNamespace(returncode=0, stdout="", stderr="")
|
|
||||||
|
|
||||||
output = io.StringIO()
|
output = io.StringIO()
|
||||||
with contextlib.redirect_stdout(output):
|
with contextlib.redirect_stdout(output):
|
||||||
exit_code = run_offline_checks.run_checks(
|
exit_code = run_offline_checks.run_checks(
|
||||||
commands=[(OFFLINE_CHECK_NAME, command)],
|
commands=[("training_console_js_syntax", ["node", "--check", "src/web/training.js"])],
|
||||||
runner=fake_runner,
|
runner=fake_runner,
|
||||||
node_available=True,
|
node_available=False,
|
||||||
env={},
|
env={},
|
||||||
)
|
)
|
||||||
|
|
||||||
self.assertEqual(names.count(OFFLINE_CHECK_NAME), 1)
|
|
||||||
self.assertEqual(command, [sys.executable, PACKAGE_SMOKE_SCRIPT])
|
|
||||||
self.assertEqual((ROOT / command[1]).resolve(), Path(training_cli.__file__).resolve())
|
|
||||||
self.assertEqual(exit_code, 0)
|
self.assertEqual(exit_code, 0)
|
||||||
self.assertEqual(calls, [(command, run_offline_checks.ROOT)])
|
self.assertIn("Offline check skipped: training_console_js_syntax", output.getvalue())
|
||||||
self.assertIn(f"Offline check passed: {OFFLINE_CHECK_NAME}", output.getvalue())
|
|
||||||
|
|
||||||
def test_training_docs_name_broader_offline_check_path(self) -> None:
|
def test_package_cli_still_emits_one_line_public_json(self) -> None:
|
||||||
for path in (DOC_EXAMPLE, README, PRODUCT_README):
|
output = io.StringIO()
|
||||||
text = path.read_text(encoding="utf-8")
|
with contextlib.redirect_stdout(output):
|
||||||
lower_text = text.lower()
|
exit_code = training_cli.main(["--input", str(FIXTURE)])
|
||||||
|
|
||||||
with self.subTest(path=path.name):
|
summary = json.loads(output.getvalue())
|
||||||
self.assertIn(OFFLINE_CHECK_COMMAND, text)
|
self.assertEqual(exit_code, 0)
|
||||||
self.assertIn("broader offline", lower_text)
|
self.assertTrue(summary["passed"])
|
||||||
self.assertIn("training case", lower_text)
|
self.assertTrue(str(summary["input"]).endswith("tests\\fixtures\\training_case_package_m6.json"))
|
||||||
|
self.assertIsNone(summary["output"])
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -1,18 +1,21 @@
|
|||||||
import ast
|
import ast
|
||||||
import io
|
import io
|
||||||
import json
|
import json
|
||||||
|
import os
|
||||||
import re
|
import re
|
||||||
import unittest
|
import unittest
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from unittest import mock
|
from unittest import mock
|
||||||
|
|
||||||
|
os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://user:pass@localhost:5432/ai_ict_test")
|
||||||
|
|
||||||
from src.api import server
|
from src.api import server
|
||||||
from src.api import training as training_api
|
from src.api import training as training_api
|
||||||
from src.services.training import TrainingCaseService
|
from src.services.training import TrainingCaseService
|
||||||
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[2]
|
ROOT = Path(__file__).resolve().parents[2]
|
||||||
TRAINING_PRODUCT_DOC = ROOT / "docs" / "training_case_engine_m1_m6_product_readme.md"
|
TRAINING_PRODUCT_DOC = ROOT / "docs" / "training_console_public_surface_v1.md"
|
||||||
INTERNAL_FIELD = "answer" + "_key"
|
INTERNAL_FIELD = "answer" + "_key"
|
||||||
PRIVATE_MARKER = "trainer-only-choice"
|
PRIVATE_MARKER = "trainer-only-choice"
|
||||||
PRIVATE_RUBRIC_MARKER = "trainer-only-rubric"
|
PRIVATE_RUBRIC_MARKER = "trainer-only-rubric"
|
||||||
|
|||||||
Reference in New Issue
Block a user