Add autonomous workflow and offline quality gates
This commit is contained in:
yuqei
@@ -6,6 +6,9 @@ AI ICT 交易辅助系统第一阶段工程实现。
|
||||
|
||||
See `docs/` for architecture, database, rule, and migration references.
|
||||
|
||||
Database connectivity defaults to a 5 second PostgreSQL connect timeout. Override with `DATABASE_CONNECT_TIMEOUT` when needed.
|
||||
Set `MIGRATIONS_OUTPUT_JSON=1` when running `run_migrations.py` for a machine-readable migration summary.
|
||||
|
||||
## Current closed loop
|
||||
|
||||
当前主线可运行链路：
|
||||
@@ -28,12 +31,145 @@ See `docs/` for architecture, database, rule, and migration references.
|
||||
- `python3 scripts/run_mss.py` - rebuild MSS structure events
|
||||
- `python3 scripts/run_fvg.py` - rebuild FVG zones
|
||||
- `python3 scripts/run_signals.py` - generate bias and signals
|
||||
- `python3 scripts/run_execution.py` - run paper execution gating and create an execution record
|
||||
- `python3 scripts/run_execution.py` - run paper execution gating, create an execution record, then auto-settle the latest execution for that signal
|
||||
- `python3 scripts/run_backtest.py` - run backtest and write results
|
||||
- `python3 scripts/run_autonomous_cycle.py` - run the unattended cycle: offline checks, database probe, readiness checks, then guarded pipeline when the database is available; prints `Next actions` for the next unattended step
|
||||
- `python3 scripts/run_database_readiness.py` - run read-only database readiness checks for connection, schema, seed data, and candle availability; prints `next_actions` and `action_plan` in JSON mode
|
||||
- `python3 scripts/run_offline_checks.py` - run database-free automated checks
|
||||
- `python3 scripts/run_migration_static_checks.py` - check migration order and required core schema without connecting to the database
|
||||
- `python3 scripts/run_script_static_checks.py` - check automation scripts for unsafe ambient environment fallback patterns
|
||||
- `python3 scripts/run_replay_file.py <candles.json|candles.jsonl|candles.csv>` - run offline no-lookahead candle replay without database access
|
||||
- `python3 scripts/run_reports.py` - build sample reports from current services
|
||||
- `python3 scripts/run_api_server.py` - start a lightweight HTTP API server on `127.0.0.1:8000`
|
||||
- `python3 scripts/run_api_server.py` - start a lightweight HTTP API server on `127.0.0.1:8000` and open `/` for the minimal dashboard
|
||||
- `python3 scripts/run_pipeline.py` - run the current end-to-end closed loop and print a structured summary
|
||||
|
||||
Optional report environment variables used by `run_reports.py`:
|
||||
|
||||
- `REPORTS_OUTPUT_JSON` - set to `1`, `true`, `yes`, or `on` for machine-readable sample report output
|
||||
|
||||
Optional utility JSON output variables:
|
||||
|
||||
- `CHECK_DB_OUTPUT_JSON`
|
||||
- `CHECK_DB_CONNECTION_OUTPUT_JSON`
|
||||
- `MIGRATION_CHECKS_OUTPUT_JSON`
|
||||
- `SCRIPT_CHECKS_OUTPUT_JSON`
|
||||
- `SHOW_MODELS_OUTPUT_JSON`
|
||||
|
||||
Optional structure script JSON output variables:
|
||||
|
||||
- `SESSION_GENERATOR_OUTPUT_JSON`
|
||||
- `SWINGS_OUTPUT_JSON`
|
||||
- `RANGES_OUTPUT_JSON`
|
||||
- `LIQUIDITY_OUTPUT_JSON`
|
||||
- `SWEEPS_OUTPUT_JSON`
|
||||
- `DISPLACEMENT_OUTPUT_JSON`
|
||||
- `MSS_OUTPUT_JSON`
|
||||
- `FVG_OUTPUT_JSON`
|
||||
|
||||
Optional backtest cost environment variables used by `run_backtest.py` and `run_pipeline.py`:
|
||||
|
||||
- `BACKTEST_FEE_RATE`
|
||||
- `BACKTEST_SLIPPAGE_RATE`
|
||||
- `BACKTEST_FIXED_COST_R`
|
||||
|
||||
Optional backtest automation gate used by `run_backtest.py` and `run_pipeline.py`:
|
||||
|
||||
- `BACKTEST_OUTPUT_JSON` - set to `1`, `true`, `yes`, or `on` for machine-readable `run_backtest.py` output
|
||||
- `BACKTEST_REQUIRE_QUALITY_GATE` - set to `1`, `true`, `yes`, or `on` to exit with code `2` when the backtest quality gate fails
|
||||
- `BACKTEST_MIN_TOTAL_SIGNALS`
|
||||
- `BACKTEST_MIN_EXECUTED_SIGNALS`
|
||||
- `BACKTEST_MIN_EXPECTANCY_R`
|
||||
- `BACKTEST_MIN_PROFIT_FACTOR_R`
|
||||
- `BACKTEST_MAX_DRAWDOWN_R`
|
||||
- `BACKTEST_MAX_CONSECUTIVE_LOSSES`
|
||||
|
||||
Optional pipeline environment variables used by `run_pipeline.py`:
|
||||
|
||||
- `PIPELINE_OUTPUT_JSON` - set to `1`, `true`, `yes`, or `on` for machine-readable pipeline error output
|
||||
|
||||
Optional replay environment variables used by `run_replay_file.py`:
|
||||
|
||||
- `OFFLINE_REPLAY_FILE` - optional candle file path used by `run_offline_checks.py` to include replay validation
|
||||
- `OFFLINE_CHECKS_OUTPUT_JSON` - set to `1`, `true`, `yes`, or `on` for machine-readable offline check summary output
|
||||
- `REPLAY_LEFT_BARS`
|
||||
- `REPLAY_RIGHT_BARS`
|
||||
- `REPLAY_MIN_CANDLES`
|
||||
- `REPLAY_MIN_SWINGS`
|
||||
- `REPLAY_MAX_GAP_MINUTES`
|
||||
- `REPLAY_OUTPUT_JSON` - set to `1`, `true`, `yes`, or `on` for machine-readable replay summary output
|
||||
|
||||
Example offline replay gate:
|
||||
|
||||
```bash
|
||||
OFFLINE_REPLAY_FILE=tests/fixtures/offline_replay_sample.jsonl \
|
||||
REPLAY_LEFT_BARS=1 REPLAY_RIGHT_BARS=1 \
|
||||
REPLAY_MIN_CANDLES=5 REPLAY_MIN_SWINGS=2 \
|
||||
python3 scripts/run_offline_checks.py
|
||||
```
|
||||
|
||||
Optional OKX backfill environment variables used by `run_okx_backfill.py`:
|
||||
|
||||
- `OKX_BACKFILL_SYMBOL` - defaults to `BTC-USDT`
|
||||
- `OKX_BACKFILL_INST_ID` - defaults to `<symbol>-SWAP`
|
||||
- `OKX_BACKFILL_PAGES` - defaults to `20`
|
||||
- `OKX_BACKFILL_LIMIT` - defaults to `100`
|
||||
- `OKX_BACKFILL_TIMEFRAMES` - defaults to `1m,5m,15m`
|
||||
- `OKX_BACKFILL_OUTPUT_JSON` - emit a machine-readable backfill summary
|
||||
|
||||
Optional autonomous cycle environment variables used by `run_autonomous_cycle.py`:
|
||||
|
||||
- `AUTONOMOUS_OUTPUT_JSON` - emit a machine-readable cycle summary
|
||||
- `AUTONOMOUS_REQUIRE_DB` - return code `2` when the database is unavailable
|
||||
- `AUTONOMOUS_RUN_PIPELINE` - set to `0` to stop after a successful database probe
|
||||
- `AUTONOMOUS_MAX_DB_PROBES` - number of database probe attempts before stopping
|
||||
- `AUTONOMOUS_PROBE_INTERVAL_SECONDS` - sleep interval between database probe attempts
|
||||
- `AUTONOMOUS_REQUIRE_SCHEMA` - enabled by default; set to `0` to skip core table readiness checks before the guarded pipeline
|
||||
- `AUTONOMOUS_APPLY_MIGRATIONS` - enabled by default; when schema is not ready, run migrations, recheck schema, then continue
|
||||
- `AUTONOMOUS_REQUIRE_DATA_READINESS` - enabled by default; set to `0` to skip read-only seed/candle readiness before the guarded pipeline
|
||||
- `AUTONOMOUS_BACKFILL_CANDLES` - enabled by default; when readiness only lacks candles, run OKX backfill, recheck readiness, then continue
|
||||
- The guarded pipeline step defaults `BACKTEST_REQUIRE_QUALITY_GATE=1` and `PIPELINE_OUTPUT_JSON=1` unless explicitly overridden
|
||||
|
||||
Current autonomous modes emit machine-readable `next_actions` and `action_plan`, including:
|
||||
|
||||
- `waiting_for_database` -> `wait_for_database_recovery`
|
||||
- `database_schema_not_ready` -> `run_migrations_before_pipeline`
|
||||
- `migration_failed` -> `inspect_migration_output_and_schema`
|
||||
- `database_readiness_failed` -> the readiness script's own next action, such as `run_okx_backfill_or_load_replay_sample`
|
||||
- `backfill_failed` -> `inspect_backfill_output_and_data_readiness`
|
||||
- `pipeline_failed` -> `inspect_pipeline_output_and_quality_gates`
|
||||
|
||||
Each `action_plan` item includes the action name, suggested command, description, and `requires_database`.
|
||||
|
||||
Optional database readiness environment variables used by `run_database_readiness.py`:
|
||||
|
||||
- `READINESS_OUTPUT_JSON`
|
||||
- `READINESS_REQUIRED_VENUES`
|
||||
- `READINESS_REQUIRED_INSTRUMENTS`
|
||||
- `READINESS_CANDLE_INSTRUMENT_ID`
|
||||
- `READINESS_CANDLE_TIMEFRAME`
|
||||
- `READINESS_MIN_CANDLES` - defaults to `50`
|
||||
- `READINESS_MIN_CANDLE_SPAN_MINUTES` - defaults to `60`
|
||||
- `READINESS_MAX_CANDLE_AGE_MINUTES` - defaults to `240`
|
||||
|
||||
Optional account risk environment variables used by `run_execution.py` and `run_pipeline.py`:
|
||||
|
||||
- `EXECUTION_OUTPUT_JSON` - set to `1`, `true`, `yes`, or `on` for machine-readable `run_execution.py` output
|
||||
- `EXECUTION_SETTLEMENT_OUTPUT_JSON` - set to `1`, `true`, `yes`, or `on` for machine-readable `run_execution_settlement.py` output
|
||||
- `EXECUTION_KILL_SWITCH`
|
||||
- `EXECUTION_DAILY_PNL_PCT`
|
||||
- `EXECUTION_CONSECUTIVE_LOSSES`
|
||||
- `EXECUTION_OPEN_POSITIONS`
|
||||
- `EXECUTION_SAME_SIDE_OPEN_POSITIONS`
|
||||
- `EXECUTION_MAX_DAILY_LOSS_PCT`
|
||||
- `EXECUTION_MAX_CONSECUTIVE_LOSSES`
|
||||
- `EXECUTION_MAX_OPEN_POSITIONS`
|
||||
- `EXECUTION_MAX_SAME_SIDE_POSITIONS`
|
||||
|
||||
Optional signal quality environment variables used by `run_signals.py` and `run_pipeline.py`:
|
||||
|
||||
- `SIGNALS_OUTPUT_JSON` - set to `1`, `true`, `yes`, or `on` for machine-readable `run_signals.py` output
|
||||
- `SIGNAL_EVIDENCE_MODE` - `minimal` by default, or `strict` to require sweep/displacement/liquidity evidence before building a signal
|
||||
|
||||
## Lightweight API helpers
|
||||
|
||||
The `src/api/` package exposes internal query helpers and HTTP routes for:
|
||||
@@ -42,8 +178,11 @@ The `src/api/` package exposes internal query helpers and HTTP routes for:
|
||||
- `GET /candles?instrument_id=1&timeframe=1m&limit=100`
|
||||
- `GET /sessions?instrument_id=1&limit=20`
|
||||
- `GET /signals?instrument_id=1&limit=20`
|
||||
- `GET /executions?instrument_id=1&limit=20`
|
||||
- `GET /backtests?instrument_id=1&limit=10`
|
||||
|
||||
Open `/` in the browser for the minimal dashboard, which now includes health, candles, signals, sessions, executions, and backtests.
|
||||
|
||||
## Validation
|
||||
|
||||
Unit tests:
|
||||
|
||||
Reference in New Issue
Block a user