ai-exchange Session

Current Goal

Current roadmap items are implemented. The next practical step is to run the new historical compatibility repair path in dry-run mode against a real database and decide whether to apply the safe repairs, or define the next roadmap increment.

Current Branch

main

Last Updated

2026-05-04 UTC

Completed In Last Session

Added a historical compatibility repair core module in src/services/maintenance/historical_compatibility.py covering three safe auto-repair classes: signal invalidation normalization, MSS meta backfill, and FVG meta compatibility backfill.
Added scripts/run_historical_compatibility_repair.py with default dry-run behavior, explicit safe-apply flags, JSON output support, and manual-review reporting for liquidity duplicate groups.
Kept liquidity duplicate handling scan-only by default; the script reports duplicate groups for review instead of auto-merging or mutating them.
Added focused unit coverage for repair planning logic, script dry-run/apply behavior, database-unavailable JSON behavior, and runtime script registry compatibility.
Documented the new historical compatibility repair script in README.md and RUNBOOK.md.
Revalidated the repository with .venv/bin/python3 -m unittest discover -s tests/unit -p "test_*.py", .venv/bin/python3 scripts/run_offline_checks.py, and .venv/bin/python3 scripts/check_local_demo_readiness.py.

Current Working State

git status --short --branch now shows ## main...origin/main with local changes in RUNBOOK.md, SESSION.md, TASKS.md, the signal/report/frontend/backtest/execution/training files touched in recent sessions, plus unrelated untracked content under docs/AI_Legal_Content_Intelligence_Upgrade_Plan_v1.md and legal_content/.
origin now uses SSH: gitea-water:yuqei/ai-exchange.git; SSH auth for this host alias works from the current shell user.
MSS candidates now carry meta.state and meta.status_reason; accepted structures use retest_held, rejected structures use retest_not_observed, and invalidated structures use old_control_reclaimed.
MSS persistence now records the latest structure state even when accepted = false, so downstream consumers no longer read stale accepted-only MSS rows.
MSS detection still collapses duplicate same-direction displacement context at the same end_ts to one deterministic candidate, preferring the highest event id.
Signal-side MSS/FVG fetches now include the latest downstream state regardless of actionability, with deterministic same-timestamp tie-breaks.
validate_signal_context now returns explicit downstream structure outcomes plus richer MSS/FVG state reasons instead of a single coarse non-actionable result.
Signal metadata now exposes structure_state, mss_state, mss_status_reason, fvg_status_reason, fvg_touch_count, and FVG data_quality_warnings.
/signals API rows now normalize key signal context fields to top-level output keys and add invalidations_normalized plus signal_context, so API consumers no longer need to inspect nested meta for either structure-state reasons or normalized invalidations.
Intraday reports now expose structure state / status_reason, FVG status_reason, and signal-side richer structure context directly in the report payload.
Post-trade review reports now expose a signal_context block with richer structure-state, normalized invalidations, and data-quality fields.
Execution audit, execution settlement, post-trade review, and backtest summary now share the same lifecycle vocabulary for missed, invalidated_before_entry, tp1_reached, tp2_closed, and stopped_out.
Backtest summary now exposes top-level lifecycle_states in addition to the older raw outcomes buckets.
Backtest summary now also exposes by_symbol, using instruments.symbol when available via repository join and signal_symbol in outcome metadata downstream.
Dashboard backtest cards now show a compact symbol-scope summary derived from summary.by_symbol.
Training case list/detail/export surfaces now expose a public read-only quality object with deterministic score, rank, and signals.
Training dashboard case cards and case detail now display the derived quality signal directly, while progress/attempt flows remain unchanged.
Training case list/export service/API surfaces now also support stable selector inputs for quality_rank, min_quality_score, and sort_by.
The generator prototype now exists as an offline-only flow: trading-derived seed JSON -> public Training draft package JSON, with empty answer content and no database writes.
Signal-side FVG selection is now explicit and deterministic: exact MSS displacement match wins, otherwise newest same-direction wins, otherwise newest overall wins.
Historical compatibility repair now exists as a script-backed path: default dry-run scan, optional safe apply for signal invalidations plus MSS/FVG meta, and manual-review-only reporting for liquidity duplicate groups.
TASKS.md now reflects the intended development sequence more explicitly: there are no active roadmap items left after the current sequence.
FVG state metadata in the dirty worktree still includes last_touch_ts, touch_count, status_reason, and gap warnings.
Replay-backed no-lookahead backtest mode remains present and validation is green.
scripts/run_dev.py still starts the persistent local API/dashboard dev server using AI_ICT_API_HOST / AI_ICT_API_PORT.
Earlier doc cleanup remains in the worktree: docs/ root contains current requirement/engineering docs, historical material is under docs/archive/, and generic workflow helpers are under docs/workflow/.
.hermes/ and geckodriver.log are ignored local artifacts.

Exact Next Step

If continuing from the current implementation, do this next:

run .venv/bin/python scripts/run_historical_compatibility_repair.py against the target database in dry-run mode and inspect candidate repair counts plus liquidity duplicate groups;
if the dry-run output is acceptable, execute .venv/bin/python scripts/run_historical_compatibility_repair.py --apply-safe-repairs in a controlled environment;
after the real repair run, decide whether the roadmap should branch into a new migration-backed cleanup phase or shift to a new feature roadmap.

Validation Commands

.venv/bin/python3 -m unittest discover -s tests/unit -p "test_*.py"
.venv/bin/python3 scripts/run_offline_checks.py
.venv/bin/python3 scripts/check_local_demo_readiness.py

Risks Or Open Questions

The dashboard now surfaces the new structure-state fields directly, but signal detail is still rendered as structured JSON rather than a dedicated custom detail card.
Legacy trade_signal.invalidations strings are now normalized for API/report/dashboard consumers, but historical rows still physically store mixed old/new string forms in the database until a later migration or cleanup pass.
The new FVG selector rule is explicit, but it still reflects one specific interpretation of “recent plus context”; if trading review later wants recency to dominate exact context in some setups, that will require another deliberate rule change rather than a repository ordering tweak.
The generator prototype is intentionally offline and read-only; it is not yet wired into repository-backed Training flows or any public generation route.
The historical repair path is script-based rather than migration-based. That keeps it safer and reversible, but it also means real data repair still depends on an explicit operator run rather than automatic deploy-time migration.
The current worktree is no longer fully clean because this session added both code and documentation changes, and there is unrelated untracked legal/planning content in the repo; decide later whether to commit or separate that unrelated content.
Training remains a maintenance-first module for now; upstream trading semantics should stabilize before adding larger Training features.

8.1 KiB Raw Blame History