# ai-exchange Session

## Current Goal

Current roadmap items are implemented. The next practical step is to run the new historical compatibility repair path in dry-run mode against a real database and decide whether to apply the safe repairs, or define the next roadmap increment.

## Current Branch

`main`

## Last Updated

2026-05-04 UTC

## Completed In Last Session

- Added a historical compatibility repair core module in [src/services/maintenance/historical_compatibility.py](/home/yuqei/project/ai-exchange/src/services/maintenance/historical_compatibility.py) covering three safe auto-repair classes: signal invalidation normalization, MSS meta backfill, and FVG meta compatibility backfill.
- Added [scripts/run_historical_compatibility_repair.py](/home/yuqei/project/ai-exchange/scripts/run_historical_compatibility_repair.py) with default dry-run behavior, explicit safe-apply flags, JSON output support, and manual-review reporting for liquidity duplicate groups.
- Kept liquidity duplicate handling scan-only by default; the script reports duplicate groups for review instead of auto-merging or mutating them.
- Added focused unit coverage for repair planning logic, script dry-run/apply behavior, database-unavailable JSON behavior, and runtime script registry compatibility.
- Documented the new historical compatibility repair script in `README.md` and `RUNBOOK.md`.
- Revalidated the repository with `.venv/bin/python3 -m unittest discover -s tests/unit -p "test_*.py"`, `.venv/bin/python3 scripts/run_offline_checks.py`, and `.venv/bin/python3 scripts/check_local_demo_readiness.py`.

## Current Working State

- `git status --short --branch` now shows `## main...origin/main` with local changes in `RUNBOOK.md`, `SESSION.md`, `TASKS.md`, the signal/report/frontend/backtest/execution/training files touched in recent sessions, plus unrelated untracked content under `docs/AI_Legal_Content_Intelligence_Upgrade_Plan_v1.md` and `legal_content/`.
- `origin` now uses SSH: `gitea-water:yuqei/ai-exchange.git`; SSH auth for this host alias works from the current shell user.
- MSS candidates now carry `meta.state` and `meta.status_reason`; accepted structures use `retest_held`, rejected structures use `retest_not_observed`, and invalidated structures use `old_control_reclaimed`.
- MSS persistence now records the latest structure state even when `accepted = false`, so downstream consumers no longer read stale accepted-only MSS rows.
- MSS detection still collapses duplicate same-direction displacement context at the same `end_ts` to one deterministic candidate, preferring the highest event id.
- Signal-side MSS/FVG fetches now include the latest downstream state regardless of actionability, with deterministic same-timestamp tie-breaks.
- `validate_signal_context` now returns explicit downstream structure outcomes plus richer MSS/FVG state reasons instead of a single coarse non-actionable result.
- Signal metadata now exposes `structure_state`, `mss_state`, `mss_status_reason`, `fvg_status_reason`, `fvg_touch_count`, and FVG `data_quality_warnings`.
- `/signals` API rows now normalize key signal context fields to top-level output keys and add `invalidations_normalized` plus `signal_context`, so API consumers no longer need to inspect nested `meta` for either structure-state reasons or normalized invalidations.
- Intraday reports now expose structure `state` / `status_reason`, FVG `status_reason`, and signal-side richer structure context directly in the report payload.
- Post-trade review reports now expose a `signal_context` block with richer structure-state, normalized invalidations, and data-quality fields.
- Execution audit, execution settlement, post-trade review, and backtest summary now share the same lifecycle vocabulary for `missed`, `invalidated_before_entry`, `tp1_reached`, `tp2_closed`, and `stopped_out`.
- Backtest summary now exposes top-level `lifecycle_states` in addition to the older raw `outcomes` buckets.
- Backtest summary now also exposes `by_symbol`, using `instruments.symbol` when available via repository join and `signal_symbol` in outcome metadata downstream.
- Dashboard backtest cards now show a compact symbol-scope summary derived from `summary.by_symbol`.
- Training case list/detail/export surfaces now expose a public read-only `quality` object with deterministic `score`, `rank`, and `signals`.
- Training dashboard case cards and case detail now display the derived quality signal directly, while progress/attempt flows remain unchanged.
- Training case list/export service/API surfaces now also support stable selector inputs for `quality_rank`, `min_quality_score`, and `sort_by`.
- The generator prototype now exists as an offline-only flow: trading-derived seed JSON -> public Training draft package JSON, with empty `answer` content and no database writes.
- Signal-side FVG selection is now explicit and deterministic: exact MSS displacement match wins, otherwise newest same-direction wins, otherwise newest overall wins.
- Historical compatibility repair now exists as a script-backed path: default dry-run scan, optional safe apply for signal invalidations plus MSS/FVG meta, and manual-review-only reporting for liquidity duplicate groups.
- `TASKS.md` now reflects the intended development sequence more explicitly:
  there are no active roadmap items left after the current sequence.
- FVG state metadata in the dirty worktree still includes `last_touch_ts`, `touch_count`, `status_reason`, and gap warnings.
- Replay-backed no-lookahead backtest mode remains present and validation is green.
- `scripts/run_dev.py` still starts the persistent local API/dashboard dev server using `AI_ICT_API_HOST` / `AI_ICT_API_PORT`.
- Earlier doc cleanup remains in the worktree: `docs/` root contains current requirement/engineering docs, historical material is under `docs/archive/`, and generic workflow helpers are under `docs/workflow/`.
- `.hermes/` and `geckodriver.log` are ignored local artifacts.

## Exact Next Step

If continuing from the current implementation, do this next:
1. run `.venv/bin/python scripts/run_historical_compatibility_repair.py` against the target database in dry-run mode and inspect candidate repair counts plus liquidity duplicate groups;
2. if the dry-run output is acceptable, execute `.venv/bin/python scripts/run_historical_compatibility_repair.py --apply-safe-repairs` in a controlled environment;
3. after the real repair run, decide whether the roadmap should branch into a new migration-backed cleanup phase or shift to a new feature roadmap.

## Validation Commands

```bash
.venv/bin/python3 -m unittest discover -s tests/unit -p "test_*.py"
.venv/bin/python3 scripts/run_offline_checks.py
.venv/bin/python3 scripts/check_local_demo_readiness.py
```

## Risks Or Open Questions

- The dashboard now surfaces the new structure-state fields directly, but signal detail is still rendered as structured JSON rather than a dedicated custom detail card.
- Legacy `trade_signal.invalidations` strings are now normalized for API/report/dashboard consumers, but historical rows still physically store mixed old/new string forms in the database until a later migration or cleanup pass.
- The new FVG selector rule is explicit, but it still reflects one specific interpretation of “recent plus context”; if trading review later wants recency to dominate exact context in some setups, that will require another deliberate rule change rather than a repository ordering tweak.
- The generator prototype is intentionally offline and read-only; it is not yet wired into repository-backed Training flows or any public generation route.
- The historical repair path is script-based rather than migration-based. That keeps it safer and reversible, but it also means real data repair still depends on an explicit operator run rather than automatic deploy-time migration.
- The current worktree is no longer fully clean because this session added both code and documentation changes, and there is unrelated untracked legal/planning content in the repo; decide later whether to commit or separate that unrelated content.
- Training remains a maintenance-first module for now; upstream trading semantics should stabilize before adding larger Training features.