Close P10 visual capture cleanup loop
This commit is contained in:
+20
-10
@@ -2,7 +2,7 @@
|
||||
|
||||
## Current Goal
|
||||
|
||||
Automation execution-layer P9 is closed locally. P9 turns alert deliveries, alert feedback, and visual-review feedback into an auditable quality rollup that is visible in the dashboard and review packet while keeping the system analysis/restricted decision-support only.
|
||||
Automation execution-layer P10 is closed locally. P10 adds visual-capture retention cleanup as an auditable operations hardening slice so stale capture artifacts can be cleaned through service, script, API, runtime automation, and workbench entry points while keeping the system analysis/restricted decision-support only.
|
||||
|
||||
## Current Branch
|
||||
|
||||
@@ -14,12 +14,13 @@ Automation execution-layer P9 is closed locally. P9 turns alert deliveries, aler
|
||||
|
||||
## Automation Status
|
||||
|
||||
- state: automation_execution_layer_p9_alert_quality_rollup_closed
|
||||
- current_phase: P9 Alert Quality Rollup
|
||||
- state: automation_execution_layer_p10_visual_capture_retention_cleanup_closed
|
||||
- current_phase: P10 Visual Capture Retention Cleanup
|
||||
- latest_p7_selector_choice: continue_periodic_analysis_decision_monitoring_with_maturity_rollup
|
||||
- latest_p9_slice: alert_quality_summary_panel
|
||||
- latest_p10_slice: visual_capture_retention_cleanup
|
||||
- latest_visual_review_script_status: idle
|
||||
- latest_runtime_visual_watch_task_status: ok
|
||||
- latest_visual_capture_cleanup_task_status: ok
|
||||
- latest_runtime_visual_watch_state: idle
|
||||
- latest_visual_snapshot_capture_status: implemented
|
||||
- latest_visual_reviews_panel_status: visible
|
||||
@@ -28,6 +29,8 @@ Automation execution-layer P9 is closed locally. P9 turns alert deliveries, aler
|
||||
- latest_alert_quality_summary_status: implemented
|
||||
- latest_review_packet_alert_quality_status: visible
|
||||
- latest_alert_quality_route_status: ok
|
||||
- latest_visual_capture_cleanup_status: implemented
|
||||
- latest_visual_capture_cleanup_route_status: ok
|
||||
- latest_runtime_blockers: none
|
||||
- guards_unchanged: execution_allowed, real_live_ready, risk_governor, trust_bucket, operator_confirmation_protocol
|
||||
|
||||
@@ -48,21 +51,26 @@ Automation execution-layer P9 is closed locally. P9 turns alert deliveries, aler
|
||||
- Added `GET /trading/alerts/quality-summary` and attached the same summary to `review_packet.alert_quality_summary`.
|
||||
- Added a dashboard `Alert Quality` panel with delivery success, alert usefulness, visual helpfulness, flags, and operator summary.
|
||||
- Applied local migrations through `042_create_alert_feedback.sql` so P9 can be exercised on the current database.
|
||||
- Added `VisualCaptureRetentionService` to remove stale `.claude/visual_watch/captures` HTML/JSON artifacts by retention window.
|
||||
- Added `scripts/run_visual_capture_cleanup.py` and `POST /trading/visual-watch/cleanup-captures`.
|
||||
- Added runtime task `visual_capture_cleanup` and a workbench `Cleanup Old Captures` button in the Visual Reviews panel.
|
||||
- Added focused unit tests for the service, script, API routing, and runtime task.
|
||||
|
||||
## Current Working State
|
||||
|
||||
- P7 is closed and remains a monitor-maturity routing layer, not a Research Council expansion.
|
||||
- P8 review/capture/feedback surfaces remain implemented and unchanged in their execution guards.
|
||||
- P9 quality rollup is implemented: recent alert deliveries, alert feedback, visual feedback, and visual-review states are now aggregated into one summary.
|
||||
- P9 dashboard visibility is implemented: the main trading workbench now exposes an `Alert Quality` panel.
|
||||
- P9 review-packet visibility is implemented: `review_packet` now carries `alert_quality_summary` for downstream operator workflows.
|
||||
- P9 quality rollup remains implemented and visible in both the dashboard and review packet.
|
||||
- P10 retention cleanup is implemented: stale visual capture HTML/JSON artifacts can now be deleted by retention policy.
|
||||
- P10 runtime visibility is implemented: `runtime_status.tasks.visual_capture_cleanup` now reports root, retention window, eligible count, deleted count, deleted bytes, and idle reasons.
|
||||
- P10 workbench/manual visibility is implemented: the Visual Reviews panel exposes a `Cleanup Old Captures` action that hits the cleanup route and refreshes the dashboard.
|
||||
- Latest `scripts/run_visual_review.py` smoke found no pending reviews: attempted=0, reviewed=0, status=idle.
|
||||
- Latest capture smoke returned `status=ignored` when no watch run was supplied, which is the expected safe no-op.
|
||||
- Latest cleanup smoke deleted one expired temporary artifact through `scripts/run_visual_capture_cleanup.py`.
|
||||
- Latest `scripts/run_trading_runtime.py` smoke returned blockers=none and includes a healthy `visual_watch` runtime task.
|
||||
- Latest local API smoke confirmed `/trading/alerts/quality-summary`, `/trading/review-packet`, dashboard HTML, and dashboard JS all expose the P9 summary successfully.
|
||||
- Latest local API smoke confirmed `/trading/visual-watch/cleanup-captures`, dashboard HTML, and dashboard JS all expose the P10 cleanup path successfully.
|
||||
|
||||
## P9 Blockers
|
||||
## P10 Blockers
|
||||
|
||||
- none
|
||||
|
||||
@@ -71,11 +79,12 @@ Automation execution-layer P9 is closed locally. P9 turns alert deliveries, aler
|
||||
- Browser screenshot capture adapter is a future enhancement, not a P8 closure blocker. The current adapter is an advisory review-packet HTML capture, and the review loop can still use an existing snapshot, an explicit `image_path`, or write an `unavailable` structured review.
|
||||
- Visual review remains advisory and must not change execution permission, live readiness, trust bucket, risk governor, or operator confirmation.
|
||||
- Alert-quality statistics remain advisory and must not change execution permission, live readiness, trust bucket, risk governor, or operator confirmation.
|
||||
- Capture cleanup remains operational only and must not change execution permission, live readiness, trust bucket, risk governor, or operator confirmation.
|
||||
- Real-live order submission remains intentionally disabled because the OKX order adapter is still not a real-live submitter.
|
||||
|
||||
## Exact Next Step
|
||||
|
||||
P9 is locally closed. Default next action is to continue periodic analysis-decision monitoring with maturity rollup and, when opening the next bounded product slice, move to P10-style automation-operations hardening such as silent windows, retry/noise control, stuck-task detection, or screenshot retention cleanup.
|
||||
P10 is locally closed. Default next action is to continue periodic analysis-decision monitoring with maturity rollup and, when opening the next bounded product slice, move to the next P11-style automation-operations hardening candidate such as silent windows / noise control or stuck-task detection.
|
||||
|
||||
## Validation Commands Used Recently
|
||||
|
||||
@@ -85,6 +94,7 @@ D:\ai-exchange\.venv\Scripts\python.exe scripts\run_script_static_checks.py
|
||||
D:\ai-exchange\.venv\Scripts\python.exe -m unittest tests.unit.test_run_until_blocked tests.unit.test_monitoring_alert_rollout tests.unit.test_runtime_script_json
|
||||
D:\ai-exchange\.venv\Scripts\python.exe scripts\run_visual_review.py
|
||||
D:\ai-exchange\.venv\Scripts\python.exe scripts\run_visual_snapshot_capture.py
|
||||
D:\ai-exchange\.venv\Scripts\python.exe scripts\run_visual_capture_cleanup.py
|
||||
D:\ai-exchange\.venv\Scripts\python.exe scripts\run_trading_runtime.py
|
||||
D:\ai-exchange\.venv\Scripts\python.exe -m unittest tests.unit.test_visual_snapshot_capture_service tests.unit.test_run_visual_snapshot_capture tests.unit.test_api_static_server
|
||||
```
|
||||
|
||||
@@ -18,18 +18,20 @@
|
||||
|
||||
- [x] P8 local closure complete
|
||||
- [x] P9 local closure complete
|
||||
- [x] P10 local closure complete
|
||||
- [ ] Continue periodic analysis-decision monitoring with maturity rollup in parallel
|
||||
- [ ] Open the next bounded P10-style automation-operations hardening slice
|
||||
- [ ] Open the next bounded P11-style automation-operations hardening slice
|
||||
- [ ] Treat browser screenshot capture and additional external media adapters as future enhancements
|
||||
|
||||
## Automation Status
|
||||
|
||||
- [x] state: automation_execution_layer_p9_alert_quality_rollup_closed
|
||||
- [x] current_phase: P9 Alert Quality Rollup
|
||||
- [x] state: automation_execution_layer_p10_visual_capture_retention_cleanup_closed
|
||||
- [x] current_phase: P10 Visual Capture Retention Cleanup
|
||||
- [x] latest_p7_selector_choice: continue_periodic_analysis_decision_monitoring_with_maturity_rollup
|
||||
- [x] latest_p9_slice: alert_quality_summary_panel
|
||||
- [x] latest_p10_slice: visual_capture_retention_cleanup
|
||||
- [x] latest_visual_review_script_status: idle
|
||||
- [x] latest_runtime_visual_watch_task_status: ok
|
||||
- [x] latest_visual_capture_cleanup_task_status: ok
|
||||
- [x] latest_visual_snapshot_capture_status: implemented
|
||||
- [x] latest_visual_reviews_panel_status: visible
|
||||
- [x] latest_visual_feedback_status: implemented
|
||||
@@ -37,6 +39,8 @@
|
||||
- [x] latest_alert_quality_summary_status: implemented
|
||||
- [x] latest_review_packet_alert_quality_status: visible
|
||||
- [x] latest_alert_quality_route_status: ok
|
||||
- [x] latest_visual_capture_cleanup_status: implemented
|
||||
- [x] latest_visual_capture_cleanup_route_status: ok
|
||||
- [x] latest_runtime_blockers: none
|
||||
- [x] no_new_research_council_phase_created: true
|
||||
|
||||
@@ -57,10 +61,16 @@
|
||||
- [x] Added `review_packet.alert_quality_summary`
|
||||
- [x] Added dashboard `Alert Quality` panel
|
||||
- [x] Applied local migrations through `042_create_alert_feedback.sql`
|
||||
- [x] Added `VisualCaptureRetentionService`
|
||||
- [x] Added `scripts/run_visual_capture_cleanup.py`
|
||||
- [x] Added `POST /trading/visual-watch/cleanup-captures`
|
||||
- [x] Added runtime `visual_capture_cleanup` task
|
||||
- [x] Added workbench `Cleanup Old Captures` action
|
||||
- [x] Added runtime `visual_watch` task
|
||||
- [x] Added focused service, script, API, and runtime tests
|
||||
- [x] Smoked the visual review script with no pending reviews
|
||||
- [x] Smoked the capture script with no watch run provided
|
||||
- [x] Smoked the cleanup script with expired temporary artifacts
|
||||
- [x] Smoked the trading runtime with blockers=none
|
||||
|
||||
## Blocked
|
||||
@@ -102,6 +112,7 @@
|
||||
- [x] `automation_execution_layer_p8_visual_watch_snapshot_capture_adapter`
|
||||
- [x] `automation_execution_layer_p8_visual_review_workbench_panel`
|
||||
- [x] `automation_execution_layer_p9_alert_quality_rollup`
|
||||
- [x] `automation_execution_layer_p10_visual_capture_retention_cleanup`
|
||||
|
||||
## Backlog
|
||||
|
||||
|
||||
@@ -392,6 +392,10 @@ candidate -> decision alert -> dispatcher -> 外部消息 -> delivery record
|
||||
- 增加 stuck task 检测
|
||||
- 增加 screenshot 存储清理策略
|
||||
|
||||
### 当前进展
|
||||
- 2026-05-12 P10 已完成 `screenshot` 存储清理闭环:新增 `VisualCaptureRetentionService`、`scripts/run_visual_capture_cleanup.py`、`POST /trading/visual-watch/cleanup-captures`、runtime `visual_capture_cleanup` 任务,以及 workbench 手动触发入口。
|
||||
- 2026-05-12 P10 本地闭环已验证:cleanup 脚本可删除过期临时文件,runtime smoke 无 blocker,本地 API/workbench smoke 可见 cleanup 路径。
|
||||
|
||||
### 验收
|
||||
- 非重大错误不打断主循环
|
||||
- delivery 失败可重试
|
||||
|
||||
@@ -644,6 +644,9 @@
|
||||
- `VISUAL_WATCH_VIEWPORT_HEIGHT`
|
||||
- `VISUAL_WATCH_STORAGE_DIR`
|
||||
- `VISUAL_REVIEW_ON_ACTIONABLE_ONLY`
|
||||
- `VISUAL_CAPTURE_RETENTION_HOURS`
|
||||
- `VISUAL_CAPTURE_CLEANUP_INTERVAL_SECONDS`
|
||||
- `VISUAL_CAPTURE_CLEANUP_MAX_DELETE_FILES`
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from src.api.visual_watch import cleanup_visual_watch_captures
|
||||
|
||||
|
||||
def main(env=None) -> int:
|
||||
env = os.environ if env is None else env
|
||||
output_json = _get_bool(env, "VISUAL_CAPTURE_CLEANUP_OUTPUT_JSON", False)
|
||||
payload = cleanup_visual_watch_captures(
|
||||
retention_hours=_get_int(env, "VISUAL_CAPTURE_CLEANUP_RETENTION_HOURS", 168),
|
||||
dry_run=_get_bool(env, "VISUAL_CAPTURE_CLEANUP_DRY_RUN", False),
|
||||
output_dir=_get_optional_value(env, "VISUAL_CAPTURE_CLEANUP_OUTPUT_DIR"),
|
||||
max_delete_files=_get_int(env, "VISUAL_CAPTURE_CLEANUP_MAX_DELETE_FILES", 500),
|
||||
)
|
||||
if output_json:
|
||||
print(json.dumps(payload, ensure_ascii=False, sort_keys=True, default=str))
|
||||
else:
|
||||
print(f"Visual capture cleanup status: {payload.get('status')}")
|
||||
print(f"Root: {payload.get('root')}")
|
||||
print(f"Retention hours: {payload.get('retention_hours')}")
|
||||
print(f"Eligible files: {payload.get('eligible_file_count')}")
|
||||
print(f"Deleted files: {payload.get('deleted_file_count')}")
|
||||
if payload.get("reason"):
|
||||
print(f"Reason: {payload.get('reason')}")
|
||||
return 0 if payload.get("status") != "failed" else 2
|
||||
|
||||
|
||||
def _get_bool(env, key: str, default: bool = False) -> bool:
|
||||
value = env.get(key)
|
||||
if value is None or value == "":
|
||||
return default
|
||||
return str(value).strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _get_int(env, key: str, default: int) -> int:
|
||||
value = env.get(key)
|
||||
if value is None or value == "":
|
||||
return default
|
||||
return int(value)
|
||||
|
||||
|
||||
def _get_optional_value(env, key: str):
|
||||
value = env.get(key)
|
||||
if value is None or value == "":
|
||||
return None
|
||||
return value
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -65,6 +65,7 @@ from src.api.research_council import (
|
||||
run_research_council,
|
||||
)
|
||||
from src.api.visual_watch import (
|
||||
cleanup_visual_watch_captures,
|
||||
capture_visual_watch_snapshot,
|
||||
get_latest_candidate_snapshot,
|
||||
get_latest_visual_watch_run,
|
||||
@@ -129,6 +130,7 @@ __all__ = [
|
||||
"get_latest_research_council_optimization_cycle",
|
||||
"get_latest_candidate_snapshot",
|
||||
"get_latest_visual_watch_run",
|
||||
"cleanup_visual_watch_captures",
|
||||
"capture_visual_watch_snapshot",
|
||||
"review_pending_visual_watch_runs",
|
||||
"review_visual_watch_run",
|
||||
|
||||
@@ -29,6 +29,7 @@ from src.api.research_council import (
|
||||
)
|
||||
from src.api.trade_plan_candidates import get_recent_trade_plan_candidates
|
||||
from src.api.visual_watch import (
|
||||
cleanup_visual_watch_captures,
|
||||
capture_visual_watch_snapshot,
|
||||
create_visual_review,
|
||||
create_visual_snapshot,
|
||||
@@ -133,6 +134,7 @@ def build_api_dependencies() -> dict:
|
||||
"trading_visual_watch_latest": get_latest_visual_watch_run,
|
||||
"trading_visual_candidate_snapshot": get_latest_candidate_snapshot,
|
||||
"trading_visual_packet_reviews": get_visual_reviews_for_packet,
|
||||
"trading_visual_cleanup_captures": cleanup_visual_watch_captures,
|
||||
"trading_visual_capture_snapshot": capture_visual_watch_snapshot,
|
||||
"trading_visual_review_pending": review_pending_visual_watch_runs,
|
||||
"trading_visual_review_run": review_visual_watch_run,
|
||||
@@ -388,6 +390,13 @@ def route_request(
|
||||
decision_packet_id = _require_int(query, "decision_packet_id")
|
||||
limit = _get_int(query, "limit", 10)
|
||||
return 200, {"items": deps["trading_visual_packet_reviews"](decision_packet_id=decision_packet_id, limit=limit)}
|
||||
if method == "POST" and path == "/trading/visual-watch/cleanup-captures":
|
||||
return 200, deps["trading_visual_cleanup_captures"](
|
||||
retention_hours=_get_body_int(body, "retention_hours", 168),
|
||||
dry_run=_get_body_bool(body, "dry_run", False),
|
||||
output_dir=_get_body_value(body, "output_dir", None),
|
||||
max_delete_files=_get_body_int(body, "max_delete_files", 500),
|
||||
)
|
||||
if method == "GET" and len(path_parts) == 5 and path_parts[:3] == ["trading", "visual-watch", "reviews"] and path_parts[4] == "feedback":
|
||||
review_id = _parse_path_int(path_parts[3], "review_id")
|
||||
return 200, {"items": deps["trading_visual_review_feedback_list"](visual_review_id=review_id, limit=_get_int(query, "limit", 20))}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from src.repositories.visual_snapshot_repository import VisualSnapshotRepository
|
||||
from src.services.feedback import AlertFeedbackService
|
||||
from src.services.visual_watch import (
|
||||
VisualCaptureRetentionService,
|
||||
VISUAL_TRIGGER_CANDIDATE_ACTIONABLE_WATCH,
|
||||
VisualReviewService,
|
||||
VisualSnapshotCaptureService,
|
||||
@@ -181,3 +182,18 @@ def create_visual_review(
|
||||
"payload": payload or {},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def cleanup_visual_watch_captures(
|
||||
*,
|
||||
retention_hours: int = 168,
|
||||
dry_run: bool = False,
|
||||
output_dir: str | None = None,
|
||||
max_delete_files: int = 500,
|
||||
) -> dict:
|
||||
return VisualCaptureRetentionService().cleanup_captures(
|
||||
retention_hours=retention_hours,
|
||||
dry_run=dry_run,
|
||||
output_dir=output_dir,
|
||||
max_delete_files=max_delete_files,
|
||||
)
|
||||
|
||||
@@ -89,6 +89,10 @@ class Settings(BaseSettings):
|
||||
trading_runtime_research_council_interval_seconds: int = 60
|
||||
trading_runtime_decision_packet_interval_seconds: int = 20
|
||||
trading_runtime_plan_interval_seconds: int = 900
|
||||
visual_capture_cleanup_enabled: bool = True
|
||||
visual_capture_cleanup_interval_seconds: int = 3600
|
||||
visual_capture_retention_hours: int = 168
|
||||
visual_capture_cleanup_max_delete_files: int = 500
|
||||
alert_dispatch_enabled: bool = True
|
||||
alert_dispatch_interval_seconds: int = 20
|
||||
alert_dispatch_max_per_run: int = 25
|
||||
|
||||
@@ -15,7 +15,7 @@ from src.services.reconciliation import BrokerExecutionReconciliationService
|
||||
from src.services.research_council import ResearchCouncilService
|
||||
from src.services.trade_plan_candidates import SetupWatcherService
|
||||
from src.services.trading_plans import TradingPlanService
|
||||
from src.services.visual_watch import VisualReviewService
|
||||
from src.services.visual_watch import VisualCaptureRetentionService, VisualReviewService
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -63,6 +63,7 @@ class TradingRuntimeScheduler:
|
||||
alert_dispatcher_service: Optional[AlertDispatcherService] = None,
|
||||
research_council_service: Optional[ResearchCouncilService] = None,
|
||||
visual_review_service: Optional[VisualReviewService] = None,
|
||||
visual_capture_retention_service: Optional[VisualCaptureRetentionService] = None,
|
||||
) -> None:
|
||||
self.market_state_service = market_state_service or MarketStateService(market_truth_supervisor=get_market_truth_supervisor())
|
||||
self.trading_plan_service = trading_plan_service or TradingPlanService()
|
||||
@@ -75,6 +76,7 @@ class TradingRuntimeScheduler:
|
||||
self.alert_dispatcher_service = alert_dispatcher_service or AlertDispatcherService()
|
||||
self.research_council_service = research_council_service or ResearchCouncilService()
|
||||
self.visual_review_service = visual_review_service or VisualReviewService()
|
||||
self.visual_capture_retention_service = visual_capture_retention_service or VisualCaptureRetentionService()
|
||||
self.symbols = [item.strip() for item in settings.trading_runtime_symbols.split(",") if item.strip()]
|
||||
self.bias_tf = settings.trading_runtime_bias_timeframe
|
||||
self.setup_tf = settings.trading_runtime_setup_timeframe
|
||||
@@ -87,6 +89,7 @@ class TradingRuntimeScheduler:
|
||||
"research_council": RuntimeTaskState("research_council", settings.trading_runtime_research_council_interval_seconds),
|
||||
"decision_packet": RuntimeTaskState("decision_packet", settings.trading_runtime_decision_packet_interval_seconds),
|
||||
"visual_watch": RuntimeTaskState("visual_watch", settings.trading_runtime_decision_packet_interval_seconds),
|
||||
"visual_capture_cleanup": RuntimeTaskState("visual_capture_cleanup", settings.visual_capture_cleanup_interval_seconds),
|
||||
"alert_dispatch": RuntimeTaskState("alert_dispatch", settings.alert_dispatch_interval_seconds),
|
||||
"plan": RuntimeTaskState("plan", settings.trading_runtime_plan_interval_seconds),
|
||||
}
|
||||
@@ -161,6 +164,7 @@ class TradingRuntimeScheduler:
|
||||
)
|
||||
self._run_decision_packet_task(symbol=symbol, instrument_id=instrument_id, now=now, blockers=blockers)
|
||||
self._run_visual_watch_task(symbol=symbol, instrument_id=instrument_id, now=now, warnings=warnings)
|
||||
self._run_visual_capture_cleanup_task(now=now, warnings=warnings)
|
||||
self._run_alert_dispatch_task(now=now, blockers=blockers, warnings=warnings)
|
||||
|
||||
self.current_blockers = _dedupe(blockers)
|
||||
@@ -501,6 +505,38 @@ class TradingRuntimeScheduler:
|
||||
blockers.append(code)
|
||||
self._record_incident(task_name=task.name, symbol="runtime", code=code, severity="blocker", error=str(exc), observed_ts=now)
|
||||
|
||||
def _run_visual_capture_cleanup_task(self, *, now: datetime, warnings: list[str]) -> None:
|
||||
task = self.tasks["visual_capture_cleanup"]
|
||||
if not _task_due(task, now):
|
||||
return
|
||||
if not settings.visual_capture_cleanup_enabled:
|
||||
task.details = {"status": "disabled"}
|
||||
_task_success(task, now)
|
||||
return
|
||||
_task_start(task, now)
|
||||
try:
|
||||
payload = self.visual_capture_retention_service.cleanup_captures(
|
||||
retention_hours=settings.visual_capture_retention_hours,
|
||||
max_delete_files=settings.visual_capture_cleanup_max_delete_files,
|
||||
dry_run=False,
|
||||
)
|
||||
task.details = {
|
||||
"status": payload.get("status"),
|
||||
"root": payload.get("root"),
|
||||
"retention_hours": payload.get("retention_hours"),
|
||||
"eligible_file_count": int(payload.get("eligible_file_count") or 0),
|
||||
"deleted_file_count": int(payload.get("deleted_file_count") or 0),
|
||||
"deleted_bytes": int(payload.get("deleted_bytes") or 0),
|
||||
"retained_file_count": int(payload.get("retained_file_count") or 0),
|
||||
"reason": payload.get("reason"),
|
||||
}
|
||||
_task_success(task, now)
|
||||
except Exception as exc:
|
||||
_task_failure(task, now, str(exc))
|
||||
code = "visual_capture_cleanup_task_failed"
|
||||
warnings.append(code)
|
||||
self._record_incident(task_name=task.name, symbol="runtime", code=code, severity="warning", error=str(exc), observed_ts=now)
|
||||
|
||||
def _record_incident(
|
||||
self,
|
||||
*,
|
||||
|
||||
@@ -10,6 +10,7 @@ from src.services.visual_watch.visual_trigger_policy_service import (
|
||||
suggest_visual_triggers_from_council,
|
||||
)
|
||||
from src.services.visual_watch.snapshot_capture_service import VisualSnapshotCaptureService
|
||||
from src.services.visual_watch.capture_retention_service import VisualCaptureRetentionService
|
||||
from src.services.visual_watch.visual_review_service import VisualReviewService
|
||||
from src.services.visual_watch.visual_watch_service import VisualWatchService
|
||||
|
||||
@@ -24,6 +25,7 @@ __all__ = [
|
||||
"suggest_visual_trigger",
|
||||
"suggest_visual_triggers_from_council",
|
||||
"VisualSnapshotCaptureService",
|
||||
"VisualCaptureRetentionService",
|
||||
"VisualReviewService",
|
||||
"VisualWatchService",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
from typing import Iterable, Optional
|
||||
|
||||
from src.services.visual_watch.snapshot_capture_service import DEFAULT_CAPTURE_ROOT, ROOT
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CaptureCleanupCandidate:
|
||||
path: Path
|
||||
size_bytes: int
|
||||
modified_ts: datetime
|
||||
|
||||
|
||||
class VisualCaptureRetentionService:
|
||||
def __init__(self, *, output_root: str | Path | None = None) -> None:
|
||||
self.output_root = _resolve_output_dir(output_root)
|
||||
|
||||
def cleanup_captures(
|
||||
self,
|
||||
*,
|
||||
output_dir: str | Path | None = None,
|
||||
retention_hours: int = 168,
|
||||
dry_run: bool = False,
|
||||
max_delete_files: int = 500,
|
||||
now: datetime | None = None,
|
||||
) -> dict:
|
||||
effective_root = _resolve_output_dir(output_dir) if output_dir is not None else self.output_root
|
||||
effective_now = now.astimezone(timezone.utc) if isinstance(now, datetime) else datetime.now(timezone.utc)
|
||||
retention_hours = max(int(retention_hours), 1)
|
||||
max_delete_files = max(int(max_delete_files), 1)
|
||||
cutoff_ts = effective_now - timedelta(hours=retention_hours)
|
||||
if not effective_root.exists():
|
||||
return {
|
||||
"status": "idle",
|
||||
"reason": "capture_root_missing",
|
||||
"root": str(effective_root),
|
||||
"retention_hours": retention_hours,
|
||||
"cutoff_ts": cutoff_ts.isoformat(),
|
||||
"dry_run": bool(dry_run),
|
||||
"candidate_file_count": 0,
|
||||
"eligible_file_count": 0,
|
||||
"deleted_file_count": 0,
|
||||
"deleted_bytes": 0,
|
||||
"retained_file_count": 0,
|
||||
"deleted_paths": [],
|
||||
}
|
||||
|
||||
candidates = list(_iter_capture_candidates(effective_root))
|
||||
eligible = [item for item in candidates if item.modified_ts <= cutoff_ts]
|
||||
deletions = eligible[:max_delete_files]
|
||||
deleted_paths = [str(item.path) for item in deletions]
|
||||
deleted_bytes = sum(item.size_bytes for item in deletions)
|
||||
if not dry_run:
|
||||
for item in deletions:
|
||||
if item.path.exists():
|
||||
item.path.unlink()
|
||||
return {
|
||||
"status": "completed",
|
||||
"root": str(effective_root),
|
||||
"retention_hours": retention_hours,
|
||||
"cutoff_ts": cutoff_ts.isoformat(),
|
||||
"dry_run": bool(dry_run),
|
||||
"candidate_file_count": len(candidates),
|
||||
"eligible_file_count": len(eligible),
|
||||
"deleted_file_count": len(deletions),
|
||||
"deleted_bytes": deleted_bytes,
|
||||
"retained_file_count": max(len(candidates) - len(deletions), 0),
|
||||
"deleted_paths": deleted_paths[:50],
|
||||
"truncated_deletions": max(len(deletions) - 50, 0),
|
||||
}
|
||||
|
||||
|
||||
def _iter_capture_candidates(root: Path) -> Iterable[CaptureCleanupCandidate]:
|
||||
for path in root.rglob("*"):
|
||||
if not path.is_file():
|
||||
continue
|
||||
if path.suffix.lower() not in {".html", ".json"}:
|
||||
continue
|
||||
stat = path.stat()
|
||||
modified_ts = datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc)
|
||||
yield CaptureCleanupCandidate(
|
||||
path=path,
|
||||
size_bytes=int(stat.st_size),
|
||||
modified_ts=modified_ts,
|
||||
)
|
||||
|
||||
|
||||
def _resolve_output_dir(path: str | Path | None) -> Path:
|
||||
if path is None or str(path).strip() == "":
|
||||
return DEFAULT_CAPTURE_ROOT
|
||||
output_dir = Path(path)
|
||||
if not output_dir.is_absolute():
|
||||
output_dir = ROOT / output_dir
|
||||
return output_dir
|
||||
@@ -642,6 +642,18 @@ function renderVisualReviews(reviewRows, decisionPacketId) {
|
||||
const items = (reviewRows?.items || []).slice(0, 10);
|
||||
const latest = items[0] || null;
|
||||
const latestFeedback = getVisualFeedbackSummary(latest);
|
||||
visualReviewsPanel.appendChild(
|
||||
buildActionBar([
|
||||
{
|
||||
label: 'Cleanup Old Captures',
|
||||
onClick: () => runAction('/trading/visual-watch/cleanup-captures', {
|
||||
retention_hours: 168,
|
||||
dry_run: false,
|
||||
max_delete_files: 500,
|
||||
}),
|
||||
},
|
||||
])
|
||||
);
|
||||
visualReviewsPanel.appendChild(
|
||||
buildInfoGrid([
|
||||
['decision packet', decisionPacketId || 'n/a'],
|
||||
|
||||
@@ -203,6 +203,28 @@ class ApiTests(unittest.TestCase):
|
||||
self.assertEqual(payload["reviewed"], 1)
|
||||
self.assertEqual(payload["instrument_id"], 1)
|
||||
|
||||
def test_route_request_visual_capture_cleanup(self) -> None:
|
||||
status, payload = route_request(
|
||||
"/trading/visual-watch/cleanup-captures",
|
||||
{},
|
||||
method="POST",
|
||||
body={"retention_hours": 72, "dry_run": True, "max_delete_files": 25},
|
||||
dependencies={
|
||||
"trading_visual_cleanup_captures": lambda **kwargs: {
|
||||
"status": "completed",
|
||||
"retention_hours": kwargs["retention_hours"],
|
||||
"dry_run": kwargs["dry_run"],
|
||||
"deleted_file_count": 0,
|
||||
"max_delete_files": kwargs["max_delete_files"],
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual(payload["retention_hours"], 72)
|
||||
self.assertTrue(payload["dry_run"])
|
||||
self.assertEqual(payload["max_delete_files"], 25)
|
||||
|
||||
def test_route_request_alert_feedback_create(self) -> None:
|
||||
status, payload = route_request(
|
||||
"/trading/alerts/7/feedback",
|
||||
|
||||
@@ -101,6 +101,7 @@ class ApiStaticServerTests(unittest.TestCase):
|
||||
"/trading/market-state",
|
||||
"/trading/candidates",
|
||||
"/trading/alerts/quality-summary",
|
||||
"/trading/visual-watch/cleanup-captures",
|
||||
"/trading/tickets",
|
||||
"/trading/executions",
|
||||
"/trading/journal",
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from scripts import run_visual_capture_cleanup
|
||||
|
||||
|
||||
class RunVisualCaptureCleanupScriptTests(unittest.TestCase):
|
||||
def test_main_outputs_json(self) -> None:
|
||||
with patch("scripts.run_visual_capture_cleanup.cleanup_visual_watch_captures") as cleanup_mock:
|
||||
cleanup_mock.return_value = {
|
||||
"status": "completed",
|
||||
"root": "captures",
|
||||
"eligible_file_count": 2,
|
||||
"deleted_file_count": 2,
|
||||
}
|
||||
output = io.StringIO()
|
||||
with contextlib.redirect_stdout(output):
|
||||
exit_code = run_visual_capture_cleanup.main(
|
||||
env={"VISUAL_CAPTURE_CLEANUP_OUTPUT_JSON": "1", "VISUAL_CAPTURE_CLEANUP_RETENTION_HOURS": "168"}
|
||||
)
|
||||
|
||||
payload = json.loads(output.getvalue())
|
||||
self.assertEqual(exit_code, 0)
|
||||
self.assertEqual(payload["status"], "completed")
|
||||
cleanup_mock.assert_called_once()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -128,6 +128,25 @@ class FakeVisualReviewService:
|
||||
return self.payload
|
||||
|
||||
|
||||
class FakeVisualCaptureRetentionService:
|
||||
def __init__(self, payload=None):
|
||||
self.payload = payload or {
|
||||
"status": "completed",
|
||||
"root": "captures",
|
||||
"retention_hours": 168,
|
||||
"eligible_file_count": 2,
|
||||
"deleted_file_count": 2,
|
||||
"deleted_bytes": 100,
|
||||
"retained_file_count": 1,
|
||||
"reason": None,
|
||||
}
|
||||
self.calls = []
|
||||
|
||||
def cleanup_captures(self, **kwargs):
|
||||
self.calls.append(kwargs)
|
||||
return dict(self.payload)
|
||||
|
||||
|
||||
class FakeResearchCouncilService:
|
||||
def __init__(self, *, should_fail=False):
|
||||
self.should_fail = should_fail
|
||||
@@ -176,6 +195,7 @@ class TradingRuntimeSchedulerTests(unittest.TestCase):
|
||||
"pending_watch_run_ids": [801],
|
||||
}
|
||||
),
|
||||
visual_capture_retention_service=FakeVisualCaptureRetentionService(),
|
||||
research_council_service=FakeResearchCouncilService(),
|
||||
)
|
||||
with patch("src.services.runtime.trading_runtime_service.settings.trading_runtime_symbols", "BTC-USDT"):
|
||||
@@ -193,6 +213,7 @@ class TradingRuntimeSchedulerTests(unittest.TestCase):
|
||||
self.assertTrue(payload["tasks"]["decision_packet"]["details"]["visual_watch_reserved"])
|
||||
self.assertEqual(payload["tasks"]["visual_watch"]["details"]["visual_reviews_created"], 1)
|
||||
self.assertEqual(payload["tasks"]["visual_watch"]["details"]["last_visual_state"], "confirm")
|
||||
self.assertEqual(payload["tasks"]["visual_capture_cleanup"]["details"]["deleted_file_count"], 2)
|
||||
self.assertEqual(payload["latest_broker_state_by_symbol"]["BTC-USDT"]["account"]["equity"], 1000.0)
|
||||
self.assertTrue(payload["tasks"]["watch"]["next_run_ts"])
|
||||
|
||||
@@ -205,6 +226,7 @@ class TradingRuntimeSchedulerTests(unittest.TestCase):
|
||||
reconciliation_service=FakeReconciliationService(),
|
||||
decision_packet_service=FakeDecisionPacketService(),
|
||||
visual_review_service=FakeVisualReviewService(),
|
||||
visual_capture_retention_service=FakeVisualCaptureRetentionService(),
|
||||
research_council_service=FakeResearchCouncilService(),
|
||||
)
|
||||
with patch("src.services.runtime.trading_runtime_service.settings.trading_runtime_symbols", "BTC-USDT"):
|
||||
@@ -228,6 +250,7 @@ class TradingRuntimeSchedulerTests(unittest.TestCase):
|
||||
decision_packet_service=FakeDecisionPacketService(),
|
||||
alert_dispatcher_service=FakeAlertDispatcherService(),
|
||||
visual_review_service=FakeVisualReviewService(),
|
||||
visual_capture_retention_service=FakeVisualCaptureRetentionService(),
|
||||
research_council_service=FakeResearchCouncilService(),
|
||||
)
|
||||
supervisor = TradingRuntimeSupervisor(scheduler=scheduler)
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from src.services.visual_watch.capture_retention_service import VisualCaptureRetentionService
|
||||
|
||||
|
||||
class VisualCaptureRetentionServiceTests(unittest.TestCase):
|
||||
def test_cleanup_deletes_old_html_and_json_files(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
old_html = root / "watch-old.html"
|
||||
old_json = root / "watch-old.json"
|
||||
new_html = root / "watch-new.html"
|
||||
old_html.write_text("old", encoding="utf-8")
|
||||
old_json.write_text("old-json", encoding="utf-8")
|
||||
new_html.write_text("new", encoding="utf-8")
|
||||
old_ts = (datetime.now(timezone.utc) - timedelta(hours=200)).timestamp()
|
||||
new_ts = (datetime.now(timezone.utc) - timedelta(hours=2)).timestamp()
|
||||
os.utime(old_html, (old_ts, old_ts))
|
||||
os.utime(old_json, (old_ts, old_ts))
|
||||
os.utime(new_html, (new_ts, new_ts))
|
||||
|
||||
service = VisualCaptureRetentionService(output_root=root)
|
||||
result = service.cleanup_captures(retention_hours=168, max_delete_files=10, dry_run=False)
|
||||
|
||||
self.assertEqual(result["status"], "completed")
|
||||
self.assertEqual(result["eligible_file_count"], 2)
|
||||
self.assertEqual(result["deleted_file_count"], 2)
|
||||
self.assertFalse(old_html.exists())
|
||||
self.assertFalse(old_json.exists())
|
||||
self.assertTrue(new_html.exists())
|
||||
|
||||
def test_cleanup_dry_run_keeps_files(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
old_html = root / "watch-old.html"
|
||||
old_html.write_text("old", encoding="utf-8")
|
||||
old_ts = (datetime.now(timezone.utc) - timedelta(hours=200)).timestamp()
|
||||
os.utime(old_html, (old_ts, old_ts))
|
||||
|
||||
service = VisualCaptureRetentionService(output_root=root)
|
||||
result = service.cleanup_captures(retention_hours=168, dry_run=True)
|
||||
|
||||
self.assertEqual(result["deleted_file_count"], 1)
|
||||
self.assertTrue(old_html.exists())
|
||||
|
||||
def test_cleanup_returns_idle_when_root_missing(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
missing = Path(tmpdir) / "missing"
|
||||
service = VisualCaptureRetentionService(output_root=missing)
|
||||
|
||||
result = service.cleanup_captures()
|
||||
|
||||
self.assertEqual(result["status"], "idle")
|
||||
self.assertEqual(result["reason"], "capture_root_missing")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user