Technosight Investment Insights
Sign Up Sign In
Operations & Runbook

10. Operational Monitoring, Troubleshooting & Best Practices

Failure recovery runbooks, log lifecycle management, and execution desk order routing protocols.

Target Audience: Site Reliability Engineers (SREs), DevOps & Head Traders

Key Takeaways for Investors & Traders

Comprehensive Health Checks

Automated verification of database connectivity, market data freshness, and disk space prior to execution.

Failure Recovery Runbooks

Step-by-step resolution paths for lockfile deadlocks, missing feeds, and zero-candidate anomalies.

Structured Log Management

Rotating log architecture with automated archiving and 30-run retention policies.

Order Generation Protocols

Pre-market staging checklists and limit-order execution guidelines for trading desks.

Executive Summary

This operational guide provides diagnostic procedures, health check checklists, and failure recovery runbooks for maintaining the daily Selected Stocks pipeline. It outlines standard protocols for handling stale data, lockfile deadlocks, zero-candidate alerts, and execution order routing.

1

Pipeline Health Checks & Failure Recovery Runbooks

Pre-Flight Verification Checklist

Prior to daily cron execution, the monitoring system verifies:

  1. Database Accessibility: Read/write ping to the QuestDB / PostgreSQL timeseries storage engine.
  2. Data Freshness: Verification that daily bar timestamps match current trading day close ($T$).
  3. Storage Capacity: Confirmation of minimum 20GB free space on /var/log and /data/signals.

Common Failure Modes & Resolution Runbook

  • Lockfile Deadlock (flock: Resource temporarily unavailable):
  • Cause: A prior pipeline run hung or was terminated abruptly without releasing /tmp/technosight_cron_daily.lock.
  • Fix: Check running processes (ps aux | grep sweep_runner). If defunct, kill the process and remove the stale lockfile (rm -f /tmp/technosight_cron_daily.lock).
  • Zero Candidate Alert (Warning: 0 candidates generated):
  • Cause: Extreme market volatility or data feed outage where all symbols failed quality thresholds.
  • Fix: Inspect logs/ for data fetch exceptions. Verify that raw closing prices were successfully ingested.
2

Execution Desk Workflow & Order Routing Protocol

For trading desks executing orders based on pipeline candidates:

  1. Pre-Market Review (08:30–09:15 EST):
  2. Access daily_signal_report.html to review top-ranked candidates.

  3. Verify that candidates have not released market-moving news (earnings or FDA approvals) in pre-market trading that invalidates closing metrics.

  4. Order Staging (09:15–09:29 EST):

  5. Stage orders using Market-on-Open (MOO) or Marketable Limit Orders with limit caps set to entry price $+0.5\%$.

  6. Input predefined stop-loss and target prices directly into broker order tickets as bracket orders.

  7. Post-Open Execution Monitoring (09:30–09:45 EST):

  8. Confirm order fills and record execution slippage relative to model entry reference prices.
  9. Adjust position sizing if opening prices gapped significantly beyond planned boundaries.