Skip to content

Harden security, reliability, and CI from adversarial review#2

Merged
Co-Messi merged 2 commits into
mainfrom
fix-roast-report
Jul 8, 2026
Merged

Harden security, reliability, and CI from adversarial review#2
Co-Messi merged 2 commits into
mainfrom
fix-roast-report

Conversation

@Co-Messi

@Co-Messi Co-Messi commented Jul 8, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes every issue in the adversarial codebase review (.roast/REPORT-latest.md): 6 High, 10 Medium, 5 Low, plus the CI/supply-chain findings.

API security (High 1, 2)

  • Non-loopback bind guard: HYPERDATA_API_HOST=0.0.0.0 is now refused unless HYPERDATA_API_KEY is set (auth enforced on all non-health routes via Authorization: Bearer or X-API-Key) or HYPERDATA_UNSAFE_PUBLIC_API=1 explicitly accepts the risk.
  • CORS: wildcard only on loopback; non-loopback binds require an HYPERDATA_CORS_ORIGINS allowlist, otherwise no CORS headers.
  • Rate limiting: per-IP sliding window (300 req/min) on all REST routes.
  • WebSocket backpressure: bounded per-client send queue + single writer task per client (slow clients drop their events, counted, instead of unbounded ensure_future fan-out). Inbound message size cap, rate cap, and bad-message disconnect threshold. Heartbeat loop logs failures instead of pass.

Reliability (High 3–5; Med 8, 9)

  • Hub startup contract: failed components land in status.failed_components, force /v1/health to degraded, and the "all components online" log is only emitted when it's true. position_scanner/market_data report startingconnected/error from their actual loop outcomes instead of a blind ready.
  • Explicit timeouts on every external HTTP call (Hyperliquid info API, HL price poll, Telegram/Discord webhooks) so a hung endpoint can't wedge a core loop.
  • Exchange payload parse guards: malformed Binance/OKX/Bybit/HL records and orderbook levels are dropped individually with per-exchange parse_errors counters (exposed in liquidation stats) — no more crash/reconnect loops on schema drift.
  • Per-venue orderflow freshness (venue_freshness(), surfaced in /v1/health) so a dead venue can't hide behind the combined feed; the status loop body is isolated so one bad get_stats() can't kill the staleness watchdog.

Data integrity (Med 1, 3–5)

  • Liquidation dedup now buckets on exchange event time, and the cascade bypass is venue-scoped (this also fixes a latent bug where bypass keys were written and read in different formats, so the bypass never fired).
  • Corrupted SQLite DBs are quarantined to data/corrupted/ with a timestamp instead of deleted.
  • schema_version table + versioned migrations; only duplicate column is treated as already-applied, anything else raises.
  • Wallet addresses validated (0x + 40 hex), lowercased, capped at 50k with LRU-style expiry.

Strategy safety (Med 2, 6, 7)

  • Smart-money ranking requires ≥10 closed trades and ≥$50k volume, exposes a 0–1 confidence per wallet plus ranking-criteria caveats in stats, and clears stale smart/dumb tiers when a wallet stops qualifying.
  • Paper trader: adds to a position are balance-checked, entry price is size-weighted, invalid signals (negative/NaN/inf size, bad action) are rejected — balance can never go negative.
  • LLM agent: first line must be exactly BUY/SELL/HOLD ("I would not BUY here" is rejected, never substring-matched), one persistent worker thread, and an LLM_MAX_EVALS_PER_HOUR budget.

Tests & CI (High 6, Low 1–5)

  • tests/test_position_scanner.py rewritten against the SQLite address store (all network mocked) and re-enabled in CI.
  • New tests/test_hardening.py (43 tests): bind guard, auth/CORS/rate-limit middleware over a live test server, WS abuse limits, dedup/cascade, malformed payloads, degraded hub startup, paper-trader invariants, LLM parsing, DB quarantine, schema versioning, alert redaction, per-venue freshness.
  • CI: ruff is now blocking (all 94 pre-existing lint errors cleaned; dashboards get a scoped E501 ignore for ASCII art), actions pinned by commit SHA, installs use requirements.lock, advisory pip-audit step added.
  • Dead copy-trading and smart-money handlers deleted from api_server.py; README/API docs foreground the exposure model; alert logs redact payloads (wallet intelligence no longer sits in rotating logs); dashboard menu input no longer mutates executor internals.

Test plan

  • python -m pytest tests/ -q — 148 passed, 2 skipped (live-marked)
  • ruff check src/ tests/ — clean
  • py_compile run_api.py run_dashboard.py + import smoke checks — pass

Fixes every issue in .roast/REPORT-latest.md (6 High, 10 Medium, 5 Low).

API security (High 1, 2):
- Refuse non-loopback bind unless HYPERDATA_API_KEY is set (Bearer /
  X-API-Key auth on all non-health routes) or HYPERDATA_UNSAFE_PUBLIC_API=1
  explicitly accepts the risk
- CORS wildcard only on loopback; HYPERDATA_CORS_ORIGINS allowlist otherwise
- Per-IP REST rate limiting (300 req/min sliding window)
- WebSocket: bounded per-client send queues with one writer task per client
  (drop-and-count instead of unbounded task fan-out), inbound message
  size/rate limits, bad-message disconnect threshold, heartbeat loop errors
  logged instead of swallowed

Reliability (High 3, 4, 5; Med 8, 9):
- Hub startup contract: component failures recorded in
  status.failed_components, surfaced in /v1/health and an honest degraded
  log line; position_scanner/market_data report 'starting' until their
  first cycle succeeds instead of a blind 'ready'
- Status-update loop body wrapped so one bad component can't kill the
  staleness watchdog; HLP z-score alerts debounced to once a minute
- Explicit aiohttp timeouts on every external HTTP call (market data,
  position scanner, HL price poll, Telegram/Discord webhooks)
- Exchange payload parse guards: malformed Binance/OKX/Bybit/HL records and
  orderbook levels are dropped individually and counted (parse_errors in
  liquidation stats) instead of crashing connection loops
- Per-venue orderflow freshness (venue_freshness) so a dead venue can't
  hide behind the combined feed; hub warns when one venue goes silent

Data integrity (Med 1, 3, 4, 5):
- Liquidation dedup buckets on exchange event time and the cascade bypass
  is venue-scoped (also fixes bypass keys that never matched)
- Corrupted SQLite files are quarantined to data/corrupted/ with a
  timestamp instead of deleted
- schema_version table with versioned migrations; only duplicate-column
  errors are treated as already-applied
- Wallet addresses validated (0x + 40 hex), normalized to lowercase, and
  capped at 50k with least-recently-seen expiry

Strategy safety (Med 2, 6, 7):
- Smart-money ranking requires 10+ trades and $50k+ volume, exposes a
  sample-size confidence score, and clears stale tiers on requalification
- Paper trader: balance-checked position adds, weighted-average entry
  price, invalid-signal rejection — balance can never go negative
- LLM agent: strict first-line-only action parsing (ambiguous output is
  rejected, never substring-matched), persistent worker thread, and an
  evals-per-hour budget (LLM_MAX_EVALS_PER_HOUR)

Tests and CI (High 6, Low 1-5):
- test_position_scanner.py rewritten against the SQLite address store and
  re-enabled in CI (all network mocked)
- New tests/test_hardening.py: bind guard, auth/CORS/rate-limit middleware,
  WS abuse limits, dedup/cascade, malformed payloads, degraded startup,
  paper-trader invariants, LLM parsing, quarantine, schema version,
  alert redaction, per-venue freshness (43 tests)
- CI: lint blocking (repo lint debt cleaned), actions pinned by SHA,
  requirements.lock used for installs, advisory pip-audit step
- Dead copy-trading/smart-money handlers deleted from api_server
- README/API docs foreground the exposure model; alert logs no longer
  persist full payloads (wallet intelligence redacted)
- run_dashboard menu input no longer mutates executor internals
@Co-Messi Co-Messi marked this pull request as ready for review July 8, 2026 03:58
@Co-Messi

Co-Messi commented Jul 8, 2026

Copy link
Copy Markdown
Owner Author

@codex

@Co-Messi

Co-Messi commented Jul 8, 2026

Copy link
Copy Markdown
Owner Author

@Verdict

@verdict-0528 verdict-0528 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What this PR does: Hardens security, reliability, and CI based on an adversarial codebase review, adding API bind guards, auth, rate limits, WS backpressure, parse guards, quarantine, schema versioning, strategy safety, and updating CI to pinned commit SHAs and blocking ruff.
Risk areas:

  • The setup-python step in the CI workflow is missing the with: block that specifies python-version, so the Python version matrix will be ignored and all jobs will use the runner's default Python; this undermines cross-version testing.
    Verdict: ⚠️ Minor concerns

Comment thread .github/workflows/ci.yml
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 P1 (High): The setup-python action is invoked without a with: block specifying python-version: ${{ matrix.python-version }}, so the Python version matrix is not applied. This means all jobs will use the runner's default Python and cross-version testing is lost. Add with: python-version: ${{ matrix.python-version }} to restore matrix testing.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b0875b3da1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ci.yml
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restore matrix Python selection

This step no longer passes python-version to actions/setup-python; the action docs state that without this input it falls back to .python-version or the runner PATH, and this repo has no .python-version. The matrix value is now only in the display name, so the two jobs won't actually exercise 3.12 and 3.13 and version-specific failures can slip through CI.

Useful? React with 👍 / 👎.

Comment thread src/data_layer/liquidation_feed.py Outdated
await self.feed.emit(event)
except Exception:
logger.debug("[bybit] failed to parse liquidation: %s", d)
except (KeyError, TypeError, ValueError):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep malformed Bybit payloads from reconnecting

Narrowing this handler to KeyError, TypeError, and ValueError leaves AttributeError uncaught, but the lines above call d.get(...) without first proving data["data"] is a dict. If Bybit sends an allLiquidation message with a non-dict data shape during schema drift or an error payload, _on_message raises out to the websocket loop and reconnects instead of recording a parse error, which is exactly the crash/reconnect path this hardening is meant to avoid.

Useful? React with 👍 / 👎.

@Co-Messi Co-Messi left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing single comment 0

Comment thread src/data_layer/liquidation_feed.py Outdated
await self.feed.emit(event)
except Exception:
logger.debug("[bybit] failed to parse liquidation: %s", d)
except (KeyError, TypeError, ValueError):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test Bybit

@Co-Messi Co-Messi left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing single comment 1

self._init_tables()
except sqlite3.DatabaseError:
logger.warning("Database corrupted at %s — recreating", self.db_path)
if conn is not None:

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test Persistence

@Co-Messi Co-Messi left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing single comment 2

Comment thread src/data_layer/alerts.py
# Log that an alert fired, not its payload — alert bodies can contain
# wallet addresses and position intelligence that must not sit in
# rotating plaintext logs.
first_line = message.strip().splitlines()[0] if message.strip() else ""

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test Alerts

@Co-Messi Co-Messi left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing single comment 3

Comment thread src/strategies/llm_agent.py Outdated
"""
lines = text.split("\n", 1)
action_word = lines[0].strip().upper()
action_word = lines[0].strip().upper().strip(".!:*# ")

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test LLMAgent

@Co-Messi Co-Messi left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing single comment 4

Comment thread .github/workflows/ci.yml Outdated
pip install pytest ruff
pip install -r requirements.lock
pip install -e . --no-deps
pip install pytest pytest-asyncio ruff pip-audit

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test CI

@Co-Messi Co-Messi left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ HyperData-Terminal Full Adversarial & Security Review

This PR successfully resolves many critical security issues (6 High, 10 Medium, 5 Low). However, a deep adversarial review of the newly introduced code has uncovered several remaining vulnerabilities across the websocket ingestion feeds, database quarantine logic, alert redactions, LLM parsing, and CI/CD supply chains.

See inline comments for details, code analogies, and recommended remediations.

Comment thread src/data_layer/liquidation_feed.py Outdated
await self.feed.emit(event)
except Exception:
logger.debug("[bybit] failed to parse liquidation: %s", d)
except (KeyError, TypeError, ValueError):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Adversarial Finding - High]: Unhandled AttributeError can crash Bybit Ingestion Feed.

Unlike OKXConnection, BybitConnection._on_message does not catch AttributeError in its except block. If Bybit streams a message where symbol or side fields are absent or null (parsed as None), the subsequent call to normalize_symbol(symbol_raw, "bybit") will execute None.upper(), throwing an AttributeError.

Since this exception is unhandled, it will propagate up and crash the websocket connection message loop, forcing a socket disconnect and reconnect loop.

Analogy: A security gate that locks itself shut and triggers a full power cycle whenever someone tries to swipe a blank card instead of just rejecting the card.

Fix: Catch AttributeError in the except tuple:

        except (KeyError, TypeError, ValueError, AttributeError):
            self.feed.record_parse_error("bybit", d)

self._init_tables()
except sqlite3.DatabaseError:
logger.warning("Database corrupted at %s — recreating", self.db_path)
if conn is not None:

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Adversarial Finding - Medium]: Database Quarantine False Positives on Lock Contention.

sqlite3.DatabaseError is the base class for all sqlite errors, including sqlite3.OperationalError (e.g. "database is locked" or "disk I/O error").

If a concurrent process (like a live dashboard view or an automated verification run) holds a lock on the database during hub startup, the integrity check or connection setup will fail with OperationalError: database is locked. The code will catch this as a corruption error, quarantine the perfectly healthy database, and start a fresh empty database.

Analogy: If your front door is stuck shut for a minute, you assume the house has collapsed, move your belongings to a temporary shed, and build a new house from scratch.

Fix: Check the exception details. Do not quarantine if the error message indicates lock contention or a busy state:

        except sqlite3.DatabaseError as exc:
            if conn is not None:
                conn.close()
            if "lock" in str(exc).lower() or "busy" in str(exc).lower():
                logger.error("Database is locked/busy — failing startup rather than quarantining")
                raise

Comment thread src/data_layer/alerts.py
# Log that an alert fired, not its payload — alert bodies can contain
# wallet addresses and position intelligence that must not sit in
# rotating plaintext logs.
first_line = message.strip().splitlines()[0] if message.strip() else ""

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Adversarial Finding - Medium]: Plaintext Wallet/Position Leaks in Alert Log Redaction.

The code redacts alert logs by writing only the first_line of the message to console/logs. However, if the alert format is updated or if the alert message has the wallet address or position intelligence on the first line (e.g., Whale alert: 0x... has a large position), this logic will fail. The plaintext wallet intelligence will be written directly into rotation logs.

Analogy: Shredding a letter but only keeping the first line because you assume the first line never contains private secrets. If the writer started with their address, your shredder is useless.

Fix: Use regex to explicitly strip or mask EVM addresses (0x[a-fA-F0-9]{40}) from the logged first line, rather than assuming it's safe:

        import re
        redacted_line = re.sub(r'0x[a-fA-F0-9]{40}', '0x...[REDACTED]', first_line)

Comment thread src/strategies/llm_agent.py Outdated
"""
lines = text.split("\n", 1)
action_word = lines[0].strip().upper()
action_word = lines[0].strip().upper().strip(".!:*# ")

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Adversarial Finding - Low/Robustness]: Split-by-newline is vulnerable to blank lines in LLM responses.

lines = text.split("\n", 1) splits the output of the LLM by the first newline. If the LLM responds with a leading blank line or formatting greeting before outputting BUY/SELL/HOLD, lines[0] will be empty. This will cause the parser to reject a completely valid decision.

Analogy: A robot mail sorter that throws away letters if there is a blank line at the top of the envelope before the address.

Fix: Split and filter out empty lines, then take the first non-empty line:

        all_lines = [l.strip() for l in text.splitlines() if l.strip()]
        if not all_lines:
            return None
        action_word = all_lines[0].upper().strip(".!:*# ")
        reason = "\n".join(all_lines[1:]) if len(all_lines) > 1 else ""

Comment thread .github/workflows/ci.yml Outdated
pip install pytest ruff
pip install -r requirements.lock
pip install -e . --no-deps
pip install pytest pytest-asyncio ruff pip-audit

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Supply Chain - Medium]: Unpinned test dependencies in CI workflow.

Installing pytest, pytest-asyncio, ruff, and pip-audit without version pinning in the CI workflow exposes the pipeline to non-deterministic test environments and package hijacking attacks. If a malicious version of any of these is published to PyPI, it will be executed automatically in your CI workflow.

Analogy: Inviting random contractors off the street into your workshop without checking their credentials or tools.

Fix: Pin the versions in the workflow file directly or include them in requirements.lock:

          pip install pytest==8.3.4 pytest-asyncio==0.25.2 ruff==0.9.1 pip-audit==2.7.3

Comment thread pyproject.toml
[tool.ruff.lint]
select = ["E", "F", "W", "I"]

[tool.ruff.lint.per-file-ignores]

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Packaging Security - Medium]: Packaging of raw config* directory.

(Reviewer note: Commenting here on the ruff addition since the setuptools section is unchanged in this diff)

Line 20 of pyproject.toml includes config* in the packaging build: include = ["src*", "config*"]. This causes any local config templates, settings, or accidental secrets stored in the config/ directory to be packaged into the distributed .whl and .tar.gz archive artifacts upon building, exposing internal file structures to downstream installers.

Analogy: Packing your private notes and house diagrams into a box of free books that you leave on the sidewalk.

Fix: Restrict finding to src only in pyproject.toml line 20:

include = ["src*"]

@Co-Messi Co-Messi left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial re-review of PR #2. The PR fixes the headline issues but ships several critical regressions and dead-code paths that defeat the stated protections. 22 findings below — CRIT/HIGH should land before merge. Verified each by tracing the relevant code path; line numbers reference the fix-roast-report branch.

Top issues:

  • !forceOrder@arr Binance stream silently produces zero events (liquidation_feed.py:165)
  • Cascade bypass self-perpetuates; size rounded to $100 = 100x false-dedup window (api_server.py:442, 482)
  • Paper trader close path lets balance go negative (paper_trader.py:208)
  • Cascade example strategy is silently non-functional (liquidation_cascade.py:47)
  • WebSocket upgrade has no Origin/CSRF check - cross-site scraping on loopback (api_server.py:686)
  • Telegram bot token + Discord webhook URL leak in logger.exception tracebacks (alerts.py:384)
  • LLM_MAX_EVALS_PER_HOUR budget consumed before the LLM call - flaky provider exhausts it in 20 min (llm_agent.py:80)
  • evaluate() blocks asyncio for up to 20s per call - paper trader stalls on every slow LLM response (llm_agent.py:111)
  • pip-audit has continue-on-error: true - known CVEs do not block CI; no Dependabot (ci.yml:37)

Please address CRIT + HIGH before merging. I'll re-review once patches land.

Additional findings (lines not in PR diff hunks — listed for reference):

  1. src/strategies/examples/liquidation_cascade.py:47 -- [CRIT] Cascade example strategy is silently non-functional....

  2. src/strategies/paper_trader.py:259 -- [HIGH] In-memory state mutated before SQLite persist; state diverges on DB error....

  3. src/data_layer/liquidation_feed.py:144 -- [HIGH] No max retry / circuit breaker on WS reconnect -- pinned CPU on dead endpoint....

  4. src/data_layer/position_scanner.py:59 -- [MED] asyncio.gather without return_exceptions=True -- one bad endpoint aborts price + meta updates together....

Comment thread src/data_layer/liquidation_feed.py Outdated
@@ -159,19 +163,25 @@ def __init__(self, feed: LiquidationFeed):

async def _on_message(self, data: Any) -> None:
if isinstance(data, dict) and data.get("e") == "forceOrder":

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[CRIT] Binance !forceOrder@arr stream produces zero events.

Handler gates on isinstance(data, dict) and data.get('e') == 'forceOrder'. !forceOrder@arr wraps each frame in a JSON array [ {...}, {...} ], so the array payload fails the type check and the early return skips BOTH the parse block AND the record_parse_error increment. Result: liquidations.get_stats(60)['by_exchange'] has no binance key, the PR's own counter stays at 0, no warning fires. Cascade alerts that key off coverage['binance'] = sampled look like Binance is feeding when it isn't.

Fix: iterate data when it's a list -- for raw in (data if isinstance(data, list) else [data]): -- before the per-record try/except.

Comment thread src/api_server.py
bypass_key = f"{ev.symbol}_{ev.side}"
bypass_key = f"{ev.symbol}_{ev.side}_{ev.exchange}"
if bypass_key in self._cascade_bypass and now < self._cascade_bypass[bypass_key]:
return False

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[CRIT] Dedup silently falls back to local clock on missing event time.

ev_ts = ev.timestamp if ev.timestamp > 0 else now violates the PR's 'buckets on exchange event time' claim. When any connector's or 0 fallback fires (Binance/Bybit/OKX/HL parse guards), the dedup hash uses local receive time. Two timestamp=0 events within 3s and matching symbol/side/exchange collide and are silently deduped.

Fix: skip dedup entirely when ev.timestamp <= 0 (treat as 'cannot safely hash'). Never substitute local clock for missing exchange time.

Comment thread src/api_server.py
bypass_key = f"{ev.symbol}_{ev.side}_{ev.exchange}"
if bypass_key in self._cascade_bypass and now < self._cascade_bypass[bypass_key]:
return False

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[CRIT] size_usd rounded to nearest $100 = 100x false-dedup window.

round(99950, -2) == round(100049, -2) == round(100050, -2) == 100000. Two distinct liquidations with sizes $99,950 vs $100,050 sharing symbol/side/exchange within 3s dedupe as the same event. In a 50+ liq/min cascade this is near-certain false-dedup. Combined with cascade bypass feedback loop (line 482), a single cascade both over-dedups legitimate events AND over-broadcasts replayed duplicates.

Fix: drop size_rounded from the hash, or use round(ev.size_usd, -1) (nearest $10), or include the full size.

Comment thread src/api_server.py Outdated
self._cascade_bypass[bypass_key] = now + self._CASCADE_BYPASS_DURATION
# Bypass dedup only for this exchange's stream: cascades on one
# venue say nothing about duplicates on another.
self._cascade_bypass[key] = now + self._CASCADE_BYPASS_DURATION

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[CRIT] Cascade bypass self-perpetuates via feedback loop.

self._cascade_bypass[key] = now + self._CASCADE_BYPASS_DURATION is re-set on every call where len(entries) >= 3 (the if entries >= 3: branch above). On WS replay, duplicates pass _is_duplicate_liq, get appended to _cascade_tracker[key], and re-trigger cascade detection -- extending the bypass to now + 30 again. A single 5-min replay floods the broadcast with 10x duplicates and the bypass NEVER expires while traffic continues. cascade_bypass dict has no size or time cap.

Fix: track first_bypass_at per key; refuse to extend past first_bypass_at + 2*CASCADE_BYPASS_DURATION (60s absolute cap). Also bound _cascade_tracker to N entries per key.

Comment thread src/strategies/paper_trader.py Outdated
@@ -178,8 +208,19 @@ def _execute_trade(self, strategy_name: str, signal: Signal) -> None:
self.balance += pos["size_usd"] + pnl

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[CRIT] Close path allows balance to go negative.

self.balance += pos['size_usd'] + pnl -- no floor. Open a short at price=100, size=$1000. Price moves to 10000. pnl = 1000 * (-99) = -99000. balance += 1000 - 99000 = -98000. Violates the PR claim 'balance can never go negative'. A long close with >100% loss has the same bug.

Fix: self.balance = max(0.0, self.balance + pos['size_usd'] + pnl) -- or enforce maintenance margin: reject close if pos['size_usd'] + pnl < 0 and force liquidate remainder at 0.

Comment thread src/data_layer/persistence.py Outdated
p.unlink()
dest = quarantine_dir / f"{p.name}.{stamp}"
try:
p.rename(dest)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[HIGH] Quarantine fallback can brick the system on a stuck DB.

p.rename(dest) may fail with OSError (cross-device EXDEV, EBUSY if held open, EACCES, ENOSPC). The catch logs and falls back to p.unlink(). If unlink also fails (Windows file handle, read-only mount), the corrupted DB remains. Then line 80+: sqlite3.connect(str(self.db_path), ...) re-opens the same corrupted file -> raises sqlite3.DatabaseError again. The except at line 59 does NOT catch this -- it propagates out of __init__ uncaught, crashing the process on startup. The 'last resort so we can still start' comment is misleading: no last-resort success path.

Fix: wrap recovery sqlite3.connect in its own try/except. On failure, open :memory: and log fatal. Use shutil.move (handles cross-device) before unlink.

Comment thread src/data_layer/hub.py Outdated
now = time.time()
if abs(hlp_zscore) > 2.0 and now - _last_hlp_alert_check > 60.0:
_last_hlp_alert_check = now
asyncio.create_task(

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[HIGH] Fire-and-forget task + dead HLP-zscore alert body.

Line 550: asyncio.create_task(self.alerts._check_hlp_zscore(self.hlp.get_stats())) -- task handle discarded. Under PyPy/debug asyncio, GC of unreferenced tasks is a real failure mode. Worse: alerts._check_hlp_zscore (lines 174-183) is a no-op -- body is pass under # HLP Z-score alerts -- DISABLED. Every status tick where |z|>2 for >60s creates and drops a coroutine that does nothing. Status log claims an alert path is in use; it is not. Cascade alerts (line 152-160) and large_liq (145-148) are also disabled bodies.

Fix: either delete the create_task block + the dead _check_* methods, OR re-enable the alert send and store the task on self._tasks so stop() awaits it.

Comment thread src/api_server.py Outdated
if bypass_key in self._cascade_bypass and now < self._cascade_bypass[bypass_key]:
return False

ev_ts = ev.timestamp if ev.timestamp > 0 else now

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[HIGH] Dedup safety depends on parse-layer divide-by-1000 convention -- undocumented invariant.

The dedup hash int(ev_ts // self._DEDUP_WINDOW) buckets on seconds. All 4 parse guards divide by 1000.0 (Binance 178, Bybit 228, OKX 270, HL 340). But this normalization is NOT enforced at the dedup layer. If a future connector adds ev_ts = d.get('time', 0) without /1000.0, a single event from that connector would hash to a different bucket than the same event from a properly-normalized connector, silently defeating cross-venue dedup.

Fix: sanity guard in _is_duplicate_liq: if not (1e9 < ev_ts < 1e11): return False # not safely dedupable. Log warning with offending exchange.

Comment thread .github/workflows/ci.yml Outdated
- name: Dependency vulnerability audit
# Advisory: a newly-published CVE in a pinned dep should be visible
# without failing unrelated PRs. Review failures in the job log.
continue-on-error: true

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[HIGH] pip-audit has continue-on-error: true -- known CVEs do not block CI. No Dependabot config.

Line 37: continue-on-error: true on the pip-audit -r requirements.lock step. PR description admits this is 'advisory'. In practice, a known CVE in any pinned dep (aiohttp, httpx, etc.) can land in main with green CI. No follow-up step parses output and fails the build. No scheduled weekly run. No Dependabot/Renovate config (.github/dependabot.yml returns 404). Lockfile is the source of truth and is never re-pinned by CI.

Fix: drop continue-on-error: true + add --strict, OR add a follow-up step that fails the job if any vuln has a fix and severity >= medium. Add Dependabot for pip and for github-actions SHAs.

Comment thread src/data_layer/market_data.py Outdated

# Every outbound request gets an explicit deadline: a hung exchange endpoint
# must fail the refresh cycle, not stall the hub's market-refresh loop forever.
HTTP_TIMEOUT = aiohttp.ClientTimeout(total=10)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MED] Hyperliquid info API uses ClientTimeout(total=10) only -- no connect/sock_connect split.

With total=10 and no connect= or sock_connect=, a slow TLS handshake (HL throttling the IP after a rate spike) can consume the entire 10s budget, leaving zero for the response body -- a hung endpoint masquerades as a 10s partial-read error. Under sustained HL throttling, both market_refresh and position_scan polls' 10s budgets collapse in lockstep. Same hole at position_scanner.py:20.

Fix: aiohttp.ClientTimeout(total=10, connect=3, sock_connect=3, sock_read=5). Apply to position_scanner.HTTP_TIMEOUT too.

@Co-Messi

Co-Messi commented Jul 8, 2026

Copy link
Copy Markdown
Owner Author

Follow-up: 4 findings from PR #2 adversarial review

Lines not in diff hunks — GitHub API only allows inline review comments on changed lines. Listed here in full.

1. src/strategies/examples/liquidation_cascade.py:47

[CRIT] Cascade example strategy is silently non-functional.

long_liq_usd = getattr(stats, "total_long_usd", 0) or 0. hub.liquidations.get_stats() returns a dict (confirmed by llm_agent.py:211 using liq_stats.get('total_count', 0)). getattr(dict, key, default) always returns the default — dict keys are not attributes. long_liq_usd is always 0. The if long_liq_usd >= 1_000_000 branch never fires. User deploys expecting cascade detection and gets zero signals with no error.

Fix: use dict access — stats.get("total_long_usd", 0) or 0. Verify the actual key name in the stats dict.

2. src/strategies/paper_trader.py:259

[HIGH] In-memory state mutated before SQLite persist; state diverges on DB error.

On open/add/close, positions dict and self.balance are mutated, THEN self._db.execute(INSERT...) runs. If sqlite3.Error fires (disk full, DB locked, schema mismatch), the trade is executed in-memory but never logged. get_portfolio() reports the new state; the trade log is missing the entry. Repeated DB errors silently corrupt portfolio view vs audit trail.

Fix: persist first, then mutate — build the trade record, insert + commit, then update positions/balance. Or wrap insert + portfolio snapshot row in a single SQLite transaction. At minimum, on DB error, roll back the in-memory mutation.

3. src/data_layer/liquidation_feed.py:144

[HIGH] No max retry / circuit breaker on WS reconnect — pinned CPU on dead endpoint.

if self._running: await asyncio.sleep(self._backoff); self._backoff = min(self._backoff * 2, self.MAX_BACKOFF). No MAX_RETRIES. A deprecated/403 WSS endpoint sits in an infinite reconnect loop, doing TLS handshake + HTTP upgrade every 60s with logger.info("[%s] reconnecting in %.1fs") every cycle. Survives stop() only via self._running = False. Competes for the event loop with _dispatch lock.

Fix: add MAX_CONSECUTIVE_FAILURES = 50 counter; after N consecutive failures without a successful message, logger.error once, stop the task, set self._ws = None so next start() rebuilds. Reset counter on any successful _on_message.

4. src/data_layer/position_scanner.py:59

[MED] asyncio.gather without return_exceptions=True — one bad endpoint aborts price + meta updates together.

await asyncio.gather(self.update_prices(), self.update_meta()). If update_prices raises (10s timeout, ClientError), the gather raises and the partially-completed update_meta (which may have returned data) is discarded. _position_scan_loop catches and sleeps 15s. A single dead endpoint means market_meta (5-min cache for maintenance margins) is never refreshed for the entire window. The batched address loop at line 76 correctly uses return_exceptions=True; this one was missed.

Fix: results = await asyncio.gather(self.update_prices(), self.update_meta(), return_exceptions=True) then consume results independently.

CI:
- Restore 'python-version: ${{ matrix.python-version }}' on setup-python
  (the matrix was display-only after the rewrite)
- Pin test tooling versions in the workflow; make pip-audit blocking
- Bump aiohttp 3.13.3 -> 3.14.1 in requirements.lock (4 published CVEs)
- Add .github/dependabot.yml (pip + github-actions, weekly)

API server:
- WebSocket upgrades with a browser Origin header are rejected unless the
  origin is allowlisted (browser WS is not gated by the same-origin policy,
  so any webpage could read the loopback stream)
- Split /v1/live (minimal, unauthenticated) from /v1/health (detailed,
  requires the API key when one is set — the payload is operational recon)
- Health/liveness paths exempt from per-IP rate limiting (LB/monitor
  probes share a NAT IP)
- Dedup: events without a plausible epoch-seconds timestamp are never
  deduped (no local-clock fallback, no ms-scale hashing); the hash uses
  the exact size instead of round(-2) so distinct ~$100k events cannot
  collapse; cascade bypass has an absolute 2x-duration cap so replayed
  duplicates cannot extend it forever; cascade tracker bounded per key
- /v1/health reports degraded when position scanner or market data loops
  are in 'error' (previously only startup failures counted)

Feeds:
- Binance forceOrder handler accepts array frames (@arr batching) instead
  of silently dropping them
- Bybit handler accepts v5 list payloads and short keys (p/v/S/s/T),
  catches AttributeError, and drops malformed records individually
- Exchange WS reconnect escalates to a single ERROR after 20 consecutive
  failures and quiets the log spam; still retries at max backoff (a hard
  stop would permanently kill the feed across a venue outage)
- Split connect/sock_read timeouts on all external HTTP deadlines so a
  slow TLS handshake can't consume the whole budget
- Position scanner price/meta updates use return_exceptions so one dead
  endpoint doesn't discard the other's result

Strategies:
- Paper trader: close credit floored at zero balance (a >100% adverse move
  on a short could previously drive balance negative); trades persist to
  SQLite BEFORE mutating the books, so a DB error can no longer diverge
  the portfolio from the audit trail
- LLM agent: evaluate() is now async and awaited by the paper trader (a
  slow LLM no longer blocks the event loop); transport-level failures
  refund their budget slot (a down provider can't exhaust the hourly
  budget; parse failures still consume theirs since tokens were billed);
  leading blank lines in responses are tolerated
- Cascade example strategy actually fires: getattr on the stats dict
  always returned 0 (and the key name was wrong) — now reads
  stats['long_volume_usd']

Persistence & alerts:
- 'database is locked/busy' raises instead of quarantining a healthy DB
  held by another process; quarantine uses shutil.move and falls back to
  an in-memory store if the corrupted file can't be moved or removed
- Alert send failures log the exception type only (aiohttp error strings
  can embed the bot-token URL); wallet addresses masked on the logged line
- Removed dead HLP z-score alert scheduling (the alert body is disabled)

Declined with rationale (see PR discussion): config/ stays in packaging
(settings.py is a runtime import, contains no secrets); smart-money tier
persistence is not wired (save_wallet/load_wallets have no callers), so
the restart-resurrection scenario cannot occur.

18 new tests cover the above; 166 passing total.
@Co-Messi

Co-Messi commented Jul 8, 2026

Copy link
Copy Markdown
Owner Author

Review triage — all findings addressed in 0961b0d

Fixed (19):

  • CI: restored python-version matrix input on setup-python (real regression, thanks); pinned test tooling versions; pip-audit is now blocking — which immediately caught 4 published CVEs in aiohttp 3.13.3, bumped to 3.14.1; added Dependabot (pip + github-actions, weekly).
  • WS Origin/CSRF: upgrades carrying a browser Origin are rejected unless allowlisted via HYPERDATA_CORS_ORIGINS (non-browser clients send no Origin and are unaffected).
  • /v1/live vs /v1/health: minimal unauthenticated liveness probe added; the detailed health payload now requires the API key when one is set. Health/live are exempt from the per-IP rate limit.
  • Dedup: no local-clock fallback — events without a plausible epoch-seconds timestamp are never deduped (also covers the ms-normalization invariant via the 1e9–1e11 sanity gate); hash uses exact size (no round(-2) collisions); cascade bypass capped at 2× duration absolute so replays can't extend it forever; tracker bounded per key.
  • Health honesty: scanner/market-data loop errors now force degraded in /v1/health.
  • Binance @arr array frames handled; Bybit v5 list payloads + short keys (p/v/S/s/T) + AttributeError all handled per-record.
  • Reconnect: escalates to a single ERROR after 20 consecutive failures and stops log spam. Deliberately keeps retrying at max backoff instead of hard-stopping: a feed that kills itself during a venue outage never recovers, and one handshake/minute is negligible. (Partial accept.)
  • Paper trader: close credit floored at zero (short blowup can't go negative); persist-first — a trade that can't be logged is not executed.
  • LLM agent: evaluate() is async and awaited by the paper trader (no more 20s event-loop stalls); transport failures refund their budget slot (parse failures still consume — tokens were billed); leading blank lines tolerated.
  • Cascade example: confirmed CRIT — getattr on a dict always returned 0 and the key name was wrong; now stats.get('long_volume_usd', 0).
  • Persistence: locked/busy raises instead of quarantining a healthy DB; shutil.move for cross-device; in-memory fallback if the corrupted file can't be moved or removed (startup can no longer crash).
  • Alerts: failure logs carry exception type only (aiohttp error strings embed the token URL — verified leak path); wallet addresses regex-masked on the logged line; timeout split connect/read everywhere.
  • position_scanner: gather(..., return_exceptions=True) with per-result logging.

Declined (3), with rationale:

  1. Remove config* from packagingconfig/settings.py is imported at runtime (from config.settings import ...); removing it breaks every install. It contains only public endpoints/symbols, no secrets.
  2. Persist tier clears in rank_all()DataStore.save_wallet/load_wallets have zero callers; wallet profiles are not persisted in the live path at all, so the restart-resurrection scenario cannot occur. If wallet persistence gets wired up later, the dirty-set batch write is the right design.
  3. Hard-stop WS reconnect after N failures — replaced with escalation + quiet retry (above); a permanent stop is a worse failure mode for a market-data terminal.

18 new tests cover the fixes; 166 passing, lint + audit clean.

@Co-Messi Co-Messi merged commit 2f7338a into main Jul 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant