From 3ee9cdaa27fc59ea0a7944e5686f68c81cfe1e4e Mon Sep 17 00:00:00 2001 From: Savvanis Spyros Date: Wed, 29 Jul 2026 23:40:23 +0300 Subject: [PATCH 1/6] fix(lantern): one malformed record could stop the estate keeping any logs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CF-18 — status_code is INTEGER and TEXT cannot hold a NUL byte, so a single crafted value made every batch insert throw, and the writer requeued the same poison for ever. Unauthenticated, from one endpoint served before the auth check. Both ingest paths are now sanitised, and a batch that fails on a data error is dropped rather than requeued, so one bad line from a trusted service cannot reproduce it either. CF-41 — the ingest quota keys on the forwarded client IP, not the tunnel socket address, which had put every remote caller in one bucket. CF-39 — Beacon watches the 8545 JSON-RPC surface: chain id asserted as 7412 on all three nodes, state roots compared three blocks below the lowest tip, and liveness requiring the RPC head to track the REST head. CF-48 — incidents render camelCase, so timestamps no longer vanish at exactly the moment Postgres is down. CF-40 — the operator documentation says what keyvault now does. It signs sweeps, so the old advice to fund the treasury by hand or disable withdrawals is gone from both files, replaced by the three steps that actually work. Compose halves of the cross-repo tickets: the seed node runs the EVM chain and publishes 8545 (CF-13), HEARTH_CHAIN_ID is one estate-wide setting with BEACON_HEARTH_CHAIN_ID defaulting to it (CF-30), and Beacon joins the origin allowlists (CF-49). VERIFICATION.md records what was executed against the remediated tree and what it returned — including an end-to-end run of the assembled stack, and a section naming what was not tested. Co-Authored-By: Claude --- .env.example | 182 ++- .github/workflows/ci.yml | 116 +- BLOCKCHAIN.md | 14 +- CHAIN-TESTPLAN.md | 24 +- DEPOSIT-SWEEP.md | 15 + ECOSYSTEM.md | 36 +- MAP.md | 35 +- ROADMAP.md | 1377 ---------------------- TESTING.md | 17 +- VERIFICATION.md | 323 +++++ docker-compose.yml | 102 +- docs/ecosystem/00-current-state.md | 402 +++++++ docs/ecosystem/01-product-vision.md | 150 +++ docs/ecosystem/02-target-architecture.md | 559 +++++++++ findings.md | 602 ---------- infra/beacon/package.json | 3 +- infra/beacon/public/app.js | 63 +- infra/beacon/src/db.js | 42 +- infra/beacon/src/env.js | 29 + infra/beacon/src/journeys/platform.js | 5 + infra/beacon/src/server.js | 26 +- infra/beacon/src/targets.js | 414 ++++++- infra/beacon/test/chainpanel.test.js | 61 + infra/beacon/test/incidents.test.js | 136 +++ infra/beacon/test/rpc.test.js | 462 ++++++++ infra/beacon/test/work.test.js | 76 ++ infra/lantern/package.json | 3 +- infra/lantern/public/app.js | 8 +- infra/lantern/src/clientip.js | 151 +++ infra/lantern/src/db.js | 27 + infra/lantern/src/env.js | 17 + infra/lantern/src/parse.js | 18 +- infra/lantern/src/sanitise.js | 103 ++ infra/lantern/src/server.js | 68 +- infra/lantern/src/store.js | 73 +- infra/lantern/test/poison.test.js | 231 ++++ infra/lantern/test/quota.test.js | 167 +++ 37 files changed, 4060 insertions(+), 2077 deletions(-) delete mode 100644 ROADMAP.md create mode 100644 VERIFICATION.md create mode 100644 docs/ecosystem/00-current-state.md create mode 100644 docs/ecosystem/01-product-vision.md create mode 100644 docs/ecosystem/02-target-architecture.md delete mode 100644 findings.md create mode 100644 infra/beacon/test/chainpanel.test.js create mode 100644 infra/beacon/test/incidents.test.js create mode 100644 infra/beacon/test/rpc.test.js create mode 100644 infra/beacon/test/work.test.js create mode 100644 infra/lantern/src/clientip.js create mode 100644 infra/lantern/src/sanitise.js create mode 100644 infra/lantern/test/poison.test.js create mode 100644 infra/lantern/test/quota.test.js diff --git a/.env.example b/.env.example index 46ad730..882c764 100644 --- a/.env.example +++ b/.env.example @@ -15,11 +15,31 @@ GAME_DATABASE_URL=postgres://cloudsforge:cloudsforge@localhost:5432/game PAY_DATABASE_URL=postgres://cloudsforge:cloudsforge@localhost:5432/pay NIMBUS_PORT=4001 +# Encrypts Nimbus's RS256 signing key at rest. REQUIRED, min 24 chars, +# placeholders refused — Nimbus does not boot without it and will not fall back +# to storing the key in the clear. Generate: openssl rand -hex 32 +# +# That key is the estate's universal forging credential: every service checks +# `iss` plus `aud: cloudsforge` and nothing else, so a token minted with it is +# admin in Pay, the game, Crucible, ForgeMint and ForgeKeyvault alike. It used to +# be plaintext JSONB in a database every service shares one Postgres role on. +# Keep this string OUT of that database and out of any backup of it — the +# separation is the whole mechanism. Same rule, same reason, as +# KEYVAULT_MASTER_SECRET below. +NIMBUS_KEY_SECRET= # Baked into every JWT and checked by every service. COMMENTED for the same # reason as the lists below: the compose default is already this exact value, # so setting it here only pins localhost onto a public deployment — see the # PUBLIC DEPLOYMENT block at the bottom, where you set the real one. # NIMBUS_ISSUER=http://localhost:4001 +# The origin Nimbus builds a password-reset link from — the one URL it produces +# that is read somewhere other than the browser that asked for it, so it comes +# from here and never from the request's Host header (which let a stranger point +# a genuine reset email at a host they controlled). COMMENTED for the same +# reason as NIMBUS_ISSUER: compose already defaults it to +# https://account.${CLOUDSFORGE_APEX}, and pinning localhost here would put a +# localhost link in every reset email a public deployment sends. +# NIMBUS_PUBLIC_URL=http://localhost:4001 # Origins the account portal may hand a sign-in code back to. Explicit list of # real app surfaces only — never a wildcard. Left COMMENTED on purpose: the # compose default already covers localhost plus the CLOUDSFORGE_APEX subdomains, @@ -31,6 +51,48 @@ ADMIN_EMAIL=admin@example.com ADMIN_PASSWORD= ADMIN_HANDLE=admin +# ─── Outbound mail (Nimbus) — password reset links ─────────────────────────── +# +# The only mail this estate sends. Set it and a user who clicks "forgot +# password" is emailed a single-use link that expires in 30 minutes. Leave +# SMTP_HOST unset and nothing is sent — the request is still recorded and an +# operator issues the link from the admin console (Users → Reset password), +# which is how it worked before this existed and is a supported way to run. +# Nimbus says which of the two it is, at info, on every boot. +# +# There is no provider code and no SDK: this is plain SMTP, so Brevo, Resend, +# SendGrid, Mailtrap, a Gmail app password and your own postfix are all the same +# four settings. Changing provider is an edit here and a restart. +# +# FREE CREDENTIALS IN TWO MINUTES (Brevo, 300 emails/day, no custom domain and +# no card): +# 1. Sign up at https://www.brevo.com and confirm the address. +# 2. SMTP & API → SMTP → "Generate a new SMTP key". +# 3. SMTP_HOST=smtp-relay.brevo.com, SMTP_PORT=587, +# SMTP_USER=, +# SMTP_PASS=. +# 4. SMTP_FROM must be an address Brevo has verified — the one you signed up +# with works immediately under Senders & IP. A custom domain is optional. +# Any other provider works with the same four variables; only the host and the +# credentials change. +# +# Unset = no mail. Everything else is only read when this is set. +SMTP_HOST= +# 587 is STARTTLS and is what every provider above documents first. 465 is +# implicit TLS and additionally needs SMTP_SECURE=true; the two are NOT +# interchangeable, and 465 with secure unset hangs until the connect timeout. +SMTP_PORT=587 +SMTP_SECURE=false +SMTP_USER= +# Treat as a secret: it can send mail as you, and nothing rate-limits that but +# the provider. Never your account password — providers issue a separate key. +SMTP_PASS= +# The From header, as a full RFC 5322 address. Must be an address the provider +# has authorised for you, or it accepts the connection and rejects the message. +SMTP_FROM=CloudsForge +# Optional. Where a reply goes, if you would rather it was not the From address. +SMTP_REPLY_TO= + GAME_PORT=4002 NIMBUS_JWKS_URL=http://localhost:4001/.well-known/jwks.json @@ -75,18 +137,45 @@ PAY_CONVERSION_SPREAD_BPS=200 # interact badly: # PAY_WITHDRAWALS_ENABLED default TRUE — withdrawals are accepted out of the box # PAY_SWEEP_ENABLED default false — the sweeper that funds the treasury -# cannot run until ForgeKeyvault grows a signable -# 'treasury' purpose (it refuses to sign for the -# 'deposit' addresses customers pay into). See -# repos/forge-pay/services/pay/src/treasury.ts +# opens no new sweeps. This is a DECISION, not a +# limitation: keyvault has had a signable `sweep` +# shape since it grew one, and its destination is a +# treasury an operator pinned inside the vault +# (forge-keyvault/src/signing.ts EvmShape, +# routes/vault.ts SWEEPABLE_FAMILIES). # # So on a fresh deployment a withdrawal is ACCEPTED, sits `pending` while pay # logs "the treasury cannot cover it" every 20s, and is then automatically # REFUNDED once PAY_WITHDRAWAL_STUCK_MINUTES passes (default 60) — safe to do, # because nothing was ever signed. Not limbo, but an hour of a user's time spent -# on an answer we could have given immediately. Until the sweeper lands, either -# fund the treasury by hand or set PAY_WITHDRAWALS_ENABLED=false so the refusal -# is instant and truthful. +# on an answer we could have given immediately. +# +# Turning sweeping on takes three things, and only the last two are env vars: +# +# 1. Pin a treasury, per chain and network, through ForgeKeyvault's Nimbus +# admin API — POST /admin/treasuries/:chain/:network/mint to create the +# address, then PUT /admin/treasuries/:chain/:network to pin it, in that +# order. Nothing sweeps to an unpinned candidate. This is an admin +# credential and a deliberate act, not a value in this file. +# 2. PAY_SWEEP_ENABLED=true. +# 3. Optionally PAY_TREASURY_TARGET_ (e.g. PAY_TREASURY_TARGET_EMBER=250) +# to hold a float. It defaults to zero, which makes the sweeper purely +# demand-driven — it moves what queued withdrawals actually need, nothing +# more. +# +# Decide (3) on custody grounds rather than convenience. A withdrawal must be +# payable to any address a user names, so the treasury's destination is +# unconstrained by design: every coin in the treasury is inside the blast radius +# of the KEYVAULT_SERVICE_TOKEN, and every coin left in a deposit address is +# outside it. A float buys immediacy on the first withdrawal after a quiet +# period — otherwise a confirmation depth of latency, ~45s on XRP, about a +# quarter of an hour on EMBER — and pays for it in standing exposure. +# +# Sweeping is available for the EVM, Ember and XRP families only. Bitcoin and +# Solana deposit addresses are still refused before decryption, because neither +# has an output policy built yet — so on those two chains the treasury really +# must be funded by hand. The full argument is DEPOSIT-SWEEP.md and the +# authoritative commentary is repos/forge-pay/services/pay/src/env.ts. # Hearth local testnet coin (EMBER) — the seed node's ETHEREUM JSON-RPC, not its # REST port (compose: http://hearth-testnet-seed:8545). Pay speaks eth_* here. # Pointed at 8645 it gets a 200 carrying the legacy getinfo shape, which every @@ -114,6 +203,27 @@ KEYVAULT_URL=http://localhost:4005 # Shared secret for forge-mint -> keyvault server-to-server calls. # REQUIRED, min 24 chars. Generate: openssl rand -hex 24 KEYVAULT_SERVICE_TOKEN= +# RPC endpoints for the chains ForgeMint deploys to. Each chain in +# SUPPORTED_CHAINS ships a working public default for both of its networks, so +# every one of these is optional — set one only to replace the public node for +# that exact chain and network, which is usually because the public node is +# rate-limiting you. +# +# The variable is per chain AND per network: RPC__, chain ids +# ethereum | bsc | polygon | arbitrum | base | solana, network mainnet | +# testnet. There is deliberately no single EVM_RPC_URL — one URL cannot serve +# five different EVM chains, and a shared value would point a Polygon deploy at +# an Ethereum endpoint. (Two such variables did exist here until CF-43. They +# were never read: the fallback that consulted them only fired when a chain had +# no baked default, which none does.) +# +# These are usually keyed provider URLs with the credential in the path +# (Infura /v3/, Alchemy /v2/). forge-mint never logs one in full — +# `safeUrl` in services/forge-mint/src/obs.ts keeps the origin and the first +# path segment and drops the rest — but treat the value as a secret in .env. +# RPC_ETHEREUM_MAINNET=https://mainnet.infura.io/v3/ +# RPC_ETHEREUM_TESTNET=https://sepolia.infura.io/v3/ +# RPC_SOLANA_MAINNET=https://.solana-mainnet.quiknode.pro// # ForgeKeyvault — per-address key custody KEYVAULT_PORT=4005 @@ -127,9 +237,6 @@ KEYVAULT_MASTER_SECRET= KEYVAULT_DATA_DIR=./.keyvault-data # Optional: mount /var/run/docker.sock to provision a container+volume per address KEYVAULT_DOCKER_SOCKET=/var/run/docker.sock -# EVM/Solana testnet RPCs (used for balance checks + testnet deploys) -EVM_RPC_URL=https://ethereum-sepolia-rpc.publicnode.com -SOLANA_RPC_URL=https://api.devnet.solana.com # Crucible — algorithmic trading (http://localhost:4006) # Serves both its SPA and its API on one port, like ForgeMint. @@ -162,6 +269,14 @@ LANTERN_TOKEN= # Days of raw log lines to keep. Grouped issues are kept for 90 regardless, # since they are small and are the thing you actually read. LANTERN_RETENTION_DAYS=7 +# Which socket peers may name the client with cf-connecting-ip or +# x-forwarded-for. /ingest/client is quota'd per client address, and behind the +# cloudflared tunnel every public request has the same socket peer — so without +# this the quota is one bucket of 120/min shared by the whole internet, and +# anyone can hold it empty. The default covers the connector (loopback, or the +# compose bridge) and trusts nothing that arrives from a public address. +# Keywords `loopback`, `private` and `all`, literal addresses, IPv4 CIDR. +# LANTERN_TRUSTED_PROXIES=loopback,private # Raise to `debug` on any service to turn detail up without a code change. # Applies to every service and to Lantern itself. LOG_LEVEL=info @@ -221,6 +336,44 @@ BEACON_PUBLIC_STATUS=false # it simply stops being an outage that an external miner has not shown up. BEACON_CHAIN_EXPECT_BLOCKS=true # +# The EIP-155 chain id every Hearth node here must report over JSON-RPC, in hex +# from eth_chainId and in decimal from net_version. +# +# 7412 is hearth-testnet, 7411 is hearth mainnet, and this is the number that +# stops a transaction signed for one from being valid on the other. A node +# refuses to start on a network it does not recognise, so the drift worth +# monitoring is the one that passes its own validation: HEARTH_CHAIN_ID set +# explicitly to something valid but wrong. Nothing then misroutes and no +# balance reads incorrectly — the loss is silent. Every testnet transaction +# becomes replayable on mainnet, and ForgeKeyvault, which resolves the expected +# id independently, begins refusing withdrawals with 403 binding_mismatch from +# a service that cannot explain why. +# +# Change this only when you change what the nodes are. It is pinned rather than +# read back from the node on purpose: a monitor that adopts the value it is +# meant to be checking asserts nothing at all. +# +# It defaults to HEARTH_CHAIN_ID below, so there is ONE number to change. +BEACON_HEARTH_CHAIN_ID=7412 +# +# The same id, for the browser wallet and the block explorer (`hearth-web`). +# +# It is separate from the node's own HEARTH_NETWORK because the bundle is a set +# of static files with no build step: nginx templates this into a +# `` at boot (repos/hearth/web/nginx.conf), and +# repos/hearth/web/assets/chain.js reads it once for the whole page. The wallet +# then binds every EIP-155 signature to it. +# +# Deliberately NOT taken from whatever eth_chainId the node reports: the wallet +# accepts a `?rpc=` override, so a node that got to choose the chain id +# would get to choose what its visitor's signature is valid on. The reported id +# is compared against this one and the mismatch is shown; it is never adopted. +# +# BEACON_HEARTH_CHAIN_ID falls back to this, so setting it here moves the +# monitor and the wallet together — the drift this replaced was a wallet +# signing for 7411 while every node in the file ran 7412. +HEARTH_CHAIN_ID=7412 +# # One webhook, fired on incident open and close and on nothing else. Plain # JSON with a `text` field, so Slack, Discord and a shell script all work. # An alert per failed probe is an alert nobody reads. @@ -243,6 +396,15 @@ BEACON_SELF_URL= # internet can reach. # NIMBUS_ISSUER=https://nimbus.cloudsforge.online # +# NIMBUS_PUBLIC_URL follows the apex too (compose defaults it to +# https://account.${CLOUDSFORGE_APEX}), so set it only if the account portal is +# reached on some other hostname. It is the origin of every password-reset link +# this estate sends, so a wrong value sends users somewhere that does not answer +# — and it is deliberately not taken from the request, because that let anyone +# who knew an address have this deployment mail them a link to a host of their +# choosing. +# NIMBUS_PUBLIC_URL=https://account.cloudsforge.online +# # Only set the lists below to REPLACE the derived defaults (they win outright — # the apex-derived and localhost origins are then dropped). Both are explicit # lists, never a wildcard: for the hand-back list any one compromised subdomain diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1b3bb6d..f250763 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -8,6 +8,79 @@ on: workflow_dispatch: jobs: + # One malformed log record — a `statusCode` of 1e30, a NUL byte on some + # container's stdout — used to abort Lantern's batch insert and then be + # re-queued, so the writer failed on every tick and the whole estate stopped + # having a durable record of anything. Nothing about that is visible from + # `docker compose config`, so it is asserted here, against a real Postgres: + # the sanitiser is only half the fix, and the other half is a flush() that + # refuses to retry a batch the database will never accept. + # + # The same job also holds the /ingest/client quota to being per *client*: it + # is keyed on the forwarded address, because behind the tunnel every public + # request shares one socket peer, and one shared bucket is one that anybody + # can empty for everybody. That test needs no database and runs here because + # this is where Lantern's suite already runs. + lantern: + name: Lantern's writer cannot be wedged, and its ingest quota is per client + runs-on: ubuntu-latest + services: + postgres: + image: postgres:17-alpine + env: + POSTGRES_USER: cloudsforge + POSTGRES_PASSWORD: cloudsforge + POSTGRES_DB: postgres + ports: [ '5432:5432' ] + options: >- + --health-cmd "pg_isready -U cloudsforge" + --health-interval 5s --health-timeout 5s --health-retries 10 + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: '22' + # npm, not pnpm: infra/lantern is not part of any pnpm workspace, it ships + # a package-lock.json, and its Dockerfile installs with `npm ci`. Using + # anything else here would test a dependency tree the image never has. + - name: Install + working-directory: infra/lantern + run: npm ci + # A database name other than `lantern`: the tests refuse to run against + # the real one, because they add a constraint to `events` to manufacture + # an unwritable batch. + - name: The log pipeline holds + working-directory: infra/lantern + env: + LANTERN_DATABASE_URL: postgres://cloudsforge:cloudsforge@127.0.0.1:5432/lantern_ci + run: npm test + + # Beacon is the thing that notices, so the failures it has to notice are the + # ones asserted here. All of them are shapes a node can be in while its REST + # port answers /info perfectly: the Ethereum JSON-RPC listener never bound + # (it fails soft — evmnode.js logs `json-rpc listen failed` and carries on), + # the chain id is valid but not ours, the id is encoded the wrong way round + # in one of the two methods that must disagree about encoding, or two nodes + # hold different history at a settled height while their tips look like lag. + # + # Against stubs on real HTTP, so this needs no chain and no database. + beacon: + name: Beacon sees a dead JSON-RPC surface and a wrong chain id + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: '22' + # npm for the same reason as Lantern above: infra/beacon is in no pnpm + # workspace, ships a package-lock.json, and its image runs `npm ci`. + - name: Install + working-directory: infra/beacon + run: npm ci + - name: The chain probes hold + working-directory: infra/beacon + run: npm test + compose: name: Compose files are valid runs-on: ubuntu-latest @@ -203,14 +276,53 @@ jobs: echo "ok: nimbus accepts its own token" curl -sf -m 15 http://localhost:4002/worlds -H "authorization: Bearer $at" >/dev/null echo "ok: the game service verified a nimbus token via JWKS" - - name: The chain mines and advances + # Both listeners, because a Hearth node runs two of them and only the REST + # one was ever asked here. `eth_*` on 8545 is the surface forge-pay's + # deposit watcher reads, the one ForgeKeyvault binds its signatures to, + # and the one that fails SOFT: `listenJsonRpc` logs `json-rpc listen + # failed` and the node goes on serving /info with a climbing height and no + # eth_* surface at all. A step that reads only 8645 passes on a stack + # where no balance can be read. + # + # And the chain id, against a literal, in BOTH encodings. This is the only + # place in the estate that pins 7412 to a *deployed* node rather than to a + # stub: `eth_chainId` is a hex QUANTITY and `net_version` is a DECIMAL + # string, both derived from the same integer by different code, so + # checking one proves nothing about the other and swapping them is what + # makes a wallet refuse the network outright. The number is written out + # here rather than read from the compose file for the reason beacon pins + # its own copy: a check that adopts the value it is checking checks + # nothing. Change it here when the network changes. + - name: The chain mines and advances, on both listeners and on chain 7412 run: | set -e + rpc() { + curl -sf -m 10 -X POST "http://localhost:$1/" \ + -H 'content-type: application/json' \ + -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"$2\",\"params\":[]}" \ + | python3 -c 'import json,sys; r=json.load(sys.stdin); e=r.get("error"); sys.exit("JSON-RPC error: %s" % e) if e else print(r["result"])' + } + # seed on 8545, miner1 on 8547, miner2 on 8549 — every node, because a + # node that came up alone on the wrong id is the one that forks. + for port in 8545 8547 8549; do + id=$(rpc $port eth_chainId) + net=$(rpc $port net_version) + [ "$id" = "0x1cf4" ] || { echo "::error::$port: eth_chainId is $id, expected 0x1cf4 (7412)"; exit 1; } + [ "$net" = "7412" ] || { echo "::error::$port: net_version is $net, expected the decimal string 7412"; exit 1; } + echo "ok: $port answers JSON-RPC on chain 7412, hex from eth_chainId and decimal from net_version" + done h1=$(curl -sf -m 10 http://localhost:8645/info | python3 -c 'import json,sys; print(json.load(sys.stdin)["height"])') + r1=$(( $(rpc 8545 eth_blockNumber) )) sleep 60 h2=$(curl -sf -m 10 http://localhost:8645/info | python3 -c 'import json,sys; print(json.load(sys.stdin)["height"])') - echo "height $h1 -> $h2" + r2=$(( $(rpc 8545 eth_blockNumber) )) + echo "REST height $h1 -> $h2 · JSON-RPC height $r1 -> $r2" [ "$h2" -gt "$h1" ] || { echo "::error::the chain did not advance"; exit 1; } + [ "$r2" -gt "$r1" ] || { echo "::error::eth_blockNumber did not advance — the height forge-pay polls for deposits is stuck"; exit 1; } + # The two listeners are the same process reading the same chain + # object, so they cannot legitimately be more than a block apart. + if [ "$r2" -lt "$h2" ]; then skew=$(( h2 - r2 )); else skew=$(( r2 - h2 )); fi + [ "$skew" -le 2 ] || { echo "::error::the listeners disagree: JSON-RPC $r2, REST $h2"; exit 1; } - name: Logs on failure if: failure() run: docker compose -f docker-compose.yml -f docker-compose.ghcr.yml logs --tail 60 diff --git a/BLOCKCHAIN.md b/BLOCKCHAIN.md index c533cba..30c39d4 100644 --- a/BLOCKCHAIN.md +++ b/BLOCKCHAIN.md @@ -161,11 +161,13 @@ unilateral exit if the platform stops cooperating. changes is that their balance is a signed claim they can enforce on-chain without our permission, rather than a number we promise to honour. -**What changes for us.** Withdrawal stops being a treasury problem. Today it is genuinely broken: -keyvault refuses to sign for `purpose:'deposit'` addresses, so payouts need a treasury, the sweeper -that funds it cannot run, and withdrawals are accepted and then refunded an hour later. A channel -close pays the user from the funding output directly. **The sweeper problem disappears rather than -being solved.** +**What changes for us.** Withdrawal stops being a treasury problem at all. It is no longer broken: +keyvault has a `sweep` shape for `purpose:'deposit'` addresses whose destination must be a pinned +treasury, so payouts still route through a treasury but the sweeper that funds it can run — it +ships off by default as a custody decision (see `MAP.md` §7), which is why an unconfigured +deployment still accepts a withdrawal and refunds it an hour later. A channel close pays the user +from the funding output directly. **The treasury stops being on the path rather than being made +to work**, and with it the standing service-token exposure that the sweep design can only bound. **Off-chain still, on purpose:** the game tick, order books, logs, matchmaking, chat. Fast, high volume, no dispute value. Putting them on a 15-second chain would be theatre. @@ -236,7 +238,7 @@ identity without exposing one. | Phase | Work | Why here | | --- | --- | --- | -| **0. Prerequisites** | keyvault gains a signable `treasury` purpose; hearth-node version discipline agreed | unblocks the sweeper today, and the fork needs the pin coordinated | +| **0. Prerequisites** | ~~keyvault gains a signable `treasury` purpose~~ **done** — the `sweep` shape and the `vault_treasuries` pin have shipped; hearth-node version discipline agreed | the sweeper is unblocked, and the fork needs the pin coordinated | | **1. Proof of reserve** | reserve records, inclusion proofs, no-credit-without-backing | cheapest real improvement; closes the live gap; needs no fork | | **2. Consensus upgrade** | locks, multisig, HTLC, tokens, record indexing + conformance across node, browser miner, keyvault | one fork, testnet disposable | | **3. Custody** | 2-of-3 addresses, recovery flows, notified migration | needs multisig | diff --git a/CHAIN-TESTPLAN.md b/CHAIN-TESTPLAN.md index 8adf3f4..d60f216 100644 --- a/CHAIN-TESTPLAN.md +++ b/CHAIN-TESTPLAN.md @@ -29,9 +29,10 @@ keyvault; and the deposit → credit → sweep → maturity accounting holds aga **That does not prove:** anything about a network larger than three cooperative nodes on one host; anything about a node joining from zero against an existing chain; anything about reorgs deeper than the two the p2p fork suite exercises; anything under load or under an adversary; -anything about wallet or explorer compatibility, both of which are currently pointed at the wrong -port (CF-13); or anything at all about the production proof-of-work parameters, which have never -executed anywhere (CF-11). +anything about wallet or explorer compatibility beyond the fact that they now reach the chain at +all (CF-13 and CF-30 landed: the bundle proxies `/eth-rpc/` to 8545 and takes its chain id from +`HEARTH_CHAIN_ID`, and Beacon's `hearth-web.eth-rpc` probes that path end to end); or anything at +all about the production proof-of-work parameters, which have never executed anywhere (CF-11). It also does not prove the chain is safe to expose. Two findings say it is not: every `eth_call` writes into the same process-wide store that holds consensus state (CF-12), and one @@ -201,8 +202,11 @@ explorer shows it. ### 4.9 Wallet MetaMask: add the network, receive, send, watch a confirmation, read a token balance. The browser wallet in `repos/hearth/web`: the same. **Pass:** no manual RPC configuration beyond the -published URL and chain id. **Today this cannot pass** — CF-13 points both the explorer and the -wallet at the REST port and at `localhost`, and CF-30 pins the wrong chain id in the browser. +published URL and chain id. The two defects that made this unpassable are fixed: CF-13 gave the +bundle a second nginx location (`/eth-rpc/` → 8545, `/rpc/` staying on 8645 for `mine.html`), and +CF-30 moved the chain id out of the JavaScript into `HEARTH_CHAIN_ID`, templated into a +`` and read once by `web/assets/chain.js`. What is still untested is +the MetaMask half, which needs a published RPC URL this estate does not have yet (§6). ### 4.10 Adversarial Each of these is a separate exercise with a separate expected refusal. @@ -263,7 +267,7 @@ deposit rule protects. Then set a target and assert only the shortfall moves. - [ ] §4.1–§4.6 and §4.8–§4.10 pass on a multi-node network. - [ ] Rate limiting and a concurrency bound in front of the public RPC. - [ ] An explorer a stranger can use, and a faucet that cannot be drained. -- [ ] CF-39 — Beacon watches 8545, asserts chain id 7412, and asserts the head advances. +- [x] CF-39 — Beacon watches 8545, asserts chain id 7412, and asserts the head advances. - [ ] The RPC URL and chain id are published somewhere users can find, and the docs no longer say the chain does not exist (CF-26). - [ ] A stated policy on what a testnet reset means, published *before* anyone depends on it. @@ -304,8 +308,12 @@ person to sign it off. **What is watched, per level.** L2 and above: head advancing on every node, heads agreeing, peer count, mempool depth, block time distribution, and — from L3 — RPC error rate and latency by method. L3 and above adds the estate's own money signals (CF-23): the sweeper's per-chain block -state and the withdrawal worker's queue age. Beacon is the place these belong, and today it -watches only the REST port (CF-39). +state and the withdrawal worker's queue age. Beacon is the place these belong. It now watches +both ports: `hearth.` reads `/info` on 8645, and `hearth.rpc.` POSTs `eth_chainId`, +`net_version` and `eth_blockNumber` to 8545 and asserts the id in both encodings, with +`hearth.rpc.state` comparing state roots at a settled height and `hearth.rpc.liveness` requiring +the JSON-RPC head to advance and to track the REST head (CF-39). The money signals are still +outstanding (CF-23). **What abort looks like.** diff --git a/DEPOSIT-SWEEP.md b/DEPOSIT-SWEEP.md index 613edf1..b47b531 100644 --- a/DEPOSIT-SWEEP.md +++ b/DEPOSIT-SWEEP.md @@ -6,6 +6,17 @@ Written against source on 2026-07-29. Every claim cites `path:line`; paths are r `repos/` unless noted. Read `findings.md`, `forge-keyvault/MAP.md` §4 and `forge-pay/MAP.md` §7 first — this document assumes them and corrects them in three places. +> **Status: items 1–3 have shipped.** ForgeKeyvault has the `sweep` EVM shape +> (`forge-keyvault/.../signing.ts`, `EvmShape`), the pin lives in `vault_treasuries` and is +> written by `PUT /admin/treasuries/:chain/:network` behind `requireNimbusAdmin` +> (`routes/admin.ts`), with `POST /admin/treasuries/:chain/:network/mint` to mint the +> candidate first. §1 below describes the deadlock **as it stood before that change** and is +> kept as the rationale, not as a current statement of behaviour — sweeping now works for the +> `evm`, `ember` and `xrp` families (`routes/vault.ts`, `SWEEPABLE_FAMILIES`) and remains +> refused for Bitcoin and Solana, which have no output policy yet. `PAY_SWEEP_ENABLED` still +> defaults false, which after this change is item 4's custody decision rather than item 1's +> blocker. + --- ## 0. The recommendation, up front @@ -42,6 +53,10 @@ says what closing it costs. ## 1. The deadlock, precisely +*Past tense as of the status note above: this is the state §0 was written to break, and every +line-cite in it predates the `sweep` shape. It is retained because the reasoning is what makes +the sweeper's default-off setting legible.* + **Keyvault will not sign for a deposit address.** `SIGNABLE_PURPOSES` is `{deployer, treasury}` (`forge-keyvault/services/forge-keyvault/src/routes/vault.ts:42`) and the check fires at `vault.ts:188-193`, returning 403 `purpose_forbidden` before the binding comparison, diff --git a/ECOSYSTEM.md b/ECOSYSTEM.md index 119f228..6087b21 100644 --- a/ECOSYSTEM.md +++ b/ECOSYSTEM.md @@ -316,9 +316,16 @@ if a customer finds it. not, and `ninety-days-after/services/game/src/routes/worlds.ts` has no create path at all. This is the designated headline revenue line. Provision it or pull the SKU. -2. **Cosmetics are localStorage-only** (`apps/game/src/lib/equip.tsx:14-19`), so +2. ~~**Cosmetics are localStorage-only** (`apps/game/src/lib/equip.tsx:14-19`), so nobody else ever sees what anyone bought — which removes the entire reason to - buy one. Fourteen SKUs sell against this. + buy one. Fourteen SKUs sell against this.~~ **Done.** A cosmetic is a + `player_cosmetics` row now: the game service checks it against Forge Pay's + entitlements before writing (a Pay outage is a 503, never "wear it anyway"), + and fans it out to `players.cosmeticStyle` for every world the account is in, + so rosters render it. `equip.tsx` reads the server, not localStorage + (`ninety-days-after/services/game/src/routes/cosmetics.ts`). Not every SKU is + delivered — the kinds with no renderer are a separate item — but the "nobody + sees it" objection is closed. 3. **ForgeMint sells two features that do not exist.** "Verified metadata" on all tiers and "Liquidity-lock helper" on Foundry appear in `MINT_OFFERS` with no implementation anywhere in `services/` or `apps/`. @@ -344,10 +351,27 @@ These are not optional and two of them are load-bearing for the thesis. and it grants: read any balance, debit any user, credit any user, liquidate any user's custodied coin. Fix the ingress, then split the single omnipotent `PAY_SERVICE_TOKEN` into per-service tokens with scopes and caps. -- **EMBER credits at zero confirmations** (`chains.ts:119-127`, - `confirmedIsExact: false`). The mine→spend loop is the ecosystem thesis and - this is the flaw that makes it unsafe. Needs a `hearth/node` change to expose - UTXO heights. **Highest-priority engineering item in this document.** +- ~~**EMBER credits at zero confirmations** (`chains.ts:119-127`, + `confirmedIsExact: false`). Needs a `hearth/node` change to expose UTXO + heights. **Highest-priority engineering item in this document.**~~ **Closed.** + Hearth is an account-model EVM chain now, so there are no UTXO heights to + expose and none are needed: forge-pay reads `eth_getBalance` at + `tip − confirmations` exactly as it does for ETH, and the tip reading has its + own variable that only ever reaches `pending` + (`forge-pay/services/pay/src/chains.ts`). The invariant to hold onto is + **there is no code path that reads at the tip and calls it confirmed**: the + block parameter is not optional in `evmBalanceProbe`. `confirmedIsExact` is + still a field on the probe (`chains.ts:204`) and the credit path still reads + it (`store.ts:1253`) — what changed is that no probe sets it false any more, + so it is a fallback nothing takes rather than a hardcoded lie. EMBER's depth is **60** blocks + (~15 min) in the shared deposit registry — the number + `hearth/docs/exchange-integration.md` §4 publishes to exchanges. **What + replaces it on this list:** depth alone does not secure a young CPU-mined + chain, and the economic controls the registry says "live in forge-pay" — + per-user deposit caps, an automatic halt on any reorg past ~5 blocks — are not + implemented there. That is now the mine→spend loop's open item, and the public + site says so rather than claiming the loop is safe at size + (`platform/apps/site/src/lib/site.ts` `LOOP_CAVEAT`). - **`KEYVAULT_MASTER_SECRET` cannot be rotated** (`crypto.ts:11-13` has no key version). A compromise is unrecoverable without regenerating every address. Key versioning before mainnet custody. diff --git a/MAP.md b/MAP.md index b807fe2..18004ff 100644 --- a/MAP.md +++ b/MAP.md @@ -190,17 +190,38 @@ It is plain JavaScript with no build step, and should stay that way for the same Read this section before deploying. -**Withdrawals are accepted before they can be funded.** `PAY_WITHDRAWALS_ENABLED` defaults to -**true** while `PAY_SWEEP_ENABLED` defaults to **false**. ForgeKeyvault refuses to sign for -`purpose:'deposit'` addresses — which is every address a customer pays into — so payouts must come -from a treasury, and the sweeper that funds it cannot run until keyvault grows a signable treasury -purpose. +**Withdrawals are accepted before they can be funded — until you decide to fund them.** +`PAY_WITHDRAWALS_ENABLED` defaults to **true** while `PAY_SWEEP_ENABLED` defaults to **false**. +Payouts come from a treasury rather than straight out of deposit addresses, and the sweeper that +funds it ships off. That default is a custody decision and not a missing feature: ForgeKeyvault +has a signable `sweep` shape whose destination must be a treasury an operator pinned inside the +vault, so the sweeper *can* run and is simply not switched on. A fresh deployment therefore accepts a withdrawal, holds it `pending` while logging "the treasury cannot cover it", and refunds it automatically once `PAY_WITHDRAWAL_STUCK_MINUTES` elapses (default 60) — safe, because nothing was ever signed. Nobody loses money; they lose an hour -waiting for an answer that could have been given at once. Fund the treasury by hand, or set -`PAY_WITHDRAWALS_ENABLED=false` so the refusal is immediate. +waiting for an answer that could have been given at once. + +Making withdrawals work takes three steps, and only the last two are environment variables: + +1. **Pin a treasury** per chain and network through keyvault's Nimbus admin API — `POST + /admin/treasuries/:chain/:network/mint` to create the address, then `PUT + /admin/treasuries/:chain/:network` to pin it, in that order. Nothing sweeps to an unpinned + candidate. +2. **`PAY_SWEEP_ENABLED=true`.** +3. Optionally **`PAY_TREASURY_TARGET_`** (`PAY_TREASURY_TARGET_EMBER=250`) to hold a float. + It defaults to zero, which leaves the sweeper purely demand-driven. + +Step 3 is the one worth arguing about. A withdrawal must be payable to any address a user names, +so the treasury's destination is unconstrained by design — every coin in the treasury is inside +the blast radius of `KEYVAULT_SERVICE_TOKEN`, and every coin left in a deposit address is outside +it. A float buys immediacy on the first withdrawal after a quiet period, at the price of standing +exposure; zero pays a confirmation depth of latency instead. Set it as a stated product decision. + +This applies to the **EVM, Ember and XRP** families. **Bitcoin and Solana** deposit addresses are +still refused before decryption — neither has an output policy built — so on those two chains the +treasury genuinely must be funded by hand. `DEPOSIT-SWEEP.md` has the full argument; +`repos/forge-pay/services/pay/src/env.ts` is the authoritative commentary. **The testnet chain state is not portable across a `MIN_TARGET` change.** Hearth raised it to lift a ceiling that capped the network at a few hundred CPUs. Replay re-validates targets, so stored diff --git a/ROADMAP.md b/ROADMAP.md deleted file mode 100644 index ec8d74a..0000000 --- a/ROADMAP.md +++ /dev/null @@ -1,1377 +0,0 @@ -# CloudsForge — Audit & Roadmap - -Full-scope audit (~28k LOC, 6 products, 18 areas), with phased improvements per -product. `PLAN.md` describes the intended design; this file records **what is -actually built** and what to do next. - -> **The code no longer lives here.** It is split across ten repositories in the -> [`cloudsforge-online`][org] organisation — nine products plus -> [`stack`][stack], which clones them all and runs the platform. This repository -> is the archive and the record of pre-split history. Paths below like -> `services/pay` now mean `forge-pay/services/pay`; see Phase 0.5 for the map. - -Audit date: 2026-07-27; revised the same day after the P0 pass, the third pass -(Hearth p2p, platform auth, price oracle, container floor) and the fourth (the -repo split, npm publishing, GHCR, and the first end-to-end deploy). Line -references are from the audit commit and have drifted where work has landed since. -Revised again 2026-07-28 for the sixth product, Crucible, and the service-token -billing surface it required in Pay. - -[org]: https://github.com/cloudsforge-online -[stack]: https://github.com/cloudsforge-online/stack - ---- - -## Headline - -The platform is substantially more real than a prototype. The game loop resolves -end to end, the blockchain has genuine Ed25519/UTXO/LWMA-difficulty/reorg code, -and ForgeMint's mainnet guard is properly enforced server-side. There is no -"it's all stubs" problem. - -There are three different problems instead: - -| | Problem | Where | -| --- | --- | --- | -| **A** | Built but unreachable — real server features with no UI | Game, Hearth | -| **B** | Money paths with correctness bugs that lose funds | Pay, ForgeMint, Keyvault | -| **C** | No operational floor — no tests, no CI, no backups, no restarts | Everywhere | - -The single largest finding: **`apps/game` calls only 7 of ~20 game endpoints.** -`/progress`, `/skills`, `/objectives`, `/achievements`, `/leaderboard`, -`/archive` and every `/communes` route have **zero UI**. XP, levels, perks, -objectives, achievements and season archives are all implemented server-side and -invisible to players. The entire retention layer already exists and is -unreachable — this is the cheapest large win available. - -**Status as of 2026-07-27 (third pass): every P0 in all three classes is closed.** - -Second pass closed classes A and B: the progression layer and communes are wired -up, the deposit money path is transactional and confirmation-aware, the token -deploy path ships real compiled contracts, and the keyvault fails closed and -binds every signature. - -Third pass closed the rest. Hearth can now reorg over the network and its web -wallet is real; the platform's session handling no longer puts tokens in URLs and -detects refresh-token theft; `shardsPerCoin` is a live multi-source oracle that -fails closed; and the container floor is multi-stage, non-root and digest-pinned. -The reorg flake is fixed and the suite is a CI gate. - -**The single most valuable finding of this pass:** the placeholder deposit rates -were paying out **1.56–2.0× the coins' real market value** (SOL was exactly -double). Every mainnet deposit would have leaked roughly half its value. - -**Fourth pass (same day): the split shipped and the platform deploys.** Nine -repositories, history preserved; three libraries published to npm; ten container -images published to GHCR; and CI that brings the whole platform up **from the -registry with no source checkouts** and drives a real user journey through it. -Two things that pass caught, both invisible until something actually ran: -`hearth-web` served an empty bind mount on any host that had not cloned the -repo, and **the deployment had `localhost` hardcoded as the JWT issuer** — every -prior verification had been against localhost, so nothing noticed. - -What remains is P1/P2 product depth, plus the items the owner holds: rotating -the leaked OpenAI key, scheduling backups off-host, choosing a mail provider -(password reset / email verification / MFA stay unbuilt until then), and -uploading the org avatar and social previews. The public-vs-private question is -settled: all nine product repositories are public. - ---- - -## Phase 0 — security & ops floor - -**Status: done (this pass), except where marked.** - -- [x] `.env` excluded from Docker build context. It held a live `sk-proj-…` key - and was reaching image layers via `COPY . .` in five single-stage service - Dockerfiles. Verified by build-context assertion; `apps/*/.env.production` - still present for Vite. -- [x] Postgres, keyvault (4005) and all three chain RPC/P2P ports bound to - `127.0.0.1` instead of `0.0.0.0`. -- [x] `restart: unless-stopped` + json-file log rotation on all 15 services (was - zero restart policies — a 3am crash stayed down until noticed, and - unrotated logs would eventually fill the disk and take Postgres with it). -- [x] `infra/backup.sh` — pg_dumpall + every `kv-*` custody volume + - `keyvault-data`. Backup→drop→restore cycle verified end to end. -- [x] Root `.github/workflows/ci.yml` — typecheck, build, hearth unit tests, - secret-hygiene guards. Previously **no CI existed**: `hearth/.github/` is - inert because Actions only discovers workflows at the repo root. -- [x] **All images rebuilt** (third pass) and asserted to contain no `.env` and - no `sk-proj-` material. `docker buildx` was the blocker and is now working - — see Cross-cutting. -- [x] **Keyvault secrets are real.** `.env` still shipped - `KEYVAULT_SERVICE_TOKEN=dev-keyvault-service-token` and a `dev-` master - secret, both of which the second pass's fail-closed check rejects — so the - keyvault *could not boot at all* from the committed env. Fresh 48/64-char - secrets generated; verified the service now starts healthy. Safe to rotate - because `vault_addresses` had **0 rows** — checked before writing. `.env` - was also world-readable (`644`) and is now `600`. -- [ ] **Rotate the leaked OpenAI key** — manual, at - platform.openai.com/api-keys. Owner-held; still not done. The rebuilt - images no longer carry it, but the key itself stays live until rotated. -- [ ] **Schedule `infra/backup.sh`** on the host and send output off-host. A - backup on the same disk is not a backup. Owner-held by choice. - -### Known gap — closed - -The Hearth e2e reorg flake (**~25% of runs**) is fixed. It was never a chain bug: -`test/e2e.js` mined its "fork" with `wallet.keys[0]`, the same key the main chain -used, so the fork's coinbase — and therefore every fork block id — was **identical -to the main branch**, and `addBlock` correctly rejected the first one as `known`. -The test then ignored `res.ok` and crashed with a confusing `TypeError` two lines -later. Fork blocks now mine to their own address, timestamps are `parent+1` so -LWMA can't hand the fork a looser target than the branch it must outweigh, and -every fork block's acceptance is asserted. Measured 6/8 before, **10/10 after**; -now a blocking CI gate alongside the new p2p suite. - ---- - -## Phase 0.5 — the `cloudsforge-online` GitHub org and repo split - -Goal: move off a single personal-account repo (`savvaniss/cloudsforge`) onto an -organisation, with each product in its own repository, **preserving history**. - -**Status: all 8 repositories are live and green**, and the dependency floor they -share is published to npm. What remains is transferring and stripping the -monorepo itself, plus two owner-held items (org avatar and social previews); the -repositories are now public. - -| Repo | Contents | -| --- | --- | -| `hearth` | the chain — node, web wallet, rust, proto, docs | -| `shared-libs` | `@cloudsforge/shared`, `@cloudsforge/ui` | -| `asset-forge` | the asset generator | -| `platform` | `services/nimbus`, `apps/site`, `apps/admin` | -| `forge-keyvault` | `services/forge-keyvault` | -| `forge-pay` | `services/pay` | -| `forge-mint` | `services/forge-mint` + `apps/forge-mint` | -| `ninety-days-after` | `services/game` + `apps/game` | -| `stack` | compose, infra, deploy, cv — clones and runs the eight above | - -Every repo preserves history (per-path `subtree split`, then `subtree add`), and -every one was verified with a **real `docker build`** before the repo was -created — not after. - -### The org is `cloudsforge-online`, not `cloudsforge` - -`github.com/cloudsforge` is taken by an unrelated account. The org is -**`cloudsforge-online`**, matching the `cloudsforge.online` domain. Every repo -URL, badge and canonical link uses that name. - -`admin:org` turned out not to be needed: creating repositories inside an org only -requires `repo` plus org-admin membership, both of which the existing token has. -Publishing packages is what needs a new scope. - -### Decisions taken - -- **8 repositories, one per product**, pairing each backend with its frontend - rather than splitting them apart: `hearth`, `ninety-days-after`, `forge-pay`, - `forge-mint`, `forge-keyvault`, `platform`, `shared-libs`, `asset-forge`. -- **`@cloudsforge/shared` and `@cloudsforge/ui` publish to public npm**, not - GitHub Packages. Cost was not the deciding factor — GitHub Packages is free at - this volume, and container storage on GHCR is free outright. Auth was: GitHub - requires a token to install *public* packages and supports only classic PATs, - which would mean a long-lived credential in seven Dockerfiles and on every dev - machine, permanently. Nothing in either package is private in practice: 37 - files across the client apps already import `shared`, so all of it — contracts, - price tables, game constants — ships in the Vite bundles and is readable in - devtools today. -- **The monorepo transfers to the org and is then stripped** to compose, infra, - deploy and docs. Strip *last*, once all 8 repos are verified: until then it is - the only working copy of the full stack. Keeping the code in it would put two - copies of every file in one org and guarantee drift. - -### The blocker, and why it was worse than it looked - -`packages/shared` had `"main": "src/index.ts"` and services run under `tsx` with -no build step. The original read was that this is merely untidy. It is not: -**`tsx` does not transpile `.ts` inside `node_modules`.** It works today only -because pnpm *symlinks* workspace packages, which sidesteps that rule entirely. -Published as raw TypeScript, all five services would boot to a `SyntaxError` — a -runtime crash on startup, invisible to typecheck and to CI. - -Solved with `publishConfig`, which rewrites `main`/`types`/`exports` in the -packed tarball only: - -``` - in a workspace -> packages/shared/src/index.ts (unchanged, verified) - from npm -> dist/index.js + dist/index.d.ts -``` - -So the dependency floor became publishable **without changing how the running -stack resolves anything**. Monorepo typecheck and build both still pass. - -**`publishConfig` field rewriting is a pnpm feature.** `npm publish` would ship -`main: src/index.ts` and break every consumer — the release workflow uses -`pnpm publish` and says why. - -### Landed - -- [x] **`cloudsforge-online/hearth`** — 4 commits of history, all 77 checks - (28 unit, 24 e2e, 25 p2p-fork) passing from a clean clone with no install - step, Docker image builds, CI green. Its workflow had drifted while inert - in the monorepo: it omitted the p2p suite and pinned Node 20 against a Node - 22 engine. Both fixed, secret-hygiene ported over. - `.gitignore` gained `.netdata/` and `.netlogs/`, which - `scripts/run-local-network.sh` writes and only the *root* `.gitignore` - had been covering. -- [x] **`cloudsforge-online/shared-libs`** — 13 commits, both packages' - history preserved via per-package `subtree split` then `subtree add`. CI - green. It does not stop at typecheck: because these publish through - `publishConfig`, a green typecheck proves nothing about the *published* - artifact, so CI packs both tarballs, asserts they carry `dist/` entry - points and no `src/`, then installs them into a clean project outside the - workspace and imports them. That is the path consumers take and the one - pnpm symlinks would otherwise hide. -- [x] **Brand marks generated** (`gpt-image-2`, which renders manifest sizes - exactly — the 1280×640 social preview needed no cropping, since there is - still no resize stage). `asset-forge` gained a per-asset `style` override: - it had been prepending the game's post-apocalyptic art direction to every - prompt, which is wrong for an org avatar. -- [x] **`cloudsforge-online/asset-forge`** — live and green, after one red first - run: the split dropped `packageManager`, which the monorepo root had been - providing. That is the first line of the per-repo checklist below. -- [x] **First `LICENSE` in the repo** (MIT, matching hearth). Required before - publishing; pnpm packs it into both tarballs automatically. - -### The dependency floor is published - -All three packages are live on npm at `0.1.0`: **`@cloudsforge/shared`**, -**`@cloudsforge/ui`**, **`@cloudsforge/hearth-node`**. The npm org `cloudsforge` -is owned by the `cloudsforge.online` account. - -**Getting there needed one thing nobody planned for: npm's publish auth changed -under us.** `npm login` succeeded and `npm whoami` answered, but the publish was -refused: - -``` -[E403] Two-factor authentication or granular access token with - bypass 2fa enabled is required to publish packages. -``` - -That is npm's November 2025 policy. Legacy tokens were removed outright, and the -`_authToken` a plain `npm login` writes authenticates reads (`whoami`, `view`) -but is refused on `PUT`. **The dead end to avoid is trying to enable 2FA:** npm's -2FA is now WebAuthn-only — a physical security key or a platform passkey — and -the TOTP-authenticator option is gone, so an account without either cannot -enrol. It does not need to. **`Bypass 2FA` is a checkbox at granular-token -creation and is not gated on having 2FA enabled**; it takes precedence over the -account and package settings regardless. Creation is web-only (the npm CLI can -list, IP-restrict and revoke tokens, but explicitly cannot mint them), so for -the record, the token that works is: - -> npmjs.com → Access Tokens → Generate New Token → **Granular** -> - Packages and scopes: **Read and write**, `Only select packages and scopes` -> → `@cloudsforge` -> - **Bypass two-factor authentication: checked** — the setting that matters, -> unchecked by default -> - Organizations: leave at no access. Org access covers teams and settings and -> confers **no** right to publish, so it is not a substitute for the above. -> - Expiration: must be ≥1 day out - -**Verified against the registry, not the workspace** — the whole point of the -`publishConfig` work was that a green workspace proves nothing about the -published artifact. Installed all three into a clean project outside the -monorepo: `shared` and `ui` resolve `./dist/index.js` + `./dist/index.d.ts` and -ship **no `src/`**, `hearth-node` ships `bin`/`src` as intended, and all three -import at runtime — 49, 11 and 10 exports respectively. That is the path -consumers take and the one pnpm symlinks would otherwise hide. - -**Next, adopt trusted publishing and drop the long-lived token.** npm supports -OIDC publishing from GitHub Actions: no `NPM_TOKEN` in org secrets, and -short-lived credentials that cannot be extracted or reused. Needs npm ≥11.5.1, -Node ≥22.14.0 and `permissions: id-token: write`. It could not have been the -first step — the trusted publisher is configured on a package's settings page, -so the package must exist first — but that precondition is now met. Free -provenance attestations additionally require the repo **and** the npm package to -be public, and both now are. - -Also owner-held, not blocking: - -- [ ] **`NPM_TOKEN`** in the org secrets, for the release workflow — or skip it - and go straight to trusted publishing, which needs no secret at all. -- [ ] **Org avatar and repo social previews.** Confirmed against the REST docs: - `PATCH /orgs/{org}` has **no** avatar parameter (`avatar_url` is - response-only), and repo social previews have no endpoint either. Both - uploads are genuinely web-only. The assets are generated and correctly - sized: - - `org-avatar.png` — 1024×1024 → github.com/organizations/cloudsforge-online/settings/profile - - `hearth/branding/social-hearth.png` — 1280×640 → the hearth repo's - Settings → Social preview - - The fields that *are* settable have been set: the org now reads - **CloudsForge**, with a description and `https://cloudsforge.online`. - Until the avatar is uploaded the org still shows GitHub's generated - identicon, which is why it looks unbranded. -- [x] **Decide public vs private. All nine product repositories are public.** - They were created private; `hearth` forced the question, because its Pages - deploy fails until it is public — free-plan orgs get Pages only on public - repos. The door does not open the other way, so treat every commit already - pushed as published: anything secret that ever reached a tracked file is - readable in the history by anyone, and rotating it is the only remedy. - Note that this changed nothing about the GHCR packages — package - visibility is set per package and inherits nothing from the repository. - -### Remaining - -1. ~~Publish `shared` + `ui` + `hearth-node` `0.1.0`.~~ **Done** — all three live - on npm and verified from a clean project outside the workspace. -2. ~~Point `services/forge-keyvault/src/chains.ts` at - `@cloudsforge/hearth-node/crypto`.~~ **Done** — the last cross-repo path - traversal in the codebase is gone; see below. -3. ~~`platform` → `forge-keyvault` → `forge-pay` → `forge-mint` → - `ninety-days-after`.~~ **Done, in that order.** The ordering paid off: each - repo's `pnpm install` and image build re-proved the published packages - against a real consumer before the next one depended on them. -4. ~~Rewire `docker-compose.yml` and strip the monorepo.~~ **Superseded and - done differently.** Rather than stripping the monorepo in place, the stack - now lives in a **ninth repository, `cloudsforge-online/stack`**, which holds - no product code at all: `scripts/clone-all.sh` clones the eight product repos - into `repos/`, and `docker-compose.yml` builds every image from those - checkouts. `docker-compose.ghcr.yml` is the deploy path — prebuilt images, - no checkouts needed. - - This is better than stripping: the monorepo stays intact as the archive - (and as the record of pre-split history), while the stack repo is a clean - thing that never contained the code it orchestrates. **The monorepo transfer - is now optional** and can happen whenever, since nothing depends on it. - - `apps/hearth-site` moved into `cloudsforge-online/hearth` as `site/` — it had - no home after the split. It is deliberately a *self-contained* pnpm project - rather than a workspace member, because `node/` and `web/` have no install - step and should not acquire one just so the site can use vite. - -**Verified end to end, twice** — once against the monorepo build and once -against the stack repo building from the eight separate checkouts: - -- register → RS256 JWT → **cross-service verification** (the game service - validating nimbus's token against its published JWKS) -- admin creates and starts a world → player joins → queues `work`/`rest`/ - `fortify` → forced tick → food 8→12, defense 0→2, XP 0→21, three reports - written. The whole simulation, in one transaction, from separately-built - images. -- price oracle pulling **live** market quotes (BTC $64,960, ETH $1,943) with the - administered EMBER rate alongside -- forge-keyvault minting an ember address through the *published* - `@cloudsforge/hearth-node`, and the running chain recognising it -- the three-node testnet mining and advancing, all nodes agreeing on the tip - -### Images publish to GHCR - -All six image-producing repos publish on push to `main` — 10 images total -(`nimbus`, `site`, `admin`, `game`, `game-client`, `pay`, `forge-mint`, -`forge-keyvault`, `hearth-node`, `hearth-site`), tagged both `main` and -`sha-`. `GITHUB_TOKEN` is sufficient; no PAT and no org secret, which is -one fewer long-lived credential than the npm side needs. - -**The deploy path is verified.** The packages are public, and the stack repo's -CI now brings the **entire platform up from `ghcr.io` with no product checkouts -present** and drives it: all 11 HTTP surfaces answer, a user registers and gets -an RS256 token, the game service verifies that token against nimbus's JWKS over -the network, the chain advances (7 → 13 in a minute), and the chain image mints -a valid `ember1…` address. If that job ever needs `repos/`, the overlay has -stopped overriding something and a service is quietly building from source. - -Getting there hit **two different permission walls that present identically** — -`docker login ghcr.io` *succeeds* and the pull then fails `403` / `denied` on the -manifest, which reads as a missing image rather than an unreadable one: - -1. **Locally the token lacked a scope.** Org ownership is not the constraint — - the token is. A browser login mints a `gho_` OAuth token carrying only the - scopes approved at that moment (`gist, read:org, repo, workflow`); GHCR wants - **`read:packages`**. `gh auth refresh -s read:packages` adds it. -2. **In CI the *repository* lacked access.** A GHCR package is readable only by - the repo that **published** it, so the `stack` repo's `GITHUB_TOKEN` was - denied despite carrying `packages: read`. - -Both were resolved by making the packages public, which needed an **org-level** -setting first: Settings → Packages → **Package Creation → Public**. Until that -is on, every package's own *Danger Zone → Change visibility* is greyed out with -"Setting is disabled by organization administrators", which does not hint at -where the real control lives. **Making a package public is one-way.** - -None of this follows the repository. A container package carries its own -visibility, so the repos going public later changed nothing here, and a package -published for the first time after that does not inherit it either — it has to -be flipped by hand like the rest. Assume a new image is unreadable until you -have pulled it from somewhere holding no credentials. - -### Two bugs the first end-to-end deploy run caught - -Both were invisible to every other check, which is the argument for the job: - -- **`hearth-web` served an empty directory.** It was a stock nginx with - `repos/hearth/web` bind-mounted, so the "no checkouts needed" claim was false - for it: deploying from images alone mounted an empty dir and nginx returned - 403. It now has its own published image and is *built*, not mounted. -- **`volumes: []` in an override does not clear the base list.** Compose - **appends** sequences on merge, so the first fix looked right and changed - nothing — the mount survived and kept shadowing the pulled image. Removing the - mount from the base file is the only thing that works. -- Images are tagged `main` and `sha-`; **nothing publishes `:latest`**, - so an untagged reference fails with `manifest unknown`. - -**Images are `linux/amd64` only** (GitHub-hosted runners). They run on an arm64 -Mac under emulation with `--platform linux/amd64`. Add -`platforms: linux/amd64,linux/arm64` to the publish workflows if native arm64 -matters. - -### The deployment was not actually deployable until now - -`NIMBUS_ISSUER`, `CORS_ORIGINS` and `PORTAL_ALLOWED_ORIGINS` were **hardcoded to -localhost** in compose. Everything was verified against localhost, so nothing -caught it — but behind the tunnel it breaks authentication in a way that -presents as a frontend bug. The issuer is baked into every JWT and verified by -every service, so real users would have received tokens whose issuer nothing on -the internet can reach, and CORS would have refused the browser before it got -that far. All three are env-driven now, defaulting to exactly the previous -localhost values, so local development is unchanged. - -`stack/README.md` carries the deploy procedure. The four things that stop it -booting or make it unsafe: `POSTGRES_PASSWORD` (must be set *before* first boot -— changing it later does not change an initialised volume), the two keyvault -secrets (≥24 chars, no placeholders, and `KEYVAULT_MASTER_SECRET` is -unrotatable), and **`PAY_PROVIDER`, which ships as `mock` and exposes -`/invoices/:id/mock-pay` — free Shards for any signed-in user.** - -### Traps found bringing the stack up - -- **A container that fails its network setup stays in a restart loop *without a - network*.** A port collision on 4001 left nimbus retrying forever against - `EAI_AGAIN postgres`, which reads exactly like a DNS bug and is not one. DNS - on the compose network was fine the whole time. `docker compose rm -sf - ` and recreate; `up -d` alone will not repair it. -- **A bind mount from an unshared host path is silently EMPTY, not an error.** - Run the stack from `/tmp` on macOS and `infra/pg-init` mounts as an empty - directory: Postgres logs `ignoring /docker-entrypoint-initdb.d/*`, initialises - with no databases, and every service then fails with `3D000 database does not - exist`. The same silence would have hollowed out the `hearth/web` and `cv` - mounts. Keep the stack repo under a Docker-shared path. - -Two things the split surfaced, both now fixed in every repo: - -- **pnpm's new-package quarantine is the `allowBuilds` trap in a new costume.** - Depending on a package published minutes earlier makes `pnpm install` write a - `minimumReleaseAgeExclude` block into `pnpm-workspace.yaml`, which then reads - back as a changed setting and fails the *next* command with - `ERR_PNPM_VERIFY_DEPS_BEFORE_RUN`. Commit the block pnpm writes and re-run - install; do not try to suppress it. -- **`.env.production` in the app packages was dead too**, not just the `ARG - VITE_*` blocks. Nothing reads `import.meta.env.VITE_*`, so those files encoded - production hostnames that were never consumed — deleted rather than ported, - since keeping them implies host selection happens at build time when it does - not. (`import.meta.env.DEV` in `apps/forge-mint` is Vite's own flag, not a - `VITE_*` var, and stays.) - -Per-repo checklist, learned the hard way on the three that are done: - -- `packageManager` **and** `engines` in `package.json` — `pnpm/action-setup` - fails with "No pnpm version is specified" without the first. `asset-forge` - lost it in the split and went red on its first run. -- `pnpm-workspace.yaml` with an explicit `allowBuilds` block, even in a - single-package repo. Without it the first `pnpm install` rewrites the file - with a `set this to true or false` placeholder that fails every later command. -- Copy `tsconfig.base.json` in — seven packages `extends` it. -- Copy `.dockerignore` — the secret-hygiene CI job asserts its `.env` entries. -- Delete the `ARG VITE_*` blocks from app Dockerfiles. No app reads - `import.meta.env.VITE_*`; hosts resolve at runtime via `cloudsforgeHosts()`. -- Verify with a real `docker build` before creating the repo, not after. - -### Landmines found while planning the remaining five - -- **Every service's `env.ts` hardcodes `resolve(…, '../../..')`** to find the - root `.env`. Post-split that resolves to nothing, dotenv loads no file, and the - service runs on defaults with `databaseUrl` undefined. Silent, not loud. -- ~~**`services/forge-keyvault/src/chains.ts:31`** `require()`s - `hearth/node/src/crypto.js` by repo-relative path.~~ **Fixed.** It now requires - `@cloudsforge/hearth-node/crypto`, so hearth stays a producer and never becomes - a consumer, and the `COPY hearth/node` lines are out of the Dockerfile. - `createRequire` is still the loader — the package is CommonJS, the service is - ESM — but the specifier is a package name now, not a path traversal. - **Verified where it actually fails, which is at runtime**: minted an ember key - and checked the address against the *chain's own in-repo* `isValidAddress` - (so drift between published and in-repo crypto would surface), then rebuilt the - image and repeated it inside the container, confirming `/repo/hearth` is gone - and the address still validates. Typecheck would have passed either way. - One side effect worth knowing: pnpm's new-package quarantine blocked the - install until it wrote `minimumReleaseAgeExclude` into `pnpm-workspace.yaml`, - which then tripped `verifyDepsBeforeRun` with "the value of the - minimumReleaseAgeExclude setting has changed" — the same - workspace-file-rewrite trap documented below, in a new costume. A second - `pnpm install` settles it. -- **`pnpm-workspace.yaml` must be copied to every repo with its `allowBuilds` - block.** This is not theoretical: the first `pnpm install` in `shared-libs` - rewrote a minimal workspace file mid-install with a - `esbuild: set this to true or false` placeholder, which then read back as a - changed setting and failed every subsequent command — exactly the trap the - monorepo's own workspace file documents. -- **`VITE_*` build args are dead code.** No app reads `import.meta.env.VITE_*`; - hosts resolve at runtime via `cloudsforgeHosts()`. Delete the `ARG`/`ENV` - blocks rather than porting them. -- **`asset-forge` wrote into four other packages** by design. Fixed: output now - resolves against a configurable `ASSET_OUT_ROOT`, defaulting to the current - behaviour. - -### Cost accepted - -The monorepo allowed atomic cross-cutting changes. The third audit pass was **one -commit touching 45 files across 8 components** — post-split that is eight -coordinated PRs plus a package release, and a version-skew failure mode that does -not exist today. CI becomes N pipelines instead of one. Both packages sit at -`0.x`, where `^0.1.0` means patch-only, which is the intended safety while the -contracts still move. `shared` is the wire contract between services and clients, -so a field rename breaks running deployments, not just builds. - ---- - -## Phase 0.8 — observability: making failures visible - -Goal: when something breaks, find out **what** broke, **where**, and **why**, -without SSH-ing to a host and reading fifteen separate `docker logs` streams. - -Prompted by the plain observation that the platform is hard to debug. It is, and -the audit below says why: the stack has *almost no error reporting at all*. Not -"the wrong kind" — none. Every one of the five services is -`Fastify({ logger: true })` and nothing else, the two most common failure modes -(bad auth, dead database) are `catch {}`, and no frontend has an error boundary, -so a render throw is a white page with nothing written down anywhere. - -### What the audit found - -**1. Logging is one word of configuration, repeated five times.** Every service -is `Fastify({ logger: true })` — nimbus `src/index.ts:47`, game `:15`, pay `:16`, -forge-mint `:13`, keyvault `:10`. That gives pino JSON on stdout, which is a -real beachhead, but with pino's bare defaults: - -- Level is `info`, hardcoded. `grep -r LOG_LEVEL` across all nine repos: **zero - hits.** There is no way to turn detail up on a running service, which is - exactly what you want at the moment you need it. -- No `base` fields — nothing in a line says which service emitted it. Fifteen - containers, and their logs are only distinguishable by which terminal you ran - `docker logs` in. -- No `redact`. Nothing leaks *today* because Fastify's default serializer drops - headers, but that is one config change away from writing `authorization` and - `x-service-token` to disk — in a service that custodies private keys. -- No `genReqId`, and **Fastify 5 defaults `requestIdHeader` to `false`**, so an - inbound `x-request-id` is ignored. Ids are a per-process counter (`req-1`, - `req-2`) that resets on restart. A request crossing site → nimbus → pay → - keyvault produces four unrelated counters, so a user-reported failure cannot - be traced through the stack even in principle. -- `NODE_ENV` is set in **no** Dockerfile, so everything runs in dev-ish defaults. - -**2. The two most common failures in production are silent by construction.** - -- `nimbus/src/tokens.ts:119`, `pay/src/auth.ts:29`, `game/src/auth.ts:28` are - each a bare `catch {}` around token verification. Expired token, bad - signature, wrong `NIMBUS_ISSUER`, **and Nimbus being down** all collapse to - one unlogged `401 invalid token`. A total auth outage is indistinguishable - from every user simultaneously typing their password wrong. This alone would - explain most of "we can't identify issues": the highest-frequency - cross-service dependency in the platform reports nothing when it fails, and - the localhost-issuer bug the last pass found (Phase 0.5) was invisible for - exactly this reason. -- `game/src/engine/tick.ts:44` is - `setInterval(() => { void tickDueWorlds() }, 30_000)`. `tickDueWorlds` opens - with an unguarded `db.select()`. If Postgres is unavailable the promise - rejects, `void` discards it, and **the world silently stops advancing** with - no output whatsoever. The game just quietly stops being a game. - -**3. No service has a `setErrorHandler`.** Zero hits across all five. So -Fastify's default handler runs, and it puts `err.message` in the response body: - -- Keyvault returns raw postgres and dockerode messages to callers - (`routes/vault.ts:149-150`), which means **connection strings including host - and user leak to clients** on a DB blip. -- The admin console reads `body.error`, which in Fastify's default 500 shape is - the literal string `"Internal Server Error"` — the useful text is in - `message`, which nothing reads. Every server error looks identical in the UI. -- 404s return `{message,error,statusCode}`, a *different* shape from the app's - own `{error,code}`, so clients parse two formats and get one of them wrong. - -**4. Nothing handles process-level failure.** No `unhandledRejection`, no -`uncaughtException`, no `SIGTERM`, in any service. Startup errors bypass pino -entirely: nimbus `index.ts:89` and game `index.ts:34` are `console.error`, and -pay/keyvault/forge-mint use top-level `await`, so a migration failure prints a -raw V8 stack that no log parser will ever pick up. The single most important -line a service ever writes — "I could not start, and here is why" — is the one -least likely to survive collection. - -**5. Money paths fail without a word.** The worst of them: - -- `forge-mint/src/routes/tokens.ts:274` — token deploy failure, *possibly after - the transaction was broadcast and real funds were spent*, sets status `failed` - and logs **nothing**. -- `forge-keyvault/src/vault/docker.ts:80` — a bare `catch {}` around vault - provisioning that **silently degrades key custody from Docker volumes to file - storage**. The one line you would want in an audit log is the one that is - discarded. Worse, `exec()` at `:51` never checks the exit code, so a failed - write still returns success — a full disk yields an empty `key.enc` and a key - that is permanently unrecoverable. -- `pay/src/routes/webhooks.ts:10` — an HMAC mismatch returns 401 with no log, so - a rotated `NOWPAYMENTS_IPN_SECRET` silently drops **every** payment. -- `pay/src/watcher.ts:67` — chain probe failures log at `debug`, i.e. below the - hardcoded `info` level, i.e. never. A dead RPC produces no output for a week. - -**6. Outbound calls have no timeouts.** mint → pay, mint → keyvault, pay → -keyvault, and every chain RPC call use bare `fetch`. Only `pay/src/pricing.ts:45` -sets `AbortSignal.timeout`. A hung dependency pins a promise forever rather than -failing loudly — the failure mode that looks like "the app is just slow" and has -no error to find. - -**7. No frontend can report anything.** No error boundary in any of the four -React apps, so a render throw is a blank white page. No `window.onerror`, no -`unhandledrejection` listener. **Browser-side failures currently produce no -record anywhere in the system** — if it breaks for a user and they don't screen- -shot it, it did not happen. Beyond that: `admin/src/lib/api.ts:102` calls -`JSON.parse` unguarded, so an HTML 502 from the tunnel renders to the user as -*"Unexpected token '<'"*; `admin/src/lib/auth.tsx:60` invokes `restore()` with -no `.catch`, so an unreachable Nimbus is an unhandled rejection and a silent -redirect loop; and mutation errors in the game client are toasts that vanish -after ~3 seconds with no way to recover the text. - -**8. There is no aggregator, and no client-side sink.** Fifteen containers, all -logging to `docker logs` with json-file rotation (Phase 0 put the rotation -there). Hearth's own compose has no rotation at all. Every frontend nginx emits -the default `combined` text format, which is the one non-JSON stream. The chain -node prints `[hearthd] ` with no timestamp and no level. -Diagnosis today means fifteen SSH tails and reading by eye. - -### The plan - -Two halves. First make every component **say** something useful; then put -something in front of it that **listens**. - -**A. `obs.ts`, vendored per service — deliberately not in `@cloudsforge/shared`.** -The shared packages publish to public npm, so putting the logger there makes a -logging fix a version bump, a publish, and eight dependency bumps before one -line reaches production. That is the wrong shape for the component you reach for -when things are already broken. It is ~150 lines that change rarely; copy it. -Each service gets: - -- pino config with `LOG_LEVEL` (default `info`), a `service` base field, a - `redact` list covering `authorization`, `x-service-token`, cookies and - password fields, and `err` serialization that keeps the stack. -- `genReqId` that **honours an inbound `x-request-id`** and mints a ULID-ish one - otherwise, plus an `onSend` hook echoing it back as a header. Every error body - grows a `requestId` field. That is the thread that stitches the four services - together, and the string a user can quote. -- `setErrorHandler` — logs the full error with request context, returns the - house `{error, code, requestId}` shape, and returns a *generic* message on - 5xx so postgres DSNs stop reaching clients. `setNotFoundHandler` for shape - consistency. -- `installProcessHandlers()` — `unhandledRejection`, `uncaughtException`, and - `SIGTERM`/`SIGINT` for a clean close, all through pino so they survive - collection. -- `bootstrap()` — wraps startup so "migration failed" is a structured line - naming the step, not a bare stack. -- `fetchJson()` with a default timeout, which logs status, latency and the - response body on failure. Replaces the bare `fetch` calls in §6. - -**B. Fix the specific swallows in §2, §5 and §6.** Not a sweep — the named -lines, each turned into a real log with context. - -**C. `obs.ts` for the frontends, same reasoning.** An `` around -each app with a real fallback (message, request id, reload), `window.onerror` -and `unhandledrejection` listeners, and an `ApiError` that distinguishes network -from 4xx from 5xx and carries the server's `requestId`. All of it reports to: - -**D. Lantern — `infra/lantern`, the aggregator.** A new container that: - -- **Collects** by tailing every container's stdout over the Docker socket - (read-only), rather than asking fifteen services to push somewhere. Nothing - needs reconfiguring to be collected, a crashing service's last words are still - captured, and there is no agent to install. It follows the same socket the - keyvault already mounts. -- **Parses** the four formats present: pino JSON, nginx (switched to a JSON - `log_format`), `[hearthd] …`, and postgres. Normalises to one record. -- **Stores** in Postgres — already running, already backed up, already healthy. - One more database, one more line in `infra/pg-init`. -- **Fingerprints** errors by service + normalised message + top stack frame, so - the UI shows *"this failure, 1,240 times, first seen 09:12"* rather than - 1,240 lines. This is the part that turns logs into issues. -- **Ingests browser errors** at `POST /ingest/client`, giving client-side - failures a home for the first time. -- **Serves a UI** at `:4010` — live tail with filters, an issues view ranked by - volume and recency, and search by `requestId` to follow one user's request - across every service it touched. - -It lives in `stack/` rather than a tenth repository on purpose: a server clones -only `stack`, so an aggregator that ships with it works on the deploy path with -no extra checkout and no GHCR image to pull. It is infrastructure, like -`infra/backup.sh`, not product. - -### Landed - -All nine repositories, verified against the stack actually running — sixteen -containers, all healthy. - -- [x] **Lantern** (`infra/lantern`, `:4010`), collecting the other fifteen - containers. It skips itself deliberately — a collector that ingests its - own output feeds any logging fault straight back into itself — which does - mean **Lantern's own failures are the one thing Lantern will not show - you**; they are in `docker logs cloudsforge-lantern`, in the same pino - shape as everything else. - Sign-in is the ordinary Nimbus one, `admin` role enforced on the token, - with `LANTERN_TOKEN` as break-glass. It creates its own database on boot, - because `infra/pg-init` only runs on a fresh volume and every existing - deployment already has one. -- [x] **`obs.ts` vendored into all five services** — `LOG_LEVEL`, a `service` - field on every line, redaction, one error shape, process-level handlers, - named startup steps, and `fetchJson` with a deadline. -- [x] **Request ids that cross services.** Verified: one `x-request-id` sent to - game, nimbus and pay comes back from all three and retrieves the whole - trace in order from Lantern's search. -- [x] **The auth-outage fix, verified by stopping nimbus.** game and pay both - answer `503 auth_unavailable` and log `cannot verify tokens — nimbus - unreachable`, where before both returned a silent 401 indistinguishable - from a bad password. The classification allowlist is written the - counter-intuitive way round on purpose: a refused connection arrives - carrying `ECONNREFUSED`, so "has an error code" does not mean "is the - caller's fault". `ERR_JWT_EXPIRED` stays a 401 — the browser's silent - refresh is triggered by that status. -- [x] **Error boundaries and global handlers in all four React apps**, plus a - plain-JS equivalent for the Hearth explorer. Browser errors reach - `/ingest/client` and group like any other. -- [x] **JSON `log_format` in every nginx**, structured output from `hearthd`. -- [x] **`exposedHeaders: ['x-request-id']`** on all five CORS configs — without - it the browser can see the header but not read it, which loses the id in - exactly the place a user would quote it back. -- [x] **Lantern in the product switcher**, admin-only, shipped as - `@cloudsforge/ui@0.1.1`. The bar reads `account.roles`; a surface that - does not pass them simply keeps the old six entries, so nothing else had - to change. Verified against the published artifact rather than the - source: non-admins resolve six products, admins seven. Hiding is not the - security boundary — Lantern checks the `admin` role on the token itself. -- [ ] **The `NPM_TOKEN` secret on `shared-libs` is dead.** The release workflow - builds and then fails to upload, so 0.1.1 was published from a developer - machine instead. Owner-held; until it is replaced, every shared-libs - release needs a human at a terminal. See the trap below — the error does - not say what it means. - -Three defects in the canonical `obs.ts` were found only because five services -adopted it at once, which is the argument for vendoring one copy rather than -five hand-written ones: `setErrorHandler`'s generic defaults to `unknown` since -Fastify 5.10 so it did not compile at all; `redact` paths are exact key matches, -so `*.apiKey` never matched `nowApiKey` and the list read as more protection -than it gave; and hosted RPC providers put the API key in the URL *path*, where -redaction cannot reach, so `fetchJson` was logging credentials — URLs now go -through `safeUrl()`. - -One more, and the reason this matters more than it used to: **postgres puts the -whole connection string in its error message**, so a single `ECONNREFUSED` wrote -the database password to the log. That was survivable when logs were fifteen -separate `docker logs` streams; it is not, now that one service aggregates and -stores them. Error messages and stacks are scrubbed before serialization. - -### Found along the way - -- **npm reports a bad publish token as `404 Not Found`, not `401`.** The - release of `@cloudsforge/ui@0.1.1` failed with: - - E404 404 Not Found - PUT https://registry.npmjs.org/@cloudsforge%2fui - - which reads as *the package does not exist* — it does, `0.1.0` was live at - the time. npm deliberately answers 404 rather than 401 on writes so that an - outsider cannot enumerate private packages by watching status codes. **On a - `PUT`, 404 means "this token may not do that."** Do not go looking for a - missing package or a broken `publishConfig`; check the credential. - Cost this pass: the build stage passes cleanly, so the failure looks like a - registry problem rather than a secrets problem, twice over. -- **A second Hearth test flake, ~1 run in 256.** `node/test/unit.js:75` tampers - a signature with `.replace(/^../, '00')` to assert it then fails - verification — which is a no-op whenever the signature already begins `00`, - and the "tampered" signature verifies. Reproduced on a clean tree. Not fixed - here; the fix is to flip the bytes to a value guaranteed to differ rather than - to a constant. -- **A leaked SSE client in the Hearth RPC.** A disconnected subscriber was never - removed from `sseClients` (`node/src/rpc.js`), so the set grew for the life of - the process. Fixed in passing — it was found only because adding a log line to - the disconnect path meant reading it. - -**Deliberately not Grafana + Loki + Promtail.** Three containers and a query -language to answer "what is broken right now", no error grouping without hand- -written rules, and no path for browser errors without a fourth component. The -bespoke version is smaller than its own compose block would be. - ---- - -## Ninety Days After (`apps/game`, `services/game`) - -Real: all six action types resolve (`engine/resolve.ts`), finite per-tile ruin -stock with contention splitting, deterministic event scheduler, XP/levels/perks, -objectives, achievements, starvation death, archive-on-season-end — all in one -transaction. - -**P0** -- [x] Wired the progression endpoints into the client — `pages/Progress.tsx` - (level/XP, objectives + claim, skill tree, achievements), - `pages/Leaderboard.tsx`, `pages/Archive.tsx`, with entry points from - WorldView and a "Past seasons" section in WorldList. Every endpoint these - screens call was verified live against a running stack. -- [x] `GET /worlds` now takes an opt-in `?status=` filter so finished seasons are - discoverable; the default (lobby + active) is unchanged. -- [x] Communes are a real system. Added `GET /communes/:cid` (commune + members + - my allowance), `withdraw` and `leave`, plus two bugs found on the way: - founding a second commune silently abandoned the first, and anyone could - deposit into any commune. Client page `pages/Communes.tsx`, linked from - WorldView. - **Governance**: withdrawing spends a `commune_credit` balance that only - depositing refills — +1 per unit given, *not* per request. The first cut - counted requests, which let a member deposit one wood a hundred times to - buy a hundred units of daily allowance; caught in review. Draws are capped - at `COMMUNE_DAILY_DRAW` per day, with `COMMUNE_JOIN_STIPEND` granted once - on joining. Leaving forfeits the balance; a departing founder hands the - commune to the longest-tenured member, and the last member out disbands it. -- [x] Fixed new-player slaughter. Added `players.joined_day` + - `SPAWN_PROTECTION_DAYS`, enforced in both the bot path and the human raid - path in `resolve.ts` — a griefer should not be able to do what bots can't. - Raider targeting is now a seeded weighted draw over distance and defense - instead of a global argmin, reusing the `seededRng` the deterministic event - scheduler already uses. Protection is human-only: applying it to bots left - established humans as the only legal targets and made the dogpile worse. - -**P1** -- Communes still never touch `resolve.ts` — no upkeep draw, no elections, no - exile, no politics. The stockpile is two-way now but inert in the simulation. -- Fuel has no sink (only mapgen seeding and steal loops). One of the three - headline "finite scarce resources" does nothing. -- Trade is non-consensual fire-and-forget (`resolve.ts:363`) — no offer/accept, - no market. -- Raiding is score-free once reputation hits 0 (`survivalScore` uses - `Math.max(0, reputation)`). -- Retention: no notifications of any kind, no live refresh (the "next report" - countdown is static until manual reload), no onboarding, cosmetics are - `localStorage`-only so nobody else ever sees them. - -**P2** -- Bots are ~45 stateless lines — no memory, no reaction to being raided. -- No season rollover; a new world must be created manually. -- Rented private worlds are never provisioned (`pay/routes/monetization.ts:99` - says "out of scope"). - ---- - -## Hearth / EMBER (`hearth/*`, `apps/hearth-site`) - -Real and well-built: non-outsourceable PoW binding, branch-aware LWMA difficulty, -median-time-past timestamps, most-cumulative-work fork choice with genuine reorg, -UTXO + Ed25519, integer emission with a Commons split. - -**P0 — both fixed this pass.** -- [x] **The web wallet is real.** `hearth/web/wallet.html` was rewritten: real - Ed25519 keygen in the tab, checksummed `ember1` address, balance and UTXOs - read live from `GET /address/:addr`, and transactions built, signed and - broadcast to `POST /tx`. Signing uses a vendored, **byte-identical** copy of - `@noble/ed25519@2.3.0` (verified against the upstream npm tarball) because - WebCrypto Ed25519 support is uneven and this is a no-build-step static site. - Keys export as PKCS#8 PEM, the same format `hearthd` uses, so they move - between browser and CLI. The fake mining timer is **removed** rather than - reimplemented — browser mining does not exist here and adding a second - overclaim to fix the first would be worse. Balance now splits spendable vs - maturing coinbase, which needed one additive read-only RPC change - (`rpc.js:_address`); without it a wallet overstates its balance and gets - sends rejected. Verified end to end against a live chain: a browser-signed - tx confirmed in block #41, a second in #62, and the node's own modules - independently accept the browser's signature and canonical bytes. -- [x] **Fork sync works over the network.** The P0 line - (`if (b.header.height <= chain.height) break`) is gone, and the protocol - gained real locator negotiation: `hello` carries `tip` so sync triggers on - any *unknown* tip rather than only a taller one, `getblocks` takes an - exponentially-spaced locator (≤32 hashes, walked back from the heaviest - stored branch) and is answered from the newest hash on the responder's own - active chain, and a new `getblock` serves a single block by hash so a **side - branch is fetchable at all**. Parentless blocks go to a bounded orphan pool - (32) and trigger a targeted parent fetch, draining transitively once the - ancestor lands. A periodic re-sync converges equal-height competing tips - instead of leaving them split forever. - **Consensus-order fix required by this:** `chain.js` now checks txs, - difficulty target and PoW *before* calling `_stateAt` on the fork path — - otherwise accepting gossiped blocks buys any remote peer a full - genesis-to-tip UTXO replay for free. - Covered by a new `test/p2p-fork.js` (25 checks) driving **two real nodes - over real sockets**: partition, compete, reconnect, reorg, side-branch - fetch, orphan drain, malformed-message and locator-cap guards. It fails - against the pre-fix code and is a blocking CI step. The pre-existing - `test/e2e.js` never called `start()`, so p2p and RPC ran in **no test at - all** — which is why this shipped. - -**P1** -- No peer discovery — hardcoded peers only, star topology on the seed. Kill the - seed and the miners partition forever. -- DoS: `_stateAt` replays the full UTXO set from genesis per fork block; mempool - `add()` is O(n²). Both are free remote CPU exhaustion. **Reduced, not removed**: - a fork block must now carry correct difficulty and valid PoW before it can - trigger a replay, so the attack costs real work — but syncing a long legitimate - fork is still O(branch × chain), and `getblocks` remains a small-request / - 200-block-response amplifier with no rate limit over time. -- Reorged transactions are dropped rather than re-pooled — silent payment loss. -- Private keys are written unencrypted to `data/wallet.json`; no HD derivation. -- Merkle odd-duplication (CVE-2012-2459); `MAX_MONEY` sits at the edge of - float53 precision. - -**P2** -- [x] **"Open the explorer" opened the marketing site.** `hearth/web/` is a - multi-page bundle whose `index.html` *is* the marketing page, with the - explorer at `explorer.html` — and the site's link pointed at the bundle - root, on `savvaniss.github.io` rather than our own host. Both that button - and the "Live explorer" footer link therefore just reloaded the homepage. - Now resolved from `cloudsforgeHosts()` like the other product URLs in the - same file, and named to the file. The tunnel already routed - `explorer.` to the bundle, so nothing else moved. Confirmed against - the live site: the root serves the marketing title, `explorer.html` serves - "Hearth Block Explorer". -- Shipped PoW params are 64 KiB / 256 steps — cache-resident, so **not** - memory-hard and GPU-farmable today. The anti-ASIC thesis is unimplemented at - shipped values. -- Marketing overclaims to reconcile: "RandomX-class VM" (it's a SHA-256 walk), - "one app — node, wallet, miner" (a Tauri skeleton). The "in-browser mining" - claim is **gone** — the wallet rewrite removed the timer, and the adjacent - "WASM light-miner" line on `hearth/web/index.html` and the matching README copy - went with it. -- `--miner-address` silently mines to the node's own wallet when the node holds no - key for the address given, because PoW binds `coinbasePub` (`miner.js:27`). A - command that looks right and quietly pays someone else; found while testing the - wallet, documented in the UI, still worth a hard error in the node. -- Decide on `hearth/rust`: ~15% of a rewrite that benchmarks a hardcoded string, - uses a different wire format and different difficulty semantics, and cannot - talk to the JS node. Finish or delete. - ---- - -## Forge Pay (`services/pay`) - -All five chains are genuinely implemented, none stubbed. - -**P0 — all fixed this pass.** -- [x] **Double-credit fixed.** `recordDepositAndCredit` now takes an address *id* - and re-reads the row `FOR UPDATE` inside the transaction, recomputing the - delta and deriving the synthetic txid from the locked total, so two ticks - can never mint two txids for the same funds. The watcher also has an - in-flight guard so a slow chain can't stack up concurrent sweeps. -- [x] **Real confirmation depth.** ETH probes the balance at `latest − 12`, SOL - `finalized`, XRP `validated`, BTC `chain_stats` (confirmed-only). Only the - confirmed figure is ever credited; what's visible at the tip is surfaced - separately as `pending` and shown in the wallet, so a deposit doesn't look - lost during the confirmation window. -- [x] **Sweep-safe.** New `swept` column; the comparison is `balance + swept` vs - the high-water mark, and an unexplained drop is logged as a `regression` - that pauses crediting for that address rather than silently forgiving it. - The false "deposit addresses never decrease" comment is gone. -- [x] **Deposits are spendable.** `POST /coins/convert` turns a confirmed coin - balance into Shards in one transaction and writes the ledger (the watcher - never did). Introduced the repo's first idempotency mechanism — an - `idempotency_keys` table + `withIdempotency` helper — and applied it to - `/spend` as well. -- [x] **Address creation unbroken**: pay now sends the `orderId` the vault - requires (`deposit:${userId}:${coin}:${network}`). -- [x] `wallets.shards` and `ledger.delta` widened INT4 → BIGINT; a few hundred - BTC of conversions overflowed the old column. - -**P1** -- [x] **Price oracle shipped; the placeholder rates are gone.** `services/pay/src/pricing.ts` - medians USD spot from four keyless public APIs (CoinGecko, Coinbase, Kraken, - Binance) on a timer, never in the request path. A round needs ≥2 good quotes - and is thrown out entirely if the sources diverge >5% — wide divergence means - one of them is wrong and there is no way to tell which. `shardsForCoinAmount` - now takes the rate as a BigInt fixed-point parameter; the static - `shardsPerCoin` field is deleted. Conversions **fail closed**: a quote older - than 15 min gets a 503 `rate_unavailable` rather than settling at a stale - price. EMBER has no market, so it is an explicitly *administered* rate - (`PAY_EMBER_USD`) that is unset in production and refused until set. A 2% - spread (`PAY_CONVERSION_SPREAD_BPS`) covers movement between quote and - settlement. The applied rate is captured on `CoinConversion.rate` and in the - ledger `reason` for reconciliation, and `GET /coins/rates` publishes the board - so the wallet UI cannot offer a conversion the server will refuse. - The placeholders were badly stale: they implied BTC $99.8k / ETH $2,994 / - SOL $150 / XRP $2.00, i.e. they were paying out 1.5–2× the coins' real worth. - **Residual gap:** the divergence gate only catches sources *disagreeing*. If - all of them move together — a genuine flash crash, or a shared upstream - feed going wrong — the median is accepted. A circuit breaker on change since - the last good quote (pause and alert rather than settle) is the next - hardening step, and the one to do before mainnet deposits are switched on. - Two pricing decisions worth a conscious sign-off: the 2% spread default, and - that shards/USD is anchored to the *worst* package in `SHARD_PACKAGES` - (pouch, 100.2/USD) so converting a deposit never beats buying outright. -- **EMBER confirmations are unenforceable.** The Hearth REST API exposes only a - tip balance with no per-UTXO height, so EMBER records `confirmations: 0` to - stay honest. Fixing it needs a `hearth/node` change (UTXO heights, or a - `?confirmations=` query). -- **`recordSweep` has no callers** and a subtle hazard when one is written: if it - is called at *broadcast* time, the swept funds are still inside the confirmed - balance for the confirmation window, so `balance + swept` double-counts and - credits a phantom deposit. Only record a sweep once it is confirmation-deep. -- A `duplicate` scan result deliberately does not advance the high-water mark, so - an address hand-reconciled the wrong way (lowering `last_seen` instead of - calling `recordSweep`) will stop crediting with only a `log.warn`. -- `PAY_PROVIDER` defaults to `'mock'`, exposing `/invoices/:id/mock-pay` — free - Shards for any authenticated user. Must hard-fail in production. -- **No refund path exists anywhere in the repo.** -- Ledger is a mutable balance column plus an append-only log that nothing ever - reconciles against it. -- `getCoinBalances` was orphaned by the new tx-scoped `walletSnapshot` — delete - it or make `walletSnapshot` call a tx-aware version, before someone fixes the - dead copy. - ---- - -## ForgeMint (`services/forge-mint`, `apps/forge-mint`) - -**P0 — fixed this pass.** -- [x] **Real compiled contracts.** `contracts/ForgeTokens.sol` (OpenZeppelin 5) - compiled by `scripts/compile-contracts.mjs` into a committed - `src/chain/erc20.generated.ts` — the repo has no Solidity toolchain and - services run under `tsx` with no build step, so there is nowhere to hang a - compile at deploy time, and bytecode in git is bytecode a reviewer can - diff. Three variants, one per tier, so the catalog copy is finally true: - Spark is fixed-supply, Forge is mintable/burnable/ownable, Foundry adds - capped + pausable. `decimals` is now a constructor argument rather than - hardcoded 18 while supply was scaled by it. - Verified by deploying all three on a local EVM and exercising - transfer/approve/mint/burn/pause and the cap revert. -- [x] **Testnet and mainnet both work end to end.** `network` is now passed to - the vault at provision time (it wasn't, so every deployer key was recorded - as testnet), and all 12 baked RPC endpoints were probed live and confirmed - to report the expected chain id / Solana cluster. -- [x] Deploy waits for the receipt and verifies contract code exists, instead of - recording a precomputed CREATE address as `deployed` before it is mined. - The tx hash is persisted the moment it is broadcast, a second deploy over - an in-flight tx is refused (`deploy_in_flight`), and `GET /status` settles - a tx whose broadcast outlived the receipt wait. Nonce now reads `pending`. - -**P1** -- **Foundry on Solana is still an overclaim.** SPL mints have no cap or pause - primitive, so `cap` is now rejected for non-EVM chains rather than silently - dropped — but the catalog still sells "Mint / burn / pausable / capped" on a - tier that includes SPL. Either implement it (mint then revoke authority) or - split the copy per family. -- No rate limiting on any route, and each `/provision` spawns a Docker container. - -Good as-is: the `confirmMainnet` guard is genuinely server-side, and there is no -shared funded deployer to drain (each order uses a user-funded address). - ---- - -## Crucible (`crucible/services/crucible`, `crucible/apps/crucible`) - -**New this pass — the sixth product.** Algorithmic trading: a backtest engine -over real exchange candles, ten strategies with curated presets, and bots that -run one in paper or live. Serves its own SPA on `:4006` beside the API, the -shape ForgeMint established. - -**What is genuinely built and verified.** -- [x] **The engine does not cheat.** A signal from bar `i`'s close fills at bar - `i + 1`'s **open**, and the feed drops the in-progress candle. Verified - against 999 real daily BTC candles by re-running with one future bar - appended and asserting every earlier fill is byte-identical. Fees (10 bps) - and slippage (5 bps) are charged by default and buy & hold is computed - over the identical window, so "did it beat holding" is always answerable. -- [x] **Ten strategies compile to one target-exposure series**, so binary trend - rules and fractional ones (grid, DCA) share one execution loop. Verified - end to end through the API: the turtle preset returned 41.3% against - 84.6% holding, 38.3% max drawdown, 31 fills — i.e. it loses to the - benchmark, which is the kind of answer the product exists to give. -- [x] **Candle cache in Postgres**, keyed `(pair, timeframe, t)`. Closed candles - are immutable so a hit is never revalidated; Binance primary, Kraken - fallback, because a geo-restricted Binance answers 451 forever rather than - something a retry fixes. -- [x] **A bot acts at most once per closed candle** — `last_candle_t` across - restarts, and a unique index on `(bot, candle, side)` at the database. The - fill row is written *before* any money moves and its id is the settlement - idempotency key, so a crash between the two cannot trade twice. -- [x] **Live is off by default** (`CRUCIBLE_LIVE_ENABLED=false`). Verified: the - create call is refused with `live_disabled` rather than failing later. -- [x] **The service creates its own database.** `infra/pg-init` only runs on a - fresh volume, so every existing deployment would otherwise need a manual - `CREATE DATABASE`. Same fix Lantern already carries. - -**Monetization — the reason this shipped.** A high-water-mark performance fee, -15%, charged only on equity above the highest the bot has ever been billed at. -The same gain is never billed twice; a recovery from a loss is free; a bot that -ends where it started has paid nothing. The fee hits the wallet, not the bot — -deducting from the bot would silently shrink the position the user chose and -make the equity curve a function of our billing. - -When the wallet is short, the shortfall is carried as `fee_owed` **and the mark -still advances**. Leaving the mark behind while also recording the debt would -bill the same gain twice, once as arrears and again as a fresh gain next pass. - -**The design decision worth recording.** The obvious build — the bot holds -Shards, the platform takes the other side — was rejected. It makes every winning -customer a liability with unbounded upside for them and unbounded exposure for -us. A live bot instead trades the coin Forge Pay **already custodies**, so the -platform's risk is custody, which it already carries, and the gains are real -Shards a fee can honestly be charged on. That decision is what forced the Pay -work below. - -**P1 — known gaps, none of them load-bearing for backtesting or paper.** -- **Live trading has never been exercised against real balances.** Everything - verified above ran with `CRUCIBLE_LIVE_ENABLED=false`. The live path is - written and typechecks; it has not moved a coin. Do not turn it on without - driving a full buy → mark → settle → stop cycle on testnet first. -- **No admin surface.** There is no operator view of running bots, aggregate - fee take, or a stuck settlement. Lantern shows the logs; nothing shows the - book. -- **The bot's `position` is a mirror of Pay's `coin_balances`, not the truth.** - Sells cap themselves against what Pay reports, so drift fails safe, but - nothing reconciles the two or alerts when they disagree. -- **One runner loop, no leader election.** Two Crucible containers would both - tick every bot. The per-candle unique index stops a double *fill*, but the - design assumes a single instance. -- **No rate limiting**, and a backtest is synchronous — 5000 candles through ten - indicators is milliseconds, but the feed call in front of it is not. -- Presets are hand-written and unbacktested as a set. Nothing checks that a - shipped preset still behaves the way its blurb claims. - ---- - -## Forge Pay — the `/internal` billing surface (new this pass) - -Pay had **no service-to-service auth at all**: every route was gated on a user's -own bearer token, which is the right default and was also a hard ceiling. A -performance fee is assessed on a timer, hours after the user last touched -anything, so there is no token to forward and minting one on their behalf would -let Crucible act as any user at any time. - -- [x] `requireServiceToken` (constant-time, both sides hashed so the comparison - is fixed-width) gating `POST /internal/{charge,credit,trade}` and - `GET /internal/wallet/:userId`. **Fails closed on an empty secret**, and - says so at boot — the symptom otherwise appears in the *calling* service's - logs as an unexplained 401, hours later. -- [x] `ledger.source` / `ledger.ref`. `reason` is prose and stays that way — - it is what a user reads on a statement — but reconciling a service's - charges against its own records should not mean parsing prose. -- [x] **Coin purchase, the missing direction.** `convertCoinToShards` had no - inverse, so a strategy could sell a position but never open one. - `buyCoinWithShards` mirrors it and settles at a **buy** rate — the same - mid price with the spread applied the other way, because quoting a - purchase at the sell rate would let anyone convert out and straight back - in at a profit, repeatedly. - -Verified by hand against the running stack: 401 without and with a wrong token; -a replayed idempotency key credits **once**; the same key with a different -amount is refused `idempotency_key_reuse`; an over-balance charge is refused -`insufficient_balance`; `source`/`ref` land on the ledger row. - -**P1.** `/internal` takes an explicit `userId`, so anything that can reach it can -move anyone's balance — it is on the compose network only and must never get a -tunnel ingress rule, exactly like ForgeKeyvault. There is still no reconciliation -job comparing the ledger against `wallets.shards`, and still no refund path. - ---- - -## ForgeKeyvault (`services/forge-keyvault`) - -AEAD is correct — fresh random 12-byte IV per encrypt, no nonce reuse. - -**P0 — fixed this pass.** -- [x] **Fails closed on secrets.** `KEYVAULT_SERVICE_TOKEN` and - `KEYVAULT_MASTER_SECRET` are now required, must be ≥24 chars, and known - placeholders are rejected by name. `.env.example` no longer ships the - literal values the service now refuses. Verified: the service will not boot - when a secret is missing, placeholder, or short. -- [x] **`/sign` is bound, not an oracle.** The caller must restate purpose, - chain, network and orderId, and each is checked against the stored row; - deposit-purpose addresses are never signable; EVM payloads must be contract - creations (`to == null`) carrying the chain id that matches the address's - own network. Verified live against a running vault: 12/12 cases, including - value-transfer-out, signing a deposit address, and cross-network replay. -- [x] Service-token comparison hashes both sides first, so it no longer leaks the - token's length. - -**P1** -- No key rotation anywhere; changing `KEYVAULT_MASTER_SECRET` permanently bricks - every custodied key. This now bites harder: the fail-closed check forces any - deployment still on the old placeholder to pick a new secret, which would - orphan its existing keys. Harmless today — `vault_addresses` does not yet exist - in the live database, so there are zero custodied keys — but a rotation path - (key versioning + re-encrypt) must land before real custody begins. -- Relatedly, `/sign` binds on `network`, and any row provisioned before this pass - would carry the old `'testnet'` default. Zero such rows exist today; if that - changes before this ships, a mainnet order would 403 with no repair path. -- Container-per-address is largely theatre: the vault holds the Docker socket - (host root) and decrypts in its own memory regardless. -- `/admin/keys/:address/reveal` returns plaintext keys to any `admin` claim with - no MFA or rate limit — an XSS in `apps/admin` exfiltrates every key. - ---- - -## Platform (`services/nimbus`, `apps/site`, `apps/admin`, `packages/*`) - -**P0 — session handling fixed this pass; account recovery deliberately deferred.** -- [x] **Rate limiting is real.** `trustProxy` is set to private ranges only - (`loopback,linklocal,uniquelocal`, `TRUST_PROXY`-overridable) rather than a - blanket `true` — the service ports are published, so trusting every peer - would let any direct caller spoof `X-Forwarded-For` and mint a fresh bucket - per request. Credential routes now carry their own limits well under the - global 200/min: login 10, register 5, refresh 30, exchange 20. -- [x] **Per-account lockout**, so an attacker rotating IPs still hits a wall: - 5 free attempts then 60s doubling to a 15-min cap, persisted in Postgres so - it survives a restart. Checked *before* the user lookup and recorded for - unknown emails too, so the response cannot be used to enumerate accounts — - verified byte-identical for a real and a nonexistent address. -- [x] **Tokens are out of the URL.** The SSO hand-back is now a one-time - exchange code: the portal redirects with `#cf_code`, the app POSTs it to - `POST /auth/exchange`, and the tokens come back in a response body. The code - is hashed at rest, single-redemption, 60s TTL, and bound to the origin it - was issued for — and it carries no tokens, so it is worthless once spent. - The fragment path is gone, not kept as a fallback. -- [x] **Wildcard allowlist removed.** `PORTAL_ALLOWED_ORIGINS` defaulted to - `*.cloudsforge.online` and was set nowhere, so the wildcard is what shipped; - any one compromised subdomain could receive a hand-off. It is now an - explicit list, set in compose and both env examples. -- [x] **Refresh reuse detection.** Rotation is a single conditional - `UPDATE ... WHERE revoked = false ... RETURNING` (the old select-then-update - let two concurrent rotations both win). Presenting a spent-but-unexpired - token is proof of theft and burns the whole `family_id` chain, forcing both - thief and victim back to a real login. - **Known tradeoff:** two *browser tabs* racing a refresh can burn a - legitimate family. All three clients dedupe in-tab, so this needs a genuine - cross-tab race; the alternative is a leeway window that re-opens a silent - theft window. Revisit if users report surprise logouts. -- [x] **SSO actually signs on once.** `GET /login` rendered the password form - unconditionally — it never read the session it had itself written to - `localStorage`, the way `/account` already did. So every product that - bounced a signed-in user to the portal asked for the password again, and - clicking "Admin Console" from the account launcher (a plain link to - another origin, carrying no hand-off code) looped straight back to a login - form. `/login` now completes the hand-off when a session exists, and - retries once after a refresh when the access token has expired. -- [x] **A rejected return URL says so.** `safeReturnUrl()` fell back to - `/account` in silence whenever an origin was missing from - `PORTAL_ALLOWED_ORIGINS`, so "Start minting" on ForgeMint landed users on - the account page looking like a broken redirect. A rejected return is now - distinguished from an absent one: the user sees a banner naming the - origin, the operator gets a `warn` log naming the variable, and a - signed-in user is not handed off past the explanation. -- [ ] **No password reset, no email verification, no MFA** — deferred by choice, - not oversight: the repo has no mail provider at all, and picking one - (Resend / SMTP / SES) is an owner decision. Note a locked-out user - consequently has no self-serve recovery beyond waiting out ≤15 min, and - account lockout remains a nuisance-DoS vector against a known email. - -**P1** -- `apps/site` and `apps/hearth-site` hardcode `signedIn: false`, so the - marketing site can never show a logged-in user. -- The newsletter capture is fake (`CTA.tsx:38` sets state, no backend). -- No analytics, no `og:image`, no sitemap/robots; SPA with no prerender, so - crawlers get an empty `#root`. - -**P2** -- [x] **Admin has user management.** Roles were writable only by `seedAdmin()` - at boot, so granting admin meant editing `ADMIN_EMAIL`/`ADMIN_PASSWORD` - and restarting — which also reset that account's password every time — - and there was no way to see who had an account. Nimbus now has three - admin-gated routes (list with search and paging, fetch one, set roles) - and the console a Users page. Two demotions are refused rather than - allowed to lock everyone out: your own admin role, and the last remaining - admin. The latter is reachable only with a stale token — claims outlive - the change by up to 15 min — so the count is taken inside a transaction - with `SELECT … FOR UPDATE`. -- Admin still has no payments/refunds and no entitlements: `forge-pay` has no - operator surface at all, so there is no backend to drive. Nor can a user be - suspended or deleted — there is no such column, only `roles`. -- The `admin` claim is read from the JWT and never re-checked against the - database (`game/src/auth.ts:37`, `keyvault/src/auth.ts:51`), so revoking - admin does nothing until the token expires. Now that roles are mutable at - runtime this matters in a way it did not when they were fixed at boot. -- Heavy duplication below the shared account bar: `Img.tsx` in three divergent - copies, `ProtectedRoute` 3×, a separate token client per app. The commune UI - added more of the same rather than extracting: `bagTotal` now exists in three - places, and the `flash` toast, the segmented tab control and the progress - meter are each copy-pasted out of `Progress.tsx` instead of living in - `components/ui.tsx` (where the same pass did correctly lift `Stepper`). -- `CONTRACTS.md` has drifted — documents none of the portal or keyvault admin - routes. -- `asset-forge`'s "post-process" resize stage does not exist — there is no - `sharp`/resize/crop code anywhere, so the manifest `size` only picks the - nearest legal API size and outputs are never resized to the intended - dimensions. (The `gpt-image-2` → `gpt-image-1` model chain is deliberate: the - newer tier first, falling back when the org isn't verified for it.) - ---- - -## Cross-cutting - -- [x] **The origin allowlists no longer default to localhost.** `CORS_ORIGINS` - (five services) and `PORTAL_ALLOWED_ORIGINS` shipped localhost-only defaults, - so any deployment that did not write both lists out by hand failed in two ways - that look nothing like a config problem: the admin console reported zero - worlds, because the browser refused its calls to the game API, and signing in - from any product landed on `/account`. Both now derive from - `CLOUDSFORGE_APEX` and keep the localhost origins alongside — an unused - allowlist entry costs nothing, a missing one is silent — so one variable makes - a public deployment work. The two lists stay distinct: CORS is browser origins - that call the APIs, `PORTAL_ALLOWED_ORIGINS` is return URLs users are handed - back to, and so lists no API service. Note `.env.example` shipped - `PORTAL_ALLOWED_ORIGINS` *uncommented* and localhost-only, so the documented - `cp .env.example .env` would have reintroduced the bug on exactly the - deployments this fixes; it is commented out now, with the reason. -- **Still near-zero tests outside `hearth/`** — and this remains the - highest-leverage cleanup left. Hearth itself is now genuinely covered - (`unit` 28, `e2e` 24, `p2p-fork` 25, all blocking in CI), and the p2p suite - proved its worth immediately: the bug it caught existed *because* `e2e.js` - never called `start()`, so no test had ever exercised p2p or RPC. Every other - service is still verified by throwaway harnesses that are not committed — - a local EVM for the contracts, a live vault for the sign bindings, a real - Postgres for the money paths, the oracle's fail-closed and divergence cases. - Those are the tests to land next; the pattern is proven, it just isn't durable. - No linter, no formatter, no error tracking, no metrics, no uptime checks. -- **No migration system** — schema is idempotent `CREATE TABLE IF NOT EXISTS` on - boot, now with a growing tail of `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` - appended by this pass (`players.joined_day`, `commune_credit`, - `deposit_addresses.swept/pending`, `token_orders.cap`, the INT4→BIGINT - widenings). This works and was verified idempotent against the populated game - database, but it is at the limit of what boot-time DDL should carry. Adopt - drizzle-kit before the next schema change. -- **`docker buildx` works now.** It was installed via Homebrew all along but - never linked as a Docker CLI plugin, so `docker buildx` answered "unknown - command" and compose silently fell back to the classic builder. One symlink - (`~/.docker/cli-plugins/docker-buildx`) fixed it; all 12 images build, and the - stack can finally be brought up to verify end to end here. If it ever reports - "unknown command" again, check the symlink before concluding it is missing. -- [x] **Base images digest-pinned** across every Dockerfile and compose entry. - The chain node moved `node:20-alpine` → `node:22-alpine`, matching the repo's - Node 22 engine pin. `postgres:17-alpine` is still floating. -- [x] **Containers run as non-root** (`USER node`, uid 1000) — services, apps and - all three chain nodes, verified live with `restarts=0`. - **Exception: `forge-keyvault` still runs as root, deliberately.** It holds the - Docker socket, which is root-owned mode 0660 with a gid that is a property of - the *host* (991 under colima here, commonly 998/999 on Linux). Baking that - number in would silently break key custody on the real host. The fix is - `group_add: [""]` in compose plus `USER node`, once the - deployment host's gid is known. -- [x] **`verifyDepsBeforeRun` restored**, and the services are multi-stage: - install in a `deps` stage, carry `node_modules` forward to a slim `runtime` - stage. Images dropped ~45%: nimbus 865→482MB, game/pay 852→463MB, keyvault - 985→595MB, forge-mint 1.06GB→663MB. - Two traps found doing it, both worth remembering: - **(a) `verifyDepsBeforeRun: true` is not a valid value in pnpm 11** — it is - silently ignored and runs no check at all. `error` is what actually enforces. - **(b)** without an explicit `allowBuilds` block, `pnpm install` *rewrites* - `pnpm-workspace.yaml` mid-install with placeholders, which then read back as a - changed setting and trip the check at runtime. - `pnpm deploy` was rejected on purpose: `@cloudsforge/shared`'s entry is - `src/index.ts`, and `deploy` materialises workspace deps as real directories - where `tsx` won't transpile TS — the symlink layout is what makes this work. -- `services/forge-keyvault/src/chains.ts` `require()`s `hearth/node/src/crypto.js` - by repo-relative path. It is not a workspace dependency, so it silently vanished - from the slimmed image and needed an explicit `COPY`. Nothing else in - `services/*/src` escapes its own tree — but this one should become a real - workspace package rather than a path traversal. -- Services ship no build artifact — `tsx` runs TS at runtime in production and - `build` is `tsc --noEmit`. CI now catches type errors that previously reached - production as runtime crashes. - ---- - -## Proposed specialized agents - -Reusable review agents worth defining in `.claude/agents/`: - -| Agent | Job | Trigger | -| --- | --- | --- | -| `money-path-auditor` | Trace every balance mutation for idempotency, transaction boundaries, confirmation depth, reorg safety, races | Diffs touching `services/pay`, `forge-mint`, ledger, watcher | -| `consensus-auditor` | Adversarial review of chain changes: fork choice, DoS cost per message, malleability, cross-impl divergence | Diffs in `hearth/node`, `hearth/rust` | -| `secrets-exposure-sentinel` | Secrets in env/compose/Dockerfiles, fail-open defaults, newly published ports, image-layer leakage | Pre-commit / pre-deploy | -| `contract-coverage` | Diff `packages/shared` contracts against real client call sites; report implemented-but-unreachable endpoints | Weekly — this is what found the 13 dead game endpoints | -| `claim-vs-code` | Check marketing copy against implementation; flag overclaims | Diffs in `apps/site`, `apps/hearth-site`, README | diff --git a/TESTING.md b/TESTING.md index e75e341..64712a3 100644 --- a/TESTING.md +++ b/TESTING.md @@ -97,8 +97,8 @@ which never comes up is a monitor which never starts. ## 3. Probes -Twenty-eight of them, every thirty seconds, all concurrent. Sequentially, twenty-eight targets -against a five-second timeout is a worst case of one hundred and forty seconds, so the monitor's +Thirty-three of them, every thirty seconds, all concurrent. Sequentially, thirty-three targets +against a five-second timeout is a worst case of a hundred and sixty-five seconds, so the monitor's resolution would collapse precisely when the network went bad. Each service gets the liveness probe everyone already has **and a deep probe that cannot answer @@ -113,8 +113,11 @@ public or already cached. Each names the dependency it proves. | `crucible.catalog` | Strategies loaded, and surfaces whether `CRUCIBLE_LIVE_ENABLED` is on. | | `lantern` | The one health route in the estate that tells the truth: it reports its Postgres connection, its write queue and its drop counter. | | `hearth.work` | A node can answer `/info` from state while being unable to build a block. Every miner needs `/mining/template` and nothing else reports on it. | +| `hearth.rpc.` | A Hearth node runs **two** listeners: REST on 8645 and Ethereum JSON-RPC on 8545. Everything above asks the first one; forge-pay, the explorer, MetaMask and Hardhat all use the second. It fails soft — `listenJsonRpc` logs `json-rpc listen failed` and the node serves REST happily with no `eth_*` surface at all — so this is the only check that can tell "the node is up" from "the RPC is up". It also asserts the chain id **in both encodings**: `eth_chainId` is a hex quantity and `net_version` is a decimal string, both derived from the same integer by different code, and getting the pair backwards is what makes a wallet refuse the network outright. | +| `hearth-web.eth-rpc` | The explorer's and the wallet's **first** call, made the way a visitor's browser makes it: `eth_chainId` through the bundle's own origin at `/eth-rpc/`, which nginx proxies to `HEARTH_ETH_RPC_UPSTREAM`. `hearth-web` proves the HTML is served and `hearth.rpc.seed` proves the node answers `eth_*`; neither proves the wire between them, and that wire is what was broken (CF-13): the proxy named the REST port, every call came back HTTP 404 with `{"err":"this is the REST API …"}`, and both probes either side stayed green while the public explorer showed nothing but errors. A wrong chain id here is amber rather than red — the per-node probe owns that verdict — and means the proxy reaches a different node. | +| `hearth.rpc.state` | Picks a settled height every node has and makes each of them name the block and the state root it holds there. `hearth.consensus` below compares tips, which can only speak about nodes that happen to be level right now — a fork *behind* the tip reads as lag for as long as the heights keep differing. Same height with a different block is that fork; the same block with a different state root is worse, because the header commits to the root, so one node is serving balances no other node agrees with. | -Four checks are **derived** — they read the other results rather than making a request, because the +Five checks are **derived** — they read the other results rather than making a request, because the question cannot be put to any single node: - **`hearth.consensus`** — two nodes at the same height with different tips is a fork, and a fork @@ -127,6 +130,12 @@ question cannot be put to any single node: check you learn to ignore — which discredits every check beside it. Height is still recorded and still graphed. `hearth.mempool` follows the same switch, because a queue with nothing mining is the expected state rather than a fault. An unreachable node is still an outage either way. +- **`hearth.rpc.liveness`** — the same question asked of the other listener, plus one only a pair of + listeners can answer. `eth_blockNumber` is the number forge-pay's deposit watcher polls, and a + watcher whose height never moves credits nothing and says nothing; and because both listeners are + the same process reading the same chain object, a JSON-RPC height sitting more than a block or two + behind that node's own REST height means one of them is serving a chain that has moved on. It + follows `BEACON_CHAIN_EXPECT_BLOCKS` for exactly the reasons `hearth.liveness` does. - **`hearth.peering`** — an isolated miner is not down. It is happily mining a chain nobody will ever accept, which is worse and looks fine from inside. - **`hearth.mempool`** — transactions are being mined rather than accumulating. @@ -291,7 +300,7 @@ chain internals. Every one of those describes the shape of an outage to whoever One outbound webhook, fired on incident open and close and on nothing else. -Twenty-eight targets probed twice a minute is about eighty thousand check results a day. A rule that +Thirty-three targets probed twice a minute is about ninety-five thousand check results a day. A rule that fires on any of them is a rule whose channel is muted in week one, after which the monitor is decorative. Journeys need two consecutive failures — ten minutes at the default cadence — because a scenario touches six services and one flake in any of them is not news. diff --git a/VERIFICATION.md b/VERIFICATION.md new file mode 100644 index 0000000..6672451 --- /dev/null +++ b/VERIFICATION.md @@ -0,0 +1,323 @@ +# VERIFICATION.md — what was run, and what it said + +The defect register is [audit.md](audit.md). The plan is [upgrade.md](upgrade.md). This is the +third document: **what was actually executed against the remediated tree, and what came back.** + +It exists because the estate has been burned twice by the gap between "the code changed" and "the +thing works". `docker compose build` returns exit 0 while cancelling every target, and a rejected +CORS preflight 204s so the handler never runs and nothing appears in any log. Both defects passed +a reading of the diff. Neither would have survived being run. + +Every number below is either something executed here, or something reported by the agent that did +the work — and the two are **labelled separately throughout**, because they are not the same kind +of evidence. Where a figure was re-derived independently, both readings are given. + +--- + +## 0. The results, in one table + +| what | result | +| --- | --- | +| Continuous integration, 6 repositories | **all checks green**, including every Docker image build | +| Typechecks, 7 workspaces | **7/7 clean** | +| Hearth full suite | **31 suites, 86,094 checks, exit 0** | +| Hearth EVM conformance (GeneralStateTests) | **20,077/20,077, 60,231 checks, 0 failed** | +| Beacon monitoring suite | **23/23** | +| Hearth PoW parameter harness | **7/7** | +| Explorer API — fixtures / in-process / external node | **177/177 · 27/27 · 21/21** | +| Faucet | **66 passing** | +| Full-block execution | **35.3 s → 5.2 s**, 212.1 MiB → 9.2 MiB | +| Tickets put through adversarial review | **29**, of which 19 confirmed clean | +| Security regressions found by that review and closed | **3**, including one account-takeover | + +Two results are worth stating in words rather than numbers. The **Docker image builds pass in CI** — +in the previous round every image in the estate silently failed to build while the build command +reported success. And the **2 GiB PoW target was measured for the first time** and found to cost +185.7 seconds per evaluation, which is a genesis-blocking finding rather than a passing test; §3. + +§6 lists what was *not* tested. It is part of the result, not an appendix to it. + +--- + +## 1. Scope + +51 items: the 49 register tickets, plus two that are not defects — outbound mail delivery for +Nimbus, and the missing JSON-RPC methods. + +| | count | +| --- | --- | +| Tickets with a recorded outcome | 40 | +| — resolved | 23 | +| — already fixed by earlier work, verified and closed | 9 | +| — partial, with the remainder written down | 8 | +| Tickets put through adversarial review | 29 | +| — confirmed: defect gone, nothing else broken | 19 | +| — incomplete: partially closed | 7 | +| — regression: defect closed, something else broken | 3 | + +The reviewer was told to assume each fix was **not** real and to walk the ticket's own +*How it fails* scenario through the new code until it either stopped or didn't. All three +regressions were sent back and closed; see §4. + +--- + +## 2. Executed here + +Run directly, not reported by an agent. + +### Typechecks — 7/7 clean + +`pnpm typecheck` at the root of each workspace: crucible, forge-keyvault, forge-mint, forge-pay, +ninety-days-after, platform, shared-libs. + +forge-mint initially failed with `ERR_PNPM_VERIFY_DEPS_BEFORE_RUN` — a test script had been added +without a matching install. `pnpm install` resolved it; the lockfile did not change and the +supply-chain policy check passed on all 363 entries. + +### Beacon — 23/23 + +`node --test` in `infra/beacon`. Covers the new JSON-RPC monitoring surface: two nodes naming the +same block with different state roots is down; a node whose JSON-RPC height lags its own REST +height is down; one block of skew is not an incident; a JSON-RPC height that stops moving +eventually goes down. + +### Hearth PoW parameters — 7/7 + +`node test/pow-params.js`. The harness parameterises the real `homefireHash` and asserts it +reproduces the configured digest *before* it measures anything, so a passing run is a measurement +of the shipped function rather than of a copy of it. + +Independent extrapolation from this run: **183.6 s** per evaluation at 2 GiB. The agent's +measured full run: **185.7 s**. 1% apart — see §3. + +### CI — all green + +Six pull requests, every check passing: + +| repo | checks | +| --- | --- | +| crucible | Docker image, Secret hygiene, "The settlement pass bills a gain once", Typecheck + build | +| forge-keyvault | Docker image, Secret hygiene, "Signing refusals and the treasury pin", Typecheck | +| forge-mint | Committed bytecode matches the source, Docker image, Secret hygiene, Typecheck + build | +| forge-pay | Docker image, Secret hygiene, Typecheck | +| ninety-days-after | Docker images, Secret hygiene, Typecheck + build | +| platform | Docker images, Secret hygiene, Typecheck + build | + +**The Docker image jobs passing is the load-bearing result here.** In the previous round every +image in the estate silently failed to build while the build command reported success, and a +"healthy" stack was running stale images. These are real builds, in CI, with a real exit code. + +--- + +## 3. The PoW measurement + +The single most consequential result, and the reason this document exists at all. + +Every Hearth block ever produced — in tests, in CI, on the compose testnet — used a **64 KiB** +scratchpad and a 256-step walk. The documented mainnet intent is **2 GiB** and 2,048+ steps: a +32,768× increase in per-attempt memory that no code had ever executed at. + +Measured, one machine, per evaluation: + +| pad | per evaluation | peak RSS | +| --- | --- | --- | +| 64 KiB — what everything has always run | 6.57 ms | — | +| 256 MiB | 24.0 s | — | +| **2 GiB — the intent** | **185.7 s** | **2,165 MiB** | + +Against a 15-second target block time, one attempt is **12× the whole block interval**. A +200-block `getblocks` page costs roughly **10.3 core-hours** to verify. The 2,048-step walk turns +out to cost 1.31×, not 8× — so the pad is the cost, and it is CPU-bound, not memory-bound. + +`node/src/params.js` now carries `POW_MAX_SCRATCH_KIB: 4096` and **refuses to start above it**. +A node configured with the launch checklist's own 2 GiB value was demonstrated declining to boot, +which is the correct failure: better than mining a chain nothing can validate. + +**Two things were deliberately not measured, and both are recorded as unmeasured** rather than +estimated: the 3-node measurement network (at 185.7 s per attempt the first block is ~13 +core-hours, so the measurement that justified the network is the one proving it cannot run), and +`P2P_BLOCK_VERIFY_BURST`, because no value works at that cost. + +This is a genesis-blocking product decision, not a bug. A mineable pad size has to be chosen, and +that choice changes Hearth's security argument. + +--- + +## 4. The three regressions + +Each was found by adversarial review, each was sent back, and each was re-read in the tree here +rather than accepted on report. + +**Password-reset delivery — account takeover.** Wiring the mailer made a latent weakness live: the +reset link was built from the request `Host` header, so anyone who knew an address could have the +deployment mail them a correctly-branded link pointing at a host of their choosing. It reproduced +against a running service with an SMTP sink. Now minted from `NIMBUS_PUBLIC_URL` regardless of how +the request was addressed, with an audit line when the two disagree — deliberately not a refusal, +because refusing tells a prober which hosts are real. + +The same review found the route had become an enumeration oracle by timing: awaiting the SMTP send +split response time **10 ms** for an unknown address against **6,015 ms** for a known one. It now +answers before the send. + +**CF-36 — orphaned refresh tokens.** The grace window minted a sibling token, but sign-out revoked +only the one presented, leaving a live server-side token nobody held for 30 days. `revokeRefreshToken` +now revokes by family. + +**CF-19 — a paused bot could be billed for a gain it never realised.** The sweep called full +assessment on non-running bots. It now carries a scope: running bots are assessed, paused and +stopped bots have arrears collected and nothing more. + +--- + +## 5. Reported by the agents that did the work + +Not re-run here. Recorded with attribution so a later reader knows which claims still want +independent confirmation. + +| suite | result | +| --- | --- | +| Hearth `pnpm test` | 31 suites, 86,094 checks, exit 0 | +| Hearth GeneralStateTests conformance | 20,077/20,077, 60,231 checks, 0 failed (~32 min) | +| explorer-api fixtures | 177/177 | +| explorer-api live chain, in process | 27/27 | +| explorer-api against an external node | 21/21 | +| faucet | 66 | + +**Full-block execution (CF-10),** same machine, same 30M-gas SSTORE transaction: +35.3 s / 212.1 MiB → **5.2 s / 9.2 MiB**. Five times the state-trie width now costs 1.00× rather +than 1.31×. The gate fails on pre-fix code (3/5) and passes now (5/5), with wall-clock expressed +as a ratio to a calibration block so a slow runner cannot flake it. One dimension is explicitly +partial: `setStorage` still materialises the storage root per write, which is about a third of the +remaining 5.2 s. Deferring it needs slot-level journalling and a rework of how revert works, and +is written up rather than half-done. + +--- + +## 5a. End-to-end, against the assembled stack + +The gap this document opened with, now closed. All 18 services rebuilt from the remediated source +and recreated; **16 images built, zero cancelled targets**. Every service reported healthy. + +The pre-existing containers had been "healthy" for 20 hours on **pre-remediation images** — none of +the resolved tickets were in them. Nothing below would have meant anything without the rebuild. + +### Password reset over SMTP — the whole loop + +Driven against a local SMTP sink, through the real service, with a throwaway account: + +| step | result | +| --- | --- | +| register | 201 | +| request reset | 202 | +| **message delivered** | captured at the relay | +| redeem token, set new password | 204 | +| sign in with the new password | 200 | +| sign in with the old password | **401** | +| replay the same token | **401** — single-use holds | + +### The forged-Host attack, closed + +A request carrying `Host: evil.attacker.test` was answered 202 like any other, and the link in the +delivered mail read `https://account.cloudsforge.online/reset#token=…` — the configured origin, not +the forged one. Nimbus wrote the audit line naming both, exactly as designed: + +``` +"audit":"password_reset_host_ignored", +"addressedAs":"http://evil.attacker.test", +"minted":"https://account.cloudsforge.online" +``` + +### The token leaks nowhere + +Both 64-character tokens were decoded out of the quoted-printable message bodies and grepped +against **all 18 container logs**. Zero occurrences. CF-17 holds in the assembled stack, not just +in the diff. + +### The timing oracle is gone + +Known address **12.3 ms**, unknown address **8.3 ms** — no separation to read. Before the fix the +same measurement split 10 ms against 6,015 ms. + +### CF-02 — the browser preflight + +`OPTIONS /cosmetics` with `Origin` and `Access-Control-Request-Method: PUT` now answers: + +``` +access-control-allow-methods: GET, HEAD, OPTIONS, POST, PUT +``` + +This is the faithful test: a preflight is exactly what a browser sends, so this is not a curl +standing in for a browser. + +### The chain + +| node | block | peers | +| --- | --- | --- | +| seed | 0x11bc | 2 | +| miner1 | 0x11bc | 1 | +| miner2 | 0x11bc | 1 | + +`eth_chainId` → `0x1cf4` (7412) and `net_version` → `7412` on the JSON-RPC port. All three nodes at +one height, so CF-27's genesis-hash handshake is admitting legitimate peers rather than +partitioning the network — the specific regression risk that change carried. `net_peerCount` and +`eth_newBlockFilter` both answer, confirming the new method families are live. + +Beacon refuses unauthenticated reads with 401, which is correct: the status page is not public. + +--- + +## 6. What was NOT verified + +The honest list. Everything here is a real gap, not a formality. + +**Outbound mail has never reached a real inbox.** The loop in §5a ran against a local SMTP sink, +which proves the code path, the link origin and the single-use redemption — but not that a real +relay accepts the message, nor that it survives SPF/DKIM/DMARC and lands somewhere other than a +spam folder. The first send through Brevo or any real provider is still ahead. + +**Most of the estate was not driven.** §5a covers password reset, the game CORS preflight and the +chain. It does **not** cover the money paths — no deposit was credited, no withdrawal reached a +terminal state, no manual sweep was refused for an overstated amount (CF-08), and no ForgeMint +order was deployed with a customer-supplied `ownerAddress` (CF-01). Those are the highest-value +remaining checks. + +**Nothing was driven through an actual browser.** The CF-02 preflight is faithful because a +preflight is a plain HTTP request, but no page was loaded, no SPA rendered, and no user journey +clicked through. + +**Seven tickets remain `INCOMPLETE`** after adversarial review: CF-08, CF-15, CF-22, CF-37, CF-38, +CF-42, CF-43. Repair passes ran afterwards and may have closed some of them; that has not been +confirmed ticket by ticket. + +**Eight tickets are `partial`,** with the remainder written down in their commit bodies rather than +in this file. + +**No load, soak or adversarial network testing** of any kind. + +**The PoW figures are one machine.** Consumer Apple silicon, one process, no thermal headroom +analysis. They establish an order of magnitude and a boot-refusing ceiling. They are not a +parameter-selection study, and choosing the real pad size needs one. + +--- + +## 7. How to re-run any of this + +```sh +# per repo +cd repos/ && pnpm install && pnpm typecheck + +# beacon +cd infra/beacon && node --test + +# hearth: everything +cd repos/hearth/node && pnpm test + +# hearth: PoW parameters (add --full to measure 2 GiB for real — minutes, and 2+ GiB of RSS) +cd repos/hearth/node && node test/pow-params.js --sweep + +# hearth: EVM conformance +cd repos/hearth/node && node test/conformance/runner.js --suite=GeneralStateTests +``` + +`--full` is not in the default path on purpose: it allocates 2 GiB and takes minutes per +evaluation, which is the finding, not a defect in the harness. diff --git a/docker-compose.yml b/docker-compose.yml index 20eda49..2e9d81d 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -76,6 +76,13 @@ x-cors-origins: &cors-origins ${CORS_ORIGINS:-http://localhost:3000,http://local # is linked from the product switcher on every front-end, so leaving it out # meant signing in from Lantern dropped you on /account instead of back in the # logs — the one place you go when something is already broken. +# +# Beacon (:4011) is on it on the same grounds, and for a while it was the only +# hostname here that was on BOTH allowlists with no row in the surface registry +# (`repos/shared-libs/packages/shared/src/products.ts`) — so nothing linked to +# it and an allowlist entry described a hand-off nobody could start. It has one +# now, `adminOnly` like Lantern and Admin, which is what makes these two lines +# describe a real journey rather than a permission for one. x-portal-allowed-origins: &portal-allowed-origins ${PORTAL_ALLOWED_ORIGINS:-http://localhost:3000,http://localhost:3001,http://localhost:3002,http://localhost:3003,http://localhost:4004,http://localhost:4006,http://localhost:4010,http://localhost:4011,https://${CLOUDSFORGE_APEX:-cloudsforge.online},https://play.${CLOUDSFORGE_APEX:-cloudsforge.online},https://admin.${CLOUDSFORGE_APEX:-cloudsforge.online},https://hearth.${CLOUDSFORGE_APEX:-cloudsforge.online},https://mint.${CLOUDSFORGE_APEX:-cloudsforge.online},https://crucible.${CLOUDSFORGE_APEX:-cloudsforge.online},https://lantern.${CLOUDSFORGE_APEX:-cloudsforge.online},https://beacon.${CLOUDSFORGE_APEX:-cloudsforge.online}} # The issuer must match the origin BROWSERS hit, not the internal one — it is @@ -147,6 +154,14 @@ services: environment: NIMBUS_DATABASE_URL: postgres://cloudsforge:${POSTGRES_PASSWORD:-cloudsforge}@postgres:5432/nimbus NIMBUS_ISSUER: *nimbus-issuer + # The origin a link that LEAVES Nimbus is built from — the password-reset + # link, emailed or handed to an operator. Not the request's Host header: + # that let anyone who knew a victim's address have this deployment's own + # relay send them a genuine reset email pointing at a host the sender + # chose. account. rather than the issuer's nimbus. because it + # is the hostname a user is meant to see; both route here (see + # repos/platform/deploy/cloudflared/config.example.yml). + NIMBUS_PUBLIC_URL: ${NIMBUS_PUBLIC_URL:-https://account.${CLOUDSFORGE_APEX:-cloudsforge.online}} CORS_ORIGINS: *cors-origins PORTAL_ALLOWED_ORIGINS: *portal-allowed-origins # The admin console reaches ForgeKeyvault THROUGH Nimbus, because keyvault @@ -165,6 +180,20 @@ services: # rely on that: env_file would otherwise hand it a .env entry pointing at # localhost, which inside this container is Nimbus itself. PAY_API_URL: http://pay:4003 + # Outbound mail, for password-reset links. env_file already carries these, + # so they are named here for one reason: Nimbus is the only service in the + # estate that sends mail, and someone looking for why a reset email never + # arrived should find the settings where the rest of Nimbus's are. The + # `:-` defaults keep an unset variable EMPTY, which the service reads as + # "no mail configured" — a supported mode that logs at info and falls back + # to operator-issued links. See .env.example, "Outbound mail". + SMTP_HOST: ${SMTP_HOST:-} + SMTP_PORT: ${SMTP_PORT:-587} + SMTP_SECURE: ${SMTP_SECURE:-false} + SMTP_USER: ${SMTP_USER:-} + SMTP_PASS: ${SMTP_PASS:-} + SMTP_FROM: ${SMTP_FROM:-} + SMTP_REPLY_TO: ${SMTP_REPLY_TO:-} ports: - "4001:4001" depends_on: @@ -426,6 +455,16 @@ services: # and "the chain answers" are different facts and pay depends on the second # one: its deposit watcher polls this RPC, and a watcher that starts against a # dead RPC credits nothing and says nothing. + # + # The healthcheck asks BOTH listeners, and the second one is the one that + # matters. There are two servers in this process — REST on 8645 and Ethereum + # JSON-RPC on 8545 — and the JSON-RPC one fails soft: `listenJsonRpc` in + # repos/hearth/node/src/evmnode.js:211 logs `json-rpc listen failed` on an + # EADDRINUSE and returns, so the node goes on serving /info with a climbing + # height and no eth_* surface at all. Since `HEARTH_TESTNET_RPC` points pay + # at 8545, a check that asks only 8645 marks that container healthy, lets the + # miners' `service_healthy` gate through, and hands forge-pay a deposit + # watcher with nothing to watch. hearth-testnet-seed: <<: *svc build: repos/hearth/node @@ -465,7 +504,10 @@ services: volumes: - testnet-seed-data:/data healthcheck: &hearthd-healthcheck - test: ["CMD","node","-e","fetch('http://localhost:8645/info').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"] + # Both listeners, and `eth_chainId` rather than a bare connection: a JSON + # body with a `result` is the cheapest question only a working JSON-RPC + # dispatcher can answer, and it costs no chain state to answer it. + test: ["CMD","node","-e","const j=async(u,o)=>{const r=await fetch(u,o);if(!r.ok)throw new Error(r.status);return r.json()};Promise.all([j('http://localhost:8645/info'),j('http://localhost:8545/',{method:'POST',headers:{'content-type':'application/json'},body:JSON.stringify({jsonrpc:'2.0',id:1,method:'eth_chainId'})}).then(b=>{if(!b.result)throw new Error('no chain id')})]).then(()=>process.exit(0),()=>process.exit(1))"] interval: 10s timeout: 5s retries: 10 @@ -541,6 +583,14 @@ services: CORS_ORIGINS: *cors-origins LOG_LEVEL: ${LOG_LEVEL:-info} LANTERN_RETENTION_DAYS: ${LANTERN_RETENTION_DAYS:-7} + # Which socket peers may name the client with cf-connecting-ip or + # x-forwarded-for, for the /ingest/client quota. Lantern has no env_file, + # so a value set in .env reaches the container only if it is named here — + # and this is a setting an operator would narrow believing it took effect. + # The default is the code's own (see clientip.js): the tunnel reaches the + # published port below from the compose bridge gateway or from loopback, + # and anything arriving from a public address speaks only for itself. + LANTERN_TRUSTED_PROXIES: ${LANTERN_TRUSTED_PROXIES:-loopback,private} # Published rather than loopback-only: the product switcher links here from # every front-end, so it has to be reachable through the tunnel. The admin # role is enforced on the token, not by the bind address. @@ -609,6 +659,28 @@ services: BEACON_HEARTH_MINER1_URL: http://hearth-testnet-miner1:8645 BEACON_HEARTH_MINER2_URL: http://hearth-testnet-miner2:8645 # + # The Ethereum JSON-RPC surface, which is a second listener in the same + # process on 8545 — the port `HEARTH_TESTNET_RPC` above points pay at, + # the one every eth_* call in the estate goes to, and the one that fails + # SOFT: evmnode.js logs `json-rpc listen failed` and carries on serving + # REST, so the node stays healthy and answers /info with a climbing + # height while nothing can read a balance from it. Probed separately from + # 8645 because "the node is up" and "the RPC is up" stopped being the + # same fact the moment there were two listeners. + BEACON_HEARTH_SEED_RPC_URL: http://hearth-testnet-seed:8545 + BEACON_HEARTH_MINER1_RPC_URL: http://hearth-testnet-miner1:8545 + BEACON_HEARTH_MINER2_RPC_URL: http://hearth-testnet-miner2:8545 + # The EIP-155 chain id every node here must report, in hex from + # eth_chainId and in decimal from net_version. 7412 is hearth-testnet; + # set 7411 for mainnet. Pinned rather than read from the node, because a + # monitor that adopts the value it is checking checks nothing. + # + # Defaults to the estate-wide `HEARTH_CHAIN_ID`, which the wallet and + # explorer bundle (`hearth-web`) also read: one setting, so a monitor + # asserting 7412 while the wallet signs for 7411 is not reachable by + # editing one line and forgetting the other. + BEACON_HEARTH_CHAIN_ID: ${BEACON_HEARTH_CHAIN_ID:-${HEARTH_CHAIN_ID:-7412}} + # # The journeys that need to be somebody, and the ones that move value. # Each is optional and each degrades to `skip` with its reason rather # than to `fail`, because a monitor that reports red for missing config @@ -668,6 +740,34 @@ services: <<: *svc build: repos/hearth/web container_name: cloudsforge-hearth-web + # The image ships defaults that already name these hosts, but they are + # written out here because they are the wiring this file is responsible for + # and because getting them wrong is invisible: the page loads, the chain is + # healthy, and every number on it is an error message. + # + # TWO PORTS, TWO PROTOCOLS. nginx.conf proxies /rpc/ to the REST + SSE + # server (8645 — /info, /events, /mining/*, used by mine.html) and /eth-rpc/ + # to the Ethereum JSON-RPC server (8545 — every eth_* call the explorer and + # wallet make). They are the same two listeners `BEACON_HEARTH_SEED_URL` and + # `BEACON_HEARTH_SEED_RPC_URL` above probe separately, for the same reason. + # Sending eth_* to 8645 gets HTTP 404 and a body that says "this is the REST + # API", which the client reports as MalformedResponse — so the explorer read + # as a dead chain while the chain was fine. + environment: + HEARTH_RPC_UPSTREAM: hearth-testnet-seed:8645 + HEARTH_ETH_RPC_UPSTREAM: hearth-testnet-seed:8545 + # The chain the bundle signs and banners for, templated into + # by nginx's sub_filter and read once by + # web/assets/chain.js. Same value and same reasoning as + # BEACON_HEARTH_CHAIN_ID: 7412 is hearth-testnet, which is the only chain + # this file runs, and it is pinned rather than read from the node because + # a wallet that adopts whatever chain id its RPC reports lets the RPC + # choose what its user's signature is bound to. + # + # `HEARTH_CHAIN_ID` is the estate's one setting for this: Beacon's + # `BEACON_HEARTH_CHAIN_ID` defaults to it too, so a mainnet deployment + # changes one line and the monitor and the wallet move together. + HEARTH_CHAIN_ID: ${HEARTH_CHAIN_ID:-7412} ports: - "8080:80" depends_on: diff --git a/docs/ecosystem/00-current-state.md b/docs/ecosystem/00-current-state.md new file mode 100644 index 0000000..aadecd9 --- /dev/null +++ b/docs/ecosystem/00-current-state.md @@ -0,0 +1,402 @@ +# 00 — Current state + +The baseline. Everything in this document was verified against source in July 2026 by +per-repository inspection, not read from documentation. Where a repository's own `MAP.md` +disagrees with its code, the code is recorded here and the drift is noted, because several +of those documents are now materially wrong and a plan built on them would be a plan for a +system that does not exist. + +Read this before any other document in `docs/ecosystem/`. Every decision in +[02-target-architecture.md](02-target-architecture.md) is a response to something recorded here. + +--- + +## 1. The estate in one page + +Eleven repositories. Nine hold product code, one (`stack`) composes them and carries two +real services in `infra/`, one (`.github`) is empty of code. + +| Repo | Deployables | Language | Purpose | Honest status | +| --- | --- | --- | --- | --- | +| `platform` | 3 — nimbus, site, admin | TS / Fastify 5 + React 19 | Identity (Nimbus), marketing site, operator console | Auth is solid. No MFA, no sessions, no orgs. | +| `forge-pay` | 1 — pay | TS / Fastify 5 | Shards, custodial coin balances, deposits, withdrawals, conversions, price oracle, entitlements, game shop | Works. **Not double-entry.** Custodial EMBER can be minted with no chain movement. | +| `forge-keyvault` | 1 — keyvault | TS / Fastify 5 | Custody: key generation, encryption, signing policy, treasury pinning | Correct signing policy. Root + Docker socket. Master secret unrotatable. | +| `forge-mint` | 1 — forge-mint (API + SPA) | TS / Fastify 5 + React 19 | ERC-20 deployment on 5 EVM chains | Deploys real contracts. Solana suspended. Mainnet closed by default. | +| `crucible` | 1 — crucible (API + SPA) | TS / Fastify 5 + React 19 | Backtesting, 10 strategies, paper + live bots, HWM performance fee | Engine complete. Live path off by default. 1 test file. | +| `ninety-days-after` | 2 — game, game-client | TS / Fastify 5 + React 19 | Asynchronous multiplayer survival simulation | Deep simulation, playable. **No game-platform abstraction at all.** | +| `hearth` | 3 + node + contracts + rust + desktop | JS (zero-dep) / React / vanilla | EMBER chain: PoW node, self-written EVM, explorer, wallet, site, faucet | Testnet in compose only. Mainnet not launched. Nothing published. | +| `asset-forge` | 0 | TS CLI | AI asset generation, 59 assets across 2 tracks | A build-time CLI in the product repo list by accident. Never deployed. | +| `shared-libs` | 0 | TS packages | `@cloudsforge/shared`, `@cloudsforge/ui` | 0.5.0/0.6.0 committed, **unpublished**; every consumer pins ^0.4.0/^0.5.0. | +| `stack` | 2 — lantern, beacon | JS (zero-framework) | Compose, routing, docs, ops console, status/journeys | Lantern and Beacon are real services living in a deployment repo. | +| `.github` | 0 | — | Org profile | Does not exist yet. | + +Eighteen containers, one host, one `docker-compose.yml` of 36 KB. `cv-web` serves a personal +site (`savvanis.life`) from the company stack. + +--- + +## 2. What is already right, and must not be rebuilt + +This list is short and load-bearing. Several proposals in earlier planning documents were +wrong because they did not read it. + +- **One account, genuinely.** One `users` table with no product column. RS256 tokens, single + audience `cloudsforge`, JWKS at a well-known path, verified independently by six services. + `platform/services/nimbus/src/db/schema.ts`, `tokens.ts`. +- **SSO handoff that is not an open redirect.** 60-second, single-use, origin-bound code, + hashed at rest, redeemed by a conditional `UPDATE … RETURNING` so a race cannot redeem it + twice; return URLs allowlisted. `nimbus/src/exchange.ts`. +- **Refresh-token family reuse detection** with a 10-second concurrent-tab grace window. +- **Signing-key rotation exists** — `active | published | retired` with a 20-minute publish + window, private JWK encrypted with AES-256-GCM under a scrypt-derived key. (`MAP.md` in + `platform` still claims there is one key and no rotation. It is wrong.) +- **Database per service, honoured.** Nine logical databases and no service reads another's + tables — verified by grep across all nine repos. Every cross-service read is HTTP. This is + the single most expensive property to retrofit and it is already true. +- **Real idempotency in the money path.** `withIdempotency` + (`forge-pay/services/pay/src/store.ts:153`) stores a request hash and the response body, + claims the key in the same transaction as the work, and replays stored JSON on a duplicate. + A different body under the same key is a 409. This is better than most production payment + code and its shape should be copied, not replaced. +- **A general billing primitive.** `/internal/charge`, `/credit`, `/trade`, + `/wallet/:userId` already let any peer service bill any user with a mandatory idempotency + key. Built for Crucible, Crucible-specific in nothing. +- **A signing policy that binds, not a signing oracle.** ForgeKeyvault's `/sign` gates on + purpose → binding (5 fields vs the stored row) → chain-id resolution → treasury pin, and + only then decrypts. A `deposit`-purpose key can sign exactly one shape (`sweep`) to exactly + one destination (the pinned treasury). `forge-keyvault/src/signing.ts`. +- **Deterministic simulation.** The game replays a world identically from a seed keyed on + `(world, day)`. `ninety-days-after/services/game/src/engine/resolve.ts`. +- **A conformance-tested EVM.** Hearth's self-written EVM passes VMTests 609/609, + GeneralStateTests 20,077/20,077, TransactionTests 188/188. +- **Beacon's journeys are already consumer-driven contract tests** — 24 multi-step functional + scenarios that skip rather than fail on missing secrets. This is the regression harness the + migration needs, and it already exists. + +--- + +## 3. What is actually wrong + +### 3.1 The system is a distributed monolith with good hygiene + +Correctly separated at the data layer, and structurally incapable of running more than one +copy of anything. Not "slow with two copies" — *incorrect* with two copies, in ways that lose +money. + +``` +$ grep -rn "pg_advisory\|SKIP LOCKED" repos/*/services infra/ +(no matches) +``` + +Eight `setInterval` timers do real work across three services, each guarded only by a +module-local boolean — a variable that is by construction invisible to a second process. + +| Service | Timer | What two replicas do | +| --- | --- | --- | +| pay | withdrawal worker | **Two withdrawals on one chain signed concurrently against the same nonce/sequence — one payment permanently lost** | +| pay | treasury sweeper | `signingBlocked` latch is per-process | +| pay | deposit watcher | Safe — address row `FOR UPDATE` + unique txid | +| pay | price oracle | Replicas quote different rates; an admin `PUT` updates one replica | +| pay | idempotency reaper | Harmless | +| crucible | bot tick | Bot state overwritten from a stale pre-trade snapshot | +| crucible | settlement sweep | **Double-billing** — `randomUUID()` settlement ids produce two different Pay idempotency keys | +| game | world tick | **Double XP and double `daysSurvived`** | + +`hasUnsettledOutbound()` (`pay/src/store.ts:1162`) is an unlocked read, so two workers both +pass it. `markWithdrawalSigned` protects a single row; it does not protect the chain's nonce. + +### 3.2 Money-losing defects live at one replica + +- **Crucible double-bills performance fees.** Settlement id is `randomUUID()`, so the Pay + idempotency key differs per attempt and Pay correctly honours both. `fee_settlements` has no + unique constraint. Races between the hourly sweep and `POST /bots/:id/actions {stop}`. +- **ForgeMint can mint a Solana token twice**, paying gas and rent both times — no + `onBroadcast`, and the `/status` settle path is gated on `chain.family === 'evm'`. (Solana + is currently suspended, which masks but does not fix it.) +- **A purchased private world is never built.** Pay debits 1,800–2,500 Shards and writes a + `private_world` entitlement; `grep -r private_world ninety-days-after/services/game/src` + returns zero. `rentPrivateWorld` is `ownOnce: false`, so it is repeat-chargeable. Pay's own + log line admits it. +- **No idempotency key on any game shop purchase.** All four are called directly from the + browser with none. `POST /spend` is the one money route that still accepts a missing key. +- **`convertCoinToEmber` credits custodial EMBER with no on-chain movement and no reserve + check**, and nothing anywhere reconciles custodial balances against custody holdings. + Custodial EMBER is withdrawable as real EMBER against coins the chain never saw. +- **`assignHomestead` lost update** — `UPDATE tiles SET owner_id` with no + `WHERE owner_id IS NULL`, so two concurrent joins land two players on one tile. + +### 3.3 There is no ledger + +`forge-pay`'s `ledger` table is single-sided: one `delta`, no account, no counter-account, +no journal grouping, no balancing invariant. Balances are materialised running columns. + +- Coin movements on the deposit-credit path write **no ledger row at all**; the audit trail + is `deposit_payments`. +- Withdrawal request, refund and convert-to-ember write `delta: 0` rows as breadcrumbs. +- There is **no reserved/available split** for user balances. A withdrawal debits immediately + and a refund credits back. +- `ledger.source` exists but only `/internal/*` writes it. **Per-product revenue is not + derivable today.** +- No reconciliation exists between Σ user balances and custody holdings, in any direction. + +This is the single largest gap in the estate and it is the reason +[02-target-architecture.md](02-target-architecture.md) makes a dedicated ledger service the +first new service built. + +### 3.4 There is no chain indexer + +Deposit detection is **balance-probing, not transaction indexing**: every 30 seconds, load +*every* address row with no pagination, call `eth_getBalance` at `latest - confirmations`, and +compare against a high-water mark. Consequences: + +- Synthetic txids (`depositPaymentTxid(coin, address, basis, total)`) — there is no real chain + transaction hash on a deposit anywhere in the system. +- No transaction history, no token transfers, no contract-deployment events, no reorg + detection, no finality model, no failed-transaction visibility. +- A regression in the observed balance freezes crediting for that address permanently until an + operator records a manual sweep by curl. There is no UI for that route. +- Bitcoin and Solana can neither be withdrawn nor swept — no output policy exists for either. + +### 3.5 Security posture + +| Finding | Where | Consequence | +| --- | --- | --- | +| `env_file: .env` on eight services hands each container **all 64 variables** | `docker-compose.yml` | The game container holds `KEYVAULT_MASTER_SECRET`. Blast radius of any compromise is total. | +| One shared `PAY_SERVICE_TOKEN` grants read/debit/credit/liquidate on **every** user | `pay/src/routes/internal.ts` | No per-caller identity; Pay's audit cannot say which service charged. | +| One shared `KEYVAULT_SERVICE_TOKEN` grants the whole custody peer surface | `forge-keyvault/src/auth.ts` | Any of three containers holding it can mint a treasury, sweep every deposit to it, then drain it. Documented as a residual risk in source. | +| `POST /admin/keys/:address/reveal` returns **any** private key in plaintext to any admin JWT | `forge-keyvault/src/routes/admin.ts:123` | Total-exfiltration primitive. Mitigation is one audit row. No approval, no rate limit, no scoping. | +| ForgeKeyvault runs as **root** with `/var/run/docker.sock` mounted read-write | `Dockerfile:29-48` | Any RCE is host takeover and full custody loss. Deliberate and documented. | +| `KEYVAULT_MASTER_SECRET` **cannot be rotated** | `crypto.ts` — `CURRENT_VERSION = 1`, no v2 branch, no re-encryption pass | A compromise is unrecoverable without regenerating every address. | +| No MFA anywhere | `platform/services/nimbus` | No TOTP, no WebAuthn, no recovery codes. | +| No session or device records | Nimbus | No "sign out everywhere", no device list, no new-device alert. | +| Nimbus's two admin proxies use bare `fetch` with no timeout | `routes/vault.ts:61`, `routes/pay.ts:73` | A hung keyvault pins the SSO service indefinitely. | +| XRP has no network binding | `forge-keyvault/src/chains.ts:141` | Same seed and address on testnet and mainnet; a signed Payment is submittable on either. | +| Flat bridge network, plaintext HTTP, no segmentation | `docker-compose.yml` | Anything on the bridge reaches `forge-keyvault:4005` and `pay:4003/internal`. | +| No rate limiting in pay, keyvault or forge-mint | — | Including unauthenticated public routes. | + +Two locks currently keep `/internal` off the internet: loopback binding in compose, and a +cloudflared path rule that returns 404 before the hostname rule. Both are asserted in CI, +because a previous configuration had `https://pay./internal/charge` live on the public +internet with one guessed token as the only protection. + +### 3.6 Operational immaturity + +- **`/health` is a liveness probe used as a readiness gate.** Every service returns a static + `{ok:true}` that never touches Postgres. A replica whose database is unreachable reports + healthy and 503s every request. `depends_on: service_healthy` across the estate rests on + this literal. +- **Migrations run in-process at boot without a lock.** Five repos ship a hand-rolled + `STEPS[]` of `CREATE TABLE IF NOT EXISTS` executed before `listen()`. No version table, no + down path, no advisory lock; failure is `process.exit(1)`. Two replicas booting together + race on `pg_class`, one raises 23505 and crash-loops. Scale-up is not slow — it is + impossible. +- **Nimbus has a split-brain signing key.** On a fresh database two replicas both generate a + keypair with different random `kid`s, `onConflictDoNothing()` conflicts on nothing, both rows + insert, and `getJwks()` does `select().limit(1)` with **no `ORDER BY`**. +- **Zero events, zero outbox, zero queue, zero retries, zero circuit breakers.** Consistency + between services is HTTP status codes plus hand-written caller-side compensation, reinvented + per call site. +- **Long work runs inside the request.** ForgeMint awaits a chain deploy for up to 180 seconds + inside the HTTP request; Crucible runs backtests synchronously inside the POST. Both are + killed by the 10-second force-exit on SIGTERM. +- **Eighteen `container_name:` entries and fixed host ports** mean `deploy.replicas` is + rejected outright. The compose file cannot express a second copy of anything. +- **Lantern collects by tailing the host Docker daemon**, so it survives neither a second host + nor Kubernetes — and it is the tool you need most on the day you move. +- **Nothing backs itself up.** `infra/backup.sh` exists, is unscheduled, and writes locally. +- **A release is seven hand-pushed git tags.** `CLOUDSFORGE_TAG` is one value shared by + fourteen images built by seven repositories; a commit sha cannot work, and a repo you forget + has no image at that version. + +### 3.7 Duplication and drift + +| Thing | Copies | Note | +| --- | --- | --- | +| `services/*/src/obs.ts` | 5 byte-identical (375 lines) + 1 fork (428) | md5 `2fcb6c10…`. Meant to be hand-recopied. | +| `apps/*/src/lib/obs.tsx` | 6 byte-identical (261 lines) | md5 `13c5932c…`. | +| Nimbus JWKS auth middleware | 5 divergent implementations (54–93 lines) | pay, game, forge-mint, crucible, keyvault. | +| `--cf-*` design tokens | source + hearth `web/assets/vendor/` (now **differs**) + lantern + beacon inline + nimbus portal string | Five copies, two already drifted. | +| App UI primitives (`components/ui.tsx`) | 3, all different | ninety-days-after 228 lines, crucible 194, forge-mint 146. | +| `env.ts` | 7, all hand-written | — | +| Ember orange | 5 values shipping | `#e8622c` canonical; `#d9812f`, `#ff5a1e`, `#ff7a2f`, `#ff4d00` also live. | + +`@cloudsforge/shared` 0.5.0 and `@cloudsforge/ui` 0.6.0 are committed and **unpublished**. +CI's `NPM_TOKEN` is dead. Every consumer pins `^0.4.0` / `^0.5.0`, and caret on `0.x` is +patch-only, so **no consumer can resolve the current version**. A minor bump costs one hand +publish plus 16 file edits across 8 repos, because these workspaces set +`verifyDepsBeforeRun: error` and a version missing from `minimumReleaseAgeExclude` breaks +every command in the repo, not just install. + +### 3.8 Sold and undelivered + +Ordered by how bad it looks if a customer finds it. + +1. **Private worlds** — 1,800–2,500 Shards, provisioned by nobody, repeat-chargeable. +2. **Four convenience items and three cosmetic kinds** — Pay sells them, nothing renders or + delivers them. The game client withheld the listings; **Pay's routes are still live**, so a + direct caller is still charged. +3. **Season pass** — 500 Shards, three ids unlocked wholesale, two of the three are undrawable + kinds, and there is no progression track. +4. **ForgeMint "verified metadata" and "liquidity-lock helper"** — in `MINT_OFFERS`, implemented + nowhere. +5. **`tokens` in the game have no sink** — awarded by objectives, spendable on nothing. +6. **Crucible earns nothing on a default deploy** — `CRUCIBLE_LIVE_ENABLED=false`. +7. **No refunds exist anywhere.** `/internal/credit` has no caller in the estate. + +### 3.9 Test coverage + +| Repo | Framework | Files | Tests | What is untested | +| --- | --- | --- | --- | --- | +| platform | `node --test` + tsx | 11 | ~72 | All admin/vault/pay route handlers; no E2E, no render tests | +| forge-pay | `node --test` | 9 | 102 | **Not wired into CI.** No DB, no HTTP, no integration | +| forge-keyvault | `node --test` | 4 | — | Bitcoin PSBT, `/sign` E2E, reveal, crypto envelope, FileVault | +| forge-mint | `node --test` | 4 | 37 | No DB, no lifecycle E2E, no contract-execution tests | +| crucible | `node --test` | **1** | 16 | **The engine, indicators, metrics, routes, runner, store, feed, SPA** | +| ninety-days-after | `node --test` | 6 | ~43 | The tick engine, every route handler, every component | +| hearth | hand-rolled | 31 suites | 20,874 vectors | Conformance is excellent; run by Beacon, not by Hearth's CI | +| shared-libs | CI pack/consume | — | — | No unit tests | +| lantern / beacon | `node --test` | 4 (new, uncommitted) | — | — | + +The one product whose core claim is numerical correctness — Crucible — has a single test file, +and it tests billing, not the engine. + +--- + +## 4. Functionality inventory — what already exists + +This is the answer to "identify if some of this already exists". Read it before proposing +anything in [08-prioritised-backlog.md](08-prioritised-backlog.md). + +| Required ecosystem capability | Exists today? | Where, and what is missing | +| --- | --- | --- | +| Registration, login | **Yes** | Nimbus. Missing: email verification, account deletion. | +| Single sign-on across products | **Yes** | Origin-bound handoff code. Missing: OIDC conformance for third parties. | +| Account settings / profile | **Partial** | Nimbus portal `/account` renders initials, handle, email, roles and a *hardcoded* launcher grid. | +| Security settings | **No** | No MFA, no device list, no session list, no security log. | +| Session management, connected devices | **No** | `refresh_tokens` rows exist; nothing surfaces or names them. | +| Account recovery | **Partial** | Password reset by email token, 30 min, single use. No MFA recovery codes, no account-recovery flow. | +| Product entitlements | **Partial** | `entitlements` table in Pay. Bearer-only — **no service can ask "does this user own X"**. No expiry, no revocation, no product dimension. | +| Notification preferences | **No** | Nothing. | +| Organisations / teams | **No** | Roles are `player \| admin` on a text array. | +| Unified product navigation | **Yes** | `CloudsForgeBar` + `ProductSwitcher` in all six SPAs, derived from one registry. | +| Unified dashboard | **No** | Nothing aggregates across products. | +| Unified activity feed | **No** | Each product has its own list. No cross-product feed, no event bus. | +| Managed wallets per chain family | **Yes** | Custody mints one flat random key per address for EVM, Ember, Solana, Bitcoin, XRP. **No HD derivation, no mnemonic.** | +| Mainnet/testnet separation | **Partial** | A column exists and is bound in signing. XRP has no network binding at all. | +| Multiple wallets, labels, primary wallet | **No** | One address per (user, coin, network), deterministic order id. | +| Wallet lifecycle (freeze, retire) | **No** | No `DELETE`, no status column. | +| External wallet connection | **No** | Nothing. | +| Wallet ownership verification / signed challenge | **No** | Nothing. | +| Receive: address, QR, pending, confirmations | **Partial** | Address + pending/confirmed counts. No QR, no explorer link on deposits (synthetic txids). | +| Send: fee review, confirm, track, explorer | **Partial** | Works for EMBER/ETH/XRP. Not for BTC or SOL. Stuck withdrawals need a curl-only admin route. | +| Private-key access / export | **Admin-only** | `POST /admin/keys/:address/reveal` returns any key to any admin. **The user cannot access their own key.** | +| Key import, rotation, recovery phrase | **No** | Nothing. | +| Double-entry ledger | **No** | §3.3. | +| Balance reservations | **No** | §3.3. | +| Reconciliation | **No** | §3.3. | +| Chain indexer | **No** | §3.4. | +| Reorg handling | **No** | §3.4. | +| Asset / brand generation | **Yes, as a CLI** | asset-forge: 30 brand + 29 game assets, `gpt-image-1`, macOS-only sizing, no API, never deployed. | +| Token deployment (EVM) | **Yes** | ForgeMint, 5 chains, 3 tiers, customer wallet owns the token, keyvault signs. | +| Token deployment (Solana) | **Suspended** | Code kept, unreachable — keyvault refuses `SetAuthority`. | +| Token page / project page | **No** | Order detail only. | +| Marketplace | **No** | Nothing. Not a single listing, offer, auction or escrow primitive exists. | +| Trading: backtest, paper, live, HWM fee | **Yes** | Crucible. Live settles against Pay's oracle, not an exchange. | +| Cross-bot portfolio, P&L export | **No** | Per-bot only. | +| Game platform (titles, shared profile, inventory) | **No** | One title, no abstraction, no `game_id` anywhere. | +| Achievements, objectives, seasons | **Yes, per-world** | Ninety Days After only. | +| Cross-game assets | **No** | — | +| Blockchain explorer | **Yes** | Hearth `web/` — hash-routed vanilla ES modules, decodes logs and ERC-20 transfers. | +| Non-custodial wallet | **Yes** | Hearth `web/wallet.html` — PBKDF2 600k + AES-256-GCM in localStorage. No seed phrase, no recovery. | +| Node + mining software | **Yes** | Homefire PoW, LWMA retarget, browser miner with a dashboard. No pools. | +| Faucet | **Built, undeployed** | `hearth/tools/faucet`, 66 checks, not in compose. | +| RPC docs, SDK | **Partial** | 41 Ethereum JSON-RPC methods. `@cloudsforge/hearth-node` 0.2.0 exports **UTXO-era** APIs only and has zero consumers. | +| Developer platform | **No** | No API keys, no OAuth clients, no webhooks, no sandbox, no docs site. | +| Notifications | **No** | One outbound SMTP path (password reset) and one Beacon incident webhook. | +| Risk / policy engine | **Partial, scattered** | Custody's purpose gate and treasury pin are a real policy layer for signing. No limits, velocity, approvals, freezes or device risk anywhere. | +| Communities, governance, treasuries | **No** | `communes` exist inside one game and **are inert** — `resolve.ts` never reads them. | +| Admin console | **Partial** | Users list + role change, vault keys, prices, withdrawals. No user create/suspend/delete. Two critical remedies are curl-only. | +| Operations console | **Yes** | Lantern: log ingest, error grouping, request-id trace, browser errors. | +| Status / synthetic monitoring | **Yes** | Beacon: 27 probes, 24 journeys, Prometheus `/metrics`, incidents, chain conformance. | +| Billing / subscriptions / invoices | **No** | Entitlements are grant-only. No subscription concept anywhere. | +| Creator payouts, revenue share | **No** | Nothing. | +| Analytics | **No** | No product analytics, no funnels, no cohorts. | + +--- + +## 5. Conflicting domain models + +Four collisions that must be resolved before any new product is built. Each is recorded here +so [04-domain-model.md](04-domain-model.md) can point at what it is replacing. + +1. **"Wallet" means three different things.** A `wallets` row in Pay (a Shards balance), a + `vault_addresses` row in custody (a keypair), and Hearth's browser wallet (a localStorage + keystore). The Forge Hub has to present one concept to a user. +2. **"Balance" has no owner.** Pay holds `wallets.shards` and `coin_balances.amount` as running + columns; custody holds the keys; the chain holds the truth; nothing compares them. Three + candidate sources of truth, no designated one. +3. **"Product" is declared in eight places** and has already drifted — the shared registry, the + Nimbus portal (twice, with different copy), a stale static page, the marketing site, the + vendored browser-obs copies, the compose anchors, and three hardcoded lists in CI. + `pull-all.sh` omits `crucible` entirely. +4. **Game rules live in a platform contract package.** `@cloudsforge/shared`'s `game.ts` is 535 + lines holding `SKILL_PERKS`, `survivalScore`, `xpToNext` and `communeWithdrawCap` — domain + logic shared between one game server and its own client, released on the same cadence as the + deposit registry that Pay and custody must agree on byte-for-byte or money is credited at + the wrong depth. + +--- + +## 6. Technical debt register + +Carried forward into [08-prioritised-backlog.md](08-prioritised-backlog.md) as `TD-*` items. + +| # | Debt | Cost of leaving it | +| --- | --- | --- | +| TD-01 | No coordination primitives (no lease, no queue, no leader election) | Cannot run two replicas of anything | +| TD-02 | Boot-time DDL with no version table | Cannot scale up; cannot roll back a schema | +| TD-03 | Single-sided ledger | Cannot reconcile, cannot report revenue, cannot dispute | +| TD-04 | Balance-probe deposit detection | No real txids, no history, no reorg handling | +| TD-05 | Shared service tokens | No per-caller audit; total blast radius | +| TD-06 | `env_file: .env` fan-out | Every container holds every secret | +| TD-07 | Vendored `obs.ts` / auth middleware ×5–6 | Every cross-cutting fix is 8 PRs | +| TD-08 | Dead `NPM_TOKEN`, `0.x` versions, `minimumReleaseAgeExclude` ritual | A contract change costs 16 file edits | +| TD-09 | SPAs inside API processes | A CSS change redeploys the trading engine | +| TD-10 | `container_name` + fixed ports | `deploy.replicas` is rejected | +| TD-11 | Static `/health` used as readiness | Rolling deploys are lossy | +| TD-12 | Long work inside HTTP requests | Deploys kill in-flight chain deploys | +| TD-13 | Lantern's Docker-socket collector | Does not survive a second host | +| TD-14 | Master secret unrotatable | Compromise is unrecoverable | +| TD-15 | `.cf31-head/` — 23 tracked stale duplicate source files in the game repo | Dead weight in git | +| TD-16 | Hearth's `SPARKS_PER_EMBER` still `1e8` while 18 decimals are specified | Latent unit bug | +| TD-17 | Hearth `site/` copy still tells the UTXO story after the EVM migration | Public page describes a chain that no longer exists | +| TD-18 | `@cloudsforge/hearth-node` exports UTXO-era APIs and has zero consumers | Published package is a lie | +| TD-19 | ~38 MB of unreferenced generated art across the estate | — | +| TD-20 | 12 art masters at 1024² against declared 512²/256² | — | + +--- + +## 7. Documentation drift found during this audit + +Recorded because these documents were used as inputs by earlier plans and are wrong. + +- `platform/MAP.md` §2.7, §7, §8 — claims no mail transport exists, that reset links are built + from the request Host, and that there is no signing-key rotation. All four claims are false + in current source. +- `forge-pay/MAP.md` — "33 handlers"; there are 35. Schema comment says treasury purpose is + `'deployer' today`; `treasury.ts:63` uses `'treasury'`. +- `forge-keyvault/MAP.md` — inline line numbers stale by ~90 lines. +- `hearth/MAP.md` §4.6, §10 — claims `jsonrpc/server.js` "is never constructed"; it is mounted + at `node/src/evmnode.js:232`. Test count says 27; there are 31 suites. +- `stack/BLOCKCHAIN.md` §1 — describes Hearth as "a pure key-to-address UTXO chain". Hearth is + now an account-model EVM chain with `0x` addresses and secp256k1. The document's entire + premise (add output predicates, not a VM) is obsolete: the VM already exists and passes + conformance. +- `stack/ECOSYSTEM.md` — describes Forge Pay as settling "through a mock provider"; the invoice + path was deleted and payments are on-chain deposit only. +- `asset-forge/MAP.md` says 27 game assets, `ECOSYSTEM.md` says 33 total; source has 59. + +The superseded platform documents (`PLAN.md`, `ECOSYSTEM.md`, `MICROSERVICES.md`, +`upgrade.md`, `audit.md`) are removed as part of this work. Their still-true content is folded +into this directory; `MICROSERVICES.md`'s Part I analysis survives as §3.1–§3.6 above and its +Part II design survives in [02-target-architecture.md](02-target-architecture.md). diff --git a/docs/ecosystem/01-product-vision.md b/docs/ecosystem/01-product-vision.md new file mode 100644 index 0000000..9858093 --- /dev/null +++ b/docs/ecosystem/01-product-vision.md @@ -0,0 +1,150 @@ +# 01 — Product vision + +What CloudsForge is for, what the products are, and what makes them one platform rather than +nine applications sharing a name. This document sets the intent that +[02-target-architecture.md](02-target-architecture.md) then serves. + +--- + +## 1. The thesis + +> **One crypto world. Mine it, hold it, forge it, trade it, sell it, play in it, build on it.** + +Almost every consumer crypto platform is an exchange with features bolted on. CloudsForge is +the inverse: a set of things worth doing, funded by a currency you can produce yourself on a +laptop, with the account, the wallet and the ledger shared across all of them. + +The loop is the product. Every arrow already exists in code except the last two: + +``` + mine EMBER on your own CPU Hearth — Homefire PoW, no farms, no pools + │ + ▼ + deposit into your CloudsForge wallet custody mints the key, the indexer confirms it + │ + ▼ + hold, convert, or reserve one ledger, double-entry, one portfolio + │ + ├──► forge a token or a brand Forge Create + ├──► run a strategy Forge Trade + ├──► play, earn, own Forge Worlds + ├──► sell it, buy someone else's Forge Market ← does not exist yet + └──► build on all of it Developer Platform ← does not exist yet + │ + ▼ + withdraw back out on-chain, or to one activity history, one set of notifications + your own external wallet +``` + +A CPU-mineable coin that is the actual funding rail for real products is a story no one else +can tell. It is roughly 80% built and 0% marketed. + +## 2. What "one platform" has to mean concretely + +The test is not whether the products share a logo. It is whether these eleven statements are +true. Today, three are. + +| # | Statement | Today | +| --- | --- | --- | +| 1 | One account signs into everything, once. | **True** | +| 2 | One identity — the same profile, handle and reputation everywhere. | Partly — one `users` row, but no profile beyond a handle | +| 3 | One wallet experience — the same receive, send and key screens whichever product you came from. | False — three different "wallet" concepts | +| 4 | One portfolio — a single number that is the truth about what you hold. | False — nothing aggregates | +| 5 | One activity history — every account, money, asset, game and governance event on one timeline. | False — no cross-product feed exists | +| 6 | One internal economy — Shards and EMBER spend and earn identically in every product. | Partly — Shards are universal; nothing earns them | +| 7 | Assets you create in one product are usable in the others. | False | +| 8 | One set of notifications, with one preference page. | False | +| 9 | One operator view — a support agent can answer any question from one place. | False | +| 10 | One financial source of truth that reconciles against the chain. | False | +| 11 | A third party can build on all of it. | False | + +Every phase in [06-ecosystem-workflow.md](06-ecosystem-workflow.md) is justified by which of +these eleven it moves from false to true. A phase that moves none of them does not ship. + +## 3. The products + +Seven customer-facing products, one control centre, one developer surface. Everything else is +spine — and spine must never appear in a product grid as a peer, because an account is not +something a person chooses, it is something they are given. + +| Surface | Verb | What it is | Built from | +| --- | --- | --- | --- | +| **Forge Hub** | — | The control centre: dashboard, portfolio, wallet, activity, settings, security. The default landing place after sign-in. | New — `hub-api` + `hub-web` | +| **Forge Network** | Mine | The EMBER chain: node, mining, explorer, faucet, network stats, RPC and SDK. | `hearth` | +| **Forge Create** | Forge | Brand generation, token deployment, project pages, launch flow. | `asset-forge` + `forge-mint` | +| **Forge Market** | Sell | Discovery, listings, auctions, offers, escrow, creator and project profiles. | New | +| **Forge Trade** | Trade | Backtesting, strategy catalogue, paper and live bots, performance reporting. | `crucible` | +| **Forge Worlds** | Play | The game platform. *Ninety Days After* is its first title, not its definition. | `ninety-days-after`, generalised | +| **Forge Pay** | Spend | Wallet, deposits, withdrawals, conversions, Shards. Presented **inside Forge Hub**, not as a separate destination. | `forge-pay`, decomposed | +| **Developer Platform** | Build | Projects, API keys, webhooks, SDK, CLI, sandbox, application directory. | New | + +**Spine, never a product:** identity, ledger, custody, indexer, policy, activity, +notifications, billing, gateway, Lantern, Beacon. + +### Why Forge Pay stops being a destination + +Forge Pay is the best-built thing in the estate and the worst-positioned. Nobody wakes up +wanting to visit a payments product. Its screens — balance, receive, send, convert, history — +are exactly the screens Forge Hub owes the user on arrival. So Forge Pay becomes the *engine* +under Hub's wallet tab, and the standalone `pay.` surface is retired from the product +switcher. The API keeps its name and its repo lineage; the destination goes away. + +## 4. The intended journey, end to end + +This is the journey named in the brief, mapped onto the surfaces that serve each step and the +phase that makes it work. Detail is in [05-user-journeys.md](05-user-journeys.md). + +| Step | Surface | Phase | +| --- | --- | --- | +| Register | Identity → Hub | P6 | +| Create or connect wallets | Hub · Wallet | P6 | +| Fund account | Hub · Receive → indexer confirms | P6, P7 | +| Hold assets | Hub · Portfolio | P6 | +| Create assets | Forge Create | P8 | +| Use assets | Forge Worlds, Forge Trade | P10 | +| Trade or sell assets | Forge Trade, Forge Market | P9, P10 | +| Earn rewards | Forge Worlds, Forge Network, Forge Market | P10 | +| Participate in communities | Communities & governance | P12 | +| Withdraw assets | Hub · Send | P6, P7 | +| Build integrations | Developer Platform | P11 | +| Return through notifications | Notifications | P13 | + +## 5. Principles + +These are the tie-breakers. When two designs are defensible, the one that satisfies more of +these wins. + +1. **The ledger is the source of truth for value; the chain is the source of truth for + ownership.** Where they disagree, the system stops and tells an operator. It never guesses. +2. **A user can always leave with their assets.** Private-key access for a wallet the user + owns is a product requirement, not a favour. The safeguards are ours to design; the right + is not ours to withhold. +3. **Do not sell what cannot be delivered.** Every SKU has a code path that delivers it, or the + SKU is withdrawn — including from the API, not just from the UI. +4. **Nothing that widens authority ships before the thing that bounds it.** The estate already + follows this rule: the treasury pin shipped in the same change as the sweep shape. +5. **Honest copy.** "Modelled — not a promise." "Fees and slippage charged, because a strategy + that only works for free does not work." This voice is an asset. Protect it. +6. **No pay-to-win.** In Forge Worlds, purchasable means cosmetic, convenience or access — + never power. Scarcity is the game. +7. **One system, many accents.** The warm ash/ember palette is distinctive. New products get an + accent, not a new visual language. +8. **Prefer the simplest architecture that preserves clear ownership, data integrity and + security boundaries.** Do not create a service per capability. +9. **Reversibility beats cleverness.** Every phase ships behind a flag with a stated rollback. + +## 6. What this vision explicitly rejects + +- **A CloudsForge exchange.** Order books, custody of other people's trading pairs and market + making are a different company with a different regulatory posture. Forge Trade settles + against a price oracle on coins we already custody; that is the boundary. +- **Selling the spine.** "CloudsForge ID" and "CloudsForge Wallet" as products is how you end + up with nine things nobody wants. +- **A rewrite of Hearth or the custody model.** Both are correct for what they are. Hearth's + chain node stays a stateful singleton; custody stays single-replica. Say so out loud rather + than discovering it during a migration. +- **A new product before the ecosystem works.** Forge Market is the one exception, and only + because it is the missing verb — "sell" — without which "create" has no destination. +- **Monetising the spine.** Wallets, transfers, key access, activity history and the account + are free forever. Revenue comes from creation, trading, play and developer scale. See + [15-monetisation-model.md](15-monetisation-model.md). diff --git a/docs/ecosystem/02-target-architecture.md b/docs/ecosystem/02-target-architecture.md new file mode 100644 index 0000000..e6eeb44 --- /dev/null +++ b/docs/ecosystem/02-target-architecture.md @@ -0,0 +1,559 @@ +# 02 — Target architecture + +The system CloudsForge is being transformed into, and the reasoning for every structural +choice. This document has decision authority: where it disagrees with any repository's +`MAP.md`, or with the deleted `MICROSERVICES.md`, this document wins. + +Read [00-current-state.md](00-current-state.md) first. Every decision below is a response to a +specific finding there, cited as `§n`. + +--- + +## 1. Shape of the target system + +``` + ┌───────────────────────────┐ + browser ────────────────────────│ gateway (Traefik) │ + │ TLS · CORS · routing │ + │ /internal refusal │ + │ serves every SPA │ + └─────────────┬─────────────┘ + │ + ┌────────────── edge / experience ─────────────┼────────────────────────────┐ + │ hub-web site market-web mint-web trade-web worlds-web explorer-web │ + │ admin-web devportal-web │ + └───────────────────────────────────────────────────────────────────────────┘ + │ + ┌────────────── aggregation ───────────────────┼────────────────────────────┐ + │ hub-api (BFF) admin-api (operator BFF) devplatform (public)│ + └───────────────────────────────────────────────────────────────────────────┘ + │ + ┌────────────── domain services ───────────────┼────────────────────────────┐ + │ │ + │ identity policy │ ledger wallet settlement pricing billing│ + │ activity notify │ custody indexer │ + │ │ │ + │ studio mint market trade worlds nda community │ + │ │ + └───────────────────────────────────────────────────────────────────────────┘ + │ + ┌────────────── substrate ─────────────────────┼────────────────────────────┐ + │ Postgres (database per service) · event bus (outbox → HTTP → inbox) │ + │ Hearth chain nodes (stateful, unreplicated) · external chain RPC │ + │ lantern (logs) beacon (synthetic) prometheus otel collector │ + └───────────────────────────────────────────────────────────────────────────┘ +``` + +Twenty-two backend services, nine frontends, one gateway. Every box is independently +deployable, independently versioned, and owns its own database. No service reads another's +tables — the one property the current estate already has and that this design refuses to give +up. + +--- + +## 2. The eighteen decisions + +Each states the decision, the alternatives considered, and why they were rejected. These are +the decisions the brief assigned to this role. + +### AD-01 · Repository topology: one repository per deployable + +**Decision.** One Git repository per independently deployable unit — 22 service repos, 9 +frontend repos — plus four library repos, `hearth`, `asset-forge`, `stack` (deployment) and +`.github` (org). Roughly 38 repositories. Full list in +[03-repository-responsibilities.md](03-repository-responsibilities.md). + +**Alternatives rejected.** +- *Platform monorepo with 24 deployables.* Genuinely cheaper: a contract change is one atomic + PR, CI tests the real integration, one tag is one rollback. It was the recommendation of the + deleted `MICROSERVICES.md`. **Rejected by the owner** in favour of maximum independence of + release and ownership. +- *Repo per bounded context (~8 repos).* The middle ground. Also rejected for the same reason. + +**What this decision costs, stated plainly.** With 38 repos, a `@cloudsforge/contracts-*` +minor bump today would cost ~48 file edits, 24 manual npm publishes, and no CI anywhere could +test the composed system. That cost is not acceptable, so choosing this topology *obliges* +the machinery in AD-02, AD-03 and AD-04. **Those three are not optional and they are the +critical path of Phase 2.** A polyrepo without them is worse than the current estate, not +better. + +### AD-02 · Contract distribution is automated, or the topology fails + +**Decision.** Four things, all in Phase 2, before any repository is split: + +1. **Fix publishing.** Publish to GitHub Packages authenticated with the workflow's own + `GITHUB_TOKEN`. The dead `NPM_TOKEN` (§3.7) is the root cause of every manual release + ritual in the estate. +2. **Go to 1.x.** Caret ranges on `0.x` are patch-only, which is why no consumer can resolve + the current contract version today. Every published package moves to `1.0.0` and evolves + additively. +3. **Renovate at org level**, grouped per contract package, auto-merging on green CI. This is + what replaces the 48 hand edits. It also removes `minimumReleaseAgeExclude` as a manual + ritual: Renovate maintains it, or the constraint is dropped for `@cloudsforge/*` scoped + packages specifically (they are first-party; the release-age gate exists to defend against + supply-chain attacks on third-party packages). +4. **Contract compatibility CI.** Every contract repo runs a `schema-diff` job that fails on a + removed field, a narrowed type or a renamed key. Additive-only is enforced by a check, not + by discipline (§3.7 shows discipline already failed). + +**Consequence.** Contract packages are **versioned artifacts with a compatibility contract**, +not shared source. A service may lag a contract version by up to two minors; the schema-diff +check is what makes that safe. + +### AD-03 · Organisation machinery replaces monorepo conveniences + +**Decision.** Build in Phase 2, in the `.github` and `stack` repos: + +| Thing | What it replaces | +| --- | --- | +| `cloudsforge/.github` reusable workflows: `service-ci.yml`, `web-ci.yml`, `publish.yml`, `secret-hygiene.yml` | 11 copies of near-identical CI, already drifted | +| `cloudsforge-service-template` + `cloudsforge-web-template` repos | Bootstrapping a new service by copy-paste | +| `cfctl` CLI in `stack` — `cfctl clone`, `cfctl up`, `cfctl release`, `cfctl doctor` | `clone-all.sh` / `pull-all.sh`, which already omit `crucible` | +| **Release manifest** — `stack/releases/.yaml` pinning an image tag per service, generated by CI, validated on `up` | `CLOUDSFORGE_TAG`, which cannot name a version across seven repos (§3.6) | +| Org-level branch protection, secret scanning, push protection, 2FA, dependency graph | All currently off | + +**The release manifest is the single most important item.** With one repo per service there is +no shared version. A release is a *manifest*, not a tag: a generated file naming exactly which +image of each of 31 services is in this release, produced by CI, committed, and the only thing +a deployment reads. Rollback is checking out the previous manifest. + +### AD-04 · Integration is tested by Beacon, not by CI in any single repo + +**Decision.** No repository can test the composed system, so the composed system is tested by +a **contract-test + synthetic-journey pair**: + +- **Consumer-driven contract tests** — each consumer publishes an expectation file to the + provider's repo; the provider's CI replays every consumer expectation against its real + handlers. A provider cannot merge a change that breaks a recorded consumer. +- **Beacon becomes the integration gate.** It already runs 24 multi-step journeys against a + live stack. Every release candidate deploys to a staging environment and must pass the full + journey suite before its manifest is promoted. Beacon's journeys grow with each phase; the + phase exit criteria in [06-ecosystem-workflow.md](06-ecosystem-workflow.md) name the + journeys each phase must add. + +This is the answer to "validate that existing functionality didn't break". It is stated in +full in [14-testing-strategy.md](14-testing-strategy.md). + +### AD-05 · Forge Hub is a new application, not a page in the identity service + +**Decision.** Forge Hub is `cloudsforge-hub-web` (SPA) + `cloudsforge-hub-api` (BFF). The +Nimbus server-rendered portal is reduced to authentication screens only — login, register, +forgot, reset, consent — and `/account` moves to Hub. + +**Why not inside `platform`/Nimbus.** Nimbus is the highest-security service in the estate: it +mints identity for every product and holds the private JWK. Hub is the richest, +fastest-changing UI surface and needs to fan out to ten services. Putting them in one process +means a dashboard tweak redeploys the token issuer, and it puts an aggregation surface inside +a security boundary. Separating them also lets Hub's BFF hold a *scoped* service credential +per upstream rather than being trusted with everything. + +**Why a BFF at all rather than the SPA calling ten services.** The dashboard needs portfolio +(ledger), balances (wallet), pending deposits (indexer), bots (trade), listings (market), +rewards (worlds), tokens (mint), activity (activity), notifications (notify), security +(identity). Ten cross-origin round trips with ten token exchanges is a bad first paint and a +CORS matrix that nobody can reason about. The BFF makes it one request, one cache policy, one +place to degrade gracefully when an upstream is down. + +### AD-06 · The ledger becomes a separate service, and Forge Pay is decomposed + +**Decision.** `forge-pay` is split into five services: + +| Service | Owns | +| --- | --- | +| `ledger` | Double-entry accounting: accounts, journal entries, postings, balances, reservations, settlement states, reversals, reconciliation, financial reporting | +| `wallet` | The user-facing money API: portfolio, deposits, receive addresses, withdrawal requests, conversions, transfers. Orchestrates ledger + custody + indexer. Owns no balances. | +| `settlement` | Outbound chain work: treasury, sweeps, transaction building, signing requests, broadcast, confirmation tracking, stuck/abandon handling | +| `pricing` | Market and administered prices, median oracle, rate history, valuation | +| `billing` | Products, prices, entitlements, subscriptions, usage records, invoices, creator payouts, revenue share | + +**Why a separate ledger rather than an isolated module inside Pay.** Three reasons, all +observable in §3.3: +1. The ledger must eventually be the financial source of truth for market settlements, trading + P&L, game purchases, creator revenue, community treasuries and platform revenue. Every one + of those has a different service as its business owner. A ledger owned by the payments + product will keep acquiring payments-shaped assumptions. +2. The ledger's change cadence must be *slow*, its review bar *high*, and its access *narrow*. + Pay's user-facing routes change constantly. They should not share a deployment. +3. Reconciliation and audit need a service that no product can write to except through a typed + posting API. A module in the same process cannot enforce that. + +**Model.** Double-entry, append-only journal, balances derived and materialised in a +`balances` projection rebuilt from the journal. Every posting carries an idempotency key, an +originating service, an actor, and a correlation id. Reversals are new entries, never updates. +Detail in [04-domain-model.md](04-domain-model.md). + +**Not event-sourced in the CQRS sense.** The journal *is* an event log for accounting purposes +and that is sufficient. A full event-sourced aggregate store adds replay tooling, snapshotting +and version migration for no gain over an append-only journal with a materialised projection. + +**Separate operational and accounting ledgers: no.** One journal, with account *types* that +distinguish user liability, platform revenue, chain custody, fee income, community treasury +and clearing. Two ledgers is two truths. + +### AD-07 · Forge Indexer is a separate service with per-family workers + +**Decision.** `cloudsforge-indexer` — one service, one database, one normalised schema, and a +worker process per chain family (EVM, Ember, Solana, Bitcoin, XRP). It replaces balance-probing +(§3.4) entirely. + +**Why not part of Forge Pay.** Because the indexer's consumers are not only Pay: Forge Market +needs settlement events, Forge Create needs contract-deployment confirmations, Forge Network +needs the explorer's data, Forge Hub needs transaction history, and the developer platform +needs address-activity webhooks. An indexer inside the payments service is an indexer that +five other products cannot use. + +**Why not part of custody.** Custody must have the smallest possible attack surface and no +outbound network dependency on twelve RPC providers. + +**Why not chain-specific services.** Five services with one normalised output is five +deployment units, five sets of retry logic and five reorg implementations. One service with +five worker *modules* sharing checkpointing, provider failover, reorg recovery and the +confirmation policy is materially simpler and is where the hard code actually is. + +**Hearth is a first-class family, not a special case.** Hearth exposes Ethereum JSON-RPC on +8545, so the EVM worker serves it with a different chain id and a different confirmation +depth (60). This is a direct dividend of Hearth's EVM migration. + +### AD-08 · Notifications are a service; delivery channels are adapters + +**Decision.** `cloudsforge-notify` owns the event → notification mapping, user preferences, +templates, localisation, deduplication, digests, delivery retries, delivery history and +operator broadcasts. Channels (in-app, email/SMTP, web push, mobile push, SMS, developer +webhooks) are adapters behind one interface. + +**Why not part of identity.** Identity already sends one email (password reset) and the +temptation is obvious. But notification preferences are a *product* surface with categories and +priorities per product, and the fan-in is the entire event bus. Putting a queue-driven +multi-channel delivery pipeline inside the token issuer is the same mistake as AD-05. + +**Developer webhooks are the same pipeline.** A webhook to a third-party application is a +delivery channel with a different addressing scheme and a signed payload. One retry policy, one +delivery-history table, one dead-letter view. + +### AD-09 · One policy service, plus a thin client library + +**Decision.** `cloudsforge-policy` is a **decision service**: callers submit a typed decision +request (`subject`, `action`, `resource`, `context`) and receive `allow | deny | challenge | +review` with reasons and obligations. It owns limits, velocity counters, trusted-address lists, +cooling-off timers, approval workflows, freezes and risk scores. `@cloudsforge/policy-client` +is a thin library with a fail-closed default and a local cache for static rules. + +**Why centralised rather than per-domain.** Because the interesting risk signals are +cross-domain: a new device (identity) plus a first withdrawal to an untrusted address (wallet) +plus a key-export request (custody) within an hour is one story, and no domain service can see +it. §3.5 shows the estate has *one* real policy layer today — custody's purpose gate — and +nothing else. + +**Why the policy service does not enforce.** It decides; callers enforce. A decision service +that also sits in the data path becomes a single point of failure for every money movement. +Fail-closed on the *narrow* set of actions where that is correct (key export, withdrawal above +a threshold, treasury spend), fail-open with an alert on the rest (rate limits, soft caps). +This split is stated per-action in [12-security-decisions.md](12-security-decisions.md). + +**Custody keeps its own signing policy.** The purpose gate, binding check and treasury pin stay +inside custody, because a signing policy that can be reached over the network is not a signing +policy. Policy service decisions are an *additional* gate, never a replacement. + +### AD-10 · Events: Postgres outbox → HTTP delivery → inbox. No broker, yet. + +**Decision.** Every service writes domain events to an `outbox` table in the same transaction +as the state change. A relay job publishes them over HTTP to subscribers registered in +`event_subscriptions`. Consumers dedupe into an `inbox` on `(topic, event_id)`. Ordering is +per `(topic, key)` only. + +**Why not a broker now.** NATS or Redis Streams is a second stateful system to operate for an +event volume in the hundreds per minute. Postgres already has `SKIP LOCKED` and transactions. + +**The trigger to adopt one, written down now so it is a measurement and not an argument:** +adopt NATS JetStream when any *one* of these is true — (a) a single topic exceeds 50 events +per second sustained, (b) more than six consumers subscribe to one topic, (c) replay of more +than 24 hours is needed operationally more than once a quarter, or (d) p99 relay lag exceeds +30 seconds for a week. Until then, the relay is a leased job like any other. + +**Event envelope is a contract.** `{ id, topic, key, occurredAt, producer, version, actor, +correlationId, payload }`. Additive-only, versioned per topic, schema-diff enforced (AD-02). + +### AD-11 · Activity is a service that consumes the bus and owns the canonical feed + +**Decision.** `cloudsforge-activity` subscribes to every domain topic and writes one canonical +`activity` record per user-visible event. Products keep their own domain records; activity +keeps the *narrative*. + +**Who owns canonical activity records: the activity service.** Not the ledger (which only sees +money), not Lantern (which sees logs, not domain facts), not each product (which is how you get +nine feeds). Activity records are immutable, typed, user-scoped and reference the owning +service's resource by URN. + +**How events are collected:** exclusively from the event bus. No product ever writes to +activity directly, because a direct write is a write that can happen without the domain change +having committed. If it is not worth an outbox row, it is not worth a feed entry. + +**Notification triggering reads the same bus, not the activity service.** Activity and +notification are two consumers of one stream, not a chain — so a notification is never blocked +on a feed write. + +### AD-12 · Account ↔ wallet binding + +**Decision.** Three record types, in three services, with one join key. + +| Concept | Owner | Record | +| --- | --- | --- | +| Managed wallet | `custody` | The keypair. Bound to `user_id`, `chain`, `network`, `purpose`, `derivation_path`. Never leaves custody. | +| Wallet registry entry | `wallet` | The user-facing wallet: label, primary flag, lifecycle state, address, chain, network, `origin = managed \| external \| watch`. | +| External wallet link | `wallet` | Address + chain + verification proof + verified-at + revoked-at. | + +**Managed wallets move to HD derivation.** Today custody generates one flat random key per +address (§4) — which is why there is no mnemonic to export, no derivation path, and no way to +give a user a recovery phrase. The target: one BIP-39 seed per (user, chain family), stored +encrypted, with addresses at `m/44'/'/'/0/`. This is what makes AD-13 +possible at all, and it is the single largest change inside custody. + +**External wallets are verified by signed challenge**, per family: EIP-4361 (Sign-In with +Ethereum) for EVM and Ember, `signMessage` for Solana, BIP-322 for Bitcoin, and a signed +memo-transaction or `sign` for XRP. A verified link may be a withdrawal destination, a token +owner, a community membership proof and a governance voting key. An unverified address may only +be a watch-only portfolio entry. + +**Wallet ownership is the identity layer for ownership across products.** ForgeMint sets the +customer's wallet as contract owner (this is already true). Market listings are owned by a +wallet, not a user id. Community token-gating reads wallet balances via the indexer. Governance +weight is computed from wallet holdings at a snapshot block. The user id remains the account +key; the wallet is the ownership key. + +### AD-13 · Private-key access: a user right, gated by a one-way lifecycle transition + +**Decision.** A user may export the private key or recovery phrase of any **managed wallet they +own**. Doing so is not a read — it is a **state transition**. The wallet moves +`active → exported`, and an exported wallet: + +- is no longer eligible to receive new deposit sweeps into the platform treasury; +- may still be used to receive and to withdraw, but is flagged in every UI as self-custodied; +- can be **retired** by the user, which stops the platform using it entirely. + +**The gate**, in order: re-authenticate with password → MFA challenge → policy-service decision +(`custody.key.export`) → **24-hour cooling-off** with a cancel link, notified on every channel +the user has → second MFA challenge on redemption → single-use, short-TTL, origin-bound reveal +token → the secret is delivered once, client-side-decrypted, never logged, never in a response +body that any proxy caches. + +**Formats:** BIP-39 mnemonic (for HD-derived families), raw private key hex, WIF (Bitcoin), +XRP family seed, and encrypted UTC/JSON keystore. The keystore is the default because it is the +only format safe to save to disk. + +**Administrative access is removed.** `POST /admin/keys/:address/reveal` (§3.5) — an +any-key-to-any-admin exfiltration primitive — is deleted. It is replaced by a **break-glass +recovery procedure** requiring two operators, a signed incident record, a hardware-token +challenge each, and an alert to every admin. It cannot be invoked from the admin console; it is +a documented runbook against an offline tool. Rationale and the full argument are in +[12-security-decisions.md](12-security-decisions.md). + +**Export audit history is user-visible.** Every export attempt, cancellation and completion +appears in the user's own security log and activity feed, and in the operator audit trail. + +### AD-14 · Marketplace settlement is dual-mode, chosen by what is being sold + +**Decision.** + +| Asset class | Settlement | Why | +| --- | --- | --- | +| Fungible tokens held custodially, Shards, EMBER | **Custodial atomic swap in the ledger** — one journal entry debiting buyer and crediting seller, escrow account in between | Instant, free, reversible by an operator during a dispute window, no gas | +| Assets owned by an external wallet | **On-chain**: escrow contract on EVM/Ember, confirmed by the indexer | The platform never holds the key; the user's own wallet is the counterparty | +| Game items, entitlements, memberships, token-gated content | **Ledger + entitlement grant**, no chain | These are platform-native. Putting them on-chain would be ceremony without benefit | +| Creator products, service subscriptions | **Billing service**, recurring | Subscriptions are a billing concept, not a marketplace one | + +**Escrow is a ledger account, not a smart contract, wherever custodial settlement applies.** +A listing reserves the seller's asset (a ledger reservation, AD-06), a purchase moves reserved +→ escrow → buyer atomically, and fees and royalties are postings in the same entry. Nothing is +"in flight" without being in an account. + +### AD-15 · Community treasuries are ledger sub-accounts with an approval policy + +**Decision.** A community treasury is a set of ledger accounts owned by a `community` subject, +with spending gated by a proposal → approval-threshold → timelock → execution flow in +`cloudsforge-community`. Execution is a ledger posting, or a withdrawal request against a +managed wallet the community owns. + +**Not on-chain multisig, initially.** Hearth has no multisig, no scripting layer and no +finality gadget. An on-chain treasury today would be a single key with extra steps. When +Hearth gains threshold signing, EMBER treasuries above a configurable value move on-chain and +the ledger becomes the mirror rather than the record. That is a stated future migration, not a +present pretence. + +### AD-16 · Developer platform is its own service, not a section of the account portal + +**Decision.** `cloudsforge-devplatform` owns developer organisations, projects, environments, +API keys, service accounts, OAuth clients, webhook endpoints and secrets, usage records, rate +limits and the application directory. `cloudsforge-devportal-web` is its console and its docs +site. + +**Why not inside identity.** Machine credentials and human credentials have different +lifecycles, different revocation semantics, different rate-limit models and different audit +requirements. Identity issues *the token*; devplatform issues *the credential that requests the +token*. Devplatform is a client of identity, exactly like every other product. + +**Public API surface is versioned and separate from internal APIs.** `api.cloudsforge.online/v1` +is a stable, documented, OpenAPI-described surface fronted by the gateway and authorised by +devplatform-issued credentials with scopes. Internal service-to-service routes are never +exposed there. (Note: `api.cloudsforge.online` currently points at the *game* API and must be +renamed to `worlds-api.` before anything depends on it.) + +### AD-17 · Runtime, deployment and versioning + +| Concern | Decision | +| --- | --- | +| **Gateway** | Traefik, label-based discovery. Deletes 18 `container_name:` entries and every fixed host port, which is what makes `deploy.replicas` legal. Owns TLS, CORS, the `/internal` refusal, rate limits at the edge, and serving every SPA. | +| **SPAs** | Built to static bundles, served by the gateway. Removed from API processes (§TD-09) and from the hand-maintained `API_PREFIXES` array. | +| **Health** | `/livez` static; `/readyz` checks Postgres, JWKS and declared upstreams. `depends_on` and load-balancer probes move to `/readyz`. | +| **Shutdown** | SIGTERM → `ready=false` → serve for one LB interval → stop claiming jobs → drain in-flight → exit. | +| **Migrations** | Versioned files, `pg_advisory_lock`, run as a one-shot job (init container / K8s Job), never in `index.ts`. Expand/contract discipline is mandatory because a rolling deploy always runs two versions against one schema. | +| **Background work** | A leased `jobs` table per service (`@cloudsforge/jobs`), claimed with `FOR UPDATE SKIP LOCKED`. Every current `setInterval` becomes a producer plus a leased job. The lease key names the *contended resource*, not the row — `chain` for withdrawals, `bot_id` for ticks, `world_id` for world ticks. | +| **Service identity** | Identity issues short-TTL RS256 service tokens with `sub=` and explicit scopes (`ledger:post`, `custody:sign`, `wallet:read`). The shared `PAY_SERVICE_TOKEN` and `KEYVAULT_SERVICE_TOKEN` are retired. Every ledger posting records the calling service. | +| **Secrets** | Per-service env files in Phase 2; Docker secrets or SOPS thereafter. No container receives a variable it does not use. This is the highest-severity item in the estate and close to free. | +| **Network** | Three compose networks: `edge` (gateway + frontends), `app` (services), `vault` (custody + ledger + settlement). Custody is reachable only from `vault`. | +| **Versioning** | Semver per repo, images tagged with the version. The release manifest (AD-03) pins one version per service. Rollback is the previous manifest. | +| **Kubernetes** | A conclusion, not a prerequisite. After the gateway and leases land, translation is mechanical — with two permanent exceptions written down now: `custody` and Hearth nodes are StatefulSets of exactly one, or they stay outside the cluster. | + +### AD-18 · What stays exactly as it is + +| Thing | Why | +| --- | --- | +| **Hearth's chain node, P2P, consensus, EVM and miner** | A chain node is a stateful singleton. Adding a node adds a validator, not capacity. It also has external contributors, its own security policy and a public audience. It stays its own repository, consumed as an npm package and an RPC endpoint. Its *supporting* surfaces — explorer, wallet, faucet, site, RPC docs, SDK — do split out (AD-19). | +| **Custody's container-per-address vault** | Correct for what it does. It is what blocks any multi-host move, and that is accepted. Single replica, permanently, written down rather than discovered. | +| **`withIdempotency` and `recordDepositAndCredit`** | Both correct. The rest of the design copies their shape. | +| **The origin-bound SSO handoff** | Correct, and safer than most OIDC deployments. It gains an OIDC-conformant *façade* for third parties (AD-16) without changing the internal mechanism. | +| **Custody's purpose gate, binding check and treasury pin** | The one real policy layer in the estate. It is extended, never bypassed. | +| **The warm ash/ember design system** | Extended with new accents, not replaced. See [assets/](assets/). | + +### AD-19 · Hearth's supporting surfaces leave the chain repository + +**Decision.** `hearth` keeps the node, the EVM, consensus, P2P, the miner, the CLI, the +contracts and the SDK. Four things move out into their own repositories: + +- `cloudsforge-explorer-web` — the block explorer (currently `hearth/web`), rebuilt on the + shared design system, fed by the indexer as well as by direct RPC. +- `cloudsforge-network-site` — the Forge Network marketing site (currently `hearth/site`, + which still tells the retired UTXO story, §TD-17). +- `cloudsforge-faucet` — the testnet faucet, which is built and tested and **not deployed**. +- The non-custodial browser wallet moves into Forge Hub as the "connect an external wallet" + path, rather than existing as a second, unrelated wallet with its own localStorage keystore. + +This is the "support sites for sure" part of the brief, and it is also what lets the explorer +carry the CloudsForge bar — today `explorer.cloudsforge.online` is the most linkable public +artifact in the estate and looks like a different company. + +--- + +## 3. Service catalogue + +Twenty-two services. Ownership, data, and the phase that creates or splits each. + +| # | Service | Owns (data) | Consumes | Phase | +| --- | --- | --- | --- | --- | +| 1 | `identity` | users, credentials, MFA factors, sessions, devices, refresh families, exchange codes, signing keys, orgs, teams, memberships, consents | policy | P3 | +| 2 | `policy` | rules, limits, velocity counters, trusted addresses, cooling-off timers, approvals, freezes, risk scores | identity, activity | P5 | +| 3 | `ledger` | accounts, journal entries, postings, balances projection, reservations, settlements, reversals, reconciliation runs | — | P4 | +| 4 | `wallet` | wallet registry, external wallet links, deposit address assignments, withdrawal requests, conversion requests | ledger, custody, indexer, pricing, policy | P4 | +| 5 | `settlement` | treasuries, sweeps, outbound transactions, broadcast state, confirmation tracking | custody, indexer, ledger, policy | P4 | +| 6 | `pricing` | price quotes, sources, administered prices, rate history | — | P4 | +| 7 | `billing` | products, prices, entitlements, subscriptions, usage records, invoices, payouts, revenue shares | ledger, identity | P4 | +| 8 | `custody` | vault addresses, HD seeds, encrypted key material, treasury pins, key events, export records | policy | P5 | +| 9 | `indexer` | blocks, transactions, receipts, logs, address activity, native + token balances, token transfers, contract deployments, reorgs, checkpoints, provider health | — | P5 | +| 10 | `activity` | activity records, inbox, feed cursors | event bus | P6 | +| 11 | `notify` | notification preferences, templates, notifications, deliveries, digests, webhook endpoints, delivery history | event bus, identity, devplatform | P13 | +| 12 | `studio` | brand kits, asset specs, generation jobs, generated assets, credits | billing, ledger | P8 | +| 13 | `mint` | token orders, deployments, token registry, token pages | custody, ledger, indexer, wallet | P3 (split), P8 (extend) | +| 14 | `market` | listings, offers, bids, auctions, orders, escrow refs, collections, verification, moderation, disputes | ledger, wallet, indexer, billing, policy | P9 | +| 15 | `trade` | strategies, backtests, bots, fills, settlements, allocations | ledger, wallet, pricing, billing | P3 (split), P10 (extend) | +| 16 | `worlds` | titles, player profiles, inventory, entitlement bridge, achievements, seasons, rewards, sanctions | identity, billing, ledger, market | P5 | +| 17 | `nda` | worlds, tiles, players, actions, reports, communes, progress, objectives, events | worlds, billing | P5 | +| 18 | `community` | communities, memberships, roles, treasury accounts, proposals, votes, delegations, executions | ledger, identity, indexer, policy | P12 | +| 19 | `devplatform` | developer orgs, projects, environments, API keys, OAuth clients, webhook endpoints, usage, quotas, applications | identity, notify | P11 | +| 20 | `hub-api` | dashboard cache, suggested actions, saved views | everything (read) | P6 | +| 21 | `admin-api` | operator actions, approvals, audit trail, feature flags, broadcast records | everything | P13 | +| 22 | `lantern` + 23 `beacon` | logs, issues / checks, journeys, incidents, conformance | — | P3 (extract) | + +Frontends: `hub-web`, `site`, `market-web`, `mint-web`, `trade-web`, `worlds-web`, +`explorer-web`, `admin-web`, `devportal-web`. + +--- + +## 4. Data ownership and partitioning + +**One database per service. No exceptions, no shared tables, no cross-service reads.** This is +already true and is the most valuable property in the estate. + +**The one cross-cutting key is `user_id`**, issued by identity. It appears in fourteen +databases as a de facto foreign key with no constraint. Two rules make that safe: + +1. **`identity.user.deleted` is a contract.** Every service that stores `user_id` subscribes, + and must acknowledge deletion within an SLA. This is also the GDPR erasure path, which does + not exist today. +2. **No service may infer anything about a user from the shape of an id.** Ids are opaque + UUIDs; profile data is fetched, never cached beyond a stated TTL. + +**Money data is partitioned by account, not by user.** Ledger accounts belong to subjects — +`user:`, `community:`, `platform:revenue`, `platform:fees`, `custody::`, +`clearing:`. This is what lets a community treasury and a platform revenue line live +in the same double-entry system without a special case. + +**Retention** is per-service and stated in [11-data-and-contract-strategy.md](11-data-and-contract-strategy.md). +The ledger keeps everything forever; Lantern keeps events 7 days; the indexer prunes +transaction bodies past a per-chain horizon but never prunes anything a ledger entry references. + +--- + +## 5. How activity is published + +One pattern, used by every service, with no exceptions: + +``` + domain change ─┐ + ├─ same transaction ─► outbox row + outbox row ─┘ + │ + ├─ relay job (leased, key = topic shard) ─► HTTP POST to each subscription + │ with HMAC signature + event id + ▼ + consumer inbox (unique on topic + event_id) ─► handler ─► its own outbox if it emits +``` + +**Topic naming:** `..` — `wallet.deposit.confirmed`, +`ledger.entry.posted`, `market.listing.sold`, `identity.device.added`, +`custody.key.exported`, `community.proposal.executed`. + +**The first events to exist, chosen by which defect each retires:** + +| Event | Consumers | Retires | +| --- | --- | --- | +| `identity.user.deleted` | all | No user-lifecycle reaction anywhere; the GDPR path | +| `ledger.entry.posted` | activity, analytics, notify | Per-product revenue not derivable (§3.3) | +| `billing.entitlement.granted` | worlds, market, community | **The private world that is never built (§3.2)** | +| `wallet.deposit.confirmed` | activity, notify, hub-api | Users learn about deposits by refreshing | +| `settlement.withdrawal.completed` | activity, notify | Same | +| `mint.deploy.confirmed` | activity, market, notify | Client polling every 4 seconds | +| `custody.key.exported` | notify, policy, admin-api | A key leaves with one log line | +| `identity.session.created` | notify, policy | No new-device alert | + +--- + +## 6. Consequences accepted + +Stated so that no one rediscovers them mid-migration. + +1. **38 repositories is a real cost.** AD-02 and AD-03 are the mitigation and they are on the + critical path. If Renovate and the release manifest are not working by the end of Phase 2, + **stop and fix them before splitting anything.** This is an explicit gate in + [06-ecosystem-workflow.md](06-ecosystem-workflow.md). +2. **No single CI can test the whole system.** AD-04 is the answer, and it means a staging + environment is now mandatory infrastructure rather than a nicety. +3. **Custody remains single-replica and is a single point of failure for signing.** Deposits + still land while it is down; withdrawals and sweeps queue. That is the correct degradation + and it must be visible in the status page. +4. **The ledger split is the riskiest migration in the plan.** It moves live balances. It is + done by dual-write with continuous reconciliation and a read-side cutover, never by a + big-bang copy. Detail in [10-migration-strategy.md](10-migration-strategy.md). +5. **HD derivation in custody cannot be retrofitted onto existing flat keys.** Existing + addresses stay flat-key and exportable; new addresses are derived. Custody carries two key + schemes indefinitely, and says so in every response. +6. **Hearth mainnet is not launched.** Everything EMBER-denominated is testnet until it is. + Nothing in this plan assumes a mainnet date. diff --git a/findings.md b/findings.md deleted file mode 100644 index 74ebc35..0000000 --- a/findings.md +++ /dev/null @@ -1,602 +0,0 @@ -# CloudsForge — audit findings and remediation plan - -Audit of all nine product repositories plus the stack composer, verified against -source rather than documentation. Every finding cites `path:line`. Typechecks pass -in platform, crucible, forge-pay, forge-mint, forge-keyvault, shared-libs and game; -Hearth's Rust builds clean under `clippy -D warnings` and its node passes 25/25 P2P -tests including real-socket partition and reorg. - -**Summary judgment.** The engineering is materially stronger than the documentation -suggests, and the documentation oversells in ways that matter. The critical problems -are concentrated in *money movement* — not in code quality. Two claims in -`ECOSYSTEM.md` are disproven below. - -Paths are relative to the stack root unless noted. - ---- - -## Status — remediated 2026-07-28 - -Everything below was worked through. All ten repositories are committed and pushed, -`@cloudsforge/shared@0.3.0` and `@cloudsforge/ui@0.4.0` are published, and the stack was -brought up and driven end to end rather than merely typechecked. - -**Verified against a running stack.** Nimbus register / `/auth/me` / refresh rotation / -family-burn-on-reuse; SSO handoff → exchange 200 with replay 401; deposit address minted; -rate board; `/packages` and `/invoices` gone (404); administered EMBER price changed through -`/admin/prices` and reflected on the board; player refused admin routes (403); the admin Vault -tab reaching keyvault through the Nimbus proxy (200, and 403 for a player); game cosmetics -GET/PUT; a Crucible backtest completing with the previously-hidden metrics and `feesPaid`; -Hearth mining a fresh chain under the raised `MIN_TARGET`; withdrawal correctly refusing a -platform-custodied destination. - -**Three things found during remediation that this audit had missed:** - -1. **No image in the estate could build.** `@fastify/static@10.1.2` pulled `minimatch@10.2.6`, - published hours earlier, and pnpm's release-age gate refused it — fatal under Docker's - `--frozen-lockfile`. Worse, `docker compose build` returns exit 0 while cancelling every - target, so this failed silently and a "healthy" stack was really running stale images. - Fixed by pinning `minimatch` to an aged `10.2.5` rather than excluding it from the gate. -2. **Withdrawal is sweep-to-treasury, not direct.** Keyvault refuses to sign for - `purpose:'deposit'` addresses and forge-pay mints every customer deposit address with that - purpose, so A4 and A5 are one piece of work. Delivered as far as keyvault permits; the - sweeper ships disabled until keyvault grows a signable treasury purpose. -3. **A new cross-repo version skew.** `@cloudsforge/hearth-node` went to 0.2.0 as a - consensus-affecting change; forge-keyvault deliberately stays on `^0.1.0`, because 0.2.0 - emits a `records` key in the signed body and signatures would differ. Do not let a blanket - re-lock pull it forward. - -**Deliberately not done, with reasons:** Hearth's PoW itself (owner chose to fix the copy — -binding the private key into the hash loop forks the chain and breaks the CI-conformance-tested -browser miner); keyvault running as root with the docker socket (a property of the deployment, -not the Dockerfile); per-user signing authorization (the service token proves "a peer service", -not "this user" — needs a design decision, and the circularity was removed in the meantime). - -**Both missing clients were then built** and driven against the running stack: a withdrawal card -and history in the game wallet (`ninety-days-after/apps/game/src/components/withdraw.tsx`) — which -moved 2.5 EMBER to a real external address and was confirmed on-chain — and a price panel in the -operator console (`platform/apps/admin/src/pages/Prices.tsx`), which sets EMBER through a Nimbus -proxy that forwards the caller's own token so attribution survives. - -Building them surfaced two more live defects, both invisible in logs because a rejected CORS -preflight 204s and the handler never runs: - -- **Nimbus advertised a narrower method set than it serves**, so `PUT /admin/users/:id/roles` - was unreachable from any browser — granting or revoking admin from the console could never - have worked. Fixed at `platform/services/nimbus/src/index.ts:82`. -- **Forge Pay had the same defect**, blocking the one call that sets a price. Fixed at - `forge-pay/services/pay/src/index.ts:23`. - -**The operational trap to know about.** `PAY_WITHDRAWALS_ENABLED` defaults to **true** while -`PAY_SWEEP_ENABLED` defaults to false (`forge-pay/services/pay/src/env.ts:63,86`). A fresh -deployment therefore accepts a withdrawal it cannot fund, because keyvault refuses to sign for the -`deposit` addresses customers pay into — which is the whole reason the sweeper exists. It does not -strand: the withdrawal sits `pending`, is retried, and is automatically refunded once -`PAY_WITHDRAWAL_STUCK_MINUTES` passes (default 60), which is safe precisely because nothing was -ever signed (`forge-pay/services/pay/src/withdrawer.ts:85-101`). So it costs a user an hour to -receive an answer that could have been immediate. Until the sweeper lands, either fund the -treasury by hand or set `PAY_WITHDRAWALS_ENABLED=false`. - -*(An earlier revision of this document said such a withdrawal parked indefinitely. It does not — -corrected after reading the refund path.)* - ---- - -## P0 — Exploitable or loses money today - -### P0-1. Free Shards on a public host -`repos/forge-pay/services/pay/src/env.ts:28` defaults `PAY_PROVIDER` to `'mock'`. -Neither `docker-compose.yml` nor `.env` overrides it, and `.env.example:29` actively -sets `PAY_PROVIDER=mock`. `POST /invoices/:id/mock-pay` -(`repos/forge-pay/services/pay/src/routes/invoices.ts:62-66`) is gated on exactly that -value, so any authenticated Nimbus user can create an invoice and settle it for free, -unlimited. `deploy/cloudflared/config.example.yml:52-56` publishes -`pay.cloudsforge.online` and blocks only `/internal/*`. - -The game client is defensive — it checks `invoice.provider === 'mock'` first -(`repos/ninety-days-after/apps/game/src/pages/Store.tsx:27`) — but that is irrelevant, -because the API is callable directly. - -**Resolution: delete the provider path entirely.** See the payment workstream below. - -### P0-2. The performance fee overcharges users -`repos/crucible/services/crucible/src/fees.ts:156` sets `highWaterMark: bot.equity` -unconditionally. The guard at `:121` only protects the case where *nothing* is due. -When a prior pass left `feeOwed > 0` and equity has since fallen below the mark, -`fee` is 0 but `due = arrears >= 1`, so execution falls through and the mark is -dragged **down** to the lower equity. The recovery back to the original mark is then -billed a second time — precisely what the header comment at `:21-26` promises cannot -happen. - -### P0-3. Two double-charge windows in settlement -- A partial charge commits under key `…:partial` (`fees.ts:68`), but the retry - re-sends the **full** `due` under `crucible:settlement:` (`fees.ts:82-90`). - Both land. -- `repos/crucible/services/crucible/src/clients/pay.ts:73` decodes only - `insufficient_balance`. Pay's `idempotency_in_flight` - (`repos/forge-pay/services/pay/src/routes/internal.ts:79`) therefore falls through - to the hard-failure path, recording `feeOwed` for a charge that later commits. - The comment at `pay.ts:71-73` says this must not be treated as a failure. - -### P0-4. Custodied BTC, XRP and EMBER are cryptographically unspendable -`repos/forge-keyvault/services/forge-keyvault/src/chains.ts:68-96` mints keypairs for -all five families, but only `signEvm` (`:104`) and `signSolana` (`:110`) exist. -`repos/forge-keyvault/services/forge-keyvault/src/routes/vault.ts:152-160` returns -`400 unknown family` for bitcoin, xrp and ember. - -Meanwhile forge-pay mints real customer **deposit** addresses for all five through -this service (`repos/forge-pay/services/pay/src/keyvault.ts:37`). Customer BTC and XRP -can therefore never be moved except by an admin revealing a plaintext key and pasting -it into an external wallet. This is the blocker for withdrawals on three of five chains. - -### P0-5. The live kill switch does not stop running bots -`env.liveEnabled` is checked only at create (`repos/crucible/.../routes/bots.ts:93`) -and start (`:188`). The sweep (`runner.ts:288`), settlement sweep (`runner.ts:311`) -and resume (`routes/bots.ts:255-262`) have no gate. Setting -`CRUCIBLE_LIVE_ENABLED=false` on a live deployment keeps moving real money. - -### P0-6. EMBER credits at zero confirmations -`repos/forge-pay/services/pay/src/chains.ts:119-127` returns the Hearth REST tip -balance *as* the confirmed reading, hardcoding `confirmedIsExact: false`. -`store.ts:751` credits on any positive delta. The declared `confirmations: 3` -(`repos/shared-libs/packages/shared/src/deposits.ts:41`) is never enforced. - -Currently unspendable — `PAY_EMBER_USD` is unset so conversion is refused -(`pricing.ts:284`) — which makes this one environment variable away from being real -money. - -**This is a forge-pay bug, not a Hearth limitation.** See the correction below. - -**Sequencing:** P0-1 → P0-6 → P0-4 → P0-2/P0-3 → P0-5. - ---- - -## Corrections to existing documentation - -Two claims that shaped prior planning are false, and two of my own mid-audit -assumptions were wrong. Recorded so they are not re-derived. - -1. **`ECOSYSTEM.md` §1.1 and `forge-pay/.../chains.ts:20-22` claim Hearth only - exposes tip balance. It does not.** `repos/hearth/node/src/rpc.js:202-220` - (`GET /address/:addr`) returns per-UTXO `height`, `spendable`, `coinbase` and - `maturesAtHeight`, plus the chain `height`. `GET /tx/:txid` returns - `confirmations` (`rpc.js:116`, `chain.js:124`). Confirmation-gated EMBER crediting - is a ~20-line change to `emberProbe`, not a Hearth feature request. - -2. **`ECOSYSTEM.md` §3 cites a stale `platform/services/nimbus/public/index.html`.** - That file no longer exists; the directory contains only `assets/`. - -3. **Crucible and ForgeMint front-ends are *not* missing from the deploy.** Neither - has a Dockerfile or compose service, which looks like two of five products being - unreachable. Both service Dockerfiles build the SPA and serve it via - `@fastify/static` (`repos/crucible/services/crucible/Dockerfile`, - `repos/crucible/services/crucible/src/index.ts:54`, and the ForgeMint equivalents). - This is correct and intentional. - -4. **Hearth's non-custodial web wallet already sends.** - `repos/hearth/web/wallet.html:138-147` and `:443-472` build, sign locally and - broadcast via `POST /tx`, using `WC.buildTx` from - `repos/hearth/web/assets/wallet-core.js`. Only the *custodial* wallet lacks send. - ---- - -## Workstream A — The crypto-native payment system - -**Direction (owner, 2026-07-28):** the ecosystem is crypto-based. No fiat or mock -payment provider. Shards are funded by on-chain deposit only. - -The audit strengthens this: **the crypto path is already built and shipping.** -`POST /deposits`, `GET /coins/rates` and `POST /coins/convert` all exist -(`repos/forge-pay/services/pay/src/routes/wallet.ts:41-85`), and the game Wallet page -already renders address generation, copy-to-clipboard, a live rate board and -idempotent convert-to-Shards -(`repos/ninety-days-after/apps/game/src/pages/Wallet.tsx:37-112`). Four of five chains -enforce genuine confirmation depth: BTC confirmed-only (`chains.ts:138`), ETH -`latest − 12` (`:150-156`), SOL finalized (`:173`), XRP validated (`:194`). - -The invoice/provider stack is therefore vestigial scaffolding *and* the security hole. -Deleting it is lower-risk than configuring it. - -### A1. Delete the provider path -Remove `routes/invoices.ts`, `providers/mock.ts`, `providers/nowpayments.ts`, the -`invoices` table (`db/schema.ts:4`, `db/migrate.ts:6`), `PAY_PROVIDER` (`env.ts:28`), -`routes/webhooks.ts`, and `SHARD_PACKAGES` / `GET /packages` (`index.ts:43`). -Note `pricing.ts:136` derives a worst-case rate from `SHARD_PACKAGES` — that needs a -replacement constant. Its only client consumer is the game Store page. - -Closes P0-1 by deletion. - -### A2. Rewrite the game Store as deposit → convert → spend -`repos/ninety-days-after/apps/game/src/pages/Store.tsx` becomes a funding page reusing -the components already present in `Wallet.tsx`. No new backend. - -### A3. Confirmation-gate EMBER -Rewrite `emberProbe` (`repos/forge-pay/services/pay/src/chains.ts:119-127`) to call -`GET /address/:addr`, compute `height − utxo.height + 1` per UTXO, and sum only those -at or above the declared depth of 3. Set `confirmedIsExact: true`. Honour `spendable` -so immature coinbase output is not credited. Closes P0-6. - -### A4. Withdrawal — send from the custodial wallet -No payout endpoint exists anywhere in forge-pay's 24 handlers. Money can enter and -never leave. Requires: -- `POST /withdrawals` in forge-pay with address validation, balance lock, fee - estimation and an idempotency key. -- Signing support in ForgeKeyvault for bitcoin, xrp and ember (**blocked on P0-4**). -- A send form in the custodial wallet UI. -- A confirmation-tracked outbound state machine mirroring the deposit watcher. - -*Open question — see Decisions Required.* Hearth's non-custodial wallet already sends; -this item is the custodial wallet. - -### A5. Treasury sweeper -`recordSweep` (`repos/forge-pay/services/pay/src/store.ts:865`) is documented as -MUST-call by `db/migrate.ts:75` and `watcher.ts:66`, but has **zero callers** across -all nine repositories. Any manual movement of funds makes `store.ts:774` return -`regression`, permanently freezing crediting for that address. Build the sweeper, or -remove the trap. - -### A6. Any-asset → EMBER conversion -Today conversion is one-directional: `convertCoinToShards` -(`repos/forge-pay/services/pay/src/routes/wallet.ts:99`). There is no coin→coin or -coin→EMBER path. - -Add a conversion whose target is EMBER, priced through the same quote layer. Design -notes: -- Route both legs through the existing USD-scaled quote (`quoteFor`, - `pricing.ts:197`) rather than adding pairwise rates: `asset → USD → EMBER`. -- Reuse the BigInt arithmetic and floor-against-the-user discipline in `amounts.ts` - and `shared/deposits.ts:208-216`. Do not introduce floats. -- Reuse the existing DB-level idempotency (`store.ts:212-247`), not a new mechanism. -- Apply `PAY_CONVERSION_SPREAD_BPS` consistently in both directions. -- EMBER credited this way is a custodial `coin_balances` row — spendable in-platform - immediately, withdrawable on-chain once A4 lands. It does not broadcast, so A6 is - independent of P0-4 and can ship before withdrawals exist. - -### A7. Administered prices with sensible defaults -The pricing layer already anticipates this. EMBER is deliberately excluded from market -quotes (`pricing.ts:11`) and quoted from configuration, tagged -`source: 'administered'` (`pricing.ts:190-195`) — the exact concept required. Today it -reads a single env var, `PAY_EMBER_USD` (`env.ts`), and refuses conversion when unset -(`pricing.ts:284-285`). - -Work: -- Move administered prices from env into a DB table, admin-settable at runtime, with a - seeded default so a fresh deployment is functional rather than refusing conversions. -- Add admin routes to forge-pay. **There is currently no admin surface in this service - at all** — `routes/` holds only deposits, internal, invoices, monetization, wallet - and webhooks. Guard with the same Nimbus `admin` role check used in - `repos/platform/services/nimbus/src/routes/admin.ts:114`. -- Add a price panel to the admin console (`repos/platform/apps/admin`). -- Keep the fail-closed posture for *market* coins: an administered override should be - an explicit, audited act, not a silent fallback when the oracle is stale. Record who - set each price and when. -- Preserve the `source` field end to end so the UI can distinguish an administered - price from a market one. Users should be able to see which is which. - -**Ordering for Workstream A:** A1 → A3 → A2 → A7 → A6 → P0-4 → A4 → A5. -A7 precedes A6 because any-asset→EMBER needs a trustworthy EMBER price to exist first. - ---- - -## P1 — Broken in the assembled stack - -### P1-1. Cosmetics can never be equipped -`docker-compose.yml` sets `PAY_API_URL` for forge-mint (`:220`) and crucible (`:258`) -but **not** for the game service (`:137-162`), and it is absent from `.env`. -`repos/ninety-days-after/services/game/src/env.ts:51` falls back to -`http://localhost:4003` — inside the container, that is the game itself. `PUT -/cosmetics` always returns 503 (`routes/cosmetics.ts:72`). One-line fix. - -### P1-2. The admin Vault tab is dead in production -`repos/platform/apps/admin/src/lib/api.ts:20` resolves `KEYVAULT_URL` to -`https://keyvault.`, but keyvault is deliberately never routed -(`deploy/cloudflared/config.example.yml:80`) and bound to `127.0.0.1:4005` -(`docker-compose.yml:299`). All three calls (`api.ts:369,373,380`) and the `/vault` -page — a permanent nav item (`components/Shell.tsx:8`) — fail off-localhost. - -**Fix by proxying vault reads through Nimbus server-side. Do not expose keyvault.** - -### P1-3. Nimbus has no password reset or change -Routes are register/login/refresh/exchange/logout/me plus admin roles -(`repos/platform/services/nimbus/src/routes/auth.ts:53-234`); the `/account` page -(`routes/portal.ts:768-844`) renders identity and a launcher only. A forgotten or -leaked password is unrecoverable by the user. Compounding this, -`repos/platform/services/nimbus/src/index.ts:28-33` rewrites the admin password hash -from env on every boot, so the plaintext must live in `.env` permanently and rotation -requires a redeploy. - -### P1-4. Hearth's difficulty ceiling caps the chain -`repos/hearth/node/src/params.js:57` sets `MIN_TARGET` at roughly 2^-20, capping work -at about 1.1M attempts per block. At ~200 H/s per core this pins difficulty at -roughly 300–500 cores; beyond that `_nextTarget` (`node/src/chain.js:225-229`) clamps, -block time falls below 15s and emission permanently accelerates. Launch-blocking for a -coin whose thesis is mass CPU mining. - -### P1-5. Hearth PoW verification is a free remote DoS -`repos/hearth/node/src/pow.js:22` fills 8,192 words with sequential SHA-256, so -`verifyPow` costs roughly one full attempt (~8,450 hashes). A peer may send 200 blocks -per `blocks` frame (`node/src/p2p.js:236`) with no rate limit. Worse, -`node/src/chain.js:241` canonically serializes up to 2 MB *before* the PoW check at -`:251`; the fork path correctly checks PoW first (`:327-329`). - -### P1-6. Additional Hearth and keyvault security items -- Plaintext Ed25519 spending keys in `localStorage` (`repos/hearth/web/wallet.html:293`, - `web/mine.html:187`), rendered into the DOM on load (`wallet.html:301`), with CSP - disabled in the desktop shell (`app-desktop/src-tauri/tauri.conf.json:21`). -- Unauthenticated DoS amplifiers on the public RPC: unbounded `readBody` - (`node/src/rpc.js:251-258`) and a full UTXO-set copy per `/mining/template` request - (`node/src/miner.js:48`) under CORS `*`. -- Keyvault Solana signing is an unconstrained signing oracle — `partialSign` on any - base64 transaction with no instruction inspection (`routes/vault.ts:156-158`). -- Keyvault EVM signing checks only `to` and `chainId`; `data`, `value` and gas fields - are unchecked, and the generic `'evm'` chain returns `null` from `chains.ts:49-53`, - which skips the chainId check entirely (`vault.ts:141`). -- Keyvault's caller-restates-the-binding check is circular: every field restated is - readable from `GET /addresses/:address` under the same service token - (`vault.ts:65-91`). It is an integrity check, not authorization. -- Keyvault runs as root with `/var/run/docker.sock` bind-mounted — any RCE is - whole-host compromise. -- Keyvault ciphertext has no key-version byte (`src/crypto.ts:20-28`), so the master - secret can never be rotated. - ---- - -## P2 — Built but not exposed - -Substantial completed work that no user can reach. - -**Crucible.** Bots are creatable only from 10 presets (`apps/crucible/src/pages/Bots.tsx:96-103`) -though the API accepts arbitrary combinations (`routes/bots.ts:127`) — roughly 10 of -~500 valid configurations, and 6 of 10 pairs are unusable. The Lab has full -strategy/pair/timeframe/slider UI (`Lab.tsx:157-233`) with no "run this as a bot" path. -Backtest history is persisted with a user index (`db/migrate.ts:85`) and its endpoints -(`routes/backtests.ts:130,132`) are never called, so every run is lost on refresh. -`cagr`, `exposure`, `bestTrade`, `worstTrade` and start/end equity are computed -(`engine/metrics.ts:102-126`) and never rendered. - -**Game.** `GET /worlds/:id/events` (`services/game/src/routes/progression.ts:201`) is -called by no client page — the storm/disease/warband/caravan arc is written every tick -and is invisible until the post-mortem archive. All five admin routes -(`services/game/src/routes/admin.ts:46,57,73,84,98`) have no UI, so no world can be -created or started without curl. - -**Hearth.** The API exposes tx lookup with confirmations, per-UTXO address detail, a -records browser and filtered SSE; the explorer (`web/index.html`) shows 4 stats and 12 -blocks with no tx search or block detail. The desktop app's three native commands -(`app-desktop/src-tauri/src/main.rs:31-75`) have zero callers, and `node_entry()` -resolves a CWD-relative path that does not exist in a bundled `.app`. The on-chain -encrypted chat (`node/src/apps/chat.js`, `node/src/box.js`) is CLI-only. - -**Nimbus.** `GET /admin/users/:id` (`routes/admin.ts:154`), `getVaultKey` and -`getWorld` are wrapped in the admin client and never called. - ---- - -## P3 — Visual coherence - -Ranked by leverage. - -### P3-1. The `.cf-dark` seam — fix first, it is in the dependency floor -`CloudsForgeBar` renders `className="cf-bar cf-dark"` -(`repos/shared-libs/packages/ui/src/index.tsx:550`). `.cf-dark` -(`packages/ui/src/tokens.css:140-147`) re-declares the **warm** ash ramp directly on -that element, and `.cf-bar`'s background reads `var(--cf-ash-950)` / `var(--cf-ash-900)` -(`packages/ui/src/ui.css:16-19`). Crucible cools the substrate at `:root` -(`repos/crucible/apps/crucible/src/index.css:26-38`), but `.cf-dark` wins on the bar -element — so the estate's best-looking app wears warm charcoal chrome on a cool -blue-grey page. - -Compounding: `.cf-dark` omits the entire semantic layer declared at `tokens.css:68-80` -(`--cf-bg`, `--cf-fg`, `--cf-accent`), so its stated purpose — surviving inside a light -host app — cannot work. - -**Fix:** have `.cf-dark` set the semantic layer, and sanction a supported substrate -override so per-product tinting is a system feature rather than a fork. - -### P3-2. The marketing site is off-palette — and it is the front door -`repos/platform/apps/site/src/index.css:9-15` declares its own ramp in which -`--color-ash-950: #0e0c0a` *is* the canonical `--cf-ash-900` -(`repos/shared-libs/packages/ui/src/tokens.css:16`). Every step differs and the whole -ramp is offset by one, so every surface renders one step too light and warmer. Admin -derives correctly from `--cf-*` (`apps/admin/src/index.css:20-25`) and the portal was -already fixed for this exact reason (`services/nimbus/src/routes/portal.ts:183-191`). -The site is the last holdout. - -### P3-3. There is no type scale and no spacing scale -`repos/shared-libs/packages/ui/src/tokens.css` is 165 lines with zero `--cf-space-*` -or `--cf-text-*`. `ui.css` then hardcodes eight ad-hoc font sizes (`:83, :106, :200, -:226, :258, :265, :272, :289`) and mixes px and rem padding. Apps cannot follow a scale -that does not exist — the system is colour, radius and shadow only. This is the root -cause of most per-app drift. - -### P3-4. The admin console has no icon or social identity -`repos/platform/apps/admin/index.html:1-12` is the only app with a bare `` — no -favicon, no `og:image`, no `theme-color` — despite `branding/favicon-cloudsforge.png` -existing. - -### P3-5. The game breaks convention three ways -Generic `favicon-32/192/512.png` and `og-image.png` -(`repos/ninety-days-after/apps/game/index.html`) against every sibling's -`favicon-.png` / `og-.png`; Google-hosted Oswald and Barlow instead -of the system stack (`index.html:47-51`); and `#150f08` hardcoded in place of -`--cf-accent-ink` across seven files (e.g. `apps/game/src/components/ui.tsx:135`). - -### P3-6. Lantern is linked from every product switcher but is a different tool -`infra/lantern/public/` is hand-rolled vanilla JS and CSS with its own palette, so -users cross from polished React chrome into something visibly other. It is also absent -from `CORS_ORIGINS` and `PORTAL_ALLOWED_ORIGINS` (`docker-compose.yml:66,72`) — no -`:4010`, no `lantern.`. - -### P3-7. Brand asset sets are uneven -Complete (favicon, mark, og, social, wordmark): crucible, hearth, platform. -Partial: forge-mint (no social, no wordmark). Minimal: forge-pay (mark only), -ninety-days-after (`mark-games.png` only). Work is already in progress on the -`brand/finish-the-set` branch. - -Separately, 12 delivered game assets violate their manifests: all six `tile-*` and six -`icon-*` in `repos/ninety-days-after/apps/game/art/` are 1024×1024 against declared -512×512 (`repos/asset-forge/src/manifest.ts:231`) and 256×256 (`:268`). Skip-if-exists -(`src/generate.ts:127`) means they never self-correct without `--force`. A further 18 -expected PNGs are missing. - -### P3-8. Registry drift inside shared-libs -The file that exists to end hand-maintained lists contains three more — -`CloudsForgeHosts` (`packages/ui/src/index.tsx:104-122`), `LOCAL_HOSTS` (`:131-149`) -and `cloudsforgeHosts` (`:172-187`) restate 14 keys three times. The `wallet` surface -declares `subdomain: 'pay'` and `devPort: 4003` (`packages/shared/src/products.ts:129-140`), -both wrong and dead, since the URL derives as `${play}/wallet` (`index.tsx:128`); `pay` -claims the same subdomain (`products.ts:225-236`). And `service` is documented as -belonging in neither switcher nor site (`products.ts:43-44`) while `admin` (`:159,165`) -and `lantern` (`:176,178`) are services with `inSwitcher: true`. - -### P3-9. Version skew -shared 0.2.1 / ui 0.3.0. crucible, ninety-days-after and asset-forge still resolve -`shared@0.2.0`. All ranges are `^`, so a re-lock fixes it — but -`minimumReleaseAgeExclude` must be hand-edited first (e.g. -`repos/ninety-days-after/pnpm-workspace.yaml:17-21` has no `0.2.1`) or -`verifyDepsBeforeRun: error` breaks every command in the workspace. - ---- - -## P4 — Claims the code does not support - -The estate's copy is deliberately honest elsewhere, which makes these stand out. - -1. **Hearth's "non-outsourceable" PoW is outsourceable.** `repos/hearth/node/src/pow.js:35` - binds only the coinbase **public** key into the seed; the private key is used once, - after the winning digest (`node/src/miner.js:94`). A pool operator can distribute - `coreHash` plus its own pubkey, collect `(nonce, digest)` from hashers who cannot - steal the reward, and sign the block itself. Non-outsourceability requires the - private key inside the hash loop. The claim is restated at - `repos/hearth/site/src/lib/hearth.ts:47,60,93,118`. -2. **"RandomX-class — each nonce compiles a program"** (`site/src/lib/hearth.ts:114-116`) - — Homefire compiles nothing; it is chained SHA-256 over 64 KiB. -3. **"Mines only on AC power or when idle"** (`:123`) — no battery or idle detection - exists anywhere in the repository. -4. **The "two-line Accept EMBER SDK" is a `setTimeout`** — `repos/hearth/web/assets/hearth-pay.js:54-58` - fabricates a txid with `Math.random()` and confirms after 1200 ms. Published via - `.github/workflows/pages.yml`. The footer discloses it; the marketing copy does not. -5. **Crucible's paper trading charges neither fee nor slippage** — - `repos/crucible/services/crucible/src/runner.ts:213,218-221` converts at the raw rate - and writes `fee: 0`, while backtests charge both (`engine/backtest.ts:103-135`). Paper - therefore systematically beats its own backtest, against the "fees and slippage - charged" claim at `apps/crucible/index.html:24`. -6. **ForgeMint sells up to 3 EVM chains** (`repos/shared-libs/packages/shared/src/forgemint.ts:154`) - but an order carries exactly one `chain` (`:210`); `routes/tokens.ts:65` only rejects - `maxChains < 1`. -7. **forge-pay's README asserts deposits are credited only at confirmation depth** — see - P0-6. -8. **The Rust core is vestigial and consensus-incompatible.** `repos/hearth/rust/hearthd/src/main.rs:16-90` - is a self-check and benchmark with no block, chain, RPC or P2P server; - `difficulty.rs:36-42` moves ±1 leading-zero bit per retarget rather than the JS - 256-bit LWMA, and `pow.rs:22` omits the coinbase pubkey from the seed. Do not treat - it as a second implementation. - ---- - -## Lower-severity items - -- **`POSTGRES_PASSWORD` is absent from `.env.example`** and defaults to `cloudsforge` - (`docker-compose.yml:97`). Postgres is loopback-only, so severity is low, but a - deployer following the example gets the default silently. -- **ForgeMint concurrent `/deploy` can double-spend gas** — `routes/tokens.ts:193` - re-admits status `'deploying'` and the in-flight guard at `:199` only fires once - `txHash` is persisted, which happens after broadcast (`chain/evm.ts:147`). -- **ForgeMint's mainnet guard permits everything** — `routes/tokens.ts:213-224` lets any - authenticated Shard-paying user deploy to mainnet with `{confirmMainnet:true}`. It is - genuinely server-side and unbypassable, but it is a confirmation checkbox, not a - guard. (No shared deployer key exists to drain — each order funds its own vault - address, `tokens.ts:142-156`.) -- **ForgeMint silent gas fallback** — `chain/evm.ts:88-107` substitutes a fixed - 1,500,000 gasLimit when estimation throws. -- **forge-pay loads the full ledger inside every money transaction** — - `store.ts:257-261` selects the entire ledger with no `LIMIT` from inside `debit()` - and `credit()` while holding the wallet lock. `idempotency_keys` has no TTL. -- **forge-pay idempotency namespace collision** — `store.ts:916` labels the - `/internal/trade` sell path `'POST /coins/convert'`, the same label the user-facing - route uses (`routes/wallet.ts:85`), so a user can pre-claim a peer service's key. -- **Game determinism gaps** — `world/mapgen.ts:39,58,59` use unseeded `Math.random()` - against the replay claim in `README.md:20`; `engine/resolve.ts:106` snapshots progress - before the transaction and upserts at `:691-721`, silently reverting concurrent - objective claims. -- **Game global `fuel` and `medicine` are seeded and never consumed** — - `world/generate.ts:21` vs `engine/resolve.ts:227-228`, against the "finite fuel and - medicine" copy at `apps/game/src/pages/Landing.tsx:9`. -- **No rate limiting in the game service**; refresh tokens in `localStorage` across all - surfaces with a 30-day TTL (`repos/platform/services/nimbus/src/tokens.ts:11`). -- **asset-forge** leads with `gpt-image-2`, absent from the pinned SDK's model union, so - every asset pays a failed round-trip before falling back (`src/generate.ts:25,87-88`); - the advertised `ASSET_MODEL=dall-e-3` fails after billing (`:78`); `ASSET_OUT_ROOT` is - read before `config()` so setting it in `.env` is ignored (`src/env.ts:20,26`); there - is no spend guard or dry-run, so a bare `pnpm generate --force` starts a ~$12 run; and - it loads the platform root `.env`, pulling compose secrets into its process - (`src/env.ts:27`). -- **Dead exports** across crucible (`market/candles.ts:111`, `store.ts:28,35,406`, - `auth.ts:57`, `clients/pay.ts:133`), forge-mint (`chain/erc20.ts:25,32`, - `chain/networks.ts:43`, `store.ts:47`) and shared-libs (`products.ts:292`, - `packages/ui/src/index.tsx:287`, `packages/shared/src/game.ts:457`). - ---- - -## What genuinely works — do not rebuild - -- **SSO is real end to end.** `signInRedirect` → portal `/login` → `/auth/login` → - `POST /portal/handoff` → origin-bound single-use code → `#cf_code` → `/auth/exchange`. - Codes are hashed, 60-second TTL, redeemed by conditional `UPDATE…RETURNING`. Return-URL - allowlisting correctly rejects open redirects. Refresh rotation with family revocation - on reuse is properly implemented (`tokens.ts:66-99`), and login throttling counts - unknown emails so it cannot be used to enumerate accounts. -- **forge-pay's money arithmetic is exact** — BigInt throughout, floors consistently - against the user, `MAX_SHARDS` enforced under `FOR UPDATE`. Idempotency is a real - primary key claimed in the same transaction as the work, not an application check. The - oracle genuinely fails closed: null past max-age, ≥2 sources required, diverged rounds - discarded, every caller mapping null to 503. -- **Crucible's backtest engine is honest.** Acts on `signals[i-1]` at `candles[i].o` — - no lookahead; fees and slippage on both sides; the benchmark pays the same entry cost; - metrics computed pre-decimation. Indicators are length-aligned with null warm-ups, - Donchian is prior-exclusive, RSI/ATR use Wilder smoothing. No IDOR anywhere — every - route resolves through `getOwnedBot`/`getOwnedBacktest`. No exchange credentials exist - at all, which is the right architecture. -- **Hearth's node is a real full node** — UTXO and Ed25519 validation, double-spend - checks, coinbase maturity, anti-inflation, MTP and future-drift bounds, all enforced on - the accept path including the fork path. Fork choice and reorg are tested over real - sockets. The browser miner is digest-identical to the node and conformance-tested in - CI; `/mining/submit` correctly refuses to trust the submitted header. -- **The game's tick loop is genuinely deterministic** — seeded mulberry32/FNV-1a - throughout, explicitly sorted action order, whole day in one transaction. Communes are - fully implemented and concurrency-safe. This is not a wireframe. -- **ForgeMint's contracts are real** — solc 0.8.26 bytecode from OpenZeppelin 5, CI - recompiles and requires a byte-identical diff, and the deploy path verifies with - `getCode` after broadcast. -- **ForgeKeyvault's secret validation is genuinely fail-closed** — no reachable dev - default, placeholders denied by name, 24-character floor, thorough log redaction - covering `*.privateKey`. -- **The stack composer is unusually well-reasoned** — loopback bindings on the sensitive - services with the reasoning recorded inline, `service_healthy` rather than - `service_started` where boot order actually matters, digest-pinned images, and CI that - already catches overlay/build drift and edge-routing mistakes. - ---- - -## Decisions taken - -Settled with the owner on 2026-07-28: - -1. **"Send" means the custodial wallet.** A4 builds outbound on-chain withdrawal in - forge-pay for custodied BTC/ETH/SOL/XRP/EMBER, surfaced in the game Wallet page. - Hearth's non-custodial wallet already sends (`repos/hearth/web/wallet.html:443-472`) - and is out of scope. A4 remains blocked on P0-4 (keyvault signing for bitcoin, xrp - and ember). -2. **Any-asset→EMBER credits a custodial balance.** A6 writes a `coin_balances` row — - spendable in-platform immediately, withdrawable on-chain once A4 lands. It does not - broadcast. This keeps A6 independent of P0-4 and lets it ship first. -3. **Administered prices cover EMBER only.** A7 does not add overrides for BTC, ETH, - SOL or XRP: the fail-closed oracle stays authoritative for coins that have a real - market. EMBER has no market, so a seeded, admin-editable price is the correct source - for it — and the existing `source: 'administered'` tag (`pricing.ts:194`) already - models exactly this. - -## Still open - -- **Hearth's non-outsourceable claim (P4-1):** fix the PoW to bind the private key into - the hash loop, or change the copy. Both are legitimate; leaving the claim as-is is not. diff --git a/infra/beacon/package.json b/infra/beacon/package.json index 875a7fd..97ab301 100644 --- a/infra/beacon/package.json +++ b/infra/beacon/package.json @@ -9,7 +9,8 @@ "start": "node src/index.js", "probe": "node src/cli.js probe", "journey": "node src/cli.js journey", - "conformance": "node src/cli.js conformance" + "conformance": "node src/cli.js conformance", + "test": "node --test" }, "dependencies": { "jose": "^5.9.6", diff --git a/infra/beacon/public/app.js b/infra/beacon/public/app.js index e36c449..cee08a1 100644 --- a/infra/beacon/public/app.js +++ b/infra/beacon/public/app.js @@ -407,7 +407,12 @@ function renderTopbar() { function renderIncidentBanner() { const banner = $('#incident-banner') - const open = (state.status?.incidents ?? []).filter((i) => !i.closed_at) + // camelCase, like every other field on the wire. `/api/status` serves the + // in-memory incidents unconditionally and `/api/incidents` falls back to + // them with Postgres down, so reading the table's snake_case here made the + // banner's timestamp blank always, and the incidents view's blank during + // exactly the outage it exists for. db.js converts at the SQL boundary now. + const open = (state.status?.incidents ?? []).filter((i) => !i.closedAt) setTabCount('incidents', open.length) if (!open.length) { banner.classList.add('is-hidden') @@ -418,7 +423,7 @@ function renderIncidentBanner() { banner.append(el('strong', null, `${open.length} open incident${open.length === 1 ? '' : 's'}`)) for (const inc of open.slice(0, 4)) { banner.append(el('span', 'banner-item', - `${inc.subject ?? inc.scope ?? 'unknown'} · ${inc.cause ?? 'cause unrecorded'} · ${since(inc.opened_at)}`)) + `${inc.subject ?? inc.scope ?? 'unknown'} · ${inc.cause ?? 'cause unrecorded'} · ${since(inc.openedAt)}`)) } const go = el('button', 'btn btn-sm', 'See all') go.onclick = () => showView('incidents') @@ -1317,6 +1322,27 @@ function renderChain() { if (c.peers && typeof c.peers === 'object') { stats.append(stat('peers', Object.entries(c.peers).map(([k, v]) => `${k} ${fmtInt(v)}`).join(' · '))) } + + // The chain id, as an assertion rather than a reading. It is here because a + // node on the wrong-but-valid id misroutes nothing — balances still read + // correctly — so the only symptom is signatures rejected somewhere else + // entirely, and the first useful question in that incident is "which chain + // does this node think it is on". + const expected = numOrNull(c.expectedChainId) + const seen = [...new Set((Array.isArray(c.nodes) ? c.nodes : []).map((nd) => numOrNull(nd.chainId)).filter((v) => v !== null))] + if (expected !== null || seen.length) { + const wrong = expected !== null && seen.some((v) => v !== expected) + stats.append( + stat( + 'chain id', + wrong ? `${seen.join('/')} — expected ${expected}` : String(expected ?? seen[0]), + wrong ? 'down' : null, + wrong + ? 'a node is reporting a chain id this deployment does not expect: every signature bound to it is bound to a different chain' + : 'the EIP-155 chain id every node here must report', + ), + ) + } box.append(stats) // --- nodes, with disagreement flagged rather than averaged away @@ -1330,7 +1356,11 @@ function renderChain() { const t = el('table', 'tbl') const thead = el('thead') const hr = el('tr') - for (const label of ['node', 'state', 'height', 'tip', 'peers']) hr.append(el('th', null, label)) + // `json-rpc` is its own column and not folded into `state`: it is a second + // listener in the same process on a different port, it fails soft, and it + // is the one forge-pay reads deposits from — so "REST up, JSON-RPC down" is + // a real and previously invisible state that has to be readable at a glance. + for (const label of ['node', 'state', 'json-rpc', 'height', 'tip', 'peers']) hr.append(el('th', null, label)) thead.append(hr) t.append(thead) const tb = el('tbody') @@ -1338,13 +1368,20 @@ function renderChain() { const height = numOrNull(nd.height) const behind = maxHeight !== null && height !== null && height !== maxHeight const onFork = forked && height === maxHeight + const rpcDown = nd.rpcState === 'down' const tr = el('tr') - if (nd.state === 'down') tr.className = 'bad' + if (nd.state === 'down' || rpcDown) tr.className = 'bad' else if (behind || onFork) tr.className = 'disagree' tr.append(el('td', null, nd.name ?? '—')) const st = el('td') st.append(glyphWithWord('state', nd.state)) tr.append(st) + const rpc = el('td') + rpc.append(glyphWithWord('state', nd.rpcState)) + rpc.title = rpcDown + ? 'the Ethereum JSON-RPC listener on 8545 is not answering. /info on 8645 can be perfect while this is dead — it is a separate listener that fails soft' + : `eth_blockNumber ${fmtInt(nd.rpcHeight)}${nd.chainId ? ` · chain ${nd.chainId}` : ''}` + tr.append(rpc) const hcell = el('td', 'num', height === null ? '—' : height.toLocaleString()) if (behind) hcell.title = `${maxHeight - height} block(s) behind the leader` tr.append(hcell) @@ -1544,10 +1581,14 @@ async function loadIncidents() { } const incidents = [...(data.incidents ?? [])].sort((a, b) => { - const aOpen = a.closed_at ? 0 : 1 - const bOpen = b.closed_at ? 0 : 1 + const aOpen = a.closedAt ? 0 : 1 + const bOpen = b.closedAt ? 0 : 1 if (aOpen !== bOpen) return bOpen - aOpen // open ones pinned to the top - return Date.parse(b.opened_at ?? '') - Date.parse(a.opened_at ?? '') + // Newest first. Against the old snake_case reads both sides parsed to NaN, + // which is not a crash but a no-op comparator: V8's stable sort then left + // the in-memory incidents in Map insertion order — oldest first, the exact + // reverse of what this line asks for, in the one view read during a cascade. + return Date.parse(b.openedAt ?? '') - Date.parse(a.openedAt ?? '') }) list.replaceChildren() @@ -1560,7 +1601,7 @@ async function loadIncidents() { function incidentCard(inc) { const box = el('div', 'inc') - const open = !inc.closed_at + const open = !inc.closedAt box.classList.add(open ? 'is-open' : 'is-closed') if (inc.severity) box.dataset.severity = String(inc.severity).toLowerCase() @@ -1569,13 +1610,13 @@ function incidentCard(inc) { if (inc.scope) head.append(el('span', 'badge group-badge', inc.scope)) head.append(el('span', 'inc-subject', inc.subject ?? '—')) head.append(el('span', 'badge', open ? 'OPEN' : 'closed')) - head.append(el('span', 'inc-when', `${since(inc.opened_at, inc.closed_at)}${open ? ' and counting' : ''}`)) + head.append(el('span', 'inc-when', `${since(inc.openedAt, inc.closedAt)}${open ? ' and counting' : ''}`)) box.append(head) box.append(el('div', 'inc-cause', inc.cause ?? 'No cause recorded.')) box.append(el('div', 'card-note', - `opened ${fmtDateTime(inc.opened_at)}${inc.closed_at ? ` · closed ${fmtDateTime(inc.closed_at)}` : ''}${inc.id ? ` · #${inc.id}` : ''}`)) - if (inc.last_error) box.append(el('pre', 'err', inc.last_error)) + `opened ${fmtDateTime(inc.openedAt)}${inc.closedAt ? ` · closed ${fmtDateTime(inc.closedAt)}` : ''}${inc.id ? ` · #${inc.id}` : ''}`)) + if (inc.lastError) box.append(el('pre', 'err', inc.lastError)) return box } diff --git a/infra/beacon/src/db.js b/infra/beacon/src/db.js index 1eccc9f..b0629c6 100644 --- a/infra/beacon/src/db.js +++ b/infra/beacon/src/db.js @@ -481,8 +481,43 @@ export async function queryStepSeries({ journey, window = '24 hours' }) { ` } +/** + * An `incidents` row in the shape the rest of Beacon speaks. + * + * The table is snake_case and every incident that never reached Postgres is + * camelCase: incidents.js:26-29 builds them that way, notify.js:32-34 sends + * them that way, and the redacted public view already publishes `openedAt` + * (server.js:265). `/api/incidents` used to serve whichever of the two it + * happened to be holding — DB rows while Postgres was up, the in-memory + * objects while it was down — so the page read `opened_at` and got + * `undefined` from precisely the fallback whose whole justification is + * working with the database gone. + * + * Normalised here, and not by making the in-memory objects snake_case, + * because this function is the only place in the process where a snake_case + * incident exists. One shape, chosen once, converted at the SQL boundary. + * `id` is deliberately unchanged: incidents.js and markNotified key on it. + * + * Exported for the same reason `chainSummary` is: it is the contract + * public/app.js reads, and a field it stops emitting is a blank line on the + * page rather than an error anywhere a test would see. + */ +export const incidentShape = (r) => ({ + id: r.id, + scope: r.scope, + subject: r.subject, + severity: r.severity, + cause: r.cause, + lastError: r.last_error, + openedAt: r.opened_at, + closedAt: r.closed_at, + failures: r.failures, + notifiedOpenAt: r.notified_open_at, + notifiedCloseAt: r.notified_close_at, +}) + export async function queryIncidents({ window = '30 days', open, limit = 100 } = {}) { - return sql` + const rows = await sql` SELECT * FROM incidents WHERE (closed_at IS NULL OR closed_at > now() - ${window}::interval) ${open === true ? sql`AND closed_at IS NULL` : sql``} @@ -490,6 +525,7 @@ export async function queryIncidents({ window = '30 days', open, limit = 100 } = ORDER BY opened_at DESC LIMIT ${Math.min(Number(limit) || 100, 500)} ` + return rows.map(incidentShape) } export async function openIncident({ scope, subject, severity, cause, lastError, at }) { @@ -501,7 +537,7 @@ export async function openIncident({ scope, subject, severity, cause, lastError, last_error = COALESCE(EXCLUDED.last_error, incidents.last_error) RETURNING * ` - return row + return row ? incidentShape(row) : null } export async function closeIncident({ scope, subject, at }) { @@ -510,7 +546,7 @@ export async function closeIncident({ scope, subject, at }) { WHERE scope = ${scope} AND subject = ${subject} AND closed_at IS NULL RETURNING * ` - return row ?? null + return row ? incidentShape(row) : null } export async function markNotified(id, field) { diff --git a/infra/beacon/src/env.js b/infra/beacon/src/env.js index 88f56a3..a19065f 100644 --- a/infra/beacon/src/env.js +++ b/infra/beacon/src/env.js @@ -114,8 +114,37 @@ export const env = { hearthSeed: process.env.BEACON_HEARTH_SEED_URL ?? 'http://hearth-testnet-seed:8645', hearthMiner1: process.env.BEACON_HEARTH_MINER1_URL ?? 'http://hearth-testnet-miner1:8645', hearthMiner2: process.env.BEACON_HEARTH_MINER2_URL ?? 'http://hearth-testnet-miner2:8645', + + // The Ethereum JSON-RPC listener, which is a SECOND listener in the same + // process on a DIFFERENT port — 8545, not the 8645 REST port above. It is + // the surface forge-pay speaks eth_* to (docker-compose.yml + // `HEARTH_TESTNET_RPC`), the one ForgeKeyvault's signatures are bound to, + // and the one a wallet or Hardhat would use. Losing it loses deposits and + // withdrawals while every /info probe stays green, so it is probed + // separately rather than assumed to follow the REST port. + hearthSeedRpc: process.env.BEACON_HEARTH_SEED_RPC_URL ?? 'http://hearth-testnet-seed:8545', + hearthMiner1Rpc: process.env.BEACON_HEARTH_MINER1_RPC_URL ?? 'http://hearth-testnet-miner1:8545', + hearthMiner2Rpc: process.env.BEACON_HEARTH_MINER2_RPC_URL ?? 'http://hearth-testnet-miner2:8545', }, + // The EIP-155 chain id this deployment's nodes must report. + // + // 7412 is `hearth-testnet`; 7411 is `hearth` (mainnet) and 7413 the + // in-process test chain — repos/hearth/node/src/params.js CHAIN_IDS is the + // register, and forge-keyvault carries its own copy in chains.ts:52. A node + // resolves the id from HEARTH_NETWORK and refuses to start on a network it + // does not know, so the drift this catches is the one that passes the node's + // own validation: an explicitly set, valid-but-wrong HEARTH_CHAIN_ID. That + // is not cosmetic. Under EIP-155 the id is the *only* thing keeping a + // testnet signature from being valid on mainnet, and keyvault refuses to + // sign a withdrawal whose chain id is not the one it expects — 403 + // binding_mismatch, with nothing on the node saying why. + // + // Set it on a mainnet deployment. There is deliberately no "whatever the + // node says" mode: a monitor that adopts the value it is meant to be + // checking asserts nothing at all. + hearthChainId: num(process.env.BEACON_HEARTH_CHAIN_ID, 7412), + // Public hostnames, probed through the edge rather than the compose network. // An internally-healthy stack with a broken tunnel is invisible to every // other check in this file, and is indistinguishable from a total outage to diff --git a/infra/beacon/src/journeys/platform.js b/infra/beacon/src/journeys/platform.js index ef71852..67224cb 100644 --- a/infra/beacon/src/journeys/platform.js +++ b/infra/beacon/src/journeys/platform.js @@ -146,6 +146,11 @@ defineJourney({ written: res.body.written, dropped: res.body.dropped, pending: res.body.pending, + // Beacon ingests from inside the compose network, so its own + // /ingest/client 429 check above can never see the *public* quota being + // exhausted — different socket peer, different bucket. This counter is + // process-wide and is the one place that shows up. + rateLimited: res.body.rateLimited, }) }) }, diff --git a/infra/beacon/src/server.js b/infra/beacon/src/server.js index f8b2aba..a364fde 100644 --- a/infra/beacon/src/server.js +++ b/infra/beacon/src/server.js @@ -175,10 +175,23 @@ function safePhases() { } } -function chainSummary(states) { +/** + * The chain panel's payload. Exported because it is the contract the panel in + * public/app.js reads, and a field the panel needs that this stops emitting is + * a silently blank column rather than an error anywhere. + */ +export function chainSummary(states) { const byName = new Map(states.map((s) => [s.name, s])) const node = (n) => { const s = byName.get(`hearth.${n}`) + // The Ethereum JSON-RPC listener is a SECOND server in the same process on + // a different port, and it fails soft (evmnode.js:211 logs `json-rpc + // listen failed` and carries on), so `hearth.` being up does not mean + // the node's eth_* surface is up. It is carried per node rather than + // folded into one chain-wide verdict because the consequence differs by + // node: losing it on the seed loses forge-pay's deposit watcher, losing it + // on a miner loses nothing anyone is currently calling. + const rpc = byName.get(`hearth.rpc.${n}`) return { name: n, state: s?.state ?? 'pending', @@ -186,6 +199,12 @@ function chainSummary(states) { tip: s?.last?.detail?.tip ?? null, peers: s?.last?.detail?.peers ?? null, mempool: s?.last?.detail?.mempool ?? null, + rpcState: rpc?.state ?? 'pending', + rpcHeight: rpc?.last?.detail?.height ?? null, + // From the JSON-RPC probe when it answered, and from /info when it did + // not — a node whose RPC is dead still reports which chain it thinks it + // is on, and during that incident that is the first thing anyone asks. + chainId: rpc?.last?.detail?.chainId ?? s?.last?.detail?.chainId ?? null, } } const nodes = ['seed', 'miner1', 'miner2'].map(node) @@ -207,6 +226,11 @@ function chainSummary(states) { blockReward: supply.blockReward ?? null, commonsTreasury: supply.commonsTreasury ?? null, burnedTotal: supply.burnedTotal ?? null, + // The id every node here must report, alongside what they do report. This + // is the expectation and not a reading: the probes assert against it, and + // the panel shows it so that "7411, expected 7412" is legible without + // anyone having to remember which number this deployment is. + expectedChainId: env.hearthChainId, } } diff --git a/infra/beacon/src/targets.js b/infra/beacon/src/targets.js index 4078411..2d4b343 100644 --- a/infra/beacon/src/targets.js +++ b/infra/beacon/src/targets.js @@ -21,15 +21,28 @@ export const DEGRADED = 'degraded' export const DOWN = 'down' /** - * A fixed, throwaway Ed25519 key, generated once per process, used only to ask - * Hearth for a mining template. It never signs anything and never receives a - * coinbase — the point is to prove the node will *issue work*, which is the + * A fixed, throwaway secp256k1 key, generated once per process, used only to + * ask Hearth for a mining template. It never signs anything and never receives + * a coinbase — the point is to prove the node will *issue work*, which is the * one thing every miner on the network needs and no `/info` field reports. + * + * It is an UNCOMPRESSED 65-byte point (`04 || X || Y`) because that is the form + * the account-model node derives an address from — chain/miner.js:136 rejects + * anything else with `pub must be a 65-byte uncompressed secp256k1 key`. This + * used to be an 88-char SPKI DER Ed25519 key, which is what the UTXO node's + * REST surface wanted (node/src/rpc.js:134); when the testnet moved to the EVM + * node the probe kept sending the old shape and `hearth.work` went permanently + * red with a 400, on a check marked critical. A monitor that is always red is a + * monitor nobody reads, so this is the shape the deployed node accepts. */ -const probePub = crypto - .generateKeyPairSync('ed25519') - .publicKey.export({ type: 'spki', format: 'der' }) - .toString('hex') +const probePub = (() => { + const jwk = crypto.generateKeyPairSync('ec', { namedCurve: 'secp256k1' }).publicKey.export({ format: 'jwk' }) + // From the JWK rather than by slicing the DER: the coordinates are named + // there, so this cannot silently start hashing a length prefix if Node ever + // changes the encoding around them. Both are 32 bytes, left-padded. + const coord = (b64) => Buffer.from(b64, 'base64url').toString('hex').padStart(64, '0') + return `04${coord(jwk.x)}${coord(jwk.y)}` +})() const ok = (detail, latencyMs, statusCode) => ({ state: UP, detail, latencyMs, statusCode }) const degraded = (why, detail, latencyMs, statusCode) => ({ state: DEGRADED, error: why, detail, latencyMs, statusCode }) @@ -71,6 +84,103 @@ function http({ name, group, url, path = '/health', method = 'GET', proves, expe } } +// --------------------------------------------------------- JSON-RPC (8545) -- +// +// The Ethereum JSON-RPC listener is a SECOND listener in the same process on a +// DIFFERENT port, and everything above this line asks the REST API on 8645. +// Three facts make that gap dangerous rather than merely untidy: +// +// 1. It fails soft. `listenJsonRpc` in repos/hearth/node/src/evmnode.js:211 +// attaches an `error` handler that *logs* `json-rpc listen failed` and +// returns; the node then serves REST perfectly happily with no JSON-RPC +// at all. +// 2. Nothing else notices. The compose healthcheck fetches `/info` on 8645, +// and `/info` reports `jsonRpc: ":8545/"` from the *configured* port +// (evmnode.js:302) whether or not anything ever bound it — so the node +// advertises an endpoint that does not exist, and `docker compose ps` +// calls it healthy. +// 3. It is the port that carries money. forge-pay's deposit watcher speaks +// `eth_*` to `hearth-testnet-seed:8545` (docker-compose.yml +// `HEARTH_TESTNET_RPC`), which is how an EMBER deposit becomes Shards. +// +// So: probed directly, and probed for the chain id as well as for being up. + +/** + * One POST carrying a JSON-RPC 2.0 batch. + * + * A batch rather than n requests because the answers then come from one + * instant on one connection — a height read a second after a chain id can + * belong to a chain the node was not on when it answered. Responses are + * matched by `id` and never by position: the spec allows a batch to come back + * in any order, and hearth's own dispatch says so at the top of + * node/src/jsonrpc/server.js. + * + * @returns {Promise<{res:object, error:string|null, results:any[]}>} the failure + * is a value, exactly as in http.js — a monitor that throws its own errors + * reports green. + */ +async function rpcBatch(url, calls, timeoutMs) { + const payload = calls.map(([method, params], i) => ({ jsonrpc: '2.0', id: i + 1, method, params: params ?? [] })) + const res = await timed(url, { method: 'POST', body: payload, timeoutMs }) + if (res.error) return { res, error: res.error, results: [] } + // JSON-RPC keeps the transport at 200 even for an RPC-level error, so a + // non-200 here is the port, not the call: nginx, a tunnel login page, or + // nothing at all. + if (res.status !== 200) return { res, error: `expected 200, got ${res.status}`, results: [] } + if (!Array.isArray(res.body)) { + // A 200 that is not a batch response is something *else* answering on this + // port. Saying which beats "undefined is not an object" three frames up — + // the legacy REST getinfo handler answering here would read as an empty + // chain rather than as a misconfiguration, which is the exact confusion + // docker-compose.yml:462-465 splits the two ports to avoid. + return { res, error: `not a JSON-RPC batch response: ${(res.text || 'empty body').slice(0, 160)}`, results: [] } + } + const byId = new Map(res.body.map((r) => [r?.id, r])) + const results = [] + for (let i = 0; i < calls.length; i++) { + const r = byId.get(i + 1) + if (!r) return { res, error: `${calls[i][0]}: nothing in the batch carried id ${i + 1}`, results: [] } + if (r.error) return { res, error: `${calls[i][0]}: JSON-RPC error ${r.error.code} ${r.error.message}`, results: [] } + results.push(r.result) + } + return { res, error: null, results } +} + +/** Build a JSON-RPC probe. `assert` has the same three return shapes as http(). */ +function jsonrpc({ name, group, url, proves, calls, assert, timeoutMs, critical = true }) { + return { + name, + group, + kind: 'jsonrpc', + critical, + proves, + url: () => url(), + async run() { + const { res, error, results } = await rpcBatch(url(), calls, timeoutMs ?? env.probeTimeoutMs) + if (error) return down(error, res.latencyMs, res.status) + + let verdict + try { + verdict = assert(results, res) + } catch (err) { + return down(`assertion threw: ${err.message}`, res.latencyMs, res.status) + } + if (typeof verdict === 'string') return down(verdict, res.latencyMs, res.status) + if (verdict?.degraded) return degraded(verdict.degraded, verdict.detail ?? {}, res.latencyMs, res.status) + return ok(verdict ?? {}, res.latencyMs, res.status) + }, + } +} + +/** A hex quantity to a Number, or null if it is not one. Heights, not wei. */ +function quantity(hex) { + if (typeof hex !== 'string' || !/^0x[0-9a-f]+$/i.test(hex)) return null + const n = Number(BigInt(hex)) + return Number.isSafeInteger(n) ? n : null +} + +const hexQuantity = (n) => `0x${n.toString(16)}` + /** A static front-end. nginx returns 200 with an HTML shell no matter what. */ function frontend({ name, url, proves }) { return http({ @@ -264,6 +374,44 @@ export const TARGETS = [ frontend({ name: 'hearth-web', url: () => u.hearthWeb, proves: 'the explorer and wallet bundle is being served' }), frontend({ name: 'cv-web', url: () => u.cv, proves: 'the CV site is being served' }), + // The explorer's and the wallet's FIRST call, made the way a visitor's + // browser makes it: through the bundle's own origin. + // + // `hearth-web` above proves the HTML is served and `hearth.rpc.seed` below + // proves the node answers eth_*; neither proves the thing that was actually + // broken, which is the wire between them. The pages default to same-origin + // `/eth-rpc/` (web/assets/explorer/rpc.js) and nginx proxies that to + // HEARTH_ETH_RPC_UPSTREAM. When that variable named the REST port — the image + // default was `hearth-testnet-seed:8645` and compose set no environment at + // all — every eth_* call got HTTP 404 and `{"err":"this is the REST API …"}`, + // which the client reports as MalformedResponse. Both targets either side of + // this line stayed green through all of it, and the public explorer showed + // nothing but errors. + // + // A 404 body is not a JSON-RPC batch, so `rpcBatch` names exactly that. + jsonrpc({ + name: 'hearth-web.eth-rpc', + group: 'frontend', + url: () => `${u.hearthWeb.replace(/\/$/, '')}/eth-rpc/`, + proves: "the explorer's same-origin JSON-RPC proxy reaches the node's 8545 listener and returns a chain id", + calls: [['eth_chainId']], + assert: ([chainIdHex]) => { + const want = env.hearthChainId + const seen = quantity(chainIdHex) + if (seen === null) { + return `eth_chainId through the explorer's /eth-rpc/ proxy returned ${JSON.stringify(chainIdHex)}, not a hex quantity` + } + // Not fatal here — `hearth.rpc.seed` owns the chain-id assertion against + // the node itself. What this one adds is that the proxy reaches THAT + // node, and a wrong id through a working proxy means it is pointed at a + // different one. + if (seen !== want) { + return { degraded: `the explorer's proxy reaches a node on chain ${seen}, not ${want}`, detail: { chainId: seen, expected: want } } + } + return { chainId: seen, chainIdHex } + }, + }), + // ---------------------------------------------------------------- chain -- ...['seed', 'miner1', 'miner2'].map((node) => http({ @@ -284,6 +432,10 @@ export const TARGETS = [ peers: b.peers ?? 0, mempool: b.mempool ?? 0, network: b.network, + // Reported, not asserted. `hearth.rpc.*` asserts the chain id, over + // the surface that signs with it; this is here so the detail panel + // says which chain a node thinks it is on without a second request. + chainId: b.chainId ?? null, mining: b.mining ?? false, hashrate: b.hashrate ?? 0, } @@ -327,6 +479,182 @@ export const TARGETS = [ } }, }), + + // ------------------------------------------------------ chain · JSON-RPC -- + ...['seed', 'miner1', 'miner2'].map((node) => + jsonrpc({ + name: `hearth.rpc.${node}`, + group: 'chain', + url: () => u[node === 'seed' ? 'hearthSeedRpc' : node === 'miner1' ? 'hearthMiner1Rpc' : 'hearthMiner2Rpc'], + // The seed is the one forge-pay reads deposits from; the miners are + // non-critical only in the sense that no service points at them, and + // they are still the fastest way to tell "this node's RPC died" from + // "the RPC broke everywhere in the last deploy". + critical: node === 'seed', + proves: 'the Ethereum JSON-RPC listener is bound, on the expected chain id, in both encodings, and reporting a height', + calls: [['web3_clientVersion'], ['eth_chainId'], ['net_version'], ['eth_blockNumber']], + assert: ([client, chainIdHex, netVersion, blockNumberHex]) => { + const want = env.hearthChainId + const wantHex = hexQuantity(want) + + // The encodings are asserted before the values, and separately, because + // they are the halves that get swapped. `eth_chainId` is a hex quantity + // and `net_version` is a DECIMAL string — the one place in the whole + // surface where hex is wrong (node/src/jsonrpc/methods.js:623) — and + // getting the pair backwards is what makes MetaMask refuse a network + // outright (repos/hearth/docs/network-config.md:68). Both are derived + // from the same integer by different code, so checking one proves + // nothing about the other. + if (typeof chainIdHex !== 'string' || !/^0x[0-9a-f]+$/i.test(chainIdHex)) { + return `eth_chainId returned ${JSON.stringify(chainIdHex)}, which is not a 0x hex quantity — a wallet cannot read this` + } + if (typeof netVersion !== 'string' || !/^[0-9]+$/.test(netVersion)) { + return `net_version returned ${JSON.stringify(netVersion)}; it must be a decimal string, not hex` + } + + // A wrong-but-valid chain id is the drift the node cannot catch itself: + // params.js refuses to start on an unknown *network*, but an explicitly + // set HEARTH_CHAIN_ID is taken at its word (params.js:47-49). It does + // not misroute anything — balances still read correctly — which is why + // it stays invisible. What it does is remove the only thing EIP-155 + // gives us: at 7411 every hearth-testnet transaction becomes replayable + // on hearth mainnet and back, so the faucet becomes a way to drain the + // account funding it. ForgeKeyvault resolves the id independently + // (forge-keyvault/src/chains.ts:52) and starts answering withdrawals + // with 403 binding_mismatch, from a service that cannot say why. + const seen = quantity(chainIdHex) + if (seen !== want) { + return `chain id is ${seen} (${chainIdHex}), expected ${want} — every signature bound to this chain is now bound to somebody else's` + } + if (netVersion !== String(want)) { + return `net_version says ${netVersion} but eth_chainId says ${seen} — the node disagrees with itself about which chain it is` + } + + const height = quantity(blockNumberHex) + if (height === null) return `eth_blockNumber returned ${JSON.stringify(blockNumberHex)}, which is not a hex quantity` + + return { + chainId: seen, + chainIdHex, + netVersion, + height, + client: typeof client === 'string' ? client : null, + } + }, + }), + ), + { + name: 'hearth.rpc.state', + group: 'chain', + kind: 'jsonrpc', + critical: true, + url: () => u.hearthSeedRpc, + // `hearth.consensus` compares tips, and a tip comparison can only speak + // about nodes that happen to be at the same height right now — two nodes a + // block apart are "lag" and are not compared at all. That leaves a fork + // *behind* the tip invisible for as long as the heights keep differing. + // + // This asks the question the other way round: pick one settled height every + // node has, and make each of them state the block and the state root it + // holds there. Same height, different block, is a fork the tip comparison + // would have called lag. Same block, different state root, is worse and + // rarer: the header hash commits to the state root, so it means two nodes + // executed the same transactions into different state and one of them is + // serving balances nobody else agrees with — which on this estate means + // deposits credited against a state root no other node has. + proves: 'the nodes computed the same state root for the same settled block, not merely the same height', + async run() { + const nodes = [ + ['seed', u.hearthSeedRpc], + ['miner1', u.hearthMiner1Rpc], + ['miner2', u.hearthMiner2Rpc], + ] + const started = performance.now() + const timeoutMs = env.probeTimeoutMs + + const tips = await Promise.all( + nodes.map(async ([name, url]) => { + const { error, results } = await rpcBatch(url, [['eth_blockNumber']], timeoutMs) + return [name, error ? null : quantity(results[0])] + }), + ) + const reachable = tips.filter(([, h]) => h !== null) + // The per-node probes above carry the outage. Two nodes is the minimum + // that can disagree, so below that there is nothing for this check to + // say and saying it in red would be reporting the same fault twice. + if (reachable.length < 2) { + return degraded( + `only ${reachable.length} node answered eth_blockNumber over JSON-RPC — nothing to compare`, + { reachable: reachable.map(([n]) => n) }, + performance.now() - started, + 0, + ) + } + + // Three blocks below the lowest tip: deep enough that a block arriving + // mid-cycle cannot make two honest nodes look like a fork, shallow enough + // that a divergence is found in a minute rather than at the next deploy. + const at = Math.max(0, Math.min(...reachable.map(([, h]) => h)) - 3) + + const held = await Promise.all( + reachable.map(async ([name]) => { + const url = nodes.find(([n]) => n === name)[1] + const { error, results } = await rpcBatch(url, [['eth_getBlockByNumber', [hexQuantity(at), false]]], timeoutMs) + return [name, error ? { error } : { block: results[0] }] + }), + ) + + const latencyMs = performance.now() - started + const roots = new Map() + for (const [name, answer] of held) { + if (answer.error) return down(`${name}: eth_getBlockByNumber(${at}) — ${answer.error}`, latencyMs, 0) + const block = answer.block + // Null is "no such block" and this height was chosen to be below every + // node's own reported tip, so a node that cannot produce it is telling + // us its index and its height disagree. + if (!block) { + return down( + `${name} reports a tip above ${at} but has no block at ${at} — its height and its block index disagree`, + latencyMs, + 0, + ) + } + roots.set(name, { hash: block.hash, stateRoot: block.stateRoot }) + } + + const [[firstName, first]] = [...roots] + for (const [name, seen] of roots) { + if (seen.hash !== first.hash) { + return down( + `fork at settled height ${at}: ${firstName} holds ${String(first.hash).slice(0, 18)} and ${name} holds ${String(seen.hash).slice(0, 18)}`, + latencyMs, + 0, + ) + } + if (seen.stateRoot !== first.stateRoot) { + return down( + `${firstName} and ${name} agree the block at ${at} is ${String(first.hash).slice(0, 18)} but disagree on its state root ` + + `(${String(first.stateRoot).slice(0, 18)} vs ${String(seen.stateRoot).slice(0, 18)}) — the header hash commits to the root, ` + + 'so one of these nodes is serving state that does not match the block it names', + latencyMs, + 0, + ) + } + } + + return ok( + { + at, + nodes: roots.size, + spread: Math.max(...reachable.map(([, h]) => h)) - Math.min(...reachable.map(([, h]) => h)), + stateRoot: String(first.stateRoot).slice(0, 18), + hash: String(first.hash).slice(0, 18), + }, + latencyMs, + 200, + ) + }, + }, ] // -------------------------------------------------------- derived checks -- @@ -390,6 +718,78 @@ export const DERIVED = [ return ok(detail) }, }, + { + name: 'hearth.rpc.liveness', + group: 'chain', + kind: 'derived', + critical: env.chainExpectBlocks, + // Two questions that only look like one, and neither is answered by + // `hearth.liveness` — that check reads the REST port. + // + // The first is whether `eth_blockNumber` moves, because that is the number + // forge-pay's deposit watcher polls: a watcher whose height never changes + // credits nothing and says nothing. + // + // The second is whether the two listeners agree. They are the same process + // reading the same chain object, so they cannot legitimately be more than a + // block apart — a JSON-RPC height sitting behind the REST height is the RPC + // half serving a snapshot of a chain that has moved on, which reads as "no + // deposits today" rather than as an outage. + proves: env.chainExpectBlocks + ? 'the JSON-RPC height advances, and matches the height the REST API reports for the same node' + : 'the JSON-RPC height matches the REST height; this deployment does not mine, so a stall is not its fault', + run(results) { + const nodes = ['seed', 'miner1', 'miner2'] + .map((n) => [n, results.get(`hearth.rpc.${n}`), results.get(`hearth.${n}`)]) + .filter(([, rpc]) => rpc && rpc.state !== DOWN && typeof rpc.detail?.height === 'number') + if (!nodes.length) return down('no node reported a height over JSON-RPC', 0) + + // Same cycle, same Promise.all, so the two readings are milliseconds + // apart on a 15s chain. One block of skew is a block landing between + // them; two is the number rounded generously in the node's favour. + for (const [name, rpc, rest] of nodes) { + if (!rest || rest.state === DOWN || typeof rest.detail?.height !== 'number') continue + const skew = rpc.detail.height - rest.detail.height + if (Math.abs(skew) > 2) { + return down( + `${name}: JSON-RPC reports height ${rpc.detail.height} while its own REST API reports ${rest.detail.height} — ` + + 'the two listeners are the same process reading the same chain, so they are serving different states', + 0, + ) + } + } + + const best = Math.max(...nodes.map(([, rpc]) => rpc.detail.height)) + const detail = { + height: best, + nodes: nodes.length, + expectBlocks: env.chainExpectBlocks, + skew: Object.fromEntries( + nodes + .filter(([, , rest]) => typeof rest?.detail?.height === 'number') + .map(([n, rpc, rest]) => [n, rpc.detail.height - rest.detail.height]), + ), + } + + const prev = lastHeight.get('rpc') + const now = Date.now() + if (prev === undefined || best > prev.height) { + lastHeight.set('rpc', { height: best, at: now }) + return ok({ ...detail, stalledSec: 0, advancedBy: prev === undefined ? 0 : best - prev.height }) + } + + const stalledSec = Math.round((now - prev.at) / 1000) + // Identical reasoning, and identical thresholds, to `hearth.liveness` + // directly above: with no local hashrate a flat height is a statement + // about the network rather than a fault in this stack. + if (!env.chainExpectBlocks) { + return ok({ ...detail, stalledSec, note: 'no local hashrate — the chain advances only when an external miner runs' }) + } + if (stalledSec > 300) return down(`eth_blockNumber has not moved in ${stalledSec}s (block time is 15s)`, 0) + if (stalledSec > 90) return degraded(`eth_blockNumber has not moved in ${stalledSec}s`, { ...detail, stalledSec }, 0) + return ok({ ...detail, stalledSec }) + }, + }, { name: 'hearth.consensus', group: 'chain', diff --git a/infra/beacon/test/chainpanel.test.js b/infra/beacon/test/chainpanel.test.js new file mode 100644 index 0000000..ed06ed4 --- /dev/null +++ b/infra/beacon/test/chainpanel.test.js @@ -0,0 +1,61 @@ +// The chain panel's payload. +// +// The probes in rpc.test.js decide whether the JSON-RPC surface is up. This +// decides whether anybody can *see* that they did: the /api/status chain +// summary is the only thing the panel in public/app.js reads, so a per-node +// verdict that never reaches it is a check nobody watches. +// +// The state each of these pins is the one the ticket is about — REST green on +// 8645, JSON-RPC dead on 8545 — which before this change was not merely +// unreported but unreportable, because no probe asked. + +import { test } from 'node:test' +import assert from 'node:assert/strict' + +process.env.BEACON_HEARTH_CHAIN_ID = '7412' +const { chainSummary } = await import('../src/server.js') + +/** The shape runner.js stores: a live `state`, and the last result under `last`. */ +const probe = (name, state, detail) => ({ name, state, last: { state, detail } }) + +test('a node whose REST port is green and whose JSON-RPC is dead says so, per node', () => { + const c = chainSummary([ + probe('hearth.seed', 'up', { height: 2933, tip: '0xaaa', peers: 2, mempool: 0, chainId: 7412 }), + probe('hearth.rpc.seed', 'down', undefined), + probe('hearth.miner1', 'up', { height: 2933, tip: '0xaaa', peers: 1, mempool: 0, chainId: 7412 }), + probe('hearth.rpc.miner1', 'up', { height: 2933, chainId: 7412 }), + ]) + + const seed = c.nodes.find((n) => n.name === 'seed') + assert.equal(seed.state, 'up', 'the REST verdict is meant to stay green — that is the whole finding') + assert.equal(seed.rpcState, 'down') + // The RPC probe returned nothing, so the chain id falls back to /info: a node + // whose eth_* surface is dead still knows which chain it thinks it is on, and + // that is the first question asked during the incident. + assert.equal(seed.chainId, 7412) + + const miner1 = c.nodes.find((n) => n.name === 'miner1') + assert.equal(miner1.rpcState, 'up') + assert.equal(miner1.rpcHeight, 2933) +}) + +test('a node that has never been probed is pending on both surfaces, not up', () => { + const c = chainSummary([]) + for (const n of c.nodes) { + assert.equal(n.state, 'pending') + assert.equal(n.rpcState, 'pending') + assert.equal(n.chainId, null) + } +}) + +test('the expected chain id is published as an expectation, not read back off a node', () => { + const c = chainSummary([ + // A node insisting it is hearth mainnet. The summary must carry both + // numbers, or the panel can only print the wrong one with nothing to + // compare it against. + probe('hearth.seed', 'up', { height: 10, tip: '0xa', chainId: 7411 }), + probe('hearth.rpc.seed', 'down', undefined), + ]) + assert.equal(c.expectedChainId, 7412) + assert.equal(c.nodes.find((n) => n.name === 'seed').chainId, 7411) +}) diff --git a/infra/beacon/test/incidents.test.js b/infra/beacon/test/incidents.test.js new file mode 100644 index 0000000..52750fb --- /dev/null +++ b/infra/beacon/test/incidents.test.js @@ -0,0 +1,136 @@ +// One incident shape, whichever side of the database it came from. +// +// `/api/status` serves `openIncidents()` unconditionally, and `/api/incidents` +// falls back to it whenever Postgres is down — the view whose entire +// justification is that it still works with the database gone. Those objects +// are camelCase; the `incidents` table is snake_case; the page read the table's +// names. So the banner's age was blank in every deployment at all times, and +// during a Postgres outage the incidents view additionally lost the duration, +// both timestamps and the error text, and listed a cascade oldest-first. +// +// Nothing here mocks Postgres. The two producers of an incident are asserted to +// agree on their keys, which is the property the page actually depends on and +// the one that silently stopped holding. + +import { test } from 'node:test' +import assert from 'node:assert/strict' +import { readFileSync } from 'node:fs' + +process.env.BEACON_DATABASE_URL ??= 'postgres://beacon:beacon@127.0.0.1:5432/beacon' + +const { record, openIncidents } = await import('../src/incidents.js') +const { incidentShape } = await import('../src/db.js') + +// The reads public/app.js performs, transcribed. Kept as data rather than as +// prose so that renaming a field here fails the test that names it. +const BANNER_READS = ['subject', 'scope', 'cause', 'openedAt', 'closedAt'] +const CARD_READS = ['severity', 'scope', 'subject', 'cause', 'openedAt', 'closedAt', 'lastError'] + +/** A row exactly as `SELECT * FROM incidents` returns it. */ +const dbRow = () => ({ + id: 41, + scope: 'target', + subject: 'forge-pay', + severity: 'major', + opened_at: new Date('2026-07-29T09:00:00Z'), + closed_at: null, + cause: 'connection refused', + last_error: 'ECONNREFUSED 10.0.0.4:8080', + failures: 3, + notified_open_at: null, + notified_close_at: null, +}) + +// The finding, reproduced: Postgres is down, so nothing was ever persisted and +// the page is rendering the in-memory objects. Before the fix every one of +// these reads was `undefined`. +test('with Postgres down the incidents the page renders carry the fields it reads', async () => { + await record({ scope: 'target', subject: 'forge-pay', transition: 'opened', cause: 'connection refused', error: 'ECONNREFUSED', severity: 'major' }) + const [inc] = openIncidents() + + for (const field of CARD_READS) { + assert.ok(field in inc, `the incidents view reads inc.${field}, and the in-memory incident has no such key`) + } + // The two that vanished, named individually — a blank timestamp is the + // symptom and `since()`/`fmtDateTime()` both render '—' for undefined, so + // this fails silently on the page and only a test can see it. + assert.ok(inc.openedAt instanceof Date, 'the banner and the card both date the incident from openedAt') + assert.equal(inc.lastError, 'ECONNREFUSED') + assert.equal(inc.closedAt, null) + + for (const key of Object.keys(inc)) { + assert.ok(!key.includes('_'), `${key} is snake_case — the wire shape is camelCase on both sides now`) + } + + await record({ scope: 'target', subject: 'forge-pay', transition: 'closed' }) +}) + +test('a row out of Postgres arrives in the same shape as one that never reached it', async () => { + const mapped = incidentShape(dbRow()) + + for (const field of CARD_READS) assert.ok(field in mapped, `the incidents view reads inc.${field}`) + assert.equal(mapped.openedAt.toISOString(), '2026-07-29T09:00:00.000Z') + assert.equal(mapped.lastError, 'ECONNREFUSED 10.0.0.4:8080') + assert.equal(mapped.closedAt, null) + // Not renamed: incidents.js reads row.id back onto the in-memory object and + // markNotified keys on it. + assert.equal(mapped.id, 41) + + for (const key of Object.keys(mapped)) { + assert.ok(!key.includes('_'), `${key} is snake_case — the SQL boundary is where that stops`) + } + + await record({ scope: 'target', subject: 'lantern', transition: 'opened', cause: 'timeout', error: 'timed out after 5000ms' }) + const [memory] = openIncidents() + // The property the whole ticket is about: `/api/incidents` returns one of + // these two depending on whether Postgres happens to be up, and the page + // cannot tell which it got. + for (const field of BANNER_READS) { + assert.equal(field in mapped, field in memory, `${field} exists on one side of the database and not the other`) + } + await record({ scope: 'target', subject: 'lantern', transition: 'closed' }) +}) + +// The console is a browser module and needs a DOM, so it cannot be imported +// here to be rendered. What can be checked is the thing that broke: it read +// field names that no incident on either side of the database has. Neither +// shape throws on a missing key and both formatters render '—' for undefined, +// so this mismatch has no runtime signal at all — which is why it survived to +// be found by an audit rather than by the page looking wrong. +test('the console reads no incident field the API does not send', () => { + const src = readFileSync(new URL('../public/app.js', import.meta.url), 'utf8') + for (const field of ['opened_at', 'closed_at', 'last_error']) { + const hits = [...src.matchAll(new RegExp(`\\.${field}\\b`, 'g'))] + assert.equal(hits.length, 0, `public/app.js still reads .${field}; incidents are camelCase on both sides now`) + } + // And it does read the camelCase ones, so the check above cannot be passed + // by deleting the feature. + for (const field of ['openedAt', 'closedAt', 'lastError']) { + assert.ok(src.includes(`inc.${field}`) || src.includes(`i.${field}`), `public/app.js never reads ${field}`) + } +}) + +// The sort at app.js:1578 is `Date.parse(b.openedAt) - Date.parse(a.openedAt)`. +// Against the old snake_case reads both sides were NaN, which is a no-op +// comparator rather than a crash: V8's stable sort left the incidents in +// `openIncidents()` Map insertion order, i.e. oldest first — the reverse of +// what the view asks for, in the one place a cascade is read. +test('a cascade of in-memory incidents sorts newest first, not oldest first', async () => { + const subjects = ['postgres', 'forge-pay', 'lantern', 'forge-keyvault'] + for (const [i, subject] of subjects.entries()) { + await record({ scope: 'target', subject, transition: 'opened', cause: 'down', at: new Date(Date.UTC(2026, 6, 29, 9, i)) }) + } + + const sorted = [...openIncidents()].sort((a, b) => { + const aOpen = a.closedAt ? 0 : 1 + const bOpen = b.closedAt ? 0 : 1 + if (aOpen !== bOpen) return bOpen - aOpen + return Date.parse(b.openedAt ?? '') - Date.parse(a.openedAt ?? '') + }) + + assert.deepEqual(sorted.map((i) => i.subject), [...subjects].reverse()) + // And the finding's actual question — which service went first — is answerable. + assert.equal(sorted.at(-1).subject, 'postgres') + + for (const subject of subjects) await record({ scope: 'target', subject, transition: 'closed' }) +}) diff --git a/infra/beacon/test/rpc.test.js b/infra/beacon/test/rpc.test.js new file mode 100644 index 0000000..bdb4386 --- /dev/null +++ b/infra/beacon/test/rpc.test.js @@ -0,0 +1,462 @@ +// The chain probes, driven against a stub node over real HTTP. +// +// The defect these exist for is not a bug in a probe — it is that the probes +// were not there. Every hearth check asked the REST port on 8645, the Ethereum +// JSON-RPC listener on 8545 was asked nothing at all, and that listener fails +// soft (repos/hearth/node/src/evmnode.js:211 logs `json-rpc listen failed` and +// carries on). So the estate could lose the surface forge-pay reads deposits +// from and every panel would stay green. +// +// A test cannot assert the absence of an oversight, so what these assert is the +// verdict: with the REST node answering perfectly, each of the four ways the +// JSON-RPC surface can be wrong — gone, on the wrong chain, encoding the chain +// id the wrong way round, disagreeing with its neighbours about history — must +// come out `down` with a message that names the fault. +// +// A stub rather than a real node because these are assertions about Beacon. +// The contract the stub implements is asserted against the real listener in +// repos/hearth: node/test/jsonrpc.js and node/test/evm-rpc.js. + +import { test } from 'node:test' +import assert from 'node:assert/strict' +import http from 'node:http' + +// ---------------------------------------------------------------- stubs -- + +const hex = (n) => `0x${n.toString(16)}` +/** Deterministic per-height, so two stubs on the same chain agree by default. */ +const hashAt = (n, chain = 'a') => `0x${chain.repeat(2)}${String(n).padStart(62, '0')}` +const rootAt = (n, chain = 'a') => `0x${chain.repeat(4)}${String(n).padStart(60, '0')}` + +function answer(state, method, params) { + switch (method) { + case 'web3_clientVersion': + return { result: 'Hearth/v0.2.0/test' } + case 'eth_chainId': + return { result: state.chainIdHex ?? hex(state.chainId) } + case 'net_version': + return { result: state.netVersion ?? String(state.chainId) } + case 'eth_blockNumber': + return { result: hex(state.height) } + case 'eth_getBlockByNumber': { + const at = Number(BigInt(params[0])) + if (at > state.height) return { result: null } + if (state.missingBelow !== undefined && at <= state.missingBelow) return { result: null } + return { + result: { + number: hex(at), + hash: (state.hash ?? hashAt)(at), + stateRoot: (state.root ?? rootAt)(at), + }, + } + } + default: + return { error: { code: -32601, message: `the method ${method} does not exist` } } + } +} + +/** + * A JSON-RPC node. `state` is mutable by the test after listening, so a stub + * can drift on to the wrong chain half way through without being restarted. + */ +async function jsonRpcStub(state) { + const server = http.createServer((req, res) => { + let body = '' + req.on('data', (c) => { + body += c + }) + req.on('end', () => { + // Something that is not a JSON-RPC endpoint answering on this port: the + // legacy REST getinfo handler, an nginx page, a tunnel login screen. + if (state.raw !== undefined) { + res.writeHead(200, { 'content-type': 'application/json' }) + res.end(state.raw) + return + } + const payload = JSON.parse(body) + const one = (msg) => { + const r = answer(state, msg.method, msg.params ?? []) + return r.error + ? { jsonrpc: '2.0', id: msg.id, error: r.error } + : { jsonrpc: '2.0', id: msg.id, result: r.result } + } + // Deliberately answered out of order. Beacon matches on `id`, and the + // spec allows any order; a probe that matched on position would pass + // every day until the first client library that did not. + const out = Array.isArray(payload) ? payload.map(one).reverse() : one(payload) + res.writeHead(200, { 'content-type': 'application/json' }) + res.end(JSON.stringify(out)) + }) + }) + await new Promise((r) => server.listen(0, '127.0.0.1', r)) + return { server, state, url: `http://127.0.0.1:${server.address().port}` } +} + +/** The REST surface on 8645 — the one that stays green while 8545 is dead. */ +async function restStub(state) { + const server = http.createServer((req, res) => { + if (!req.url.startsWith('/info')) { + // Verbatim from node/src/evmnode.js: the REST server's answer to anything + // it does not route, including every eth_* POST that lands here because a + // proxy named 8645 where it meant 8545. + res.writeHead(404, { 'content-type': 'application/json' }).end( + JSON.stringify({ + err: 'this is the REST API — the Ethereum JSON-RPC endpoint is a different port', + jsonRpc: 'port 8545, path /', + }), + ) + return + } + res.writeHead(200, { 'content-type': 'application/json' }) + res.end( + JSON.stringify({ + network: 'hearth-testnet', + chainId: state.chainId, + height: state.height, + tip: (state.hash ?? hashAt)(state.height), + peers: 2, + mempool: 0, + mining: true, + hashrate: 1, + // Printed from the *configured* port whether or not anything bound it — + // evmnode.js:302. This field is why a dead listener is invisible. + jsonRpc: ':8545/', + }), + ) + }) + await new Promise((r) => server.listen(0, '127.0.0.1', r)) + return { server, state, url: `http://127.0.0.1:${server.address().port}` } +} + +/** + * The explorer's own nginx, reduced to the one thing that was wrong with it: + * `/eth-rpc/` forwards to whatever upstream it was configured with. Pointed at + * a JSON-RPC stub it is the fixed deployment; pointed at a REST stub it is the + * shipped one, where the visitor's first `eth_chainId` came back as HTTP 404 + * and `{"err":"this is the REST API …"}` while every other panel stayed green. + */ +async function webStub(upstream) { + const server = http.createServer((req, res) => { + let body = '' + req.on('data', (c) => { + body += c + }) + req.on('end', async () => { + if (!req.url.startsWith('/eth-rpc/')) { + res.writeHead(404).end('nginx: no such path') + return + } + // proxy_pass http://UPSTREAM/ — the prefix is stripped, as nginx strips it. + const up = await fetch(upstream + '/', { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body, + }) + const text = await up.text() + res.writeHead(up.status, { 'content-type': up.headers.get('content-type') ?? 'application/json' }) + res.end(text) + }) + }) + await new Promise((r) => server.listen(0, '127.0.0.1', r)) + return { server, url: `http://127.0.0.1:${server.address().port}` } +} + +const close = (s) => new Promise((r) => s.server.close(r)) + +// ------------------------------------------------------------- harness -- + +// env.js reads process.env once, at import. The import of targets.js is +// therefore dynamic and happens after this line, or every URL below would be a +// container name that does not resolve. +process.env.BEACON_HEARTH_CHAIN_ID = '7412' +process.env.BEACON_PROBE_TIMEOUT_MS = '3000' + +const { env } = await import('../src/env.js') +const { TARGETS, DERIVED } = await import('../src/targets.js') + +const target = (name) => { + const t = [...TARGETS, ...DERIVED].find((x) => x.name === name) + assert.ok(t, `no such target: ${name}`) + return t +} + +/** Point Beacon at a set of stubs for the length of one test. */ +function aim({ seedRpc, miner1Rpc, miner2Rpc, seed, web }) { + if (seedRpc !== undefined) env.urls.hearthSeedRpc = seedRpc + if (miner1Rpc !== undefined) env.urls.hearthMiner1Rpc = miner1Rpc + if (miner2Rpc !== undefined) env.urls.hearthMiner2Rpc = miner2Rpc + if (seed !== undefined) env.urls.hearthSeed = seed + if (web !== undefined) env.urls.hearthWeb = web +} + +// --------------------------------------------------------------- tests -- + +test('a healthy node passes, and reports the chain id in both encodings', async () => { + const node = await jsonRpcStub({ chainId: 7412, height: 2933 }) + try { + aim({ seedRpc: node.url }) + const r = await target('hearth.rpc.seed').run() + assert.equal(r.state, 'up', r.error) + assert.equal(r.detail.chainId, 7412) + assert.equal(r.detail.chainIdHex, '0x1cf4') + assert.equal(r.detail.netVersion, '7412') + assert.equal(r.detail.height, 2933) + } finally { + await close(node) + } +}) + +// The finding, reproduced. This is the whole ticket in one test: the REST port +// answers a climbing height, and the JSON-RPC listener never bound. +test('a JSON-RPC listener that never bound is down, while the REST port stays up', async () => { + const rest = await restStub({ chainId: 7412, height: 2933 }) + // Bound and immediately released: the port is real, nothing is listening on + // it, which is exactly the state evmnode.js leaves behind after an EADDRINUSE. + const dead = await jsonRpcStub({ chainId: 7412, height: 2933 }) + const deadUrl = dead.url + await close(dead) + + try { + aim({ seed: rest.url, seedRpc: deadUrl }) + + const restResult = await target('hearth.seed').run() + assert.equal(restResult.state, 'up', 'the REST probe is meant to stay green — that is the point') + assert.equal(restResult.detail.height, 2933) + + const rpcResult = await target('hearth.rpc.seed').run() + assert.equal(rpcResult.state, 'down') + assert.match(rpcResult.error, /ECONNREFUSED|fetch failed|timed out/) + } finally { + await close(rest) + } +}) + +test('a valid but wrong chain id is down, and the message names both ids', async () => { + // 7411 is hearth mainnet. A node running the testnet with HEARTH_CHAIN_ID + // set to it passes the node's own validation (params.js:47-49) and makes + // every testnet signature replayable on mainnet. + const node = await jsonRpcStub({ chainId: 7411, height: 10 }) + try { + aim({ seedRpc: node.url }) + const r = await target('hearth.rpc.seed').run() + assert.equal(r.state, 'down') + assert.match(r.error, /7411/) + assert.match(r.error, /7412/) + } finally { + await close(node) + } +}) + +test('net_version encoded as hex is down — the one place hex is wrong', async () => { + const node = await jsonRpcStub({ chainId: 7412, height: 10, netVersion: '0x1cf4' }) + try { + aim({ seedRpc: node.url }) + const r = await target('hearth.rpc.seed').run() + assert.equal(r.state, 'down') + assert.match(r.error, /net_version/) + assert.match(r.error, /decimal/) + } finally { + await close(node) + } +}) + +test('eth_chainId encoded as a decimal string is down', async () => { + const node = await jsonRpcStub({ chainId: 7412, height: 10, chainIdHex: '7412' }) + try { + aim({ seedRpc: node.url }) + const r = await target('hearth.rpc.seed').run() + assert.equal(r.state, 'down') + assert.match(r.error, /eth_chainId/) + assert.match(r.error, /hex quantity/) + } finally { + await close(node) + } +}) + +test('the two encodings disagreeing with each other is down', async () => { + const node = await jsonRpcStub({ chainId: 7412, height: 10, netVersion: '7411' }) + try { + aim({ seedRpc: node.url }) + const r = await target('hearth.rpc.seed').run() + assert.equal(r.state, 'down') + assert.match(r.error, /disagrees with itself/) + } finally { + await close(node) + } +}) + +test('something that is not a JSON-RPC endpoint answering 200 is down, and says so', async () => { + // The legacy REST getinfo handler mounted on 8545 by mistake. It answers 200 + // with valid JSON, which is why this has to be checked rather than assumed. + const node = await jsonRpcStub({ raw: '{"network":"hearth-testnet","height":2933}' }) + try { + aim({ seedRpc: node.url }) + const r = await target('hearth.rpc.seed').run() + assert.equal(r.state, 'down') + assert.match(r.error, /not a JSON-RPC batch response/) + } finally { + await close(node) + } +}) + +test('three nodes on one chain agree on the state root at a settled height', async () => { + const nodes = await Promise.all([ + jsonRpcStub({ chainId: 7412, height: 2933 }), + jsonRpcStub({ chainId: 7412, height: 2931 }), + jsonRpcStub({ chainId: 7412, height: 2932 }), + ]) + try { + aim({ seedRpc: nodes[0].url, miner1Rpc: nodes[1].url, miner2Rpc: nodes[2].url }) + const r = await target('hearth.rpc.state').run() + assert.equal(r.state, 'up', r.error) + // Three below the lowest tip, so every node has it. + assert.equal(r.detail.at, 2928) + assert.equal(r.detail.nodes, 3) + assert.equal(r.detail.spread, 2) + } finally { + await Promise.all(nodes.map(close)) + } +}) + +test('a fork behind the tip is down, at heights where the tip comparison sees only lag', async () => { + // The heights differ, so `hearth.consensus` — which compares only nodes that + // are at the *same* height — has nothing to compare and reports green. + const nodes = await Promise.all([ + jsonRpcStub({ chainId: 7412, height: 2933 }), + jsonRpcStub({ chainId: 7412, height: 2931, hash: (n) => hashAt(n, 'b'), root: (n) => rootAt(n, 'b') }), + jsonRpcStub({ chainId: 7412, height: 2932 }), + ]) + try { + aim({ seedRpc: nodes[0].url, miner1Rpc: nodes[1].url, miner2Rpc: nodes[2].url }) + + const consensus = target('hearth.consensus').run( + new Map([ + ['hearth.seed', { state: 'up', detail: { height: 2933, tip: '0xaa' } }], + ['hearth.miner1', { state: 'up', detail: { height: 2931, tip: '0xbb' } }], + ['hearth.miner2', { state: 'up', detail: { height: 2932, tip: '0xaa' } }], + ]), + ) + assert.equal(consensus.state, 'up', 'the tip comparison is supposed to miss this — that is why the new check exists') + + const r = await target('hearth.rpc.state').run() + assert.equal(r.state, 'down') + assert.match(r.error, /fork at settled height 2928/) + } finally { + await Promise.all(nodes.map(close)) + } +}) + +test('two nodes naming the same block with different state roots is down', async () => { + const nodes = await Promise.all([ + jsonRpcStub({ chainId: 7412, height: 2933 }), + jsonRpcStub({ chainId: 7412, height: 2933, root: (n) => rootAt(n, 'c') }), + ]) + try { + aim({ seedRpc: nodes[0].url, miner1Rpc: nodes[1].url, miner2Rpc: 'http://127.0.0.1:1' }) + const r = await target('hearth.rpc.state').run() + assert.equal(r.state, 'down') + assert.match(r.error, /disagree on its state root/) + } finally { + await Promise.all(nodes.map(close)) + } +}) + +test('fewer than two nodes answering is amber, not red — the per-node probes carry that', async () => { + const node = await jsonRpcStub({ chainId: 7412, height: 2933 }) + try { + aim({ seedRpc: node.url, miner1Rpc: 'http://127.0.0.1:1', miner2Rpc: 'http://127.0.0.1:1' }) + const r = await target('hearth.rpc.state').run() + assert.equal(r.state, 'degraded') + } finally { + await close(node) + } +}) + +test('a node whose JSON-RPC height lags its own REST height is down', () => { + const results = new Map([ + ['hearth.rpc.seed', { state: 'up', detail: { height: 2900 } }], + ['hearth.seed', { state: 'up', detail: { height: 2933 } }], + ]) + const r = target('hearth.rpc.liveness').run(results) + assert.equal(r.state, 'down') + assert.match(r.error, /2900/) + assert.match(r.error, /2933/) +}) + +test('one block of skew between the two listeners is not an incident', () => { + const results = new Map([ + ['hearth.rpc.seed', { state: 'up', detail: { height: 2932 } }], + ['hearth.seed', { state: 'up', detail: { height: 2933 } }], + ]) + const r = target('hearth.rpc.liveness').run(results) + assert.equal(r.state, 'up', r.error) + assert.equal(r.detail.skew.seed, -1) +}) + +test('a JSON-RPC height that stops moving eventually goes down', () => { + const check = target('hearth.rpc.liveness') + const at = (height) => new Map([['hearth.rpc.seed', { state: 'up', detail: { height } }]]) + + assert.equal(check.run(at(2933)).state, 'up') + // Same height on the next cycle is not yet a stall: the recorded timestamp is + // seconds old, and one missed block on a 15s chain is ordinary. + const second = check.run(at(2933)) + assert.equal(second.state, 'up') + assert.equal(second.detail.stalledSec, 0) + // And it advances again. + assert.equal(check.run(at(2934)).detail.advancedBy, 1) +}) + +// ------------------------------------- the explorer's own path to the chain -- +// +// CF-13. The two probes either side of this one — `hearth-web` (the bundle is +// served) and `hearth.rpc.seed` (the node answers eth_*) — were both green for +// the entire life of the defect. What was broken was the wire between them. + +test("the explorer's /eth-rpc/ proxy is up when it reaches the JSON-RPC port", async () => { + const node = await jsonRpcStub({ chainId: 7412, height: 2933 }) + const web = await webStub(node.url) + try { + aim({ web: web.url }) + const r = await target('hearth-web.eth-rpc').run() + assert.equal(r.state, 'up', r.error) + assert.equal(r.detail.chainId, 7412) + assert.equal(r.detail.chainIdHex, '0x1cf4') + } finally { + await close(web) + await close(node) + } +}) + +// The finding, reproduced: HEARTH_RPC_UPSTREAM named the REST port, so the +// visitor's first eth_chainId came back as a 404 the client cannot read. +test("the explorer's proxy pointed at the REST port is down, and quotes the node saying so", async () => { + const rest = await restStub({ chainId: 7412, height: 2933 }) + const web = await webStub(rest.url) + try { + aim({ web: web.url }) + const r = await target('hearth-web.eth-rpc').run() + assert.equal(r.state, 'down') + assert.match(r.error, /expected 200, got 404/) + } finally { + await close(web) + await close(rest) + } +}) + +// A proxy that works but reaches somebody else's node. Amber, not red: the +// per-node probe owns the chain-id verdict, and two reds for one fault is how +// a status page teaches its reader to skim. +test("the explorer's proxy reaching a node on the wrong chain is degraded", async () => { + const node = await jsonRpcStub({ chainId: 7411, height: 12 }) + const web = await webStub(node.url) + try { + aim({ web: web.url }) + const r = await target('hearth-web.eth-rpc').run() + assert.equal(r.state, 'degraded') + assert.match(r.error, /chain 7411, not 7412/) + } finally { + await close(web) + await close(node) + } +}) diff --git a/infra/beacon/test/work.test.js b/infra/beacon/test/work.test.js new file mode 100644 index 0000000..7092800 --- /dev/null +++ b/infra/beacon/test/work.test.js @@ -0,0 +1,76 @@ +// The mining-work probe, against a stub that validates its `pub` the way the +// deployed node does. +// +// `hearth.work` is a critical chain probe, and it was permanently red: it sent +// an 88-char SPKI DER Ed25519 key, which is what the UTXO node's REST surface +// wanted (repos/hearth/node/src/rpc.js:134), while the testnet now runs the +// account-model node, whose template issuer rejects anything that is not a +// 65-byte uncompressed secp256k1 point (repos/hearth/node/src/chain/miner.js:136) +// with a 400. Nothing caught it because no test in this repo ever ran a probe +// against a node, and a check that is red every cycle stops being read. +// +// So the assertion is the key SHAPE, at the one place it is decided, against a +// stub that refuses the shape the real node refuses. + +import { test } from 'node:test' +import assert from 'node:assert/strict' +import http from 'node:http' + +process.env.BEACON_PROBE_TIMEOUT_MS = '3000' +const { env } = await import('../src/env.js') +const { TARGETS } = await import('../src/targets.js') + +const work = TARGETS.find((t) => t.name === 'hearth.work') +assert.ok(work, 'no hearth.work target') + +/** The node's own validation, and nothing else. */ +async function templateStub() { + const server = http.createServer((req, res) => { + const url = new URL(req.url, 'http://x') + if (!url.pathname.startsWith('/mining/template')) { + res.writeHead(404).end('{}') + return + } + const pub = (url.searchParams.get('pub') || '').toLowerCase() + const bytes = /^[0-9a-f]*$/.test(pub) && pub.length % 2 === 0 ? Buffer.from(pub, 'hex') : Buffer.alloc(0) + if (bytes.length !== 65 || bytes[0] !== 4) { + res.writeHead(400, { 'content-type': 'application/json' }) + res.end(JSON.stringify({ err: 'pub must be a 65-byte uncompressed secp256k1 key' })) + return + } + res.writeHead(200, { 'content-type': 'application/json' }) + res.end( + JSON.stringify({ + templateId: '5aa3055b764a7ba2e3c1e161f4c0cffe', + height: 3023, + coreHash: '06bb4f8b1fdd35d7db0bc8fd8284fbe7b70fed843f1e5e79ed51662d45f61feb', + target: '001b8c992537abf8f974a167e1ba179db471185325487b0d61aab6da718c70ac', + reward: '5997844660000000000', + txCount: 0, + scratchKiB: 512, + }), + ) + }) + await new Promise((r) => server.listen(0, '127.0.0.1', r)) + return { server, url: `http://127.0.0.1:${server.address().port}` } +} + +test('the work probe asks for a template with a key the account-model node accepts', async () => { + const node = await templateStub() + try { + env.urls.hearthSeed = node.url + const r = await work.run() + assert.equal(r.state, 'up', r.error) + assert.equal(r.detail.height, 3023) + } finally { + await new Promise((r) => node.server.close(r)) + } +}) + +test('the probe key is an uncompressed secp256k1 point, not a DER-wrapped one', () => { + // Read off the built URL rather than the module private, because the URL is + // what the node sees and the wrapping is exactly what was wrong before. + const pub = new URL(work.url(), 'http://x').searchParams.get('pub') + assert.match(pub, /^04[0-9a-f]{128}$/) + assert.equal(Buffer.from(pub, 'hex').length, 65) +}) diff --git a/infra/lantern/package.json b/infra/lantern/package.json index a0f390f..4b78df0 100644 --- a/infra/lantern/package.json +++ b/infra/lantern/package.json @@ -6,7 +6,8 @@ "description": "Log collection and error grouping for the CloudsForge stack.", "engines": { "node": ">=22" }, "scripts": { - "start": "node src/index.js" + "start": "node src/index.js", + "test": "node --test" }, "dependencies": { "jose": "^5.9.6", diff --git a/infra/lantern/public/app.js b/infra/lantern/public/app.js index 2645ab7..a75cc04 100644 --- a/infra/lantern/public/app.js +++ b/infra/lantern/public/app.js @@ -202,7 +202,13 @@ async function loadHealth() { const st = await fetch('/health').then((r) => r.json()).catch(() => null) if (st) { - $('#ingest-rate').textContent = `${st.ingested.toLocaleString()} lines${st.dropped ? ` · ${st.dropped} dropped` : ''}` + // `rate-limited` alongside `dropped` on purpose: a browser report that the + // quota refused is a log line you are missing, and the browser client + // swallows the failure, so this counter is the only trace it leaves. + $('#ingest-rate').textContent = + `${st.ingested.toLocaleString()} lines` + + `${st.dropped ? ` · ${st.dropped} dropped` : ''}` + + `${st.rateLimited ? ` · ${st.rateLimited} rate-limited` : ''}` } } diff --git a/infra/lantern/src/clientip.js b/infra/lantern/src/clientip.js new file mode 100644 index 0000000..d9719fb --- /dev/null +++ b/infra/lantern/src/clientip.js @@ -0,0 +1,151 @@ +// Which address a public request actually came from. +// +// `req.socket.remoteAddress` is the far end of the TCP connection, and for +// every request that arrives through the cloudflared tunnel that is the +// connector, not the browser. Keying the /ingest/client quota on it gave the +// entire internet one shared bucket of 120 requests a minute: 60 people hitting +// a broken release exhaust it in seconds, and — because the endpoint is routed +// before authorise() — anyone at all can hold it empty indefinitely at two +// requests a second, silently disabling browser error reporting for the whole +// estate for as long as they care to keep going. +// +// The forwarded headers fix that, but only from a peer we put there. A +// forwarded header is a request header like any other: from an untrusted peer +// it is whatever the caller typed, and a caller that chooses its own quota key +// has no quota at all. So the peer is checked against the trusted-proxy list +// first, and an unrecognised peer speaks only for itself. + +import { env } from './env.js' + +// Long enough for the longest IPv6 text form with an embedded IPv4 tail +// (45 characters). Anything longer is not an address, and is not going to +// become a Map key here. +const MAX_ADDR = 45 + +const V4 = /^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/ +const V6 = /^[0-9a-f:]+$/ + +/** + * Reduce an address to the one spelling we compare and key on, or '' if it is + * not an address at all. + * + * Node reports an IPv4 peer on a dual-stack listener as `::ffff:10.0.0.1`, so + * the same machine has two spellings and would otherwise get two buckets and + * miss a literal entry in the trusted list. + */ +export function normaliseIp(value) { + if (typeof value !== 'string') return '' + let s = value.trim().toLowerCase() + if (s.length > MAX_ADDR + 8) return '' // + room for the ::ffff: prefix + const bracketed = /^\[([^\]]*)\]/.exec(s) // [::1]:443, as a Forwarded header spells it + if (bracketed) s = bracketed[1] + const zone = s.indexOf('%') // fe80::1%eth0 — the scope is local to the host that wrote it + if (zone !== -1) s = s.slice(0, zone) + if (s.startsWith('::ffff:') && V4.test(s.slice(7))) s = s.slice(7) + if (V4.test(s)) return v4ToInt(s) === null ? '' : s + if (s.length <= MAX_ADDR && s.includes(':') && V6.test(s)) return s + return '' +} + +/** Dotted quad to a 32-bit integer, or null if any octet is out of range. */ +function v4ToInt(addr) { + const m = V4.exec(addr) + if (!m) return null + let n = 0 + for (let i = 1; i <= 4; i++) { + const octet = Number(m[i]) + if (octet > 255) return null + n = n * 256 + octet + } + return n +} + +function inV4Cidr(addr, base, bits) { + const a = v4ToInt(addr) + const b = v4ToInt(base) + if (a === null || b === null || !(bits >= 0 && bits <= 32)) return false + if (bits === 0) return true + const mask = (0xffffffff << (32 - bits)) >>> 0 + return ((a & mask) >>> 0) === ((b & mask) >>> 0) +} + +// The two keyword sets. `private` is the useful default: the tunnel connector +// reaches a published container port from the Docker bridge gateway (172.16/12 +// under compose) or from loopback, while a request that came straight off the +// public internet has a public source address and is therefore never trusted +// to rename itself. +const PRIVATE_V4 = ['10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16', '169.254.0.0/16'] + +function matchesKeyword(addr, keyword) { + const isV6 = addr.includes(':') + switch (keyword) { + case 'all': + return true + case 'loopback': + return isV6 ? addr === '::1' : inV4Cidr(addr, '127.0.0.0', 8) + case 'private': + // fc00::/7 (unique local) and fe80::/10 (link local) are the IPv6 + // equivalents of the RFC1918 ranges below, and are what a dual-stack + // Docker network or a host-local proxy actually presents. + if (isV6) return /^f[cd]/.test(addr) || /^fe[89ab]/.test(addr) || addr === '::1' + return PRIVATE_V4.some((cidr) => { + const [base, bits] = cidr.split('/') + return inV4Cidr(addr, base, Number(bits)) + }) + default: + return false + } +} + +/** + * @param {string} addr a normalised address (see normaliseIp) + * @param {string[]} rules env.trustedProxies + */ +export function isTrustedProxy(addr, rules = env.trustedProxies) { + if (!addr) return false + for (const rule of rules) { + if (matchesKeyword(addr, rule)) return true + if (rule.includes('/')) { + const [base, bits] = rule.split('/') + // IPv6 CIDR is deliberately not implemented rather than half implemented: + // a prefix match that quietly does the wrong thing on the address family + // it was written for is worse than one that never matches. Use the + // keywords or a literal address. + if (inV4Cidr(addr, base, Number(bits))) return true + continue + } + if (normaliseIp(rule) === addr) return true + } + return false +} + +/** The leftmost entry of a possibly-repeated, possibly-listed header. */ +function leftmost(value) { + const raw = Array.isArray(value) ? value[0] : value + if (typeof raw !== 'string') return '' + return raw.slice(0, 512).split(',')[0] +} + +/** + * The address to attribute this request to: quota key, and nothing else. + * + * `cf-connecting-ip` is preferred over `x-forwarded-for` because Cloudflare + * *sets* the former at the edge, overwriting whatever the client sent, while it + * *appends* to the latter — so behind the tunnel the leftmost x-forwarded-for + * entry is still a value the caller chose. Where there is no Cloudflare the + * header is absent and the ordinary forwarded chain is used instead. + * + * A forwarded value that is not an address is ignored rather than used as a + * key, so a peer that is trusted but sloppy cannot mint unbounded Map entries. + */ +export function clientIpFor(req, rules = env.trustedProxies) { + const peer = normaliseIp(req.socket?.remoteAddress ?? '') + if (!isTrustedProxy(peer, rules)) return peer || 'unknown' + const headers = req.headers ?? {} + return ( + normaliseIp(leftmost(headers['cf-connecting-ip'])) || + normaliseIp(leftmost(headers['x-forwarded-for'])) || + peer || + 'unknown' + ) +} diff --git a/infra/lantern/src/db.js b/infra/lantern/src/db.js index aaf313b..9becdf3 100644 --- a/infra/lantern/src/db.js +++ b/infra/lantern/src/db.js @@ -104,6 +104,33 @@ export async function migrate() { setDbHealthy(true) } +/** + * Is this write failure the *batch's* fault rather than the database's? + * + * The distinction decides whether store.flush() re-queues a failed batch or + * throws it away, so it is deliberately an allowlist of the classes that can + * never succeed on a retry, not the inverse test the register suggested + * ("anything that is not a connection failure"). That inverse test discards + * logs it should have kept: `53100 disk full` and `57P03 cannot connect now` + * both clear on their own, and `42P01 undefined_table` is the ordinary state of + * the world for the first few seconds after a boot with Postgres still starting + * — index.js is retrying migrate() while the writer is already ticking. + * + * class 22 — data exception. The poison this exists for: 22P02 for `1e30` or + * `1.5` in an INTEGER column, 22021 for a NUL byte in TEXT, 22P05 + * for one in JSONB, 22003 for an overflow, 22007 for a bad + * timestamp. sanitise.js should mean we never see these; this is + * the second lock, for the value nobody thought of. + * class 23 — integrity constraint violation. Same batch, same outcome, every + * time it is tried. + * 54000 — program limit exceeded: a row or index entry too large to store. + */ +export function isPoisonError(err) { + const code = err?.code + if (typeof code !== 'string') return false + return code.startsWith('22') || code.startsWith('23') || code === '54000' +} + /** Batch insert. One statement per flush, not per event. */ export async function insertEvents(rows) { if (!rows.length) return diff --git a/infra/lantern/src/env.js b/infra/lantern/src/env.js index e3b0302..ff78a91 100644 --- a/infra/lantern/src/env.js +++ b/infra/lantern/src/env.js @@ -48,6 +48,23 @@ export const env = { .map((s) => s.trim()) .filter(Boolean), + // Which socket peers are allowed to name the client on a request's behalf. + // + // Lantern is published on 0.0.0.0:4010 and reached through the cloudflared + // tunnel, so the peer of every public request is the connector — the Docker + // bridge gateway or loopback, one address for the whole internet. Forwarded + // headers are believed from those peers and from nowhere else: a header from + // an untrusted peer is whatever the caller typed, and a caller that can pick + // its own quota key does not have a quota. See clientip.js. + // + // Keywords `loopback` and `private`, literal addresses, and IPv4 CIDR are + // understood; `all` trusts every peer and should only be set when something + // else guarantees Lantern's port is unreachable except through a proxy. + trustedProxies: (process.env.LANTERN_TRUSTED_PROXIES ?? 'loopback,private') + .split(',') + .map((s) => s.trim().toLowerCase()) + .filter(Boolean), + // Backfill on first connect to a container, in lines. Keeps a restart of // Lantern from losing the context of an incident already in progress. tailOnConnect: num(process.env.LANTERN_TAIL_ON_CONNECT, 200), diff --git a/infra/lantern/src/parse.js b/infra/lantern/src/parse.js index 26f3803..a1da6d4 100644 --- a/infra/lantern/src/parse.js +++ b/infra/lantern/src/parse.js @@ -5,6 +5,8 @@ // configured writes a bare V8 stack to stderr, and that is exactly the line you // most need to keep. +import { clampLatency, clampStatusCode } from './sanitise.js' + const ANSI = /\[[0-9;]*m/g const PINO_LEVELS = { 10: 'trace', 20: 'debug', 30: 'info', 40: 'warn', 50: 'error', 60: 'fatal' } @@ -57,12 +59,16 @@ function fromPino(o) { // Fastify's own request/response lines: pull the useful bits into columns so // they are filterable without digging through JSON. const route = o.req?.url ?? fields.route ?? null - const statusCode = o.res?.statusCode ?? fields.statusCode ?? null + // `fields.statusCode` is any JSON key of that name in any container's log + // line, not just Fastify's — so it is arbitrary, and it lands in an INTEGER + // column. A `typeof x === 'number'` test admits NaN, 1.5 and 1e30, each of + // which fails the whole batch insert. See sanitise.js. + const statusCode = clampStatusCode(o.res?.statusCode ?? fields.statusCode ?? null) const latency = o.responseTime ?? fields.responseTime ?? null let level = PINO_LEVELS[o.level] ?? (typeof o.level === 'string' ? o.level : 'info') // A 5xx logged at `info` by a request-completion hook is still an incident. - if (level === 'info' && typeof statusCode === 'number' && statusCode >= 500) level = 'error' + if (level === 'info' && statusCode !== null && statusCode >= 500) level = 'error' return { source: 'pino', @@ -71,8 +77,8 @@ function fromPino(o) { msg: String(o.msg ?? err?.message ?? '(no message)'), reqId: o.reqId ? String(o.reqId) : (fields.requestId ?? null), route: route ? String(route) : null, - statusCode: typeof statusCode === 'number' ? statusCode : null, - latencyMs: typeof latency === 'number' ? latency : null, + statusCode, + latencyMs: clampLatency(latency), err: err ? { type: err.type ?? err.name ?? 'Error', @@ -85,7 +91,7 @@ function fromPino(o) { } function fromNginx(o) { - const status = Number(o.status) + const status = clampStatusCode(Number(o.status)) const level = status >= 500 ? 'error' : status >= 400 ? 'warn' : 'info' const latency = Number(o.request_time) return { @@ -95,7 +101,7 @@ function fromNginx(o) { msg: `${o.method ?? '-'} ${o.uri ?? o.request ?? '-'} ${o.status ?? '-'}`, reqId: o.request_id || null, route: o.uri ?? null, - statusCode: Number.isFinite(status) ? status : null, + statusCode: status, latencyMs: Number.isFinite(latency) ? Math.round(latency * 1000) : null, err: null, fields: o, diff --git a/infra/lantern/src/sanitise.js b/infra/lantern/src/sanitise.js new file mode 100644 index 0000000..e5e73bb --- /dev/null +++ b/infra/lantern/src/sanitise.js @@ -0,0 +1,103 @@ +// Remove the values Postgres will refuse, before they can reach a batch insert. +// +// insertEvents() writes a whole flush as one statement, so a single value a +// column cannot hold aborts the entire batch — every service's logs, not just +// the bad line. flush() then put that same batch back on the queue, so it +// failed again on the next tick, and the next, for as long as the record kept +// being re-queued. One unauthenticated POST to /ingest/client was enough to +// stop Lantern persisting anything at all. +// +// Sanitising happens in store.ingest(), which is the one chokepoint both +// producers pass through — the Docker collector via parse.js, and the browser +// sink at POST /ingest/client — rather than in either caller, because a third +// producer added later would otherwise reopen the hole. The callers clamp their +// own statusCode as well; that is a readability choice, not the defence. +// +// Three hazards, each confirmed against postgres:17 (the SQLSTATEs are real): +// +// status_code is INTEGER, and `Number.isFinite` is not the same question. +// 1e30 and 1.5 are both finite, and Postgres answers 22P02 +// (invalid_text_representation) to either, or 22003 to a bare overflow. +// +// TEXT cannot hold U+0000 at all — 22021, `invalid byte sequence for +// encoding "UTF8": 0x00` — and JSONB cannot either, as 22P05. Nothing in the +// stack strips it: the collector removes ANSI escapes and stops there, so any +// container that prints a NUL byte on stdout poisons the batch with no +// attacker involved. +// +// An invalid Date makes the driver throw a RangeError while serialising, and +// a RangeError carries no SQLSTATE — so it would read as a connection fault +// and be retried forever. It cannot happen from today's producers; it is +// handled here so that it cannot start happening from tomorrow's. + +/** Values nested deeper than this in `fields` are dropped rather than walked. */ +const MAX_DEPTH = 12 + +/** + * U+0000, built rather than typed. A raw NUL in this file would be invisible in + * every diff and every grep — which is exactly how the character got as far as + * a Postgres parameter in the first place. + */ +const NUL = String.fromCharCode(0) + +/** + * The `status_code` INTEGER column, or null. + * + * Deliberately narrower than "an integer that fits in int4": a status code is + * three digits, and anything else in that column is a bug in the producer, not + * data worth keeping. Non-numbers — a string `"200"` lifted out of some unknown + * JSON shape — are rejected too, which is what the old `typeof` test did. + */ +export function clampStatusCode(v) { + return Number.isInteger(v) && v >= 0 && v < 1000 ? v : null +} + +/** NUMERIC will take almost anything, but not NaN dressed up as a latency. */ +export function clampLatency(v) { + return typeof v === 'number' && Number.isFinite(v) ? v : null +} + +/** Strip U+0000. The `includes` test keeps the common case allocation-free. */ +export function stripNul(s) { + return s.includes(NUL) ? s.split(NUL).join('') : s +} + +/** Deep-clean a `fields`/JSONB value: strings scrubbed, everything else kept. */ +function cleanValue(v, depth) { + if (typeof v === 'string') return stripNul(v) + if (v === null || typeof v !== 'object') return v + if (depth >= MAX_DEPTH) return null + if (Array.isArray(v)) return v.map((x) => cleanValue(x, depth + 1)) + const out = {} + // A NUL in a *key* is as fatal as one in a value: JSONB has no way to + // represent either. + for (const [k, x] of Object.entries(v)) out[stripNul(k)] = cleanValue(x, depth + 1) + return out +} + +/** Columns that are plain TEXT and are copied from the record verbatim. */ +const TEXT_FIELDS = ['service', 'container', 'source', 'stream', 'level', 'msg', 'reqId', 'route', 'raw'] + +/** + * Make one record safe to insert. Mutates in place — this runs on every line + * from every container, and a copy per line is a cost with nothing to show for + * it. Returns the record so callers can chain. + */ +export function sanitiseRecord(rec) { + if (!(rec.ts instanceof Date) || Number.isNaN(rec.ts.getTime())) rec.ts = new Date() + + rec.statusCode = clampStatusCode(rec.statusCode) + rec.latencyMs = clampLatency(rec.latencyMs) + + for (const k of TEXT_FIELDS) if (typeof rec[k] === 'string') rec[k] = stripNul(rec[k]) + + if (rec.err && typeof rec.err === 'object') { + for (const k of ['type', 'message', 'stack']) { + if (typeof rec.err[k] === 'string') rec.err[k] = stripNul(rec.err[k]) + } + } + + if (rec.fields && typeof rec.fields === 'object') rec.fields = cleanValue(rec.fields, 0) + + return rec +} diff --git a/infra/lantern/src/server.js b/infra/lantern/src/server.js index 9b608c3..2052ad7 100644 --- a/infra/lantern/src/server.js +++ b/infra/lantern/src/server.js @@ -12,6 +12,8 @@ import { env } from './env.js' import { logger } from './log.js' import { authorise, authMode } from './auth.js' import { ingest, subscribe, recent, stats, dbHealthy } from './store.js' +import { clampStatusCode, stripNul } from './sanitise.js' +import { clientIpFor } from './clientip.js' import { queryEvents, queryIssues, queryStats, queryTimeline, resolveIssue } from './db.js' import { followedServices } from './collector.js' @@ -55,23 +57,39 @@ async function readBody(req, limit = 64 * 1024) { } // A misbehaving page must not be able to fill the store. Per-IP, per-minute. +// +// "Per-IP" means the *client's* IP — see clientip.js. Keyed on the socket peer +// it was one bucket for the whole internet, which any one caller could empty +// and hold empty, and every browser in the estate then shared the 429. const clientQuota = new Map() +let rateLimited = 0 function quotaExceeded(ip) { const now = Date.now() const slot = clientQuota.get(ip) if (!slot || now - slot.start > 60_000) { clientQuota.set(ip, { start: now, count: 1 }) - if (clientQuota.size > 10_000) clientQuota.clear() + // Drop the windows that have already expired before resorting to clearing + // the map: a wholesale clear forgives every caller currently over quota, + // and now that the key is the real client address there are enough distinct + // keys in normal traffic for a flood to trigger it deliberately. + if (clientQuota.size > 10_000) { + for (const [key, s] of clientQuota) if (now - s.start > 60_000) clientQuota.delete(key) + if (clientQuota.size > 10_000) clientQuota.clear() + } return false } slot.count++ - return slot.count > 120 + if (slot.count > 120) { + rateLimited++ + return true + } + return false } /** Browser-reported failures. The only push path into Lantern. */ async function handleClientIngest(req, res) { const cors = corsHeaders(req) - const ip = req.socket.remoteAddress ?? 'unknown' + const ip = clientIpFor(req) if (quotaExceeded(ip)) return send(res, 429, { error: 'rate limited' }, cors) let payload @@ -81,6 +99,14 @@ async function handleClientIngest(req, res) { return send(res, 400, { error: 'invalid body', detail: err.message }, cors) } + // This endpoint is served before authorise(), so everything below is attacker + // controlled. Every string is bounded and stripped of NUL bytes, and the + // status code is coerced to something the INTEGER column can actually hold — + // `Number.isFinite` used to be the whole check here, and 1e30 satisfies it. + // store.ingest() sanitises again for the collector's sake; this is not + // redundant so much as it is the same rule stated where the danger is. + const text = (v, max) => stripNul(String(v).slice(0, max)) + const items = Array.isArray(payload) ? payload : [payload] let accepted = 0 for (const item of items.slice(0, 50)) { @@ -88,24 +114,24 @@ async function handleClientIngest(req, res) { const level = ['error', 'warn', 'info', 'fatal'].includes(item.level) ? item.level : 'error' ingest({ ts: new Date(), - service: `web:${String(item.app ?? 'unknown').slice(0, 40)}`, + service: `web:${text(item.app ?? 'unknown', 40)}`, container: 'browser', source: 'client', stream: 'browser', level, - msg: String(item.message ?? '(no message)').slice(0, 2000), - reqId: item.requestId ? String(item.requestId).slice(0, 80) : null, - route: item.route ? String(item.route).slice(0, 500) : null, - statusCode: Number.isFinite(item.statusCode) ? item.statusCode : null, + msg: text(item.message ?? '(no message)', 2000), + reqId: item.requestId ? text(item.requestId, 80) : null, + route: item.route ? text(item.route, 500) : null, + statusCode: clampStatusCode(item.statusCode), latencyMs: null, err: { - type: String(item.type ?? 'BrowserError').slice(0, 120), - message: String(item.message ?? '').slice(0, 2000), - stack: item.stack ? String(item.stack).slice(0, 8000) : null, + type: text(item.type ?? 'BrowserError', 120), + message: text(item.message ?? '', 2000), + stack: item.stack ? text(item.stack, 8000) : null, }, fields: { - url: String(item.url ?? '').slice(0, 500), - userAgent: String(req.headers['user-agent'] ?? '').slice(0, 300), + url: text(item.url ?? '', 500), + userAgent: text(req.headers['user-agent'] ?? '', 300), release: item.release ?? null, userId: item.userId ?? null, context: item.context ?? null, @@ -157,7 +183,12 @@ async function serveStatic(res, pathname) { } } -export function startServer() { +/** + * @param {number} [port] overridden only by the tests, which need an ephemeral + * port: a suite that binds 4010 fails on any machine already running Lantern, + * which is every machine you would want to run it on. + */ +export function startServer(port = env.port) { const server = http.createServer(async (req, res) => { const url = new URL(req.url, `http://${req.headers.host ?? 'lantern'}`) const path = url.pathname @@ -175,6 +206,11 @@ export function startServer() { dropped: stats.dropped, buffered: stats.buffered, pending: stats.pending, + // Reports refused by the /ingest/client quota. Published because the + // browser client swallows a failed report by design, so this counter + // is the only place a refusal is recorded anywhere: without it, "no + // browser errors" and "every browser error rejected" read alike. + rateLimited, uptimeSec: Math.round((Date.now() - stats.startedAt) / 1000), }) } @@ -242,8 +278,8 @@ export function startServer() { } }) - server.listen(env.port, '0.0.0.0', () => { - logger.info('lantern listening', { port: env.port, authMode: authMode() }) + server.listen(port, '0.0.0.0', () => { + logger.info('lantern listening', { port: server.address()?.port ?? port, authMode: authMode() }) if (authMode() === 'open') { logger.warn('NO AUTH CONFIGURED — anyone who can reach this port can read every log line. Set NIMBUS_JWKS_URL or LANTERN_TOKEN.') } diff --git a/infra/lantern/src/store.js b/infra/lantern/src/store.js index 63eaeca..e8f6b3e 100644 --- a/infra/lantern/src/store.js +++ b/infra/lantern/src/store.js @@ -7,7 +7,8 @@ import { env } from './env.js' import { logger } from './log.js' import { fingerprint as fingerprintOf, issueTitle } from './fingerprint.js' -import { insertEvents, upsertIssues, setDbHealthy, dbHealthy } from './db.js' +import { sanitiseRecord } from './sanitise.js' +import { insertEvents, upsertIssues, setDbHealthy, dbHealthy, isPoisonError } from './db.js' const ring = [] const subscribers = new Set() @@ -48,6 +49,10 @@ export function recent({ limit = 200, service, level, q } = {}) { /** @param {object} rec a parsed record, already carrying `service`. */ export function ingest(rec) { + // Before anything else, and before the fingerprint: a NUL byte in `msg` also + // reaches the `issues` table through the group title, so scrubbing after + // grouping would leave the second insert poisoned. See sanitise.js. + sanitiseRecord(rec) rec.fingerprint = fingerprintOf(rec) stats.ingested++ @@ -66,8 +71,10 @@ export function ingest(rec) { // outage during a log storm would otherwise turn a logging problem into an // out-of-memory kill, taking down the one service still telling you anything. if (pending.length >= env.bufferSize * 4) { - pending.splice(0, pending.length - env.bufferSize * 2) - dropped++ + // Count the records lost, not the number of times we trimmed: /health + // publishes this as `dropped`, and "1" for ten thousand discarded lines is + // the wrong thing to read during an incident. + dropped += pending.splice(0, pending.length - env.bufferSize * 2).length } pending.push(rec) @@ -94,24 +101,76 @@ export function ingest(rec) { } } -async function flush() { +/** Exported for the tests; the writer below is the only production caller. */ +export async function flush() { if (!pending.length && !issueCounts.size) return const rows = pending const groups = [...issueCounts.values()] pending = [] issueCounts = new Map() + let wroteEvents = false try { await insertEvents(rows) + wroteEvents = true await upsertIssues(groups) stats.written += rows.length setDbHealthy(true) } catch (err) { const wasHealthy = dbHealthy() + + // A batch Postgres will never accept must not come back. Re-queueing one is + // what turned a single malformed record into a total outage of the write + // path: the same rows failed on every tick, forever, and every honest line + // behind them waited in a queue that only ever emptied by being trimmed. + // Data errors are the batch's fault and are dropped; everything else — + // connection loss, `undefined_table` while the schema is still being + // created at boot, disk full — is the database's fault and is retried, + // because those rows would have been written a second later. + if (isPoisonError(err)) { + if (wroteEvents) { + stats.written += rows.length + logger.error('dropped unwritable issue groups', { err, groups: groups.length }) + } else { + dropped += rows.length + logger.error('dropped an unwritable batch', { err, rows: rows.length }) + } + // Postgres answered, so it is up. Saying otherwise would send /api/events + // to the ring buffer over a fault that has nothing to do with the DB. + setDbHealthy(true) + failuresSinceOk = 0 + return + } + setDbHealthy(false) - // Put the batch back so a transient outage doesn't lose it, but only once — - // repeated failure must not compound into an unbounded retry queue. - if (pending.length < env.bufferSize) pending = rows.concat(pending) + if (wroteEvents) { + // The events are committed and only the issue upsert failed. Re-queueing + // the rows here would write every line in the batch a second time. + stats.written += rows.length + } else if (pending.length < env.bufferSize) { + // Put the batch back so a transient outage doesn't lose it, but only + // once — repeated failure must not compound into an unbounded queue. + pending = rows.concat(pending) + } else { + dropped += rows.length + } + // The grouped counts have to be put back too. They used to be dropped on + // the floor here — `issueCounts` was already replaced with an empty Map — + // so every failed flush permanently lost the grouping for its window, and + // the `issues` table has no way to reconstruct it from the events. + // Bounded like the row queue above: distinct fingerprints are few in + // practice, but "few in practice" is not a memory guarantee during a long + // outage with something generating a fresh error message per line. + for (const g of groups) { + const existing = issueCounts.get(g.fingerprint) + if (!existing) { + if (issueCounts.size < env.bufferSize) issueCounts.set(g.fingerprint, g) + } else { + existing.count += g.count + if (g.ts > existing.ts) existing.ts = g.ts + if (!existing.stack && g.stack) existing.stack = g.stack + } + } // Report the transition, then stay quiet. A retry loop that logs a stack // trace every second turns one outage into the loudest thing in the file // and buries the failure that caused it. diff --git a/infra/lantern/test/poison.test.js b/infra/lantern/test/poison.test.js new file mode 100644 index 0000000..c0c6694 --- /dev/null +++ b/infra/lantern/test/poison.test.js @@ -0,0 +1,231 @@ +// One malformed log record used to stop Lantern writing anything at all. +// +// `status_code` is an INTEGER and TEXT/JSONB cannot hold U+0000, so a record +// carrying either aborted the whole batch insert — and flush() put the same +// batch straight back on the queue, to fail again on the next tick and every +// tick after. These tests hold both halves of the fix: nothing unwritable gets +// into a batch, and a batch that is unwritable anyway does not come back. +// +// The database half runs only against a scratch database (see `scratchUrl`). +// Everything else is pure and always runs: +// +// node --test # the pure tests +// LANTERN_DATABASE_URL=postgres://…/lantern_test node --test + +import { test, describe, before, after } from 'node:test' +import assert from 'node:assert/strict' + +import { clampLatency, clampStatusCode, sanitiseRecord, stripNul } from '../src/sanitise.js' +import { parseLine } from '../src/parse.js' +import { isPoisonError } from '../src/db.js' + +const NUL = String.fromCharCode(0) + +describe('clampStatusCode', () => { + test('accepts the three-digit codes the column exists for', () => { + for (const n of [0, 1, 200, 404, 503, 999]) assert.equal(clampStatusCode(n), n) + }) + + test('rejects the finite numbers that fail the INSERT', () => { + // Both of these passed the old `Number.isFinite` / `typeof === number` + // tests. 1e30 answers 22P02 from Postgres; so does 1.5. + assert.equal(clampStatusCode(1e30), null) + assert.equal(clampStatusCode(1.5), null) + assert.equal(clampStatusCode(-1), null) + assert.equal(clampStatusCode(1000), null) + assert.equal(clampStatusCode(2 ** 31), null) + }) + + test('rejects the non-numbers', () => { + for (const v of [NaN, Infinity, -Infinity, '200', null, undefined, {}, []]) { + assert.equal(clampStatusCode(v), null) + } + }) +}) + +describe('clampLatency', () => { + test('keeps a real measurement and drops the rest', () => { + assert.equal(clampLatency(12.5), 12.5) + assert.equal(clampLatency(0), 0) + for (const v of [NaN, Infinity, '12', null]) assert.equal(clampLatency(v), null) + }) +}) + +describe('stripNul', () => { + test('removes U+0000 and leaves everything else alone', () => { + assert.equal(stripNul(`a${NUL}b`), 'ab') + assert.equal(stripNul(`${NUL}${NUL}`), '') + const clean = 'nothing to do here — é 中 ' + assert.equal(stripNul(clean), clean) + }) +}) + +describe('sanitiseRecord', () => { + test('scrubs every column a NUL byte can reach', () => { + const rec = sanitiseRecord({ + ts: new Date('2026-07-29T00:00:00Z'), + service: `web:x${NUL}`, + msg: `hi${NUL}there`, + route: `/a${NUL}/b`, + raw: `{"m":"${NUL}"}`, + statusCode: 200, + err: { type: `Err${NUL}`, message: `m${NUL}`, stack: `s${NUL}` }, + fields: { [`k${NUL}1`]: `v${NUL}1`, nested: { deep: [`a${NUL}`, 3, null] } }, + }) + + assert.equal(rec.service, 'web:x') + assert.equal(rec.msg, 'hithere') + assert.equal(rec.route, '/a/b') + assert.equal(rec.raw, '{"m":""}') + assert.deepEqual(rec.err, { type: 'Err', message: 'm', stack: 's' }) + assert.deepEqual(rec.fields, { k1: 'v1', nested: { deep: ['a', 3, null] } }) + }) + + test('clamps the numeric columns', () => { + const rec = sanitiseRecord({ ts: new Date(), statusCode: 1e30, latencyMs: NaN }) + assert.equal(rec.statusCode, null) + assert.equal(rec.latencyMs, null) + }) + + test('replaces a timestamp the driver would throw on', () => { + // An invalid Date makes postgres.js raise a RangeError, which carries no + // SQLSTATE — it would read as a connection fault and be retried forever. + const rec = sanitiseRecord({ ts: new Date('not a date'), msg: 'x' }) + assert.ok(rec.ts instanceof Date) + assert.ok(!Number.isNaN(rec.ts.getTime())) + }) +}) + +describe('parseLine', () => { + test('a container that logs a fractional statusCode does not poison the batch', () => { + // No attacker needed: `fields.statusCode` is any JSON key of that name in + // any container's output. + const rec = parseLine(JSON.stringify({ level: 30, time: Date.now(), msg: 'odd', statusCode: 1.5 })) + assert.equal(rec.statusCode, null) + }) + + test('a 1e30 statusCode does not poison the batch either', () => { + const rec = parseLine(JSON.stringify({ level: 30, time: Date.now(), msg: 'odd', statusCode: 1e30 })) + assert.equal(rec.statusCode, null) + }) + + test('a real 5xx is still lifted and still escalates the level', () => { + const rec = parseLine(JSON.stringify({ level: 30, time: Date.now(), msg: 'request completed', res: { statusCode: 503 } })) + assert.equal(rec.statusCode, 503) + assert.equal(rec.level, 'error') + }) + + test('nginx status codes are clamped the same way', () => { + const rec = parseLine(JSON.stringify({ log: 'nginx', remote_addr: '10.0.0.1', status: 1e30, uri: '/x' })) + assert.equal(rec.statusCode, null) + }) +}) + +describe('isPoisonError', () => { + test('data and constraint errors are the batch’s fault', () => { + for (const code of ['22P02', '22003', '22021', '22P05', '22007', '23505', '23502', '54000']) { + assert.equal(isPoisonError({ code }), true, code) + } + }) + + test('everything the database will recover from is retried', () => { + // 42P01 in particular: the ordinary state of the world for the first + // seconds after boot, while index.js is still retrying migrate(). + for (const code of ['08006', '08003', '57P03', '57P01', '53100', '53300', '40001', '42P01', '3D000']) { + assert.equal(isPoisonError({ code }), false, code) + } + assert.equal(isPoisonError(new Error('socket hang up')), false) + assert.equal(isPoisonError(undefined), false) + }) +}) + +// --------------------------------------------------------------- integration -- + +/** + * Refuse to run against the real database. These tests write rows and add a + * constraint to `events`; doing that to a live Lantern would be a worse + * outage than the one they are here to prevent. + */ +function scratchUrl() { + const url = process.env.LANTERN_DATABASE_URL + if (!url || !url.startsWith('postgres')) return null + try { + const name = new URL(url).pathname.slice(1) + return name && name !== 'lantern' ? url : null + } catch { + return null + } +} + +describe('against a real Postgres', { skip: scratchUrl() ? false : 'set LANTERN_DATABASE_URL to a scratch database' }, () => { + let db + let store + let reachable = false + + before(async () => { + db = await import('../src/db.js') + store = await import('../src/store.js') + try { + await db.migrate() + await db.sql`DELETE FROM events` + await db.sql`DELETE FROM issues` + reachable = true + } catch (err) { + reachable = false + console.error('postgres unreachable, integration tests will fail:', err.message) + } + }) + + after(async () => { + if (reachable) await db.sql`ALTER TABLE events DROP CONSTRAINT IF EXISTS cf18_poison`.catch(() => {}) + await db.sql.end({ timeout: 5 }).catch(() => {}) + }) + + test('the record from the ticket is accepted and persisted', async () => { + store.ingest({ + ts: new Date(), + service: 'web:x', + container: 'browser', + source: 'client', + level: 'error', + msg: `hi${NUL}`, + statusCode: 1e30, + err: { type: 'BrowserError', message: `hi${NUL}`, stack: null }, + fields: { context: { note: `a${NUL}b` } }, + raw: '{"app":"x"}', + }) + await store.flush() + + assert.equal(store.dbHealthy(), true) + const [row] = await db.sql`SELECT msg, status_code, fields FROM events WHERE service = 'web:x'` + assert.equal(row.msg, 'hi') + assert.equal(row.status_code, null) + assert.deepEqual(row.fields.context, { note: 'ab' }) + }) + + test('a batch Postgres will never accept is dropped, not re-queued', async () => { + // Manufacture a poison record the sanitiser cannot know about: a CHECK + // constraint standing in for whatever tomorrow's unwritable value is. + await db.sql`ALTER TABLE events ADD CONSTRAINT cf18_poison CHECK (msg <> 'poison')` + + const before = store.stats.written + store.ingest({ ts: new Date(), service: 'a', source: 'test', level: 'info', msg: 'poison', fields: {} }) + store.ingest({ ts: new Date(), service: 'a', source: 'test', level: 'info', msg: 'behind the poison', fields: {} }) + await store.flush() + + // The batch is gone and the writer is not wedged: Postgres answered, so it + // is still healthy, and nothing is left on the queue to fail again. + assert.equal(store.stats.written, before) + assert.equal(store.stats.pending, 0) + assert.equal(store.dbHealthy(), true) + + await db.sql`ALTER TABLE events DROP CONSTRAINT cf18_poison` + + // The next line written after the poison is the one that matters: before + // the fix it queued behind a batch that could never drain. + store.ingest({ ts: new Date(), service: 'a', source: 'test', level: 'info', msg: 'after the poison', fields: {} }) + await store.flush() + const rows = await db.sql`SELECT msg FROM events WHERE service = 'a'` + assert.deepEqual(rows.map((r) => r.msg), ['after the poison']) + }) +}) diff --git a/infra/lantern/test/quota.test.js b/infra/lantern/test/quota.test.js new file mode 100644 index 0000000..00bef25 --- /dev/null +++ b/infra/lantern/test/quota.test.js @@ -0,0 +1,167 @@ +// The /ingest/client quota is per client, not per socket. +// +// Keyed on `req.socket.remoteAddress` it was one bucket of 120 requests a +// minute for the entire internet, because through the cloudflared tunnel every +// public request has the same socket peer. Two things fell out of that: 60 +// browsers reporting a bad release emptied the bucket in seconds and lost +// exactly the reports the endpoint exists to collect, and — /ingest/client +// being routed before authorise() — any unauthenticated caller could hold it +// empty at two requests a second and turn browser error reporting off for the +// whole estate. Both are reproduced below. +// +// node --test +// +// The end-to-end half binds an ephemeral port on loopback and speaks real HTTP +// to it; nothing here touches Postgres (the writer is never started, so ingest +// only fills the in-memory queue). + +import { test, describe, before, after } from 'node:test' +import assert from 'node:assert/strict' + +import { clientIpFor, isTrustedProxy, normaliseIp } from '../src/clientip.js' + +describe('normaliseIp', () => { + test('collapses the spellings of one address', () => { + assert.equal(normaliseIp('::ffff:10.0.0.1'), '10.0.0.1') + assert.equal(normaliseIp('[::1]'), '::1') + assert.equal(normaliseIp('FE80::1%eth0'), 'fe80::1') + assert.equal(normaliseIp(' 203.0.113.7 '), '203.0.113.7') + }) + + test('rejects everything that is not an address', () => { + // The forwarded value becomes a Map key, so anything that gets through here + // is something a caller can mint buckets with. + for (const v of ['', 'not-an-ip', '203.0.113.999', '1.2.3', 'x'.repeat(200), ' + + From 99231fca816f55d9092745914c0163454d86938f Mon Sep 17 00:00:00 2001 From: Savvanis Spyros Date: Thu, 30 Jul 2026 00:38:25 +0300 Subject: [PATCH 6/6] fix(ci): address the nodes as 127.0.0.1, the way compose publishes them The chain step timed out against miner1 and reported it as a JSONDecodeError, because curl -sf prints nothing on a timeout and the python meant to read a chain id got an empty string. Every hearthd port here is published as 127.0.0.1::, so no IPv6 listener exists, and localhost resolves to ::1 first -- every call pays a failed IPv6 connect before falling back. Measured locally: 0.003s to 127.0.0.1:8545 against 3.3s to localhost:8547. A slower runner crosses the -m 10 budget and the step fails on a healthy chain. Co-Authored-By: Claude --- .github/workflows/ci.yml | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3106c4e..7f76b54 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -328,8 +328,17 @@ jobs: - name: The chain mines and advances, on both listeners and on chain 7412 run: | set -e + # 127.0.0.1, never `localhost`. Every hearthd port in this file is + # published as `127.0.0.1::`, so there is no IPv6 + # listener — and `localhost` resolves to ::1 first, which means each + # call pays a failed IPv6 connect before falling back. Measured on a + # developer machine that is 0.003s against 127.0.0.1:8545 and 3.3s + # against localhost:8547; on a slower runner it exceeds the -m 10 + # budget, curl -sf then prints nothing, and the failure surfaces as an + # unreadable JSONDecodeError from the python that was meant to parse a + # chain id. rpc() { - curl -sf -m 10 -X POST "http://localhost:$1/" \ + curl -sf -m 10 -X POST "http://127.0.0.1:$1/" \ -H 'content-type: application/json' \ -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"$2\",\"params\":[]}" \ | python3 -c 'import json,sys; r=json.load(sys.stdin); e=r.get("error"); sys.exit("JSON-RPC error: %s" % e) if e else print(r["result"])' @@ -343,10 +352,10 @@ jobs: [ "$net" = "7412" ] || { echo "::error::$port: net_version is $net, expected the decimal string 7412"; exit 1; } echo "ok: $port answers JSON-RPC on chain 7412, hex from eth_chainId and decimal from net_version" done - h1=$(curl -sf -m 10 http://localhost:8645/info | python3 -c 'import json,sys; print(json.load(sys.stdin)["height"])') + h1=$(curl -sf -m 10 http://127.0.0.1:8645/info | python3 -c 'import json,sys; print(json.load(sys.stdin)["height"])') r1=$(( $(rpc 8545 eth_blockNumber) )) sleep 60 - h2=$(curl -sf -m 10 http://localhost:8645/info | python3 -c 'import json,sys; print(json.load(sys.stdin)["height"])') + h2=$(curl -sf -m 10 http://127.0.0.1:8645/info | python3 -c 'import json,sys; print(json.load(sys.stdin)["height"])') r2=$(( $(rpc 8545 eth_blockNumber) )) echo "REST height $h1 -> $h2 · JSON-RPC height $r1 -> $r2" [ "$h2" -gt "$h1" ] || { echo "::error::the chain did not advance"; exit 1; }