diff --git a/README.md b/README.md index 90d91c2..561e7fe 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,27 @@ # Hogflare -Hogflare +Hogflare -Hogflare is a Cloudflare Workers ingestion layer for PostHog SDKs. It supports PostHog-style ingestion, stateful persons/groups, and SDK feature flags, then streams events and person snapshots into Cloudflare Pipelines so data lands in R2 as Iceberg/Parquet. +Hogflare is a Cloudflare Workers ingestion layer for PostHog SDKs. It supports PostHog-style ingestion, stateful persons and groups, SDK feature flags, and a read-only session replay explorer, then streams events and person snapshots into Cloudflare Pipelines so data lands in R2 as Iceberg/Parquet. -#### What works today +![Hogflare replay explorer](docs/assets/replay-explorer.jpg) -- Ingestion endpoints: `/capture`, `/identify`, `/alias`, `/batch`, `/e`, `/engage`, `/groups`, `/s` -- Persons and groups: `$set`, `$set_once`, `$unset`, aliasing, and group properties -- SDK config and feature flags: `/array/:token/config`, `/flags`, and `/decide` are evaluated in the Worker -- Request enrichment: Cloudflare IP/geo fields added when missing -- Queryable people: append-only person snapshots can be written to a separate Iceberg table +## What Works Today + +- PostHog-compatible ingestion endpoints: `/capture`, `/identify`, `/alias`, `/batch`, `/e`, `/engage`, `/groups`, `/s` +- Persons and groups: `$set`, `$set_once`, `$unset`, aliasing, group properties, and group slots +- SDK config and feature flags: `/array/:token/config`, `/flags`, and `/decide` +- Session replay ingestion and read-only replay explorer backed by R2 SQL over Iceberg rows +- Request enrichment with Cloudflare IP and geo fields +- Queryable event and person snapshots in R2 Data Catalog-backed Iceberg tables + +## Docs + +- [Deployment](docs/deployment.md): Cloudflare Pipeline setup, Wrangler config, secrets, deployment, verification, local fake pipeline, and cleanup. +- [Session Replay](docs/session-replay.md): replay ingestion, explorer UI, API routes, filters, and local demo commands. +- [PostHog Compatibility](docs/posthog-compatibility.md): SDK setup, endpoint behavior, persons, groups, feature flags, signing, and enrichment. +- [Import Existing PostHog Data](docs/import-posthog.md): host-side backfill importer for existing PostHog projects. +- [Data Model](docs/data-model.md): event and person row shapes plus DuckDB/R2 SQL query examples. ## Architecture @@ -39,660 +50,40 @@ flowchart TB Worker -->|"person snapshots"| PersonsPipeline["Persons Pipeline"] EventsPipeline --> EventsR2["R2 Data Catalog
events table"] PersonsPipeline --> PersonsR2["R2 Data Catalog
persons table"] -``` - -## Why? - -PostHog is a nice-to-use web & product analytics platform. However, self-hosting PostHog is prohibitively complex so most users seem to rely on the cloud offering. This is an alternative for cost-conscious data folks & businesses interested in a low maintenance way to ingest web & product analytics directly into a managed data lake. - -A [hobby deployment of PostHog](https://github.com/PostHog/posthog/blob/master/docker-compose.hobby.yml) includes: postgres, redis, redis7, clickhouse, zookeeper, kafka, worker, web, plugins, proxy, objectstorage, seaweedfs, asyncmigrationscheck, temporal, elasticsearch, temporal-admin-tools, temporal-ui, temporal-django-worker, cyclotron-janitor, capture, replay-capture, property-defs-rs, livestream, feature-flags, cymbal - -Admittedly, PostHog does a *lot* more than this package, but some folks really just want the basics! - -## Quick start (Cloudflare) - -1. Create R2 Data Catalog-backed Pipelines resources. -2. Copy `wrangler.toml.example` to `wrangler.toml` and set the stream endpoints. -3. Set Wrangler secrets. -4. Build and deploy the Worker. -5. Send a capture/identify verification flow and query the Iceberg tables. - -The examples below use stable table names for a fresh deployment: `default.hogflare_events` and `default.hogflare_persons`. If you use versioned names during migration, substitute those names consistently in the sink commands and queries. - -### Create Pipelines Resources - -Set these values before creating sinks: - -```bash -export R2_BUCKET="" -export R2_CATALOG_TOKEN="" -``` - -`R2_CATALOG_TOKEN` is the token used by R2 Data Catalog/R2 SQL clients such as DuckDB or PyIceberg. The bucket must have R2 Data Catalog enabled before creating `r2-data-catalog` sinks. - -Create the events stream, sink, and pipeline: - -```bash -bunx wrangler pipelines streams create hogflare_events_stream \ - --schema-file scripts/events-pipeline-schema.json \ - --http-enabled true \ - --http-auth true - -bunx wrangler pipelines sinks create hogflare_events_sink \ - --type r2-data-catalog \ - --bucket "$R2_BUCKET" \ - --namespace default \ - --table hogflare_events \ - --catalog-token "$R2_CATALOG_TOKEN" \ - --roll-interval 60 - -bunx wrangler pipelines create hogflare_events_pipeline \ - --sql "INSERT INTO hogflare_events_sink SELECT * FROM hogflare_events_stream;" -``` - -Create the persons stream, sink, and pipeline if you want queryable people in Iceberg: - -```bash -bunx wrangler pipelines streams create hogflare_persons_stream \ - --schema-file scripts/persons-pipeline-schema.json \ - --http-enabled true \ - --http-auth true - -bunx wrangler pipelines sinks create hogflare_persons_sink \ - --type r2-data-catalog \ - --bucket "$R2_BUCKET" \ - --namespace default \ - --table hogflare_persons \ - --catalog-token "$R2_CATALOG_TOKEN" \ - --roll-interval 60 - -bunx wrangler pipelines create hogflare_persons_pipeline \ - --sql "INSERT INTO hogflare_persons_sink SELECT * FROM hogflare_persons_stream;" -``` - -Each stream creation command prints an HTTP endpoint like `https://.ingest.cloudflare.com`. Use those endpoints in `wrangler.toml`. - -### Wrangler config - -Copy the example and fill in the stream endpoints: - -```bash -cp wrangler.toml.example wrangler.toml -``` - -```toml -name = "hogflare" -main = "build/index.js" # generated entrypoint from worker-build for the Rust worker -compatibility_date = "2025-01-09" - -[vars] -CLOUDFLARE_PIPELINE_ENDPOINT = "https://.ingest.cloudflare.com" -CLOUDFLARE_PERSONS_PIPELINE_ENDPOINT = "https://.ingest.cloudflare.com" -CLOUDFLARE_PIPELINE_TIMEOUT_SECS = "10" - -# Optional -# POSTHOG_TEAM_ID = "1" -# POSTHOG_GROUP_TYPE_0 = "company" -# POSTHOG_GROUP_TYPE_1 = "team" -# POSTHOG_GROUP_TYPE_2 = "project" -# POSTHOG_GROUP_TYPE_3 = "org" -# POSTHOG_GROUP_TYPE_4 = "workspace" -# POSTHOG_SESSION_RECORDING_ENDPOINT = "/s/" - -[[durable_objects.bindings]] -name = "PERSONS" -class_name = "PersonDurableObject" - -[[durable_objects.bindings]] -name = "PERSON_ID_COUNTER" -class_name = "PersonIdCounterDurableObject" - -[[durable_objects.bindings]] -name = "GROUPS" -class_name = "GroupDurableObject" - -[[migrations]] -tag = "v1" -new_sqlite_classes = ["PersonDurableObject"] - -[[migrations]] -tag = "v2" -new_sqlite_classes = ["PersonIdCounterDurableObject", "GroupDurableObject"] -``` - -### Configuration Reference - -| Setting | Required | Notes | -| --- | --- | --- | -| `CLOUDFLARE_PIPELINE_ENDPOINT` | Yes | Events stream HTTP endpoint from `wrangler pipelines streams create`. | -| `CLOUDFLARE_PIPELINE_AUTH_TOKEN` | Yes, for authenticated streams | Bearer token used for events stream HTTP ingest. | -| `CLOUDFLARE_PERSONS_PIPELINE_ENDPOINT` | No | Persons stream endpoint. Set this to write person snapshots to Iceberg. | -| `CLOUDFLARE_PERSONS_PIPELINE_AUTH_TOKEN` | No | Falls back to `CLOUDFLARE_PIPELINE_AUTH_TOKEN` when omitted. | -| `CLOUDFLARE_PIPELINE_TIMEOUT_SECS` | No | Defaults to 10 seconds. | -| `POSTHOG_API_KEY` | No | Default project token returned by `/decide` when request/header token is absent. | -| `POSTHOG_TEAM_ID` | No | Optional team id attached to event and person rows. | -| `POSTHOG_GROUP_TYPE_0..4` | No | Maps PostHog group types to `group0..group4`; set `POSTHOG_GROUP_TYPE_0=company` to populate `group0` for company groups. | -| `POSTHOG_SESSION_RECORDING_ENDPOINT` | No | Returned in `/decide` session recording config. | -| `POSTHOG_SIGNING_SECRET` | No | Enables HMAC request signature checks. | -| `PERSON_DEBUG_TOKEN` | No | Enables `/__debug/person/:id` for deployment verification. | -| `HOGFLARE_FEATURE_FLAGS` | No | JSON flag config used by `/decide` and `/flags`. | - -### Secrets - -Use a Cloudflare API token that can write to Pipelines for `CLOUDFLARE_PIPELINE_AUTH_TOKEN`. The same token can usually be reused for the persons stream. - -```bash -bunx wrangler secret put CLOUDFLARE_PIPELINE_AUTH_TOKEN -# Optional. If omitted, the persons pipeline uses CLOUDFLARE_PIPELINE_AUTH_TOKEN. -bunx wrangler secret put CLOUDFLARE_PERSONS_PIPELINE_AUTH_TOKEN - -# Optional. -bunx wrangler secret put POSTHOG_SIGNING_SECRET -bunx wrangler secret put PERSON_DEBUG_TOKEN -bunx wrangler secret put HOGFLARE_FEATURE_FLAGS -``` - -### Deploy - -```bash -worker-build --release -bunx wrangler deploy -``` - -## Verify Deployment - -```bash -export HOGFLARE_URL="https://.workers.dev" -export HOGFLARE_API_KEY="phc_verify_$(date -u +%Y%m%d%H%M%S)" -export HOGFLARE_ANON_ID="${HOGFLARE_API_KEY}_anon" -export HOGFLARE_USER_ID="${HOGFLARE_API_KEY}_user" -``` - -Send an anonymous capture: - -```bash -curl -X POST "$HOGFLARE_URL/capture" \ - -H "Content-Type: application/json" \ - -d "{ - \"api_key\": \"$HOGFLARE_API_KEY\", - \"event\": \"verify-anon-capture\", - \"distinct_id\": \"$HOGFLARE_ANON_ID\", - \"properties\": { - \"\$set\": { \"initial_referrer\": \"docs\" }, - \"\$set_once\": { \"first_seen_source\": \"readme\" } - } - }" -``` - -Identify the user and link the anonymous ID: - -```bash -curl -X POST "$HOGFLARE_URL/identify" \ - -H "Content-Type: application/json" \ - -d "{ - \"api_key\": \"$HOGFLARE_API_KEY\", - \"distinct_id\": \"$HOGFLARE_USER_ID\", - \"properties\": { - \"\$anon_distinct_id\": \"$HOGFLARE_ANON_ID\", - \"\$set\": { \"email\": \"verify@example.com\", \"plan\": \"pro\" }, - \"\$set_once\": { \"signup_source\": \"readme\" } - } - }" -``` - -Send a post-identify capture: - -```bash -curl -X POST "$HOGFLARE_URL/capture" \ - -H "Content-Type: application/json" \ - -d "{ - \"api_key\": \"$HOGFLARE_API_KEY\", - \"event\": \"verify-identified-capture\", - \"distinct_id\": \"$HOGFLARE_USER_ID\", - \"properties\": { \"button\": \"verify\" } - }" -``` - -Wait for the sink roll interval, then query R2 SQL: - -```bash -export R2_WAREHOUSE="_" -export WRANGLER_R2_SQL_AUTH_TOKEN="$R2_CATALOG_TOKEN" - -bunx wrangler r2 sql query "$R2_WAREHOUSE" \ - "select event, distinct_id, person_id, person_properties - from default.hogflare_events - where api_key = '$HOGFLARE_API_KEY' - order by created_at asc" - -bunx wrangler r2 sql query "$R2_WAREHOUSE" \ - "select operation, canonical_distinct_id, person_id, distinct_ids, merged_properties - from default.hogflare_persons - where api_key = '$HOGFLARE_API_KEY' - order by updated_at asc" -``` - -Expected result: the three event rows share one `person_id`, and the persons table has `capture`, `identify`, `capture` snapshots. After identify, `distinct_ids` should include both the anonymous and identified IDs. - -## HMAC signing (optional) - -If `POSTHOG_SIGNING_SECRET` is set, requests must include a valid signature. - -```bash -payload='[ - { - "api_key": "phc_example", - "event": "purchase", - "distinct_id": "user_12345", - "properties": { "amount": 29.99 } - } -]' - -signature=$(printf '%s' "$payload" | openssl dgst -sha256 -hmac "$POSTHOG_SIGNING_SECRET" | awk '{print $2}') - -curl -X POST https://.workers.dev/capture \ - -H "Content-Type: application/json" \ - -H "X-POSTHOG-SIGNATURE: sha256=$signature" \ - -d "$payload" -``` - -Note: `X-HUB-SIGNATURE` with `sha1=` is also accepted for GitHub-style webhook compatibility. - -## PostHog SDK config - -### Browser (posthog-js) - -```js -import posthog from "posthog-js"; - -posthog.init("", { - api_host: "https://.workers.dev", - capture_pageview: true, -}); -``` - -### Server (posthog-node) - -```js -import { PostHog } from "posthog-node"; - -const client = new PostHog("", { - host: "https://.workers.dev", -}); - -client.capture({ - distinctId: "user_123", - event: "purchase", - properties: { amount: 29.99 }, -}); - -await client.shutdown(); -``` - -### Other SDKs - -Set the SDK host/base URL to your Worker (`https://.workers.dev`) and use your project API key. Most SDKs use either `api_host` (browser/mobile) or `host` (server). - -## Local development (fake pipeline) - -The repo includes a lightweight fake pipeline (FastAPI + DuckDB) used by tests. - -```bash -docker compose up --build -d fake-pipeline -``` - -```bash -# .env.local (not committed) -CLOUDFLARE_PIPELINE_ENDPOINT=http://127.0.0.1:8088/ -CLOUDFLARE_PERSONS_PIPELINE_ENDPOINT=http://127.0.0.1:8088/ -CLOUDFLARE_PIPELINE_TIMEOUT_SECS=5 -``` - -```bash -cargo run -``` - -## Query data (DuckDB) - -```sql -INSTALL httpfs; -INSTALL iceberg; -LOAD httpfs; -LOAD iceberg; - -CREATE SECRET r2_catalog_secret ( - TYPE ICEBERG, - TOKEN '' -); -ATTACH '_' AS iceberg_catalog ( - TYPE ICEBERG, - ENDPOINT 'https://catalog.cloudflarestorage.com//' -); - -SELECT count(*) FROM iceberg_catalog.default.hogflare_events; -SELECT count(*) FROM iceberg_catalog.default.hogflare_persons; -SELECT * FROM iceberg_catalog.default.hogflare_persons LIMIT 5; + ReplayUI["Replay Explorer"] -->|"R2 SQL"| EventsR2 ``` -If you used versioned table names during a migration, substitute those names here. - -## Cleanup +## Why -Delete Pipelines resources in dependency order: pipelines first, then streams and sinks. +PostHog is a nice-to-use web and product analytics platform. Self-hosting PostHog is prohibitively complex, so most users rely on the cloud offering. Hogflare is an alternative for cost-conscious data teams and businesses that want a low-maintenance way to ingest web and product analytics directly into a managed data lake. -```bash -bunx wrangler pipelines list -bunx wrangler pipelines delete --force +A [hobby deployment of PostHog](https://github.com/PostHog/posthog/blob/master/docker-compose.hobby.yml) includes postgres, redis, redis7, clickhouse, zookeeper, kafka, worker, web, plugins, proxy, objectstorage, seaweedfs, asyncmigrationscheck, temporal, elasticsearch, temporal-admin-tools, temporal-ui, temporal-django-worker, cyclotron-janitor, capture, replay-capture, property-defs-rs, livestream, feature-flags, and cymbal. -bunx wrangler pipelines streams list -bunx wrangler pipelines streams delete --force +PostHog does much more than this package, but some teams only need the warehouse-first basics. -bunx wrangler pipelines sinks list -bunx wrangler pipelines sinks delete --force -``` +## Replay Demo -`wrangler r2 sql query` is read-only. To drop an Iceberg table from R2 Data Catalog, use the Iceberg catalog API. One local option is PyIceberg: +The branch includes a local replay fixture that makes the explorer usable without Cloudflare credentials: ```bash -R2_CATALOG_TOKEN="" uv run --with pyiceberg python - <<'PY' -import os -from pyiceberg.catalog.rest import RestCatalog - -catalog = RestCatalog( - name="hogflare", - warehouse="_", - uri="https://catalog.cloudflarestorage.com//", - token=os.environ["R2_CATALOG_TOKEN"], -) - -catalog.drop_table(("default", ""), purge_requested=True) -PY +REPLAY_DEMO_PORT=4666 bun scripts/replay_demo_stub.mjs ``` -## Import existing PostHog data - -Hogflare includes a host-side importer for backfilling an existing PostHog project into the same Cloudflare Pipeline sink used by the Worker. It reads PostHog's private API with a personal API key, then writes normalized rows to the pipeline: - -- persons as `$identify` rows -- groups as `$groupidentify` rows -- historical events from HogQL with original `timestamp`, `created_at`, and PostHog event `uuid` when available - -The importer writes historical rows directly to the pipeline. It does not mutate Worker Durable Object state. - -Required inputs: - ```bash -export POSTHOG_PROJECT_ID="" -export POSTHOG_PERSONAL_API_KEY="phx_..." -export CLOUDFLARE_PIPELINE_ENDPOINT="https://.ingest.cloudflare.com" -export CLOUDFLARE_PIPELINE_AUTH_TOKEN="" # if your stream requires it +APP_ADDR=127.0.0.1:4567 \ +CLOUDFLARE_PIPELINE_ENDPOINT=http://127.0.0.1:4666/ \ +HOGFLARE_REPLAY_ACCOUNT_ID=demo-account \ +HOGFLARE_REPLAY_BUCKET=demo-bucket \ +HOGFLARE_REPLAY_R2_SQL_TOKEN=demo-token \ +HOGFLARE_REPLAY_R2_SQL_ENDPOINT=http://127.0.0.1:4666/ \ +HOGFLARE_REPLAY_EVENTS_TABLE=default.hogflare_events \ +HOGFLARE_REPLAY_QUERY_LIMIT=500 \ +cargo run --bin hogflare ``` -Optional inputs: +Open: -```bash -export POSTHOG_HOST="https://us.posthog.com" # or https://eu.posthog.com / self-hosted URL -export POSTHOG_ENVIRONMENT_ID="" # recommended for current PostHog persons/groups APIs -export HOGFLARE_API_KEY="phc_..." -export POSTHOG_TEAM_ID="1" -export POSTHOG_GROUP_TYPE_0="company" -export IMPORT_FROM="2025-01-01" -export IMPORT_TO="2025-02-01" -export IMPORT_BATCH_SIZE="500" -export IMPORT_PERSONS_OFFSET="0" # resume guardrails -export IMPORT_EVENTS_OFFSET="0" -export IMPORT_EVENTS_AFTER_TIMESTAMP="2024-09-21T03:24:11Z" -export IMPORT_EVENTS_AFTER_UUID="0192129b-c354-77b4-b496-9be7ec571fb4" -export IMPORT_EVENT_UUIDS_FILE="/tmp/missing-event-uuids.txt" -export IMPORT_EVENT_WINDOW_DAYS="7" -export IMPORT_EVENT_WINDOW_HOURS="6" # use days or hours, not both -export IMPORT_MAX_PERSONS="1000" # optional guardrails for smoke tests -export IMPORT_MAX_GROUPS="1000" -export IMPORT_MAX_EVENTS="1000" -export IMPORT_STATE_FILE=".hogflare-import-state.jsonl" -export IMPORT_TARGET_ACCOUNT_ID="" -export IMPORT_TARGET_BUCKET="" -export IMPORT_TARGET_TABLE="default.hogflare_events_v3" -export WRANGLER_R2_SQL_AUTH_TOKEN="" -export IMPORT_CLOUDFLARE_API_TOKEN="" # optional auto flush discovery -export IMPORT_PIPELINE_FLUSH_SECS="300" # fallback if Pipelines read is unavailable -``` - -Production imports require R2 SQL target checks by default. The importer uses stable import -keys, queries the target before each batch, and skips rows that are already present. -Cloudflare Pipeline/R2 is append-only and does not enforce uniqueness by itself. Passing -`--no-target-check` or `IMPORT_TARGET_CHECKS=false` opts out and should only be used for -local tests. - -Retry behavior is intentionally conservative. Import sends are not blindly retried after a -transport or response error because the pipeline may have accepted the batch even if the -client did not receive the response. The importer aligns its wait window to the Cloudflare -Pipeline sink rolling policy when `IMPORT_CLOUDFLARE_API_TOKEN` can read Pipelines. Without -that API access, it uses `IMPORT_PIPELINE_FLUSH_SECS`, defaulting conservatively to 300 -seconds. The wait is `max(60s, 2 * flush + 30s)`, unless `IMPORT_TARGET_WAIT_SECS` is set -explicitly. - -The local state file makes normal same-machine resumes cheap, but it is not a substitute for -target checks if the state file is lost, multiple importers run concurrently, or a send has an -unknown commit state. - -Run a dry run first: - -```bash -cargo run --bin import_posthog -- --dry-run -``` - -Run the import: - -```bash -cargo run --bin import_posthog +```text +http://127.0.0.1:4567/replay?api_key=phc_demo&distinct_id=replay-user&limit=100&session_id=demo-session-1&at_ms=1500 ``` - -You can also pass flags instead of env vars: - -```bash -cargo run --bin import_posthog -- \ - --posthog-host https://us.posthog.com \ - --project-id 12345 \ - --environment-id 67890 \ - --personal-api-key "$POSTHOG_PERSONAL_API_KEY" \ - --pipeline-endpoint "$CLOUDFLARE_PIPELINE_ENDPOINT" \ - --pipeline-auth-token "$CLOUDFLARE_PIPELINE_AUTH_TOKEN" \ - --hogflare-api-key phc_example \ - --from 2025-01-01 \ - --to 2025-02-01 \ - --persons-offset 0 \ - --events-offset 0 \ - --events-after-timestamp 2024-09-21T03:24:11Z \ - --events-after-uuid 0192129b-c354-77b4-b496-9be7ec571fb4 \ - --event-uuids-file /tmp/missing-event-uuids.txt \ - --event-window-hours 6 \ - --max-persons 1000 \ - --max-groups 1000 \ - --max-events 1000 \ - --import-state-file .hogflare-import-state.jsonl \ - --target-account-id "$CLOUDFLARE_ACCOUNT_ID" \ - --target-bucket hogflare \ - --target-table default.hogflare_events_v3 \ - --target-auth-token "$WRANGLER_R2_SQL_AUTH_TOKEN" \ - --cloudflare-api-token "$CLOUDFLARE_API_TOKEN" -``` - -Use `--skip-persons`, `--skip-groups`, or `--skip-events` to import only part of the project. -Use `--skip-person-output` when resuming an event import after person rows were already written; it still loads people for event hydration. - -## PostHog compatibility - -### Ingestion endpoints - -- `/capture` (single or batch payloads) -- `/identify` -- `/alias` -- `/batch` (mixed events) -- `/e` (event payloads) -- `/engage` -- `/groups` -- `/s` (session replay payloads) - -### Persons - -Identify, capture `$set` / `$set_once` / `$unset`, and alias events update a person record stored in a Durable Object. The record tracks distinct_id aliases, person properties, and a sequential `id` plus a UUID. Events include: - -- `person_id` (the person UUID) -- `person_created_at` -- `person_properties` - -The Durable Object is the source of truth for the current person record. When `CLOUDFLARE_PERSONS_PIPELINE_ENDPOINT` is configured, Hogflare also writes append-only person snapshots to the persons pipeline so the state is queryable in Iceberg. - -### Groups - -- `/groups` (`$groupidentify` payloads) are forwarded. -- Group properties are stored in a Group DO and attached to events as `group_properties`. -- Group slots (`group0`..`group4`) are mapped by `POSTHOG_GROUP_TYPE_0..4`. - -### Session replay - -- SDK config advertises `sessionRecording: false` when `POSTHOG_SESSION_RECORDING_ENDPOINT` is unset, so the Worker can keep replay off remotely. Set `POSTHOG_SESSION_RECORDING_ENDPOINT=/s/` to turn replay on and route uploads through Hogflare's ingestion path. -- `/s` accepts PostHog replay payloads, including gzip/gzip-js compressed browser SDK requests. -- Modern `$snapshot` payloads are normalized to `$snapshot_items` rows before they are sent through Cloudflare Pipelines into R2. -- Legacy raw chunk payloads are still accepted as `$snapshot` rows. - -### Feature flags - -Feature flags and SDK remote config are evaluated in the Worker and exposed via `/array/:token/config`, `/decide`, and `/flags`. - -Configuration is a JSON blob in `HOGFLARE_FEATURE_FLAGS`. It can be either: - -- `{ "flags": [ ... ] }` -- `[ ... ]` (array of flag definitions) - -Supported fields per flag: - -| Field | Type | Notes | -| --- | --- | --- | -| `key` | string | Flag key | -| `active` | bool | Defaults to `true` | -| `type` | `"boolean"` \| `"multivariate"` | Defaults to boolean | -| `rollout_percentage` | number | 0–100 | -| `variants` | array | `[{ key, rollout_percentage, payload? }]` | -| `payload` | json | Used for boolean flags | -| `variant_payloads` | map | `{ "variant_key": { ... } }` | -| `conditions` | array | See filters below | -| `group_type` | string | Enables group-based rollout | -| `evaluation_environments` | array | Optional env gating | -| `salt` | string | Optional bucketing salt | -| `id`, `version`, `description` | metadata | Returned in flag details | - -Filters support these operators: - -- `eq` (default), `is_not` -- `in`, `not_in` -- `contains` -- `regex` -- `is_set` -- `gt`, `gte`, `lt`, `lte` - -Value comparisons coerce strings/booleans/numbers when possible (e.g. `"21"` >= `18`). - -Request fields honored by `/flags` and `/decide`: - -- `flag_keys_to_evaluate` — only evaluate these keys -- `evaluation_environments` — only evaluate flags whose `evaluation_environments` includes one of these -- `person_properties`, `group_properties`, `groups` — override state for evaluation - -#### Bucketing - -Rollout bucketing is deterministic: - -- Hash: `sha1("{salt}:{hash_id}")` -- `hash_id` is `distinct_id` for person flags, or the group key when `group_type` is set -- Bucket = `hash % 100` (0–99) -- `salt` defaults to the flag `key` if not provided - -Example: - -```json -{ - "flags": [ - { - "key": "pro-flag", - "active": true, - "rollout_percentage": 100, - "id": 12, - "version": 3, - "description": "Pro users", - "salt": "pro-flag-salt", - "conditions": [ - { - "properties": [ - { "key": "plan", "value": ["pro", "enterprise"], "operator": "in" }, - { "key": "age", "value": 18, "operator": "gte" } - ] - } - ], - "payload": { "tier": "pro" } - } - ] -} -``` - -Limitations: cohorts and event-based filters are not supported. - -### Signing - -- If `POSTHOG_SIGNING_SECRET` is set, requests must include a valid HMAC signature. - -### Enrichment - -Hogflare adds Cloudflare request data into `properties` when those keys are not already present: - -- `$ip` from `CF-Connecting-IP` -- `$geoip_*` from Cloudflare request metadata (country, city, region, lat/long, timezone) -- `cf_*` fields: `cf_asn`, `cf_as_organization`, `cf_colo`, `cf_metro_code`, `cf_ray` - -## Event shape in R2 - -Each row is a `PipelineEvent` with these columns: - -| Field | Type / Notes | -| --- | --- | -| `uuid` | string (UUID v4) | -| `team_id` | int64 (optional) | -| `source` | string | -| `event` | string | -| `distinct_id` | string | -| `timestamp` | RFC3339 timestamp (optional) | -| `created_at` | RFC3339 timestamp | -| `properties` | JSON | -| `context` | JSON | -| `person_id` | string (person UUID) | -| `person_created_at` | RFC3339 timestamp | -| `person_properties` | JSON | -| `group0..group4` | string (group key slots) | -| `group_properties` | JSON (by group type) | -| `api_key` | string | -| `extra` | JSON | - -## Person shape in R2 - -Each row is a `PersonPipelineRecord` snapshot with these columns: - -| Field | Type / Notes | -| --- | --- | -| `uuid` | string (snapshot UUID v4) | -| `team_id` | int64 (optional) | -| `source` | string | -| `operation` | capture, identify, alias, engage, session_recording | -| `person_id` | string (person UUID) | -| `person_int_id` | int64 | -| `canonical_distinct_id` | string | -| `distinct_ids` | string list / array | -| `created_at` | person creation timestamp | -| `updated_at` | snapshot timestamp | -| `version` | person version | -| `properties` | JSON `$set` properties | -| `properties_set_once` | JSON `$set_once` properties | -| `merged_properties` | JSON merged person properties | -| `api_key` | string | -| `source_event_uuid` | event row UUID that produced the snapshot | diff --git a/docs/assets/replay-explorer.jpg b/docs/assets/replay-explorer.jpg new file mode 100644 index 0000000..ca9f608 Binary files /dev/null and b/docs/assets/replay-explorer.jpg differ diff --git a/docs/data-model.md b/docs/data-model.md new file mode 100644 index 0000000..d782834 --- /dev/null +++ b/docs/data-model.md @@ -0,0 +1,72 @@ +# Data Model + +## Query Data + +```sql +INSTALL httpfs; +INSTALL iceberg; +LOAD httpfs; +LOAD iceberg; + +CREATE SECRET r2_catalog_secret ( + TYPE ICEBERG, + TOKEN '' +); + +ATTACH '_' AS iceberg_catalog ( + TYPE ICEBERG, + ENDPOINT 'https://catalog.cloudflarestorage.com//' +); + +SELECT count(*) FROM iceberg_catalog.default.hogflare_events; +SELECT count(*) FROM iceberg_catalog.default.hogflare_persons; +SELECT * FROM iceberg_catalog.default.hogflare_persons LIMIT 5; +``` + +If you used versioned table names during a migration, substitute those names here. + +## Event Shape In R2 + +Each row is a `PipelineEvent` with these columns: + +| Field | Type / Notes | +| --- | --- | +| `uuid` | string (UUID v4) | +| `team_id` | int64 (optional) | +| `source` | string | +| `event` | string | +| `distinct_id` | string | +| `timestamp` | RFC3339 timestamp (optional) | +| `created_at` | RFC3339 timestamp | +| `properties` | JSON | +| `context` | JSON | +| `person_id` | string (person UUID) | +| `person_created_at` | RFC3339 timestamp | +| `person_properties` | JSON | +| `group0..group4` | string (group key slots) | +| `group_properties` | JSON by group type | +| `api_key` | string | +| `extra` | JSON | + +## Person Shape In R2 + +Each row is a `PersonPipelineRecord` snapshot with these columns: + +| Field | Type / Notes | +| --- | --- | +| `uuid` | string (snapshot UUID v4) | +| `team_id` | int64 (optional) | +| `source` | string | +| `operation` | capture, identify, alias, engage, session_recording | +| `person_id` | string (person UUID) | +| `person_int_id` | int64 | +| `canonical_distinct_id` | string | +| `distinct_ids` | string list / array | +| `created_at` | person creation timestamp | +| `updated_at` | snapshot timestamp | +| `version` | person version | +| `properties` | JSON `$set` properties | +| `properties_set_once` | JSON `$set_once` properties | +| `merged_properties` | JSON merged person properties | +| `api_key` | string | +| `source_event_uuid` | event row UUID that produced the snapshot | diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..9e9b5b7 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,284 @@ +# Deployment + +## Quick Start + +1. Create R2 Data Catalog-backed Pipelines resources. +2. Copy `wrangler.toml.example` to `wrangler.toml` and set the stream endpoints. +3. Set Wrangler secrets. +4. Build and deploy the Worker. +5. Send a capture/identify verification flow and query the Iceberg tables. + +The examples below use stable table names for a fresh deployment: `default.hogflare_events` and `default.hogflare_persons`. If you use versioned names during migration, substitute those names consistently in the sink commands and queries. + +## Create Pipeline Resources + +Set these values before creating sinks: + +```bash +export R2_BUCKET="" +export R2_CATALOG_TOKEN="" +``` + +`R2_CATALOG_TOKEN` is the token used by R2 Data Catalog/R2 SQL clients such as DuckDB or PyIceberg. The bucket must have R2 Data Catalog enabled before creating `r2-data-catalog` sinks. + +Create the events stream, sink, and pipeline: + +```bash +bunx wrangler pipelines streams create hogflare_events_stream \ + --schema-file scripts/events-pipeline-schema.json \ + --http-enabled true \ + --http-auth true + +bunx wrangler pipelines sinks create hogflare_events_sink \ + --type r2-data-catalog \ + --bucket "$R2_BUCKET" \ + --namespace default \ + --table hogflare_events \ + --catalog-token "$R2_CATALOG_TOKEN" \ + --roll-interval 60 + +bunx wrangler pipelines create hogflare_events_pipeline \ + --sql "INSERT INTO hogflare_events_sink SELECT * FROM hogflare_events_stream;" +``` + +Create the persons stream, sink, and pipeline if you want queryable people in Iceberg: + +```bash +bunx wrangler pipelines streams create hogflare_persons_stream \ + --schema-file scripts/persons-pipeline-schema.json \ + --http-enabled true \ + --http-auth true + +bunx wrangler pipelines sinks create hogflare_persons_sink \ + --type r2-data-catalog \ + --bucket "$R2_BUCKET" \ + --namespace default \ + --table hogflare_persons \ + --catalog-token "$R2_CATALOG_TOKEN" \ + --roll-interval 60 + +bunx wrangler pipelines create hogflare_persons_pipeline \ + --sql "INSERT INTO hogflare_persons_sink SELECT * FROM hogflare_persons_stream;" +``` + +Each stream creation command prints an HTTP endpoint like `https://.ingest.cloudflare.com`. Use those endpoints in `wrangler.toml`. + +## Wrangler Config + +Copy the example and fill in the stream endpoints: + +```bash +cp wrangler.toml.example wrangler.toml +``` + +```toml +name = "hogflare" +main = "build/index.js" # generated entrypoint from worker-build for the Rust worker +compatibility_date = "2025-01-09" + +[vars] +CLOUDFLARE_PIPELINE_ENDPOINT = "https://.ingest.cloudflare.com" +CLOUDFLARE_PERSONS_PIPELINE_ENDPOINT = "https://.ingest.cloudflare.com" +CLOUDFLARE_PIPELINE_TIMEOUT_SECS = "10" + +# Optional +# POSTHOG_TEAM_ID = "1" +# POSTHOG_GROUP_TYPE_0 = "company" +# POSTHOG_GROUP_TYPE_1 = "team" +# POSTHOG_GROUP_TYPE_2 = "project" +# POSTHOG_GROUP_TYPE_3 = "org" +# POSTHOG_GROUP_TYPE_4 = "workspace" +# POSTHOG_SESSION_RECORDING_ENDPOINT = "/s/" + +[[durable_objects.bindings]] +name = "PERSONS" +class_name = "PersonDurableObject" + +[[durable_objects.bindings]] +name = "PERSON_ID_COUNTER" +class_name = "PersonIdCounterDurableObject" + +[[durable_objects.bindings]] +name = "GROUPS" +class_name = "GroupDurableObject" + +[[migrations]] +tag = "v1" +new_sqlite_classes = ["PersonDurableObject"] + +[[migrations]] +tag = "v2" +new_sqlite_classes = ["PersonIdCounterDurableObject", "GroupDurableObject"] +``` + +## Configuration Reference + +| Setting | Required | Notes | +| --- | --- | --- | +| `CLOUDFLARE_PIPELINE_ENDPOINT` | Yes | Events stream HTTP endpoint from `wrangler pipelines streams create`. | +| `CLOUDFLARE_PIPELINE_AUTH_TOKEN` | Yes, for authenticated streams | Bearer token used for events stream HTTP ingest. | +| `CLOUDFLARE_PERSONS_PIPELINE_ENDPOINT` | No | Persons stream endpoint. Set this to write person snapshots to Iceberg. | +| `CLOUDFLARE_PERSONS_PIPELINE_AUTH_TOKEN` | No | Falls back to `CLOUDFLARE_PIPELINE_AUTH_TOKEN` when omitted. | +| `CLOUDFLARE_PIPELINE_TIMEOUT_SECS` | No | Defaults to 10 seconds. | +| `POSTHOG_API_KEY` | No | Default project token returned by `/decide` when request/header token is absent. | +| `POSTHOG_TEAM_ID` | No | Optional team id attached to event and person rows. | +| `POSTHOG_GROUP_TYPE_0..4` | No | Maps PostHog group types to `group0..group4`; set `POSTHOG_GROUP_TYPE_0=company` to populate `group0` for company groups. | +| `POSTHOG_SESSION_RECORDING_ENDPOINT` | No | Returned in `/decide` session recording config. | +| `HOGFLARE_REPLAY_ACCOUNT_ID` | No | Enables the `/replay` UI API when set with bucket and token. | +| `HOGFLARE_REPLAY_BUCKET` | No | R2 bucket name backing the R2 Data Catalog warehouse. | +| `HOGFLARE_REPLAY_R2_SQL_TOKEN` | No | R2 SQL/Data Catalog token used server-side to query replay rows. Store as a secret. | +| `HOGFLARE_REPLAY_EVENTS_TABLE` | No | Iceberg events table queried by replay APIs. Defaults to `default.hogflare_events`. | +| `HOGFLARE_REPLAY_QUERY_LIMIT` | No | Maximum snapshot rows a replay API request can read. Defaults to `5000`. | +| `POSTHOG_SIGNING_SECRET` | No | Enables HMAC request signature checks. | +| `PERSON_DEBUG_TOKEN` | No | Enables `/__debug/person/:id` for deployment verification. | +| `HOGFLARE_FEATURE_FLAGS` | No | JSON flag config used by `/decide` and `/flags`. | + +## Secrets + +Use a Cloudflare API token that can write to Pipelines for `CLOUDFLARE_PIPELINE_AUTH_TOKEN`. The same token can usually be reused for the persons stream. + +```bash +bunx wrangler secret put CLOUDFLARE_PIPELINE_AUTH_TOKEN +# Optional. If omitted, the persons pipeline uses CLOUDFLARE_PIPELINE_AUTH_TOKEN. +bunx wrangler secret put CLOUDFLARE_PERSONS_PIPELINE_AUTH_TOKEN + +# Optional. +bunx wrangler secret put POSTHOG_SIGNING_SECRET +bunx wrangler secret put PERSON_DEBUG_TOKEN +bunx wrangler secret put HOGFLARE_FEATURE_FLAGS +bunx wrangler secret put HOGFLARE_REPLAY_R2_SQL_TOKEN +``` + +## Deploy + +```bash +worker-build --release +bunx wrangler deploy +``` + +## Verify Deployment + +```bash +export HOGFLARE_URL="https://.workers.dev" +export HOGFLARE_API_KEY="phc_verify_$(date -u +%Y%m%d%H%M%S)" +export HOGFLARE_ANON_ID="${HOGFLARE_API_KEY}_anon" +export HOGFLARE_USER_ID="${HOGFLARE_API_KEY}_user" +``` + +Send an anonymous capture: + +```bash +curl -X POST "$HOGFLARE_URL/capture" \ + -H "Content-Type: application/json" \ + -d "{ + \"api_key\": \"$HOGFLARE_API_KEY\", + \"event\": \"verify-anon-capture\", + \"distinct_id\": \"$HOGFLARE_ANON_ID\", + \"properties\": { + \"\$set\": { \"initial_referrer\": \"docs\" }, + \"\$set_once\": { \"first_seen_source\": \"readme\" } + } + }" +``` + +Identify the user and link the anonymous ID: + +```bash +curl -X POST "$HOGFLARE_URL/identify" \ + -H "Content-Type: application/json" \ + -d "{ + \"api_key\": \"$HOGFLARE_API_KEY\", + \"distinct_id\": \"$HOGFLARE_USER_ID\", + \"properties\": { + \"\$anon_distinct_id\": \"$HOGFLARE_ANON_ID\", + \"\$set\": { \"email\": \"verify@example.com\", \"plan\": \"pro\" }, + \"\$set_once\": { \"signup_source\": \"readme\" } + } + }" +``` + +Send a post-identify capture: + +```bash +curl -X POST "$HOGFLARE_URL/capture" \ + -H "Content-Type: application/json" \ + -d "{ + \"api_key\": \"$HOGFLARE_API_KEY\", + \"event\": \"verify-identified-capture\", + \"distinct_id\": \"$HOGFLARE_USER_ID\", + \"properties\": { \"button\": \"verify\" } + }" +``` + +Wait for the sink roll interval, then query R2 SQL: + +```bash +export R2_WAREHOUSE="_" +export WRANGLER_R2_SQL_AUTH_TOKEN="$R2_CATALOG_TOKEN" + +bunx wrangler r2 sql query "$R2_WAREHOUSE" \ + "select event, distinct_id, person_id, person_properties + from default.hogflare_events + where api_key = '$HOGFLARE_API_KEY' + order by created_at asc" + +bunx wrangler r2 sql query "$R2_WAREHOUSE" \ + "select operation, canonical_distinct_id, person_id, distinct_ids, merged_properties + from default.hogflare_persons + where api_key = '$HOGFLARE_API_KEY' + order by updated_at asc" +``` + +Expected result: the three event rows share one `person_id`, and the persons table has `capture`, `identify`, `capture` snapshots. After identify, `distinct_ids` should include both the anonymous and identified IDs. + +## Local Development + +The repo includes a lightweight fake pipeline used by tests. + +```bash +docker compose up --build -d fake-pipeline +``` + +```bash +# .env.local (not committed) +CLOUDFLARE_PIPELINE_ENDPOINT=http://127.0.0.1:8088/ +CLOUDFLARE_PERSONS_PIPELINE_ENDPOINT=http://127.0.0.1:8088/ +CLOUDFLARE_PIPELINE_TIMEOUT_SECS=5 +``` + +```bash +cargo run --bin hogflare +``` + +## Cleanup + +Delete Pipelines resources in dependency order: pipelines first, then streams and sinks. + +```bash +bunx wrangler pipelines list +bunx wrangler pipelines delete --force + +bunx wrangler pipelines streams list +bunx wrangler pipelines streams delete --force + +bunx wrangler pipelines sinks list +bunx wrangler pipelines sinks delete --force +``` + +`wrangler r2 sql query` is read-only. To drop an Iceberg table from R2 Data Catalog, use the Iceberg catalog API. One local option is PyIceberg: + +```bash +R2_CATALOG_TOKEN="" uv run --with pyiceberg python - <<'PY' +import os +from pyiceberg.catalog.rest import RestCatalog + +catalog = RestCatalog( + name="hogflare", + warehouse="_", + uri="https://catalog.cloudflarestorage.com//", + token=os.environ["R2_CATALOG_TOKEN"], +) + +catalog.drop_table(("default", ""), purge_requested=True) +PY +``` diff --git a/docs/import-posthog.md b/docs/import-posthog.md new file mode 100644 index 0000000..cafaf3f --- /dev/null +++ b/docs/import-posthog.md @@ -0,0 +1,98 @@ +# Import Existing PostHog Data + +Hogflare includes a host-side importer for backfilling an existing PostHog project into the same Cloudflare Pipeline sink used by the Worker. It reads PostHog's private API with a personal API key, then writes normalized rows to the pipeline: + +- persons as `$identify` rows +- groups as `$groupidentify` rows +- historical events from HogQL with original `timestamp`, `created_at`, and PostHog event `uuid` when available + +The importer writes historical rows directly to the pipeline. It does not mutate Worker Durable Object state. + +## Required Inputs + +```bash +export POSTHOG_PROJECT_ID="" +export POSTHOG_PERSONAL_API_KEY="phx_..." +export CLOUDFLARE_PIPELINE_ENDPOINT="https://.ingest.cloudflare.com" +export CLOUDFLARE_PIPELINE_AUTH_TOKEN="" # if your stream requires it +``` + +## Optional Inputs + +```bash +export POSTHOG_HOST="https://us.posthog.com" # or https://eu.posthog.com / self-hosted URL +export POSTHOG_ENVIRONMENT_ID="" # recommended for current PostHog persons/groups APIs +export HOGFLARE_API_KEY="phc_..." +export POSTHOG_TEAM_ID="1" +export POSTHOG_GROUP_TYPE_0="company" +export IMPORT_FROM="2025-01-01" +export IMPORT_TO="2025-02-01" +export IMPORT_BATCH_SIZE="500" +export IMPORT_PERSONS_OFFSET="0" # resume guardrails +export IMPORT_EVENTS_OFFSET="0" +export IMPORT_EVENTS_AFTER_TIMESTAMP="2024-09-21T03:24:11Z" +export IMPORT_EVENTS_AFTER_UUID="0192129b-c354-77b4-b496-9be7ec571fb4" +export IMPORT_EVENT_UUIDS_FILE="/tmp/missing-event-uuids.txt" +export IMPORT_EVENT_WINDOW_DAYS="7" +export IMPORT_EVENT_WINDOW_HOURS="6" # use days or hours, not both +export IMPORT_MAX_PERSONS="1000" # optional guardrails for smoke tests +export IMPORT_MAX_GROUPS="1000" +export IMPORT_MAX_EVENTS="1000" +export IMPORT_STATE_FILE=".hogflare-import-state.jsonl" +export IMPORT_TARGET_ACCOUNT_ID="" +export IMPORT_TARGET_BUCKET="" +export IMPORT_TARGET_TABLE="default.hogflare_events_v3" +export WRANGLER_R2_SQL_AUTH_TOKEN="" +export IMPORT_CLOUDFLARE_API_TOKEN="" # optional auto flush discovery +export IMPORT_PIPELINE_FLUSH_SECS="300" # fallback if Pipelines read is unavailable +``` + +Production imports require R2 SQL target checks by default. The importer uses stable import keys, queries the target before each batch, and skips rows that are already present. Cloudflare Pipeline/R2 is append-only and does not enforce uniqueness by itself. Passing `--no-target-check` or `IMPORT_TARGET_CHECKS=false` opts out and should only be used for local tests. + +Retry behavior is intentionally conservative. Import sends are not blindly retried after a transport or response error because the pipeline may have accepted the batch even if the client did not receive the response. The importer aligns its wait window to the Cloudflare Pipeline sink rolling policy when `IMPORT_CLOUDFLARE_API_TOKEN` can read Pipelines. Without that API access, it uses `IMPORT_PIPELINE_FLUSH_SECS`, defaulting conservatively to 300 seconds. The wait is `max(60s, 2 * flush + 30s)`, unless `IMPORT_TARGET_WAIT_SECS` is set explicitly. + +The local state file makes normal same-machine resumes cheap, but it is not a substitute for target checks if the state file is lost, multiple importers run concurrently, or a send has an unknown commit state. + +## Dry Run + +```bash +cargo run --bin import_posthog -- --dry-run +``` + +## Run Import + +```bash +cargo run --bin import_posthog +``` + +You can also pass flags instead of env vars: + +```bash +cargo run --bin import_posthog -- \ + --posthog-host https://us.posthog.com \ + --project-id 12345 \ + --environment-id 67890 \ + --personal-api-key "$POSTHOG_PERSONAL_API_KEY" \ + --pipeline-endpoint "$CLOUDFLARE_PIPELINE_ENDPOINT" \ + --pipeline-auth-token "$CLOUDFLARE_PIPELINE_AUTH_TOKEN" \ + --hogflare-api-key phc_example \ + --from 2025-01-01 \ + --to 2025-02-01 \ + --persons-offset 0 \ + --events-offset 0 \ + --events-after-timestamp 2024-09-21T03:24:11Z \ + --events-after-uuid 0192129b-c354-77b4-b496-9be7ec571fb4 \ + --event-uuids-file /tmp/missing-event-uuids.txt \ + --event-window-hours 6 \ + --max-persons 1000 \ + --max-groups 1000 \ + --max-events 1000 \ + --import-state-file .hogflare-import-state.jsonl \ + --target-account-id "$CLOUDFLARE_ACCOUNT_ID" \ + --target-bucket hogflare \ + --target-table default.hogflare_events_v3 \ + --target-auth-token "$WRANGLER_R2_SQL_AUTH_TOKEN" \ + --cloudflare-api-token "$CLOUDFLARE_API_TOKEN" +``` + +Use `--skip-persons`, `--skip-groups`, or `--skip-events` to import only part of the project. Use `--skip-person-output` when resuming an event import after person rows were already written; it still loads people for event hydration. diff --git a/docs/posthog-compatibility.md b/docs/posthog-compatibility.md new file mode 100644 index 0000000..c964f14 --- /dev/null +++ b/docs/posthog-compatibility.md @@ -0,0 +1,180 @@ +# PostHog Compatibility + +## SDK Config + +### Browser + +```js +import posthog from "posthog-js"; + +posthog.init("", { + api_host: "https://.workers.dev", + capture_pageview: true, +}); +``` + +### Server + +```js +import { PostHog } from "posthog-node"; + +const client = new PostHog("", { + host: "https://.workers.dev", +}); + +client.capture({ + distinctId: "user_123", + event: "purchase", + properties: { amount: 29.99 }, +}); + +await client.shutdown(); +``` + +### Other SDKs + +Set the SDK host/base URL to your Worker (`https://.workers.dev`) and use your project API key. Most SDKs use either `api_host` for browser/mobile clients or `host` for server clients. + +## Ingestion Endpoints + +- `/capture` accepts single event payloads. +- `/identify` links and updates people. +- `/alias` creates aliases. +- `/batch` accepts mixed event payloads. +- `/e` accepts browser event payloads. +- `/engage` accepts people updates. +- `/groups` accepts `$groupidentify` payloads. +- `/s` accepts session replay payloads. + +## Persons + +Identify, capture `$set` / `$set_once` / `$unset`, and alias events update a person record stored in a Durable Object. The record tracks distinct ID aliases, person properties, and a sequential `id` plus a UUID. + +Events include: + +- `person_id` +- `person_created_at` +- `person_properties` + +The Durable Object is the source of truth for the current person record. When `CLOUDFLARE_PERSONS_PIPELINE_ENDPOINT` is configured, Hogflare also writes append-only person snapshots to the persons pipeline so the state is queryable in Iceberg. + +## Groups + +- `/groups` (`$groupidentify` payloads) are forwarded. +- Group properties are stored in a Group DO and attached to events as `group_properties`. +- Group slots (`group0`..`group4`) are mapped by `POSTHOG_GROUP_TYPE_0..4`. + +## Feature Flags + +Feature flags and SDK remote config are evaluated in the Worker and exposed via `/array/:token/config`, `/decide`, and `/flags`. + +Configuration is a JSON blob in `HOGFLARE_FEATURE_FLAGS`. It can be either: + +- `{ "flags": [ ... ] }` +- `[ ... ]` + +Supported fields per flag: + +| Field | Type | Notes | +| --- | --- | --- | +| `key` | string | Flag key | +| `active` | bool | Defaults to `true` | +| `type` | `"boolean"` or `"multivariate"` | Defaults to boolean | +| `rollout_percentage` | number | 0 to 100 | +| `variants` | array | `[{ key, rollout_percentage, payload? }]` | +| `payload` | JSON | Used for boolean flags | +| `variant_payloads` | map | `{ "variant_key": { ... } }` | +| `conditions` | array | See filters below | +| `group_type` | string | Enables group-based rollout | +| `evaluation_environments` | array | Optional environment gating | +| `salt` | string | Optional bucketing salt | +| `id`, `version`, `description` | metadata | Returned in flag details | + +Filters support these operators: + +- `eq` (default), `is_not` +- `in`, `not_in` +- `contains` +- `regex` +- `is_set` +- `gt`, `gte`, `lt`, `lte` + +Value comparisons coerce strings, booleans, and numbers when possible. For example, `"21"` can be compared with `18`. + +Request fields honored by `/flags` and `/decide`: + +- `flag_keys_to_evaluate` +- `evaluation_environments` +- `person_properties` +- `group_properties` +- `groups` + +## Bucketing + +Rollout bucketing is deterministic: + +- Hash: `sha1("{salt}:{hash_id}")` +- `hash_id` is `distinct_id` for person flags, or the group key when `group_type` is set +- Bucket: `hash % 100` +- `salt` defaults to the flag `key` if not provided + +Example: + +```json +{ + "flags": [ + { + "key": "pro-flag", + "active": true, + "rollout_percentage": 100, + "id": 12, + "version": 3, + "description": "Pro users", + "salt": "pro-flag-salt", + "conditions": [ + { + "properties": [ + { "key": "plan", "value": ["pro", "enterprise"], "operator": "in" }, + { "key": "age", "value": 18, "operator": "gte" } + ] + } + ], + "payload": { "tier": "pro" } + } + ] +} +``` + +Limitations: cohorts and event-based filters are not supported. + +## HMAC Signing + +If `POSTHOG_SIGNING_SECRET` is set, requests must include a valid signature. + +```bash +payload='[ + { + "api_key": "phc_example", + "event": "purchase", + "distinct_id": "user_12345", + "properties": { "amount": 29.99 } + } +]' + +signature=$(printf '%s' "$payload" | openssl dgst -sha256 -hmac "$POSTHOG_SIGNING_SECRET" | awk '{print $2}') + +curl -X POST https://.workers.dev/capture \ + -H "Content-Type: application/json" \ + -H "X-POSTHOG-SIGNATURE: sha256=$signature" \ + -d "$payload" +``` + +`X-HUB-SIGNATURE` with `sha1=` is also accepted for GitHub-style webhook compatibility. + +## Enrichment + +Hogflare adds Cloudflare request data into `properties` when those keys are not already present: + +- `$ip` from `CF-Connecting-IP` +- `$geoip_*` from Cloudflare request metadata +- `cf_*` fields: `cf_asn`, `cf_as_organization`, `cf_colo`, `cf_metro_code`, `cf_ray` diff --git a/docs/session-replay.md b/docs/session-replay.md new file mode 100644 index 0000000..3259f1c --- /dev/null +++ b/docs/session-replay.md @@ -0,0 +1,87 @@ +# Session Replay + +![Hogflare replay explorer](assets/replay-explorer.jpg) + +Hogflare stores replay uploads in the same events table as analytics events, then serves a read-only replay explorer from `/replay`. The UI is for product analytics workflows: browse recent sessions, search events, inspect funnel drop-offs, review computed friction signals, and follow a person journey. + +## Ingestion + +- SDK config advertises `sessionRecording: false` when `POSTHOG_SESSION_RECORDING_ENDPOINT` is unset. +- Set `POSTHOG_SESSION_RECORDING_ENDPOINT=/s/` to route replay uploads through Hogflare. +- `/s` accepts PostHog replay payloads, including gzip/gzip-js compressed browser SDK requests. +- Modern `$snapshot` payloads are normalized to `$snapshot_items` rows before they are sent through Cloudflare Pipelines into R2. +- Legacy raw chunk payloads are still accepted as `$snapshot` rows. + +## Replay API Config + +Replay APIs require: + +| Setting | Notes | +| --- | --- | +| `HOGFLARE_REPLAY_ACCOUNT_ID` | Cloudflare account id for the R2 SQL endpoint. | +| `HOGFLARE_REPLAY_BUCKET` | R2 bucket name backing the R2 Data Catalog warehouse. | +| `HOGFLARE_REPLAY_R2_SQL_TOKEN` | R2 SQL/Data Catalog token. Store as a secret. | +| `HOGFLARE_REPLAY_EVENTS_TABLE` | Optional Iceberg table. Defaults to `default.hogflare_events`. | +| `HOGFLARE_REPLAY_QUERY_LIMIT` | Optional maximum rows read per replay API request. Defaults to `5000`. | +| `HOGFLARE_REPLAY_R2_SQL_ENDPOINT` | Optional override used by tests and local demos. | + +The token stays server-side in the Worker. The browser only calls Hogflare's replay API. + +## Routes + +- `/replay` serves the explorer UI. +- `/replay/api/sessions` lists replay sessions by reading `$snapshot_items` and legacy `$snapshot` rows from Iceberg through R2 SQL. +- `/replay/api/events` searches analytics events while excluding replay recording rows. +- `/replay/api/funnels` classifies sessions as converted, stuck, or dropped for an ordered `steps` list of event names. +- `/replay/api/friction` computes replay-derived signals such as rage clicks, dead clicks, form thrash, long idle gaps, repeated navigation, and deep scroll without follow-up. +- `/replay/api/person` joins a distinct ID's replay sessions and analytics events into one journey timeline. +- `/replay/api/sessions/:session_id` returns normalized rrweb events plus an activity timeline for one session. + +## Filters + +These query parameters can narrow replay reads: + +- `api_key` +- `distinct_id` +- `session_id` +- `url` +- `event_name` +- `steps` +- `signal` +- `date_from` +- `date_to` +- `min_duration_secs` +- `max_duration_secs` +- `min_events` +- `max_events` +- `limit` + +Session deep links use `session_id` plus `at_ms`. + +## Local Demo + +Start the replay SQL demo stub: + +```bash +REPLAY_DEMO_PORT=4666 bun scripts/replay_demo_stub.mjs +``` + +Start Hogflare against the stub: + +```bash +APP_ADDR=127.0.0.1:4567 \ +CLOUDFLARE_PIPELINE_ENDPOINT=http://127.0.0.1:4666/ \ +HOGFLARE_REPLAY_ACCOUNT_ID=demo-account \ +HOGFLARE_REPLAY_BUCKET=demo-bucket \ +HOGFLARE_REPLAY_R2_SQL_TOKEN=demo-token \ +HOGFLARE_REPLAY_R2_SQL_ENDPOINT=http://127.0.0.1:4666/ \ +HOGFLARE_REPLAY_EVENTS_TABLE=default.hogflare_events \ +HOGFLARE_REPLAY_QUERY_LIMIT=500 \ +cargo run --bin hogflare +``` + +Open: + +```text +http://127.0.0.1:4567/replay?api_key=phc_demo&distinct_id=replay-user&limit=100&session_id=demo-session-1&at_ms=1500 +``` diff --git a/scripts/replay_demo_stub.mjs b/scripts/replay_demo_stub.mjs new file mode 100644 index 0000000..bc94d6d --- /dev/null +++ b/scripts/replay_demo_stub.mjs @@ -0,0 +1,497 @@ +const port = Number(process.env.REPLAY_DEMO_PORT || 4666); + +let nextNodeId = 1; + +function id() { + return nextNodeId++; +} + +function text(textContent) { + return { + type: 3, + textContent, + id: id(), + }; +} + +function textWithId(nodeId, textContent) { + return { + type: 3, + textContent, + id: nodeId, + }; +} + +function element(tagName, attributes = {}, childNodes = []) { + return { + type: 2, + tagName, + attributes, + childNodes, + id: id(), + }; +} + +function elementWithId(nodeId, tagName, attributes = {}, childNodes = []) { + return { + type: 2, + tagName, + attributes, + childNodes, + id: nodeId, + }; +} + +const css = ` + * { box-sizing: border-box; } + body { + margin: 0; + min-height: 100vh; + background: #f4f7fb; + color: #101418; + font-family: Inter, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; + } + .shell { + display: grid; + min-height: 100vh; + grid-template-rows: 64px 1fr; + } + .top { + display: flex; + align-items: center; + justify-content: space-between; + border-bottom: 1px solid #d9e1ea; + background: #ffffff; + padding: 0 32px; + } + .brand { + font-size: 18px; + font-weight: 760; + } + .nav { + display: flex; + gap: 24px; + color: #627081; + font-size: 14px; + font-weight: 650; + } + .content { + display: grid; + grid-template-columns: minmax(420px, 0.95fr) minmax(430px, 0.75fr); + gap: 28px; + padding: 32px; + } + .hero, .checkout { + border: 1px solid #d9e1ea; + border-radius: 8px; + background: #ffffff; + box-shadow: 0 20px 60px rgba(30, 44, 60, 0.08); + } + .hero { + padding: 34px; + } + .eyebrow { + color: #087a55; + font-size: 12px; + font-weight: 760; + letter-spacing: 0.04em; + text-transform: uppercase; + } + h1 { + max-width: 700px; + margin: 14px 0 14px; + font-size: 44px; + line-height: 1.02; + letter-spacing: 0; + } + .lede { + max-width: 640px; + color: #5f6d7a; + font-size: 17px; + line-height: 1.5; + } + .plans { + display: grid; + grid-template-columns: repeat(3, minmax(0, 1fr)); + gap: 14px; + margin-top: 28px; + } + .plan { + min-height: 164px; + border: 1px solid #d9e1ea; + border-radius: 8px; + padding: 18px; + } + .plan.active { + border-color: #087a55; + background: #e8f6ef; + } + .plan strong { + display: block; + font-size: 15px; + } + .price { + margin: 12px 0; + font-size: 28px; + font-weight: 760; + } + .plan span { + color: #627081; + font-size: 13px; + line-height: 1.45; + } + .checkout { + align-self: start; + padding: 24px; + } + .checkout h2 { + margin: 0 0 6px; + font-size: 22px; + } + .checkout p { + margin: 0 0 22px; + color: #627081; + } + .notice { + margin: 0 0 16px; + border: 1px solid #c7d5e4; + border-radius: 6px; + background: #f4f8fc; + color: #4d5965; + padding: 10px 12px; + font-size: 13px; + font-weight: 650; + } + .notice.active { + border-color: #d68b21; + background: #fff5e6; + color: #804d08; + } + .field { + display: grid; + gap: 6px; + margin-bottom: 14px; + } + label { + color: #4d5965; + font-size: 12px; + font-weight: 720; + } + input { + height: 42px; + border: 1px solid #ccd6e0; + border-radius: 6px; + padding: 0 12px; + color: #101418; + font: inherit; + } + .cta { + width: 100%; + height: 44px; + border: 0; + border-radius: 6px; + background: #111827; + color: #ffffff; + font-size: 14px; + font-weight: 760; + } + .cta[disabled] { + background: #6b7280; + } + .summary { + display: grid; + gap: 9px; + margin-top: 18px; + border-top: 1px solid #d9e1ea; + padding-top: 18px; + color: #627081; + font-size: 13px; + } + .summary div { + display: flex; + justify-content: space-between; + } + .summary strong { + color: #101418; + } +`; + +const emailInputId = id(); +const companyInputId = id(); +const noticeId = id(); +const noticeTextId = id(); +const ctaButtonId = id(); +const ctaTextId = id(); + +function snapshotNode() { + return { + type: 0, + childNodes: [ + { + type: 1, + name: "html", + publicId: "", + systemId: "", + id: id(), + }, + element("html", { lang: "en" }, [ + element("head", {}, [ + element("title", {}, [text("Acme Analytics Checkout")]), + element("style", {}, [text(css)]), + ]), + element("body", {}, [ + element("div", { class: "shell" }, [ + element("header", { class: "top" }, [ + element("div", { class: "brand" }, [text("Acme Analytics")]), + element("nav", { class: "nav" }, [ + element("span", {}, [text("Pricing")]), + element("span", {}, [text("Docs")]), + element("span", {}, [text("Support")]), + ]), + ]), + element("main", { class: "content" }, [ + element("section", { class: "hero" }, [ + element("div", { class: "eyebrow" }, [text("Replay demo account")]), + element("h1", {}, [text("Understand every product moment without guessing.")]), + element("p", { class: "lede" }, [ + text( + "This is a real rrweb snapshot used by the Hogflare demo, with pricing cards, a checkout form, and recorded interactions." + ), + ]), + element("div", { class: "plans" }, [ + element("article", { class: "plan" }, [ + element("strong", {}, [text("Starter")]), + element("div", { class: "price" }, [text("$49")]), + element("span", {}, [text("Basic ingestion and replay for smaller teams.")]), + ]), + element("article", { class: "plan active" }, [ + element("strong", {}, [text("Pro")]), + element("div", { class: "price" }, [text("$149")]), + element("span", {}, [text("Event search, funnels, replay, and friction signals.")]), + ]), + element("article", { class: "plan" }, [ + element("strong", {}, [text("Scale")]), + element("div", { class: "price" }, [text("Custom")]), + element("span", {}, [text("Warehouse-first analytics for high-volume teams.")]), + ]), + ]), + ]), + element("aside", { class: "checkout" }, [ + element("h2", {}, [text("Start Pro trial")]), + element("p", {}, [text("Recorded checkout path for replay verification.")]), + elementWithId(noticeId, "div", { class: "notice" }, [ + textWithId(noticeTextId, "Ready. Fill the form to continue."), + ]), + element("div", { class: "field" }, [ + element("label", { for: "email" }, [text("Work email")]), + { + type: 2, + tagName: "input", + attributes: { + id: "email", + type: "email", + value: "", + placeholder: "you@company.com", + }, + childNodes: [], + id: emailInputId, + }, + ]), + element("div", { class: "field" }, [ + element("label", { for: "company" }, [text("Company")]), + { + type: 2, + tagName: "input", + attributes: { + id: "company", + type: "text", + value: "", + placeholder: "Company name", + }, + childNodes: [], + id: companyInputId, + }, + ]), + elementWithId(ctaButtonId, "button", { class: "cta" }, [ + textWithId(ctaTextId, "Continue to payment"), + ]), + element("div", { class: "summary" }, [ + element("div", {}, [element("span", {}, [text("Plan")]), element("strong", {}, [text("Pro")])]), + element("div", {}, [element("span", {}, [text("Seats")]), element("strong", {}, [text("8")])]), + element("div", {}, [element("span", {}, [text("Due today")]), element("strong", {}, [text("$0")])]), + ]), + ]), + ]), + ]), + ]), + ]), + ], + id: id(), + }; +} + +const demoEvents = [ + { + type: 4, + timestamp: 1_000, + data: { + href: "https://app.test/pricing", + width: 1365, + height: 768, + }, + }, + { + type: 2, + timestamp: 1_100, + data: { + node: snapshotNode(), + initialOffset: { left: 0, top: 0 }, + }, + }, + { type: 3, timestamp: 1_700, data: { source: 2, type: 5, x: 815, y: 255 } }, + { type: 3, timestamp: 2_100, data: { source: 5, id: emailInputId, text: "nico@atm.com", isChecked: false } }, + { type: 3, timestamp: 2_700, data: { source: 5, id: companyInputId, text: "ATM.COM", isChecked: false } }, + { type: 3, timestamp: 3_400, data: { source: 2, type: 2, x: 948, y: 517 } }, + { + type: 3, + timestamp: 3_650, + data: { + source: 0, + texts: [{ id: ctaTextId, value: "Checking workspace..." }], + attributes: [{ id: ctaButtonId, attributes: { class: "cta", disabled: "true" } }], + removes: [], + adds: [], + }, + }, + { type: 3, timestamp: 4_050, data: { source: 2, type: 2, x: 950, y: 518 } }, + { type: 3, timestamp: 4_450, data: { source: 2, type: 2, x: 951, y: 516 } }, + { + type: 3, + timestamp: 4_950, + data: { + source: 0, + texts: [ + { id: noticeTextId, value: "Payment method missing. Add card details to continue." }, + { id: ctaTextId, value: "Continue to payment" }, + ], + attributes: [ + { id: noticeId, attributes: { class: "notice active" } }, + { id: ctaButtonId, attributes: { class: "cta" } }, + ], + removes: [], + adds: [], + }, + }, + { type: 3, timestamp: 6_200, data: { source: 3, x: 0, y: 720 } }, +]; + +const journeyEvents = [ + { type: 4, timestamp: 1_000, data: { href: "https://app.test/home", width: 1365, height: 768 } }, + { + type: 2, + timestamp: 1_100, + data: { + node: snapshotNode(), + initialOffset: { left: 0, top: 0 }, + }, + }, + { type: 3, timestamp: 1_700, data: { source: 3, x: 0, y: 420 } }, +]; + +const rows = [ + { + uuid: "demo-chunk-1", + event: "$snapshot", + distinct_id: "replay-user", + created_at: "2026-05-22T10:00:01Z", + api_key: "phc_demo", + properties: { + session_id: "demo-session-1", + events: demoEvents, + }, + }, + { + uuid: "demo-chunk-2", + event: "$snapshot", + distinct_id: "journey-user", + created_at: "2026-05-22T10:03:00Z", + api_key: "phc_demo", + properties: { + session_id: "journey-session", + events: journeyEvents, + }, + }, + { + uuid: "event-pricing", + event: "Viewed Pricing", + distinct_id: "replay-user", + created_at: "2026-05-22T10:00:02Z", + api_key: "phc_demo", + properties: { + "$session_id": "demo-session-1", + "$current_url": "https://app.test/pricing", + plan: "pro", + }, + }, + { + uuid: "event-checkout", + event: "Checkout Started", + distinct_id: "replay-user", + created_at: "2026-05-22T10:00:05Z", + api_key: "phc_demo", + properties: { + "$session_id": "demo-session-1", + "$current_url": "https://app.test/checkout", + plan: "pro", + }, + }, + { + uuid: "event-stuck-1", + event: "Viewed Pricing", + distinct_id: "stuck-user", + created_at: "2026-05-22T10:01:00Z", + api_key: "phc_demo", + properties: { + "$session_id": "stuck-session", + "$current_url": "https://app.test/pricing", + }, + }, + { + uuid: "event-stuck-2", + event: "Viewed Pricing", + distinct_id: "stuck-user", + created_at: "2026-05-22T10:01:03Z", + api_key: "phc_demo", + properties: { + "$session_id": "stuck-session", + "$current_url": "https://app.test/pricing", + }, + }, + { + uuid: "event-journey", + event: "Product Viewed", + distinct_id: "journey-user", + created_at: "2026-05-22T10:03:05Z", + api_key: "phc_demo", + properties: { + "$session_id": "journey-session", + "$current_url": "https://app.test/product", + sku: "sku_123", + }, + }, +]; + +Bun.serve({ + port, + async fetch(request) { + let payload = {}; + try { + payload = await request.json(); + } catch {} + console.log(payload.query || "no query"); + return Response.json({ result: { rows } }); + }, +}); + +console.log(`replay sql demo stub listening on ${port}`); +await new Promise(() => {}); diff --git a/src/config.rs b/src/config.rs index 9b19b6c..ce7f4cd 100644 --- a/src/config.rs +++ b/src/config.rs @@ -7,6 +7,7 @@ use thiserror::Error; use url::Url; use crate::feature_flags::FeatureFlagStore; +use crate::replay::{ReplayConfig, ReplayConfigError}; #[derive(Debug, Clone)] pub struct Config { @@ -20,6 +21,7 @@ pub struct Config { pub posthog_group_types: [Option; 5], pub posthog_project_api_key: Option, pub session_recording_endpoint: Option, + pub replay: Option, pub posthog_signing_secret: Option, pub person_debug_token: Option, pub feature_flags: FeatureFlagStore, @@ -39,6 +41,8 @@ pub enum ConfigError { InvalidTeamId { value: String, message: String }, #[error("failed to parse HOGFLARE_FEATURE_FLAGS: {0}")] InvalidFeatureFlags(String), + #[error(transparent)] + Replay(#[from] ReplayConfigError), } impl Config { @@ -131,6 +135,7 @@ impl Config { .var("POSTHOG_SESSION_RECORDING_ENDPOINT") .ok() .map(|v| v.to_string()); + let replay = replay_config_from_worker_env(env)?; let posthog_signing_secret = env .secret("POSTHOG_SIGNING_SECRET") .ok() @@ -168,6 +173,7 @@ impl Config { posthog_group_types, posthog_project_api_key, session_recording_endpoint, + replay, posthog_signing_secret, person_debug_token, feature_flags, @@ -262,6 +268,7 @@ impl Config { let posthog_project_api_key = env::var("POSTHOG_API_KEY").ok(); let session_recording_endpoint = env::var("POSTHOG_SESSION_RECORDING_ENDPOINT").ok(); + let replay = replay_config_from_env()?; let posthog_signing_secret = env::var("POSTHOG_SIGNING_SECRET").ok(); let person_debug_token = env::var("PERSON_DEBUG_TOKEN").ok(); let feature_flags = match env::var("HOGFLARE_FEATURE_FLAGS") { @@ -286,9 +293,122 @@ impl Config { posthog_group_types, posthog_project_api_key, session_recording_endpoint, + replay, posthog_signing_secret, person_debug_token, feature_flags, }) } } + +#[cfg(target_arch = "wasm32")] +fn replay_config_from_worker_env(env: &worker::Env) -> Result, ConfigError> { + let account_id = env + .var("HOGFLARE_REPLAY_ACCOUNT_ID") + .ok() + .map(|value| value.to_string()); + let bucket_name = env + .var("HOGFLARE_REPLAY_BUCKET") + .ok() + .map(|value| value.to_string()); + let auth_token = env + .secret("HOGFLARE_REPLAY_R2_SQL_TOKEN") + .ok() + .map(|secret| secret.to_string()) + .or_else(|| { + env.var("HOGFLARE_REPLAY_R2_SQL_TOKEN") + .ok() + .map(|value| value.to_string()) + }); + + let configured = account_id.is_some() || bucket_name.is_some() || auth_token.is_some(); + if !configured { + return Ok(None); + } + + let events_table = env + .var("HOGFLARE_REPLAY_EVENTS_TABLE") + .ok() + .map(|value| value.to_string()); + let query_limit = parse_replay_query_limit( + env.var("HOGFLARE_REPLAY_QUERY_LIMIT") + .ok() + .map(|value| value.to_string()), + )?; + let endpoint = parse_optional_url( + "HOGFLARE_REPLAY_R2_SQL_ENDPOINT", + env.var("HOGFLARE_REPLAY_R2_SQL_ENDPOINT") + .ok() + .map(|value| value.to_string()), + )?; + + ReplayConfig::new( + account_id.ok_or(ConfigError::MissingVar("HOGFLARE_REPLAY_ACCOUNT_ID"))?, + bucket_name.ok_or(ConfigError::MissingVar("HOGFLARE_REPLAY_BUCKET"))?, + auth_token.ok_or(ConfigError::MissingVar("HOGFLARE_REPLAY_R2_SQL_TOKEN"))?, + events_table, + query_limit, + endpoint, + ) + .map(Some) + .map_err(ConfigError::from) +} + +#[cfg(not(target_arch = "wasm32"))] +fn replay_config_from_env() -> Result, ConfigError> { + let account_id = env::var("HOGFLARE_REPLAY_ACCOUNT_ID").ok(); + let bucket_name = env::var("HOGFLARE_REPLAY_BUCKET").ok(); + let auth_token = env::var("HOGFLARE_REPLAY_R2_SQL_TOKEN").ok(); + + let configured = account_id.is_some() || bucket_name.is_some() || auth_token.is_some(); + if !configured { + return Ok(None); + } + + let events_table = env::var("HOGFLARE_REPLAY_EVENTS_TABLE").ok(); + let query_limit = parse_replay_query_limit(env::var("HOGFLARE_REPLAY_QUERY_LIMIT").ok())?; + let endpoint = parse_optional_url( + "HOGFLARE_REPLAY_R2_SQL_ENDPOINT", + env::var("HOGFLARE_REPLAY_R2_SQL_ENDPOINT").ok(), + )?; + + ReplayConfig::new( + account_id.ok_or(ConfigError::MissingVar("HOGFLARE_REPLAY_ACCOUNT_ID"))?, + bucket_name.ok_or(ConfigError::MissingVar("HOGFLARE_REPLAY_BUCKET"))?, + auth_token.ok_or(ConfigError::MissingVar("HOGFLARE_REPLAY_R2_SQL_TOKEN"))?, + events_table, + query_limit, + endpoint, + ) + .map(Some) + .map_err(ConfigError::from) +} + +fn parse_replay_query_limit(value: Option) -> Result, ConfigError> { + value + .map(|value| { + value.parse::().map_err(|err| { + ConfigError::Replay(ReplayConfigError::InvalidQueryLimit { + value, + message: err.to_string(), + }) + }) + }) + .transpose() +} + +fn parse_optional_url( + name: &'static str, + value: Option, +) -> Result, ConfigError> { + value + .map(|value| { + Url::parse(&value).map_err(|err| { + ConfigError::Replay(ReplayConfigError::InvalidEndpoint { + value: format!("{name}={value}"), + message: err.to_string(), + }) + }) + }) + .transpose() +} diff --git a/src/extractors.rs b/src/extractors.rs index 0f1e161..8e141b8 100644 --- a/src/extractors.rs +++ b/src/extractors.rs @@ -909,6 +909,7 @@ mod tests { AppState { pipeline: Arc::new(pipeline), persons_pipeline: None, + replay: None, posthog_team_id: None, decide_api_token: None, session_recording_endpoint: None, diff --git a/src/lib.rs b/src/lib.rs index ca9658f..63334b5 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -7,13 +7,14 @@ pub mod importer; pub mod models; pub mod persons; pub mod pipeline; +pub mod replay; use std::{collections::HashMap, sync::Arc}; use axum::{ extract::{Path, State}, http::{HeaderMap, StatusCode}, - response::IntoResponse, + response::{Html, IntoResponse}, routing::{get, post}, Json, Router, }; @@ -33,6 +34,10 @@ use persons::{ NoopPersonStore, PersonAlias, PersonError, PersonStore, PersonUpdate, }; use pipeline::{PersonPipelineRecord, PipelineClient, PipelineError, PipelineEvent}; +use replay::{ + ReplayClient, ReplayError, ReplayEventsQuery, ReplayFrictionQuery, ReplayFunnelQuery, + ReplayPersonQuery, ReplaySessionEventsQuery, ReplaySessionsQuery, +}; use serde::{Deserialize, Serialize}; use serde_json::{json, Value}; use thiserror::Error; @@ -57,6 +62,7 @@ use tower_service::Service; pub(crate) struct AppState { pub(crate) pipeline: Arc, pub(crate) persons_pipeline: Option>, + pub(crate) replay: Option>, pub(crate) posthog_team_id: Option, pub(crate) decide_api_token: Option, pub(crate) session_recording_endpoint: Option, @@ -168,6 +174,12 @@ pub async fn run_with_config(config: Config) -> Result<(), RunError> { .map(Arc::new) }) .transpose()?; + let replay = config + .replay + .clone() + .map(ReplayClient::new) + .transpose()? + .map(Arc::new); info!( endpoint = %config.pipeline_endpoint, @@ -187,10 +199,11 @@ pub async fn run_with_config(config: Config) -> Result<(), RunError> { let listener = TcpListener::bind(config.address).await?; info!(address = %config.address, "listening for requests"); - serve_with_person_pipeline( + serve_with_person_pipeline_and_replay( listener, Arc::new(pipeline), persons_pipeline, + replay, config.posthog_team_id, Arc::new(NoopGroupStore), GroupTypeMap::new(config.posthog_group_types.clone()), @@ -259,10 +272,25 @@ pub async fn fetch( let group_store: Arc = groups::store_from_env(&env); let group_type_map = GroupTypeMap::new(config.posthog_group_types.clone()); let feature_flags = Arc::new(config.feature_flags); + let replay = match config.replay.clone() { + Some(config) => match ReplayClient::new(config) { + Ok(client) => Some(Arc::new(client)), + Err(err) => { + error!(error = %err, "failed to create replay client"); + let body = Json(ErrorResponse { + status: 0, + error: err.to_string(), + }); + return Ok((StatusCode::INTERNAL_SERVER_ERROR, body).into_response()); + } + }, + None => None, + }; - let mut router = build_router_with_person_pipeline( + let mut router = build_router_with_person_pipeline_and_replay( Arc::new(pipeline), persons_pipeline, + replay, config.posthog_team_id, group_store, group_type_map, @@ -331,10 +359,41 @@ pub fn build_router_with_person_pipeline( person_debug_token: Option, feature_flags: Arc, person_store: Arc, +) -> Router { + build_router_with_person_pipeline_and_replay( + pipeline, + persons_pipeline, + None, + posthog_team_id, + group_store, + group_type_map, + decide_api_token, + session_recording_endpoint, + signing_secret, + person_debug_token, + feature_flags, + person_store, + ) +} + +pub fn build_router_with_person_pipeline_and_replay( + pipeline: Arc, + persons_pipeline: Option>, + replay: Option>, + posthog_team_id: Option, + group_store: Arc, + group_type_map: GroupTypeMap, + decide_api_token: Option, + session_recording_endpoint: Option, + signing_secret: Option, + person_debug_token: Option, + feature_flags: Arc, + person_store: Arc, ) -> Router { router(build_state( pipeline, persons_pipeline, + replay, posthog_team_id, group_store, group_type_map, @@ -355,6 +414,7 @@ pub async fn serve(listener: TcpListener, pipeline: Arc) -> Resu pipeline, None, None, + None, Arc::new(NoopGroupStore), GroupTypeMap::default(), None, @@ -410,10 +470,43 @@ pub async fn serve_with_person_pipeline( signing_secret: Option, person_debug_token: Option, feature_flags: Arc, +) -> Result<(), RunError> { + serve_with_person_pipeline_and_replay( + listener, + pipeline, + persons_pipeline, + None, + posthog_team_id, + group_store, + group_type_map, + decide_api_token, + session_recording_endpoint, + signing_secret, + person_debug_token, + feature_flags, + ) + .await +} + +#[cfg(not(target_arch = "wasm32"))] +pub async fn serve_with_person_pipeline_and_replay( + listener: TcpListener, + pipeline: Arc, + persons_pipeline: Option>, + replay: Option>, + posthog_team_id: Option, + group_store: Arc, + group_type_map: GroupTypeMap, + decide_api_token: Option, + session_recording_endpoint: Option, + signing_secret: Option, + person_debug_token: Option, + feature_flags: Arc, ) -> Result<(), RunError> { let state = build_state( pipeline, persons_pipeline, + replay, posthog_team_id, group_store, group_type_map, @@ -445,6 +538,17 @@ fn router(state: AppState) -> Router { .route("/array/:token/config.js", get(array_config_js)) .route("/s", post(session_recording)) .route("/s/", post(session_recording)) + .route("/replay", get(replay_ui)) + .route("/replay/", get(replay_ui)) + .route("/replay/api/sessions", get(replay_sessions)) + .route("/replay/api/events", get(replay_events)) + .route("/replay/api/funnels", get(replay_funnels)) + .route("/replay/api/friction", get(replay_friction)) + .route("/replay/api/person", get(replay_person)) + .route( + "/replay/api/sessions/:session_id", + get(replay_session_events), + ) .route("/__debug/person/:id", get(debug_person)) .route("/healthz", get(health)) .with_state(state); @@ -462,6 +566,7 @@ fn router(state: AppState) -> Router { fn build_state( pipeline: Arc, persons_pipeline: Option>, + replay: Option>, posthog_team_id: Option, group_store: Arc, group_type_map: GroupTypeMap, @@ -475,6 +580,7 @@ fn build_state( AppState { pipeline, persons_pipeline, + replay, posthog_team_id, group_store, group_type_map, @@ -1620,6 +1726,133 @@ async fn health() -> impl IntoResponse { Json(json!({ "status": "ok" })) } +#[cfg_attr(target_arch = "wasm32", worker::send)] +async fn replay_ui() -> impl IntoResponse { + Html(replay::replay_ui_html()) +} + +#[cfg_attr(target_arch = "wasm32", worker::send)] +async fn replay_sessions( + State(state): State, + axum::extract::Query(query): axum::extract::Query, +) -> impl IntoResponse { + let Some(client) = state.replay.as_ref() else { + return replay_error_response(ReplayError::NotConfigured); + }; + + match client.list_sessions(query).await { + Ok(response) => Json(response).into_response(), + Err(err) => replay_error_response(err), + } +} + +#[cfg_attr(target_arch = "wasm32", worker::send)] +async fn replay_events( + State(state): State, + axum::extract::Query(query): axum::extract::Query, +) -> impl IntoResponse { + let Some(client) = state.replay.as_ref() else { + return replay_error_response(ReplayError::NotConfigured); + }; + + match client.search_events(query).await { + Ok(response) => Json(response).into_response(), + Err(err) => replay_error_response(err), + } +} + +#[cfg_attr(target_arch = "wasm32", worker::send)] +async fn replay_funnels( + State(state): State, + axum::extract::Query(query): axum::extract::Query, +) -> impl IntoResponse { + let Some(client) = state.replay.as_ref() else { + return replay_error_response(ReplayError::NotConfigured); + }; + + match client.search_funnel(query).await { + Ok(response) => Json(response).into_response(), + Err(err) => replay_error_response(err), + } +} + +#[cfg_attr(target_arch = "wasm32", worker::send)] +async fn replay_friction( + State(state): State, + axum::extract::Query(query): axum::extract::Query, +) -> impl IntoResponse { + let Some(client) = state.replay.as_ref() else { + return replay_error_response(ReplayError::NotConfigured); + }; + + match client.search_friction(query).await { + Ok(response) => Json(response).into_response(), + Err(err) => replay_error_response(err), + } +} + +#[cfg_attr(target_arch = "wasm32", worker::send)] +async fn replay_person( + State(state): State, + axum::extract::Query(query): axum::extract::Query, +) -> impl IntoResponse { + let Some(client) = state.replay.as_ref() else { + return replay_error_response(ReplayError::NotConfigured); + }; + + match client.person_journey(query).await { + Ok(response) => Json(response).into_response(), + Err(err) => replay_error_response(err), + } +} + +#[cfg_attr(target_arch = "wasm32", worker::send)] +async fn replay_session_events( + State(state): State, + Path(session_id): Path, + axum::extract::Query(query): axum::extract::Query, +) -> impl IntoResponse { + let Some(client) = state.replay.as_ref() else { + return replay_error_response(ReplayError::NotConfigured); + }; + + match client.get_session(&session_id, query).await { + Ok(response) => Json(response).into_response(), + Err(err) => replay_error_response(err), + } +} + +fn replay_error_response(err: ReplayError) -> axum::response::Response { + let status = match &err { + ReplayError::NotConfigured => StatusCode::SERVICE_UNAVAILABLE, + ReplayError::SessionNotFound(_) => StatusCode::NOT_FOUND, + ReplayError::UnexpectedResponse { .. } + | ReplayError::Transport(_) + | ReplayError::InvalidResponse(_) + | ReplayError::MissingRows + | ReplayError::InvalidRow(_) => StatusCode::BAD_GATEWAY, + #[cfg(not(target_arch = "wasm32"))] + ReplayError::ClientBuild(_) => StatusCode::INTERNAL_SERVER_ERROR, + #[cfg(target_arch = "wasm32")] + ReplayError::RequestBuild(_) | ReplayError::Serialize(_) => { + StatusCode::INTERNAL_SERVER_ERROR + } + }; + + if status.is_server_error() { + error!(error = %err, "replay request failed"); + } + + ( + status, + Json(ErrorResponse { + status: 0, + error: err.to_string(), + }), + ) + .into_response() +} + #[cfg_attr(target_arch = "wasm32", worker::send)] async fn debug_person( State(state): State, @@ -1659,6 +1892,8 @@ pub enum RunError { #[error(transparent)] Pipeline(#[from] PipelineError), #[error(transparent)] + Replay(#[from] ReplayError), + #[error(transparent)] Io(#[from] std::io::Error), #[error("server error: {0}")] Serve(String), diff --git a/src/replay.rs b/src/replay.rs new file mode 100644 index 0000000..83b44b8 --- /dev/null +++ b/src/replay.rs @@ -0,0 +1,2689 @@ +use std::collections::{HashMap, HashSet}; +use std::io::Read; +use std::time::Duration; + +use base64::{engine::general_purpose::STANDARD as BASE64_STANDARD, Engine}; +use chrono::{DateTime, Utc}; +use flate2::read::GzDecoder; +use http::StatusCode; +use serde::{Deserialize, Serialize}; +use serde_json::{json, Map, Value}; +use thiserror::Error; +use url::Url; + +#[cfg(not(target_arch = "wasm32"))] +use reqwest::Client; + +#[cfg(target_arch = "wasm32")] +use worker::{ + wasm_bindgen::JsValue, wasm_bindgen_futures::spawn_local, AbortController, Delay, Fetch, + Headers, Method, Request, RequestInit, +}; + +const DEFAULT_REPLAY_TABLE: &str = "default.hogflare_events"; +const DEFAULT_REPLAY_QUERY_LIMIT: usize = 5_000; +const REPLAY_SQL_TIMEOUT: Duration = Duration::from_secs(20); +const SNAPSHOT_EVENT: &str = "$snapshot"; +const SNAPSHOT_ITEMS_EVENT: &str = "$snapshot_items"; + +#[derive(Debug, Clone)] +pub struct ReplayConfig { + pub account_id: String, + pub bucket_name: String, + pub auth_token: String, + pub events_table: String, + pub query_limit: usize, + pub endpoint: Url, +} + +impl ReplayConfig { + pub fn new( + account_id: String, + bucket_name: String, + auth_token: String, + events_table: Option, + query_limit: Option, + endpoint: Option, + ) -> Result { + let events_table = events_table.unwrap_or_else(|| DEFAULT_REPLAY_TABLE.to_string()); + validate_sql_identifier(&events_table)?; + + let endpoint = match endpoint { + Some(endpoint) => endpoint, + None => Url::parse(&format!( + "https://api.sql.cloudflarestorage.com/api/v1/accounts/{}/r2-sql/query/{}", + account_id, bucket_name + )) + .map_err(|err| ReplayConfigError::InvalidEndpoint { + value: "".to_string(), + message: err.to_string(), + })?, + }; + + Ok(Self { + account_id, + bucket_name, + auth_token, + events_table, + query_limit: query_limit.unwrap_or(DEFAULT_REPLAY_QUERY_LIMIT), + endpoint, + }) + } +} + +#[derive(Debug, Error)] +pub enum ReplayConfigError { + #[error("invalid replay events table `{value}`")] + InvalidEventsTable { value: String }, + #[error("invalid replay endpoint `{value}`: {message}")] + InvalidEndpoint { value: String, message: String }, + #[error("invalid replay query limit `{value}`: {message}")] + InvalidQueryLimit { value: String, message: String }, +} + +#[derive(Debug, Clone)] +pub struct ReplayClient { + config: ReplayConfig, + #[cfg(not(target_arch = "wasm32"))] + client: Client, +} + +impl ReplayClient { + pub fn new(config: ReplayConfig) -> Result { + #[cfg(not(target_arch = "wasm32"))] + let client = Client::builder() + .timeout(REPLAY_SQL_TIMEOUT) + .build() + .map_err(ReplayError::ClientBuild)?; + + Ok(Self { + config, + #[cfg(not(target_arch = "wasm32"))] + client, + }) + } + + pub async fn list_sessions( + &self, + query: ReplaySessionsQuery, + ) -> Result { + let mut rows = self.query_snapshot_rows(&query).await?; + if let Some(session_id) = query.session_id.as_ref().filter(|value| !value.is_empty()) { + rows.retain(|row| session_id_for_row(row) == *session_id); + } + let mut response = build_sessions_response(rows, self.config.query_limit); + apply_session_summary_filters(&mut response.sessions, &query); + + if let Some(event_name) = query.event_name.as_ref().filter(|value| !value.is_empty()) { + let event_query = ReplayEventsQuery { + api_key: query.api_key.clone(), + distinct_id: query.distinct_id.clone(), + session_id: query.session_id.clone(), + event_name: Some(event_name.clone()), + url: query.url.clone(), + date_from: query.date_from.clone(), + date_to: query.date_to.clone(), + limit: Some(self.config.query_limit), + }; + let events = self.query_event_rows(&event_query).await?; + let matching_sessions = matching_session_keys_from_events(events); + response.sessions.retain(|session| { + matching_sessions.session_ids.contains(&session.session_id) + || matching_sessions + .distinct_ids + .contains(&session.distinct_id) + }); + } + + response.sessions.truncate(query.limit_or_default()); + Ok(response) + } + + pub async fn get_session( + &self, + session_id: &str, + query: ReplaySessionEventsQuery, + ) -> Result { + let rows = self + .query_snapshot_rows(&ReplaySessionsQuery { + api_key: query.api_key, + distinct_id: query.distinct_id, + session_id: Some(session_id.to_string()), + url: None, + event_name: None, + date_from: None, + date_to: None, + min_duration_secs: None, + max_duration_secs: None, + min_events: None, + max_events: None, + limit: query.limit, + }) + .await?; + + build_session_events_response(session_id, rows, query.at_ms) + } + + pub async fn search_events( + &self, + query: ReplayEventsQuery, + ) -> Result { + let rows = self.query_event_rows(&query).await?; + Ok(build_events_response(rows, query.limit_or_default())) + } + + pub async fn search_funnel( + &self, + query: ReplayFunnelQuery, + ) -> Result { + let rows = self.query_event_rows(&query.to_events_query()).await?; + Ok(build_funnel_response( + rows, + query.steps_vec(), + query.limit_or_default(), + )) + } + + pub async fn search_friction( + &self, + query: ReplayFrictionQuery, + ) -> Result { + let sessions_query = query.to_sessions_query(); + let mut rows = self.query_snapshot_rows(&sessions_query).await?; + retain_snapshot_rows_for_session(&mut rows, sessions_query.session_id.as_deref()); + + let mut session_response = build_sessions_response(rows.clone(), self.config.query_limit); + apply_session_summary_filters(&mut session_response.sessions, &sessions_query); + let allowed_sessions = session_response + .sessions + .into_iter() + .map(|session| session.session_id) + .collect::>(); + rows.retain(|row| allowed_sessions.contains(&session_id_for_row(row))); + + Ok(build_friction_response( + rows, + query.signal.as_deref(), + query.limit_or_default(), + )) + } + + pub async fn person_journey( + &self, + query: ReplayPersonQuery, + ) -> Result { + let sessions_query = query.to_sessions_query(); + let events_query = query.to_events_query(); + let mut snapshot_rows = self.query_snapshot_rows(&sessions_query).await?; + retain_snapshot_rows_for_session(&mut snapshot_rows, sessions_query.session_id.as_deref()); + + let mut sessions = build_sessions_response(snapshot_rows, self.config.query_limit).sessions; + apply_session_summary_filters(&mut sessions, &sessions_query); + sessions.truncate(query.limit_or_default()); + + let events = build_events_response( + self.query_event_rows(&events_query).await?, + query.limit_or_default(), + ) + .events; + + Ok(build_person_journey_response( + query.distinct_id.clone().filter(|value| !value.is_empty()), + sessions, + events, + query.limit_or_default(), + )) + } + + async fn query_snapshot_rows( + &self, + query: &ReplaySessionsQuery, + ) -> Result, ReplayError> { + let limit = query.limit_or_default().min(self.config.query_limit); + let sql = build_snapshot_rows_sql(&self.config.events_table, query, limit); + let response = self.query_sql(&sql).await?; + let rows = rows_from_r2_sql_response(response)?; + let mut rows = rows + .into_iter() + .map(ReplaySnapshotRow::try_from) + .collect::, _>>()?; + + rows.retain(|row| { + row.event + .as_deref() + .map(is_replay_snapshot_event) + .unwrap_or(true) + }); + if let Some(api_key) = query.api_key.as_ref().filter(|value| !value.is_empty()) { + rows.retain(|row| row.api_key.as_deref() == Some(api_key.as_str())); + } + if let Some(distinct_id) = query.distinct_id.as_ref().filter(|value| !value.is_empty()) { + rows.retain(|row| row.distinct_id == *distinct_id); + } + + Ok(rows) + } + + async fn query_event_rows( + &self, + query: &ReplayEventsQuery, + ) -> Result, ReplayError> { + let limit = query.limit_or_default().min(self.config.query_limit); + let sql = build_event_rows_sql(&self.config.events_table, query, limit); + let response = self.query_sql(&sql).await?; + let rows = rows_from_r2_sql_response(response)?; + let mut rows = rows + .into_iter() + .map(ReplayEventRow::try_from) + .collect::, _>>()?; + + rows.retain(|row| !is_replay_snapshot_event(&row.event)); + if let Some(event_name) = query.event_name.as_ref().filter(|value| !value.is_empty()) { + rows.retain(|row| row.event == *event_name); + } + if let Some(api_key) = query.api_key.as_ref().filter(|value| !value.is_empty()) { + rows.retain(|row| row.api_key.as_deref() == Some(api_key.as_str())); + } + if let Some(distinct_id) = query.distinct_id.as_ref().filter(|value| !value.is_empty()) { + rows.retain(|row| row.distinct_id == *distinct_id); + } + if let Some(session_id) = query.session_id.as_ref().filter(|value| !value.is_empty()) { + rows.retain(|row| { + session_id_from_value(&row.properties).as_deref() == Some(session_id) + }); + } + if let Some(url) = query.url.as_ref().filter(|value| !value.is_empty()) { + rows.retain(|row| { + event_url_from_properties(&row.properties) + .map(|candidate| contains_case_insensitive(&candidate, url)) + .unwrap_or(false) + }); + } + + Ok(rows) + } + + async fn query_sql(&self, sql: &str) -> Result { + #[cfg(not(target_arch = "wasm32"))] + { + let response = self + .client + .post(self.config.endpoint.clone()) + .bearer_auth(&self.config.auth_token) + .json(&json!({ "query": sql })) + .send() + .await + .map_err(ReplayError::Transport)?; + + let status = StatusCode::from_u16(response.status().as_u16()) + .unwrap_or(StatusCode::INTERNAL_SERVER_ERROR); + let body = response.text().await.unwrap_or_default(); + + if !status.is_success() { + return Err(ReplayError::UnexpectedResponse { status, body }); + } + + serde_json::from_str(&body).map_err(ReplayError::InvalidResponse) + } + + #[cfg(target_arch = "wasm32")] + { + let headers = Headers::new(); + headers + .set("content-type", "application/json") + .map_err(ReplayError::RequestBuild)?; + headers + .set( + "authorization", + &format!("Bearer {}", self.config.auth_token), + ) + .map_err(ReplayError::RequestBuild)?; + + let body = + serde_json::to_string(&json!({ "query": sql })).map_err(ReplayError::Serialize)?; + let mut init = RequestInit::new(); + init.with_method(Method::Post); + init.with_headers(headers); + init.with_body(Some(JsValue::from_str(&body))); + + let request = Request::new_with_init(self.config.endpoint.as_str(), &init) + .map_err(ReplayError::RequestBuild)?; + let controller = AbortController::default(); + let signal = controller.signal(); + spawn_local(async move { + Delay::from(REPLAY_SQL_TIMEOUT).await; + controller.abort(); + }); + + let mut response = Fetch::Request(request) + .send_with_signal(&signal) + .await + .map_err(ReplayError::Transport)?; + + let status = StatusCode::from_u16(response.status_code()) + .unwrap_or(StatusCode::INTERNAL_SERVER_ERROR); + let body = response.text().await.unwrap_or_default(); + + if !status.is_success() { + return Err(ReplayError::UnexpectedResponse { status, body }); + } + + serde_json::from_str(&body).map_err(ReplayError::InvalidResponse) + } + } +} + +#[derive(Debug, Error)] +pub enum ReplayError { + #[error("session replay is not configured")] + NotConfigured, + #[error("failed to create replay HTTP client: {0}")] + #[cfg(not(target_arch = "wasm32"))] + ClientBuild(#[source] reqwest::Error), + #[error("failed to query replay rows: {0}")] + #[cfg(not(target_arch = "wasm32"))] + Transport(#[source] reqwest::Error), + #[error("failed to query replay rows: {0}")] + #[cfg(target_arch = "wasm32")] + Transport(#[source] worker::Error), + #[error("failed to build replay request: {0}")] + #[cfg(target_arch = "wasm32")] + RequestBuild(#[source] worker::Error), + #[error("failed to serialize replay request: {0}")] + #[cfg(target_arch = "wasm32")] + Serialize(#[source] serde_json::Error), + #[error("replay query responded with {status}: {body}")] + UnexpectedResponse { status: StatusCode, body: String }, + #[error("invalid replay query response: {0}")] + InvalidResponse(#[source] serde_json::Error), + #[error("replay query response did not include rows")] + MissingRows, + #[error("invalid replay row: {0}")] + InvalidRow(String), + #[error("session `{0}` was not found")] + SessionNotFound(String), +} + +#[derive(Debug, Deserialize)] +pub struct ReplaySessionsQuery { + #[serde(default)] + pub api_key: Option, + #[serde(default)] + pub distinct_id: Option, + #[serde(default)] + pub session_id: Option, + #[serde(default)] + pub url: Option, + #[serde(default)] + pub event_name: Option, + #[serde(default)] + pub date_from: Option, + #[serde(default)] + pub date_to: Option, + #[serde(default)] + pub min_duration_secs: Option, + #[serde(default)] + pub max_duration_secs: Option, + #[serde(default)] + pub min_events: Option, + #[serde(default)] + pub max_events: Option, + #[serde(default)] + pub limit: Option, +} + +impl ReplaySessionsQuery { + fn limit_or_default(&self) -> usize { + self.limit + .unwrap_or(250) + .clamp(1, DEFAULT_REPLAY_QUERY_LIMIT) + } +} + +#[derive(Debug, Deserialize)] +pub struct ReplaySessionEventsQuery { + #[serde(default)] + pub api_key: Option, + #[serde(default)] + pub distinct_id: Option, + #[serde(default)] + pub at_ms: Option, + #[serde(default)] + pub limit: Option, +} + +#[derive(Debug, Deserialize)] +pub struct ReplayEventsQuery { + #[serde(default)] + pub api_key: Option, + #[serde(default)] + pub distinct_id: Option, + #[serde(default)] + pub session_id: Option, + #[serde(default)] + pub event_name: Option, + #[serde(default)] + pub url: Option, + #[serde(default)] + pub date_from: Option, + #[serde(default)] + pub date_to: Option, + #[serde(default)] + pub limit: Option, +} + +impl ReplayEventsQuery { + fn limit_or_default(&self) -> usize { + self.limit + .unwrap_or(100) + .clamp(1, DEFAULT_REPLAY_QUERY_LIMIT) + } +} + +#[derive(Debug, Deserialize)] +pub struct ReplayFunnelQuery { + #[serde(default)] + pub api_key: Option, + #[serde(default)] + pub distinct_id: Option, + #[serde(default)] + pub session_id: Option, + #[serde(default)] + pub url: Option, + #[serde(default)] + pub steps: Option, + #[serde(default)] + pub date_from: Option, + #[serde(default)] + pub date_to: Option, + #[serde(default)] + pub limit: Option, +} + +impl ReplayFunnelQuery { + fn limit_or_default(&self) -> usize { + self.limit + .unwrap_or(100) + .clamp(1, DEFAULT_REPLAY_QUERY_LIMIT) + } + + fn steps_vec(&self) -> Vec { + split_steps(self.steps.as_deref()) + } + + fn to_events_query(&self) -> ReplayEventsQuery { + ReplayEventsQuery { + api_key: self.api_key.clone(), + distinct_id: self.distinct_id.clone(), + session_id: self.session_id.clone(), + event_name: None, + url: self.url.clone(), + date_from: self.date_from.clone(), + date_to: self.date_to.clone(), + limit: Some(self.limit_or_default()), + } + } +} + +#[derive(Debug, Deserialize)] +pub struct ReplayFrictionQuery { + #[serde(default)] + pub api_key: Option, + #[serde(default)] + pub distinct_id: Option, + #[serde(default)] + pub session_id: Option, + #[serde(default)] + pub url: Option, + #[serde(default)] + pub signal: Option, + #[serde(default)] + pub date_from: Option, + #[serde(default)] + pub date_to: Option, + #[serde(default)] + pub min_duration_secs: Option, + #[serde(default)] + pub max_duration_secs: Option, + #[serde(default)] + pub min_events: Option, + #[serde(default)] + pub max_events: Option, + #[serde(default)] + pub limit: Option, +} + +impl ReplayFrictionQuery { + fn limit_or_default(&self) -> usize { + self.limit + .unwrap_or(100) + .clamp(1, DEFAULT_REPLAY_QUERY_LIMIT) + } + + fn to_sessions_query(&self) -> ReplaySessionsQuery { + ReplaySessionsQuery { + api_key: self.api_key.clone(), + distinct_id: self.distinct_id.clone(), + session_id: self.session_id.clone(), + url: self.url.clone(), + event_name: None, + date_from: self.date_from.clone(), + date_to: self.date_to.clone(), + min_duration_secs: self.min_duration_secs, + max_duration_secs: self.max_duration_secs, + min_events: self.min_events, + max_events: self.max_events, + limit: Some(self.limit_or_default()), + } + } +} + +#[derive(Debug, Deserialize)] +pub struct ReplayPersonQuery { + #[serde(default)] + pub api_key: Option, + #[serde(default)] + pub distinct_id: Option, + #[serde(default)] + pub session_id: Option, + #[serde(default)] + pub url: Option, + #[serde(default)] + pub event_name: Option, + #[serde(default)] + pub date_from: Option, + #[serde(default)] + pub date_to: Option, + #[serde(default)] + pub limit: Option, +} + +impl ReplayPersonQuery { + fn limit_or_default(&self) -> usize { + self.limit + .unwrap_or(100) + .clamp(1, DEFAULT_REPLAY_QUERY_LIMIT) + } + + fn to_sessions_query(&self) -> ReplaySessionsQuery { + ReplaySessionsQuery { + api_key: self.api_key.clone(), + distinct_id: self.distinct_id.clone(), + session_id: self.session_id.clone(), + url: self.url.clone(), + event_name: self.event_name.clone(), + date_from: self.date_from.clone(), + date_to: self.date_to.clone(), + min_duration_secs: None, + max_duration_secs: None, + min_events: None, + max_events: None, + limit: Some(self.limit_or_default()), + } + } + + fn to_events_query(&self) -> ReplayEventsQuery { + ReplayEventsQuery { + api_key: self.api_key.clone(), + distinct_id: self.distinct_id.clone(), + session_id: self.session_id.clone(), + event_name: self.event_name.clone(), + url: self.url.clone(), + date_from: self.date_from.clone(), + date_to: self.date_to.clone(), + limit: Some(self.limit_or_default()), + } + } +} + +#[derive(Debug, Serialize, Clone, PartialEq)] +pub struct ReplaySessionSummary { + pub session_id: String, + pub distinct_id: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub api_key: Option, + pub first_seen: DateTime, + pub last_seen: DateTime, + pub duration_ms: i64, + pub chunk_count: usize, + pub event_count: usize, + #[serde(skip_serializing_if = "Option::is_none")] + pub first_url: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub last_url: Option, +} + +#[derive(Debug, Serialize)] +pub struct ReplaySessionsResponse { + pub sessions: Vec, +} + +#[derive(Debug, Serialize)] +pub struct ReplaySessionEventsResponse { + pub session: ReplaySessionSummary, + pub events: Vec, + pub activity: Vec, + pub chunks: Vec, + #[serde(skip_serializing_if = "Option::is_none")] + pub replay_start_ms: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub replay_end_ms: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub replay_anchor_ms: Option, +} + +#[derive(Debug, Serialize, Clone, PartialEq)] +pub struct ReplayChunkSummary { + pub uuid: String, + pub created_at: DateTime, + pub event_count: usize, + pub source_shape: String, +} + +#[derive(Debug, Serialize, Clone, PartialEq)] +pub struct ReplayActivityItem { + pub id: String, + pub timestamp: i64, + pub offset_ms: i64, + pub kind: String, + pub label: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub detail: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub url: Option, + pub replay_anchor_ms: i64, +} + +#[derive(Debug, Serialize)] +pub struct ReplayEventsResponse { + pub events: Vec, +} + +#[derive(Debug, Serialize)] +pub struct ReplayFunnelResponse { + pub steps: Vec, + pub sessions: Vec, +} + +#[derive(Debug, Serialize, Clone, PartialEq)] +pub struct ReplayFunnelSession { + pub distinct_id: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub api_key: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub session_id: Option, + pub status: String, + pub completed_steps: usize, + pub step_count: usize, + #[serde(skip_serializing_if = "Option::is_none")] + pub current_step: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub drop_off_step: Option, + pub first_seen: DateTime, + pub last_seen: DateTime, + #[serde(skip_serializing_if = "Option::is_none")] + pub url: Option, + pub replay_anchor_ms: i64, + pub steps: Vec, +} + +#[derive(Debug, Serialize, Clone, PartialEq)] +pub struct ReplayFunnelStepEvent { + pub step_index: usize, + pub event: String, + pub created_at: DateTime, + #[serde(skip_serializing_if = "Option::is_none")] + pub session_id: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub url: Option, +} + +#[derive(Debug, Serialize)] +pub struct ReplayFrictionResponse { + pub sessions: Vec, +} + +#[derive(Debug, Serialize, Clone, PartialEq)] +pub struct ReplayFrictionSession { + pub session: ReplaySessionSummary, + pub score: usize, + pub signals: Vec, +} + +#[derive(Debug, Serialize, Clone, PartialEq)] +pub struct ReplayFrictionSignal { + pub kind: String, + pub label: String, + pub detail: String, + pub severity: String, + pub count: usize, + pub timestamp: i64, + pub replay_anchor_ms: i64, + #[serde(skip_serializing_if = "Option::is_none")] + pub url: Option, +} + +#[derive(Debug, Serialize)] +pub struct ReplayPersonJourneyResponse { + #[serde(skip_serializing_if = "Option::is_none")] + pub distinct_id: Option, + pub sessions: Vec, + pub events: Vec, + pub timeline: Vec, +} + +#[derive(Debug, Serialize, Clone, PartialEq)] +pub struct ReplayPersonTimelineItem { + pub kind: String, + pub title: String, + pub timestamp: DateTime, + #[serde(skip_serializing_if = "Option::is_none")] + pub session_id: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub url: Option, + pub replay_anchor_ms: i64, + pub detail: String, +} + +#[derive(Debug, Serialize, Clone, PartialEq)] +pub struct ReplayAnalyticsEvent { + pub uuid: String, + pub event: String, + pub distinct_id: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub api_key: Option, + pub created_at: DateTime, + #[serde(skip_serializing_if = "Option::is_none")] + pub session_id: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub url: Option, + pub replay_anchor_ms: i64, + pub properties: Vec, +} + +#[derive(Debug, Serialize, Clone, PartialEq)] +pub struct ReplayEventProperty { + pub key: String, + pub value: String, +} + +#[derive(Debug, Clone, PartialEq)] +pub struct ReplaySnapshotRow { + pub uuid: String, + pub event: Option, + pub distinct_id: String, + pub created_at: DateTime, + pub api_key: Option, + pub properties: Value, +} + +impl TryFrom for ReplaySnapshotRow { + type Error = ReplayError; + + fn try_from(value: Value) -> Result { + let object = value + .as_object() + .ok_or_else(|| ReplayError::InvalidRow("expected row object".to_string()))?; + + let uuid = string_field(object, "uuid") + .ok_or_else(|| ReplayError::InvalidRow("missing uuid".to_string()))?; + let event = string_field(object, "event"); + let distinct_id = string_field(object, "distinct_id") + .ok_or_else(|| ReplayError::InvalidRow("missing distinct_id".to_string()))?; + let created_at_raw = string_field(object, "created_at") + .ok_or_else(|| ReplayError::InvalidRow("missing created_at".to_string()))?; + let created_at = parse_datetime(&created_at_raw)?; + let api_key = string_field(object, "api_key"); + let properties = object + .get("properties") + .map(parse_jsonish_value) + .transpose()? + .unwrap_or(Value::Null); + + Ok(Self { + uuid, + event, + distinct_id, + created_at, + api_key, + properties, + }) + } +} + +#[derive(Debug, Clone, PartialEq)] +pub struct ReplayEventRow { + pub uuid: String, + pub event: String, + pub distinct_id: String, + pub created_at: DateTime, + pub api_key: Option, + pub properties: Value, +} + +impl TryFrom for ReplayEventRow { + type Error = ReplayError; + + fn try_from(value: Value) -> Result { + let object = value + .as_object() + .ok_or_else(|| ReplayError::InvalidRow("expected row object".to_string()))?; + + let uuid = string_field(object, "uuid") + .ok_or_else(|| ReplayError::InvalidRow("missing uuid".to_string()))?; + let event = string_field(object, "event") + .ok_or_else(|| ReplayError::InvalidRow("missing event".to_string()))?; + let distinct_id = string_field(object, "distinct_id") + .ok_or_else(|| ReplayError::InvalidRow("missing distinct_id".to_string()))?; + let created_at_raw = string_field(object, "created_at") + .ok_or_else(|| ReplayError::InvalidRow("missing created_at".to_string()))?; + let created_at = parse_datetime(&created_at_raw)?; + let api_key = string_field(object, "api_key"); + let properties = object + .get("properties") + .map(parse_jsonish_value) + .transpose()? + .unwrap_or(Value::Null); + + Ok(Self { + uuid, + event, + distinct_id, + created_at, + api_key, + properties, + }) + } +} + +#[derive(Debug)] +struct ExtractedReplayEvents { + events: Vec, + source_shape: String, +} + +pub fn build_snapshot_rows_sql(table: &str, query: &ReplaySessionsQuery, limit: usize) -> String { + let mut clauses = vec![format!( + "event in ({}, {})", + sql_string_literal(SNAPSHOT_EVENT), + sql_string_literal(SNAPSHOT_ITEMS_EVENT) + )]; + if let Some(api_key) = query.api_key.as_ref().filter(|value| !value.is_empty()) { + clauses.push(format!("api_key = {}", sql_string_literal(api_key))); + } + if let Some(distinct_id) = query.distinct_id.as_ref().filter(|value| !value.is_empty()) { + clauses.push(format!("distinct_id = {}", sql_string_literal(distinct_id))); + } + push_date_clauses( + &mut clauses, + query.date_from.as_deref(), + query.date_to.as_deref(), + ); + + format!( + "select uuid, event, distinct_id, created_at, properties, api_key from {table} where {} order by created_at desc limit {limit}", + clauses.join(" and ") + ) +} + +pub fn build_event_rows_sql(table: &str, query: &ReplayEventsQuery, limit: usize) -> String { + let mut clauses = vec![format!( + "event not in ({}, {})", + sql_string_literal(SNAPSHOT_EVENT), + sql_string_literal(SNAPSHOT_ITEMS_EVENT) + )]; + if let Some(api_key) = query.api_key.as_ref().filter(|value| !value.is_empty()) { + clauses.push(format!("api_key = {}", sql_string_literal(api_key))); + } + if let Some(distinct_id) = query.distinct_id.as_ref().filter(|value| !value.is_empty()) { + clauses.push(format!("distinct_id = {}", sql_string_literal(distinct_id))); + } + if let Some(event_name) = query.event_name.as_ref().filter(|value| !value.is_empty()) { + clauses.push(format!("event = {}", sql_string_literal(event_name))); + } + push_date_clauses( + &mut clauses, + query.date_from.as_deref(), + query.date_to.as_deref(), + ); + + format!( + "select uuid, event, distinct_id, created_at, properties, api_key from {table} where {} order by created_at desc limit {limit}", + clauses.join(" and ") + ) +} + +pub fn build_sessions_response( + rows: Vec, + limit: usize, +) -> ReplaySessionsResponse { + let mut sessions: HashMap = HashMap::new(); + + for row in rows { + let session_id = session_id_for_row(&row); + let extracted = extract_rrweb_events(&row.properties); + let stats = extracted + .as_ref() + .map(|value| replay_event_stats(&value.events)) + .unwrap_or_default(); + + let entry = sessions + .entry(session_id.clone()) + .or_insert_with(|| SessionAccumulator::new(session_id, row.clone())); + entry.add(row, stats); + } + + let mut summaries: Vec = sessions + .into_values() + .map(SessionAccumulator::finish) + .collect(); + summaries.sort_by(|a, b| b.last_seen.cmp(&a.last_seen)); + summaries.truncate(limit); + + ReplaySessionsResponse { + sessions: summaries, + } +} + +pub fn build_session_events_response( + session_id: &str, + rows: Vec, + replay_anchor_ms: Option, +) -> Result { + let mut events = Vec::new(); + let mut chunks = Vec::new(); + let mut matched_rows = Vec::new(); + + for row in rows { + if session_id_for_row(&row) != session_id { + continue; + } + let extracted = extract_rrweb_events(&row.properties).unwrap_or(ExtractedReplayEvents { + events: Vec::new(), + source_shape: "unrecognized".to_string(), + }); + chunks.push(ReplayChunkSummary { + uuid: row.uuid.clone(), + created_at: row.created_at, + event_count: extracted.events.len(), + source_shape: extracted.source_shape, + }); + events.extend(extracted.events); + matched_rows.push(row); + } + + if matched_rows.is_empty() { + return Err(ReplayError::SessionNotFound(session_id.to_string())); + } + + events.sort_by_key(rrweb_timestamp); + chunks.sort_by_key(|chunk| chunk.created_at); + let activity = build_activity(&events); + let replay_start_ms = events.iter().filter_map(rrweb_timestamp_opt).min(); + let replay_end_ms = events.iter().filter_map(rrweb_timestamp_opt).max(); + let replay_anchor_ms = replay_anchor_ms.or(Some(0)); + + let session = build_sessions_response(matched_rows, 1) + .sessions + .into_iter() + .next() + .ok_or_else(|| ReplayError::SessionNotFound(session_id.to_string()))?; + + Ok(ReplaySessionEventsResponse { + session, + events, + activity, + chunks, + replay_start_ms, + replay_end_ms, + replay_anchor_ms, + }) +} + +pub fn build_events_response(rows: Vec, limit: usize) -> ReplayEventsResponse { + let mut events = rows + .into_iter() + .map(ReplayAnalyticsEvent::from) + .collect::>(); + events.sort_by(|a, b| b.created_at.cmp(&a.created_at)); + events.truncate(limit); + ReplayEventsResponse { events } +} + +pub fn build_funnel_response( + rows: Vec, + steps: Vec, + limit: usize, +) -> ReplayFunnelResponse { + if steps.is_empty() { + return ReplayFunnelResponse { + steps, + sessions: Vec::new(), + }; + } + + let mut grouped: HashMap> = HashMap::new(); + for row in rows { + let event = ReplayAnalyticsEvent::from(row); + let key = event + .session_id + .clone() + .unwrap_or_else(|| format!("person:{}", event.distinct_id)); + grouped.entry(key).or_default().push(event); + } + + let mut sessions = grouped + .into_values() + .filter_map(|mut events| { + events.sort_by(|a, b| a.created_at.cmp(&b.created_at)); + funnel_session_from_events(&events, &steps) + }) + .collect::>(); + sessions.sort_by(|a, b| b.last_seen.cmp(&a.last_seen)); + sessions.truncate(limit); + + ReplayFunnelResponse { steps, sessions } +} + +pub fn build_friction_response( + rows: Vec, + signal_filter: Option<&str>, + limit: usize, +) -> ReplayFrictionResponse { + let mut grouped: HashMap> = HashMap::new(); + for row in rows { + grouped + .entry(session_id_for_row(&row)) + .or_default() + .push(row); + } + + let normalized_signal_filter = signal_filter + .filter(|value| !value.is_empty()) + .map(|value| value.to_ascii_lowercase()); + let mut sessions = Vec::new(); + for (session_id, rows) in grouped { + let Ok(response) = build_session_events_response(&session_id, rows, None) else { + continue; + }; + let mut signals = detect_friction_signals(&response.events); + if let Some(filter) = normalized_signal_filter.as_deref() { + signals.retain(|signal| signal.kind == filter); + } + if signals.is_empty() { + continue; + } + let score = signals.iter().map(friction_signal_weight).sum(); + sessions.push(ReplayFrictionSession { + session: response.session, + score, + signals, + }); + } + + sessions.sort_by(|a, b| { + b.score + .cmp(&a.score) + .then_with(|| b.session.last_seen.cmp(&a.session.last_seen)) + }); + sessions.truncate(limit); + ReplayFrictionResponse { sessions } +} + +pub fn build_person_journey_response( + distinct_id: Option, + sessions: Vec, + events: Vec, + limit: usize, +) -> ReplayPersonJourneyResponse { + let mut timeline = Vec::new(); + + for session in &sessions { + timeline.push(ReplayPersonTimelineItem { + kind: "session".to_string(), + title: "Replay session".to_string(), + timestamp: session.first_seen, + session_id: Some(session.session_id.clone()), + url: session + .first_url + .clone() + .or_else(|| session.last_url.clone()), + replay_anchor_ms: 0, + detail: format!( + "{} rrweb events, {} chunks, {}ms", + session.event_count, session.chunk_count, session.duration_ms + ), + }); + } + + for event in &events { + timeline.push(ReplayPersonTimelineItem { + kind: "event".to_string(), + title: event.event.clone(), + timestamp: event.created_at, + session_id: event.session_id.clone(), + url: event.url.clone(), + replay_anchor_ms: event.replay_anchor_ms, + detail: event_property_line(event), + }); + } + + timeline.sort_by(|a, b| b.timestamp.cmp(&a.timestamp)); + timeline.truncate(limit); + + ReplayPersonJourneyResponse { + distinct_id, + sessions, + events, + timeline, + } +} + +pub fn rows_from_r2_sql_response(value: Value) -> Result, ReplayError> { + if let Value::Array(rows) = value { + return Ok(rows); + } + + let candidates = [ + "/rows", + "/data", + "/result/rows", + "/result/data", + "/result/results", + "/result/0/results", + "/result/0/rows", + "/result/0/data", + ]; + + for pointer in candidates { + if let Some(Value::Array(rows)) = value.pointer(pointer) { + return Ok(rows.clone()); + } + } + + Err(ReplayError::MissingRows) +} + +pub fn replay_ui_html() -> &'static str { + include_str!("replay_ui.html") +} + +impl From for ReplayAnalyticsEvent { + fn from(row: ReplayEventRow) -> Self { + let session_id = session_id_from_value(&row.properties); + let url = event_url_from_properties(&row.properties); + let properties = summarize_event_properties(&row.properties); + + Self { + uuid: row.uuid, + event: row.event, + distinct_id: row.distinct_id, + api_key: row.api_key, + created_at: row.created_at, + session_id, + url, + replay_anchor_ms: row.created_at.timestamp_millis(), + properties, + } + } +} + +fn funnel_session_from_events( + events: &[ReplayAnalyticsEvent], + steps: &[String], +) -> Option { + let first = events.first()?; + let last = events.last()?; + let mut next_step = 0; + let mut repeated_current_step = 0; + let mut matched_steps = Vec::new(); + + for event in events { + if next_step < steps.len() && event.event == steps[next_step] { + matched_steps.push(ReplayFunnelStepEvent { + step_index: next_step + 1, + event: event.event.clone(), + created_at: event.created_at, + session_id: event.session_id.clone(), + url: event.url.clone(), + }); + next_step += 1; + continue; + } + + if next_step > 0 && next_step < steps.len() && event.event == steps[next_step - 1] { + repeated_current_step += 1; + } + } + + if next_step == 0 { + return None; + } + + let anchor = matched_steps + .last() + .map(|event| event.created_at.timestamp_millis()) + .unwrap_or_else(|| first.created_at.timestamp_millis()); + let status = if next_step == steps.len() { + "converted" + } else if repeated_current_step > 0 { + "stuck" + } else { + "dropped" + }; + + Some(ReplayFunnelSession { + distinct_id: first.distinct_id.clone(), + api_key: first.api_key.clone(), + session_id: first.session_id.clone(), + status: status.to_string(), + completed_steps: next_step, + step_count: steps.len(), + current_step: matched_steps.last().map(|event| event.event.clone()), + drop_off_step: steps.get(next_step).cloned(), + first_seen: first.created_at, + last_seen: last.created_at, + url: matched_steps + .last() + .and_then(|event| event.url.clone()) + .or_else(|| events.iter().find_map(|event| event.url.clone())), + replay_anchor_ms: anchor, + steps: matched_steps, + }) +} + +fn detect_friction_signals(events: &[Value]) -> Vec { + let start = events + .iter() + .filter_map(rrweb_timestamp_opt) + .min() + .unwrap_or(0); + let clicks = replay_clicks(events, start); + let mut signals = Vec::new(); + + if let Some(signal) = detect_rage_click(&clicks, start) { + signals.push(signal); + } + if let Some(signal) = detect_dead_clicks(events, &clicks, start) { + signals.push(signal); + } + if let Some(signal) = detect_repeated_navigation(events, start) { + signals.push(signal); + } + if let Some(signal) = detect_long_idle(events, start) { + signals.push(signal); + } + if let Some(signal) = detect_form_thrash(events, start) { + signals.push(signal); + } + if let Some(signal) = detect_missed_cta(events, &clicks, start) { + signals.push(signal); + } + + signals.sort_by_key(|signal| signal.timestamp); + signals +} + +#[derive(Debug, Clone)] +struct ReplayClick { + timestamp: i64, + x: Option, + y: Option, + url: Option, +} + +#[derive(Debug, Clone)] +struct ReplayScroll { + timestamp: i64, + y: i64, + url: Option, +} + +fn replay_clicks(events: &[Value], start: i64) -> Vec { + events + .iter() + .filter(|event| rrweb_source(event) == Some(2)) + .filter(|event| { + matches!( + event.pointer("/data/type").and_then(Value::as_i64), + Some(2 | 4) + ) + }) + .filter_map(|event| { + let timestamp = rrweb_timestamp_opt(event)?; + Some(ReplayClick { + timestamp, + x: event.pointer("/data/x").and_then(Value::as_i64), + y: event.pointer("/data/y").and_then(Value::as_i64), + url: url_from_rrweb_event(event).or_else(|| current_url_at(events, timestamp)), + }) + }) + .filter(|click| click.timestamp >= start) + .collect() +} + +fn detect_rage_click(clicks: &[ReplayClick], start: i64) -> Option { + for (index, first) in clicks.iter().enumerate() { + let cluster = clicks + .iter() + .skip(index) + .take_while(|click| click.timestamp.saturating_sub(first.timestamp) <= 2_000) + .filter(|click| clicks_are_near(first, click, 25)) + .count(); + if cluster >= 3 { + return Some(friction_signal( + "rage_click", + "Rage clicks", + format!("{cluster} clicks inside 2s near the same point"), + "high", + cluster, + first.timestamp, + start, + first.url.clone(), + )); + } + } + None +} + +fn detect_dead_clicks( + events: &[Value], + clicks: &[ReplayClick], + start: i64, +) -> Option { + let dead = clicks + .iter() + .filter(|click| { + !events.iter().any(|event| { + let timestamp = rrweb_timestamp(event); + timestamp > click.timestamp + && timestamp.saturating_sub(click.timestamp) <= 1_500 + && is_click_follow_up(event) + }) + }) + .collect::>(); + let first = dead.first()?; + + Some(friction_signal( + "dead_click", + "Dead clicks", + format!( + "{} {} had no DOM, navigation, or input follow-up", + dead.len(), + if dead.len() == 1 { "click" } else { "clicks" } + ), + if dead.len() >= 3 { "high" } else { "medium" }, + dead.len(), + first.timestamp, + start, + first.url.clone(), + )) +} + +fn detect_repeated_navigation(events: &[Value], start: i64) -> Option { + let navigations = events + .iter() + .filter(|event| event.get("type").and_then(Value::as_i64) == Some(4)) + .filter_map(|event| { + let timestamp = rrweb_timestamp_opt(event)?; + let url = url_from_rrweb_event(event)?; + Some((timestamp, url)) + }) + .collect::>(); + + let mut counts: HashMap = HashMap::new(); + for (_, url) in &navigations { + *counts.entry(url.clone()).or_default() += 1; + } + if let Some((url, count)) = counts.into_iter().find(|(_, count)| *count >= 3) { + let timestamp = navigations + .iter() + .find(|(_, candidate)| candidate == &url) + .map(|(timestamp, _)| *timestamp) + .unwrap_or(start); + return Some(friction_signal( + "repeated_navigation", + "Repeated navigation", + format!("{count} visits to the same URL"), + "medium", + count, + timestamp, + start, + Some(url), + )); + } + + for window in navigations.windows(3) { + if window[0].1 == window[2].1 && window[0].1 != window[1].1 { + return Some(friction_signal( + "repeated_navigation", + "Back and forth navigation", + "Returned to the same URL after one intervening page".to_string(), + "medium", + 3, + window[0].0, + start, + Some(window[0].1.clone()), + )); + } + } + + None +} + +fn detect_long_idle(events: &[Value], start: i64) -> Option { + let mut timestamps = events + .iter() + .filter_map(rrweb_timestamp_opt) + .collect::>(); + timestamps.sort_unstable(); + let (before, gap) = timestamps + .windows(2) + .map(|window| (window[0], window[1].saturating_sub(window[0]))) + .max_by_key(|(_, gap)| *gap)?; + + if gap < 30_000 { + return None; + } + + Some(friction_signal( + "long_idle", + "Long idle", + format!("{}s gap before the next recorded action", gap / 1_000), + "medium", + 1, + before, + start, + current_url_at(events, before), + )) +} + +fn detect_form_thrash(events: &[Value], start: i64) -> Option { + let inputs = events + .iter() + .filter(|event| rrweb_source(event) == Some(5)) + .filter_map(rrweb_timestamp_opt) + .collect::>(); + + for (index, first) in inputs.iter().enumerate() { + let count = inputs + .iter() + .skip(index) + .take_while(|timestamp| timestamp.saturating_sub(*first) <= 10_000) + .count(); + if count >= 3 { + return Some(friction_signal( + "form_thrash", + "Form thrash", + format!("{count} input changes inside 10s"), + "high", + count, + *first, + start, + current_url_at(events, *first), + )); + } + } + + None +} + +fn detect_missed_cta( + events: &[Value], + clicks: &[ReplayClick], + start: i64, +) -> Option { + let deepest = events + .iter() + .filter(|event| rrweb_source(event) == Some(3)) + .filter_map(|event| { + let timestamp = rrweb_timestamp_opt(event)?; + let y = event.pointer("/data/y").and_then(Value::as_i64)?; + Some(ReplayScroll { + timestamp, + y, + url: current_url_at(events, timestamp), + }) + }) + .max_by_key(|scroll| scroll.y)?; + + if deepest.y < 1_200 + || clicks + .iter() + .any(|click| click.timestamp > deepest.timestamp) + { + return None; + } + + Some(friction_signal( + "missed_cta", + "Deep scroll without follow-up", + format!("Scrolled to y={} with no later click", deepest.y), + "low", + 1, + deepest.timestamp, + start, + deepest.url, + )) +} + +fn friction_signal( + kind: &str, + label: &str, + detail: String, + severity: &str, + count: usize, + timestamp: i64, + start: i64, + url: Option, +) -> ReplayFrictionSignal { + ReplayFrictionSignal { + kind: kind.to_string(), + label: label.to_string(), + detail, + severity: severity.to_string(), + count, + timestamp, + replay_anchor_ms: timestamp.saturating_sub(start), + url, + } +} + +fn friction_signal_weight(signal: &ReplayFrictionSignal) -> usize { + match signal.kind.as_str() { + "rage_click" => 5, + "form_thrash" => 4, + "repeated_navigation" => 3, + "dead_click" | "long_idle" => 2, + "missed_cta" => 1, + _ => 1, + } +} + +fn clicks_are_near(left: &ReplayClick, right: &ReplayClick, radius: i64) -> bool { + match (left.x, left.y, right.x, right.y) { + (Some(left_x), Some(left_y), Some(right_x), Some(right_y)) => { + left_x.saturating_sub(right_x).abs() <= radius + && left_y.saturating_sub(right_y).abs() <= radius + } + _ => true, + } +} + +fn is_click_follow_up(event: &Value) -> bool { + match event.get("type").and_then(Value::as_i64) { + Some(2 | 4) => true, + Some(3) => matches!(rrweb_source(event), Some(0 | 5)), + _ => false, + } +} + +fn rrweb_source(event: &Value) -> Option { + if event.get("type").and_then(Value::as_i64) != Some(3) { + return None; + } + event.pointer("/data/source").and_then(Value::as_i64) +} + +fn current_url_at(events: &[Value], timestamp: i64) -> Option { + events + .iter() + .filter(|event| rrweb_timestamp(event) <= timestamp) + .filter_map(url_from_rrweb_event) + .last() +} + +fn event_property_line(event: &ReplayAnalyticsEvent) -> String { + if event.properties.is_empty() { + return event + .url + .clone() + .or_else(|| event.session_id.clone()) + .unwrap_or_else(|| event.distinct_id.clone()); + } + + event + .properties + .iter() + .take(3) + .map(|property| format!("{}={}", property.key, property.value)) + .collect::>() + .join(", ") +} + +fn split_steps(value: Option<&str>) -> Vec { + value + .unwrap_or_default() + .split(|character| matches!(character, ',' | '\n' | '>')) + .map(str::trim) + .filter(|step| !step.is_empty()) + .map(ToString::to_string) + .collect() +} + +fn retain_snapshot_rows_for_session(rows: &mut Vec, session_id: Option<&str>) { + if let Some(session_id) = session_id.filter(|value| !value.is_empty()) { + rows.retain(|row| session_id_for_row(row) == session_id); + } +} + +fn apply_session_summary_filters( + sessions: &mut Vec, + query: &ReplaySessionsQuery, +) { + if let Some(url) = query.url.as_ref().filter(|value| !value.is_empty()) { + sessions.retain(|session| { + session + .first_url + .as_ref() + .or(session.last_url.as_ref()) + .map(|candidate| contains_case_insensitive(candidate, url)) + .unwrap_or(false) + }); + } + + if let Some(min_duration_secs) = query.min_duration_secs { + sessions.retain(|session| session.duration_ms >= min_duration_secs.saturating_mul(1_000)); + } + if let Some(max_duration_secs) = query.max_duration_secs { + sessions.retain(|session| session.duration_ms <= max_duration_secs.saturating_mul(1_000)); + } + if let Some(min_events) = query.min_events { + sessions.retain(|session| session.event_count >= min_events); + } + if let Some(max_events) = query.max_events { + sessions.retain(|session| session.event_count <= max_events); + } +} + +#[derive(Debug, Default)] +struct MatchingSessionKeys { + session_ids: HashSet, + distinct_ids: HashSet, +} + +fn matching_session_keys_from_events(rows: Vec) -> MatchingSessionKeys { + let mut keys = MatchingSessionKeys::default(); + for row in rows { + if let Some(session_id) = session_id_from_value(&row.properties) { + keys.session_ids.insert(session_id); + } + keys.distinct_ids.insert(row.distinct_id); + } + keys +} + +fn extract_rrweb_events(value: &Value) -> Option { + let parsed = parse_jsonish_value(value).ok()?; + let mut candidates = Vec::new(); + collect_event_candidates(&parsed, &mut candidates); + + if let Some(events) = candidates.into_iter().find(|events| !events.is_empty()) { + return Some(ExtractedReplayEvents { + events, + source_shape: "json-events".to_string(), + }); + } + + let chunk = parsed + .pointer("/data/chunk") + .or_else(|| parsed.pointer("/chunk")) + .and_then(Value::as_str)?; + let compression = parsed + .pointer("/data/compression") + .or_else(|| parsed.pointer("/compression")) + .and_then(Value::as_str); + let decoded = decode_chunk_payload(chunk, compression)?; + let mut chunk_candidates = Vec::new(); + collect_event_candidates(&decoded, &mut chunk_candidates); + + chunk_candidates + .into_iter() + .find(|events| !events.is_empty()) + .map(|events| ExtractedReplayEvents { + events, + source_shape: "base64-chunk".to_string(), + }) +} + +fn collect_event_candidates(value: &Value, candidates: &mut Vec>) { + if is_rrweb_event(value) { + candidates.push(vec![value.clone()]); + } + + if let Value::Array(items) = value { + if items.iter().all(is_rrweb_event) { + candidates.push(items.clone()); + } + } + + for pointer in [ + "/$snapshot_items", + "/$snapshot_data", + "/events", + "/snapshots", + "/batch", + "/data", + "/data/$snapshot_items", + "/data/$snapshot_data", + "/data/events", + "/data/snapshots", + "/data/batch", + "/payload/events", + ] { + if let Some(candidate) = value.pointer(pointer) { + if let Value::Array(items) = candidate { + if items.iter().all(is_rrweb_event) { + candidates.push(items.clone()); + } + } else if is_rrweb_event(candidate) { + candidates.push(vec![candidate.clone()]); + } + } + } +} + +fn is_replay_snapshot_event(event: &str) -> bool { + matches!(event, SNAPSHOT_EVENT | SNAPSHOT_ITEMS_EVENT) +} + +fn decode_chunk_payload(chunk: &str, compression: Option<&str>) -> Option { + let bytes = BASE64_STANDARD + .decode(chunk) + .ok() + .or_else(|| Some(chunk.as_bytes().to_vec()))?; + let text = if compression + .map(|value| value.to_ascii_lowercase().contains("gzip")) + .unwrap_or(false) + || bytes.starts_with(&[0x1f, 0x8b]) + { + let mut decoder = GzDecoder::new(bytes.as_slice()); + let mut decoded = String::new(); + decoder.read_to_string(&mut decoded).ok()?; + decoded + } else { + String::from_utf8(bytes).ok()? + }; + + serde_json::from_str(&text).ok() +} + +fn is_rrweb_event(value: &Value) -> bool { + value + .as_object() + .map(|object| { + object.get("type").and_then(Value::as_i64).is_some() + && object.get("timestamp").and_then(Value::as_i64).is_some() + }) + .unwrap_or(false) +} + +fn session_id_for_row(row: &ReplaySnapshotRow) -> String { + session_id_from_value(&row.properties).unwrap_or_else(|| row.distinct_id.clone()) +} + +fn session_id_from_value(value: &Value) -> Option { + let parsed = parse_jsonish_value(value).ok()?; + for pointer in [ + "/session_id", + "/sessionId", + "/$session_id", + "/recording_id", + "/session_recording_id", + "/data/session_id", + "/data/sessionId", + "/data/$session_id", + "/data/metadata/session_id", + "/data/metadata/sessionId", + "/data/metadata/$session_id", + ] { + if let Some(session_id) = parsed.pointer(pointer).and_then(Value::as_str) { + if !session_id.is_empty() { + return Some(session_id.to_string()); + } + } + } + None +} + +fn first_url_from_events(events: &[Value]) -> Option { + for event in events { + if let Some(value) = url_from_rrweb_event(event) { + return Some(value); + } + } + None +} + +fn last_url_from_events(events: &[Value]) -> Option { + events.iter().rev().find_map(url_from_rrweb_event) +} + +fn url_from_rrweb_event(event: &Value) -> Option { + for pointer in [ + "/data/href", + "/data/source", + "/data/attributes/href", + "/data/node/root/childNodes/0/attributes/href", + ] { + if let Some(value) = event.pointer(pointer).and_then(Value::as_str) { + if value.starts_with("http://") || value.starts_with("https://") { + return Some(value.to_string()); + } + } + } + None +} + +#[derive(Debug, Default)] +struct ReplayEventStats { + event_count: usize, + first_url: Option, + last_url: Option, + start_ms: Option, + end_ms: Option, +} + +fn replay_event_stats(events: &[Value]) -> ReplayEventStats { + ReplayEventStats { + event_count: events.len(), + first_url: first_url_from_events(events), + last_url: last_url_from_events(events), + start_ms: events.iter().filter_map(rrweb_timestamp_opt).min(), + end_ms: events.iter().filter_map(rrweb_timestamp_opt).max(), + } +} + +fn rrweb_timestamp(value: &Value) -> i64 { + value.get("timestamp").and_then(Value::as_i64).unwrap_or(0) +} + +fn rrweb_timestamp_opt(value: &Value) -> Option { + value.get("timestamp").and_then(Value::as_i64) +} + +fn build_activity(events: &[Value]) -> Vec { + let start = events + .iter() + .filter_map(rrweb_timestamp_opt) + .min() + .unwrap_or(0); + events + .iter() + .enumerate() + .filter_map(|(index, event)| activity_for_event(index, event, start)) + .collect() +} + +fn activity_for_event(index: usize, event: &Value, start: i64) -> Option { + let event_type = event.get("type").and_then(Value::as_i64)?; + let timestamp = rrweb_timestamp_opt(event)?; + let url = url_from_rrweb_event(event); + let (kind, label, detail) = match event_type { + 2 => ("snapshot", "DOM snapshot".to_string(), None), + 4 => ( + "navigation", + "Page context".to_string(), + url.clone().or_else(|| string_pointer(event, "/data/href")), + ), + 5 => ( + "custom", + string_pointer(event, "/data/tag").unwrap_or_else(|| "Custom marker".to_string()), + string_pointer(event, "/data/payload"), + ), + 3 => incremental_activity(event)?, + _ => return None, + }; + + Some(ReplayActivityItem { + id: format!("rrweb-{index}"), + timestamp, + offset_ms: timestamp.saturating_sub(start), + kind: kind.to_string(), + label, + detail, + url, + replay_anchor_ms: timestamp.saturating_sub(start), + }) +} + +fn incremental_activity(event: &Value) -> Option<(&'static str, String, Option)> { + let source = event.pointer("/data/source").and_then(Value::as_i64)?; + match source { + 0 => Some(("dom", "DOM changed".to_string(), None)), + 2 => Some(mouse_activity(event)), + 3 => Some(( + "scroll", + "Scrolled".to_string(), + xy_detail(event.pointer("/data/x"), event.pointer("/data/y")), + )), + 4 => Some(( + "viewport", + "Viewport resized".to_string(), + size_detail(event.pointer("/data/width"), event.pointer("/data/height")), + )), + 5 => Some(("input", "Input changed".to_string(), None)), + 7 => Some(("media", "Media interaction".to_string(), None)), + 11 => Some(("technical", "Browser log".to_string(), None)), + _ => None, + } +} + +fn mouse_activity(event: &Value) -> (&'static str, String, Option) { + let label = match event.pointer("/data/type").and_then(Value::as_i64) { + Some(0) => "Mouse up", + Some(1) => "Mouse down", + Some(2) => "Click", + Some(3) => "Context menu", + Some(4) => "Double click", + Some(5) => "Focus", + Some(6) => "Blur", + Some(7) => "Touch start", + Some(9) => "Touch end", + _ => "Mouse interaction", + }; + + ( + "interaction", + label.to_string(), + xy_detail(event.pointer("/data/x"), event.pointer("/data/y")), + ) +} + +fn xy_detail(x: Option<&Value>, y: Option<&Value>) -> Option { + Some(format!( + "{}, {}", + x.and_then(Value::as_i64)?, + y.and_then(Value::as_i64)? + )) +} + +fn size_detail(width: Option<&Value>, height: Option<&Value>) -> Option { + Some(format!( + "{} x {}", + width.and_then(Value::as_i64)?, + height.and_then(Value::as_i64)? + )) +} + +fn event_url_from_properties(properties: &Value) -> Option { + for pointer in [ + "/$current_url", + "/current_url", + "/url", + "/href", + "/page_url", + "/properties/$current_url", + "/properties/current_url", + "/properties/url", + "/properties/href", + ] { + if let Some(value) = properties.pointer(pointer).and_then(Value::as_str) { + if value.starts_with("http://") + || value.starts_with("https://") + || value.starts_with('/') + { + return Some(value.to_string()); + } + } + } + None +} + +fn summarize_event_properties(properties: &Value) -> Vec { + let Some(object) = properties.as_object() else { + return Vec::new(); + }; + let preferred = [ + "$current_url", + "current_url", + "url", + "path", + "pathname", + "$browser", + "$device_type", + "$os", + ]; + let mut summary = Vec::new(); + let mut seen = HashSet::new(); + + for key in preferred { + if let Some(value) = object.get(key).and_then(display_property_value) { + seen.insert(key.to_string()); + summary.push(ReplayEventProperty { + key: key.to_string(), + value, + }); + } + } + + for (key, value) in object { + if summary.len() >= 6 { + break; + } + if seen.contains(key) || key.starts_with('$') { + continue; + } + if let Some(value) = display_property_value(value) { + summary.push(ReplayEventProperty { + key: key.clone(), + value, + }); + } + } + + summary +} + +fn display_property_value(value: &Value) -> Option { + match value { + Value::String(value) if !value.is_empty() => Some(value.clone()), + Value::Number(value) => Some(value.to_string()), + Value::Bool(value) => Some(value.to_string()), + _ => None, + } +} + +fn string_pointer(value: &Value, pointer: &str) -> Option { + value.pointer(pointer).and_then(|value| match value { + Value::String(value) if !value.is_empty() => Some(value.clone()), + Value::Number(value) => Some(value.to_string()), + Value::Bool(value) => Some(value.to_string()), + _ => None, + }) +} + +fn parse_jsonish_value(value: &Value) -> Result { + match value { + Value::String(raw) => serde_json::from_str(raw) + .map_err(|err| ReplayError::InvalidRow(format!("invalid JSON string: {err}"))), + other => Ok(other.clone()), + } +} + +fn parse_datetime(value: &str) -> Result, ReplayError> { + DateTime::parse_from_rfc3339(value) + .map(|dt| dt.with_timezone(&Utc)) + .or_else(|_| { + DateTime::parse_from_str(value, "%Y-%m-%d %H:%M:%S%.f %z") + .map(|dt| dt.with_timezone(&Utc)) + }) + .map_err(|err| ReplayError::InvalidRow(format!("invalid created_at `{value}`: {err}"))) +} + +fn string_field(object: &Map, key: &str) -> Option { + object.get(key).and_then(|value| match value { + Value::String(value) if !value.is_empty() => Some(value.clone()), + Value::Number(value) => Some(value.to_string()), + _ => None, + }) +} + +fn sql_string_literal(value: &str) -> String { + format!("'{}'", value.replace('\'', "''")) +} + +fn push_date_clauses(clauses: &mut Vec, date_from: Option<&str>, date_to: Option<&str>) { + if let Some(date_from) = date_from.filter(|value| !value.is_empty()) { + clauses.push(format!("created_at >= {}", sql_string_literal(date_from))); + } + if let Some(date_to) = date_to.filter(|value| !value.is_empty()) { + clauses.push(format!("created_at <= {}", sql_string_literal(date_to))); + } +} + +fn contains_case_insensitive(value: &str, needle: &str) -> bool { + value + .to_ascii_lowercase() + .contains(&needle.to_ascii_lowercase()) +} + +fn validate_sql_identifier(value: &str) -> Result<(), ReplayConfigError> { + if value + .split('.') + .all(|part| !part.is_empty() && part.chars().all(|c| c.is_ascii_alphanumeric() || c == '_')) + { + Ok(()) + } else { + Err(ReplayConfigError::InvalidEventsTable { + value: value.to_string(), + }) + } +} + +#[derive(Debug)] +struct SessionAccumulator { + session_id: String, + distinct_id: String, + api_key: Option, + first_seen: DateTime, + last_seen: DateTime, + replay_start_ms: Option, + replay_end_ms: Option, + chunk_count: usize, + event_count: usize, + first_url: Option, + last_url: Option, +} + +impl SessionAccumulator { + fn new(session_id: String, row: ReplaySnapshotRow) -> Self { + Self { + session_id, + distinct_id: row.distinct_id, + api_key: row.api_key, + first_seen: row.created_at, + last_seen: row.created_at, + replay_start_ms: None, + replay_end_ms: None, + chunk_count: 0, + event_count: 0, + first_url: None, + last_url: None, + } + } + + fn add(&mut self, row: ReplaySnapshotRow, stats: ReplayEventStats) { + self.first_seen = self.first_seen.min(row.created_at); + self.last_seen = self.last_seen.max(row.created_at); + self.chunk_count += 1; + self.event_count += stats.event_count; + self.replay_start_ms = min_optional_i64(self.replay_start_ms, stats.start_ms); + self.replay_end_ms = max_optional_i64(self.replay_end_ms, stats.end_ms); + if self.api_key.is_none() { + self.api_key = row.api_key; + } + if self.first_url.is_none() { + self.first_url = stats.first_url; + } + if stats.last_url.is_some() { + self.last_url = stats.last_url; + } + } + + fn finish(self) -> ReplaySessionSummary { + let duration_ms = match (self.replay_start_ms, self.replay_end_ms) { + (Some(start), Some(end)) => end.saturating_sub(start), + _ => self + .last_seen + .signed_duration_since(self.first_seen) + .num_milliseconds() + .max(0), + }; + ReplaySessionSummary { + session_id: self.session_id, + distinct_id: self.distinct_id, + api_key: self.api_key, + first_seen: self.first_seen, + last_seen: self.last_seen, + duration_ms, + chunk_count: self.chunk_count, + event_count: self.event_count, + first_url: self.first_url, + last_url: self.last_url, + } + } +} + +fn min_optional_i64(left: Option, right: Option) -> Option { + match (left, right) { + (Some(left), Some(right)) => Some(left.min(right)), + (Some(left), None) => Some(left), + (None, Some(right)) => Some(right), + (None, None) => None, + } +} + +fn max_optional_i64(left: Option, right: Option) -> Option { + match (left, right) { + (Some(left), Some(right)) => Some(left.max(right)), + (Some(left), None) => Some(left), + (None, Some(right)) => Some(right), + (None, None) => None, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use chrono::TimeZone; + + fn row(uuid: &str, session_id: &str, distinct_id: &str, timestamp: i64) -> ReplaySnapshotRow { + ReplaySnapshotRow { + uuid: uuid.to_string(), + event: Some("$snapshot".to_string()), + distinct_id: distinct_id.to_string(), + created_at: Utc.timestamp_millis_opt(timestamp).unwrap(), + api_key: Some("phc_test".to_string()), + properties: json!({ + "session_id": session_id, + "events": [ + { "type": 4, "timestamp": timestamp, "data": { "href": "https://app.test/start" } } + ] + }), + } + } + + fn row_with_events( + uuid: &str, + session_id: &str, + distinct_id: &str, + events: Vec, + ) -> ReplaySnapshotRow { + ReplaySnapshotRow { + uuid: uuid.to_string(), + event: Some("$snapshot".to_string()), + distinct_id: distinct_id.to_string(), + created_at: Utc.timestamp_millis_opt(1_000).unwrap(), + api_key: Some("phc_test".to_string()), + properties: json!({ + "session_id": session_id, + "events": events, + }), + } + } + + fn snapshot_items_row( + uuid: &str, + session_id: &str, + distinct_id: &str, + events: Vec, + ) -> ReplaySnapshotRow { + ReplaySnapshotRow { + uuid: uuid.to_string(), + event: Some("$snapshot_items".to_string()), + distinct_id: distinct_id.to_string(), + created_at: Utc.timestamp_millis_opt(1_000).unwrap(), + api_key: Some("phc_test".to_string()), + properties: json!({ + "$session_id": session_id, + "$snapshot_items": events, + }), + } + } + + fn analytics_row( + uuid: &str, + event: &str, + session_id: &str, + distinct_id: &str, + timestamp: i64, + ) -> ReplayEventRow { + ReplayEventRow { + uuid: uuid.to_string(), + event: event.to_string(), + distinct_id: distinct_id.to_string(), + created_at: Utc.timestamp_millis_opt(timestamp).unwrap(), + api_key: Some("phc_test".to_string()), + properties: json!({ + "$session_id": session_id, + "$current_url": "https://app.test/pricing", + }), + } + } + + #[test] + fn builds_filtered_snapshot_sql_with_escaped_literals() { + let sql = build_snapshot_rows_sql( + "default.hogflare_events", + &ReplaySessionsQuery { + api_key: Some("phc_o'hare".to_string()), + distinct_id: Some("user-1".to_string()), + session_id: None, + url: None, + event_name: None, + date_from: None, + date_to: None, + min_duration_secs: None, + max_duration_secs: None, + min_events: None, + max_events: None, + limit: None, + }, + 100, + ); + + assert_eq!( + sql, + "select uuid, event, distinct_id, created_at, properties, api_key from default.hogflare_events where event in ('$snapshot', '$snapshot_items') and api_key = 'phc_o''hare' and distinct_id = 'user-1' order by created_at desc limit 100" + ); + } + + #[test] + fn builds_filtered_event_sql_with_date_range() { + let sql = build_event_rows_sql( + "default.hogflare_events", + &ReplayEventsQuery { + api_key: Some("phc_test".to_string()), + distinct_id: Some("user-1".to_string()), + session_id: None, + event_name: Some("Checkout Started".to_string()), + url: None, + date_from: Some("2026-05-22T00:00:00Z".to_string()), + date_to: Some("2026-05-23T00:00:00Z".to_string()), + limit: None, + }, + 50, + ); + + assert_eq!( + sql, + "select uuid, event, distinct_id, created_at, properties, api_key from default.hogflare_events where event not in ('$snapshot', '$snapshot_items') and api_key = 'phc_test' and distinct_id = 'user-1' and event = 'Checkout Started' and created_at >= '2026-05-22T00:00:00Z' and created_at <= '2026-05-23T00:00:00Z' order by created_at desc limit 50" + ); + } + + #[test] + fn extracts_sessions_from_rows() { + let response = build_sessions_response( + vec![ + row("chunk-2", "session-a", "user-a", 2_000), + row("chunk-1", "session-a", "user-a", 1_000), + row("chunk-3", "session-b", "user-b", 3_000), + ], + 10, + ); + + assert_eq!(response.sessions.len(), 2); + assert_eq!(response.sessions[0].session_id, "session-b"); + assert_eq!(response.sessions[1].session_id, "session-a"); + assert_eq!(response.sessions[1].chunk_count, 2); + assert_eq!(response.sessions[1].event_count, 2); + assert_eq!( + response.sessions[1].first_url.as_deref(), + Some("https://app.test/start") + ); + } + + #[test] + fn applies_session_summary_filters() { + let mut response = build_sessions_response( + vec![ + row("chunk-2", "session-a", "user-a", 2_000), + row("chunk-1", "session-a", "user-a", 1_000), + row("chunk-3", "session-b", "user-b", 3_000), + ], + 10, + ); + + apply_session_summary_filters( + &mut response.sessions, + &ReplaySessionsQuery { + api_key: None, + distinct_id: None, + session_id: None, + url: Some("app.test/start".to_string()), + event_name: None, + date_from: None, + date_to: None, + min_duration_secs: Some(1), + max_duration_secs: Some(1), + min_events: Some(2), + max_events: Some(2), + limit: None, + }, + ); + + assert_eq!(response.sessions.len(), 1); + assert_eq!(response.sessions[0].session_id, "session-a"); + } + + #[test] + fn extracts_sorted_rrweb_events_from_session_chunks() { + let response = build_session_events_response( + "session-a", + vec![ + row("chunk-2", "session-a", "user-a", 2_000), + row("chunk-1", "session-a", "user-a", 1_000), + row("chunk-other", "session-b", "user-b", 500), + ], + None, + ) + .unwrap(); + + assert_eq!(response.session.session_id, "session-a"); + assert_eq!(response.chunks.len(), 2); + assert_eq!(response.events.len(), 2); + assert_eq!(response.activity.len(), 2); + assert_eq!(response.replay_start_ms, Some(1_000)); + assert_eq!(response.replay_end_ms, Some(2_000)); + assert_eq!(response.events[0]["timestamp"], 1_000); + assert_eq!(response.events[1]["timestamp"], 2_000); + } + + #[test] + fn extracts_normalized_snapshot_items_rows() { + let response = build_session_events_response( + "session-normalized", + vec![snapshot_items_row( + "chunk-normalized", + "session-normalized", + "user-normalized", + vec![json!({ "type": 4, "timestamp": 1_500, "data": { "href": "https://app.test/pricing" } })], + )], + None, + ) + .unwrap(); + + assert_eq!(response.session.session_id, "session-normalized"); + assert_eq!(response.session.event_count, 1); + assert_eq!(response.events.len(), 1); + assert_eq!(response.events[0]["timestamp"], 1_500); + assert_eq!( + response.session.first_url.as_deref(), + Some("https://app.test/pricing") + ); + } + + #[test] + fn builds_analytics_event_response_without_raw_properties() { + let response = build_events_response( + vec![ReplayEventRow { + uuid: "event-1".to_string(), + event: "Checkout Started".to_string(), + distinct_id: "user-a".to_string(), + created_at: Utc.timestamp_millis_opt(2_000).unwrap(), + api_key: Some("phc_test".to_string()), + properties: json!({ + "$session_id": "session-a", + "$current_url": "https://app.test/checkout", + "plan": "pro", + "nested": { "ignored": true } + }), + }], + 10, + ); + + assert_eq!(response.events.len(), 1); + assert_eq!(response.events[0].session_id.as_deref(), Some("session-a")); + assert_eq!( + response.events[0].url.as_deref(), + Some("https://app.test/checkout") + ); + assert!(response.events[0] + .properties + .iter() + .any(|property| property.key == "plan" && property.value == "pro")); + assert!(!response.events[0] + .properties + .iter() + .any(|property| property.key == "nested")); + } + + #[test] + fn classifies_funnel_sessions() { + let response = build_funnel_response( + vec![ + analytics_row("a1", "Viewed Pricing", "converted", "user-a", 1_000), + analytics_row("a2", "Checkout Started", "converted", "user-a", 2_000), + analytics_row("a3", "Paid", "converted", "user-a", 3_000), + analytics_row("b1", "Viewed Pricing", "stuck", "user-b", 4_000), + analytics_row("b2", "Viewed Pricing", "stuck", "user-b", 5_000), + analytics_row("c1", "Viewed Pricing", "dropped", "user-c", 6_000), + analytics_row("d1", "Signed Up", "ignored", "user-d", 7_000), + ], + vec![ + "Viewed Pricing".to_string(), + "Checkout Started".to_string(), + "Paid".to_string(), + ], + 10, + ); + + assert_eq!(response.sessions.len(), 3); + let converted = response + .sessions + .iter() + .find(|session| session.session_id.as_deref() == Some("converted")) + .unwrap(); + assert_eq!(converted.status, "converted"); + assert_eq!(converted.completed_steps, 3); + assert_eq!(converted.drop_off_step, None); + + let stuck = response + .sessions + .iter() + .find(|session| session.session_id.as_deref() == Some("stuck")) + .unwrap(); + assert_eq!(stuck.status, "stuck"); + assert_eq!(stuck.completed_steps, 1); + assert_eq!(stuck.drop_off_step.as_deref(), Some("Checkout Started")); + + let dropped = response + .sessions + .iter() + .find(|session| session.session_id.as_deref() == Some("dropped")) + .unwrap(); + assert_eq!(dropped.status, "dropped"); + assert_eq!(dropped.completed_steps, 1); + } + + #[test] + fn computes_friction_signals_from_rrweb_events() { + let response = build_friction_response( + vec![row_with_events( + "chunk-friction", + "session-friction", + "user-friction", + vec![ + json!({ "type": 4, "timestamp": 1_000, "data": { "href": "https://app.test/pricing" } }), + json!({ "type": 3, "timestamp": 1_100, "data": { "source": 2, "type": 2, "x": 42, "y": 52 } }), + json!({ "type": 3, "timestamp": 1_700, "data": { "source": 2, "type": 2, "x": 44, "y": 50 } }), + json!({ "type": 3, "timestamp": 2_200, "data": { "source": 2, "type": 2, "x": 43, "y": 51 } }), + json!({ "type": 3, "timestamp": 3_000, "data": { "source": 5 } }), + json!({ "type": 3, "timestamp": 3_700, "data": { "source": 5 } }), + json!({ "type": 3, "timestamp": 4_200, "data": { "source": 5 } }), + json!({ "type": 3, "timestamp": 40_000, "data": { "source": 3, "x": 0, "y": 1_600 } }), + json!({ "type": 4, "timestamp": 45_000, "data": { "href": "https://app.test/help" } }), + json!({ "type": 4, "timestamp": 46_000, "data": { "href": "https://app.test/pricing" } }), + json!({ "type": 4, "timestamp": 47_000, "data": { "href": "https://app.test/help" } }), + ], + )], + None, + 10, + ); + + assert_eq!(response.sessions.len(), 1); + let kinds = response.sessions[0] + .signals + .iter() + .map(|signal| signal.kind.as_str()) + .collect::>(); + assert!(kinds.contains("rage_click")); + assert!(kinds.contains("form_thrash")); + assert!(kinds.contains("long_idle")); + assert!(kinds.contains("repeated_navigation")); + assert!(kinds.contains("missed_cta")); + assert!(response.sessions[0].score >= 10); + } + + #[test] + fn builds_person_journey_without_raw_event_properties() { + let sessions = + build_sessions_response(vec![row("chunk-1", "session-a", "user-a", 1_000)], 10) + .sessions; + let events = build_events_response( + vec![ReplayEventRow { + uuid: "event-1".to_string(), + event: "Checkout Started".to_string(), + distinct_id: "user-a".to_string(), + created_at: Utc.timestamp_millis_opt(2_000).unwrap(), + api_key: Some("phc_test".to_string()), + properties: json!({ + "$session_id": "session-a", + "$current_url": "https://app.test/checkout", + "plan": "pro", + "raw_object": { "hidden": true } + }), + }], + 10, + ) + .events; + + let response = + build_person_journey_response(Some("user-a".to_string()), sessions, events, 10); + + assert_eq!(response.distinct_id.as_deref(), Some("user-a")); + assert_eq!(response.sessions.len(), 1); + assert_eq!(response.events.len(), 1); + assert_eq!(response.timeline.len(), 2); + assert!(response + .timeline + .iter() + .any(|item| item.kind == "event" && item.detail.contains("plan=pro"))); + assert!(!response + .timeline + .iter() + .any(|item| item.detail.contains("raw_object"))); + } + + #[test] + fn extracts_events_from_base64_gzip_chunk() { + let encoded = { + use flate2::{write::GzEncoder, Compression}; + use std::io::Write; + + let mut encoder = GzEncoder::new(Vec::new(), Compression::default()); + encoder + .write_all(br#"[{"type":4,"timestamp":1000,"data":{}}]"#) + .unwrap(); + BASE64_STANDARD.encode(encoder.finish().unwrap()) + }; + + let extracted = extract_rrweb_events(&json!({ + "data": { + "chunk": encoded, + "compression": "gzip" + } + })) + .unwrap(); + + assert_eq!(extracted.source_shape, "base64-chunk"); + assert_eq!(extracted.events.len(), 1); + assert_eq!(extracted.events[0]["type"], 4); + } + + #[test] + fn reads_rows_from_common_r2_sql_response_shapes() { + let row = json!({ + "uuid": "row-1", + "distinct_id": "user-1", + "created_at": "2026-05-22T10:00:00Z", + "properties": "{}" + }); + + assert_eq!( + rows_from_r2_sql_response(json!({ "result": { "rows": [row.clone()] } })).unwrap(), + vec![row.clone()] + ); + assert_eq!( + rows_from_r2_sql_response(json!({ "data": [row.clone()] })).unwrap(), + vec![row] + ); + } + + #[test] + fn validates_table_identifier() { + assert!(validate_sql_identifier("default.hogflare_events").is_ok()); + assert!(validate_sql_identifier("default.hogflare_events;drop").is_err()); + } +} diff --git a/src/replay_ui.html b/src/replay_ui.html new file mode 100644 index 0000000..4b67e82 --- /dev/null +++ b/src/replay_ui.html @@ -0,0 +1,2235 @@ + + + + + + Hogflare Replay Explorer + + + + + +
+ + +
+
+
+ No session selected + Search replay sessions or choose an analytics event. +
+
+ + +
+
+
+
+
+ Ready + Load sessions, then select a recording to replay. +
+
+
+
+ + +
+ + + + diff --git a/tests/helpers/mod.rs b/tests/helpers/mod.rs index 4749bec..4ef4677 100644 --- a/tests/helpers/mod.rs +++ b/tests/helpers/mod.rs @@ -5,7 +5,7 @@ use std::{net::SocketAddr, sync::Arc, time::Duration}; use axum::{extract::State, http::StatusCode, routing::post, Json, Router}; use hogflare::pipeline::PipelineClient; use reqwest::{Client, Url}; -use serde_json::Value; +use serde_json::{json, Value}; use tokio::{ net::TcpListener, process::Command, @@ -50,6 +50,45 @@ pub async fn spawn_pipeline_stub( Ok((endpoint, receiver, handle)) } +pub async fn spawn_replay_sql_stub( + rows: Vec, +) -> Result<(Url, mpsc::Receiver, JoinHandle<()>), Box> { + let (sender, receiver) = mpsc::channel(16); + + #[derive(Clone)] + struct StubState { + rows: Vec, + sender: mpsc::Sender, + } + + async fn handle_query( + State(state): State, + Json(payload): Json, + ) -> Json { + if let Some(query) = payload.get("query").and_then(Value::as_str) { + let _ = state.sender.send(query.to_string()).await; + } + + Json(json!({ "result": { "rows": state.rows } })) + } + + let app = Router::new() + .route("/", post(handle_query)) + .with_state(StubState { rows, sender }); + + let listener = TcpListener::bind("127.0.0.1:0").await?; + let address = listener.local_addr()?; + let endpoint = Url::parse(&format!("http://{}/", address))?; + + let handle = tokio::spawn(async move { + if let Err(err) = axum::serve(listener, app.into_make_service()).await { + eprintln!("replay sql stub terminated: {err}"); + } + }); + + Ok((endpoint, receiver, handle)) +} + pub async fn spawn_app( pipeline_endpoint: Url, ) -> Result<(SocketAddr, JoinHandle<()>), Box> { @@ -185,6 +224,40 @@ pub async fn spawn_app_with_runtime_options_and_person_pipeline( Ok((address, server_handle)) } +pub async fn spawn_app_with_replay( + pipeline_endpoint: Url, + replay_config: hogflare::replay::ReplayConfig, +) -> Result<(SocketAddr, JoinHandle<()>), Box> { + let pipeline_client = PipelineClient::new(pipeline_endpoint, None, Duration::from_secs(5))?; + let replay_client = hogflare::replay::ReplayClient::new(replay_config)?; + + let listener = TcpListener::bind("127.0.0.1:0").await?; + let address = listener.local_addr()?; + + let server_handle = tokio::spawn(async move { + if let Err(err) = hogflare::serve_with_person_pipeline_and_replay( + listener, + Arc::new(pipeline_client), + None, + Some(Arc::new(replay_client)), + None, + Arc::new(hogflare::groups::MemoryGroupStore::new()), + hogflare::groups::GroupTypeMap::default(), + None, + None, + None, + None, + Arc::new(hogflare::feature_flags::FeatureFlagStore::empty()), + ) + .await + { + eprintln!("hogflare server terminated: {err}"); + } + }); + + Ok((address, server_handle)) +} + pub async fn wait_for_events( receiver: &mut mpsc::Receiver>, ) -> Result, Box> { diff --git a/tests/posthog_endpoints.rs b/tests/posthog_endpoints.rs index ee9819f..71d407b 100644 --- a/tests/posthog_endpoints.rs +++ b/tests/posthog_endpoints.rs @@ -6,11 +6,11 @@ use chrono::Utc; use flate2::{write::GzEncoder, Compression}; use helpers::{ cleanup, spawn_app_with_options, spawn_app_with_options_and_debug, - spawn_app_with_options_debug_and_person_pipeline, spawn_app_with_runtime_options, - spawn_pipeline_stub, wait_for_events, + spawn_app_with_options_debug_and_person_pipeline, spawn_app_with_replay, + spawn_app_with_runtime_options, spawn_pipeline_stub, spawn_replay_sql_stub, wait_for_events, }; use hmac::{Hmac, Mac}; -use hogflare::{feature_flags::FeatureFlagStore, groups::GroupTypeMap}; +use hogflare::{feature_flags::FeatureFlagStore, groups::GroupTypeMap, replay::ReplayConfig}; use reqwest::{Client, StatusCode}; use serde_json::{json, Value}; use sha2::Sha256; @@ -772,6 +772,313 @@ async fn posthog_compatibility_endpoints_forward_events() -> Result<(), Box Result<(), Box> { + let rows = vec![ + json!({ + "uuid": "chunk-2", + "event": "$snapshot_items", + "distinct_id": "replay-user", + "created_at": "2026-05-22T10:00:02Z", + "api_key": "phc_replay", + "properties": { + "$session_id": "session-1", + "$snapshot_items": [ + { "type": 4, "timestamp": 2000, "data": { "href": "https://app.test/page" } } + ] + } + }), + json!({ + "uuid": "chunk-1", + "event": "$snapshot_items", + "distinct_id": "replay-user", + "created_at": "2026-05-22T10:00:01Z", + "api_key": "phc_replay", + "properties": { + "$session_id": "session-1", + "$snapshot_items": [ + { "type": 2, "timestamp": 1000, "data": { "node": { "id": 1 } } } + ] + } + }), + json!({ + "uuid": "chunk-other", + "event": "$snapshot", + "distinct_id": "other-user", + "created_at": "2026-05-22T10:00:03Z", + "api_key": "phc_other", + "properties": { + "session_id": "session-other", + "events": [ + { "type": 4, "timestamp": 3000, "data": {} } + ] + } + }), + json!({ + "uuid": "event-1", + "event": "Checkout Started", + "distinct_id": "replay-user", + "created_at": "2026-05-22T10:00:02Z", + "api_key": "phc_replay", + "properties": { + "$session_id": "session-1", + "$current_url": "https://app.test/page", + "plan": "pro" + } + }), + json!({ + "uuid": "funnel-1", + "event": "Viewed Pricing", + "distinct_id": "funnel-user", + "created_at": "2026-05-22T10:01:00Z", + "api_key": "phc_replay", + "properties": { + "$session_id": "funnel-session", + "$current_url": "https://app.test/pricing" + } + }), + json!({ + "uuid": "funnel-2", + "event": "Checkout Started", + "distinct_id": "funnel-user", + "created_at": "2026-05-22T10:01:10Z", + "api_key": "phc_replay", + "properties": { + "$session_id": "funnel-session", + "$current_url": "https://app.test/checkout" + } + }), + json!({ + "uuid": "funnel-3", + "event": "Viewed Pricing", + "distinct_id": "stuck-user", + "created_at": "2026-05-22T10:02:00Z", + "api_key": "phc_replay", + "properties": { + "$session_id": "stuck-session", + "$current_url": "https://app.test/pricing" + } + }), + json!({ + "uuid": "funnel-4", + "event": "Viewed Pricing", + "distinct_id": "stuck-user", + "created_at": "2026-05-22T10:02:03Z", + "api_key": "phc_replay", + "properties": { + "$session_id": "stuck-session", + "$current_url": "https://app.test/pricing" + } + }), + json!({ + "uuid": "chunk-friction", + "event": "$snapshot_items", + "distinct_id": "friction-user", + "created_at": "2026-05-22T10:03:00Z", + "api_key": "phc_replay", + "properties": { + "$session_id": "friction-session", + "$snapshot_items": [ + { "type": 4, "timestamp": 1000, "data": { "href": "https://app.test/pricing" } }, + { "type": 3, "timestamp": 1100, "data": { "source": 2, "type": 2, "x": 42, "y": 52 } }, + { "type": 3, "timestamp": 1700, "data": { "source": 2, "type": 2, "x": 44, "y": 50 } }, + { "type": 3, "timestamp": 2200, "data": { "source": 2, "type": 2, "x": 43, "y": 51 } }, + { "type": 3, "timestamp": 40000, "data": { "source": 3, "x": 0, "y": 1600 } } + ] + } + }), + json!({ + "uuid": "chunk-journey", + "event": "$snapshot_items", + "distinct_id": "journey-user", + "created_at": "2026-05-22T10:04:00Z", + "api_key": "phc_replay", + "properties": { + "$session_id": "journey-session", + "$snapshot_items": [ + { "type": 4, "timestamp": 1000, "data": { "href": "https://app.test/start" } } + ] + } + }), + json!({ + "uuid": "journey-event", + "event": "Product Viewed", + "distinct_id": "journey-user", + "created_at": "2026-05-22T10:04:10Z", + "api_key": "phc_replay", + "properties": { + "$session_id": "journey-session", + "$current_url": "https://app.test/product", + "sku": "sku_123" + } + }), + ]; + + let (pipeline_endpoint, _pipeline_rx, pipeline_handle) = spawn_pipeline_stub().await?; + let (replay_endpoint, mut replay_queries, replay_handle) = spawn_replay_sql_stub(rows).await?; + let replay_config = ReplayConfig::new( + "account-id".to_string(), + "bucket-name".to_string(), + "r2-sql-token".to_string(), + Some("default.hogflare_events".to_string()), + Some(500), + Some(replay_endpoint), + )?; + let (address, server_handle) = spawn_app_with_replay(pipeline_endpoint, replay_config).await?; + + let base_url = format!("http://{}", address); + let client = Client::builder().timeout(Duration::from_secs(5)).build()?; + + let ui_response = client.get(format!("{}/replay", base_url)).send().await?; + assert!(ui_response.status().is_success()); + let ui = ui_response.text().await?; + assert!(ui.contains("Replay Explorer")); + assert!(ui.contains("aria-label=\"Hogflare\"")); + assert!(ui.contains("id=\"status\" hidden")); + assert!(ui.contains("Activity timeline")); + assert!(ui.contains("Date range")); + assert!(ui.contains("type=\"datetime-local\"")); + assert!(ui.contains("data-date-preset=\"24h\"")); + assert!(ui.contains("aria-label=\"Hide filters\"")); + assert!(ui.contains("aria-label=\"Hide context\"")); + assert!(ui.contains(">Search<")); + assert!(ui.contains("Funnel steps")); + assert!(ui.contains("Friction signal")); + assert!(ui.contains("Person journey")); + assert!(ui.contains("rrweb-player")); + assert!(!ui.contains("Raw summary")); + + let sessions_response = client + .get(format!("{}/replay/api/sessions", base_url)) + .query(&[ + ("api_key", "phc_replay"), + ("distinct_id", "replay-user"), + ("limit", "25"), + ]) + .send() + .await?; + assert!(sessions_response.status().is_success()); + let sessions: Value = sessions_response.json().await?; + assert_eq!(sessions["sessions"].as_array().unwrap().len(), 1); + assert_eq!(sessions["sessions"][0]["session_id"], "session-1"); + assert_eq!(sessions["sessions"][0]["chunk_count"], 2); + assert_eq!(sessions["sessions"][0]["event_count"], 2); + assert_eq!(sessions["sessions"][0]["duration_ms"], 1000); + + let first_query = tokio::time::timeout(Duration::from_secs(2), replay_queries.recv()) + .await? + .expect("expected replay sql query"); + assert!(first_query.contains("from default.hogflare_events")); + assert!(first_query.contains("api_key = 'phc_replay'")); + assert!(first_query.contains("distinct_id = 'replay-user'")); + assert!(first_query.contains("event in ('$snapshot', '$snapshot_items')")); + + let replay_events_response = client + .get(format!("{}/replay/api/events", base_url)) + .query(&[ + ("api_key", "phc_replay"), + ("distinct_id", "replay-user"), + ("event_name", "Checkout Started"), + ("url", "page"), + ]) + .send() + .await?; + assert!(replay_events_response.status().is_success()); + let replay_events: Value = replay_events_response.json().await?; + assert_eq!(replay_events["events"].as_array().unwrap().len(), 1); + assert_eq!(replay_events["events"][0]["event"], "Checkout Started"); + assert_eq!(replay_events["events"][0]["session_id"], "session-1"); + assert_eq!(replay_events["events"][0]["url"], "https://app.test/page"); + assert_eq!( + replay_events["events"][0]["properties"][0]["key"], + "$current_url" + ); + + let event_query = tokio::time::timeout(Duration::from_secs(2), replay_queries.recv()) + .await? + .expect("expected replay event sql query"); + assert!(event_query.contains("event not in ('$snapshot', '$snapshot_items')")); + assert!(event_query.contains("event = 'Checkout Started'")); + + let events_response = client + .get(format!("{}/replay/api/sessions/session-1", base_url)) + .query(&[ + ("api_key", "phc_replay"), + ("distinct_id", "replay-user"), + ("at_ms", "250"), + ]) + .send() + .await?; + assert!(events_response.status().is_success()); + let events: Value = events_response.json().await?; + assert_eq!(events["session"]["session_id"], "session-1"); + assert_eq!(events["chunks"].as_array().unwrap().len(), 2); + assert_eq!(events["events"].as_array().unwrap().len(), 2); + assert_eq!(events["activity"].as_array().unwrap().len(), 2); + assert_eq!(events["replay_anchor_ms"], 250); + assert_eq!(events["events"][0]["timestamp"], 1000); + assert_eq!(events["events"][1]["timestamp"], 2000); + + let funnel_response = client + .get(format!("{}/replay/api/funnels", base_url)) + .query(&[ + ("api_key", "phc_replay"), + ("steps", "Viewed Pricing,Checkout Started"), + ("limit", "25"), + ]) + .send() + .await?; + assert!(funnel_response.status().is_success()); + let funnel: Value = funnel_response.json().await?; + assert_eq!(funnel["steps"].as_array().unwrap().len(), 2); + assert!(funnel["sessions"] + .as_array() + .unwrap() + .iter() + .any( + |session| session["session_id"] == "funnel-session" && session["status"] == "converted" + )); + assert!(funnel["sessions"] + .as_array() + .unwrap() + .iter() + .any(|session| session["session_id"] == "stuck-session" && session["status"] == "stuck")); + + let friction_response = client + .get(format!("{}/replay/api/friction", base_url)) + .query(&[("api_key", "phc_replay"), ("distinct_id", "friction-user")]) + .send() + .await?; + assert!(friction_response.status().is_success()); + let friction: Value = friction_response.json().await?; + assert_eq!(friction["sessions"].as_array().unwrap().len(), 1); + assert!(friction["sessions"][0]["signals"] + .as_array() + .unwrap() + .iter() + .any(|signal| signal["kind"] == "rage_click")); + + let person_response = client + .get(format!("{}/replay/api/person", base_url)) + .query(&[("api_key", "phc_replay"), ("distinct_id", "journey-user")]) + .send() + .await?; + assert!(person_response.status().is_success()); + let person: Value = person_response.json().await?; + assert_eq!(person["sessions"].as_array().unwrap().len(), 1); + assert_eq!(person["events"].as_array().unwrap().len(), 1); + assert!(person["timeline"] + .as_array() + .unwrap() + .iter() + .any(|item| item["kind"] == "event" && item["title"] == "Product Viewed")); + + cleanup(server_handle, pipeline_handle).await; + replay_handle.abort(); + let _ = replay_handle.await; + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn anonymous_identify_transition_enriches_events_and_person( ) -> Result<(), Box> { diff --git a/wrangler.toml.example b/wrangler.toml.example index f873a29..d97f903 100644 --- a/wrangler.toml.example +++ b/wrangler.toml.example @@ -6,6 +6,10 @@ compatibility_date = "2025-01-09" CLOUDFLARE_PIPELINE_ENDPOINT = "https://.ingest.cloudflare.com" CLOUDFLARE_PERSONS_PIPELINE_ENDPOINT = "https://.ingest.cloudflare.com" CLOUDFLARE_PIPELINE_TIMEOUT_SECS = "10" +# Optional replay UI Iceberg query config. Store HOGFLARE_REPLAY_R2_SQL_TOKEN as a secret. +# HOGFLARE_REPLAY_ACCOUNT_ID = "" +# HOGFLARE_REPLAY_BUCKET = "" +# HOGFLARE_REPLAY_EVENTS_TABLE = "default.hogflare_events" [[durable_objects.bindings]] name = "PERSONS"