Add Sidemantic product analytics

Cargo.toml

@@ -35,7 +35,7 @@ regex = "1"
 axum = { version = "0.7", default-features = false, features = ["json", "query", "tokio", "http1"] }
 dotenvy = "0.15"
 reqwest = { version = "0.12", features = ["json", "rustls-tls"] }
-tokio = { version = "1.37", features = ["macros", "rt-multi-thread", "process", "net", "time"] }
+tokio = { version = "1.37", features = ["macros", "rt-multi-thread", "process", "net", "time", "io-util", "sync"] }
 tracing-subscriber = { version = "0.3", features = ["env-filter"] }
 
 [dev-dependencies]

README.md

@@ -71,7 +71,7 @@ flowchart TB
     EventsPipeline --> EventsR2["R2 Data Catalog<br/>events table"]
     PersonsPipeline --> PersonsR2["R2 Data Catalog<br/>persons table"]
 
-    ReplayUI["Replay Explorer"] -->|"R2 SQL"| EventsR2
+    WebUI["Hogflare UI"] -->|"R2 SQL"| EventsR2
     Models["Semantic Models<br/>models/*.yml"] --> EventsR2
     Models --> PersonsR2
     Consumers["DuckDB / R2 SQL / BI"] --> Models

docs/product-analytics.md

@@ -0,0 +1,28 @@
+# Product Analytics
+
+Hogflare serves product analytics from `/analytics`. The dashboard uses the repo's Sidemantic `events`, `pageviews`, and `sessions` models through the native Sidemantic/DuckDB Iceberg bridge.
+
+## API Config
+
+Product analytics shares the same R2 Data Catalog warehouse credentials as replay, but uses analytics-specific names when provided:
+
+| Setting | Notes |
+| --- | --- |
+| `HOGFLARE_ANALYTICS_ACCOUNT_ID` | Cloudflare account id for the R2 Data Catalog warehouse. Falls back to `HOGFLARE_REPLAY_ACCOUNT_ID`. |
+| `HOGFLARE_ANALYTICS_BUCKET` | R2 bucket name backing the warehouse. Falls back to `HOGFLARE_REPLAY_BUCKET`. |
+| `HOGFLARE_ANALYTICS_R2_SQL_TOKEN` | R2 SQL/Data Catalog token. Store as a secret. Falls back to `HOGFLARE_REPLAY_R2_SQL_TOKEN`. |
+| `HOGFLARE_ANALYTICS_EVENTS_TABLE` | Iceberg events table. Falls back to `HOGFLARE_REPLAY_EVENTS_TABLE`. |
+| `HOGFLARE_ANALYTICS_PERSONS_TABLE` | Optional Iceberg persons table. Defaults are inferred from the events table. |
+| `HOGFLARE_ANALYTICS_MODEL_DIR` | Optional Sidemantic model directory. Defaults to `models`. |
+| `HOGFLARE_ANALYTICS_SIDEMANTIC_SCRIPT` | Optional override for the native analytics worker script. |
+| `HOGFLARE_ANALYTICS_PREAGG` | Optional Sidemantic pre-aggregation switch. Defaults to enabled. Set to `0` to disable. |
+| `HOGFLARE_ANALYTICS_PREAGG_SCHEMA` | Optional DuckDB schema for Sidemantic rollup tables. Defaults to `sidemantic_preagg`. |
+
+## Routes
+
+- `/` serves the Hogflare app and opens product analytics by default.
+- `/analytics` serves the product analytics view.
+- `/analytics/api/charts` returns overview metrics, a focused trend, and semantic breakdowns including domains, referrers, browser, country, region, and city leaderboards.
+- `/replay` serves the replay feature.
+
+At worker startup, Sidemantic materializes known count-based chart shapes into daily pre-aggregation rollups and automatically routes eligible queries through those tables. `metric`, `dimension`, and `granularity` choose the focused chart. `semantic_filters` carries clickable cross-filter state as a JSON object of semantic dimensions to values. Analytics leaderboards use a fixed top-10 row cap plus an Others row.

docs/session-replay.md

@@ -2,7 +2,7 @@
 
 ![Hogflare replay explorer](assets/replay-explorer.jpg)
 
-Hogflare stores replay uploads in the same events table as analytics events, then serves a read-only replay explorer from `/replay`. The UI is for product analytics workflows: browse recent sessions, search events, inspect funnel drop-offs, review computed friction signals, and follow a person journey.
+Hogflare stores replay uploads in the same events table as product events, then serves a read-only replay explorer from `/replay`. The replay feature is for browsing recent sessions, searching events, inspecting funnel drop-offs, reviewing computed friction signals, and following a person journey.
 
 ## Ingestion
 
@@ -27,14 +27,16 @@ Replay APIs require:
 
 The token stays server-side in the Worker. The browser only calls Hogflare's replay API.
 
+Product analytics has a separate first-class route and API. See [product analytics](product-analytics.md).
+
 ## Routes
 
 - `/replay` serves the explorer UI.
 - `/replay/api/sessions` lists replay sessions by reading `$snapshot_items` and legacy `$snapshot` rows from Iceberg through R2 SQL.
-- `/replay/api/events` searches analytics events while excluding replay recording rows.
+- `/replay/api/events` searches product events while excluding replay recording rows.
 - `/replay/api/funnels` classifies sessions as converted, stuck, or dropped for an ordered `steps` list of event names.
 - `/replay/api/friction` computes replay-derived signals such as rage clicks, dead clicks, form thrash, long idle gaps, repeated navigation, and deep scroll without follow-up.
-- `/replay/api/person` joins a distinct ID's replay sessions and analytics events into one journey timeline.
+- `/replay/api/person` joins a distinct ID's replay sessions and product events into one journey timeline.
 - `/replay/api/sessions/:session_id` returns normalized rrweb events plus an activity timeline for one session.
 
 ## Filters

models/events.yml

@@ -54,7 +54,6 @@ models:
           group3,
           group4,
           group_properties,
-          api_key,
           extra,
           coalesce(
             json_extract_string(properties, '$.$session_id'),
@@ -87,6 +86,14 @@ models:
             json_extract_string(properties, '$.$referrer'),
             json_extract_string(properties, '$.referrer')
           ) as referrer,
+          nullif(regexp_extract(
+            coalesce(
+              json_extract_string(properties, '$.$referrer'),
+              json_extract_string(properties, '$.referrer')
+            ),
+            '^(?:[a-zA-Z][a-zA-Z0-9+.-]*://)?([^/?#]+)',
+            1
+          ), '') as referrer_domain,
           coalesce(
             json_extract_string(properties, '$.$utm_source'),
             json_extract_string(properties, '$.utm_source'),
@@ -188,11 +195,11 @@ models:
       select
         *,
         coalesce(session_id, actor_id || ':' || strftime(event_time, '%Y-%m-%d')) as session_key,
-        event_type not in ('$identify', '$groupidentify', '$create_alias', '$engage', '$snapshot') as is_capture_event,
+        event_type not in ('$identify', '$groupidentify', '$create_alias', '$engage', '$snapshot', '$snapshot_items') as is_capture_event,
         event_type in ('$pageview', 'page_view', '$screen', 'screen') as is_pageview_event,
         event_type in ('$identify', '$engage') as is_person_mutation_event,
         event_type = '$groupidentify' as is_group_event,
-        event_type = '$snapshot' as is_session_recording_event,
+        event_type in ('$snapshot', '$snapshot_items') as is_session_recording_event,
         resolved_person_id is not null as has_person,
         group0 is not null or group1 is not null or group2 is not null or group3 is not null or group4 is not null as has_group,
         session_id is not null as has_session_id
@@ -240,9 +247,6 @@ models:
       - name: team_id
         type: numeric
         description: Optional PostHog team id assigned by the Worker.
-      - name: api_key
-        type: categorical
-        description: PostHog project API key.
       - name: distinct_id
         type: categorical
         description: Original PostHog distinct id.
@@ -285,6 +289,9 @@ models:
       - name: referrer
         type: categorical
         description: Browser referrer.
+      - name: referrer_domain
+        type: categorical
+        description: Domain extracted from the browser referrer.
       - name: utm_source
         type: categorical
         description: UTM source.
@@ -326,15 +333,19 @@ models:
         description: Country code from Cloudflare enrichment or event properties.
       - name: geo_region
         type: categorical
+        parent: geo_country_code
         description: Region/subdivision from Cloudflare enrichment or event properties.
       - name: geo_city
         type: categorical
+        parent: geo_region
         description: City from Cloudflare enrichment or event properties.
       - name: geo_timezone
         type: categorical
+        parent: geo_country_code
         description: Timezone from Cloudflare enrichment.
       - name: cf_colo
         type: categorical
+        parent: geo_country_code
         description: Cloudflare colo.
       - name: cf_asn
         type: numeric
@@ -479,8 +490,8 @@ models:
       - name: snapshot_events
         agg: count
         filters:
-          - "event_type = '$snapshot'"
-        description: Session recording snapshot events.
+          - "event_type in ('$snapshot', '$snapshot_items')"
+        description: Session recording snapshot and normalized snapshot-item events.
       - name: grouped_events
         agg: count
         filters:
@@ -500,10 +511,6 @@ models:
         agg: count_distinct
         sql: coalesce(group0, group1, group2, group3, group4)
         description: Distinct group keys across populated group slots.
-      - name: unique_api_keys
-        agg: count_distinct
-        sql: api_key
-        description: Distinct PostHog project API keys.
       - name: power_users
         type: cohort
         entity: actor_id
@@ -524,6 +531,64 @@ models:
         agg: count
         description: Count of actors with at least two sessions in the query scope.
 
+    pre_aggregations:
+      - name: analytics_hourly
+        measures:
+          - event_count
+          - capture_events
+          - pageviews
+          - snapshot_events
+        dimensions:
+          - event_type
+          - pathname
+          - host
+          - referrer_domain
+          - referrer
+          - browser
+          - os
+          - device_type
+          - geo_country_code
+          - geo_region
+          - geo_city
+          - geo_timezone
+          - cf_asn
+          - utm_source
+          - utm_campaign
+        time_dimension: event_time
+        granularity: hour
+        partition_granularity: day
+        refresh_key:
+          every: 1 hour
+          incremental: false
+      - name: analytics_daily
+        measures:
+          - event_count
+          - capture_events
+          - pageviews
+          - snapshot_events
+        dimensions:
+          - event_type
+          - pathname
+          - host
+          - referrer_domain
+          - referrer
+          - browser
+          - os
+          - device_type
+          - geo_country_code
+          - geo_region
+          - geo_city
+          - geo_timezone
+          - cf_asn
+          - utm_source
+          - utm_campaign
+        time_dimension: event_time
+        granularity: day
+        partition_granularity: month
+        refresh_key:
+          every: 1 hour
+          incremental: false
+
     segments:
       - name: capture_events
         sql: is_capture_event
@@ -546,4 +611,3 @@ models:
       - name: session_recordings
         sql: is_session_recording_event
         description: Session recording snapshot events.
-

models/pageviews.yml

@@ -48,20 +48,29 @@ models:
               group2,
               group3,
               group4,
-              api_key,
               coalesce(json_extract_string(properties, '$.$session_id'), json_extract_string(properties, '$.session_id')) as session_id,
               coalesce(json_extract_string(properties, '$.$current_url'), json_extract_string(properties, '$.current_url'), json_extract_string(properties, '$.url')) as current_url,
               coalesce(json_extract_string(properties, '$.$pathname'), json_extract_string(properties, '$.pathname'), json_extract_string(properties, '$.path')) as pathname,
               coalesce(json_extract_string(properties, '$.$host'), json_extract_string(properties, '$.host')) as host,
               coalesce(json_extract_string(properties, '$.$title'), json_extract_string(properties, '$.title')) as page_title,
               coalesce(json_extract_string(properties, '$.$referrer'), json_extract_string(properties, '$.referrer')) as referrer,
+              nullif(regexp_extract(
+                coalesce(json_extract_string(properties, '$.$referrer'), json_extract_string(properties, '$.referrer')),
+                '^(?:[a-zA-Z][a-zA-Z0-9+.-]*://)?([^/?#]+)',
+                1
+              ), '') as referrer_domain,
               coalesce(json_extract_string(properties, '$.$utm_source'), json_extract_string(properties, '$.utm_source'), json_extract_string(properties, '$.$initial_utm_source')) as utm_source,
               coalesce(json_extract_string(properties, '$.$utm_medium'), json_extract_string(properties, '$.utm_medium'), json_extract_string(properties, '$.$initial_utm_medium')) as utm_medium,
               coalesce(json_extract_string(properties, '$.$utm_campaign'), json_extract_string(properties, '$.utm_campaign'), json_extract_string(properties, '$.$initial_utm_campaign')) as utm_campaign,
               coalesce(json_extract_string(properties, '$.$browser'), json_extract_string(properties, '$.browser')) as browser,
               coalesce(json_extract_string(properties, '$.$os'), json_extract_string(properties, '$.os')) as os,
               coalesce(json_extract_string(properties, '$.$device_type'), json_extract_string(properties, '$.device_type')) as device_type,
-              coalesce(json_extract_string(properties, '$.$geoip_country_code'), json_extract_string(properties, '$.country')) as geo_country_code
+              coalesce(json_extract_string(properties, '$.$geoip_country_code'), json_extract_string(properties, '$.country')) as geo_country_code,
+              coalesce(json_extract_string(properties, '$.$geoip_subdivision_1_code'), json_extract_string(properties, '$.region')) as geo_region,
+              coalesce(json_extract_string(properties, '$.$geoip_city_name'), json_extract_string(properties, '$.city')) as geo_city,
+              json_extract_string(properties, '$.$geoip_time_zone') as geo_timezone,
+              json_extract_string(properties, '$.cf_colo') as cf_colo,
+              try_cast(json_extract_string(properties, '$.cf_asn') as bigint) as cf_asn
             from {{ events_table }}
             left join identity_map on distinct_id = identity_map.linked_distinct_id
           )
@@ -105,6 +114,8 @@ models:
         type: categorical
       - name: referrer
         type: categorical
+      - name: referrer_domain
+        type: categorical
       - name: utm_source
         type: categorical
       - name: utm_medium
@@ -119,6 +130,20 @@ models:
         type: categorical
       - name: geo_country_code
         type: categorical
+      - name: geo_region
+        type: categorical
+        parent: geo_country_code
+      - name: geo_city
+        type: categorical
+        parent: geo_region
+      - name: geo_timezone
+        type: categorical
+        parent: geo_country_code
+      - name: cf_colo
+        type: categorical
+        parent: geo_country_code
+      - name: cf_asn
+        type: numeric
       - name: event_time
         type: time
         granularity: day
@@ -140,3 +165,54 @@ models:
         sql: resolved_person_id
         description: Distinct resolved persons with pageviews.
 
+    pre_aggregations:
+      - name: analytics_hourly
+        measures:
+          - pageviews
+        dimensions:
+          - event_type
+          - pathname
+          - host
+          - referrer_domain
+          - referrer
+          - browser
+          - os
+          - device_type
+          - geo_country_code
+          - geo_region
+          - geo_city
+          - geo_timezone
+          - cf_asn
+          - utm_source
+          - utm_campaign
+        time_dimension: event_time
+        granularity: hour
+        partition_granularity: day
+        refresh_key:
+          every: 1 hour
+          incremental: false
+      - name: analytics_daily
+        measures:
+          - pageviews
+        dimensions:
+          - event_type
+          - pathname
+          - host
+          - referrer_domain
+          - referrer
+          - browser
+          - os
+          - device_type
+          - geo_country_code
+          - geo_region
+          - geo_city
+          - geo_timezone
+          - cf_asn
+          - utm_source
+          - utm_campaign
+        time_dimension: event_time
+        granularity: day
+        partition_granularity: month
+        refresh_key:
+          every: 1 hour
+          incremental: false