Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,10 @@ jobs:
- name: Install JS test dependencies
run: bun install --cwd tests/js_client --frozen-lockfile

- name: Install Playwright browser
working-directory: tests/js_client
run: bunx playwright install --with-deps chromium

- name: Verify Docker Compose
run: docker compose version

Expand Down
13 changes: 9 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ Hogflare is a Cloudflare Workers ingestion layer for PostHog SDKs. It supports P

#### What works today

- Ingestion endpoints: `/capture`, `/identify`, `/alias`, `/batch`, `/e`, `/engage`, `/groups`
- Ingestion endpoints: `/capture`, `/identify`, `/alias`, `/batch`, `/e`, `/engage`, `/groups`, `/s`
- Persons and groups: `$set`, `$set_once`, `$unset`, aliasing, and group properties
- Feature flags: `/flags` and `/decide` are evaluated in the Worker (used by PostHog SDKs)
- SDK config and feature flags: `/array/:token/config`, `/flags`, and `/decide` are evaluated in the Worker
- Request enrichment: Cloudflare IP/geo fields added when missing
- Queryable people: append-only person snapshots can be written to a separate Iceberg table

Expand Down Expand Up @@ -137,6 +137,7 @@ CLOUDFLARE_PIPELINE_TIMEOUT_SECS = "10"
# POSTHOG_GROUP_TYPE_2 = "project"
# POSTHOG_GROUP_TYPE_3 = "org"
# POSTHOG_GROUP_TYPE_4 = "workspace"
# POSTHOG_SESSION_RECORDING_ENDPOINT = "/s/"

[[durable_objects.bindings]]
name = "PERSONS"
Expand Down Expand Up @@ -532,6 +533,7 @@ Use `--skip-person-output` when resuming an event import after person rows were
- `/e` (event payloads)
- `/engage`
- `/groups`
- `/s` (session replay payloads)

### Persons

Expand All @@ -551,11 +553,14 @@ The Durable Object is the source of truth for the current person record. When `C

### Session replay

- `/s` stores raw session recording chunks only.
- SDK config advertises `sessionRecording: false` when `POSTHOG_SESSION_RECORDING_ENDPOINT` is unset, so the Worker can keep replay off remotely. Set `POSTHOG_SESSION_RECORDING_ENDPOINT=/s/` to turn replay on and route uploads through Hogflare's ingestion path.
- `/s` accepts PostHog replay payloads, including gzip/gzip-js compressed browser SDK requests.
- Modern `$snapshot` payloads are normalized to `$snapshot_items` rows before they are sent through Cloudflare Pipelines into R2.
- Legacy raw chunk payloads are still accepted as `$snapshot` rows.

### Feature flags

Feature flags are evaluated in the Worker and exposed via `/decide` and `/flags`.
Feature flags and SDK remote config are evaluated in the Worker and exposed via `/array/:token/config`, `/decide`, and `/flags`.

Configuration is a JSON blob in `HOGFLARE_FEATURE_FLAGS`. It can be either:

Expand Down
51 changes: 51 additions & 0 deletions src/extractors.rs
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ pub struct PostHogBatchPayload {
pub batch: BatchRequest,
}

pub struct PostHogRawPayload {
pub items: Vec<Value>,
pub sent_at: Option<DateTime<Utc>>,
}

pub struct RequestEnrichment {
properties: Map<String, Value>,
}
Expand Down Expand Up @@ -291,6 +296,52 @@ impl FromRequest<AppState, Body> for PostHogBatchPayload {
}
}

#[async_trait]
impl FromRequest<AppState, Body> for PostHogRawPayload {
type Rejection = PayloadExtractorError;

async fn from_request(
request: Request<Body>,
state: &AppState,
) -> Result<Self, Self::Rejection> {
let (parts, body) = request.into_parts();
let headers = parts.headers;
let query = parts.uri.query().map(str::to_string);
let bytes = to_bytes(body, usize::MAX)
.await
.map_err(PayloadExtractorError::BodyRead)?;

verify_signature(&headers, &bytes, state.signing_secret.as_deref())?;
let decoded = decode_content_encoding(&headers, &bytes)?;
let query_params = parse_query_params(query.as_deref())?;
let query_compression = query_param(&query_params, "compression")
.or_else(|| query_param(&query_params, "compression_method"));
let mut payloads =
parse_posthog_body_with_compression::<Value>(&headers, &decoded, query_compression)?;

if let Some(api_key) = header_api_key(&headers) {
apply_api_key_to_values(&mut payloads, &api_key);
}

Ok(PostHogRawPayload {
items: payloads,
sent_at: header_sent_at(&headers),
})
}
}

fn apply_api_key_to_values(items: &mut [Value], api_key: &str) {
for item in items {
let Value::Object(map) = item else {
continue;
};
if map.get("api_key").is_none() && map.get("token").is_none() && map.get("$token").is_none()
{
map.insert("api_key".to_string(), Value::String(api_key.to_string()));
}
}
}

fn decode_content_encoding(
headers: &HeaderMap,
body: &[u8],
Expand Down
Loading
Loading