Client performance telemetry: always-on instrument, Loki ingest, and Grafana dashboard#5597
Open
habdelra wants to merge 30 commits into
Open
Client performance telemetry: always-on instrument, Loki ingest, and Grafana dashboard#5597habdelra wants to merge 30 commits into
habdelra wants to merge 30 commits into
Conversation
The Linear MCP server registers as 'Linear' (capitalized), so the existing lowercase mcp__linear__* allowlist entries never match its tools. Add capitalized twins for the approved read tools, plus the read-only GitHub MCP tools that remote sessions use for PR reads and the session-start index-cache fetch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UFceD1sbstj9M4K8xRL62R
itemsThatReference only scanned boxel_index_working and prerendered_html, so a dependent whose row exists only in production boxel_index (index imported from a production-tables-only dump, or rows staged under a since-changed schema) was never re-rendered when a module it depends on changed. Its indexed computed values stayed stale until the file was touched individually. Scan production too, excluding promoted tombstones, and collapse duplicates in the existing (url, type) dedup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UFceD1sbstj9M4K8xRL62R
A render that runs inside the very batch that re-indexes a file reads the pre-swap production index row, so a card linking the file indexed the file's previous contentHash/contentSize — and any reindex re-serialized the same stale values instead of healing them. realm_file_meta is written in the same critical section as the file's bytes, so prefer it when serving file-meta documents; the indexed values remain fallbacks for files never hashed at write time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UFceD1sbstj9M4K8xRL62R
An instance's initial index visit can expand a searchable FileDef link
as { id } only, so the baseline snapshot now comes from the fan-out
re-render that follows the file's first write — the same path the
rewrite assertion exercises. Also let QUNIT_FILTER narrow a run to
individual tests, complementing TEST_MODULES/TEST_FILES.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UFceD1sbstj9M4K8xRL62R
State why realm_file_meta is authoritative and how the indexed values lag, rather than describing what would go wrong without the preference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UFceD1sbstj9M4K8xRL62R
A whitespace-only value became an active filter matching nothing, silently running no tests. Trim and only apply when non-empty, matching the TEST_MODULES handling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UFceD1sbstj9M4K8xRL62R
The tools resolve as mcp__Linear__* in this environment; the lowercase mcp__linear__* twins never matched. Keep the capitalized set and drop the dead duplicates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UFceD1sbstj9M4K8xRL62R
Accept batched client performance beacons (authenticated by the realm-server session token) and re-emit each event as a single structured JSON log line on the boxel:client-perf channel, which the existing stdout -> alloy -> Loki pipeline carries to Grafana. The handler is schema-agnostic beyond an envelope check: it caps body size (413) and event count (400), and the trusted envelope fields (channel, authenticated user id) are applied last so a per-event field cannot spoof them. Serve Document-Policy: js-profiling on the host HTML so the client telemetry's sampled JS self-profiler can capture wedge stacks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A passive singleton instrument that measures client-side performance and ships signal-gated, batched events to the realm-server /_client-telemetry endpoint. It records six event types: server-request latency (correlated to server logs via the logging correlation id), response deserialization cost, card-to- interactive latency (with card-id attribution), main-thread wedges (a heartbeat gap of 10s or more, with long-animation-frame / longtask attribution and, on sampled sessions, JS self-profiler stacks), loader/store rebuilds (with the invalidated trigger module), and per-realm-event reload churn. The instrument is disabled under tests (opt-in via enableForTest) and inside a prerender tab, feature-detects every web API it uses, and tears down every timer, observer, and listener. It stays quiet while the tab is hidden (throttled timers are not wedges), bounds each flush to the server's caps, and samples the JS self-profiler per session so continuous profiling cost is not paid by every client. Every hook is internally guarded so a telemetry failure never disturbs the path that emitted the event. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Visualizes the boxel:client-perf telemetry from Loki: an overview row of headline health, a collapsible row per event type, a by-user row surfacing which accounts are hitting issues (wedges, worst card-to-interactive, and signal volume per user), and a wedge breakdown that names the blocking frame (function plus source URL and character offset, resolvable via source map) and lists the worst blocking scripts per wedge. A wedge is labelled throughout as a main-thread freeze of 10s or more. Every panel is driven by shared env / user / session template variables so selecting a user or session drills the whole board into one client. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Grafana previewPreview deployed for 1 dashboard in the staging Grafana.
Dashboards: Preview is torn down automatically when this PR is closed or merged. (Run: https://github.com/cardstack/boxel/actions/runs/30103584541) |
Contributor
Observability diff (vs staging)Diff truncated (104770 bytes; limit 60000). Full diff: https://github.com/cardstack/boxel/actions/runs/30103584519 Show diffdiff --git a/tmp/committed-canon.tBqnTP/dashboards/boxel-status/client-performance.json b/tmp/committed-canon.tBqnTP/dashboards/boxel-status/client-performance.json
new file mode 100644
index 0000000..582cd2c
--- /dev/null
+++ b/tmp/committed-canon.tBqnTP/dashboards/boxel-status/client-performance.json
@@ -0,0 +1,3067 @@
+{
+ "apiVersion": "dashboard.grafana.app/v1beta1",
+ "kind": "Dashboard",
+ "metadata": {
+ "annotations": {
+ "grafana.app/folder": "defd2d156sav4d"
+ },
+ "name": "boxel-client-perf"
+ },
+ "spec": {
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": {
+ "type": "grafana",
+ "uid": "-- Grafana --"
+ },
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "description": "Client-side performance telemetry (channel boxel:client-perf) re-emitted by realm-server as structured JSON log lines and read from Loki. Overview headline signals plus a collapsible row per event type; the User/Session variables drill the whole board into a single client.",
+ "editable": true,
+ "fiscalYearStartMonth": 0,
+ "graphTooltip": 0,
+ "links": [],
+ "panels": [
+ {
+ "collapsed": false,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 0
+ },
+ "id": 2,
+ "panels": [],
+ "title": "Overview",
+ "type": "row"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Distinct client sessions that emitted any boxel:client-perf event in the selected time range. Drill filters (user/session) narrow this.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "blue"
+ },
+ {
+ "color": "green",
+ "value": 1
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 6,
+ "x": 0,
+ "y": 1
+ },
+ "id": 3,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "center",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "count(count by (session_id) (count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__range])))",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Active sessions",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Count of card-load events in the selected range.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 6,
+ "x": 6,
+ "y": 1
+ },
+ "id": 4,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "center",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__range]))",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Card loads (range)",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Count of wedge events (main-thread stalls) in the selected range. Non-zero is worth investigating in the wedge row below. A wedge is a main-thread freeze of 10s or more (the 100ms heartbeat missed a tick by ≥10s). Sub-second jank is not counted here; see a keepalive's max_gap_ms for that.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "yellow",
+ "value": 1
+ },
+ {
+ "color": "red",
+ "value": 10
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 6,
+ "x": 12,
+ "y": 1
+ },
+ "id": 5,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "center",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"wedge\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__range]))",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Wedges ≥10s (range)",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Count of rebuild events (loader/module refetch cycles) in the selected range.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "yellow",
+ "value": 5
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 6,
+ "x": 18,
+ "y": 1
+ },
+ "id": 6,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "center",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"rebuild\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__range]))",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Rebuilds (range)",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "p95 card-to-interactive latency (card-load settle_ms) alongside p95 server-request duration_ms. The headline responsiveness signals.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "ms",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "ms"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 5
+ },
+ "id": 7,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.95, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap settle_ms [$__interval]) by ()",
+ "legendFormat": "card settle p95",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.95, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__interval]) by ()",
+ "legendFormat": "server-request p95",
+ "queryType": "range",
+ "refId": "B"
+ }
+ ],
+ "title": "Interactivity p95 (ms)",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Rate of the signal-gated event types that flag anomalies. wedge and rebuild spikes correlate with degraded client sessions.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "events",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 5
+ },
+ "id": 8,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"wedge\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__interval]))",
+ "legendFormat": "wedge",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"rebuild\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__interval]))",
+ "legendFormat": "rebuild",
+ "queryType": "range",
+ "refId": "B"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"realm-event\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__interval]))",
+ "legendFormat": "realm-event",
+ "queryType": "range",
+ "refId": "C"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__interval]))",
+ "legendFormat": "card-load",
+ "queryType": "range",
+ "refId": "D"
+ }
+ ],
+ "title": "Signal event rate (events / interval)",
+ "type": "timeseries"
+ },
+ {
+ "collapsed": true,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 14
+ },
+ "id": 13,
+ "panels": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Card-to-interactive time (settle_ms): p50/p95/p99 across all card-load events in each interval.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "ms",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "ms"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 15
+ },
+ "id": 9,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.50, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap settle_ms [$__interval]) by ()",
+ "legendFormat": "p50",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.95, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap settle_ms [$__interval]) by ()",
+ "legendFormat": "p95",
+ "queryType": "range",
+ "refId": "B"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.99, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap settle_ms [$__interval]) by ()",
+ "legendFormat": "p99",
+ "queryType": "range",
+ "refId": "C"
+ }
+ ],
+ "title": "card-load settle_ms percentiles",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Network/loading time (loading_ms): p50/p95/p99 across all card-load events in each interval.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "ms",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "ms"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 15
+ },
+ "id": 10,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.50, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap loading_ms [$__interval]) by ()",
+ "legendFormat": "p50",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.95, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap loading_ms [$__interval]) by ()",
+ "legendFormat": "p95",
+ "queryType": "range",
+ "refId": "B"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.99, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap loading_ms [$__interval]) by ()",
+ "legendFormat": "p99",
+ "queryType": "range",
+ "refId": "C"
+ }
+ ],
+ "title": "card-load loading_ms percentiles",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Top 20 card_id values by worst observed settle_ms over the selected range. Instant query; widen the time range to survey more history.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "custom": {
+ "align": "auto",
+ "cellOptions": {
+ "type": "auto"
+ },
+ "filterable": true,
+ "inspect": false
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "red",
+ "value": 80
+ }
+ ]
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 9,
+ "w": 12,
+ "x": 0,
+ "y": 23
+ },
+ "id": 11,
+ "options": {
+ "cellHeight": "sm",
+ "footer": {
+ "countRows": false,
+ "fields": "",
+ "reducer": [
+ "sum"
+ ],
+ "show": false
+ },
+ "showHeader": true
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(20, max by (card_id) (max_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap settle_ms [$__range])))",
+ "legendFormat": "",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Slowest card loads by card_id (max settle_ms)",
+ "transformations": [
+ {
+ "id": "labelsToFields",
+ "options": {}
+ },
+ {
+ "id": "merge",
+ "options": {}
+ },
+ {
+ "id": "organize",
+ "options": {
+ "excludeByName": {
+ "Time": true
+ },
+ "renameByName": {
+ "Value #A": "max settle_ms"
+ }
+ }
+ }
+ ],
+ "type": "table"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Card-load event rate per interval, plus the average num_loads (link/dependency loads) per card-load event.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 9,
+ "w": 12,
+ "x": 12,
+ "y": 23
+ },
+ "id": 12,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__interval]))",
+ "legendFormat": "card-loads",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "avg_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"card-load\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap num_loads [$__interval]) by ()",
+ "legendFormat": "avg num_loads",
+ "queryType": "range",
+ "refId": "B"
+ }
+ ],
+ "title": "card-load rate & avg num_loads",
+ "type": "timeseries"
+ }
+ ],
+ "title": "card-load",
+ "type": "row"
+ },
+ {
+ "collapsed": true,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 24
+ },
+ "id": 19,
+ "panels": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "p95 duration_ms grouped by endpoint. Isolates which realm-server endpoints dominate client-observed request latency.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "ms",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "ms"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 25
+ },
+ "id": 14,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.95, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__interval]) by (endpoint)",
+ "legendFormat": "{{endpoint}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "server-request duration p95 by endpoint",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "p50/p95/p99 of duration_ms across all server-request events in each interval.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "ms",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "ms"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 25
+ },
+ "id": 15,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.50, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__interval]) by ()",
+ "legendFormat": "p50",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.95, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__interval]) by ()",
+ "legendFormat": "p95",
+ "queryType": "range",
+ "refId": "B"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.99, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__interval]) by ()",
+ "legendFormat": "p99",
+ "queryType": "range",
+ "refId": "C"
+ }
+ ],
+ "title": "server-request duration percentiles (all endpoints)",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Rate of 4xx/5xx server-request responses grouped by status code.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "errors",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 33
+ },
+ "id": 16,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum by (status) (count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | status=~\"[45]..\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__interval]))",
+ "legendFormat": "{{status}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "server-request error rate by status",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Per-endpoint worst-case duration_ms (query A) and retried-request count (query B) over the selected range. Instant queries merged on endpoint.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "custom": {
+ "align": "auto",
+ "cellOptions": {
+ "type": "auto"
+ },
+ "filterable": true,
+ "inspect": false
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "red",
+ "value": 80
+ }
+ ]
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 33
+ },
+ "id": 17,
+ "options": {
+ "cellHeight": "sm",
+ "footer": {
+ "countRows": false,
+ "fields": "",
+ "reducer": [
+ "sum"
+ ],
+ "show": false
+ },
+ "showHeader": true
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(15, max by (endpoint) (max_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__range])))",
+ "legendFormat": "",
+ "queryType": "instant",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum by (endpoint) (count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | retried=\"true\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__range]))",
+ "legendFormat": "",
+ "queryType": "instant",
+ "refId": "B"
+ }
+ ],
+ "title": "Slowest & most-retried endpoints",
+ "transformations": [
+ {
+ "id": "labelsToFields",
+ "options": {}
+ },
+ {
+ "id": "merge",
+ "options": {}
+ },
+ {
+ "id": "organize",
+ "options": {
+ "excludeByName": {
+ "Time": true
+ },
+ "renameByName": {
+ "Value #A": "max duration_ms",
+ "Value #B": "retried"
+ }
+ }
+ }
+ ],
+ "type": "table"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Raw server-request telemetry lines, newest first. Each carries correlation_id for cross-referencing realm-server request logs.",
+ "gridPos": {
+ "h": 10,
+ "w": 24,
+ "x": 0,
+ "y": 41
+ },
+ "id": 18,
+ "options": {
+ "dedupStrategy": "none",
+ "enableLogDetails": true,
+ "prettifyLogMessage": false,
+ "showCommonLabels": false,
+ "showLabels": false,
+ "showTime": true,
+ "sortOrder": "Descending",
+ "wrapLogMessage": true
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "{service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"server-request\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\"",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "server-request log lines (correlation_id)",
+ "type": "logs"
+ }
+ ],
+ "title": "server-request",
+ "type": "row"
+ },
+ {
+ "collapsed": true,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 42
+ },
+ "id": 22,
+ "panels": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "p95 deserialize duration_ms grouped by card_type. Surfaces card types whose document deserialization is expensive.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "ms",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "ms"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 43
+ },
+ "id": 20,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.95, {service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"deserialize\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__interval]) by (card_type)",
+ "legendFormat": "{{card_type}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "deserialize duration p95 by card_type",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Per card_type: avg duration_ms (A), avg doc_bytes (B), avg included_count (C) over the range. Intended as a cost table to spot large-document / high-fan-in card types (a scatter of duration vs size).",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "thresholds"
+ },
+ "custom": {
+ "align": "auto",
+ "cellOptions": {
+ "type": "auto"
+ },
+ "filterable": true,
+ "inspect": false
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ },
+ {
+ "color": "red",
+ "value": 80
+ }
+ ]
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 43
+ },
+ "id": 21,
+ "options": {
+ "cellHeight": "sm",
+ "footer": {
+ "countRows": false,
+ "fields": "",
+ "reducer": [
+ "sum"
+ ],
+ "show": false
+ },
+ "showHeader": true
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "avg by (card_type) (avg_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"deserialize\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap duration_ms [$__range]))",
+ "legendFormat": "",
+ "queryType": "instant",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "avg by (card_type) (avg_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"deserialize\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap doc_bytes [$__range]))",
+ "legendFormat": "",
+ "queryType": "instant",
+ "refId": "B"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "avg by (card_type) (avg_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"deserialize\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" | unwrap included_count [$__range]))",
+ "legendFormat": "",
+ "queryType": "instant",
+ "refId": "C"
+ }
+ ],
+ "title": "deserialize cost by card_type (duration vs bytes vs included)",
+ "transformations": [
+ {
+ "id": "labelsToFields",
+ "options": {}
+ },
+ {
+ "id": "merge",
+ "options": {}
+ },
+ {
+ "id": "organize",
+ "options": {
+ "excludeByName": {
+ "Time": true
+ },
+ "renameByName": {
+ "Value #A": "avg duration_ms",
+ "Value #B": "avg doc_bytes",
+ "Value #C": "avg included_count"
+ }
+ }
+ }
+ ],
+ "type": "table"
+ }
+ ],
+ "title": "deserialize",
+ "type": "row"
+ },
+ {
+ "collapsed": true,
+ "description": "A wedge is a main-thread freeze of 10s or more (the 100ms heartbeat missed a tick by ≥10s). Sub-second jank is not counted here; see a keepalive's max_gap_ms for that.",
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 44
+ },
+ "id": 25,
+ "panels": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Wedge event rate per interval (left axis) and the worst blocked_ms observed per interval (right axis, ms). A wedge is a main-thread stall; blocked_ms is how long the thread was blocked. A wedge is a main-thread freeze of 10s or more (the 100ms heartbeat missed a tick by ≥10s). Sub-second jank is not counted here; see a keepalive's max_gap_ms for that.",
+ "fieldConfig": {
+ "defaults": {
+ "color": {
+ "mode": "palette-classic"
+ },
+ "custom": {
+ "axisLabel": "wedges",
+ "axisPlacement": "auto",
+ "barAlignment": 0,
+ "drawStyle": "line",
+ "fillOpacity": 10,
+ "gradientMode": "none",
+ "hideFrom": {
+ "legend": false,
+ "tooltip": false,
+ "viz": false
+ },
+ "lineInterpolation": "linear",
+ "lineWidth": 1,
+ "pointSize": 5,
+ "scaleDistribution": {
+ "type": "linear"
+ },
+ "showPoints": "never",
+ "spanNulls": false,
+ "stacking": {
+ "group": "A",
+ "mode": "none"
+ },
+ "thresholdsStyle": {
+ "mode": "off"
+ }
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green"
+ }
+ ]
+ },
+ "unit": "short"
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "max blocked_ms"
+ },
+ "properties": [
+ {
+ "id": "custom.axisPlacement",
+ "value": "right"
+ },
+ {
+ "id": "unit",
+ "value": "ms"
+ }
+ ]
+ }
+ ]
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 24,
+ "x": 0,
+ "y": 45
+ },
+ "id": 23,
+ "options": {
+ "legend": {
+ "calcs": [
+ "mean",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "pluginVersion": "12.4.3",
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum(count_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"wedge\" | matrix_user_id=~\"$matrix_user_id\" | session_id=~\".*${session_id}.*\" [$__interval]))",
+ "legendFormat": "wedges",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "max_over_time({service=\"realm-server\", env=\"$env\"} | json | channel=\"boxel:client-perf\" | event_type=\"wedge\" | matrix_user_id=~\"$mat |
Contributor
Preview deploymentsHost Test Results 1 files 1 suites 3h 7m 20s ⏱️ Results for commit 6276be4. Realm Server Test Results 1 files ±0 1 suites ±0 11m 1s ⏱️ +32s Results for commit 6276be4. ± Comparison against earlier commit a885dca. |
Grafana names an instant metric query's numeric column Value #<refId>; an organize transformation renames it to what it measures (wedges, events, max settle_ms, ...) and hides the redundant per-row timestamp. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 754b7a17a6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The images are referenced from the PR description via commit-pinned raw URLs; the working tree no longer needs to carry them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an always-on client performance telemetry pipeline end-to-end: a host-side instrument that batches events and POSTs them to realm-server, a realm-server ingest route that re-emits events as structured log lines for Loki, and a Grafana dashboard to explore the resulting signal.
Changes:
- Add
/_client-telemetryrealm-server route + handler that authenticates via JWT and logs one JSON line per event to theboxel:client-perfchannel. - Introduce a host
client-telemetryservice (eagerly started) plus hooks in network/store/loader paths to emit server-request, deserialize, rebuild, realm-event, card-load, wedge, and keepalive events. - Add a Loki-backed Grafana dashboard for client performance drilldowns, plus enable
Document-Policy: js-profilingin realm-server HTML responses and host dev servers.
Reviewed changes
Copilot reviewed 10 out of 11 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/realm-server/routes.ts | Wires a new authenticated POST /_client-telemetry route. |
| packages/realm-server/handlers/serve-index.ts | Adds Document-Policy: js-profiling on index HTML responses. |
| packages/realm-server/handlers/handle-client-telemetry.ts | Implements ingest: validates envelope and logs each event as a JSON line. |
| packages/observability/grafanactl/resources/dashboards/boxel-status/client-performance.json | Adds a new “Client Performance” Grafana dashboard using Loki queries over structured logs. |
| packages/host/vite.config.mjs | Enables Document-Policy: js-profiling in Vite dev/preview headers. |
| packages/host/tests/integration/client-telemetry-test.ts | Adds integration tests for the telemetry service’s gating and event shapes. |
| packages/host/app/services/store.ts | Emits deserialize + realm-event/rebuild telemetry from store lifecycle and invalidation handling. |
| packages/host/app/services/network.ts | Adds server-request timing middleware into authedFetch. |
| packages/host/app/services/client-telemetry.ts | New telemetry service: buffering, flush loop, wedge detection, optional JS self-profiler sampling, and request timing middleware. |
| packages/host/app/lib/gc-card-store.ts | Extends recent card doc load diagnostics with outcome for telemetry attribution. |
| packages/host/app/instance-initializers/register-client-telemetry.ts | Eagerly boots and tears down telemetry at app-instance lifecycle boundaries. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The end-to-end full-reindex test passed with or without the file-meta serving change, so it never exercised the regression it claimed to guard. Bug #2 is guarded by the serving-precedence test, which fails without the fix. Remove the redundant test and its now-unused searchDocForIndexEntry helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UFceD1sbstj9M4K8xRL62R
- Cap each flush under the ~64KB fetch(keepalive) body budget, measured in UTF-8
bytes (a chunk between 64KB and the old 200KB char cap was silently rejected
by the browser and its events dropped).
- Reject oversized telemetry on the declared Content-Length before buffering the
whole request body; keep the post-read byte check as a fallback.
- Dashboard: an empty session textbox now matches all sessions
(`session_id=~".*${session_id}.*"`) instead of only an empty id.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The deserialize event's doc_bytes was JSON.stringify(doc).length — an O(size) re-serialization of the whole included graph on the hot deserialize path, purely to size it. fetchJSON now stamps the response Content-Length on the doc (O(1), non-enumerable), and the telemetry hook reads that instead. duration_ms (the actual deserialize cost) is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Referenced from the PR description via a commit-pinned raw URL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ender - Scope card-load `loaded_ids`/`slowest_loads` to the load window by stamping each recent doc-load with the load generation and filtering to generations past the window start, so a slow background reload from an earlier window is not misattributed to the card being opened. - Stamp a logging correlation id on the outgoing request in the server-request timing middleware when one is absent, so real-user beacons carry a correlation id that joins to server-side `realm:requests` logs (previously only minted for prerender traffic). - Defer the wedge emit to a macrotask and only emit when the heartbeat gap is corroborated by a LoAF/longtask observation in the window, suppressing spurious wedges from timer coalescing after the tab was backgrounded. - Race the card-load window against a watchdog so a stuck load cannot hold the window open for the tab's lifetime and suppress every later card-load. - Drop the prior 401 attempt from the buffer when a retried request succeeds, so a transient auth refresh does not read as a server error. - Skip the deserialize/rebuild/realm-event hooks inside a prerender tab. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Give each table panel's instant-query value column a descriptive name via an organize transform, so panels no longer render an opaque "Value #A" header. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Card-load windows aren't measured while the tab is hidden (the heartbeat returns early), so `#lastLoadGeneration` freezes there while background reloads advance `store.loadGeneration`. On resume the visibility handler reset the heartbeat clock but not the generation cursor, so the first visible heartbeat saw a large generation jump and opened a card-load window spanning the whole hidden period — attributing background reloads to the now-focused card with a near-zero settle time. Fast-forward the generation cursor on resume, mirroring the heartbeat-clock reset, so a hidden-period jump never opens a window. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The early size check rejected only an oversized *declared* length; a request
with no Content-Length yielded `Number('')` → 0, which passed the check, and
the body was then read fully into memory before the post-read byte check ran —
so a chunked or unbounded body bypassed the cap. A telemetry beacon is always a
materialized JSON string whose client declares a length, so require a finite
positive Content-Length up front (411 otherwise) and reject an oversized
declaration before reading. The post-read byte check stays as defense in depth.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A genuine main-thread freeze always leaves a LoAF/longtask entry, but that observer callback can be delivered a little after the heartbeat that detected the gap. Dropping an uncorroborated gap immediately could therefore discard a real freeze whose entry simply hadn't arrived yet. Re-check a couple of times across a short grace window before concluding the gap was an OS sleep/suspend wake — a real sleep runs no task and never corroborates, so it is still dropped, while a lagging observer entry now has time to land. Guard the deferred emit on the started flag so a pending re-check after teardown is a no-op. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Covers the security-critical surface: authentication (401 without a token), the one-JSON-line-per-event emission and 204 response, and the trusted-envelope ordering — the JWT-authenticated user and the fixed channel win over client-supplied per-event fields, so a beacon cannot spoof either. Also asserts the size caps (411 with no declared length, 413 over the byte cap) and the input-validation rejections (malformed JSON, missing events array, over the event-count cap). The emitted lines are captured by tapping the shared boxel:client-perf logger's methodFactory. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Persist search sheet state across close/reopen Add an in-memory, session-scoped SearchSheetState service that holds the operator-mode search sheet's query, selected types/realms, sort, view, and pagination, plus a snapshot of the last results. The sheet's @mode-gated subtree is still destroyed on close (no background work while closed); on reopen it rehydrates from the service, redisplays the cached results snapshot immediately, and re-runs the query to refresh. Persistence is opt-in via a `@persist` flag passed only by the search sheet, so the card choosers keep their own ephemeral state. The service registers with the reset service, so logout/realm reset clears it; a page reload also clears it (in-memory, no localStorage). resetState now runs only on explicit Cancel/Escape (removed from plain close/blur and result-select), and reopening a persisted search opens straight to the results view. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix backtracking assertions in persisted search sheet The search sheet's result snapshot and the search resource each wrote tracked state during the render that consumed it, tripping Glimmer's mutate-after-consume assertion and failing host tests as a global error. - Capture the main-results snapshot in a microtask instead of during the modifier's render, and key it to the current wire query so a reopen only redisplays results for the same search (never stale rows under a different term, and an idle sheet never clobbers a snapshot). Extract the snapshot-vs-live decision into a pure, unit-tested helper. - Start SearchEntriesResource's search a microtask after modify so the task's isRunning write lands outside the consuming render — the case that bites when a SearchResults mounts with a query already set (the sheet reopening restored, or a chooser rendering recents). Load tracking stays synchronous via a Deferred so prerender readiness is unaffected. - Update the reopen expectation in the interact-submode acceptance test to the persisted behavior, and add acceptance + unit coverage for persist, Cancel/Escape reset, empty-then-close, and service reset. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Seed search sheet results on reopen and persist scroll position Reopening the operator-mode search sheet with an unchanged query now seeds the freshly-mounted search resource from the retained snapshot and skips the fetch, so the prior results show immediately with no re-run and no "Searching…" flash. A one-shot guard keeps the seed from re-applying, so any query change still runs a fresh search. The results-list scroll offset is captured on scroll and restored on reopen (session-only). The seeded resource holds the rows directly, so the display-time resolveMainResults snapshot path is removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Update search-sheet blur tests to expect persistence The search sheet now keeps its query and filters on a plain close (click outside); only an explicit Cancel or Escape clears. Update the two operator-mode UI tests that asserted the old reset-on-blur behavior: - realm filter: selecting a realm then clicking outside and reopening keeps the realm checked. - type filter: a search term plus a type selection survive click-outside, so reopening returns to the results view with both still applied. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Report loading across the deferred search start The code-mode playground autogenerates a blank instance when it sees a search that has settled with no results. Starting the search resource's task a microtask after modify() (needed so a SearchResults mounting with a query already set doesn't flip isRunning mid-render and trip Glimmer's backtracking assertion) left a window where a just-set query read as settled-with-no-results — so the playground autogenerated "Untitled" instances, breaking the playground / spec / AI-assistant suites. Keep the deferred start, but have isLoading also report loading across that gap: it's true whenever a query is set and no run has completed yet, read from the untracked #previousQuery / hasCompletedFullRun fields so nothing tracked is mutated mid-render. The seed branch marks the run complete synchronously, so a seeded reopen still presents as settled with no flash. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Rework search-sheet persistence to a service-owned resource The session-scoped service now owns the search query and the SearchEntriesResource itself, so the results survive a sheet close/reopen without being snapshotted, re-seeded, or revalidated — the resource simply outlives the sheet's subtree, and its realm subscriptions stay live while closed. - Service derives mainQuery from its own inputs (idle when no term, type, or realm), so a type-filtered reopen never runs unfiltered and resetState() collapses the resource to idle. - SearchResults takes a @resource arg instead of @seed/@onSnapshot; PanelContent passes the service resource when persisting and keeps its own query derivation for the card choosers. - An externally-triggered search resets state and remounts the panel via a trigger epoch, so a trigger while the sheet is open re-reads the freshly-reset filter display. - isLoading settles after a failed run (no permanent "Searching…"), and a pending debounce is cancelled on Cancel/Escape. - Delete the snapshot/seed machinery and its tests; add coverage for the query derivation, failed-run settling, and clean triggered search. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Decouple search panel from sheet service; fix scroll restore Search-panel components no longer know about the search-sheet service. PanelContent is now controlled/uncontrolled via plain args (mainSearchResource, viewId/onViewIdChange, pagination, scrollTop/ onScrollTopChange); the sheet — which owns the service — passes them in. SearchPanel drops the `persist` flag. Replace the bespoke rAF-countdown scroll restore with a reusable persist-scroll-position modifier that re-applies the offset on every geometry change (MutationObserver + ResizeObserver) and only records gesture-driven scrolls, so restore no longer races the sheet's open transition and late-hydrating rows (flash-to-top / stuck-at-top). Adds deterministic integration tests for the modifier. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Hide search results until the reopen scroll restore sticks Restoring the persisted offset on reopen painted its intermediate states: the list appeared at the top mid-animation, then jumped or drifted as the offset re-applied against still-hydrating rows. The modifier now tags the container `data-scroll-restore-pending` while a non-zero saved offset is in flight, and the search sheet's content hides behind the tag and fades in on removal — the list reveals already in place. The tag comes off when an applied offset survives read-back and the container box has been quiet ~100ms (a stick against the mid-transition tiny viewport is trivial and must not count), on the first user gesture, or after a 1s cap when the offset is unreachable because the list got shorter. Restore runs once per element: recording writes the live offset back into the tracked state feeding the `scrollTop` arg, which re-runs the modifier on every recorded scroll — a re-install is recording-only (no tag, no observers, no re-applies), so the user's own scrolling is never hidden or fought. Close/reopen mounts a new element, so each reopen still restores exactly once. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Reopen filter-only searches to their results view Gate the reopen-to-results decision on the search-sheet state service's own hasActiveSearch (term OR type OR realm) instead of re-deriving "active" from the search term alone. A filter-only search — code mode's "Find instances" sets a type with an empty term — now reopens to its live, type-filtered results rather than the recents-only compact prompt. Cover it with a type-only unit case on hasActiveSearch/mainQuery and an acceptance test that reopens a filter-only search to the results view. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix import order after SessionService rename Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Reopen realm-only search to results in ui integration test The reopen gate keys on hasActiveSearch (term OR type OR realm), so a realm-filtered search with no term reopens to the results view, not the compact prompt. Update the integration test's stale expectation to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ments-khnwpn Fix indexed values going stale: module-edit fan-out misses and one-generation-old file content meta
…meout' into cs-12288-client-perf-telemetry
habdelra
changed the base branch from
main
to
flaky-base-realm-fetch-header-timeout
July 24, 2026 15:02
burieberry
reviewed
Jul 24, 2026
burieberry
left a comment
burieberry
left a comment
Contributor
There was a problem hiding this comment.
Code review
Two findings after verifying each against the diff. (Four other candidates were investigated and dropped — a search-entries race, an index-writer "third full-table scan", an oversized-telemetry-event, and the full-reindex own_write label — none survived verification, either factually wrong or with severity overstated.)
🐛 persist-scroll-position.ts — momentum/inertial scroll offsets aren't persisted on touch devices
The restore-suppression gate swallows inertial scroll events, so on touch devices the persisted offset is the position a frame or two after finger-lift, not the final resting position. Reopening the sheet then restores to the wrong place after any flick-scroll.
The chain:
restoringis a plain local reset totrueon every modifier run (let restoring = true), andrecordis gated on!restoring.restoringis cleared tofalseonly bystopRestoring, which is attached exclusively to gesture events:['wheel', 'touchmove', 'pointerdown', 'keydown']. Thescrolllistener runsrecordand never clears it.record→onChange(scrollTop)writes back the trackedresultsScrollTopthat thescrollToparg reads, which re-runs the modifier and re-armsrestoring = true(as the header comment notes).
So after a flick: the last touchmove clears restoring, the first momentum scroll events record and trigger a write-back, the modifier re-installs with restoring = true, and every remaining momentum scroll (iOS inertia can run ~1s+) is dropped because momentum emits no further gesture event to clear the gate.
Keeping restoring = true on re-install is clearly deliberate — it suppresses a layout-induced scroll reset right after re-install. But it also discards the real momentum settle. Options:
- Only re-arm
restoringon a genuine re-install (first install / an actual restore-target change), not on a self-triggered write-back re-run. - Or clear
restoringon the firstscrollafter a re-install once the offset matches the restored target, so subsequent momentum scrolls record.
Either way, worth a test asserting a gesture-less scroll following a write-back re-install is recorded — the current suite only asserts the opposite (gesture-less scrolls ignored), which is why this slipped through.
ℹ️ Non-blocking — realm.ts file-meta serve now does an unconditional DB lookup
The (searchHash === undefined || searchSize === undefined) guard was dropped, so getContentMeta() (an uncached SELECT on realm_file_meta) now runs on every file-meta serve even when the indexed searchDoc already carries both values. It's a single indexed-PK lookup so the cost is small, and the staleness rationale is well-commented — just flagging that the added per-request query isn't mentioned in the comments. Fine to leave as-is if the round-trip is acceptable on that path.
🤖 Review generated with Claude Code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
The client had no performance signal in the field. When a user's tab froze on "Loading card…" or went unresponsive for minutes, there was nothing to diagnose it after the fact, and everyday client cost — server round-trips, response deserialization, card-to-interactive latency, loader rebuilds — was invisible. This adds an always-on, passive client performance instrument, an ingest endpoint, and a Grafana dashboard, so real client behavior is measurable and attributable to a specific user or session.
Pipeline: host instrument →
fetch(keepalive)POST/_client-telemetry→ structuredboxel:client-perflog line → alloy → Loki → Grafana (the same log pipeline every server log already uses — no new infrastructure).What it measures
Six event types:
realm:requests/realm:search-timinglogs via the logging correlation id.function @ source-url:char-offset(source-map-resolvable, so it's readable for minified builds) plus the worst few blocking scripts, and — on profiler-sampled sessions — real JS self-profiler call stacks.foo.gtschanged").Extending it — new telemetry is welcome
The six event types are a starting set, not a closed schema. The instrument, the ingest route, and the dashboard are all event-type-agnostic: the route re-emits whatever event objects it's handed as
boxel:client-perfJSON lines, and the dashboard reads fields at query time with| json. Adding a new kind of client signal is a small, local change — define the event shape, record it from the relevant hook, and add a panel that reads its fields. Devs should feel welcome to instrument other parts of the client this way: Matrix sync latency and reconnect churn, AI assistant round-trips and streaming responsiveness, command execution timings — anything where the client's real behavior is otherwise invisible in the field.Sampling cadence vs. network cadence (decoupled)
Stats are gathered at their natural cadence, independent of when the client talks to the server:
pagehide, wherekeepalivelets the last POST complete, capped under the browser's ~64KB keepalive body budget).Design (live invariants)
enableForTest) and inside a prerender tab; every web API is feature-detected; every timer / observer / listener is torn down.env,service) are Loki stream labels;session_id/matrix_user_id/card_id/urlride inside the JSON line and are extracted at query time with| json, so per-user/per-session drill-down stays possible without exploding Loki's index.Dashboard
An overview row of headline health, a collapsible row per event type, a by-user row that surfaces which accounts are hitting issues (wedges, worst card-to-interactive, and signal volume per user), and a wedge breakdown that names the blocking frame and lists the worst blocking scripts per wedge. Shared
env/matrix_user_id/session_idtemplate variables drill the whole board into a single client. Works in local dev via the observabilitydocker-composestack +./scripts/apply.sh --env local.Screenshots below are the local dashboard driven by synthetic telemetry.
Overview + per-event-type rows
By user — who is hitting issues
Recent wedges with their blocking scripts (
fn @ url:char, source-map-resolvable)Testing
/_client-telemetryroute caps body size (early on Content-Length, then a byte fallback) and event count, authenticates viajwtMiddleware, and emits one JSON line per event. Type + lint clean.| json/unwrap/quantile_over_timequeries, per-user drill-down, and wedge-frame panel all render.Follow-ups to consider
Not committed to in this PR — options worth weighing later:
server-requesttelemetry volume if the Loki ingestion cost turns out to matter at scale.🤖 Generated with Claude Code