S4X5: budget threshold alerts + SLI/SLO dashboards - #7
Closed
braghettos wants to merge 5 commits into
Closed
Conversation
Budget mechanism (alerts/budget/): a budget is a threshold on rated showback data per Org/Tenant/Service/Tag. Adds the budgets table + budget_status view DDL, an evaluator CronJob that emits breach/warning JSON log lines (heartbeat-canary pattern), and a HyperDX bootstrap script wiring in-portal + email delivery through the existing OTel -> ClickHouse -> HyperDX alert pipeline. Budget definitions are seeded by the consuming assembly; only the mechanism ships here. SLI/SLO exposure (dashboards/slo/ + docs/SLO.md): availability, latency and error-rate SLI queries over otel_traces/otel_logs, configurable SLO targets, error-budget burn-rate query, dashboard layout, and an SLO breach-alert bootstrap script. docs/SLO.md holds the reference definitions and a perf-results capture template. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Owner
Author
Minor fixes (verdict: real; verification pending)Budget
|
Renders {{database}} and actually executes ddl/001+002 against a
throwaway ClickHouse (clickhouse-local, docker-image fallback) with stub
showback_daily/showback_daily_by_tag rollups mirroring the showback
engine schemas, then asserts budget_status materializes the expected
rows: ok/warning/breached classification, ReplacingMergeTree FINAL
supersede semantics, tag-scoped budgets (tag_value filtering), daily vs
monthly period windows, enabled=0 exclusion and the no-spend LEFT JOIN
path. Verified green locally via the docker fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…es (D19a)
Two review findings on the budget evaluator CronJob:
- CLICKHOUSE_DATABASE was hardcoded to 'default' while the DDL renders a
{{database}} token (showback engine convention, default 'showback'),
so the CronJob queried a database where budget_status does not exist
(reproduced: UNKNOWN_TABLE, exit 60). Default is now 'showback' with a
comment binding it to the rendered DDL database.
- The bespoke stdout line (level/msg/component + fragile grep/cut row
parsing) is replaced by a line following the OTel Logs Data Model:
timestamp, severity_text/severity_number (warning->WARN/13,
breached->ERROR/17), body, trace_id/span_id and semconv-style
attributes (event.name, service.name, krateo.budget.*), so a
collector json parser can map it 1:1 onto an OTLP LogRecord while
HyperDX keeps querying Body as JSON. The evaluator is a scheduled
origin with no inbound trace context/baggage, so each run mints one
trace_id shared by all lines of the run (scalar subquery) plus a
per-line span_id. The whole line is built inside ClickHouse
(toJSONString escapes all user-supplied values; FORMAT TSVRaw keeps
lines atomic) - the shell no longer parses row data.
Verified end-to-end: the exact command block extracted from this YAML
was run in the CronJob image against a live clickhouse-server container
seeded with the DDL + sample data; output lines are valid JSON (jq),
32-hex shared trace_id / 16-hex span_ids, correct severity mapping, and
hostile values (quote/backslash/newline) stay escaped and atomic.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both bootstrap scripts logged a soft warning on non-2xx and kept going, so a partial bootstrap looked like success. Failures (saved search or alert creation, or a 2xx response with no id) are now recorded and the script ends with a FAILED summary and exit 1. Failures are collected in a temp file because the helpers run inside $(...) subshells where a counter would be lost. Also fixes a real capture bug the loud-failure test exposed: the 'Creating saved search' log line was written to stdout inside the command-substituted helper, polluting the captured saved-search id (the alert payload's savedSearchId contained the log line). It now goes to stderr; verified against a mock HyperDX that the alert payload carries the bare id, that an all-500 run attempts no alert POSTs and exits 1, and that partial/success paths behave (0 on success, 1 on partial). Budget saved-search filters and groupBy updated to the evaluator's new OTel line shape: JSONExtractString(Body,'attributes','krateo.budget.status') / 'krateo.budget.id' (extraction verified in ClickHouse against a real emitted line). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…D19a line shape
- Setup now shows rendering {{database}} (the previous command piped the
raw file with an unresolved token) and states that the CronJob's
CLICKHOUSE_DATABASE must match the rendered database.
- Document tests/smoke-budget-ddl.sh and the bootstrap's loud-failure
behaviour.
- Document the OTel Logs Data Model line shape (severity mapping,
per-run trace_id / per-line span_id, why a scheduled origin has no
inbound context/baggage).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner
Author
|
Applied the review minor-fixes (4 commits): 1. DDL actually executed — added 2. Database mismatch fixed — reproduced the bug first: with the old 3. HyperDX bootstraps fail loudly — both bootstrap scripts now record every non-2xx (and 2xx-without-id) step and end with a FAILED summary + exit 1, so a partial bootstrap can't look like success. Testing this against a mock HyperDX exposed a real pre-existing bug: the helper's log line was captured by 4. D19a line shape — the evaluator now emits an OTel Logs Data Model line (timestamp, severity_text/number with warning→WARN/13 breached→ERROR/17, body, trace_id/span_id, semconv-style attributes Verification: the exact command block extracted from the CronJob YAML was executed in the CronJob image against a live clickhouse-server container seeded with the DDL — valid JSON per line (jq), 32-hex shared trace_id / 16-hex span_ids, hostile values (quote/backslash/newline) stay escaped and atomic. Still unverified against a live HyperDX API (none in this environment): the alert payload field names remain best-effort. 🤖 Generated with Claude Code |
Comment on lines
+61
to
+141
| - name: evaluator | ||
| image: clickhouse/clickhouse-server:24.8-alpine | ||
| # ClickHouse builds the complete OTel-shaped JSON line per row | ||
| # (concat + toJSONString: all user-supplied values are | ||
| # JSON-escaped server-side; no shell parsing of row data). | ||
| # The scalar subquery is evaluated once, so every line of a | ||
| # run shares one trace_id; rand64() gives a per-line span_id. | ||
| # FORMAT TSVRaw = one unescaped line per row (toJSONString | ||
| # never emits raw control characters, lines stay atomic). | ||
| command: | ||
| - sh | ||
| - -c | ||
| - | | ||
| set -eu | ||
| clickhouse-client \ | ||
| --host "${CLICKHOUSE_HOST}" \ | ||
| --port "${CLICKHOUSE_PORT}" \ | ||
| --user "${CLICKHOUSE_USER}" \ | ||
| --password "${CLICKHOUSE_PASSWORD}" <<SQL | ||
| WITH (SELECT lower(hex(randomString(16)))) AS run_trace_id | ||
| SELECT concat( | ||
| '{"timestamp":"', formatDateTime(now(), '%Y-%m-%dT%H:%i:%SZ', 'UTC'), '",', | ||
| '"trace_id":"', run_trace_id, '",', | ||
| '"span_id":"', lower(hex(rand64())), '",', | ||
| '"severity_text":"', if(status = 'breached', 'ERROR', 'WARN'), '",', | ||
| '"severity_number":', if(status = 'breached', '17', '13'), ',', | ||
| '"body":"budget threshold crossed",', | ||
| '"attributes":', toJSONString(map( | ||
| 'event.name', 'krateo.budget.threshold', | ||
| 'service.name', 'krateo-budget-evaluator', | ||
| 'krateo.budget.id', budget_id, | ||
| 'krateo.budget.org', org, | ||
| 'krateo.budget.tenant', tenant, | ||
| 'krateo.budget.service', service, | ||
| 'krateo.budget.tag_key', tag_key, | ||
| 'krateo.budget.tag_value', tag_value, | ||
| 'krateo.budget.period', period, | ||
| 'krateo.budget.amount', toString(amount), | ||
| 'krateo.budget.currency', currency, | ||
| 'krateo.budget.spend', toString(spend), | ||
| 'krateo.budget.spend_ratio', toString(round(spend_ratio, 4)), | ||
| 'krateo.budget.status', status)), | ||
| '}') | ||
| FROM ${CLICKHOUSE_DATABASE}.budget_status | ||
| WHERE status != 'ok' | ||
| FORMAT TSVRaw | ||
| SQL | ||
| env: | ||
| - name: CLICKHOUSE_HOST | ||
| value: clickhouse.clickhouse-system.svc.cluster.local | ||
| - name: CLICKHOUSE_PORT | ||
| value: "9000" | ||
| # Must match the {{database}} the ddl/ files were rendered | ||
| # with (= the showback engine database; its default is | ||
| # `showback`), otherwise the budget_status view is not found. | ||
| - name: CLICKHOUSE_DATABASE | ||
| value: showback | ||
| - name: CLICKHOUSE_USER | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: otel-clickhouse-credentials | ||
| key: username | ||
| - name: CLICKHOUSE_PASSWORD | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: otel-clickhouse-credentials | ||
| key: password | ||
| resources: | ||
| limits: | ||
| cpu: 100m | ||
| memory: 128Mi | ||
| requests: | ||
| cpu: 20m | ||
| memory: 64Mi | ||
| securityContext: | ||
| readOnlyRootFilesystem: true | ||
| runAsNonRoot: true | ||
| runAsUser: 65534 | ||
| allowPrivilegeEscalation: false | ||
| capabilities: | ||
| drop: [ALL] |
Comment on lines
+61
to
+141
| - name: evaluator | ||
| image: clickhouse/clickhouse-server:24.8-alpine | ||
| # ClickHouse builds the complete OTel-shaped JSON line per row | ||
| # (concat + toJSONString: all user-supplied values are | ||
| # JSON-escaped server-side; no shell parsing of row data). | ||
| # The scalar subquery is evaluated once, so every line of a | ||
| # run shares one trace_id; rand64() gives a per-line span_id. | ||
| # FORMAT TSVRaw = one unescaped line per row (toJSONString | ||
| # never emits raw control characters, lines stay atomic). | ||
| command: | ||
| - sh | ||
| - -c | ||
| - | | ||
| set -eu | ||
| clickhouse-client \ | ||
| --host "${CLICKHOUSE_HOST}" \ | ||
| --port "${CLICKHOUSE_PORT}" \ | ||
| --user "${CLICKHOUSE_USER}" \ | ||
| --password "${CLICKHOUSE_PASSWORD}" <<SQL | ||
| WITH (SELECT lower(hex(randomString(16)))) AS run_trace_id | ||
| SELECT concat( | ||
| '{"timestamp":"', formatDateTime(now(), '%Y-%m-%dT%H:%i:%SZ', 'UTC'), '",', | ||
| '"trace_id":"', run_trace_id, '",', | ||
| '"span_id":"', lower(hex(rand64())), '",', | ||
| '"severity_text":"', if(status = 'breached', 'ERROR', 'WARN'), '",', | ||
| '"severity_number":', if(status = 'breached', '17', '13'), ',', | ||
| '"body":"budget threshold crossed",', | ||
| '"attributes":', toJSONString(map( | ||
| 'event.name', 'krateo.budget.threshold', | ||
| 'service.name', 'krateo-budget-evaluator', | ||
| 'krateo.budget.id', budget_id, | ||
| 'krateo.budget.org', org, | ||
| 'krateo.budget.tenant', tenant, | ||
| 'krateo.budget.service', service, | ||
| 'krateo.budget.tag_key', tag_key, | ||
| 'krateo.budget.tag_value', tag_value, | ||
| 'krateo.budget.period', period, | ||
| 'krateo.budget.amount', toString(amount), | ||
| 'krateo.budget.currency', currency, | ||
| 'krateo.budget.spend', toString(spend), | ||
| 'krateo.budget.spend_ratio', toString(round(spend_ratio, 4)), | ||
| 'krateo.budget.status', status)), | ||
| '}') | ||
| FROM ${CLICKHOUSE_DATABASE}.budget_status | ||
| WHERE status != 'ok' | ||
| FORMAT TSVRaw | ||
| SQL | ||
| env: | ||
| - name: CLICKHOUSE_HOST | ||
| value: clickhouse.clickhouse-system.svc.cluster.local | ||
| - name: CLICKHOUSE_PORT | ||
| value: "9000" | ||
| # Must match the {{database}} the ddl/ files were rendered | ||
| # with (= the showback engine database; its default is | ||
| # `showback`), otherwise the budget_status view is not found. | ||
| - name: CLICKHOUSE_DATABASE | ||
| value: showback | ||
| - name: CLICKHOUSE_USER | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: otel-clickhouse-credentials | ||
| key: username | ||
| - name: CLICKHOUSE_PASSWORD | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: otel-clickhouse-credentials | ||
| key: password | ||
| resources: | ||
| limits: | ||
| cpu: 100m | ||
| memory: 128Mi | ||
| requests: | ||
| cpu: 20m | ||
| memory: 64Mi | ||
| securityContext: | ||
| readOnlyRootFilesystem: true | ||
| runAsNonRoot: true | ||
| runAsUser: 65534 | ||
| allowPrivilegeEscalation: false | ||
| capabilities: | ||
| drop: [ALL] |
Comment on lines
+61
to
+141
| - name: evaluator | ||
| image: clickhouse/clickhouse-server:24.8-alpine | ||
| # ClickHouse builds the complete OTel-shaped JSON line per row | ||
| # (concat + toJSONString: all user-supplied values are | ||
| # JSON-escaped server-side; no shell parsing of row data). | ||
| # The scalar subquery is evaluated once, so every line of a | ||
| # run shares one trace_id; rand64() gives a per-line span_id. | ||
| # FORMAT TSVRaw = one unescaped line per row (toJSONString | ||
| # never emits raw control characters, lines stay atomic). | ||
| command: | ||
| - sh | ||
| - -c | ||
| - | | ||
| set -eu | ||
| clickhouse-client \ | ||
| --host "${CLICKHOUSE_HOST}" \ | ||
| --port "${CLICKHOUSE_PORT}" \ | ||
| --user "${CLICKHOUSE_USER}" \ | ||
| --password "${CLICKHOUSE_PASSWORD}" <<SQL | ||
| WITH (SELECT lower(hex(randomString(16)))) AS run_trace_id | ||
| SELECT concat( | ||
| '{"timestamp":"', formatDateTime(now(), '%Y-%m-%dT%H:%i:%SZ', 'UTC'), '",', | ||
| '"trace_id":"', run_trace_id, '",', | ||
| '"span_id":"', lower(hex(rand64())), '",', | ||
| '"severity_text":"', if(status = 'breached', 'ERROR', 'WARN'), '",', | ||
| '"severity_number":', if(status = 'breached', '17', '13'), ',', | ||
| '"body":"budget threshold crossed",', | ||
| '"attributes":', toJSONString(map( | ||
| 'event.name', 'krateo.budget.threshold', | ||
| 'service.name', 'krateo-budget-evaluator', | ||
| 'krateo.budget.id', budget_id, | ||
| 'krateo.budget.org', org, | ||
| 'krateo.budget.tenant', tenant, | ||
| 'krateo.budget.service', service, | ||
| 'krateo.budget.tag_key', tag_key, | ||
| 'krateo.budget.tag_value', tag_value, | ||
| 'krateo.budget.period', period, | ||
| 'krateo.budget.amount', toString(amount), | ||
| 'krateo.budget.currency', currency, | ||
| 'krateo.budget.spend', toString(spend), | ||
| 'krateo.budget.spend_ratio', toString(round(spend_ratio, 4)), | ||
| 'krateo.budget.status', status)), | ||
| '}') | ||
| FROM ${CLICKHOUSE_DATABASE}.budget_status | ||
| WHERE status != 'ok' | ||
| FORMAT TSVRaw | ||
| SQL | ||
| env: | ||
| - name: CLICKHOUSE_HOST | ||
| value: clickhouse.clickhouse-system.svc.cluster.local | ||
| - name: CLICKHOUSE_PORT | ||
| value: "9000" | ||
| # Must match the {{database}} the ddl/ files were rendered | ||
| # with (= the showback engine database; its default is | ||
| # `showback`), otherwise the budget_status view is not found. | ||
| - name: CLICKHOUSE_DATABASE | ||
| value: showback | ||
| - name: CLICKHOUSE_USER | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: otel-clickhouse-credentials | ||
| key: username | ||
| - name: CLICKHOUSE_PASSWORD | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: otel-clickhouse-credentials | ||
| key: password | ||
| resources: | ||
| limits: | ||
| cpu: 100m | ||
| memory: 128Mi | ||
| requests: | ||
| cpu: 20m | ||
| memory: 64Mi | ||
| securityContext: | ||
| readOnlyRootFilesystem: true | ||
| runAsNonRoot: true | ||
| runAsUser: 65534 | ||
| allowPrivilegeEscalation: false | ||
| capabilities: | ||
| drop: [ALL] |
Comment on lines
+61
to
+141
| - name: evaluator | ||
| image: clickhouse/clickhouse-server:24.8-alpine | ||
| # ClickHouse builds the complete OTel-shaped JSON line per row | ||
| # (concat + toJSONString: all user-supplied values are | ||
| # JSON-escaped server-side; no shell parsing of row data). | ||
| # The scalar subquery is evaluated once, so every line of a | ||
| # run shares one trace_id; rand64() gives a per-line span_id. | ||
| # FORMAT TSVRaw = one unescaped line per row (toJSONString | ||
| # never emits raw control characters, lines stay atomic). | ||
| command: | ||
| - sh | ||
| - -c | ||
| - | | ||
| set -eu | ||
| clickhouse-client \ | ||
| --host "${CLICKHOUSE_HOST}" \ | ||
| --port "${CLICKHOUSE_PORT}" \ | ||
| --user "${CLICKHOUSE_USER}" \ | ||
| --password "${CLICKHOUSE_PASSWORD}" <<SQL | ||
| WITH (SELECT lower(hex(randomString(16)))) AS run_trace_id | ||
| SELECT concat( | ||
| '{"timestamp":"', formatDateTime(now(), '%Y-%m-%dT%H:%i:%SZ', 'UTC'), '",', | ||
| '"trace_id":"', run_trace_id, '",', | ||
| '"span_id":"', lower(hex(rand64())), '",', | ||
| '"severity_text":"', if(status = 'breached', 'ERROR', 'WARN'), '",', | ||
| '"severity_number":', if(status = 'breached', '17', '13'), ',', | ||
| '"body":"budget threshold crossed",', | ||
| '"attributes":', toJSONString(map( | ||
| 'event.name', 'krateo.budget.threshold', | ||
| 'service.name', 'krateo-budget-evaluator', | ||
| 'krateo.budget.id', budget_id, | ||
| 'krateo.budget.org', org, | ||
| 'krateo.budget.tenant', tenant, | ||
| 'krateo.budget.service', service, | ||
| 'krateo.budget.tag_key', tag_key, | ||
| 'krateo.budget.tag_value', tag_value, | ||
| 'krateo.budget.period', period, | ||
| 'krateo.budget.amount', toString(amount), | ||
| 'krateo.budget.currency', currency, | ||
| 'krateo.budget.spend', toString(spend), | ||
| 'krateo.budget.spend_ratio', toString(round(spend_ratio, 4)), | ||
| 'krateo.budget.status', status)), | ||
| '}') | ||
| FROM ${CLICKHOUSE_DATABASE}.budget_status | ||
| WHERE status != 'ok' | ||
| FORMAT TSVRaw | ||
| SQL | ||
| env: | ||
| - name: CLICKHOUSE_HOST | ||
| value: clickhouse.clickhouse-system.svc.cluster.local | ||
| - name: CLICKHOUSE_PORT | ||
| value: "9000" | ||
| # Must match the {{database}} the ddl/ files were rendered | ||
| # with (= the showback engine database; its default is | ||
| # `showback`), otherwise the budget_status view is not found. | ||
| - name: CLICKHOUSE_DATABASE | ||
| value: showback | ||
| - name: CLICKHOUSE_USER | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: otel-clickhouse-credentials | ||
| key: username | ||
| - name: CLICKHOUSE_PASSWORD | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: otel-clickhouse-credentials | ||
| key: password | ||
| resources: | ||
| limits: | ||
| cpu: 100m | ||
| memory: 128Mi | ||
| requests: | ||
| cpu: 20m | ||
| memory: 64Mi | ||
| securityContext: | ||
| readOnlyRootFilesystem: true | ||
| runAsNonRoot: true | ||
| runAsUser: 65534 | ||
| allowPrivilegeEscalation: false | ||
| capabilities: | ||
| drop: [ALL] |
Owner
Author
|
Superseded by krateo-platformops/clickstack-chart#29 — re-homed to the current observability repo. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements S4 + X5 (tier-a mechanism).
alerts/budget/—budgetstable +budget_statusview DDL over the showback daily rollups, evaluator CronJob emitting breach/warning log lines (heartbeat-canary pattern), HyperDX bootstrap for warning + breach alerts with in-portal + email channels. No budget instances shipped (assembly config).dashboards/slo/— availability/latency/error-rate SLI queries overotel_traces/otel_logs, configurable SLO targets, burn-rate query, breach-alert bootstrap;docs/SLO.mdreference + perf-results capture template.Ref: braghettos/krateo-acmp#11, braghettos/krateo-acmp#21. Draft for review.