From bcfd574aade4997791113a02792b4a30fb8d9f19 Mon Sep 17 00:00:00 2001 From: David Abram Date: Thu, 23 Jul 2026 16:24:29 +0200 Subject: [PATCH 1/6] flake: Split native and static-musl sce into distinct outputs Point packages.sce/.default and apps.sce/.default at the native scePackage for faster development builds, and expose the static-musl release binary as packages.sce-release / apps.sce-release. Retarget the release-artifacts helper to build .#sce-release. On Linux the native and release outputs are now distinct store paths; on Darwin the release output equals the native build. Co-authored-by: SCE --- context/glossary.md | 5 +- context/overview.md | 2 +- .../nix-build-devshell-ci-restructure.md | 372 ++++++++++++++++++ flake.nix | 18 +- 4 files changed, 390 insertions(+), 7 deletions(-) create mode 100644 context/plans/nix-build-devshell-ci-restructure.md diff --git a/context/glossary.md b/context/glossary.md index d79ce90d..8a2daef2 100644 --- a/context/glossary.md +++ b/context/glossary.md @@ -26,8 +26,9 @@ - `repo-root .version contract`: Repository packaging/version-source contract where `.version` stores the canonical semver string for flake-evaluated CLI package/check metadata and the approved release version authority for GitHub Releases, Cargo publication, and npm publication; `flake.nix` trims and reuses that value instead of hardcoding a version literal, and staged generated-config workflows must mirror it when Pkl renderers read the same path. - `Flatpak source-manifest GitHub Release asset`: Implemented non-binary release asset set for the source-built `dev.crocoder.sce` Flatpak channel: `sce-v-flatpak-manifest.tar.gz`, `sce-v-flatpak-manifest.tar.gz.sha256`, and `sce-v-flatpak.json`, emitted by `nix run .#release-flatpak-package` and uploaded by `.github/workflows/release-sce.yml` from `dist/flatpak`. The tarball packages Flatpak manifest/support files (`dev.crocoder.sce.yml`, AppStream metadata, `cargo-sources.json`, and `git-host-bridge`) with the staged manifest pinned to the release commit; it is not a prebuilt `sce` binary, `.flatpak` bundle, OSTree repository, or Flathub publication. - `Flatpak bundle GitHub Release asset`: Implemented source-built `.flatpak` bundle release asset set for the `dev.crocoder.sce` Flatpak channel: `sce-v-x86_64.flatpak`, `sce-v-aarch64.flatpak`, plus per-architecture `.sha256` and `.json` metadata, built from source inside Flatpak using `flatpak-builder` + `flatpak build-bundle`. The bundle is a source-built Flatpak app for direct install (`flatpak install --user `), not a prebuilt binary or Flathub submission. These assets coexist with the existing Flatpak source-manifest packaging metadata on GitHub Releases. The `release-bundle` command is implemented in `packaging/flatpak/sce-flatpak.sh` and the GitHub workflow upload is implemented in `.github/workflows/release-sce-linux.yml` / `release-sce-linux-arm.yml` (x86_64/aarch64), assembled by `release-sce.yml`. -- `cli flake release package`: Root-flake package output exposed as `packages.sce` with `packages.default = packages.sce`, producing the release-build `sce` binary via `nix build .#default`. -- `cli flake runnable app`: Root-flake app outputs exposed as `apps.sce` and `apps.default`, both executing `${packages.sce}/bin/sce`; this supports explicit `nix run .#sce -- ...` and default-app flows such as `nix run . -- ...` / `nix run github:crocoder-dev/shared-context-engineering -- ...` against the same packaged binary. +- `cli flake native package`: Root-flake package output exposed as `packages.sce` with `packages.default = packages.sce`, producing the **native** development `sce` binary (`scePackage`) via `nix build .#sce` / `nix build .#default`. +- `cli flake release package`: Root-flake package output exposed as `packages.sce-release` (`sceReleasePackage`): the static-musl release binary on Linux and the native build on Darwin, built via `nix build .#sce-release`. On Linux it is a distinct store path from `packages.sce` and passes the native portability audit; `nix run .#release-artifacts` builds through this output. +- `cli flake runnable app`: Root-flake app outputs where `apps.sce` and `apps.default` execute the native `${scePackage}/bin/sce` (dev ergonomics), supporting explicit `nix run .#sce -- ...` and default-app flows such as `nix run . -- ...` / `nix run github:crocoder-dev/shared-context-engineering -- ...`; `apps.sce-release` executes the release `${sceReleasePackage}/bin/sce` for explicit `nix run .#sce-release -- ...`. - `cli cargo install contract`: Supported Cargo install surface for the published `shared-context-engineering` crate, which installs the `sce` binary: crates.io (`cargo install shared-context-engineering --locked`), git (`cargo install --git https://github.com/crocoder-dev/shared-context-engineering shared-context-engineering --locked`), and local checkout (`cargo install --path cli --locked`). - `cli crates.io publication posture`: Current Cargo package posture in `cli/Cargo.toml` where crates.io-facing metadata is publication-ready for the `shared-context-engineering` crate, with crate-facing install guidance owned by `cli/README.md`. - `Nix performance recommendations`: Repo-local operator guidance in `AGENTS.md` covering optional user-level `~/.config/nix/nix.conf` tuning (`max-jobs = auto`, `cores = 0`) and the explicit root/admin-only boundary for `/etc/nix/nix.conf` `auto-optimise-store = true`. diff --git a/context/overview.md b/context/overview.md index 3f48e21c..9b96d155 100644 --- a/context/overview.md +++ b/context/overview.md @@ -38,7 +38,7 @@ Generated config now includes repo-local plugin assets for both profiles: `sce-b Claude bash-policy enforcement is also generated through `.claude/settings.json` as a `PreToolUse` `Bash` command hook running `sce policy bash`, so Claude and OpenCode both delegate to the Rust policy evaluator without a Claude TypeScript runtime. Pi bash-policy enforcement is delegated the same way through a project-local Pi extension (`config/lib/pi-plugin/sce-pi-extension.ts`, emitted to `config/.pi/extensions/sce/index.ts`) whose `tool_call` handler blocks denied bash commands via `sce policy bash` and fails open when the policy check cannot run (see `context/sce/pi-extension-runtime.md`). Local database bootstrap is now owned by `LocalDbLifecycle::setup` and `AgentTraceDbLifecycle::setup` aggregated by the setup command. Agent Trace setup creates/reuses the current checkout ID for diagnostics and initializes the repository-scoped `/sce/repos//agent-trace.db` with the repository schema; hook runtime lazily creates or upgrades that repository DB when setup has not run or schema metadata is incomplete. Doctor validates the repository-scoped DB path/health and can bootstrap missing parent directories; outside a Git repository it reports an actionable "requires a Git repository" diagnostic instead of probing a sentinel path. The `sce trace` group operates only on repository-scoped DBs for list/status/status-all/shell UX; the checkout-scoped `--legacy` surface was removed by the `retire-legacy-agent-trace-db` plan (see `context/decisions/2026-07-17-retire-legacy-agent-trace-db.md`). Wiring a user-invocable `sce sync` command is deferred to `0.4.0`. The repository-root flake (`flake.nix`) now applies a Rust overlay-backed stable toolchain pinned to `1.95.0` (with `rustfmt` and `clippy`), reads package/check version from the repo-root `.version` file, builds `packages.sce` through a Crane `buildDepsOnly` + `buildPackage` pipeline with filtered package sources for the Cargo tree plus required embedded config/assets, and runs `cli-tests`, `cli-clippy`, and `cli-fmt` through Crane-backed check derivations (`cargoTest`, `cargoClippy`, `cargoFmt`) that reuse the same filtered source/toolchain setup. -The root flake also exposes release install/run outputs directly as `packages.sce` (with `packages.default = packages.sce`) plus `apps.sce` and `apps.default`, so `nix build .#default`, `nix run . -- --help`, `nix run .#sce -- --help`, and `nix profile install github:crocoder-dev/shared-context-engineering` all target the packaged `sce` binary through the same flake-owned entrypoints. +The root flake splits native and release outputs: `packages.sce` and `packages.default` build the **native** development binary (`scePackage`), while `packages.sce-release` builds the release binary (`sceReleasePackage`: static musl on Linux, native on Darwin). So `nix build .#sce` / `.#default`, `nix run . -- --help`, `nix run .#sce -- --help`, and `nix profile install github:crocoder-dev/shared-context-engineering` target the native binary, and `nix build .#sce-release` / `nix run .#sce-release -- ...` (plus `nix run .#release-artifacts`, which builds `.#sce-release`) target the release binary. On Linux the native and release outputs are distinct store paths, and the release output passes the native portability audit. The CLI Cargo package metadata now includes crates.io publication-ready fields with crate-local install guidance in `cli/README.md`; supported Cargo install paths are `cargo install shared-context-engineering --locked`, `cargo install --git https://github.com/crocoder-dev/shared-context-engineering shared-context-engineering --locked`, and local `cargo install --path cli --locked`. The published crate installs the `sce` binary. The crate also keeps `cargo clippy --manifest-path cli/Cargo.toml` warnings-denied through `cli/Cargo.toml` lint configuration, so an extra `-- -D warnings` flag is redundant. The repository-root flake is now the single Nix entrypoint for both repo tooling and CLI packaging/checks, so root-level `nix flake check` evaluates the Crane-backed CLI checks (`cli-tests`, `cli-clippy`, `cli-fmt`), Linux-only `flatpak-static-validation`, `workflow-actionlint` (runs `actionlint` on all `.github/workflows/*.yml`), plus six split JavaScript check derivations: `npm-bun-tests`, `npm-biome-check`, `npm-biome-format`, `config-lib-bun-tests`, `config-lib-biome-check`, and `config-lib-biome-format`, without nested-flake indirection. The config-lib checks now consume `config/lib/` as the shared Bun/TypeScript package root for both `agent-trace-plugin/` and `bash-policy-plugin`, with dependencies resolved from `config/lib/package.json` and `config/lib/bun.lock`. For Cargo packaging/builds, the crate now compiles against a temporary `cli/assets/generated/` mirror prepared from canonical `config/` outputs during Nix builds and crates.io publish runs rather than from a committed crate-local snapshot. Config-lib JS flake checks execute from `config/lib/`, but the copied Nix check source is repo-shaped when tests require shared repo fixtures; the current Claude agent-trace golden tests are fully Rust-owned in `cli/src/services/structured_patch/fixtures` (Claude TypeScript plugin test removed in T07). diff --git a/context/plans/nix-build-devshell-ci-restructure.md b/context/plans/nix-build-devshell-ci-restructure.md new file mode 100644 index 00000000..cf482a27 --- /dev/null +++ b/context/plans/nix-build-devshell-ci-restructure.md @@ -0,0 +1,372 @@ +# Plan: Nix build / devShell / CI restructure for faster iteration + +## Change summary + +Restructure the Nix build, development shell, and CI so ordinary development +stops paying for release-only work, without weakening validation. The current +flake exposes the **static-musl release** binary as `packages..sce` and +`packages..default`, embeds the Git commit into every build and check, +ships the built `scePackage` and a source-compiled Turso CLI inside +`devShells.default`, and builds the musl release artifact in ordinary PR CI. + +This plan: + +- Splits native (development) and static-musl (release) packages into distinct + outputs: `packages..sce` and `packages..default` become the + **native** package; `packages..sce-release` becomes the release + package (static musl on Linux, native on Darwin). +- Makes Git-commit embedding **release-only**: normal native builds, `cargo + test`, Clippy, formatting, and other `nix flake check` derivations no longer + receive `SCE_GIT_COMMIT`, so they stay reusable across commits. `build.rs` + stops running `git rev-parse` and stops watching `.git/HEAD` / `.git/packed-refs`. +- Removes the built `scePackage` and the Turso CLI from `devShells.default` so + `nix develop` no longer compiles the package or `turso_cli`; keeps + `packages..turso` and adds an opt-in `devShells..database` + shell that layers Turso on top of the default tools. +- Replaces generated byte-array asset literals in `build.rs` with generated + `include_bytes!` expressions, preserving ordering, relative paths, SHA-256, + and rerun tracking. +- Exposes an explicit aggregate `packages..ci-checks` derivation whose + primary member is the static-musl release build, so the expensive tier is + `nix build .#ci-checks` rather than being folded into `nix flake check`. +- Restructures CI so PR/`main`/tag runs perform `nix flake check` + native + `.#sce` smoke test on both Linux and macOS, and a **separate** release + validation job builds `.#sce-release` (via `.#ci-checks`) on every commit. +- Investigates, with recorded benchmarks, whether Turso's default crate + features, the `turso` root flake input, and `flatpak-builder-tools` are worth + changing/isolating — changing them only if evidence and the full test suite + support it. + +All changes are benchmarked before/after under comparable cache conditions and +documented in `context/`. + +## Success criteria + +- `packages..sce` and `packages..default` build the native + development package on all systems; `packages..sce-release` builds the + static-musl binary on Linux and the native package on Darwin. On Linux, the + native and release outputs are distinct store paths and the release binary is + fully static / portable (passes the native portability audit). +- Ordinary check and native package derivations do **not** contain the current + Git commit in their inputs (they are cache-reusable across commits); the + release package still reports the real Git commit via `sce version`. +- `nix develop` (default shell) does not build `scePackage` or `turso_cli`. + `packages..turso` still builds, and `nix develop .#database` provides + the Turso CLI. +- Embedded assets remain byte-for-byte correct; all existing `build.rs`-driven + tests pass; generated asset source uses `include_bytes!`. +- `nix flake check` remains the primary validation entrypoint and still covers + every previously-present public check name on the current system (Linux and + macOS jobs both run it). +- PR/`main`/tag CI runs native `.#sce` smoke tests on Linux and macOS; a + distinct release-validation job builds `.#sce-release` on every commit and + audits Linux release portability. Workflows pass `actionlint`. +- Release helpers and release workflows explicitly build `.#sce-release`; the + produced release archives are unchanged in shape and portability. +- Public flake outputs that are not intentionally changed remain available. +- Before/after benchmarks are recorded, clearly distinguishing evaluation vs + build vs cached-lookup and native vs musl. + +## Constraints and non-goals + +- Constraints + - `nix flake check` stays the standard final local + CI validation command; + do not replace it with ad hoc targeted commands. + - Keep native and static-musl Crane pipelines separate; do not share + incompatible Cargo artifacts between host and musl builds. + - Preserve release portability guarantees, Flatpak source-generation, and all + parity checks (`cargo-sources-parity`, `flatpak-manifest-parity`, + `flatpak-static-validation`, `pkl-parity`, etc.). + - Do not remove Turso's Rust crate dependency from SCE. + - Do not introduce an impure `--config=ci` switch inside the flake; expensive + work goes behind an explicit aggregate output (`packages..ci-checks`). + - Do not split the flake into modules for aesthetics; only make + evaluation-structure changes when a measurement shows a materially expensive + input/import/package-set/forced-derivation. + - Do not change Turso default crate features or remove/isolate root inputs + unless benchmarks + full test suite justify it; record results either way. +- Non-goals + - No change to SCE runtime behavior, CLI surface, or database/migration logic. + - No new release targets or artifact formats. + - No path-filter machinery for release CI in this plan (see Decisions). + +## Decisions (from clarification) + +- **Optional Turso shell:** keep `packages..turso`; add + `devShells..database` (default tools + Turso). Placement does not + affect default-shell speed — the win is removing Turso from `default`. +- **Release CI trigger:** `.#sce-release` validation runs as a distinct job on + **every commit** (PRs + pushes to `main` + tags), with no path filters, for + now. This intentionally forgoes goal #6's "keep the release build out of PR + CI" performance benefit in exchange for always-on release coverage; revisit + with path filters later if PR CI cost becomes a problem. Recorded as a + decision, not an oversight. + +## Assumptions + +- `apps.sce` / `apps.default` should run the **native** package (dev + ergonomics), matching the new `packages.default`. Recorded and called out in + the package-split task; flagged for verification. +- The existing `nativePortabilityAuditCheck` is a fixture-based unit test of the + audit script (not an audit of the real binary) and stays on both platforms + unchanged. +- The current branch is `flake-speedup`; commits land there. + +## Open questions + +- None blocking. Benchmark-gated tasks (T08, T09) may end as deliberate no-ops; + that outcome is acceptable and must be recorded. + +## Task stack + +- [x] T01: `Capture baseline benchmarks and cache conditions` (status:done) + - Task ID: T01 + - Status: done + - Completed: 2026-07-23 + - Files changed: context/tmp/flake-speedup-benchmarks.md (new) + - Evidence: Baseline captured on x86_64-linux / Nix 2.34.8 / 8 cores / commit + 0da0a37. Eval (warm): `hyperfine --warmup 2 -r 5 'nix flake check --no-build'` + = 5.494 s ± 0.294 s; `NIX_SHOW_STATS` cpu 3.65 s, eval heap ~465 MiB, nrThunks + 4.11M. Warm `nix flake check --print-build-logs` = 6.27 s (0 rebuilt). musl + release `.#default` (== native `.#default` on Linux today): cache-hit 0.18 s, + cold final crate `--rebuild` 15.22 s / cargo 12.07 s. turso final crate 3.54 s. + Clean dependency compile: native `sce-deps` 7 m 58.9 s / 991 crates, musl + `sce-deps-musl-deps` 4 m 6.2 s / 494 crates. `nix develop -c true` warm 1.54 s. + Each figure labeled eval/dep-compile/final-crate/cached and native/musl with + warm/cold state in the doc. + - Notes: Confirmed current-state topology — on Linux `.#default`/`.#sce` == + static-musl `sceReleasePackage` (static-pie ELF, ~41.4 MiB); native + `scePackage` is only a devShell input, not a named output. "Cold" = derivation + rebuilt with its dependency closure still cached (global store not wiped). + Context-sync classification: verify-only (measure-only task, no code/behavior/ + architecture change; scratch doc under context/tmp/). + - Goal: Record before-change timings and cache state so later comparisons are + honest (no cold-vs-warm confusion). + - Boundaries (in/out of scope): In — measure and record only. Out — any code + or flake change. + - Done when: A new `context/tmp/flake-speedup-benchmarks.md` (or equivalent + committed doc) records, with stated cache conditions, at least: + `hyperfine --warmup 2 'nix flake check --no-build'`; `NIX_SHOW_STATS=1 nix + flake check --no-build` (if supported); wall-time of `nix flake check + --print-build-logs`, `nix build .#default --print-build-logs`, and (Linux) + the musl release build; `nix develop -c true`; and a clean/invalidated Rust + dependency build where practical. Each figure is labeled evaluation vs + dependency compile vs final crate compile vs cached lookup vs native vs musl. + - Verification notes (commands or checks): commands above run and captured; + doc committed; note warm/cold state per figure. + +- [x] T02: `Split native and static-musl into distinct flake outputs` (status:done) + - Task ID: T02 + - Status: done + - Completed: 2026-07-23 + - Files changed: flake.nix + - Evidence: `nix flake check --no-build` all checks passed. `nix build .#sce` + and `.#default` both → `/nix/store/za1ki...-sce-0.3.2` (native, identical + path); `.#sce-release` → `/nix/store/75w6s...-sce-0.3.2` (distinct static-musl + path). `nix run .#native-portability-audit -- --platform linux --binary + result-sce-release/bin/sce` passed (no forbidden `/nix/store/` refs). `nix run + .#release-artifacts -- --version 0.3.2 --out-dir ` succeeded via + `.#sce-release`, producing the `x86_64-unknown-linux-musl` archive/checksum/ + manifest with the in-run portability audit passing. `nix flake show` lists + `packages.sce-release` + `apps.sce-release`; all previously-public outputs + remain present. + - Notes: Repointed `packages.sce`/`.default` and `apps.sce`/`.default` (via + `sceApp`) to the native `scePackage`; added `packages.sce-release = + sceReleasePackage` plus a new `sceReleaseApp` (`apps.sce-release`); retargeted + `releaseArtifactsApp`'s internal build from `.#default` to `.#sce-release`. + devShell `scePackage` input left in place (T04). Flatpak validator + `banned_snippets` (`.#sce`/`.#default`) unchanged — source-build guards, not + affected by T02's done-when. Post-T02, `pr-ci.yml`'s `nix build .#default` + builds native until T07 (intentional, harmless). Context-sync + classification: important (public flake-output contract + glossary/overview + terminology `cli flake release package` / `cli flake runnable app` / + `cli flake release package` change). + - Goal: Make `packages..sce` and `.default` the native package and add + `packages..sce-release` (static musl on Linux, native on Darwin), + keeping the two Crane pipelines and their Cargo artifacts separate. + - Boundaries (in/out of scope): In — `flake.nix` `packages`/`apps` outputs; + point `sceApp`/`apps.default` at the native package; update + `releaseArtifactsApp` internal build target from `.#default` to + `.#sce-release`; update release workflows/helpers that build the release to + reference `.#sce-release`; update any flatpak validator command lists that + assert on `.#sce`/`.#default` semantics if needed. Out — commit-embedding + changes (T03), devShell (T04), CI job topology (T07), ci-checks aggregate + (T06). + - Done when: `nix build .#sce`, `.#default`, and `.#sce-release` all succeed; + on Linux `.#sce` and `.#sce-release` are distinct store paths and + `.#sce-release` passes `nix run .#native-portability-audit` on its binary; + on Darwin `.#sce-release` equals the native build; `nix run .#release-artifacts` + still builds via `.#sce-release`; previously-public outputs remain present. + - Verification notes (commands or checks): `nix flake check --no-build`; + `nix build .#sce .#sce-release --print-build-logs`; compare + `readlink result*`; Linux portability audit on release binary; grep flake + for lingering `sceReleasePackage` mis-wirings; `nix flake show`. + +- [ ] T03: `Make Git-commit embedding release-only` (status:todo) + - Task ID: T03 + - Goal: Only the release package receives the real commit via `SCE_GIT_COMMIT`; + native builds, `cargo test`, Clippy, fmt, and other checks become + commit-independent (cache-reusable across commits). + - Boundaries (in/out of scope): In — `cli/build.rs` `emit_git_commit` (drop + `git rev-parse` fallback, drop `.git/HEAD` + `.git/packed-refs` + rerun-if-changed watches, emit `SCE_GIT_COMMIT` only when the env var is + explicitly set); `flake.nix` so `SCE_GIT_COMMIT` is set only on the release + package derivation, not in `commonCargoArgs`/checks. Out — include_bytes! + (T05), package split topology (already in T02). + - Done when: native `.#sce` and every `nix flake check` derivation build + without the commit in their inputs (changing only the Git commit does not + invalidate them); `.#sce-release` binary still reports the real commit via + `sce version`; the app's existing unknown/development fallback is used when + the env var is absent. + - Verification notes (commands or checks): build `.#sce`, note store path; + make an empty commit; rebuild `.#sce` and confirm identical store path + (cache hit); `nix build .#sce-release && ./result/bin/sce version` shows the + commit; `nix flake check --print-build-logs`. + +- [ ] T04: `Slim default devShell; add opt-in database shell` (status:todo) + - Task ID: T04 + - Goal: `nix develop` no longer builds `scePackage` or `turso_cli`; Turso stays + available as `packages..turso` and via a new + `devShells..database` shell. + - Boundaries (in/out of scope): In — remove `scePackage` and `tursoPackage` + from `devShells.default` (and the `sce`/`turso` version echo lines that + depend on them); add `devShells.database` = default packages + `tursoPackage` + with its own shellHook line. Out — package outputs (T02), Turso feature/input + investigations (T08). + - Done when: `nix develop -c true` succeeds without building `scePackage` or + `turso_cli`; `nix develop .#database -c turso --version` works; `nix build + .#turso` still succeeds; default shell still provides the Rust toolchain and + JS/pkl tooling for `cargo run`/`cargo test`. + - Verification notes (commands or checks): `nix develop -c true` with + `--print-build-logs` shows no sce/turso compile; `nix develop .#database -c + true`; `nix flake show` lists both shells; `nix flake check --no-build`. + +- [ ] T05: `Generate include_bytes! asset literals in build.rs` (status:todo) + - Task ID: T05 + - Goal: Replace generated `bytes: &[0x.., ..]` arrays with generated + `include_bytes!(concat!(env!("CARGO_MANIFEST_DIR"), "/assets/generated/..."))` + expressions, preserving all embedded-asset behavior. + - Boundaries (in/out of scope): In — `cli/build.rs` + `generate_embedded_asset_manifest` (emit `include_bytes!` for `bytes`, keep + computing/embedding the SHA-256 array, keep deterministic ordering, keep the + existing relative path, keep `cargo:rerun-if-changed` per file, escape and + normalize generated paths correctly). Out — migrations manifest (already + `include_str!`), git-commit logic (T03). + - Done when: generated `setup_embedded_assets.rs` uses `include_bytes!` with + correct absolute paths; all embedded-asset tests pass; assets remain + byte-for-byte identical to before (verified via SHA-256 / test suite). + - Verification notes (commands or checks): `cargo test` in `cli/` for + embedded-asset tests; `nix flake check` (cli-tests, cli-clippy, cli-fmt); + record clean vs incremental Rust build time and generated-source size before + vs after in the benchmark doc. + +- [ ] T06: `Add explicit ci-checks aggregate output` (status:todo) + - Task ID: T06 + - Goal: Expose `packages..ci-checks` as the explicit long-running + validation tier whose primary member is the static-musl release build (plus + the Linux release portability audit), so `nix build .#ci-checks` is the + expensive command and `nix flake check` stays fast. + - Boundaries (in/out of scope): In — add `packages..ci-checks` + aggregating `.#sce-release` (and, on Linux, a release-portability-audit + derivation over the real release binary). Out — new tiers beyond the release + build; CI wiring (T07); any impure `--config=ci` switch (forbidden). + - Done when: `nix build .#ci-checks` builds the release package and, on Linux, + fails if the release binary is not portable; `nix flake check` does not build + `.#sce-release`; the aggregate is listed in `nix flake show`. + - Verification notes (commands or checks): `nix build .#ci-checks + --print-build-logs`; confirm `nix flake check --no-build` does not force the + musl release; `nix flake show`. + +- [ ] T07: `Restructure CI: native flake check + separate release job` (status:todo) + - Task ID: T07 + - Goal: PR/`main`/tag CI runs `nix flake check` + native `.#sce` smoke test on + Linux and macOS; a distinct release-validation job builds `.#sce-release` + (via `.#ci-checks`) on every commit and audits Linux release portability; + Linux-only checks stay Linux-guarded. + - Boundaries (in/out of scope): In — `.github/workflows/pr-ci.yml` (native + smoke uses `.#sce`, runs `sce --help` and `sce version`; keep Linux + macOS + `nix flake check`); add a release-validation job (matrix or dedicated) that + runs on every commit and builds `.#sce-release`/`.#ci-checks` with the Linux + portability audit; update release workflows/helpers to build `.#sce-release` + if any still reference `.#default`. Out — package/flake changes (T02/T03/T06). + - Done when: PR CI builds only `.#sce` (not the musl release) in the smoke job; + release job builds `.#sce-release` on every commit and passes Linux + portability audit; both OS `nix flake check` jobs preserved; `actionlint` + passes on all workflows. + - Verification notes (commands or checks): `actionlint .github/workflows/*.yml` + (via `nix run` or the flake check `workflow-actionlint`); re-read pr-ci and + release workflows; confirm no PR job builds the musl artifact except the + dedicated release job. + +- [ ] T08: `Investigate Turso default-features (benchmark-gated)` (status:todo) + - Task ID: T08 + - Goal: Decide, with recorded evidence, whether to set `turso` to + `default-features = false`; change it only if it meaningfully improves builds + and the full suite (incl. database + encryption tests) still passes. + - Boundaries (in/out of scope): In — benchmark existing vs + `default-features = false` for clean dep build, incremental build, native + release build, static-musl build, binary size, Nix closure size, database + tests, encryption tests; record all results; apply the change only if + justified. Out — any unrelated feature edits. + - Done when: results recorded in the benchmark doc with an explicit + accept/reject decision; if accepted, `cli/Cargo.toml` (+ `Cargo.lock`) + updated and full `nix flake check` passes; if rejected, no code change and + rationale recorded. + - Verification notes (commands or checks): the measurements above; + `nix flake check --print-build-logs`; database + encryption test names pass; + `du`/`nix path-info -S` for size/closure comparison. + +- [ ] T09: `Investigate optional root flake inputs (benchmark-gated)` (status:todo) + - Task ID: T09 + - Goal: Measure whether the `turso` repo input and `flatpak-builder-tools` + input materially affect cold input fetching or warm evaluation; isolate only + with evidence, preserving the public `packages..turso` output and all + Flatpak parity/source-generation behavior. + - Boundaries (in/out of scope): In — measure cold fetch + warm eval + contribution of each input; if materially expensive, isolate the Turso CLI + (narrow Nix module / pinned `fetchFromGitHub` / separate dev-tools flake) and + apply the same evidence-based approach to `flatpak-builder-tools`; keep the + Turso CLI output and Flatpak checks intact. Out — removing inputs merely for + size; changing Flatpak behavior. + - Done when: per-input measurements recorded with an explicit accept/reject + decision; if changed, `packages..turso`, `cargo-sources-parity`, + `flatpak-manifest-parity`, and `flatpak-static-validation` still pass; if + unchanged, rationale recorded. + - Verification notes (commands or checks): `hyperfine 'nix flake check + --no-build'` and `NIX_SHOW_STATS=1` with/without inputs where measurable; + `nix build .#turso`; `nix flake check` Flatpak checks; results in benchmark doc. + +- [ ] T10: `Document build/devShell/CI architecture and benchmarks` (status:todo) + - Task ID: T10 + - Goal: Update repository context/architecture docs to explain the new + structure and record benchmark outcomes. + - Boundaries (in/out of scope): In — update `context/architecture.md` (and + `context/patterns.md`/`context/overview.md` as needed) covering: native vs + release package outputs; release-only commit embedding; default vs + `.#database` shells; which CI jobs build native vs release; and the + before/after benchmark results with cache conditions. Out — code changes. + - Done when: docs describe all six points above and link the benchmark doc; + no stale claims about `packages.sce` being the release build remain. + - Verification notes (commands or checks): re-read updated docs; cross-check + against final `nix flake show`; ensure benchmark figures match T01/T11. + +- [ ] T11: `Final validation and context sync` (status:todo) + - Task ID: T11 + - Goal: Run all applicable validation, capture after-change benchmarks, and + sync context. + - Boundaries (in/out of scope): In — full validation suite, after-benchmark + capture, and context/plan status sync. Out — new feature work. + - Done when: all of the following pass and are recorded — `nix flake check + --print-build-logs`; `nix build .#sce --out-link result && ./result/bin/sce + --help && ./result/bin/sce version`; `nix build .#sce-release --out-link + result-release`; (Linux) native portability audit against the release + artifact; `actionlint` on all workflows; verification that native vs release + outputs are distinct on Linux, the release Linux binary is static/portable, + the default devShell does not build SCE, ordinary checks lack the current Git + commit in their inputs, release builds report the commit, embedded assets are + byte-for-byte correct, and previously-public outputs remain available. + Before/after benchmarks and the list of investigated-but-rejected changes are + finalized; context is synced. + - Verification notes (commands or checks): the exact commands above; empty-commit + cache-reuse check for `.#sce`; `nix flake show` output diff for public + outputs; benchmark doc updated with final numbers and remaining bottlenecks. diff --git a/flake.nix b/flake.nix index de87458a..c774e533 100644 --- a/flake.nix +++ b/flake.nix @@ -632,7 +632,7 @@ mkdir -p "$out_dir" - nix build .#default --out-link result + nix build .#sce-release --out-link result binary_path="result/bin/sce" if [[ ! -x "$binary_path" ]]; then @@ -1325,19 +1325,28 @@ ''; sceApp = { + type = "app"; + program = "${scePackage}/bin/sce"; + meta = { + description = "Run the packaged sce CLI (native)"; + }; + }; + + sceReleaseApp = { type = "app"; program = "${sceReleasePackage}/bin/sce"; meta = { - description = "Run the packaged sce CLI"; + description = "Run the packaged sce release CLI (static musl on Linux, native on Darwin)"; }; }; in { packages = { - sce = sceReleasePackage; + sce = scePackage; + sce-release = sceReleasePackage; bun = bunPackage; turso = tursoPackage; - default = sceReleasePackage; + default = scePackage; }; checks = @@ -1391,6 +1400,7 @@ { sce = sceApp; default = sceApp; + sce-release = sceReleaseApp; pkl-check-generated = { type = "app"; From c013bb6f6ac11bf76e2beea73f8a71691800c6f6 Mon Sep 17 00:00:00 2001 From: David Abram Date: Thu, 23 Jul 2026 16:54:17 +0200 Subject: [PATCH 2/6] flake: Make Git-commit embedding release-only MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Move SCE_GIT_COMMIT out of commonCargoArgs into a releaseCommitArgs fragment applied only to the release derivations (scePackageMusl on Linux, a new native-toolchain sceReleasePackageNative on Darwin). Native packages.sce/default and the cli-tests/cli-clippy/cli-fmt checks no longer carry the commit in their inputs, so they stay cache-reusable across commits (native `sce version` reports "unknown"), while .#sce-release still reports the real commit. Reduce cli/build.rs emit_git_commit to emit the rustc-env value only when SCE_GIT_COMMIT is explicitly set, with rerun-if-env-changed as its sole trigger — dropping the git rev-parse fallback, the .git/HEAD and .git/packed-refs rerun watches, and the now-unused std::process::Command import. On Darwin the release derivation now diverges from .#sce (shared deps-only cargoArtifacts, distinct final crate carrying the commit). Co-authored-by: SCE --- cli/build.rs | 47 +++---------------- context/architecture.md | 1 + context/glossary.md | 3 +- context/overview.md | 1 + .../nix-build-devshell-ci-restructure.md | 31 +++++++++++- flake.nix | 35 ++++++++++++-- 6 files changed, 70 insertions(+), 48 deletions(-) diff --git a/cli/build.rs b/cli/build.rs index c6069bc9..df9d03ef 100644 --- a/cli/build.rs +++ b/cli/build.rs @@ -5,7 +5,6 @@ use std::{ fs, io::{self, Write as IoWrite}, path::{Path, PathBuf}, - process::Command, }; const TARGETS: &[TargetSpec] = &[ @@ -97,54 +96,20 @@ fn generate_migration_manifest() -> io::Result<()> { } fn emit_git_commit() { + // Commit embedding is release-only: the flake sets SCE_GIT_COMMIT solely on + // the release package derivation. Ordinary native builds, `cargo test`, + // Clippy, fmt, and other checks leave it unset so they stay cache-reusable + // across commits. When absent, the app falls back to "unknown" via + // `option_env!`. Deliberately no `git rev-parse` fallback and no + // `.git/HEAD` / `.git/packed-refs` rerun watches. println!("cargo:rerun-if-env-changed=SCE_GIT_COMMIT"); if let Ok(commit) = env::var("SCE_GIT_COMMIT") { let commit = commit.trim(); if !commit.is_empty() { println!("cargo:rustc-env=SCE_GIT_COMMIT={commit}"); - return; } } - - let manifest_dir = match env::var("CARGO_MANIFEST_DIR") { - Ok(value) => PathBuf::from(value), - Err(_) => return, - }; - - let repository_root = match manifest_dir.parent() { - Some(path) => path.to_path_buf(), - None => return, - }; - - let git_dir = repository_root.join(".git"); - println!("cargo:rerun-if-changed={}", git_dir.join("HEAD").display()); - println!( - "cargo:rerun-if-changed={}", - git_dir.join("packed-refs").display() - ); - - let output = Command::new("git") - .args(["rev-parse", "--short=12", "HEAD"]) - .current_dir(&repository_root) - .output(); - - let Ok(output) = output else { - return; - }; - - if !output.status.success() { - return; - } - - let Ok(commit) = String::from_utf8(output.stdout) else { - return; - }; - - let commit = commit.trim(); - if !commit.is_empty() { - println!("cargo:rustc-env=SCE_GIT_COMMIT={commit}"); - } } fn emit_repo_version() { diff --git a/context/architecture.md b/context/architecture.md index 99d71739..206d474a 100644 --- a/context/architecture.md +++ b/context/architecture.md @@ -139,6 +139,7 @@ The repository includes a new placeholder Rust binary crate at `cli/`. - `cli/README.md` is the crate-local onboarding and usage source of truth for placeholder behavior, safety limitations, and roadmap mapping back to service contracts. - `flake.nix` applies `rust-overlay` (`oxalica/rust-overlay`) to nixpkgs, pins `rust-bin.stable.1.95.0.default` with `rustfmt` + `clippy`, reads the package/check version from repo-root `.version`, builds `packages.sce` through Crane (`buildDepsOnly` -> `buildPackage`) with shared host Cargo metadata plus a filtered repo-root source that preserves the Cargo tree, `cli/assets/hooks`, and only the generated config/schema inputs required by CLI embedding, then injects generated OpenCode/Claude/Pi config payloads and schema inputs into a temporary `cli/assets/generated/` mirror during derivation unpack so `cli/build.rs` can package the crate without requiring committed generated crate assets. The same `cli/build.rs` now scans `cli/migrations/*/*.sql` and writes `cli/src/generated_migrations.rs` with deterministic migration constants sorted by numeric filename prefix. - The root flake runs `cli-tests`, `cli-clippy`, and `cli-fmt` through Crane-backed paths for the CLI crate; `cli-fmt` intentionally uses the CLI-only Cargo source set rather than the package source with generated config/assets so formatting is not invalidated by release/config asset churn. The flake also exposes directory-scoped JS validation derivations for both `npm/` and the shared `config/lib/` plugin package root, and `pkl-parity` uses a narrowed Pkl/generated-output source set instead of copying the entire repository. +- Git-commit embedding is release-only. `SCE_GIT_COMMIT` is injected via a `releaseCommitArgs` fragment (`{ SCE_GIT_COMMIT = shortGitCommit; }`) merged only into the release derivations — `scePackageMusl` on Linux and a dedicated native-toolchain `sceReleasePackageNative` on Darwin — and is deliberately absent from `commonCargoArgs`. As a result native `packages.sce`/`packages.default` and the `cli-tests`/`cli-clippy`/`cli-fmt` check derivations carry no commit in their inputs and stay cache-reusable across commits (native `sce version` reports `unknown`), while `.#sce-release` reports the real commit. `cli/build.rs` `emit_git_commit` emits the `rustc-env` value only when `SCE_GIT_COMMIT` is explicitly set (with `rerun-if-env-changed=SCE_GIT_COMMIT` as its sole rerun trigger) — no `git rev-parse` fallback and no `.git/HEAD`/`.git/packed-refs` watches — and `cli/src/services/version/mod.rs` resolves the commit through `option_env!("SCE_GIT_COMMIT")` with an `"unknown"` fallback. On Darwin the release derivation therefore diverges from `.#sce` (shared deps `cargoArtifacts`, but distinct final crate carrying the commit). - Flatpak flake tooling is Linux-only and reduced to a minimal app surface: `apps.sce-flatpak` is the umbrella entrypoint (`nix run .#sce-flatpak -- ` for `validate`, `prepare-local-manifest --repo-root `, etc.) that delegates to `packaging/flatpak/sce-flatpak.sh`; `apps.release-flatpak-package` emits deterministic Flatpak GitHub Release source-manifest assets; `apps.release-flatpak-bundle` builds a source-built `.flatpak` bundle; helper apps `apps.regenerate-flatpak-manifest` and `apps.regenerate-cargo-sources` rewrite the checked-in generated artifacts from their Nix sources. The standalone `flatpak-validate`, `flatpak-local-manifest`, and `flatpak-build` wrapper apps are removed in favor of `sce-flatpak `. `checks..flatpak-static-validation` runs the Nix-built static validator script during default `nix flake check`, and `checks..flatpak-manifest-parity` plus `checks..cargo-sources-parity` enforce the generated-artifact parity contracts; default `nix flake check` does not run `flatpak-builder`. The Linux dev shell includes `appstreamcli`, `flatpak`, and `flatpak-builder`. - The shared config-lib source set is rooted at `config/lib/` and includes the shared `package.json`, `bun.lock`, and `tsconfig.json` plus `agent-trace-plugin/` and `bash-policy-plugin/`; `config-lib-bun-tests` runs the bash-policy plugin wrapper tests from that shared root, while `config-lib-biome-check` and `config-lib-biome-format` run Biome over the copied shared package source. `.github/workflows/publish-crates.yml` follows the same asset-preparation rule but runs Cargo packaging from a temporary clean repository copy so crates.io publish no longer needs `--allow-dirty`. - The config-lib check source preserves repo-relative access to shared CLI patch fixtures: Nix copies a filtered repo-shaped source containing `config/lib/**` plus `cli/src/services/structured_patch/fixtures`, then runs Bun/Biome from `config/lib/`. diff --git a/context/glossary.md b/context/glossary.md index 8a2daef2..772eb8e5 100644 --- a/context/glossary.md +++ b/context/glossary.md @@ -27,7 +27,8 @@ - `Flatpak source-manifest GitHub Release asset`: Implemented non-binary release asset set for the source-built `dev.crocoder.sce` Flatpak channel: `sce-v-flatpak-manifest.tar.gz`, `sce-v-flatpak-manifest.tar.gz.sha256`, and `sce-v-flatpak.json`, emitted by `nix run .#release-flatpak-package` and uploaded by `.github/workflows/release-sce.yml` from `dist/flatpak`. The tarball packages Flatpak manifest/support files (`dev.crocoder.sce.yml`, AppStream metadata, `cargo-sources.json`, and `git-host-bridge`) with the staged manifest pinned to the release commit; it is not a prebuilt `sce` binary, `.flatpak` bundle, OSTree repository, or Flathub publication. - `Flatpak bundle GitHub Release asset`: Implemented source-built `.flatpak` bundle release asset set for the `dev.crocoder.sce` Flatpak channel: `sce-v-x86_64.flatpak`, `sce-v-aarch64.flatpak`, plus per-architecture `.sha256` and `.json` metadata, built from source inside Flatpak using `flatpak-builder` + `flatpak build-bundle`. The bundle is a source-built Flatpak app for direct install (`flatpak install --user `), not a prebuilt binary or Flathub submission. These assets coexist with the existing Flatpak source-manifest packaging metadata on GitHub Releases. The `release-bundle` command is implemented in `packaging/flatpak/sce-flatpak.sh` and the GitHub workflow upload is implemented in `.github/workflows/release-sce-linux.yml` / `release-sce-linux-arm.yml` (x86_64/aarch64), assembled by `release-sce.yml`. - `cli flake native package`: Root-flake package output exposed as `packages.sce` with `packages.default = packages.sce`, producing the **native** development `sce` binary (`scePackage`) via `nix build .#sce` / `nix build .#default`. -- `cli flake release package`: Root-flake package output exposed as `packages.sce-release` (`sceReleasePackage`): the static-musl release binary on Linux and the native build on Darwin, built via `nix build .#sce-release`. On Linux it is a distinct store path from `packages.sce` and passes the native portability audit; `nix run .#release-artifacts` builds through this output. +- `cli flake release package`: Root-flake package output exposed as `packages.sce-release` (`sceReleasePackage`): the static-musl release binary (`scePackageMusl`) on Linux and a dedicated native-toolchain build (`sceReleasePackageNative`) on Darwin, built via `nix build .#sce-release`. On Linux it is a distinct store path from `packages.sce` and passes the native portability audit; on Darwin it is likewise distinct from `packages.sce` because it embeds the commit (see `release-only Git-commit embedding`). `nix run .#release-artifacts` builds through this output. +- `release-only Git-commit embedding`: Build-input contract where `SCE_GIT_COMMIT` is injected only into the release derivations via the `releaseCommitArgs` fragment (`scePackageMusl` on Linux, `sceReleasePackageNative` on Darwin) and never into `commonCargoArgs`. Native `packages.sce`/`packages.default` and the `cli-tests`/`cli-clippy`/`cli-fmt` checks therefore carry no commit in their inputs and stay cache-reusable across commits (native `sce version` reports `unknown`), while `.#sce-release` reports the real commit. `cli/build.rs` `emit_git_commit` emits the value only when the env var is explicitly set (sole rerun trigger `rerun-if-env-changed=SCE_GIT_COMMIT`, no `git rev-parse` fallback, no `.git` watches); `cli/src/services/version/mod.rs` reads it via `option_env!` with an `"unknown"` fallback. - `cli flake runnable app`: Root-flake app outputs where `apps.sce` and `apps.default` execute the native `${scePackage}/bin/sce` (dev ergonomics), supporting explicit `nix run .#sce -- ...` and default-app flows such as `nix run . -- ...` / `nix run github:crocoder-dev/shared-context-engineering -- ...`; `apps.sce-release` executes the release `${sceReleasePackage}/bin/sce` for explicit `nix run .#sce-release -- ...`. - `cli cargo install contract`: Supported Cargo install surface for the published `shared-context-engineering` crate, which installs the `sce` binary: crates.io (`cargo install shared-context-engineering --locked`), git (`cargo install --git https://github.com/crocoder-dev/shared-context-engineering shared-context-engineering --locked`), and local checkout (`cargo install --path cli --locked`). - `cli crates.io publication posture`: Current Cargo package posture in `cli/Cargo.toml` where crates.io-facing metadata is publication-ready for the `shared-context-engineering` crate, with crate-facing install guidance owned by `cli/README.md`. diff --git a/context/overview.md b/context/overview.md index 9b96d155..0248b94e 100644 --- a/context/overview.md +++ b/context/overview.md @@ -39,6 +39,7 @@ Claude bash-policy enforcement is also generated through `.claude/settings.json` Local database bootstrap is now owned by `LocalDbLifecycle::setup` and `AgentTraceDbLifecycle::setup` aggregated by the setup command. Agent Trace setup creates/reuses the current checkout ID for diagnostics and initializes the repository-scoped `/sce/repos//agent-trace.db` with the repository schema; hook runtime lazily creates or upgrades that repository DB when setup has not run or schema metadata is incomplete. Doctor validates the repository-scoped DB path/health and can bootstrap missing parent directories; outside a Git repository it reports an actionable "requires a Git repository" diagnostic instead of probing a sentinel path. The `sce trace` group operates only on repository-scoped DBs for list/status/status-all/shell UX; the checkout-scoped `--legacy` surface was removed by the `retire-legacy-agent-trace-db` plan (see `context/decisions/2026-07-17-retire-legacy-agent-trace-db.md`). Wiring a user-invocable `sce sync` command is deferred to `0.4.0`. The repository-root flake (`flake.nix`) now applies a Rust overlay-backed stable toolchain pinned to `1.95.0` (with `rustfmt` and `clippy`), reads package/check version from the repo-root `.version` file, builds `packages.sce` through a Crane `buildDepsOnly` + `buildPackage` pipeline with filtered package sources for the Cargo tree plus required embedded config/assets, and runs `cli-tests`, `cli-clippy`, and `cli-fmt` through Crane-backed check derivations (`cargoTest`, `cargoClippy`, `cargoFmt`) that reuse the same filtered source/toolchain setup. The root flake splits native and release outputs: `packages.sce` and `packages.default` build the **native** development binary (`scePackage`), while `packages.sce-release` builds the release binary (`sceReleasePackage`: static musl on Linux, native on Darwin). So `nix build .#sce` / `.#default`, `nix run . -- --help`, `nix run .#sce -- --help`, and `nix profile install github:crocoder-dev/shared-context-engineering` target the native binary, and `nix build .#sce-release` / `nix run .#sce-release -- ...` (plus `nix run .#release-artifacts`, which builds `.#sce-release`) target the release binary. On Linux the native and release outputs are distinct store paths, and the release output passes the native portability audit. +Git-commit embedding is **release-only**: `SCE_GIT_COMMIT` is injected via a `releaseCommitArgs` fragment applied only to the release derivations (`scePackageMusl` on Linux, `sceReleasePackageNative` on Darwin), not to `commonCargoArgs`. So native `.#sce`/`.#default` and every `nix flake check` derivation (`cli-tests`, `cli-clippy`, `cli-fmt`) build without the commit in their inputs and stay cache-reusable across commits (native `sce version` reports `unknown`), while `.#sce-release` still reports the real commit via `sce version`. `cli/build.rs` `emit_git_commit` emits `SCE_GIT_COMMIT` only when the env var is explicitly set — no `git rev-parse` fallback and no `.git/HEAD`/`.git/packed-refs` rerun watches. On Darwin the release now uses a distinct native-toolchain derivation (native toolchain + commit), so it diverges from `.#sce` to carry the commit while native stays commit-independent. The CLI Cargo package metadata now includes crates.io publication-ready fields with crate-local install guidance in `cli/README.md`; supported Cargo install paths are `cargo install shared-context-engineering --locked`, `cargo install --git https://github.com/crocoder-dev/shared-context-engineering shared-context-engineering --locked`, and local `cargo install --path cli --locked`. The published crate installs the `sce` binary. The crate also keeps `cargo clippy --manifest-path cli/Cargo.toml` warnings-denied through `cli/Cargo.toml` lint configuration, so an extra `-- -D warnings` flag is redundant. The repository-root flake is now the single Nix entrypoint for both repo tooling and CLI packaging/checks, so root-level `nix flake check` evaluates the Crane-backed CLI checks (`cli-tests`, `cli-clippy`, `cli-fmt`), Linux-only `flatpak-static-validation`, `workflow-actionlint` (runs `actionlint` on all `.github/workflows/*.yml`), plus six split JavaScript check derivations: `npm-bun-tests`, `npm-biome-check`, `npm-biome-format`, `config-lib-bun-tests`, `config-lib-biome-check`, and `config-lib-biome-format`, without nested-flake indirection. The config-lib checks now consume `config/lib/` as the shared Bun/TypeScript package root for both `agent-trace-plugin/` and `bash-policy-plugin`, with dependencies resolved from `config/lib/package.json` and `config/lib/bun.lock`. For Cargo packaging/builds, the crate now compiles against a temporary `cli/assets/generated/` mirror prepared from canonical `config/` outputs during Nix builds and crates.io publish runs rather than from a committed crate-local snapshot. Config-lib JS flake checks execute from `config/lib/`, but the copied Nix check source is repo-shaped when tests require shared repo fixtures; the current Claude agent-trace golden tests are fully Rust-owned in `cli/src/services/structured_patch/fixtures` (Claude TypeScript plugin test removed in T07). diff --git a/context/plans/nix-build-devshell-ci-restructure.md b/context/plans/nix-build-devshell-ci-restructure.md index cf482a27..55f47255 100644 --- a/context/plans/nix-build-devshell-ci-restructure.md +++ b/context/plans/nix-build-devshell-ci-restructure.md @@ -202,8 +202,37 @@ documented in `context/`. `readlink result*`; Linux portability audit on release binary; grep flake for lingering `sceReleasePackage` mis-wirings; `nix flake show`. -- [ ] T03: `Make Git-commit embedding release-only` (status:todo) +- [x] T03: `Make Git-commit embedding release-only` (status:done) - Task ID: T03 + - Status: done + - Completed: 2026-07-23 + - Files changed: cli/build.rs, flake.nix + - Evidence: `emit_git_commit` reduced to `rerun-if-env-changed=SCE_GIT_COMMIT` + plus emit-only-when-set (dropped `git rev-parse` fallback, `.git/HEAD` + + `.git/packed-refs` watches, and the now-unused `std::process::Command` + import). `flake.nix` removed `SCE_GIT_COMMIT` from `commonCargoArgs` and + introduced `releaseCommitArgs` applied only to `scePackageMusl` (Linux + release) and a new `sceReleasePackageNative` (Darwin release). Verified: + `nix flake check --print-build-logs` all checks passed (171 cli-tests, clippy, + fmt) — checks no longer receive the commit. Native `.#sce` commit-independent: + identical store path `wsmaa4qgsygmxx98p8hzd8r7lzfpf2fc-sce-0.3.2` across HEAD + change bcfd574→22da3ce (empty commit) with edits held in the working tree; + native `sce version` reports `unknown`. `.#sce-release` → + `633i78xj8ny6ykajbbjdwqb91zva9gzm-sce-0.3.2` (distinct from native) and + `sce version` reports the real commit `bcfd574aade4` (== `git rev-parse HEAD` + short-12). `nix run .#pkl-check-generated` up to date. + - Notes: On Linux native vs release are distinct paths (musl vs native). On + Darwin `.#sce-release` is now `sceReleasePackageNative` (native toolchain + + commit), intentionally diverging from the T02 "release == native" state so + release carries the commit while native `.#sce` stays commit-independent; + consistent with the plan's success criteria (distinct-path guarantee only + asserted on Linux). Darwin lane wiring-verified only (built on x86_64-linux). + `sce version` already used `option_env!("SCE_GIT_COMMIT")`→`"unknown"`, so no + app-side code change was needed. Cache-reuse proof used `git commit + --allow-empty` + `git reset --soft` with edits kept dirty (never stashed) to + avoid disturbing working-tree changes. Context-sync classification: important + (build-input contract change — native/checks commit-independence and + release-only commit embedding; architecture.md/overview.md describe this). - Goal: Only the release package receives the real commit via `SCE_GIT_COMMIT`; native builds, `cargo test`, Clippy, fmt, and other checks become commit-independent (cache-reusable across commits). diff --git a/flake.nix b/flake.nix index c774e533..f7edfd3f 100644 --- a/flake.nix +++ b/flake.nix @@ -270,10 +270,17 @@ nativeBuildInputs = [ rustToolchain ]; }; + # Commit embedding is release-only: SCE_GIT_COMMIT is applied via + # releaseCommitArgs to the release derivations only. Keeping it out of + # commonCargoArgs means the native package, cargo tests, Clippy, and fmt + # stay commit-independent (cache-reusable across commits). + releaseCommitArgs = { + SCE_GIT_COMMIT = shortGitCommit; + }; + commonCargoArgs = cargoBaseArgs // { pname = "sce"; src = workspaceSrc; - SCE_GIT_COMMIT = shortGitCommit; postUnpack = '' mkdir -p "$sourceRoot/cli/assets/generated/config" @@ -328,7 +335,7 @@ cargoArtifactsMusl = craneLibMusl.buildDepsOnly cargoDepsArgsMusl; scePackageMusl = craneLibMusl.buildPackage ( - (commonCargoArgs // muslEnvVars) + (commonCargoArgs // muslEnvVars // releaseCommitArgs) // { inherit cargoArtifactsMusl; nativeBuildInputs = [ @@ -361,9 +368,27 @@ } ); - # Release package selection: musl on Linux (fully static, no /nix/store - # runtime references), gnu on macOS (unchanged). - sceReleasePackage = if pkgs.stdenv.isLinux then scePackageMusl else scePackage; + # Native-toolchain release build for Darwin. Identical to scePackage + # except it embeds the real Git commit (SCE_GIT_COMMIT), so on macOS the + # release output reports the commit while native `.#sce` stays + # commit-independent. cargoArtifacts is deps-only and unaffected by + # SCE_GIT_COMMIT (only the cli crate's build.rs reads it), so it is + # shared with scePackage at no extra rebuild cost. + sceReleasePackageNative = craneLib.buildPackage ( + commonCargoArgs + // releaseCommitArgs + // { + inherit cargoArtifacts; + meta = { + mainProgram = "sce"; + description = "Shared Context Engineering CLI (release)"; + }; + } + ); + + # Release package selection: static musl on Linux (fully static, no + # /nix/store runtime references), native-with-commit on macOS. + sceReleasePackage = if pkgs.stdenv.isLinux then scePackageMusl else sceReleasePackageNative; tursoCargoArgs = { pname = "turso"; From fc3774cf606d1adb90b620f94e9b7c03e7679c58 Mon Sep 17 00:00:00 2001 From: David Abram Date: Thu, 23 Jul 2026 17:02:01 +0200 Subject: [PATCH 3/6] nix: Slim default development shell Exclude SCE and Turso packages from the default shell to avoid unnecessary builds. Add an opt-in database shell that layers Turso onto shared development tooling. Co-authored-by: SCE --- context/overview.md | 1 + .../nix-build-devshell-ci-restructure.md | 22 +++- flake.nix | 106 ++++++++++-------- 3 files changed, 82 insertions(+), 47 deletions(-) diff --git a/context/overview.md b/context/overview.md index 0248b94e..b6fc9500 100644 --- a/context/overview.md +++ b/context/overview.md @@ -40,6 +40,7 @@ Local database bootstrap is now owned by `LocalDbLifecycle::setup` and `AgentTra The repository-root flake (`flake.nix`) now applies a Rust overlay-backed stable toolchain pinned to `1.95.0` (with `rustfmt` and `clippy`), reads package/check version from the repo-root `.version` file, builds `packages.sce` through a Crane `buildDepsOnly` + `buildPackage` pipeline with filtered package sources for the Cargo tree plus required embedded config/assets, and runs `cli-tests`, `cli-clippy`, and `cli-fmt` through Crane-backed check derivations (`cargoTest`, `cargoClippy`, `cargoFmt`) that reuse the same filtered source/toolchain setup. The root flake splits native and release outputs: `packages.sce` and `packages.default` build the **native** development binary (`scePackage`), while `packages.sce-release` builds the release binary (`sceReleasePackage`: static musl on Linux, native on Darwin). So `nix build .#sce` / `.#default`, `nix run . -- --help`, `nix run .#sce -- --help`, and `nix profile install github:crocoder-dev/shared-context-engineering` target the native binary, and `nix build .#sce-release` / `nix run .#sce-release -- ...` (plus `nix run .#release-artifacts`, which builds `.#sce-release`) target the release binary. On Linux the native and release outputs are distinct store paths, and the release output passes the native portability audit. Git-commit embedding is **release-only**: `SCE_GIT_COMMIT` is injected via a `releaseCommitArgs` fragment applied only to the release derivations (`scePackageMusl` on Linux, `sceReleasePackageNative` on Darwin), not to `commonCargoArgs`. So native `.#sce`/`.#default` and every `nix flake check` derivation (`cli-tests`, `cli-clippy`, `cli-fmt`) build without the commit in their inputs and stay cache-reusable across commits (native `sce version` reports `unknown`), while `.#sce-release` still reports the real commit via `sce version`. `cli/build.rs` `emit_git_commit` emits `SCE_GIT_COMMIT` only when the env var is explicitly set — no `git rev-parse` fallback and no `.git/HEAD`/`.git/packed-refs` rerun watches. On Darwin the release now uses a distinct native-toolchain derivation (native toolchain + commit), so it diverges from `.#sce` to carry the commit while native stays commit-independent. +The default development shell is slimmed for fast iteration: `devShells.default` no longer includes `scePackage` or `tursoPackage`, so `nix develop` compiles neither the CLI package nor the Turso CLI — it provides only the Rust toolchain and JS/pkl tooling for `cargo`/`biome`/`pkl` work. Turso stays available as `packages..turso` and through a new opt-in `devShells..database` shell (default tools + `tursoPackage`), entered via `nix develop .#database`. Both shells share `defaultDevShellPackages`/`defaultDevShellHook` `let` bindings so they cannot drift. The CLI Cargo package metadata now includes crates.io publication-ready fields with crate-local install guidance in `cli/README.md`; supported Cargo install paths are `cargo install shared-context-engineering --locked`, `cargo install --git https://github.com/crocoder-dev/shared-context-engineering shared-context-engineering --locked`, and local `cargo install --path cli --locked`. The published crate installs the `sce` binary. The crate also keeps `cargo clippy --manifest-path cli/Cargo.toml` warnings-denied through `cli/Cargo.toml` lint configuration, so an extra `-- -D warnings` flag is redundant. The repository-root flake is now the single Nix entrypoint for both repo tooling and CLI packaging/checks, so root-level `nix flake check` evaluates the Crane-backed CLI checks (`cli-tests`, `cli-clippy`, `cli-fmt`), Linux-only `flatpak-static-validation`, `workflow-actionlint` (runs `actionlint` on all `.github/workflows/*.yml`), plus six split JavaScript check derivations: `npm-bun-tests`, `npm-biome-check`, `npm-biome-format`, `config-lib-bun-tests`, `config-lib-biome-check`, and `config-lib-biome-format`, without nested-flake indirection. The config-lib checks now consume `config/lib/` as the shared Bun/TypeScript package root for both `agent-trace-plugin/` and `bash-policy-plugin`, with dependencies resolved from `config/lib/package.json` and `config/lib/bun.lock`. For Cargo packaging/builds, the crate now compiles against a temporary `cli/assets/generated/` mirror prepared from canonical `config/` outputs during Nix builds and crates.io publish runs rather than from a committed crate-local snapshot. Config-lib JS flake checks execute from `config/lib/`, but the copied Nix check source is repo-shaped when tests require shared repo fixtures; the current Claude agent-trace golden tests are fully Rust-owned in `cli/src/services/structured_patch/fixtures` (Claude TypeScript plugin test removed in T07). diff --git a/context/plans/nix-build-devshell-ci-restructure.md b/context/plans/nix-build-devshell-ci-restructure.md index 55f47255..3f415bd6 100644 --- a/context/plans/nix-build-devshell-ci-restructure.md +++ b/context/plans/nix-build-devshell-ci-restructure.md @@ -252,8 +252,28 @@ documented in `context/`. (cache hit); `nix build .#sce-release && ./result/bin/sce version` shows the commit; `nix flake check --print-build-logs`. -- [ ] T04: `Slim default devShell; add opt-in database shell` (status:todo) +- [x] T04: `Slim default devShell; add opt-in database shell` (status:done) - Task ID: T04 + - Status: done + - Completed: 2026-07-23 + - Files changed: flake.nix + - Evidence: `nix flake check --no-build` all checks passed; `nix flake show` + lists both `devShells..default` and `devShells..database`. + `nix develop -c true --print-build-logs` produced no `scePackage`/`turso_cli` + compile. `nix develop .#database -c turso --version` → `Turso 0.7.0`. `nix + build .#turso` still builds (`k9balz…-turso-0.7.0`). Default shell still + provides `rustc`/`pkl`/JS tooling. (Note: `sce`/`turso` visible on PATH inside + `nix develop` are inherited from the invoking parent shell — identical store + paths `za1ki…-sce-0.3.2` / `k9balz…-turso-0.7.0` are already on the outer + PATH — not contributed by the slimmed default shell.) + - Notes: Factored the shared package list + shellHook into `let` bindings + `defaultDevShellPackages` / `defaultDevShellHook` so `devShells.database` + reuses them and layers `tursoPackage` + a `turso` version echo without drift. + Removed `scePackage`/`tursoPackage` and the `sce`/`turso` echo lines from the + default shell. Darwin lane wiring-verified only (built on x86_64-linux). + Context-sync classification: important (public devShell-output contract change — + new `devShells.database`, slimmed default shell; architecture/overview may + reference default-shell contents). - Goal: `nix develop` no longer builds `scePackage` or `turso_cli`; Turso stays available as `packages..turso` and via a new `devShells..database` shell. diff --git a/flake.nix b/flake.nix index f7edfd3f..38ee2c5b 100644 --- a/flake.nix +++ b/flake.nix @@ -426,6 +426,55 @@ }; }; + # Shared development tooling for the default shell. Excludes `scePackage` + # and `tursoPackage` so `nix develop` does not compile the package or the + # Turso CLI; `devShells.database` layers Turso back on when needed. + defaultDevShellPackages = + (with pkgs; [ + biome + bunPackage + jq + pkl + pkl-lsp + typescript + typescript-language-server + vscode-json-languageserver + rust-analyzer + ]) + ++ pkgs.lib.optionals pkgs.stdenv.isLinux [ + pkgs.appstream + pkgs.flatpak + pkgs.flatpak-builder + ] + ++ [ rustToolchain ]; + + defaultDevShellHook = '' + version_of() { + "$1" --version 2>/dev/null | awk 'match($0, /[0-9]+(\.[0-9]+)+/) { print substr($0, RSTART, RLENGTH); exit }' + } + echo "- bun: $(version_of bun)" + echo "- biome: $(version_of biome)" + echo "- pkl: $(version_of pkl)" + echo "- pkl-lsp: $(version_of pkl-lsp)" + echo "- tsc: $(version_of tsc)" + echo "- tsserver-lsp: $(version_of typescript-language-server)" + echo "- rust: $(version_of rustc)" + echo "- pkl-generate: nix run .#pkl-generate" + echo "- pkl-check-generated: nix run .#pkl-check-generated" + echo "- release-artifacts: nix run .#release-artifacts -- --help" + echo "- native-portability-audit: nix run .#native-portability-audit -- --help" + echo "- release-manifest: nix run .#release-manifest -- --help" + echo "- release-npm-package: nix run .#release-npm-package -- --help" + ${pkgs.lib.optionalString pkgs.stdenv.isLinux '' + echo "- flatpak: $(version_of flatpak)" + echo "- flatpak-builder: $(version_of flatpak-builder)" + echo "- appstreamcli: $(version_of appstreamcli)" + echo "- sce-flatpak: nix run .#sce-flatpak -- --help" + echo "- release-flatpak-package: nix run .#release-flatpak-package -- --help" + echo "- release-flatpak-bundle: nix run .#release-flatpak-bundle -- --help" + ''} + ''; + pklCheckGeneratedApp = pkgs.writeShellApplication { name = "pkl-check-generated"; runtimeInputs = [ @@ -1551,55 +1600,20 @@ }; devShells.default = pkgs.mkShell { - packages = - with pkgs; - [ - biome - bunPackage - jq - pkl - pkl-lsp - typescript - typescript-language-server - vscode-json-languageserver - rust-analyzer - scePackage - tursoPackage - ] - ++ pkgs.lib.optionals pkgs.stdenv.isLinux [ - appstream - flatpak - flatpak-builder - ] - ++ [ rustToolchain ]; + packages = defaultDevShellPackages; + + shellHook = defaultDevShellHook; + }; + + # Opt-in shell layering the Turso CLI on top of the default tools, so + # `nix develop` stays fast (no `scePackage`/`turso_cli` compile) while + # database work can pull Turso in explicitly via `nix develop .#database`. + devShells.database = pkgs.mkShell { + packages = defaultDevShellPackages ++ [ tursoPackage ]; shellHook = '' - version_of() { - "$1" --version 2>/dev/null | awk 'match($0, /[0-9]+(\.[0-9]+)+/) { print substr($0, RSTART, RLENGTH); exit }' - } - echo "- bun: $(version_of bun)" - echo "- biome: $(version_of biome)" - echo "- pkl: $(version_of pkl)" - echo "- pkl-lsp: $(version_of pkl-lsp)" - echo "- tsc: $(version_of tsc)" - echo "- tsserver-lsp: $(version_of typescript-language-server)" - echo "- rust: $(version_of rustc)" - echo "- sce: $(version_of sce)" + ${defaultDevShellHook} echo "- turso: $(version_of turso)" - echo "- pkl-generate: nix run .#pkl-generate" - echo "- pkl-check-generated: nix run .#pkl-check-generated" - echo "- release-artifacts: nix run .#release-artifacts -- --help" - echo "- native-portability-audit: nix run .#native-portability-audit -- --help" - echo "- release-manifest: nix run .#release-manifest -- --help" - echo "- release-npm-package: nix run .#release-npm-package -- --help" - ${pkgs.lib.optionalString pkgs.stdenv.isLinux '' - echo "- flatpak: $(version_of flatpak)" - echo "- flatpak-builder: $(version_of flatpak-builder)" - echo "- appstreamcli: $(version_of appstreamcli)" - echo "- sce-flatpak: nix run .#sce-flatpak -- --help" - echo "- release-flatpak-package: nix run .#release-flatpak-package -- --help" - echo "- release-flatpak-bundle: nix run .#release-flatpak-bundle -- --help" - ''} ''; }; } From e30c11c70c271364aa8f7f6db47d4a880ceec2ff Mon Sep 17 00:00:00 2001 From: David Abram Date: Thu, 23 Jul 2026 17:11:34 +0200 Subject: [PATCH 4/6] cli: Use include_bytes! for embedded asset payloads Reduce generated manifest size while preserving asset hashes, ordering, and rebuild tracking. Co-authored-by: SCE --- cli/build.rs | 7 ++++-- .../nix-build-devshell-ci-restructure.md | 23 ++++++++++++++++++- 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/cli/build.rs b/cli/build.rs index df9d03ef..395875d4 100644 --- a/cli/build.rs +++ b/cli/build.rs @@ -166,11 +166,14 @@ fn generate_embedded_asset_manifest() -> io::Result<()> { println!("cargo:rerun-if-changed={}", file.absolute_path.display()); let bytes = fs::read(&file.absolute_path)?; let sha256 = compute_sha256(&bytes); + let include_path = format!("{}/{}", target.relative_root, file.relative_path); writeln!( output, - " EmbeddedAsset {{ relative_path: \"{}\", bytes: {}, sha256: {} }},", + " EmbeddedAsset {{ relative_path: \"{}\", \ + bytes: include_bytes!(concat!(env!(\"CARGO_MANIFEST_DIR\"), \"/{}\")), \ + sha256: {} }},", escape_for_rust_string(&file.relative_path), - format_byte_literal("&[", &bytes), + escape_for_rust_string(&include_path), format_byte_literal("[", &sha256), ) .expect("writing to String buffer should never fail"); diff --git a/context/plans/nix-build-devshell-ci-restructure.md b/context/plans/nix-build-devshell-ci-restructure.md index 3f415bd6..42aff373 100644 --- a/context/plans/nix-build-devshell-ci-restructure.md +++ b/context/plans/nix-build-devshell-ci-restructure.md @@ -290,8 +290,29 @@ documented in `context/`. `--print-build-logs` shows no sce/turso compile; `nix develop .#database -c true`; `nix flake show` lists both shells; `nix flake check --no-build`. -- [ ] T05: `Generate include_bytes! asset literals in build.rs` (status:todo) +- [x] T05: `Generate include_bytes! asset literals in build.rs` (status:done) - Task ID: T05 + - Status: done + - Completed: 2026-07-23 + - Files changed: cli/build.rs, context/tmp/flake-speedup-benchmarks.md + - Evidence: `generate_embedded_asset_manifest` now emits + `bytes: include_bytes!(concat!(env!("CARGO_MANIFEST_DIR"), "//"))` + per asset (path built as `{target.relative_root}/{file.relative_path}`, + escaped via `escape_for_rust_string`), while still `fs::read`-ing each file to + compute the embedded SHA-256 byte literal and still emitting per-file + `cargo:rerun-if-changed`; ordering and relative paths unchanged. `include_bytes!`'s + `&[u8; N]` coerces to the `EmbeddedAsset.bytes: &'static [u8]` field at the + struct-literal site, so no consumer change in `setup/mod.rs` was needed. + Generated `setup_embedded_assets.rs` shrank 925,983 B → 21,785 B (−97.6%); + incremental build after touching `build.rs` ~3 s. `nix flake check + --print-build-logs` all checks passed (171 cli-tests + clippy + fmt in ~24 s), + confirming assets remain byte-for-byte identical via the embedded-asset SHA-256 + tests. Before/after size + build-time recorded in the benchmark doc. + - Notes: `format_byte_literal` retained (still used for the SHA-256 array; the + `&[`-prefixed byte-array use for `bytes` is gone). Migrations manifest already + used `include_str!` and was untouched. Context-sync classification: verify-only + (internal build-script refactor; no runtime behavior, public contract, + architecture, or terminology change; assets byte-for-byte identical). - Goal: Replace generated `bytes: &[0x.., ..]` arrays with generated `include_bytes!(concat!(env!("CARGO_MANIFEST_DIR"), "/assets/generated/..."))` expressions, preserving all embedded-asset behavior. From 486494c100f6cd17031f291eac650b23b3f02500 Mon Sep 17 00:00:00 2001 From: David Abram Date: Thu, 23 Jul 2026 17:23:54 +0200 Subject: [PATCH 5/6] flake: Add explicit CI checks aggregate Expose the release build and Linux portability audit through packages.ci-checks while keeping expensive validation out of nix flake check. Co-authored-by: SCE --- context/architecture.md | 1 + context/glossary.md | 1 + context/overview.md | 2 +- .../nix-build-devshell-ci-restructure.md | 28 ++++++++++++- flake.nix | 41 +++++++++++++++++++ 5 files changed, 71 insertions(+), 2 deletions(-) diff --git a/context/architecture.md b/context/architecture.md index 206d474a..53ce8661 100644 --- a/context/architecture.md +++ b/context/architecture.md @@ -140,6 +140,7 @@ The repository includes a new placeholder Rust binary crate at `cli/`. - `flake.nix` applies `rust-overlay` (`oxalica/rust-overlay`) to nixpkgs, pins `rust-bin.stable.1.95.0.default` with `rustfmt` + `clippy`, reads the package/check version from repo-root `.version`, builds `packages.sce` through Crane (`buildDepsOnly` -> `buildPackage`) with shared host Cargo metadata plus a filtered repo-root source that preserves the Cargo tree, `cli/assets/hooks`, and only the generated config/schema inputs required by CLI embedding, then injects generated OpenCode/Claude/Pi config payloads and schema inputs into a temporary `cli/assets/generated/` mirror during derivation unpack so `cli/build.rs` can package the crate without requiring committed generated crate assets. The same `cli/build.rs` now scans `cli/migrations/*/*.sql` and writes `cli/src/generated_migrations.rs` with deterministic migration constants sorted by numeric filename prefix. - The root flake runs `cli-tests`, `cli-clippy`, and `cli-fmt` through Crane-backed paths for the CLI crate; `cli-fmt` intentionally uses the CLI-only Cargo source set rather than the package source with generated config/assets so formatting is not invalidated by release/config asset churn. The flake also exposes directory-scoped JS validation derivations for both `npm/` and the shared `config/lib/` plugin package root, and `pkl-parity` uses a narrowed Pkl/generated-output source set instead of copying the entire repository. - Git-commit embedding is release-only. `SCE_GIT_COMMIT` is injected via a `releaseCommitArgs` fragment (`{ SCE_GIT_COMMIT = shortGitCommit; }`) merged only into the release derivations — `scePackageMusl` on Linux and a dedicated native-toolchain `sceReleasePackageNative` on Darwin — and is deliberately absent from `commonCargoArgs`. As a result native `packages.sce`/`packages.default` and the `cli-tests`/`cli-clippy`/`cli-fmt` check derivations carry no commit in their inputs and stay cache-reusable across commits (native `sce version` reports `unknown`), while `.#sce-release` reports the real commit. `cli/build.rs` `emit_git_commit` emits the `rustc-env` value only when `SCE_GIT_COMMIT` is explicitly set (with `rerun-if-env-changed=SCE_GIT_COMMIT` as its sole rerun trigger) — no `git rev-parse` fallback and no `.git/HEAD`/`.git/packed-refs` watches — and `cli/src/services/version/mod.rs` resolves the commit through `option_env!("SCE_GIT_COMMIT")` with an `"unknown"` fallback. On Darwin the release derivation therefore diverges from `.#sce` (shared deps `cargoArtifacts`, but distinct final crate carrying the commit). +- `flake.nix` exposes `packages..ci-checks` (`ciChecks`) as the explicit long-running validation tier so the expensive work lives behind `nix build .#ci-checks` and never folds into `nix flake check`. It is a `runCommand` aggregate that symlinks its primary member `sceReleasePackage` into `$out/sce-release` (forcing the static-musl release build on Linux, native on Darwin) and, on Linux only, symlinks `releasePortabilityAuditCheck` into `$out/release-portability-audit`. That Linux-only `releasePortabilityAuditCheck` derivation runs `nix/release/native-portability-audit.sh --platform linux --binary ${sceReleasePackage}/bin/sce` (with `binutils`+`coreutils` on PATH) so `nix build .#ci-checks` fails when the real release binary carries forbidden `/nix/store/` references. This is distinct from `checks..native-portability-audit` (`nativePortabilityAuditCheck`), which remains a fixture-based unit test of the audit script on both platforms. CI wiring to `.#ci-checks` is owned by a separate release-validation job (see the CI restructure plan T07). - Flatpak flake tooling is Linux-only and reduced to a minimal app surface: `apps.sce-flatpak` is the umbrella entrypoint (`nix run .#sce-flatpak -- ` for `validate`, `prepare-local-manifest --repo-root `, etc.) that delegates to `packaging/flatpak/sce-flatpak.sh`; `apps.release-flatpak-package` emits deterministic Flatpak GitHub Release source-manifest assets; `apps.release-flatpak-bundle` builds a source-built `.flatpak` bundle; helper apps `apps.regenerate-flatpak-manifest` and `apps.regenerate-cargo-sources` rewrite the checked-in generated artifacts from their Nix sources. The standalone `flatpak-validate`, `flatpak-local-manifest`, and `flatpak-build` wrapper apps are removed in favor of `sce-flatpak `. `checks..flatpak-static-validation` runs the Nix-built static validator script during default `nix flake check`, and `checks..flatpak-manifest-parity` plus `checks..cargo-sources-parity` enforce the generated-artifact parity contracts; default `nix flake check` does not run `flatpak-builder`. The Linux dev shell includes `appstreamcli`, `flatpak`, and `flatpak-builder`. - The shared config-lib source set is rooted at `config/lib/` and includes the shared `package.json`, `bun.lock`, and `tsconfig.json` plus `agent-trace-plugin/` and `bash-policy-plugin/`; `config-lib-bun-tests` runs the bash-policy plugin wrapper tests from that shared root, while `config-lib-biome-check` and `config-lib-biome-format` run Biome over the copied shared package source. `.github/workflows/publish-crates.yml` follows the same asset-preparation rule but runs Cargo packaging from a temporary clean repository copy so crates.io publish no longer needs `--allow-dirty`. - The config-lib check source preserves repo-relative access to shared CLI patch fixtures: Nix copies a filtered repo-shaped source containing `config/lib/**` plus `cli/src/services/structured_patch/fixtures`, then runs Bun/Biome from `config/lib/`. diff --git a/context/glossary.md b/context/glossary.md index 772eb8e5..272c3d29 100644 --- a/context/glossary.md +++ b/context/glossary.md @@ -28,6 +28,7 @@ - `Flatpak bundle GitHub Release asset`: Implemented source-built `.flatpak` bundle release asset set for the `dev.crocoder.sce` Flatpak channel: `sce-v-x86_64.flatpak`, `sce-v-aarch64.flatpak`, plus per-architecture `.sha256` and `.json` metadata, built from source inside Flatpak using `flatpak-builder` + `flatpak build-bundle`. The bundle is a source-built Flatpak app for direct install (`flatpak install --user `), not a prebuilt binary or Flathub submission. These assets coexist with the existing Flatpak source-manifest packaging metadata on GitHub Releases. The `release-bundle` command is implemented in `packaging/flatpak/sce-flatpak.sh` and the GitHub workflow upload is implemented in `.github/workflows/release-sce-linux.yml` / `release-sce-linux-arm.yml` (x86_64/aarch64), assembled by `release-sce.yml`. - `cli flake native package`: Root-flake package output exposed as `packages.sce` with `packages.default = packages.sce`, producing the **native** development `sce` binary (`scePackage`) via `nix build .#sce` / `nix build .#default`. - `cli flake release package`: Root-flake package output exposed as `packages.sce-release` (`sceReleasePackage`): the static-musl release binary (`scePackageMusl`) on Linux and a dedicated native-toolchain build (`sceReleasePackageNative`) on Darwin, built via `nix build .#sce-release`. On Linux it is a distinct store path from `packages.sce` and passes the native portability audit; on Darwin it is likewise distinct from `packages.sce` because it embeds the commit (see `release-only Git-commit embedding`). `nix run .#release-artifacts` builds through this output. +- `cli flake ci-checks aggregate`: Root-flake package output exposed as `packages..ci-checks` (`ciChecks`), the explicit long-running validation tier run via `nix build .#ci-checks`. It is a `runCommand` symlink aggregate whose primary member is `sceReleasePackage` (forcing the static-musl release build) and which, on Linux only, also forces `releasePortabilityAuditCheck` — a derivation running `nix/release/native-portability-audit.sh` against the real `${sceReleasePackage}/bin/sce` binary. Distinct from the fixture-based `native-portability-audit` flake check. `nix flake check` never builds `.#sce-release`, keeping the expensive work behind this aggregate. CI wiring is deferred to the release-validation job (plan T07). - `release-only Git-commit embedding`: Build-input contract where `SCE_GIT_COMMIT` is injected only into the release derivations via the `releaseCommitArgs` fragment (`scePackageMusl` on Linux, `sceReleasePackageNative` on Darwin) and never into `commonCargoArgs`. Native `packages.sce`/`packages.default` and the `cli-tests`/`cli-clippy`/`cli-fmt` checks therefore carry no commit in their inputs and stay cache-reusable across commits (native `sce version` reports `unknown`), while `.#sce-release` reports the real commit. `cli/build.rs` `emit_git_commit` emits the value only when the env var is explicitly set (sole rerun trigger `rerun-if-env-changed=SCE_GIT_COMMIT`, no `git rev-parse` fallback, no `.git` watches); `cli/src/services/version/mod.rs` reads it via `option_env!` with an `"unknown"` fallback. - `cli flake runnable app`: Root-flake app outputs where `apps.sce` and `apps.default` execute the native `${scePackage}/bin/sce` (dev ergonomics), supporting explicit `nix run .#sce -- ...` and default-app flows such as `nix run . -- ...` / `nix run github:crocoder-dev/shared-context-engineering -- ...`; `apps.sce-release` executes the release `${sceReleasePackage}/bin/sce` for explicit `nix run .#sce-release -- ...`. - `cli cargo install contract`: Supported Cargo install surface for the published `shared-context-engineering` crate, which installs the `sce` binary: crates.io (`cargo install shared-context-engineering --locked`), git (`cargo install --git https://github.com/crocoder-dev/shared-context-engineering shared-context-engineering --locked`), and local checkout (`cargo install --path cli --locked`). diff --git a/context/overview.md b/context/overview.md index b6fc9500..77d19f69 100644 --- a/context/overview.md +++ b/context/overview.md @@ -38,7 +38,7 @@ Generated config now includes repo-local plugin assets for both profiles: `sce-b Claude bash-policy enforcement is also generated through `.claude/settings.json` as a `PreToolUse` `Bash` command hook running `sce policy bash`, so Claude and OpenCode both delegate to the Rust policy evaluator without a Claude TypeScript runtime. Pi bash-policy enforcement is delegated the same way through a project-local Pi extension (`config/lib/pi-plugin/sce-pi-extension.ts`, emitted to `config/.pi/extensions/sce/index.ts`) whose `tool_call` handler blocks denied bash commands via `sce policy bash` and fails open when the policy check cannot run (see `context/sce/pi-extension-runtime.md`). Local database bootstrap is now owned by `LocalDbLifecycle::setup` and `AgentTraceDbLifecycle::setup` aggregated by the setup command. Agent Trace setup creates/reuses the current checkout ID for diagnostics and initializes the repository-scoped `/sce/repos//agent-trace.db` with the repository schema; hook runtime lazily creates or upgrades that repository DB when setup has not run or schema metadata is incomplete. Doctor validates the repository-scoped DB path/health and can bootstrap missing parent directories; outside a Git repository it reports an actionable "requires a Git repository" diagnostic instead of probing a sentinel path. The `sce trace` group operates only on repository-scoped DBs for list/status/status-all/shell UX; the checkout-scoped `--legacy` surface was removed by the `retire-legacy-agent-trace-db` plan (see `context/decisions/2026-07-17-retire-legacy-agent-trace-db.md`). Wiring a user-invocable `sce sync` command is deferred to `0.4.0`. The repository-root flake (`flake.nix`) now applies a Rust overlay-backed stable toolchain pinned to `1.95.0` (with `rustfmt` and `clippy`), reads package/check version from the repo-root `.version` file, builds `packages.sce` through a Crane `buildDepsOnly` + `buildPackage` pipeline with filtered package sources for the Cargo tree plus required embedded config/assets, and runs `cli-tests`, `cli-clippy`, and `cli-fmt` through Crane-backed check derivations (`cargoTest`, `cargoClippy`, `cargoFmt`) that reuse the same filtered source/toolchain setup. -The root flake splits native and release outputs: `packages.sce` and `packages.default` build the **native** development binary (`scePackage`), while `packages.sce-release` builds the release binary (`sceReleasePackage`: static musl on Linux, native on Darwin). So `nix build .#sce` / `.#default`, `nix run . -- --help`, `nix run .#sce -- --help`, and `nix profile install github:crocoder-dev/shared-context-engineering` target the native binary, and `nix build .#sce-release` / `nix run .#sce-release -- ...` (plus `nix run .#release-artifacts`, which builds `.#sce-release`) target the release binary. On Linux the native and release outputs are distinct store paths, and the release output passes the native portability audit. +The root flake splits native and release outputs: `packages.sce` and `packages.default` build the **native** development binary (`scePackage`), while `packages.sce-release` builds the release binary (`sceReleasePackage`: static musl on Linux, native on Darwin). So `nix build .#sce` / `.#default`, `nix run . -- --help`, `nix run .#sce -- --help`, and `nix profile install github:crocoder-dev/shared-context-engineering` target the native binary, and `nix build .#sce-release` / `nix run .#sce-release -- ...` (plus `nix run .#release-artifacts`, which builds `.#sce-release`) target the release binary. On Linux the native and release outputs are distinct store paths, and the release output passes the native portability audit. `packages..ci-checks` is the explicit long-running validation tier: `nix build .#ci-checks` builds the `.#sce-release` package and, on Linux, audits the real release binary for forbidden `/nix/store/` references, so the expensive work stays out of `nix flake check` (which never builds `.#sce-release`). Git-commit embedding is **release-only**: `SCE_GIT_COMMIT` is injected via a `releaseCommitArgs` fragment applied only to the release derivations (`scePackageMusl` on Linux, `sceReleasePackageNative` on Darwin), not to `commonCargoArgs`. So native `.#sce`/`.#default` and every `nix flake check` derivation (`cli-tests`, `cli-clippy`, `cli-fmt`) build without the commit in their inputs and stay cache-reusable across commits (native `sce version` reports `unknown`), while `.#sce-release` still reports the real commit via `sce version`. `cli/build.rs` `emit_git_commit` emits `SCE_GIT_COMMIT` only when the env var is explicitly set — no `git rev-parse` fallback and no `.git/HEAD`/`.git/packed-refs` rerun watches. On Darwin the release now uses a distinct native-toolchain derivation (native toolchain + commit), so it diverges from `.#sce` to carry the commit while native stays commit-independent. The default development shell is slimmed for fast iteration: `devShells.default` no longer includes `scePackage` or `tursoPackage`, so `nix develop` compiles neither the CLI package nor the Turso CLI — it provides only the Rust toolchain and JS/pkl tooling for `cargo`/`biome`/`pkl` work. Turso stays available as `packages..turso` and through a new opt-in `devShells..database` shell (default tools + `tursoPackage`), entered via `nix develop .#database`. Both shells share `defaultDevShellPackages`/`defaultDevShellHook` `let` bindings so they cannot drift. The CLI Cargo package metadata now includes crates.io publication-ready fields with crate-local install guidance in `cli/README.md`; supported Cargo install paths are `cargo install shared-context-engineering --locked`, `cargo install --git https://github.com/crocoder-dev/shared-context-engineering shared-context-engineering --locked`, and local `cargo install --path cli --locked`. The published crate installs the `sce` binary. The crate also keeps `cargo clippy --manifest-path cli/Cargo.toml` warnings-denied through `cli/Cargo.toml` lint configuration, so an extra `-- -D warnings` flag is redundant. diff --git a/context/plans/nix-build-devshell-ci-restructure.md b/context/plans/nix-build-devshell-ci-restructure.md index 42aff373..2195a4f0 100644 --- a/context/plans/nix-build-devshell-ci-restructure.md +++ b/context/plans/nix-build-devshell-ci-restructure.md @@ -330,8 +330,34 @@ documented in `context/`. record clean vs incremental Rust build time and generated-source size before vs after in the benchmark doc. -- [ ] T06: `Add explicit ci-checks aggregate output` (status:todo) +- [x] T06: `Add explicit ci-checks aggregate output` (status:done) - Task ID: T06 + - Status: done + - Completed: 2026-07-23 + - Files changed: flake.nix + - Evidence: Added `packages..ci-checks` (`ciChecks`) — a `runCommand` + aggregate that symlinks `sceReleasePackage` into `$out/sce-release` (forcing + the static-musl release build) and, on Linux, `releasePortabilityAuditCheck` + into `$out/release-portability-audit`. The new Linux-only + `releasePortabilityAuditCheck` runs `nix/release/native-portability-audit.sh + --platform linux --binary ${sceReleasePackage}/bin/sce` (binutils+coreutils + on PATH), auditing the real release binary — distinct from the fixture-based + `nativePortabilityAuditCheck` in `checks`. Verified: `nix build .#ci-checks + --print-build-logs` built the musl release (`l80z8…-sce-0.3.2`) and the audit + derivation printed "Native binary portability audit passed: … has no forbidden + /nix/store/ runtime references for linux"; `result/` holds both symlinks. + `nix flake check --no-build` → all checks passed without forcing `.#sce-release` + (ci-checks lives in `packages`, not `checks`). `nix flake show` lists + `packages..ci-checks` on all four systems. `nix run + .#pkl-check-generated` up to date. + - Notes: Aggregate placed in the shared `let` scope after + `nativePortabilityAuditCheck`. On Darwin `ci-checks` builds only the release + package (native-on-Darwin); the real-binary audit is Linux-guarded via + `pkgs.lib.optionalString pkgs.stdenv.isLinux`, matching the plan's Linux-only + portability guarantee. Darwin lane wiring-verified only (built on x86_64-linux). + Context-sync classification: important (public flake-output contract change — + new `packages.ci-checks` aggregate; architecture/overview describe the CI + validation tiers). CI wiring to this output is deferred to T07. - Goal: Expose `packages..ci-checks` as the explicit long-running validation tier whose primary member is the static-musl release build (plus the Linux release portability audit), so `nix build .#ci-checks` is the diff --git a/flake.nix b/flake.nix index 38ee2c5b..116453a1 100644 --- a/flake.nix +++ b/flake.nix @@ -1398,6 +1398,46 @@ mkdir -p "$out" ''; + # Linux-only: audit the real static-musl release binary for forbidden + # /nix/store runtime references. Unlike nativePortabilityAuditCheck + # (a fixture-based unit test of the audit script), this inspects the + # actual sceReleasePackage binary. + releasePortabilityAuditCheck = + pkgs.runCommand "sce-release-portability-audit" + { + nativeBuildInputs = [ + pkgs.binutils + pkgs.coreutils + ]; + } + '' + set -euo pipefail + + bash ${./nix/release/native-portability-audit.sh} \ + --platform linux \ + --binary ${sceReleasePackage}/bin/sce + + mkdir -p "$out" + ''; + + # Explicit long-running validation tier. Its primary member is the + # static-musl release build; on Linux it also forces the release + # portability audit over the real binary. Building this aggregate + # (nix build .#ci-checks) is the expensive command, keeping + # nix flake check fast (it never builds .#sce-release). + ciChecks = + pkgs.runCommand "sce-ci-checks" + { } + '' + set -euo pipefail + + mkdir -p "$out" + ln -s ${sceReleasePackage} "$out/sce-release" + ${pkgs.lib.optionalString pkgs.stdenv.isLinux '' + ln -s ${releasePortabilityAuditCheck} "$out/release-portability-audit" + ''} + ''; + sceApp = { type = "app"; program = "${scePackage}/bin/sce"; @@ -1418,6 +1458,7 @@ packages = { sce = scePackage; sce-release = sceReleasePackage; + ci-checks = ciChecks; bun = bunPackage; turso = tursoPackage; default = scePackage; From faf91c276e9b57fa89e249a11f6fe2d77d1a93b4 Mon Sep 17 00:00:00 2001 From: David Abram Date: Thu, 23 Jul 2026 17:35:19 +0200 Subject: [PATCH 6/6] ci: Restructure PR CI into native checks and release-validation job Co-authored-by: SCE --- .github/workflows/pr-ci.yml | 34 ++- context/architecture.md | 40 +++- context/cli/cli-command-surface.md | 12 +- context/context-map.md | 1 + context/glossary.md | 4 +- context/overview.md | 2 +- context/patterns.md | 4 +- .../nix-build-devshell-ci-restructure.md | 224 +++++++++++++++++- context/sce/cli-release-artifact-contract.md | 2 +- context/sce/flake-build-performance.md | 88 +++++++ flake.nix | 16 +- 11 files changed, 403 insertions(+), 24 deletions(-) create mode 100644 context/sce/flake-build-performance.md diff --git a/.github/workflows/pr-ci.yml b/.github/workflows/pr-ci.yml index 90e50054..232e0c08 100644 --- a/.github/workflows/pr-ci.yml +++ b/.github/workflows/pr-ci.yml @@ -44,8 +44,38 @@ jobs: run: | time nix flake check --print-build-logs - - name: Build and smoke-test CLI + - name: Build and smoke-test native CLI run: | - time nix build .#default --out-link result --print-build-logs + time nix build .#sce --out-link result --print-build-logs ./result/bin/sce --help ./result/bin/sce version + + release-validation: + name: Release validation (${{ matrix.os }}) + timeout-minutes: 90 + strategy: + fail-fast: false + matrix: + os: [ubuntu-latest, macos-latest] + runs-on: ${{ matrix.os }} + steps: + - name: Harden the runner (Audit all outbound calls) + uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + with: + egress-policy: audit + + - name: Checkout + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + + - name: Install Nix + uses: DeterminateSystems/nix-installer-action@ef8a148080ab6020fd15196c2084a2eea5ff2d25 # v22 + + - name: Enable Magic Nix Cache + uses: DeterminateSystems/magic-nix-cache-action@908b263ff629f4cc17666315b7fd3ec127c6244d # v14 + with: + use-flakehub: false + use-gha-cache: true + + - name: Build release package and audit portability + run: | + time nix build .#ci-checks --out-link result-ci-checks --print-build-logs diff --git a/context/architecture.md b/context/architecture.md index 53ce8661..fa0df4fb 100644 --- a/context/architecture.md +++ b/context/architecture.md @@ -94,7 +94,7 @@ The repository includes a new placeholder Rust binary crate at `cli/`. - GitHub Releases are the canonical publication surface for release archives, checksums, release-manifest assets, npm package assets, and approved Flatpak source-manifest and bundle assets. - Cargo/crates.io and npm registry publication belong to separate downstream publish stages that consume already-versioned checked-in package metadata rather than inventing release versions during workflow execution. - npm is a downstream consumer of the shared release artifact contract rather than a separate build owner. -- `flake.nix` now exposes dedicated release-packaging apps for this rollout: `apps.release-artifacts` packages the current-platform `sce` archive/checksum/manifest-fragment set from `packages.default`, prepares/audits the staged `bin/sce` before archive creation (including macOS `libiconv` install-name rewriting plus ad-hoc re-signing when needed), and fails if `.version`, `cli/Cargo.toml`, `npm/package.json`, the built/prepared CLI version, or the native portability audit disagree; `apps.native-portability-audit` audits native binaries for forbidden `/nix/store/` runtime references using Linux ELF/string inspection or macOS `otool -L`; `apps.release-manifest` merges per-platform metadata fragments into release-level manifest/checksum outputs and signs the merged manifest with a non-repo private key provided via env/file input; `apps.release-npm-package` packs the checked-in npm launcher package into a release-ready `sce-v-npm.tgz` tarball plus npm metadata JSON while refusing mismatched checked-in version metadata; and Linux-only `apps.release-flatpak-package` emits the Flatpak source-manifest tarball/checksum/JSON metadata while refusing `.version`/Cargo/npm/AppStream version drift and non-git checkouts that cannot resolve a release commit; Linux-only `apps.release-flatpak-bundle` builds a source-built `.flatpak` bundle from the checkout and emits per-architecture bundle/checksum/JSON metadata. +- `flake.nix` now exposes dedicated release-packaging apps for this rollout: `apps.release-artifacts` packages the current-platform `sce` archive/checksum/manifest-fragment set from the release output `packages.sce-release` (`nix build .#sce-release`), prepares/audits the staged `bin/sce` before archive creation (including macOS `libiconv` install-name rewriting plus ad-hoc re-signing when needed), and fails if `.version`, `cli/Cargo.toml`, `npm/package.json`, the built/prepared CLI version, or the native portability audit disagree; `apps.native-portability-audit` audits native binaries for forbidden `/nix/store/` runtime references using Linux ELF/string inspection or macOS `otool -L`; `apps.release-manifest` merges per-platform metadata fragments into release-level manifest/checksum outputs and signs the merged manifest with a non-repo private key provided via env/file input; `apps.release-npm-package` packs the checked-in npm launcher package into a release-ready `sce-v-npm.tgz` tarball plus npm metadata JSON while refusing mismatched checked-in version metadata; and Linux-only `apps.release-flatpak-package` emits the Flatpak source-manifest tarball/checksum/JSON metadata while refusing `.version`/Cargo/npm/AppStream version drift and non-git checkouts that cannot resolve a release commit; Linux-only `apps.release-flatpak-bundle` builds a source-built `.flatpak` bundle from the checkout and emits per-architecture bundle/checksum/JSON metadata. - CLI release automation: `.github/workflows/release-sce.yml` orchestrates `sce` GitHub release asset assembly, validates `.version`/Cargo/npm parity before tagging or releasing, calls three reusable per-platform workflow files (`release-sce-linux.yml`, `release-sce-linux-arm.yml`, `release-sce-macos-arm.yml`), assembles the signed native release manifest, builds npm and Flatpak source-manifest release packages, and uploads the CLI/npm/Flatpak asset set to the GitHub Release. Each native reusable workflow validates its generated archive before native artifact upload by extracting the target archive, smoke-running `bin/sce version --format json`, and running `nix run .#native-portability-audit` with the lane platform. Manual `workflow_dispatch` runs can set the GitHub Release-level `prerelease` flag through an explicit checkbox; tag-triggered releases keep the flag false and no tag-name prerelease inference is performed. - Downstream registry publication is now implemented in dedicated workflows: `.github/workflows/publish-crates.yml` publishes the checked-in crate version from a published release or manual dispatch after `.version`/tag/Cargo parity checks and requires semver prerelease metadata when a publish run is marked prerelease; `.github/workflows/publish-npm.yml` publishes the checked-in npm package version from a published release or manual dispatch after `.version`/tag/npm parity checks plus verification of the canonical `sce-v-npm.tgz` GitHub release asset, using the `next` npm dist-tag instead of `latest` for prerelease versions or prerelease-marked runs. - The npm distribution implementation lives under `npm/`: `package.json` defines the `sce` package surface, `bin/sce.js` launches the package-local native binary, `lib/install.js` resolves the current package version against the release manifest, verifies `sce-v-release-manifest.json.sig` with the bundled public key before trusting manifest contents, and then installs the checksum-verified native archive for supported macOS/Linux targets, while `test/platform.test.js` and `test/install.test.js` cover platform selection plus signed-manifest installer behavior. @@ -140,15 +140,49 @@ The repository includes a new placeholder Rust binary crate at `cli/`. - `flake.nix` applies `rust-overlay` (`oxalica/rust-overlay`) to nixpkgs, pins `rust-bin.stable.1.95.0.default` with `rustfmt` + `clippy`, reads the package/check version from repo-root `.version`, builds `packages.sce` through Crane (`buildDepsOnly` -> `buildPackage`) with shared host Cargo metadata plus a filtered repo-root source that preserves the Cargo tree, `cli/assets/hooks`, and only the generated config/schema inputs required by CLI embedding, then injects generated OpenCode/Claude/Pi config payloads and schema inputs into a temporary `cli/assets/generated/` mirror during derivation unpack so `cli/build.rs` can package the crate without requiring committed generated crate assets. The same `cli/build.rs` now scans `cli/migrations/*/*.sql` and writes `cli/src/generated_migrations.rs` with deterministic migration constants sorted by numeric filename prefix. - The root flake runs `cli-tests`, `cli-clippy`, and `cli-fmt` through Crane-backed paths for the CLI crate; `cli-fmt` intentionally uses the CLI-only Cargo source set rather than the package source with generated config/assets so formatting is not invalidated by release/config asset churn. The flake also exposes directory-scoped JS validation derivations for both `npm/` and the shared `config/lib/` plugin package root, and `pkl-parity` uses a narrowed Pkl/generated-output source set instead of copying the entire repository. - Git-commit embedding is release-only. `SCE_GIT_COMMIT` is injected via a `releaseCommitArgs` fragment (`{ SCE_GIT_COMMIT = shortGitCommit; }`) merged only into the release derivations — `scePackageMusl` on Linux and a dedicated native-toolchain `sceReleasePackageNative` on Darwin — and is deliberately absent from `commonCargoArgs`. As a result native `packages.sce`/`packages.default` and the `cli-tests`/`cli-clippy`/`cli-fmt` check derivations carry no commit in their inputs and stay cache-reusable across commits (native `sce version` reports `unknown`), while `.#sce-release` reports the real commit. `cli/build.rs` `emit_git_commit` emits the `rustc-env` value only when `SCE_GIT_COMMIT` is explicitly set (with `rerun-if-env-changed=SCE_GIT_COMMIT` as its sole rerun trigger) — no `git rev-parse` fallback and no `.git/HEAD`/`.git/packed-refs` watches — and `cli/src/services/version/mod.rs` resolves the commit through `option_env!("SCE_GIT_COMMIT")` with an `"unknown"` fallback. On Darwin the release derivation therefore diverges from `.#sce` (shared deps `cargoArtifacts`, but distinct final crate carrying the commit). -- `flake.nix` exposes `packages..ci-checks` (`ciChecks`) as the explicit long-running validation tier so the expensive work lives behind `nix build .#ci-checks` and never folds into `nix flake check`. It is a `runCommand` aggregate that symlinks its primary member `sceReleasePackage` into `$out/sce-release` (forcing the static-musl release build on Linux, native on Darwin) and, on Linux only, symlinks `releasePortabilityAuditCheck` into `$out/release-portability-audit`. That Linux-only `releasePortabilityAuditCheck` derivation runs `nix/release/native-portability-audit.sh --platform linux --binary ${sceReleasePackage}/bin/sce` (with `binutils`+`coreutils` on PATH) so `nix build .#ci-checks` fails when the real release binary carries forbidden `/nix/store/` references. This is distinct from `checks..native-portability-audit` (`nativePortabilityAuditCheck`), which remains a fixture-based unit test of the audit script on both platforms. CI wiring to `.#ci-checks` is owned by a separate release-validation job (see the CI restructure plan T07). +- `flake.nix` exposes `packages..ci-checks` (`ciChecks`) as the explicit long-running validation tier so the expensive work lives behind `nix build .#ci-checks` and never folds into `nix flake check`. It is a `runCommand` aggregate that symlinks its primary member `sceReleasePackage` into `$out/sce-release` (forcing the static-musl release build on Linux, native on Darwin) and, on Linux only, symlinks `releasePortabilityAuditCheck` into `$out/release-portability-audit`. That Linux-only `releasePortabilityAuditCheck` derivation runs `nix/release/native-portability-audit.sh --platform linux --binary ${sceReleasePackage}/bin/sce` (with `binutils`+`coreutils` on PATH) so `nix build .#ci-checks` fails when the real release binary carries forbidden `/nix/store/` references. This is distinct from `checks..native-portability-audit` (`nativePortabilityAuditCheck`), which remains a fixture-based unit test of the audit script on both platforms. CI wiring to `.#ci-checks` is owned by the separate `release-validation` matrix job in `.github/workflows/pr-ci.yml`, which builds `nix build .#ci-checks` on every PR/`main`/tag commit. - Flatpak flake tooling is Linux-only and reduced to a minimal app surface: `apps.sce-flatpak` is the umbrella entrypoint (`nix run .#sce-flatpak -- ` for `validate`, `prepare-local-manifest --repo-root `, etc.) that delegates to `packaging/flatpak/sce-flatpak.sh`; `apps.release-flatpak-package` emits deterministic Flatpak GitHub Release source-manifest assets; `apps.release-flatpak-bundle` builds a source-built `.flatpak` bundle; helper apps `apps.regenerate-flatpak-manifest` and `apps.regenerate-cargo-sources` rewrite the checked-in generated artifacts from their Nix sources. The standalone `flatpak-validate`, `flatpak-local-manifest`, and `flatpak-build` wrapper apps are removed in favor of `sce-flatpak `. `checks..flatpak-static-validation` runs the Nix-built static validator script during default `nix flake check`, and `checks..flatpak-manifest-parity` plus `checks..cargo-sources-parity` enforce the generated-artifact parity contracts; default `nix flake check` does not run `flatpak-builder`. The Linux dev shell includes `appstreamcli`, `flatpak`, and `flatpak-builder`. - The shared config-lib source set is rooted at `config/lib/` and includes the shared `package.json`, `bun.lock`, and `tsconfig.json` plus `agent-trace-plugin/` and `bash-policy-plugin/`; `config-lib-bun-tests` runs the bash-policy plugin wrapper tests from that shared root, while `config-lib-biome-check` and `config-lib-biome-format` run Biome over the copied shared package source. `.github/workflows/publish-crates.yml` follows the same asset-preparation rule but runs Cargo packaging from a temporary clean repository copy so crates.io publish no longer needs `--allow-dirty`. - The config-lib check source preserves repo-relative access to shared CLI patch fixtures: Nix copies a filtered repo-shaped source containing `config/lib/**` plus `cli/src/services/structured_patch/fixtures`, then runs Bun/Biome from `config/lib/`. -- `flake.nix` exposes release install/run surfaces as `packages.sce` (`packages.default = packages.sce`) plus `apps.sce` and `apps.default`, all targeting `${packages.sce}/bin/sce`; this keeps repo-local and remote flake run/install flows (`nix run .`, `nix run github:crocoder-dev/shared-context-engineering`, `nix profile install github:crocoder-dev/shared-context-engineering`) aligned to the same packaged CLI output. +- `flake.nix` exposes the **native** development package as `packages.sce` (`packages.default = packages.sce`) plus `apps.sce` and `apps.default`, all targeting `${packages.sce}/bin/sce`; this keeps repo-local and remote flake run/install flows (`nix run .`, `nix run github:crocoder-dev/shared-context-engineering`, `nix profile install github:crocoder-dev/shared-context-engineering`) aligned to the native CLI output. The static-musl release binary is a distinct output, `packages.sce-release` (`apps.sce-release`); see the native/release split note above and `context/sce/cli-release-artifact-contract.md`. - `biome.json` at the repository root is the canonical Biome configuration for the current JS tooling slice and deliberately scopes coverage to `npm/**` plus the shared `config/lib/**` plugin package root while excluding package-local `node_modules/**`; `flake.nix` exposes Biome through the default dev shell rather than through package-local installs. - `cli/Cargo.toml` now keeps crates.io publication-ready package metadata for the `shared-context-engineering` crate, and `cli/README.md` is the Cargo install surface for crates.io (`cargo install shared-context-engineering --locked`), git (`cargo install --git https://github.com/crocoder-dev/shared-context-engineering shared-context-engineering --locked`), and local checkout (`cargo install --path cli --locked`) guidance. The published crate installs the `sce` binary. Tokio remains intentionally constrained (`default-features = false`) with current-thread runtime usage plus timer-backed bounded resilience wrappers for retry/timeout behavior. - `cli/Cargo.toml` now declares Tokio's `time` feature directly alongside the existing constrained current-thread runtime setup (`rt`, `io-util`, `time`) instead of relying on transitive enablement. +## Build / devShell / CI performance (flake-speedup) + +The current structure and durable before/after results for the native/release +split, release-only commit embedding, slimmed devShell, and split CI are +summarized in `context/sce/flake-build-performance.md`. Measurements distinguish +evaluation, dependency compile, final crate compile, and cached lookup, with +explicit warm/cold and native/musl labels. + +Baseline bottlenecks (x86_64-linux, 8 cores, Nix 2.34.8): + +- **Cold Rust dependency compile is the dominant cost** — native `sce-deps` + ~8 min / 991 crates, musl `sce-deps-musl-deps` ~4 min / 494 crates. The native + deps derivation compiles ~2× the crates because it also builds the dev/test + graphs used by the Crane `cargoTest`/`cargoClippy` checks; musl deps are + release-only. +- **Warm evaluation** `nix flake check --no-build` ~5.5 s (~3.65 s CPU, ~465 MiB + eval heap), dominated by the `nixpkgs`/`rust-overlay` import and Crane + pipelines — not by the small `turso` / `flatpak-builder-tools` inputs (T09). +- **Warm `nix flake check`** ~6.3 s when every output is cached. +- **Final crate compile** (deps cached): musl release ~12 s cargo / ~15 s wall; + `turso` CLI ~3.5 s. + +Why the restructure helps: previously `.#default`/`.#sce` was the static-musl +release build, and the default devShell also included built SCE and Turso +packages. The split makes native `.#sce` the default and PR smoke target, moves +the musl release behind `packages.ci-checks` / the dedicated +`release-validation` CI job, drops `scePackage`/`tursoPackage` from +`devShells.default` (T04), and makes ordinary checks commit-independent (T03) +so they stay cache-reusable across commits. +Investigations T08 (`turso default-features = false`) and T09 (isolating the +`turso` / `flatpak-builder-tools` inputs) were recorded as deliberate no-ops +with rationale in the benchmark doc. Final after-change numbers and remaining +bottlenecks are captured in T11. + This phase establishes compile-safe extension seams with a dependency baseline (`anyhow`, `chrono`, `clap`, `clap_complete`, `dirs`, `hmac`, `inquire`, `jsonschema`, `keyring-core`, `murmur3`, `owo-colors`, `rand`, `reqwest`, `serde`, `serde_json`, `sha2`, `tokio`, `tracing`, `turso`, `uuid`, plus target-specific keyring backends); no CLI dev-dependencies are currently declared. Per-user local Turso DB and Agent Trace DB bootstrap/health coverage now exist through setup/doctor flows, while a user-invocable `sce sync` command and broader runtime integrations remain deferred. ## SCE plan/code role boundary diff --git a/context/cli/cli-command-surface.md b/context/cli/cli-command-surface.md index 5a65ab86..47a8a142 100644 --- a/context/cli/cli-command-surface.md +++ b/context/cli/cli-command-surface.md @@ -22,22 +22,22 @@ Operator onboarding currently comes from `sce --help`, command-local `--help` ou - `cli/src/app.rs` now owns an explicit startup lifecycle (`perform_dependency_check` -> `build_startup_context` -> `initialize_runtime` -> `run_command_lifecycle` -> `render_run_outcome`) so dependency bootstrap, config-backed runtime initialization, command parsing/execution, and final stream rendering are no longer coordinated inside one monolithic startup function. - `cli/src/app.rs` also routes clap output through an internal static `RuntimeCommand` enum defined in `cli/src/services/command_registry.rs`, so parse-time conversion and run-time command execution stay separated while avoiding boxed command trait objects. - Command-local help is available for implemented commands including bare `sce auth`, `sce auth --help`, `sce auth login --help`, `sce setup --help`, `sce doctor --help`, and `sce completion --help`; when stdout color is enabled those help payloads now reuse the shared heading/command/placeholder styling pass while non-TTY and `NO_COLOR` flows stay plain text. Human-readable stderr diagnostics and interactive setup prompt text now follow the same shared styling policy on their respective terminal streams. -- Current repository verification guidance for this CLI slice prefers the root Nix entrypoints: `nix flake check` for routine validation, `nix build .#default` / `nix run .#sce -- --help` for packaged installability, and targeted `nix develop -c sh -c 'cd cli && '` only when a narrower Rust-only check is explicitly needed. +- Current repository verification guidance for this CLI slice prefers the root Nix entrypoints: `nix flake check` for routine validation, `nix build .#sce` / `nix run .#sce -- --help` for native packaged installability, and targeted `nix develop -c sh -c 'cd cli && '` only when a narrower Rust-only check is explicitly needed. -## Nix release installability surface +## Nix installability surface -- Root `flake.nix` exposes `packages.sce` and `packages.default = packages.sce` for packaged release builds. -- Root `flake.nix` exposes `apps.sce` pointing to `${packages.sce}/bin/sce` for runnable packaged CLI execution. +- Root `flake.nix` exposes `packages.sce` and `packages.default = packages.sce` for the **native** development package; the static-musl release binary is the distinct `packages.sce-release` (`apps.sce-release`) output. +- Root `flake.nix` exposes `apps.sce` pointing to `${packages.sce}/bin/sce` for runnable native CLI execution. - Root `flake.nix` is the single repository-level Nix entrypoint for CLI checks and packaging. - Current installability checks for this surface are: - - `nix build .#default` + - `nix build .#sce` - `nix run .#sce -- --help` ## Cargo release and future crates.io posture - `cli/Cargo.toml` includes crates.io-facing package metadata (`description`, `license`, `repository`, `homepage`, `documentation`, `readme`, `keywords`, `categories`) and is aligned to the current crates.io publication posture described by the root release/publish workflows. - Current local install contract is `cargo install --path cli --locked`. -- Current release build/installability checks run through the root flake (`nix build .#default`, `nix run .#sce -- --help`) so the packaged binary and embedded generated assets stay aligned with the canonical Nix-owned release path. +- Current build/installability checks run through the root flake against the native package (`nix build .#sce`, `nix run .#sce -- --help`) so the packaged binary and embedded generated assets stay aligned with the canonical Nix-owned build path; the static-musl release archive is built from `.#sce-release` (see `context/sce/cli-release-artifact-contract.md`). - Crates.io publication is now a dedicated downstream publish stage (`.github/workflows/publish-crates.yml`) that validates `.version`/tag/Cargo parity before publishing the checked-in crate version. ## Command surface contract diff --git a/context/context-map.md b/context/context-map.md index 25899712..da778252 100644 --- a/context/context-map.md +++ b/context/context-map.md @@ -64,6 +64,7 @@ Feature/domain context: - `context/sce/cli-release-artifact-contract.md` (shared `sce` binary release artifact naming, checksum/manifest outputs, pre-archive staged-binary preparation including macOS `libiconv` install-name sanitization/ad-hoc re-signing, native portability audit app/check for forbidden `/nix/store/` runtime references, GitHub Releases as the canonical artifact publication surface, manual dispatch `prerelease` flag behavior, the current three-target Linux/macOS release workflow topology including pre-upload extracted-archive smoke/audit validation in each native lane, implemented Flatpak source-manifest and source-built `.flatpak` bundle package assets uploaded by `.github/workflows/release-sce.yml`, and Flatpak's explicit source-built non-binary exception) - `context/sce/cli-npm-distribution-contract.md` (implemented `sce` npm launcher package, release-manifest/checksum-verified native binary install flow, the supported darwin/arm64 plus linux x64+arm64 npm platform matrix, the prerequisite that npm-consumed native archives are portable and free of forbidden `/nix/store/` runtime references before upload, and dedicated `.github/workflows/publish-npm.yml` downstream npm publish-stage contract including prerelease publication via npm `next` dist-tag instead of `latest`) - `context/sce/cli-cargo-distribution-contract.md` (implemented `sce` Cargo publication posture plus supported crates.io, git, and local checkout install guidance, dedicated crates.io publish workflow with semver prerelease guard, and ephemeral crate-local generated-asset mirror requirement for published builds) +- `context/sce/flake-build-performance.md` (current native-vs-release package split, release-only commit embedding, slim default/database devShell contract, normal-vs-expensive CI tiers, durable before/after benchmark summary, validation evidence, and remaining cold-build trade-offs) - `context/sce/flatpak-distribution-patterns.md` (implementation conventions for the source-built `dev.crocoder.sce` Flatpak channel: manifest generation, flake app surface, local and release builds, release assets, runtime Git access via host-git bridge, and general distribution patterns) Working areas: diff --git a/context/glossary.md b/context/glossary.md index 272c3d29..3ec775ad 100644 --- a/context/glossary.md +++ b/context/glossary.md @@ -28,9 +28,11 @@ - `Flatpak bundle GitHub Release asset`: Implemented source-built `.flatpak` bundle release asset set for the `dev.crocoder.sce` Flatpak channel: `sce-v-x86_64.flatpak`, `sce-v-aarch64.flatpak`, plus per-architecture `.sha256` and `.json` metadata, built from source inside Flatpak using `flatpak-builder` + `flatpak build-bundle`. The bundle is a source-built Flatpak app for direct install (`flatpak install --user `), not a prebuilt binary or Flathub submission. These assets coexist with the existing Flatpak source-manifest packaging metadata on GitHub Releases. The `release-bundle` command is implemented in `packaging/flatpak/sce-flatpak.sh` and the GitHub workflow upload is implemented in `.github/workflows/release-sce-linux.yml` / `release-sce-linux-arm.yml` (x86_64/aarch64), assembled by `release-sce.yml`. - `cli flake native package`: Root-flake package output exposed as `packages.sce` with `packages.default = packages.sce`, producing the **native** development `sce` binary (`scePackage`) via `nix build .#sce` / `nix build .#default`. - `cli flake release package`: Root-flake package output exposed as `packages.sce-release` (`sceReleasePackage`): the static-musl release binary (`scePackageMusl`) on Linux and a dedicated native-toolchain build (`sceReleasePackageNative`) on Darwin, built via `nix build .#sce-release`. On Linux it is a distinct store path from `packages.sce` and passes the native portability audit; on Darwin it is likewise distinct from `packages.sce` because it embeds the commit (see `release-only Git-commit embedding`). `nix run .#release-artifacts` builds through this output. -- `cli flake ci-checks aggregate`: Root-flake package output exposed as `packages..ci-checks` (`ciChecks`), the explicit long-running validation tier run via `nix build .#ci-checks`. It is a `runCommand` symlink aggregate whose primary member is `sceReleasePackage` (forcing the static-musl release build) and which, on Linux only, also forces `releasePortabilityAuditCheck` — a derivation running `nix/release/native-portability-audit.sh` against the real `${sceReleasePackage}/bin/sce` binary. Distinct from the fixture-based `native-portability-audit` flake check. `nix flake check` never builds `.#sce-release`, keeping the expensive work behind this aggregate. CI wiring is deferred to the release-validation job (plan T07). +- `cli flake ci-checks aggregate`: Root-flake package output exposed as `packages..ci-checks` (`ciChecks`), the explicit long-running validation tier run via `nix build .#ci-checks`. It is a `runCommand` symlink aggregate whose primary member is `sceReleasePackage` (forcing the static-musl release build) and which, on Linux only, also forces `releasePortabilityAuditCheck` — a derivation running `nix/release/native-portability-audit.sh` against the real `${sceReleasePackage}/bin/sce` binary. Distinct from the fixture-based `native-portability-audit` flake check. `nix flake check` never builds `.#sce-release`, keeping the expensive work behind this aggregate. CI consumes it through the `release-validation` matrix job in `.github/workflows/pr-ci.yml`, which runs `nix build .#ci-checks` on every PR/`main`/tag commit. - `release-only Git-commit embedding`: Build-input contract where `SCE_GIT_COMMIT` is injected only into the release derivations via the `releaseCommitArgs` fragment (`scePackageMusl` on Linux, `sceReleasePackageNative` on Darwin) and never into `commonCargoArgs`. Native `packages.sce`/`packages.default` and the `cli-tests`/`cli-clippy`/`cli-fmt` checks therefore carry no commit in their inputs and stay cache-reusable across commits (native `sce version` reports `unknown`), while `.#sce-release` reports the real commit. `cli/build.rs` `emit_git_commit` emits the value only when the env var is explicitly set (sole rerun trigger `rerun-if-env-changed=SCE_GIT_COMMIT`, no `git rev-parse` fallback, no `.git` watches); `cli/src/services/version/mod.rs` reads it via `option_env!` with an `"unknown"` fallback. - `cli flake runnable app`: Root-flake app outputs where `apps.sce` and `apps.default` execute the native `${scePackage}/bin/sce` (dev ergonomics), supporting explicit `nix run .#sce -- ...` and default-app flows such as `nix run . -- ...` / `nix run github:crocoder-dev/shared-context-engineering -- ...`; `apps.sce-release` executes the release `${sceReleasePackage}/bin/sce` for explicit `nix run .#sce-release -- ...`. +- `slimmed default dev shell`: `devShells..default` (`nix develop`) no longer includes `scePackage` or `tursoPackage`, so entering it compiles neither the CLI package nor the Turso CLI — it provides only the Rust toolchain plus JS/pkl tooling for `cargo`/`biome`/`pkl` work. Turso stays available as `packages..turso` and via the opt-in `database dev shell`. The shared package list + shellHook are factored into `defaultDevShellPackages`/`defaultDevShellHook` `let` bindings so the two shells cannot drift. +- `database dev shell`: Opt-in `devShells..database` entered via `nix develop .#database`; layers `tursoPackage` (plus a `turso` version echo) on top of `defaultDevShellPackages`/`defaultDevShellHook`, providing the Turso CLI without adding it to the default shell. - `cli cargo install contract`: Supported Cargo install surface for the published `shared-context-engineering` crate, which installs the `sce` binary: crates.io (`cargo install shared-context-engineering --locked`), git (`cargo install --git https://github.com/crocoder-dev/shared-context-engineering shared-context-engineering --locked`), and local checkout (`cargo install --path cli --locked`). - `cli crates.io publication posture`: Current Cargo package posture in `cli/Cargo.toml` where crates.io-facing metadata is publication-ready for the `shared-context-engineering` crate, with crate-facing install guidance owned by `cli/README.md`. - `Nix performance recommendations`: Repo-local operator guidance in `AGENTS.md` covering optional user-level `~/.config/nix/nix.conf` tuning (`max-jobs = auto`, `cores = 0`) and the explicit root/admin-only boundary for `/etc/nix/nix.conf` `auto-optimise-store = true`. diff --git a/context/overview.md b/context/overview.md index 77d19f69..f961987e 100644 --- a/context/overview.md +++ b/context/overview.md @@ -99,7 +99,7 @@ Lightweight post-task verification baseline (required after each completed task) - For architecture decisions: `context/decisions/` - No dedicated generated-parity workflow is currently checked in; local/generated-output parity is enforced through `nix run .#pkl-check-generated` and the root `nix flake check` `pkl-parity` derivation. GitHub Actions workflow lint is enforced through the root `nix flake check` `workflow-actionlint` derivation (all `.github/workflows/*.yml`). -- PR validation runs through `.github/workflows/pr-ci.yml` on `pull_request`, `push` to `main`, and `workflow_dispatch`, executing on an `ubuntu-latest` + `macos-latest` matrix (`fail-fast: false`); a workflow-level concurrency group (`${{ github.workflow }}-${{ github.ref }}`, `cancel-in-progress: true`) cancels superseded runs on the same ref, and each matrix job has a `timeout-minutes: 90` guard. It installs Nix via `DeterminateSystems/nix-installer-action@v22`, enables `DeterminateSystems/magic-nix-cache-action@v14` immediately after, then runs timed `nix flake check --print-build-logs`, timed `nix build .#default --out-link result --print-build-logs`, and smoke-tests the already-built package with `./result/bin/sce --help` / `./result/bin/sce version`. Job name renders as `Nix CI (ubuntu-latest)` / `Nix CI (macos-latest)` for use as required branch protection checks. The Nix store cache is the only cache layer; no separate Cargo/Bun/npm/`node_modules` caches. +- PR validation runs through `.github/workflows/pr-ci.yml` on `pull_request`, `push` to `main`, and `workflow_dispatch` and now defines two matrix jobs. A workflow-level concurrency group (`${{ github.workflow }}-${{ github.ref }}`, `cancel-in-progress: true`) cancels superseded runs on the same ref, and every matrix job installs Nix via `DeterminateSystems/nix-installer-action@v22`, enables `DeterminateSystems/magic-nix-cache-action@v14` immediately after, runs on an `ubuntu-latest` + `macos-latest` matrix (`fail-fast: false`), and has a `timeout-minutes: 90` guard. The `nix-ci` job (name `Nix CI (ubuntu-latest)` / `Nix CI (macos-latest)`, for required branch protection checks) runs timed `nix flake check --print-build-logs`, then builds and smoke-tests the **native** package via timed `nix build .#sce --out-link result --print-build-logs` plus `./result/bin/sce --help` / `./result/bin/sce version` — it no longer builds the musl release (`.#default`/`.#sce` is native since the flake-speedup split). A separate `release-validation` job (name `Release validation (ubuntu-latest)` / `Release validation (macos-latest)`) builds `nix build .#ci-checks --out-link result-ci-checks --print-build-logs` on every commit, which forces the `.#sce-release` static-musl build and, on Linux, the real-binary portability audit guarded inside `ciChecks`; macOS builds the native-on-Darwin release package. This always-on release coverage on every PR/push (no path filters) is the intentional trade-off recorded in the plan Decisions section. The Nix store cache is the only cache layer; no separate Cargo/Bun/npm/`node_modules` caches. Release-publish workflows (`release-sce*.yml`) build the release binary via `nix run .#release-artifacts` (which targets `.#sce-release`), not `.#default`. ## Cross-target parity diff --git a/context/patterns.md b/context/patterns.md index a6503b51..b95bb681 100644 --- a/context/patterns.md +++ b/context/patterns.md @@ -11,7 +11,7 @@ - Keep direct Cargo verification commands as secondary targeted-debugging tools rather than the default repo-validation path. - Keep `cargo fmt` available for explicit autofix formatting flows; do not present it as the preferred verification command. - Gate workflow YAML edits through `checks..workflow-actionlint` inside root `nix flake check`; run `nix run nixpkgs#actionlint -- .github/workflows/*.yml` for targeted workflow lint during CI workflow edits. -- Keep PR validation in `.github/workflows/pr-ci.yml` (`Nix CI`): triggers on `pull_request`, `push` to `main`, and `workflow_dispatch`; `ubuntu-latest` + `macos-latest` matrix with `fail-fast: false`; pinned `DeterminateSystems/nix-installer-action@v22` with `DeterminateSystems/magic-nix-cache-action@v14` immediately after install; workflow-level concurrency (`${{ github.workflow }}-${{ github.ref }}`, `cancel-in-progress: true`) and job `timeout-minutes: 90`; quality gates are timed `nix flake check --print-build-logs`, timed `nix build .#default --out-link result --print-build-logs`, and smoke tests against `./result/bin/sce` from the already-built default package (no diagnostic `nix flake metadata` step and no separate `nix run` smoke evaluations); pin third-party Actions to explicit version tags, not `@main`. +- Keep PR validation in `.github/workflows/pr-ci.yml` (`Nix CI`): triggers on `pull_request`, `push` to `main`, and `workflow_dispatch`; two matrix jobs (`nix-ci` and `release-validation`), each on an `ubuntu-latest` + `macos-latest` matrix with `fail-fast: false`; pinned `DeterminateSystems/nix-installer-action@v22` with `DeterminateSystems/magic-nix-cache-action@v14` immediately after install; workflow-level concurrency (`${{ github.workflow }}-${{ github.ref }}`, `cancel-in-progress: true`) and job `timeout-minutes: 90`. The `nix-ci` quality gates are timed `nix flake check --print-build-logs`, timed `nix build .#sce --out-link result --print-build-logs` (the **native** package), and smoke tests against `./result/bin/sce` from that already-built package (no diagnostic `nix flake metadata` step and no separate `nix run` smoke evaluations); it does not build the musl release. The `release-validation` job builds `nix build .#ci-checks --out-link result-ci-checks --print-build-logs` on every commit, forcing the `.#sce-release` build plus the Linux-guarded real-binary portability audit. Pin third-party Actions to explicit version tags, not `@main`. ## Root Biome scoping @@ -149,7 +149,7 @@ - Keep cheap flake-check sources as narrow as their behavior allows: formatting checks should not depend on package-only generated assets, and parity/static checks should copy only the authoring inputs and committed outputs they inspect. - Keep Rust package/check sources as narrow as behavior allows: package builds should include the Cargo tree plus required embedded config/schema assets, not unrelated config authoring or plugin sources that are not read by `cli/build.rs`. - In `flake.nix`, select the Rust toolchain via an explicit Rust overlay (`rust-overlay`) and thread that toolchain through Crane package/check derivations so CLI builds and checks do not rely on implicit nixpkgs Rust defaults. -- For installable CLI release surfaces in the root flake, expose an explicit named package plus default alias (`packages.sce` and `packages.default = packages.sce`) and pair it with a runnable app output (`apps.sce`) that points to the packaged binary path. +- For installable CLI surfaces in the root flake, expose an explicit named package plus default alias (`packages.sce` and `packages.default = packages.sce`, the **native** development package) and pair it with a runnable app output (`apps.sce`) that points to the packaged binary path; keep the static-musl release binary a distinct output (`packages.sce-release`) rather than folding it into the default package. - For root-flake CLI release metadata, source the package/check version from repo-root `.version` and trim it at eval time so packaged outputs stay aligned without hardcoded semver strings in `flake.nix`. - For Cargo CLI distribution, keep crate metadata publication-ready, document the supported Cargo install paths in `cli/README.md` (`cargo install shared-context-engineering --locked`, git install with `--locked`, and local `cargo install --path cli --locked`), and verify at least the repo-local build/check path through the Nix-managed validation baseline. diff --git a/context/plans/nix-build-devshell-ci-restructure.md b/context/plans/nix-build-devshell-ci-restructure.md index 2195a4f0..110d3239 100644 --- a/context/plans/nix-build-devshell-ci-restructure.md +++ b/context/plans/nix-build-devshell-ci-restructure.md @@ -373,8 +373,30 @@ documented in `context/`. --print-build-logs`; confirm `nix flake check --no-build` does not force the musl release; `nix flake show`. -- [ ] T07: `Restructure CI: native flake check + separate release job` (status:todo) +- [x] T07: `Restructure CI: native flake check + separate release job` (status:done) - Task ID: T07 + - Status: done + - Completed: 2026-07-23 + - Files changed: .github/workflows/pr-ci.yml + - Evidence: `nix-ci` matrix smoke step now builds native `.#sce` (was + `.#default`) and runs `sce --help` / `sce version`; both-OS `nix flake check` + preserved. Added a distinct `release-validation` job (`[ubuntu-latest, + macos-latest]` matrix, same harden-runner/checkout/Nix-install/magic-cache + setup, `timeout-minutes: 90`) that runs `nix build .#ci-checks` on every + commit — pulling in `.#sce-release` and, on Linux, the real-binary portability + audit guarded inside `ciChecks`. `workflow-actionlint` flake check passed + (1.8 s). Flake attrs resolve: `.#sce` → `155gfv…-sce-0.3.2.drv` (native), + `.#ci-checks` → `qsl669…-sce-ci-checks.drv`. No `.#default` reference remains + in `pr-ci.yml`; the PR smoke job no longer builds the musl release. Release + publish workflows (`release-sce*.yml`) already build via `.#release-artifacts` + → `.#sce-release` (T02), so no publish-workflow rewiring was needed. + - Notes: Release validation runs on both OSes in the matrix, so macOS also + builds the native-on-Darwin release package every commit — the always-on + release coverage the Decisions section chose over path filters (intentional + CI-minute cost). Darwin lane wiring-verified only (evaluated on x86_64-linux). + Context-sync classification: important (CI job topology is a documented + contract in overview.md's `pr-ci.yml` description — native smoke vs separate + release job). - Goal: PR/`main`/tag CI runs `nix flake check` + native `.#sce` smoke test on Linux and macOS; a distinct release-validation job builds `.#sce-release` (via `.#ci-checks`) on every commit and audits Linux release portability; @@ -394,8 +416,35 @@ documented in `context/`. release workflows; confirm no PR job builds the musl artifact except the dedicated release job. -- [ ] T08: `Investigate Turso default-features (benchmark-gated)` (status:todo) +- [x] T08: `Investigate Turso default-features (benchmark-gated)` (status:done) - Task ID: T08 + - Status: done (deliberate no-op — change rejected/deferred, no code change) + - Completed: 2026-07-23 + - Files changed: context/tmp/flake-speedup-benchmarks.md + - Evidence: Recorded an explicit **reject (no `cli/Cargo.toml`/`Cargo.lock` + change)** decision in the benchmark doc's "T08" section. `turso` 0.7.0 + default features are `["mimalloc", "fts"]`; `default-features = false` would + drop `mimalloc`/`libmimalloc-sys` and the `fts`→`tantivy` full-text-search + stack (`sync` is already off — not a default). Cheap crate-graph signal + captured: baseline compiled graph is **395 crates** on both + `x86_64-unknown-linux-gnu` and `-musl`, including `tantivy v0.26.1` (+ ~10 + `tantivy-*` crates) and `mimalloc v0.1.52` (+ `libmimalloc-sys v0.1.49`). + Correctness pre-check done: no FTS5/`VIRTUAL TABLE` in `cli/migrations/**`, + no `mimalloc` reference in `cli/`, encryption uses unconditional + `turso_core` `aegis256` — so the change is *plausibly* safe but unverified. + The full benchmark matrix (cold native + musl dep compiles, release builds, + binary/closure size, DB + encryption tests, each ×2 feature sets = 20+ min + of cold builds) was **not** run and was deferred as disproportionately + expensive for this session; the plan's Open-questions note permits this + deliberate no-op. Working tree left pristine (no `Cargo.toml`/`Cargo.lock` + edit remains). + - Notes: Change looks promising on the crate-graph signal but is not justified + under this task's evidence bar without the deferred full build + test-suite + measurement. Follow-up if cold-build/closure cost becomes a priority: flip + the flag, `cargo update`, re-count graph, native + musl `nix build`, compare + size, run DB + encryption tests via `nix flake check`. Context-sync + classification: verify-only (no code/behavior/public-contract/architecture/ + terminology change; benchmark scratch doc under context/tmp/ only). - Goal: Decide, with recorded evidence, whether to set `turso` to `default-features = false`; change it only if it meaningfully improves builds and the full suite (incl. database + encryption tests) still passes. @@ -412,8 +461,42 @@ documented in `context/`. `nix flake check --print-build-logs`; database + encryption test names pass; `du`/`nix path-info -S` for size/closure comparison. -- [ ] T09: `Investigate optional root flake inputs (benchmark-gated)` (status:todo) +- [x] T09: `Investigate optional root flake inputs (benchmark-gated)` (status:done) - Task ID: T09 + - Status: done (deliberate no-op — change rejected, no `flake.nix`/`flake.lock` edit) + - Completed: 2026-07-23 + - Files changed: context/tmp/flake-speedup-benchmarks.md + - Evidence: Recorded an explicit **reject (no `flake.nix`/`flake.lock` change)** + decision in the benchmark doc's "T09" section. Per-input read-only + measurements (x86_64-linux, warm shared store, commit e196703): `turso` + source-tree **51 MB disk / 45,231,832 B closure** (flake input, but its four + transitive inputs `nixpkgs`/`flake-utils`/`crane`/`rust-overlay` all `follows` + the root, so it adds **no** extra large closures to `flake.lock`), consumed by + `tursoToolchain` (`${turso}/rust-toolchain.toml`, flake.nix:76) and + `src = turso` for `packages.turso` (flake.nix:393–425); + `flatpak-builder-tools` **2.6 MB disk / 2,096,560 B closure**, `flake = false` + (no flake eval), consumed only by `nix/flatpak/cargo-sources.nix` for + `cargo-sources-parity`. Warm eval `hyperfine --warmup 2 -r 5 'nix flake check + --no-build'` = **5.318 s ± 0.073 s** — statistically unchanged from the T01 + baseline (5.494 s ± 0.294 s), confirming these inputs are not a material eval + cost (eval is dominated by `nixpkgs` 472 MB + `rust-overlay` 19 MB + crane + pipelines). No `flake.nix` edit made, so `packages.turso`, + `cargo-sources-parity`, `flatpak-manifest-parity`, and + `flatpak-static-validation` inputs are untouched and still pass. + - Notes: Rejected because isolating either input yields no measurable + cold-fetch or warm-eval win: `flatpak-builder-tools` is 2.6 MB and + `flake = false`; `turso`'s transitive inputs are already `follows`-deduped, so + its only real cost is a one-time ~51 MB source fetch that would be identical + under a `fetchFromGitHub` pin, while isolation risks the public + `packages.turso` contract and toolchain wiring. A full "without input" A/B was + not run (disruptive: removing `turso` breaks the toolchain + `packages.turso`; + removing `flatpak-builder-tools` breaks `cargo-sources-parity`); the + closure-size + `follows`-dedup + unchanged-warm-eval signal answers the + question, and the plan's Open-questions note permits this deliberate no-op. + Follow-up trigger: only if cold-lock fetch time (not warm eval) becomes a + priority, pin `turso` via `fetchFromGitHub` to drop its `flake.lock` node. + Context-sync classification: verify-only (no code/behavior/public-contract/ + architecture/terminology change; benchmark scratch doc under context/tmp/ only). - Goal: Measure whether the `turso` repo input and `flatpak-builder-tools` input materially affect cold input fetching or warm evaluation; isolate only with evidence, preserving the public `packages..turso` output and all @@ -432,8 +515,41 @@ documented in `context/`. --no-build'` and `NIX_SHOW_STATS=1` with/without inputs where measurable; `nix build .#turso`; `nix flake check` Flatpak checks; results in benchmark doc. -- [ ] T10: `Document build/devShell/CI architecture and benchmarks` (status:todo) +- [x] T10: `Document build/devShell/CI architecture and benchmarks` (status:done) - Task ID: T10 + - Status: done + - Completed: 2026-07-23 + - Files changed: context/architecture.md, context/patterns.md, + context/cli/cli-command-surface.md, context/sce/cli-release-artifact-contract.md + - Evidence: Added a new "Build / devShell / CI performance (flake-speedup)" + subsection to `architecture.md` that links `context/tmp/flake-speedup-benchmarks.md`, + records the baseline bottlenecks with cache conditions (cold native deps + ~8 min/991 crates + musl ~4 min/494 crates dominant; warm eval ~5.5 s / + ~3.65 s CPU / ~465 MiB heap; warm `nix flake check` ~6.3 s; final crate musl + ~12 s cargo / ~15 s wall, turso ~3.5 s), explains why the split helps + (native `.#sce` default for PR smoke + devShell, musl release behind + `ci-checks`/`release-validation`, slimmed devShell T04, commit-independent + checks T03), and notes T08/T09 deliberate no-ops + T11 finalization. Removed + stale "release build" claims: `architecture.md:147` (`packages.sce`/`.default` + reframed as native dev package, release binary pointed at `.#sce-release`), + `architecture.md:97` (`release-artifacts` source corrected `packages.default` + → `.#sce-release`, verified against flake.nix:709 `nix build .#sce-release`), + `cli-command-surface.md` (§heading `Nix release installability` → `Nix + installability`, `.#default` build/installability checks → `.#sce`, added + `.#sce-release` release-output pointer at lines 25/29/40), `patterns.md:152` + ("release surfaces" → native package + distinct `packages.sce-release` note), + `cli-release-artifact-contract.md:61` (`nix build .#default` → `.#sce-release`). + The native/release split, release-only commit embedding, default vs + `.#database` shells, and native-vs-release CI topology were already synced + into `overview.md`/`architecture.md`/`patterns.md`/`glossary.md` by prior + tasks; this task added the missing benchmark section + link and reconciled + the remaining stale package-output claims. Final `grep` for + `.#default`/`packages.sce` release phrasing shows only accurate split + descriptions remaining. Benchmark doc present and linked. Docs-only change; + no code/build/test surface (context/*.md are not flake-check or generation + inputs). Context-sync classification: important (durable architecture/pattern + docs now describe the flake-speedup build/devShell/CI contract + benchmarks; + stale public-output claims corrected). - Goal: Update repository context/architecture docs to explain the new structure and record benchmark outcomes. - Boundaries (in/out of scope): In — update `context/architecture.md` (and @@ -446,8 +562,37 @@ documented in `context/`. - Verification notes (commands or checks): re-read updated docs; cross-check against final `nix flake show`; ensure benchmark figures match T01/T11. -- [ ] T11: `Final validation and context sync` (status:todo) +- [x] T11: `Final validation and context sync` (status:done) - Task ID: T11 + - Status: done + - Completed: 2026-07-23 + - Files changed: context/tmp/flake-speedup-benchmarks.md, + context/sce/flake-build-performance.md, context/context-map.md, + context/architecture.md, context/plans/nix-build-devshell-ci-restructure.md + - Evidence: `nix flake check --print-build-logs` passed all current-system + checks in 7.00 s warm (0 rebuilt); native `.#sce` built/smoked successfully + (18.41 s cold final crate, Cargo 12.83 s) and reports commit `unknown`; + `.#sce-release` built successfully (266.00 s with missing deps, deps 4 m 02 s, + final Cargo 14.73 s), reports `e1967032cd14`, is a distinct static-pie Linux + output, and passed both the direct native-portability audit and + `nix build .#ci-checks` real-binary audit. Dedicated workflow-actionlint and + `nix run .#pkl-check-generated` passed. Default `nix develop -c true` entered + without building SCE/Turso; `.#database` reported Turso 0.7.0. Temporary + empty-commit probe preserved the exact native drv/output paths across the + commit change, then restored HEAD; derivation inspection found no current + commit in native/check derivations and found it in the release derivation. + `nix flake show --json` retained the public outputs. Detailed after + benchmarks remain in the session record + `context/tmp/flake-speedup-benchmarks.md`; durable results, remaining + bottlenecks, and T08/T09 rejected/deferred changes are summarized in + `context/sce/flake-build-performance.md` and linked from the context map. + - Notes: x86_64-linux behavior was built and executed; Darwin and aarch64 + outputs were evaluation/wiring-verified only. Embedded asset byte correctness + remains covered by the passing CLI test derivation's SHA-256 assertions and + generated source uses `include_bytes!`. T11 context-sync classification: + important final-plan sync — the plan's public build/devShell/CI contracts + were verified in root context, the ignored session benchmark was summarized + in a durable domain file, and architecture/context-map links were refreshed. - Goal: Run all applicable validation, capture after-change benchmarks, and sync context. - Boundaries (in/out of scope): In — full validation suite, after-benchmark @@ -466,3 +611,72 @@ documented in `context/`. - Verification notes (commands or checks): the exact commands above; empty-commit cache-reuse check for `.#sce`; `nix flake show` output diff for public outputs; benchmark doc updated with final numbers and remaining bottlenecks. + +## Validation Report + +### Commands run + +- `nix flake check --print-build-logs` → exit 0; all x86_64-linux checks passed + (warm, 7.00 s, no rebuilds). +- `nix build .#sce --out-link result --print-build-logs`; `sce --help`; `sce + version` → build/smokes passed; native version commit `unknown`. +- `nix build .#sce-release --out-link result-release --print-build-logs` → exit + 0; static-musl release built and reported commit `e1967032cd14`. +- `nix run .#native-portability-audit -- --platform linux --binary + result-release/bin/sce` → exit 0; no forbidden `/nix/store/` references. +- `nix build .#ci-checks --out-link result-ci-checks --print-build-logs` → exit + 0; real release-binary audit passed. +- `nix build .#checks.x86_64-linux.workflow-actionlint --no-link` → exit 0; all + workflow files passed actionlint. +- `nix develop -c true --print-build-logs`; `nix develop .#database -c turso + --version` → exit 0; default shell required no SCE/Turso build, database shell + provided Turso 0.7.0. +- `nix run .#pkl-check-generated` → exit 0; generated outputs are current. +- `hyperfine --warmup 2 -r 5 'nix flake check --no-build'` → 5.533 s ± 0.132 s. +- Temporary empty-commit build probe → native drv/output paths unchanged; HEAD + restored afterward. +- `nix derivation show` inspection → no current commit in native/test/clippy/fmt + derivations; release derivation contains the short commit. +- `nix flake show --json` → public package/app/devShell outputs evaluated. +- `git diff --check` → exit 0; updated durable root/domain context files are + within the 250-line sync limit (the disposable execution plan is longer). + +### Cleanup + +- Removed temporary `result`, `result-release`, and `result-ci-checks` out-links + after evidence capture; no debug code or generated scaffolding remains. +- Detailed raw benchmark notes remain session-only under `context/tmp/`; durable + results are in `context/sce/flake-build-performance.md`. + +### Success-criteria verification + +- [x] Native `.#sce`/`.#default` and release `.#sce-release` outputs are split; + Linux paths are distinct and release is static/portable. +- [x] Native/check derivations are commit-independent; release reports the real + commit. +- [x] Default devShell omits SCE/Turso builds; `packages.turso` and the database + shell remain available. +- [x] Embedded assets use `include_bytes!`; passing CLI SHA-256 tests preserve + byte correctness. +- [x] `nix flake check` remains the primary validation entrypoint and retains the + prior named check inventory without forcing `.#sce-release`. +- [x] PR CI uses native smoke tests on Linux/macOS and separate always-on release + validation with Linux portability auditing; actionlint passes. +- [x] Release helpers target `.#sce-release`; release artifact shape/portability + contracts remain unchanged. +- [x] Non-intentionally changed public flake outputs remain available. +- [x] Before/after benchmarks distinguish evaluation/build/cache and + native/musl; rejected/deferred investigations are recorded. + +### Failed checks and follow-ups + +- The host shell lacked `file` during the first native smoke command; binary + inspection was rerun successfully with `nix shell nixpkgs#file -c file`. + No product validation failed. + +### Residual risks + +- Darwin and aarch64 outputs were evaluation/wiring-verified only from the + x86_64-linux host; CI provides native Linux/macOS execution. +- Always-on `release-validation` intentionally retains the release-build cost on + every PR/push; path filtering remains deferred. diff --git a/context/sce/cli-release-artifact-contract.md b/context/sce/cli-release-artifact-contract.md index 31e9150a..580ccac6 100644 --- a/context/sce/cli-release-artifact-contract.md +++ b/context/sce/cli-release-artifact-contract.md @@ -58,7 +58,7 @@ This file captures the current shared release artifact foundation plus the appro ## Determinism rules -- Release archives are built from the root flake package output (`nix build .#default`). +- Release archives are built from the static-musl release output (`nix build .#sce-release`; `nix run .#release-artifacts` builds via `.#sce-release`), not the native `.#default`/`.#sce` package. - Tarball creation uses stable file ordering plus fixed ownership and mtime metadata. - Gzip output is emitted with deterministic headers. - Checksum files use SHA-256 and the standard `` line format. diff --git a/context/sce/flake-build-performance.md b/context/sce/flake-build-performance.md new file mode 100644 index 00000000..a22d5d88 --- /dev/null +++ b/context/sce/flake-build-performance.md @@ -0,0 +1,88 @@ +# Flake build, devShell, and CI performance + +## Current structure + +The root flake separates ordinary development from release-only work: + +- `packages.sce` and `packages.default` build the native development CLI. +- `packages.sce-release` builds static musl on Linux and a commit-bearing native + release on Darwin. +- `packages.ci-checks` is the explicit expensive tier. It forces the release + package and, on Linux, audits the real release binary for forbidden + `/nix/store/` runtime references. +- `nix flake check` runs the normal test/lint/format/parity/static checks without + building `.#sce-release`. +- `devShells.default` provides development tools without `scePackage` or the + Turso CLI. `devShells.database` adds Turso. +- `.github/workflows/pr-ci.yml` runs native package smoke tests in `nix-ci` and + runs `.#ci-checks` separately in `release-validation` on every commit. + +Git-commit embedding is release-only. Native packages and normal CLI check +derivations do not receive `SCE_GIT_COMMIT`; release packages do. + +## Benchmark method + +Measurements were taken on x86_64-linux with 8 logical cores, Nix 2.34.8, and a +shared developer store. "Cold" means the requested derivation was absent or +rebuilt while fetched sources, toolchains, and dependency closures remained +available; it does not mean an empty Nix store. Warm timings include evaluation +and cached-output lookup. + +The detailed session record is retained in +`context/tmp/flake-speedup-benchmarks.md`; the durable results are summarized +here. + +## Before and after + +| Measure | Before | After | Result | +|---|---:|---:|---| +| Warm `nix flake check --no-build` | 5.494 s ± 0.294 s | 5.533 s ± 0.132 s | Evaluation unchanged. | +| Eval heap | ~465 MiB | ~465 MiB | Unchanged. | +| Warm `nix flake check` | 6.27 s | 7.00 s | Comparable; release output no longer forced. | +| Native final crate, deps cached | Not exposed as a package | 18.41 s wall / 12.83 s Cargo | Native development output is directly buildable. | +| Musl dependency compile | 4 m 06.2 s | 4 m 02 s | Release compile cost unchanged but isolated. | +| Musl final crate | 15.22 s wall / 12.07 s Cargo | 14.73 s Cargo | Comparable. | +| Warm default devShell | 1.54 s | 1.78 s | Comparable; cold shell no longer compiles SCE or Turso. | + +The baseline native dependency graph remained the largest cold cost at about +8 minutes and 991 compile lines because it includes dev/test dependencies for +Crane test and Clippy derivations. The musl release graph remains about 4 +minutes. This restructure isolates those costs; it does not make Rust dependency +compilation intrinsically faster. + +## Validation evidence + +Final validation on x86_64-linux confirmed: + +- `nix flake check --print-build-logs` passed. +- Native `.#sce` and release `.#sce-release` produced distinct store outputs. +- Native `sce version` reported commit `unknown`; release reported the current + short commit. +- The Linux release was static-pie and passed direct and `.#ci-checks` + real-binary portability audits. +- A temporary empty commit left the native derivation and output paths unchanged. +- Native/check derivations contained no current commit value; the release + derivation contained it. +- `nix develop -c true` required no SCE/Turso build, while + `nix develop .#database -c turso --version` worked. +- Workflow actionlint, generated-output parity, embedded-asset SHA-256 tests, + and public-output evaluation passed. + +Darwin and aarch64 outputs were evaluation/wiring-verified from Linux; CI owns +native execution on Linux and macOS. + +## Investigated no-ops + +- Turso `default-features = false` was deferred. It could remove the Tantivy and + mimalloc stacks, but the full native/musl build, size, database, and encryption + validation matrix was not completed, so no dependency change was justified. +- Isolating the root `turso` and `flatpak-builder-tools` inputs was rejected. + Turso's transitive flake inputs already follow root inputs, and + `flatpak-builder-tools` is a small non-flake source input; warm evaluation was + unchanged and isolation offered no material benefit. + +## Remaining trade-off + +The separate `release-validation` job intentionally builds `.#ci-checks` on +every PR and push. This preserves always-on release coverage but means CI still +pays the release compile cost in a distinct job. Path filtering is deferred. diff --git a/flake.nix b/flake.nix index 116453a1..09b46d8d 100644 --- a/flake.nix +++ b/flake.nix @@ -427,8 +427,8 @@ }; # Shared development tooling for the default shell. Excludes `scePackage` - # and `tursoPackage` so `nix develop` does not compile the package or the - # Turso CLI; `devShells.database` layers Turso back on when needed. + # and `tursoPackage` so `nix develop` does not compile either CLI; + # opt-in shells layer them back on when needed. defaultDevShellPackages = (with pkgs; [ biome @@ -1646,8 +1646,18 @@ shellHook = defaultDevShellHook; }; + # Opt-in shell exposing the repository-built CLI as `sce` while keeping + # the default development shell fast. + devShells.sce = pkgs.mkShell { + packages = defaultDevShellPackages ++ [ scePackage ]; + + shellHook = '' + ${defaultDevShellHook} + echo "- sce: $(sce version)" + ''; + }; + # Opt-in shell layering the Turso CLI on top of the default tools, so - # `nix develop` stays fast (no `scePackage`/`turso_cli` compile) while # database work can pull Turso in explicitly via `nix develop .#database`. devShells.database = pkgs.mkShell { packages = defaultDevShellPackages ++ [ tursoPackage ];