Skip to content

[deckhouse-cli] Add pull summary after d8 mirror pull#381

Merged
ldmonster merged 11 commits into
mainfrom
feat/add-summary-stats-after-pulling
Jun 4, 2026
Merged

[deckhouse-cli] Add pull summary after d8 mirror pull#381
ldmonster merged 11 commits into
mainfrom
feat/add-summary-stats-after-pulling

Conversation

@Glitchy-Sheep

Copy link
Copy Markdown
Contributor

[mirror] Add an end-of-pull summary to d8 mirror pull

ℹ️ What

Prints a short, framed summary at the end of every d8 mirror pull (and --dry-run):
what was pulled (or, in dry-run, what will be) - platform, installer, security databases,
modules and packages.

A --verbose-summary flag adds a per-module and per-package breakdown with their resolved versions.

❓ Why

Before this, a pull ended with hundreds of Pulling ...@sha256:... lines and no recap.
To learn the one thing we actually need - which releases, module and package
versions will be mirrored
(and whether the bundle is complete) - you had to scroll the log.

The original ask was exactly that: in --dry-run, see which platform release and which
module/package versions will be pulled, honouring all filters (--include-module /
--exclude-module / --include-package / --no-platform / --include-platform / ...),
without reading sha lines.

📝 How it works

  • Each phase (platform / installer / security / modules / packages) records what it
    resolved/pulled - versions, VEX attestations, bundle sizes - from data the pull
    already has. No extra registry requests.
  • Counts are taken before packing (packing deletes the layouts as it tars them).
  • The summary renders on every outcome, so you always see what landed:
    • success and dry-run,
    • Ctrl+C -> partial summary, marked cancelled,
    • a hard failure (e.g. network died) -> partial summary, marked FAILED, then the error.
      A phase that never ran shows not pulled.
  • Colour is semantic and only when attached to a terminal; pipes and files get plain text
    (set FORCE_COLOR=1 to force colour into a file).

📊 What the summary shows

  • Edition - the source edition (CE/EE/...), parsed from the source path; omitted for
    a custom registry with no edition segment.
  • Platform - the release version(s), and how many channels they map to.
  • Installer - its tag (latest or a pinned --installer-tag).
  • Security - N/M databases (green when complete, yellow when partial).
  • Modules - module count and the total VEX; with --verbose-summary, each module's
    versions and VEX count.
  • Packages - same as modules, for Deckhouse packages: package count and total VEX; with
    --verbose-summary, each package's versions and VEX count.
  • Bundle (real pull) - each artifact with its size and chunk count, plus a TOTAL.
    Package archives (package-<name>.tar, the shared package-versions.tar) are listed here
    automatically.

Versions are sorted newest-first (semver) everywhere they appear.
(See examples below)


🔄 Before / after

Before - the run just ended with the streaming log; no recap:

... [318/319] Pulling registry.../modules/code@sha256:...  succeeded
... [319/319] Pulling registry.../modules/code@sha256:...  succeeded

Pull packages

FORCE_COLOR=1 d8 mirror pull \
      --source dev-registry.deckhouse.io/sys/deckhouse-oss \
      --no-platform --no-security-db --no-modules --no-installer \
      --include-package 'boardmaps@>=0.0.0' \
      --include-package 'cossacks-server@>=0.0.0' \
      --verbose-summary \
      <bundle-dir>
2026-06-04 AT 11 25@2x

Different platform

FORCE_COLOR=1 d8 mirror pull \
      --source registry.deckhouse.ru/deckhouse/ee \
      --include-platform '>=v1.71.0 <v1.76.0' --no-modules --dry-run --verbose-summary \
      <bundle-dir>
2026-06-04 AT 11 26@2x

Dry-Run for Community Edition

FORCE_COLOR=1 d8 mirror pull \
      --source registry.deckhouse.ru/deckhouse/ce \
      --dry-run --verbose-summary \
      <bundle-dir>
2026-06-04 AT 11 27@2x

Dry-Run for Enterprise Edition

FORCE_COLOR=1 d8 mirror pull \
      --source registry.deckhouse.ru/deckhouse/ee \
      --dry-run --verbose-summary \
      <bundle-dir>
2026-06-04 AT 11 28@2x

✏️ Main changes

  • New summary model and renderer (formatting only; emitted through the existing logger).
  • Pull() now returns (*PullSummary, error) - always a non-nil summary, so a partial run
    (cancel or hard error) still renders instead of showing only a raw error.
  • Per-phase Stats() that captures counts/versions before packing, for modules and
    packages
    .
  • --verbose-summary flag (output only - it does not change what is pulled).
  • Counting helpers in pkg/registry/image (manifest count + a matched subset for VEX).
  • Two latent bugs fixed while wiring this up: platform AsList() returned empty (reflection
    looked for the wrong field type), and dry-run under-counted module versions.

ℹ️ Also: packages mirroring no longer crashes on registries without packages

While adding the packages summary, surfaced and fixed a regression from the packages
mirroring support (#376): the packages phase failed the whole pull against any registry
that has no /packages repository - including the public registry.deckhouse.ru/deckhouse/ee
and /ce. The intended graceful-skip only matched ErrImageNotFound, but a missing repo is
reported as a NAME_UNKNOWN transport error.

Fix: treat NAME_UNKNOWN the same as ErrImageNotFound ("no packages here, skip the phase")
in both validatePackagesAccess and discoverPackageNames, mirroring the already-graceful
PullPackageVersions. With this, a pull against a packages-less registry succeeds and the
summary simply shows Packages: 0 packages.


🧪 Tests

  • Unit: rendering in every state (dry-run / cancelled / failed / not-pulled / verbose / VEX /
    versions) for modules and packages, semver sorting, colour gating, the counting helpers,
    and outcome classification.
  • Packages Stats(): dry-run accounting and the before-packing capture (the count must
    survive packing).
  • Regression for the graceful-skip fix: a missing /packages repo (both NAME_UNKNOWN and
    ErrImageNotFound) skips the phase instead of failing the pull.
  • Verified end-to-end against real registries: dry-run and real pulls render the expected
    versions / VEX / sizes (including a real packages pull); the summary adds no registry requests.

- PullSummary plus per-component stat types (platform, installer, security, modules, bundle)
- holds resolved versions, the VEX subset and bundle sizes for the end-of-pull report

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- each phase records its image/VEX counts and resolved versions before packing
- fix platform AsList that returned no sub-layouts (reflection looked for the wrong field type)

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- Pull returns a non-nil summary even on error, so partial runs still report
- map each services stats into the summary

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- framed block with versions, VEX, security N/M, bundle sizes and edition
- semver-sorted versions, semantic colour gated on TTY / FORCE_COLOR

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- render the summary on success, dry-run, cancel and hard failure
- --verbose-summary adds the per-module breakdown (output only)

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- cover dry-run, cancelled, failed, not-pulled, verbose, VEX and version sorting
- colour gating and the edition line

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- per-service Stats survive packing; manifest counting; bundle stats
- outcome classification and AsList sub-layout enumeration

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- add PackagesStats/PackageStat model and a Packages field to PullSummary
- capture per-package versions and manifest counts in the packages service
- render a Packages section (aggregate + per-package versions/VEX in verbose)

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
- treat a NAME_UNKNOWN transport error as "no packages", like ErrImageNotFound
- apply the graceful skip in both validatePackagesAccess and discoverPackageNames
- add a regression test covering both signals

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep Glitchy-Sheep self-assigned this Jun 4, 2026
@Glitchy-Sheep Glitchy-Sheep added the enhancement New feature or request label Jun 4, 2026
@Glitchy-Sheep Glitchy-Sheep marked this pull request as ready for review June 4, 2026 08:40
@Glitchy-Sheep Glitchy-Sheep requested a review from ldmonster as a code owner June 4, 2026 08:40
@ldmonster ldmonster merged commit 7f5674c into main Jun 4, 2026
9 checks passed
@ldmonster ldmonster deleted the feat/add-summary-stats-after-pulling branch June 4, 2026 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants