Skip to content

backup: add gc command - #1341

Merged
bigbes merged 1 commit into
bigbes/gh-no-vshard-updatefrom
bigbes/tntp-8671-backup-gc
Jul 31, 2026
Merged

backup: add gc command#1341
bigbes merged 1 commit into
bigbes/gh-no-vshard-updatefrom
bigbes/tntp-8671-backup-gc

Conversation

@bigbes

@bigbes bigbes commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Add tt backup gc, the retention-based cleanup of a backup storage. It builds a deletion plan out of the manifest chain and only then executes it, so --dry-run shows exactly what a real run would remove.

Retention combines --keep-full N (the last N healthy chains) and --keep-days D: a backup goes only when no rule keeps it, and without either flag nothing is deleted at all. Deletion cascades and cuts a chain from its newest end only, so a kept increment always keeps the backups it is recovered from, and a full backup goes only with its whole chain. Inside one backup the manifest goes before its archives, and inside a chain the newest backup goes first, so an interrupted run leaves a valid chain prefix and collectable garbage rather than manifests pointing at nothing. Before anything is deleted the plan is filtered once more against every manifest that stays behind, since a manifest may name a previous backup in another chain or an archive another backup uploaded, and chain.Build treats neither as a problem.

Four things are never deleted, whatever the rules say. The chain holding the newest manifest, because an upload may be appending an increment to it. The newest chain that can still be recovered from, so a storage whose newest manifest belongs to a tail without its full backup does not lose every usable backup and keep the unusable one. An archive whose backup id is newer than every stored manifest — or any archive at all when no manifest is stored yet — since archives are uploaded before the manifest that references them. And the archives of a structurally invalid manifest: only the manifest says which archives a backup refers to, so one that cannot be believed makes its archives unclassifiable rather than abandoned. All of this is stated in the output, together with the consequence: gc cannot empty a storage.

Dangling archives are removed once older than --orphan-age (24h by default, an upload has to outlive it), each after a direct GET of its manifest, because LIST is not read-after-write consistent.

Chains with problems are left to tt backup verify to report, except a tail whose full backup is gone and whose every manifest is older than --orphan-age: nothing can be appending to it and nothing else would ever collect it, so it is collected even when it holds the newest manifest.

One supporting change falls out of this: ClusterManifest.Validate now requires creation_time, which the schema has always marked mandatory. Every retention rule reads it, and a missing one is either "ancient" or "immortal" depending on the rule.

Stacked on #1338.

@bigbes
bigbes force-pushed the bigbes/tntp-8671-backup-gc branch from e099f16 to 674d280 Compare July 31, 2026 07:02
@bigbes
bigbes force-pushed the bigbes/tntp-8671-backup-gc branch from 674d280 to b009611 Compare July 31, 2026 07:48
@bigbes
bigbes requested a review from patapenka-alexey July 31, 2026 08:27
@bigbes
bigbes force-pushed the bigbes/tntp-8671-backup-gc branch from b009611 to 9c92a63 Compare July 31, 2026 08:52
Add `tt backup gc`, the retention-based cleanup of a backup storage. It
builds a deletion plan out of the manifest chain and only then executes
it, so `--dry-run` shows what a real run would remove.

Retention combines `--keep-full N` (the last N healthy chains) and
`--keep-days D`: a backup goes only when no rule keeps it, and without
either flag nothing is deleted at all. Deletion cascades and cuts a
chain from its newest end only, so a kept increment always keeps the
backups it is recovered from; a full backup goes only with its whole
chain. Inside one backup the manifest goes before its archives, and
inside a chain the newest backup goes first, so an interrupted run
leaves a valid chain prefix and collectable garbage rather than
manifests pointing at nothing. Before anything is deleted the plan is
filtered once more against every manifest that stays behind, since a
manifest may name a previous backup in another chain or an archive
another backup uploaded, and chain.Build treats neither as a problem.

Three things are never deleted by the retention rules. The chain holding
the newest manifest, because an upload may be appending an increment to
it. The newest chain that can still be recovered from, so that a storage
whose newest manifest belongs to a tail without its full backup does not
end up losing every usable backup and keeping the unusable one. And an
archive whose backup id is newer than every stored manifest, or any
archive at all when no manifest is stored yet, since archives are
uploaded before the manifest that references them. All three are stated
in the output, together with the consequence: gc cannot empty a storage.

Dangling archives are removed once older than `--orphan-age` (24h by
default, an upload has to outlive it), each after a direct GET of its
manifest, because LIST is not read-after-write consistent.

Chains with problems are left to `tt backup verify` to report, except a
tail whose full backup is gone and whose every manifest is older than
`--orphan-age`: nothing can be appending to it and nothing else would
ever collect it, so it is collected even when it holds the newest
manifest.

An archive is also left alone when the manifest of its backup is
structurally invalid. Only the manifest says which archives a backup
refers to, so one that cannot be believed makes its archives merely
unclassifiable, not abandoned - and gc is the one command whose mistakes
cannot be undone.

One supporting change falls out of this: `ClusterManifest.Validate` now
requires `creation_time`, which the schema has always marked mandatory.
Every retention rule reads it, and a missing one is either "ancient" or
"immortal" depending on the rule.

Closes TNTP-8671
@bigbes
bigbes force-pushed the bigbes/tntp-8671-backup-gc branch from 9c92a63 to 9afa0b3 Compare July 31, 2026 10:25
@bigbes
bigbes merged commit 6b2c2db into master Jul 31, 2026
25 checks passed
@bigbes
bigbes deleted the bigbes/tntp-8671-backup-gc branch July 31, 2026 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants