Allow capping data to N blocks with API retention NET-896#77
Conversation
Add an optional `max_blocks` to the Api config that trims the tail to keep at most `max_blocks` from the head.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
| data_sources: Vec<ReqwestDataClient>, | ||
| // Static safety cap on retained blocks for Api-controlled datasets: even | ||
| // if the portal stops advancing the floor, the tail is trimmed to keep at | ||
| // most this many blocks behind the tip. `None` means grow indefinitely. |
There was a problem hiding this comment.
I think it would be valuable to mention that it's not a hard limit, but rather soft limit, as it may happen that the first chunk has accumulated a lot of blocks, which would go above the max_blocks limit. And example: max_blocks = 100K, first chunk has [0..150K], when the trigger fires at 250K, we get window 0..250K which is 2.5x times larger than the expected max_blocks limit.
There was a problem hiding this comment.
Mentioned, but I don't think it's related to this PR directly. The same happens with API retention — once set to block N, it usually keeps a couple of blocks below N.
define-null
left a comment
define-null
left a comment
There was a problem hiding this comment.
Approving this PR, as max_blocks seems to be a best_effort thing, but in my view it will be hard to reason about with this soft limit approach.
- Document that `max_blocks` is a soft limit - Extract `trim_floor` and cover the capping rule with tests - Simplify the comments
9aa609f to
0676fde
Compare
2 participants
Goal
If the archival data got stuck, the storage grows until running out of disk space leading to unpleasant consequences.
We can't reliably prevent it from happening. But we can sacrifice older blocks to keep serving the real-time data. Much less users will be affected by a gap than by a stale head.
Config changes