fix(feed): Improve NVD feed download resilience#3343
Conversation
dcaravel
changed the title
📝 WalkthroughWalkthroughThe NVD feed loader gains a 5-attempt exponential-backoff retry loop in ChangesNVD Feed Loader Retry & Timeout
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/vulnloader/nvdloader/loader_feed.go`:
- Around line 56-67: downloadFeedForYear is retrying all failures from
fetchFeed, including permanent 4xx responses, so classify errors in fetchFeed
and only retry transport errors and retryable HTTP statuses like 429/5xx. Update
the retry loop in downloadFeedForYear to inspect the error type/status before
sleeping, and preserve the existing maxRetries/backoff behavior for transient
failures while failing fast on non-retryable client errors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Central YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 91cd5d80-4950-43fd-8954-1a3ee79d9319
📒 Files selected for processing (2)
pkg/vulnloader/nvdloader/loader_api.gopkg/vulnloader/nvdloader/loader_feed.go
| for attempt := 1; ; attempt++ { | ||
| var err error | ||
| apiFeed, err = fetchFeed(url, year) | ||
| if err == nil { | ||
| break | ||
| } | ||
| if attempt >= maxRetries { | ||
| return errors.Wrapf(err, "failed to download feed for year %d after %d attempts", year, attempt) | ||
| } | ||
| log.Warnf("Feed year %d: attempt %d failed: %v; retrying in %s", year, attempt, err, backoff) | ||
| time.Sleep(backoff) | ||
| backoff *= 2 |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Stop retrying permanent HTTP client errors.
fetchFeed turns every non-200 into the same generic error, so downloadFeedForYear also retries 400/401/403/404 responses. With a 6-minute client timeout plus 10→80s backoff, one permanent 4xx can stall a single year for 30+ minutes before failing. Please classify errors so only transport failures and retryable statuses (for example 429/5xx) loop.
Also applies to: 101-103
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/vulnloader/nvdloader/loader_feed.go` around lines 56 - 67,
downloadFeedForYear is retrying all failures from fetchFeed, including permanent
4xx responses, so classify errors in fetchFeed and only retry transport errors
and retryable HTTP statuses like 429/5xx. Update the retry loop in
downloadFeedForYear to inspect the error type/status before sleeping, and
preserve the existing maxRetries/backoff behavior for transient failures while
failing fast on non-retryable client errors.
Source: Path instructions
2 participants
Summary
queryWithBackoffContext
CI was failing with
stream error: stream ID 1; INTERNAL_ERROR; received from peerwhen downloading NVD feeds. The NVD server intermittently kills HTTP/2 streams mid-transfer. The previous code had no retries and no logging beyond the opaque error, making failures hard to diagnose and impossible to recover from.Testing
CI and tested locally