feat(ingest): shared connector builder + opt-in in-process sync scheduler (#428) - #435
Merged
Merged
Conversation
…uler (#428) Deduplicate connector construction into internal/connectors.Build, reused by the CLI (buildConnector) and MCP (importFunc) as thin wrappers, and add an opt-in background sync in cmd/http. - internal/connectors.Build(ctx, kb, cfg, source, path, ocr) is the single place connectors are built; OAuth resolution (#246) lives inside, OCR is passed in by the caller so the shell-out stays in the cmd layer. CLI + MCP now delegate to it (net -100 lines, one place to edit a connector). - cmd/http autoSync: with SYNC_INTERVAL set (e.g. 15m), re-ingest every configured REMOTE connector on a timer — the in-process form of `kb sync` (#415) — incrementally, honoring INGEST_PRUNE_DELETED. markdown is skipped (autoImport already sweeps it). Default off (empty SYNC_INTERVAL) so the minimal single-user stack is unchanged. - Config: Ingest.SyncInterval + SyncInterval() + SYNC_INTERVAL env. - Tests: connectors.Build construction/error cases (no DB); SyncInterval parse + env. SYSTEM.md decision log + .env.example documented. Push adapters (fsnotify/webhooks) remain a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #428.
What
Two things, both keeping the minimal single-user stack unchanged (opt-in):
Shared connector builder. The ~150-line connector-construction switch was duplicated in the CLI (
buildConnector) and the MCP server (importFunc). It now lives once ininternal/connectors.Build(ctx, kb, cfg, source, path, ocr); both adapters are thin wrappers. OAuth token resolution (Per-source credential / OAuth store #246) lives inside the builder; the OCR fallback is passed in by the caller so theexecshell-out stays in the cmd layer. Net −100 lines, one place to edit a connector.Opt-in in-process sync scheduler.
cmd/httpgainsautoSync(mirrors the existingautoImport): withSYNC_INTERVALset (e.g.15m), the HTTP server re-ingests every configured remote connector on a timer — the in-process form ofkb sync(Background streaming-sync scheduler + upsert re-ingest + push adapters (follow-up to #395) #415) — incrementally, honoringINGEST_PRUNE_DELETED. markdown is skipped (already swept byautoImport).Default off —
SYNC_INTERVALempty ⇒ the scheduler never starts, minimal config unchanged.Tests
internal/connectors:Buildconstruction + error cases (no DB needed — non-OAuth sources don't touch the core).internal/config:SyncInterval()parse +SYNC_INTERVALenv.Docs
.env.exampledocumentSYNC_INTERVAL.Follow-up
🤖 Generated with Claude Code