feat: event-driven ingestion via inotify (phase 1)#3
Open
Meldrey wants to merge 6 commits into
Open
Conversation
Extract _ensure_base_tables() with the generic flex storage contract (chunks, sources, edges, meta, presets, FTS) from the CC-specific extension tables (tool_ops, message types, delegations, soft_ops, file bodies). _ensure_core_tables() remains as a backward-compatible wrapper that calls both. This separation lets non-coding-agent modules (e.g. Matrix, Telegram) bootstrap cells without CC-specific table baggage. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New insert_base_chunk(conn, chunk, source_type) writes only to the generic _raw_chunks and _edges_source tables. Any flex module can call this without needing CC-specific extension tables. insert_chunk_atom() now delegates to insert_base_chunk() for the base write, then adds CC-specific tables (types_message, tool_ops, delegations, soft_ops) as before. No behavior change for existing callers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add substrate parameter ('claude_code' or 'base') that controls which
tables are created. Base substrate gets only the generic chunk-atom
tables + content store. CC substrate adds coding-agent extension tables
and SOMA, matching previous behavior.
bootstrap_claude_code_cell remains as a backward-compatible alias.
Default substrate='claude_code' means zero behavior change for existing
callers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Modules can now declare substrate='base' in their spec to get a clean generic cell without CC-specific tables or enrichment. Changes: - coding_agent_install.py: read substrate from spec, pass to bootstrap_cell(), conditionally apply enrichment stubs/pipeline and use appropriate contract validator - contract.py: add REQUIRED_BASE_TABLES and validate_base_cell() for non-coding-agent modules - __init__.py: extract BASE_ENRICHMENT_STUBS (_ops, _views) from ENRICHMENT_STUBS. Full CC stubs = base + CC-specific. Backward compatible -- ENRICHMENT_STUBS still exports everything. This enables non-coding-agent modules (Matrix, Telegram, etc.) to use the flex module system without inheriting CC-specific schema, enrichment, or validation requirements. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the 2-second polling loop as the primary change detection mechanism with inotify via watchdog. The daemon now detects JSONL writes in ~500ms (debounce window) instead of waiting for the next stat() sweep. - New FlexWatcher class (watcher.py) wraps watchdog.Observer with debounced drain pattern — observer thread records changes, main thread syncs - daemon.py builds watcher on startup, registers ~/.claude/projects/ and all registry watch cells, passes to daemon_loop() - daemon_loop gains event-driven path (0.25s tick, drain-only) alongside polling fallback (2s tick, full rglob scan) - Polling demoted to 60s integrity scan when inotify is active - Graceful fallback: FLEX_DISABLE_INOTIFY, --no-inotify, or missing watchdog all revert to original polling behavior with zero changes - watchdog added as optional dependency under [watch] extra Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add flex:dev:core and flex:dev:modules under ai/skills/dev/ — reference skills for contributors working on flex internals. Covers daemon architecture, watcher integration, registry, module specs, substrate types, worker patterns, and the watch/refresh lifecycle. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
watchdogFlexWatcherclass wrapswatchdog.Observerwith debounced drain — observer thread records(path, timestamp), main thread syncs only changed filesdaemon_loopgains event-driven path (0.25s tick, drain-only) with polling demoted to 60s integrity scanFLEX_DISABLE_INOTIFY,--no-inotify, or missingwatchdogrevert to original polling with zero behavioral changeflex:dev:coreandflex:dev:modulescontributor reference skills underai/skills/dev/Context
From the event-driven ingestion proposal: the daemon's
scan_sessions()doesrglob("*.jsonl")+stat()on every file every 2 seconds. On a system with 2,460 sessions, that's O(n) filesystem work per tick. inotify does it in O(1) — the kernel notifies us only when something actually changes.This is Phase 1: proof of concept alongside existing polling. Phase 2+ (event queue, module watch declarations, stream drivers) builds on this foundation.
Depends on
feat/base-substrateEnvironment knobs
FLEX_DISABLE_INOTIFYFLEX_DEBOUNCE_MS500FLEX_INTEGRITY_INTERVAL60Test plan
FlexWatcherunit tests: create/modify detection, pattern filtering, debounce, symlink rejectionsync_session_messagesinserts chunksFLEX_DISABLE_INOTIFY=1correctly disables watcherast.parse()🤖 Generated with Claude Code