Support subquery moves across AND/OR/NOT WHERE clauses by robacourt · Pull Request #4051 · electric-sql/electric

robacourt · 2026-03-24T17:14:02Z

Summary

This PR lets shapes with boolean combinations around subqueries stay live when dependency rows move. Previously, shapes that used AND, OR, or NOT around a subquery would return 409 on a move, forcing the client to throw away the shape and resync it from scratch. For large shapes that could mean a very expensive full resync; this PR avoids that by reconciling those changes in-stream.

The core of the PR is a rewrite of the subquery foundation. Move-ins are now handled as exact log splices: buffer outer-table transactions, run the move-in query, and insert its results into the shape log at the precise point where they become valid. That gives subqueries a much more reliable ordering model, removes a large class of move-handling edge cases, and gives us a stable base for richer boolean WHERE support over subqueries.

Compatibility

This PR changes the wire protocol.

The sync service from this branch is not compatible with elixir-client from before this PR. For the TanStack DB client, this requires:

@tanstack/db >= 0.6.2
@tanstack/electric-db-collection >= 0.3.0

The protocol change is needed because DNF-based subquery shapes need more than tags alone. Rows can now stay in the shape for multiple independent reasons, so the server sends real active_conditions values and position-aware move-in broadcast control messages. That lets clients update the truth value of the affected DNF positions for rows they already have locally, re-evaluate inclusion correctly, and avoid unnecessary deletes or full resyncs.

Architecture

compile each shape's WHERE clause into a DNF plan and use that plan consistently for routing, move planning, SQL generation, tags, and active_conditions
replace the previous subquery reverse-indexing approach with a per-shape SubqueryIndex that each shape consumer seeds and updates from its own dependency views, keeping the reverse index exactly in sync with consumer state
refactor Electric.Shapes.Consumer around explicit event handlers and ordered effects, with a single move queue and explicit buffering/splice handling
emit DNF-aware row metadata and position-aware move broadcasts so visibility can change incrementally instead of forcing invalidation

Secondary changes

update the materializer and clients to understand DNF-aware tags, active_conditions, and move broadcasts
expand coverage across DNF planning, routing, move ordering, resume behavior, and boolean/subquery edge cases
update the oracle test harness and generators to cover the new boolean/subquery behavior more accurately

codecov · 2026-04-15T17:37:44Z

Codecov Report

❌ Patch coverage is 85.41667% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.72%. Comparing base (3026244) to head (ec39d48).
⚠️ Report is 21 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...kages/elixir-client/lib/electric/client/message.ex	0.00%	7 Missing ⚠️
...s/elixir-client/lib/electric/client/tag_tracker.ex	94.87%	4 Missing ⚠️
packages/elixir-client/lib/electric/client/poll.ex	66.66%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4051      +/-   ##
==========================================
+ Coverage   68.29%   68.72%   +0.43%     
==========================================
  Files         127      156      +29     
  Lines       15988    17244    +1256     
  Branches     3855     3911      +56     
==========================================
+ Hits        10919    11851     +932     
- Misses       5065     5389     +324     
  Partials        4        4

Flag	Coverage Δ
elixir	`78.79% <85.41%> (?)`
elixir-client	`78.79% <85.41%> (?)`
packages/agents	`38.39% <ø> (ø)`
packages/agents-runtime	`81.34% <ø> (ø)`
packages/agents-server	`65.55% <ø> (-0.21%)`	⬇️
packages/agents-server-ui	`0.00% <ø> (ø)`
packages/electric-ax	`30.46% <ø> (ø)`
packages/experimental	`87.73% <ø> (ø)`
packages/react-hooks	`86.48% <ø> (ø)`
packages/start	`82.83% <ø> (ø)`
packages/typescript-client	`94.30% <ø> (ø)`
packages/y-electric	`56.05% <ø> (?)`
typescript	`68.08% <ø> (-0.22%)`	⬇️
unit-tests	`68.72% <85.41%> (+0.43%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

alco

Intermediate review while I keep looking through the rest of the changes.

What exactly is a subquery dependency polarity and what does it affect?

alco · 2026-04-23T11:09:31Z

  """
  @callback write_move_in_snapshot!(
-              Enumerable.t({key :: String.t(), value :: Querying.json_iodata()}),
+              Enumerable.t(row()),


The row() type is defined as list(). Can we make its definition reflect the actual value type it represents?

Addressed in 8827e20 (PR #4646).

alco · 2026-04-23T13:01:49Z

+          is_subquery: boolean(),
+          negated: boolean(),


Weird naming inconsistency. Both are boolean fields, should rather be named subquery? and negated?.

Addressed in fd7748e (PR #4646).

alco · 2026-04-23T13:07:28Z

+    case :ets.lookup(table(stack_ref), :last_broadcast_lsn) do
+      [{:last_broadcast_lsn, lsn}] -> lsn
+      [] -> 0
+    end
+  rescue
+    ArgumentError -> 0


Nit: since we have this rescue clause, we could use :ets.lookup_element:

Suggested change

case :ets.lookup(table(stack_ref), :last_broadcast_lsn) do

[{:last_broadcast_lsn, lsn}] -> lsn

[] -> 0

end

rescue

ArgumentError -> 0

:ets.lookup_element(table(stack_ref), :last_broadcast_lsn, 2)

rescue

ArgumentError -> 0

Addressed in 91f8f17 (PR #4646).

alco · 2026-04-23T13:08:23Z

+      last_lsn = get_last_broadcast_lsn(stack_ref)
+
+      if last_lsn > 0 do
+        send(self(), {:global_last_seen_lsn, last_lsn})


Since this is sending to self(), it should be mentioned in the function doc that callers are expected to process this message after a successful function call.

Addressed in dd8cb5c (PR #4646).

alco · 2026-04-23T13:09:13Z

+    test "return 400 if same subquery is used with both positive and negative polarity", %{
+      opts: opts
+    } do
+      assert %{status: 400} =


Does this also return a descriptive message? I think we should have that and assert against that in tests for easy reference.

Good call — the 400 returns a descriptive message under the where field. Added an assertion against it in 1545bb3 (PR #4646).

alco · 2026-04-23T13:10:56Z

+      tag2 =
+        :crypto.hash(:md5, ctx.stack_id <> req.handle <> "v:2") |> Base.encode16(case: :lower)


This looks like a good candidate for a helper function, to avoid repeating the same invocation with base encoding throughout the test code.

Done in 2a7fff5 (PR #4646). Extracted a value_tag/3 helper and routed all 12 call sites through it. It independently mirrors SubqueryTags.make_value_hash_raw/3 rather than calling it, so the assertions still document the expected wire format instead of reusing the code under test.

alco · 2026-04-23T13:22:07Z

  end

  @impl GenServer
+  def handle_continue({:init_consumer, config}, state) do


This is dead code, we no longer have :init_consumer continuation in this module. It got replaced with the def handle_info({:initialize_shape, shape, opts}, state) clause some time ago.

Addressed in 82ecbc2 (PR #4646).

alco · 2026-04-23T13:39:58Z

         {:ok, reduced} <-
           build_bool_chain(
-             %{name: "or", impl: &pg_and/2, strict?: false},
+             %{name: "and", impl: &pg_and/2, strict?: false},


Is this a sneaked-in bug fix for BETWEEN?

Before this PR it was a latent bug: the only consumer that ran was the evaluator, which uses implementation, so BETWEEN gave correct results. This PR introduces the code paths that key off name — DNF decomposition and re-generating WHERE SQL for subquery move-in queries — at which point the mislabel becomes a real correctness bug for any shape whose WHERE clause uses BETWEEN.

alco · 2026-04-23T14:16:30Z

+      end
+    end)
+    |> case do
+      {:ok, effects} -> {:ok, effects |> Enum.reverse() |> List.flatten()}


Effects could alternatively be accumulated in an order-preserving way using Stream.concat() and here converted to a list with Enum.to_list().

I left this one as-is. Each transaction_to_effects returns 0 or 1 Effects.AppendChanges, so the accumulator is a list of at most single-element lists and Enum.reverse |> List.flatten stays cheap and clear. A Stream.concat chain built up in the reduce would be nested per-txn and, to my eye, less readable here without a measurable win — happy to revisit if you feel strongly.

alco · 2026-04-23T14:18:33Z

+        error
+
+      converted ->
+        {:ok, converted |> Enum.reverse() |> List.flatten() |> mark_last_change()}


While this code was still in Consumer and/or ChangeHandling, the last change could be marked before reversing the list of changes. We can preserve that behaviour here as well, no? Instead of doing reverse + rebuild the list just to update its last entry.

Done in 6aa7581 (PR #4646). Conversions are now flattened into the reverse-order accumulator as they're produced, so the head is the last emitted change — it's marked before a single Enum.reverse/1, dropping the List.flatten/1 pass and the List.pop_at(-1) + ++ rebuild.

robacourt · 2026-04-23T15:15:50Z

What exactly is a subquery dependency polarity and what does it affect?

positive: x IN subquery
negative: x NOT IN subquery, NOT(condition AND x IN subquery)
negatives turn move outs from the subquery to move-ins for the outer subquery
also to expand the filter during move-ins queries in flight:

for positive we need the union of the before and after move-in subquery view (include enough for before and after)
for negative we need the intersection of the before and after subquery view (don't exclude more until after completion)

github-actions · 2026-04-27T13:15:04Z

This PR has been released! 🚀

The following packages include changes from this PR:

@core/elixir-client@0.10.0
@core/sync-service@1.6.0

Thanks for contributing to Electric!

@alco

## Summary Follow-up to the merged subqueries PR #4051, addressing reviewer (@alco) feedback that came in after merge. Each commit maps 1:1 to a review comment so they can be reviewed independently. ## Changes | Commit | Review comment | | --- | --- | | `8827e20` Make storage `row()` type reflect actual value type | [r3130325864](#4051 (comment)) | | `fd7748e` Rename `position_info` boolean fields to `subquery?`/`negated?` | [r3130956496](#4051 (comment)) | | `91f8f17` Use `:ets.lookup_element/3` in `get_last_broadcast_lsn` | [r3130990531](#4051 (comment)) | | `dd8cb5c` Document `self()` message sent by `subscribe_to_global_lsn_updates` | [r3130996392](#4051 (comment)) | | `82ecbc2` Remove dead `:init_consumer` `handle_continue` clause | [r3131087003](#4051 (comment)) | | `1545bb3` Assert mixed-polarity subquery error message in router test | [r3131001659](#4051 (comment)) | | `6aa7581` Mark last converted change before reversing | [r3131488531](#4051 (comment)) | | `2a7fff5` Extract `value_tag/3` helper in router test | [r3131013153](#4051 (comment)) | These are internal-only changes (type docs, field renames, a dead-code removal, an ETS lookup simplification, test assertions/helpers, and a list-building cleanup) with no user-facing behavior change, so no changeset is included. A few comments were discussed and intentionally left unchanged: the `Stream.concat` suggestion ([r3131472813](#4051 (comment)), the accumulated lists are tiny and `reverse |> flatten` is clearer) and the `BETWEEN` question ([r3131209403](#4051 (comment)), confirmed as an intentional latent-bug fix required by this PR's `name`-based DNF decomposition / SQL generation). --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

icehaunter added the load-test label Mar 25, 2026

robacourt removed the load-test label Mar 25, 2026

robacourt force-pushed the rob/simple-subqueries-with-dnf branch 2 times, most recently from c7da8bb to f33c781 Compare March 25, 2026 14:38

robacourt added load-test and removed load-test labels Mar 25, 2026

robacourt force-pushed the rob/simple-subqueries-with-dnf branch 2 times, most recently from 6669eb8 to a1e0fc1 Compare March 26, 2026 12:38

robacourt removed the load-test label Mar 26, 2026

robacourt force-pushed the rob/simple-subqueries-with-dnf branch from 226f9b3 to 203f4fc Compare March 26, 2026 16:51

robacourt force-pushed the rob/simple-subqueries-with-dnf branch 6 times, most recently from 67f228e to 438e6cf Compare April 13, 2026 14:22

robacourt force-pushed the rob/simple-subqueries-with-dnf branch 3 times, most recently from 66d52fe to fb8a7d9 Compare April 15, 2026 15:40

electric-sql deleted a comment from github-actions Bot Apr 15, 2026

electric-sql deleted a comment from netlify Bot Apr 15, 2026

electric-sql deleted a comment from github-actions Bot Apr 15, 2026

electric-sql deleted a comment from codecov Bot Apr 15, 2026

robacourt changed the title ~~DRAFT: simple subqueries with DNF~~ DRAFT: keep boolean subquery shapes live with DNF move handling Apr 18, 2026

robacourt changed the title ~~DRAFT: keep boolean subquery shapes live with DNF move handling~~ Support subquery moves across AND/OR/NOT WHERE clauses Apr 18, 2026

robacourt force-pushed the rob/simple-subqueries-with-dnf branch from 413acaf to 4208d42 Compare April 18, 2026 20:46

robacourt marked this pull request as ready for review April 20, 2026 06:35

robacourt requested a review from icehaunter April 20, 2026 06:35

robacourt added the claude label Apr 22, 2026

alco reviewed Apr 23, 2026

View reviewed changes

alco mentioned this pull request Apr 23, 2026

Normalize shape key generation from where clause #3256

Open

alco approved these changes Apr 23, 2026

View reviewed changes

robacourt added 13 commits April 23, 2026 17:47

Compile compound subquery predicates to DNF

da00676

Emit DNF row metadata and move-in queries

7b4e8f0

Route subquery shapes through SubqueryIndex

48748c7

Rewrite subquery move handling around exact log splicing

5e881df

Update clients for DNF move metadata

6476a10

Expand oracle coverage for compound subquery predicates

c3583c2

Add changeset for compound subquery move support

d992639

Restore functionality

3b262da

Remove unused electric_protocol_version

0625d97

Fix move-in queries for repeated dependency positions

f8a0af5

Fix bpchar tag hashing in DNF queries

bc403af

Inline start_effects

07e1a0c

Fix legacy move-out handling without active conditions

ec39d48

robacourt force-pushed the rob/simple-subqueries-with-dnf branch from 801d714 to ec39d48 Compare April 23, 2026 16:48

robacourt merged commit a04b259 into main Apr 27, 2026
55 of 60 checks passed

robacourt deleted the rob/simple-subqueries-with-dnf branch April 27, 2026 11:04

alco mentioned this pull request May 18, 2026

Emit telemetry spans for move_in_query snapshot operations #4303

Merged

robacourt mentioned this pull request Jun 23, 2026

Address review feedback from #4051 #4646

Merged

electric-sql deleted a comment from claude Bot Jun 23, 2026

		tag2 =
		:crypto.hash(:md5, ctx.stack_id <> req.handle <> "v:2") \|> Base.encode16(case: :lower)

Conversation

robacourt commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Compatibility

Architecture

Secondary changes

Uh oh!

codecov Bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alco left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robacourt commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

robacourt commented Mar 24, 2026 •

edited

Loading

codecov Bot commented Apr 15, 2026 •

edited

Loading

robacourt commented Apr 23, 2026 •

edited

Loading