feat(web)!: replace the jeep-sqlite web implementation with @sqlite.org/sqlite-wasm + OPFS - #694
Open
kklem0 wants to merge 27 commits into
Open
feat(web)!: replace the jeep-sqlite web implementation with @sqlite.org/sqlite-wasm + OPFS#694kklem0 wants to merge 27 commits into
kklem0 wants to merge 27 commits into
Conversation
Contributor
Author
|
Three commits pushed to the branch since the review map above: two fixes and a docs correction, all found by running the branch on a physical iPhone and inside a real Vite app rather than in CI.
|
Pin @sqlite.org/sqlite-wasm at 3.53.0-build1 (SQLite 3.53.0) as a runtime dependency. The floor is 3.50, which is where pauseVfs/unpauseVfs landed. Add vitest, @vitest/browser, @vitest/browser-playwright and playwright as dev dependencies: OPFS sync access handles and dedicated workers do not exist in jsdom, so the web suite has to run in a real browser. @rollup/plugin-terser is for the prebuilt worker bundle, which ships as a built artifact rather than being minified by the consumer. No behaviour change yet.
…ngine src/web.ts was a 589-line pass-through to a <jeep-sqlite> DOM element, whose engine (sql.js in memory plus localforage image persistence) has been dormant since 2024-08 and has neither FTS5 nor OPFS. The core method set now runs on @sqlite.org/sqlite-wasm in a dedicated worker. Two durability tiers, one engine: - tier 1 keeps databases in OPFS through the opfs-sahpool VFS, which needs neither COOP/COEP headers nor SharedArrayBuffer and therefore works inside Capacitor WebViews and on ordinary hosting. - tier 2 is the automatic fallback where OPFS sync access handles are missing: the same engine, with databases in :memory: and whole-file images in IndexedDB. The image format is identical, so a database moves between tiers. A rejection from the VFS install is classified before any fallback. A pool held by another browsing context reports NoModificationAllowedError, which is indistinguishable from "this platform has no OPFS" unless inspected; falling back on it would open an empty database over the user's real data and then flush that empty image over the stored one. Only a genuine capability gap reaches tier 2, and a navigator.locks gate keeps a second context out before the install is attempted at all. Other behaviour worth calling out: - changes() and last_insert_rowid() are read inside the same worker op as the statement that produced them, so concurrent calls cannot interleave between the write and the read. - rows stay plain objects and BLOBs stay Uint8Array in both directions, so reorderRows remains a no-op for web results. - readonly connections become real on web: the read-only open flag on tier 1, PRAGMA query_only on tier 2, which has to open writable in order to deserialize its image. - errors keep their message instead of being stringified into "Error: Error:". - upgrade statements run on open, ported from electron-utils/utilsUpgrade.ts, with an in-memory pre-upgrade image standing in for that port's file backup. A failed ladder restores the image and fails the open rather than silently serving the pre-upgrade version. Rollup emits dist/web-worker.js as a classic, non-module worker with dist/sqlite3.wasm beside it. OPFS sync access handles land in Firefox 111 but module workers only in Firefox 114, so an ESM worker would fail to load on browsers that can otherwise run tier 1. setSqliteWorkerFactory and setSqliteWebOptions are additive exports for bundlers that cannot resolve the shipped worker; neither touches CapacitorSQLitePlugin. The eleven JSON, sync-table and asset methods still throw "Not implemented on web." pending their port.
Vitest browser mode against real Chromium, because OPFS sync access handles and dedicated workers do not exist in jsdom. Every contract test is parameterised over tier 1 and tier 2, tier 2 being forced through the documented configuration hook, and they drive the public SQLiteConnection and SQLiteDBConnection wrappers unmodified so that those are part of the surface under test rather than assumed correct. Coverage: the core method set, transactions including rollback, the upgrade version ladder and its failure path, BLOB round-trips, int64 as BigInt, FTS5 create/match/snippet, readonly enforcement at the engine rather than only at the wrapper, connection consistency reconciliation, tier classification including the busy-pool case that must not downgrade, the statement splitter, and a run of the shipped dist/web-worker.js rather than the TypeScript sources. browser.fileParallelism stays on. opfs-sahpool is single-owner per origin, which looks like it would collide across parallel files, but each file runs in its own browser context with its own storage partition. The consequence is the opposite of the obvious one: multi-context scenarios have to be built inside a single file with several workers. eslint and prettier are scoped to the real source roots. Both were globbing the whole working tree, so any scratch directory in a checkout failed npm run lint for reasons unrelated to the plugin. Coverage on a clean checkout is unchanged.
On tier 1 the pool name already isolates one store from another, so two stores configured with different pool names keep separate databases. Tier 2 wrote every image into a single origin-wide IndexedDB database, so the same two stores shared their data. Nobody would expect isolation to depend on which durability tier the browser happened to select. The IndexedDB name is now derived from the pool name. Nothing has to migrate: the tier 2 store has not been released.
copyFromAssets accepts a .zip entry in databases.json as an archive of databases, so the web implementation needs an inflater. fflate over jszip: MIT rather than dual MIT-or-GPL, which is a complication an MIT plugin does not need; 21.6 KiB gzip against 27.7; and it has a streaming API should the zip path ever need one.
isJsonValid, importFromJson and exportToJson, ported from electron/src/electron-utils/ImportExportJson with jeep-sqlite as the cross-check. The two turned out to be the same logic under the same names, which makes the row semantics below the real contract rather than one implementation's habits: - mode "full" drops and recreates; mode "partial" inserts rows whose key is absent and updates the rest. - In partial mode a row carrying sql_deleted = 1 deletes the local row, which is how a soft delete replicates from a server into a client database. - An UPDATE whose stored row already matches is skipped, so importing the same payload twice reports no changes rather than churning last_modified. - A value arriving as an array of numbers is a BLOB and becomes a Uint8Array. exportToJson is BigInt-aware, which is web-specific. sqlite-wasm returns int64 values above 2^53 as BigInt and JSON.stringify throws TypeError on those, so an export that ignored it would hand back an object the caller cannot serialise. Casting to Number would be worse: it silently corrupts exactly the values BigInt exists to protect. Values inside the safe-integer range stay numbers, anything outside becomes its decimal string, and SQLite's INTEGER affinity converts that back to the identical int64 on import. Also adds the worker event channel the facade forwards to notifyListeners, restoring sqliteImportProgressEvent and sqliteExportProgressEvent. Encrypted payloads are refused with an explicit message rather than half-imported; there is no SQLCipher in wasm.
createSyncTable, setSyncDate, getSyncDate and deleteExportedRows, plus the DELETE rewrite in the statement layer, ported from electron-utils with jeep-sqlite as the cross-check. A database opts into the sync protocol by giving its tables both a last_modified and a sql_deleted column. From then on a DELETE marks the row instead of removing it, so the next export can tell the server about it, and deleteExportedRows is what finally reclaims rows a completed export already carried away. The rewritten statement carries AND sql_deleted = 0, which keeps it idempotent: deleting an already-marked row reports no changes rather than touching last_modified again. The gate keys off the columns rather than off sync_table, matching the port source. Keying off the table would mean a DELETE behaved differently either side of a createSyncTable call that is documented as bookkeeping. The answer is cached per connection and invalidated on any DDL, since computing it costs a PRAGMA per table. Connection.run gains rewriteDeletes, the equivalent of the port source's fromJson flag, because the plugin's own deletes must not be rewritten. deleteExportedRows exists precisely to reclaim marked rows, and a JSON import replicating an upstream deletion has already been told the row is gone; rewriting either would turn it into a silent no-op.
… flows
The last four M2 methods, restoring the three remaining web events:
sqliteHTTPRequestEndedEvent, sqlitePickDatabaseEndedEvent and
sqliteSaveDatabaseToDiskEvent.
Downloads stream. A plain database is fed to importDb's async-callback form one
network chunk at a time, so a multi-hundred-megabyte bundle costs one chunk of
memory rather than its own size. The fetch runs inside the worker, so no
MessagePort is involved: the pull protocol with backpressure exists for the
case where the main thread owns the source, and here producer and consumer are
the same thread. Zip archives are the exception and are materialised, because
inflating requires the archive in memory; that is a property of the container,
not of this path, and it is why zip assets should stay modest.
copyFromAssets accepts both manifest shapes, a bare array and
{ databaseList: [...] }, rather than guessing which one an existing app ships.
getFromLocalDiskToStore and saveToLocalDisk live on the main thread because
they are DOM flows: one opens a file picker, the other triggers a download.
The boundary between the plugin and that chrome is injectable through the
additive setSqliteLocalDiskAdapter export, so the logic either side of it is
testable and an app with its own file UI can supply one. setSqliteWebOptions
gains assetsPath for apps that do not host databases under assets/databases/.
Every ported method on both tiers, 76 further tests. copyFromAssets and getFromHTTPRequest fetch from inside the worker, which has its own global scope, so stubbing fetch on the main thread reaches nothing. The suite runs a small in-memory fixture endpoint on the vitest dev server instead: a test builds a real database through the plugin, PUTs its bytes to /__fixture/..., and the worker fetches them back over real HTTP with a real streaming body. Nothing leaves localhost and no product hook exists purely for tests. The file picker and the download sink are replaced through the injectable adapter, which covers both events, cancellation, overwrite = false and a save-then-pick round trip. The browser's own picker dialog stays manual-only: it needs a user gesture no automated browser will supply. Two tests earn their keep by having already caught bugs: an int64 above 2^53 exported and re-imported without precision loss, and deleteExportedRows physically removing rows, which the soft-delete rewrite had been silently turning into a no-op.
verify:web runs the browser test suite now, and vitest browser mode needs a Playwright browser that npm ci does not install.
The previous web implementation kept every database as a whole-file image in a localforage store. The first initWebStore() reads that store with raw IndexedDB, imports each image through the same path a downloaded database takes, verifies it with PRAGMA integrity_check, and only then deletes the legacy store. The rules come from jeep-sqlite 2.8.0 driven in a real browser rather than from reading it: the legacy database is at version 2 with a second object store that localforage adds itself, a database opened but never saved leaves its key with an undefined value, and an interrupted version upgrade leaves a backup- copy behind. None of the three blocks retiring the store. Nothing is deleted while any database failed to import or failed its integrity check, nothing is ever written over a name already in the store, and the run happens at most once: retrying would re-import old images over databases the app has been writing to since.
The seeded store is shaped like the real one, down to localforage's own second object store, and the database image is jeep-sqlite's own output rather than a stand-in. Twenty-seven cases across both tiers: the happy path, orphan and lone backup keys, never-saved placeholders, undecodable values, corrupt and half-written images, two keys claiming one database, a live database of the same name, the once-only marker and the skip option.
…ernals Nothing imports jeep-sqlite any more. The localforage and sql.js rollup externals and globals date from a web implementation deleted in 2021.
There is no element to define, no package to install alongside the plugin and no sql-wasm.wasm to copy: the worker and sqlite3.wasm ship in dist/. Documents what was never written down: the two durability tiers and what saveToStore means on each, both browser floors including the one below which the plugin does not load at all rather than degrading, the single-owner tab rule, the automatic migration, BigInt for int64 above 2^53, and that read-only connections now work on web. Corrects the storage location and the WAL claim in the API reference, whose opening prose docgen does not regenerate. JSDoc only in definitions.ts, no signature changed; the docgen output it regenerates is committed alongside.
CHANGELOG follows the generated shape, breaking changes first. info_releases gains a CAPACITOR 8 bucket, since the file was still stale at Capacitor 4, and supersedes the 3.2.3-1 entry whose initWebStore snippet still shows the element bootstrap.
sqlite defaults PRAGMA foreign_keys to OFF, per connection, and nothing here ever turned it on. A web database therefore ignored every constraint its own schema declared: no cascade, no RESTRICT, no check on insert. Both port sources set it at open (electron utilsSQLite.ts:63, jeep Database.open), and the JSON-import and upgrade paths in this repo already toggled it off and back on as though ON were the resting state. BREAKING CHANGE: foreign key constraints declared in a schema are now enforced on the web platform, as they already were on Android, iOS and Electron. An app whose data violates its own declared constraints will begin receiving constraint errors on insert and update, and ON DELETE actions will start firing where they previously did nothing.
… RETURNING Five defects in the rewrite, all of them silent. A double-quoted table name extracted as WHERE, because stripNoise blanks quoted identifiers along with string literals and a greedy match read straight past them. The rewrite then targeted a table of that name, the per-table check found nothing, and the statement ran as an ordinary DELETE: the rows were destroyed and the server was never told. A string literal in the clause produced SQL that did not parse, since the blanked copy was returned rather than the original. RETURNING did the same, landing inside the generated WHERE. An unbracketed clause let AND bind tighter than OR, so the guard covered the last disjunct only and an already-deleted row was marked again on every repeat, republishing its tombstone. A table whose name is a reserved word made an internal PRAGMA a syntax error, which switched soft deletes off for the whole database. The rewrite is also gated on the target table now rather than on the database: a schema where one table is tracked and another is not is legal, and rewriting a DELETE against the second produced SET sql_deleted = 1 on a table with no such column.
A DELETE on a sync-tracked database is recorded rather than performed, so sqlite never fires the ON DELETE actions the schema declares. A deleted parent therefore left its children live, and the next export told the server the parent was gone while still reporting the children: the two ends diverged with no error on either. Relationships come from PRAGMA foreign_key_list rather than a regex over sqlite_master. The port source's pattern reads only the first matching table, so a parent with two referencing tables cascaded into one of them, misses column-level REFERENCES entirely, and matches any table whose DDL merely mentions the name. It also propagates a single level, leaving grandchildren live under a deleted grandparent. The walk recurses, and marks each row before descending, which is what makes a cyclic or self-referencing schema terminate: a marked row no longer satisfies sql_deleted = 0 and cannot be reached twice. A WITHOUT ROWID table on the receiving end of an ON DELETE action is reported rather than skipped. A referencing table with no sql_deleted column is left to sqlite, which runs the real action when deleteExportedRows performs the physical delete. A DELETE inside an execute() batch now takes the same path, as it does in the port source; without that, which entry point the app used decided whether a deletion reached the server. DDL issued through run() invalidates the cached schema, so a foreign key added that way is visible to the next delete.
The two durability tiers do not read each other. A browser that lacked OPFS sync access handles stores whole-file images in IndexedDB; when the same browser later gains them, tier selection picks tier 1, the pool is empty, and getDatabaseList returns nothing while the user's data sits in IndexedDB with no way back. That is not hypothetical: it is the normal update path for exactly the devices that run on tier 2, Android WebView reaching M132 and iOS 16.3 to 16.4. Every tier 1 init now adopts any image left in the fallback store through the same verified path the jeep-sqlite migration uses, checking integrity_check before removing the original. That ordering is what makes it safe to run on every boot instead of once: nothing is deleted until its replacement has been read back, so an interrupted run resumes on the next start and a browser that drops back to tier 2 still finds whatever has not moved. The pool wins a name conflict, because a database can only be in the pool if it was written after the flip. The losing image is kept rather than deleted: the only way both can exist is a promotion that failed and an app that then created the name itself, and in that case the image may be the copy that matters. The adoption contract the migration introduced moves to its own module, since both callers now need it.
… restart WKWebView invalidates OPFS access handles when the OS suspends the app, so the store cannot be left open across a background. The worker ops for this have existed since M1 but nothing drove them: there was no appStateChange wiring, no pause or resume on the facade, and no recovery path. pauseWebStore closes every connection and pauses the VFS, in that order, because pauseVfs throws SQLITE_MISUSE while any database is open. resumeWebStore unpauses and reopens what it closed, and falls through to tearing the worker down and starting a fresh one when it cannot, which is the recovery path 6.7 made binding and which a real suspension can require. A transaction open at pause time cannot survive the close, so it is rolled back and named on the console rather than silently abandoned. Auto-wiring uses visibilitychange rather than @capacitor/app, which is not a dependency of this plugin and should not become one for a single listener, and is installed only when running on a native Capacitor platform: a browser tab going hidden is not a suspension, and closing every connection on a tab switch would be a bug rather than a protection. Apps that want the more precise signal can call the three methods themselves, which is safe alongside the listener because all three are idempotent.
The message blamed bundler resolution for every failure, including the one F1 measured on WebView 61, where the real cause is a SyntaxError inside the shipped worker because the engine is below the floor this plugin supports. Telling those two apart is the difference between a five-minute bundler fix and an unsupported browser. A parse failure is now reported as an unsupported engine, naming the floors and saying that no build target in the app can change it, since the syntax belongs to @sqlite.org/sqlite-wasm and BigInt is a runtime dependency of int64. Anything else keeps the bundler message. Both quote what the browser actually reported: a guess that hides its evidence is worse than no guess.
…rwrite An overwrite with a full import has to close and unlink before it can rebuild, but it never reopened. The app was still holding those connections, so its next statement failed with "Database X is not open" while the facade registry still believed the connection was open. The closed set is now recorded and restored once the import has finished.
Adds the Backgrounding under Capacitor section, the tier-promotion paragraph under Durability tiers, and two entries under Limitations: that foreign keys are enforced, and what the soft-delete cascade needs from a schema. Extends the unreleased 8.2.0 notes with the behaviour changes and the fixes that came with them.
pauseWebStore recorded the connections it had closed only after its worker round trip resolved, and resumeWebStore returned a no-op when it found nothing recorded. On a physical iPhone the OS freezes the page inside that round trip and then delivers the foreground signal first, so the resume ran and returned before the pause completed, and the pause then closed every connection with nothing left to reopen them. Every query afterwards failed with "Database X is not open" until the app went round the background cycle a second time. Two of three real background cycles on an iPhone 16 Pro hit it, including one of only 30 seconds with the screen on, so it is the ordinary case rather than an edge of long suspensions: nothing keeps a hidden WKWebView running, and whether a round trip started at visibilitychange completes is the OS's decision. pauseWebStore now records the pause as soon as it starts, and resumeWebStore waits for that before deciding there is nothing to restore. restartWebStore deliberately does not wait: it is the recovery path and has to stay reachable even when a pause cannot finish. The tests drive the same interleaving without a device, both as two unawaited calls and as the unawaited pause the plugin's own visibilitychange listener issues.
A worker URL that does not resolve is answered with the application's own HTML, and the browser then reports "Uncaught SyntaxError: Unexpected token '<'". The classifier matched SyntaxError first, so a Vite build on Chrome 150 was told its browser is below Chrome 80 and that changing the build target cannot help, while the one message that names the fix, the bundler branch with setSqliteWorkerFactory, was suppressed. HTML is now recognised before the parse-failure branch and reported as WORKER_LOAD_FAILED. The signal is unambiguous: an old engine fails on the syntax it cannot parse, as WebView 61 did with "Unexpected token .", never on a tag.
Web-Usage.md claimed that Vite, the Angular builder and webpack 5 all resolve and emit the worker and the wasm without help. Vite 7 does not, measured with a real build of a consumer app: the plugin's worker URL is computed at runtime rather than written as a literal new URL(..., import.meta.url), so Vite rewrites it to point at a module of the plugin's own that it emits as an asset, web-worker.js is never emitted, and the request 404s. Serving both files from the web root and pointing the plugin at them is now the documented default because it always works, with a note that Vite needs it and that Capacitor copies the web root into the native apps on sync. The claim about the other bundlers is gone rather than restated: it was never measured. The troubleshooting section separates a worker that 404s from an engine below the floor, since the two produce the same SyntaxError and only one of them is the user's browser.
Contributor
Author
|
Rebased onto |
kklem0
force-pushed
the
feat/web-sqlite-wasm
branch
from
August 1, 2026 19:52
88869e9 to
5149665
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the web implementation (a pass-through to the dormant jeep-sqlite) with an in-repo engine on
@sqlite.org/sqlite-wasm: dedicated worker,opfs-sahpooldurability (no COOP/COEP), an automatic same-engine IndexedDB fallback tier for older browsers, a one-time verified migration of existing user data out of the jeep store, full 36-method web parity through the unmodified connection wrappers, and 259 browser-mode tests on both tiers wired into CI.definitions.tssignatures are untouched: no impact on iOS, Android, Electron.BREAKING CHANGE (web):
PRAGMA foreign_keysis now ON for every connection (previously never enabled, so declared constraints were silently ignored). Markedfix(web)!in the series.Review map, in landing order:
2a6b07d..78f716c)a6aca1e..ef72c8f)deaae48..aaef3c0)ef557dc..ede2a11)Every commit builds; tests land with their features or as the milestone suites.
To run:
npm ci && npx playwright install chromium && npm run build && npx vitest run.Context and discussion: #693. The lineage defects found while porting (the quoted-identifier real delete, the
ORprecedence gap, the Electron cascade issues) are described there; we are happy to file them as standalone issues as well.Added after opening:
7d6b882documents the tier 2 conditions, their measured market share (about 1.4% of global usage), and why the fallback tier exists (Capacitor 8 supports iOS 15+ and WebViews from Chrome 60, both below the tier 1 floor; Safari private browsing has no OPFS on any version).