Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions docs/data-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,18 @@

All data storage is cross-browser (Chrome, Firefox, Safari) — none depends on the service worker.

| Storage | What it stores | Purpose | Cross-browser |
|---|---|---|---|
| **IndexedDB** | Photo objects (full-size base64 image + thumbnail + all metadata), album objects, geo caches | Primary data store — the app reads from here on every load | Yes |
| **Disk (matrix-data.json)** | JSON dump of all IndexedDB content | Backup that survives browser data clearing; auto-saved by `serve.py` | Yes |
| **Disk (matrix-photos/)** | Individual image files (`{id}.jpg`, `{id}_thumb.jpg`) extracted from base64 | Used by the export flow (avoids embedding huge base64 in JSON) and for lightbox display after import | Yes |
| Storage | What it stores | Purpose |
|---|---|---|
| **matrix-photos/** | Full-size images (`{id}.jpg`) and thumbnails (`{id}_thumb.jpg`) | Source of truth for image data — written on upload, served directly for display |
| **IndexedDB** | Photo metadata (coords, date, camera, notes) + file path references to `matrix-photos/`, album objects | Source of truth for metadata — the app reads from here on every load |
| **matrix-data.json** | JSON backup of metadata, albums, and geo caches (file paths, not image data) | Backup that survives browser data clearing; auto-saved by `serve.py` |

**IndexedDB** is the source of truth during normal use. `matrix-data.json` and `matrix-photos/` are redundant backups maintained by `serve.py`. If you clear browser data, the app offers to restore from `matrix-data.json` on next load.
## How it works

## Scalability note
**Image data** lives on disk in `matrix-photos/`. IndexedDB stores only metadata and a path reference (e.g. `matrix-photos/p_123.jpg`), keeping the database lean (~50KB/photo vs ~4MB with embedded base64). This lets the app handle tens of thousands of photos without bloating browser storage.

IndexedDB currently stores full-size images as base64 inside each photo record. At scale (thousands of large photos), this can cause high memory usage at startup since all records are loaded into memory. A future refactor will move full-size images to `matrix-photos/` only, keeping IndexedDB lean (metadata + thumbnails). See the [planned refactor](../plans/reactive-bubbling-creek.md) for details.
**On upload**, `processFiles()` saves the full-size image to disk via `POST /api/photos/{id}`, verifies the file is servable, then stores the file path in IndexedDB.

**On load**, the app reads metadata from IndexedDB and references images by their disk paths. If IndexedDB is empty (e.g. after clearing browser data), the app offers to restore metadata from `matrix-data.json`. The image files in `matrix-photos/` remain intact regardless.

**Auto-save** (`scheduleAutoSave()`) writes metadata to `matrix-data.json` via `POST /api/data` after any data-modifying action. This file contains file paths — not image data — so it stays small.
2 changes: 1 addition & 1 deletion index.html
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
</div>
<div class="settings-item" onclick="emptyTileCache()">
<span class="settings-item-icon">🗺</span>
<div><div class="settings-item-label">Empty Map Cache</div><div class="settings-item-sub">Clear cached map tiles from disk</div></div>
<div><div class="settings-item-label">Empty Map Cache</div><div class="settings-item-sub" id="cache-size-status">Clear cached map tiles from disk</div></div>
</div>
</div>
</div>
Expand Down
10 changes: 10 additions & 0 deletions js/data.js
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,15 @@ function toggleSettingsMenu(e) {
const dd = document.getElementById('settings-dropdown');
dd.classList.toggle('open');
updateAutoSaveIndicator();
if (dd.classList.contains('open')) updateCacheSizeStatus();
}

function updateCacheSizeStatus() {
fetch('/api/tiles/size').then(r => r.json()).then(d => {
const mb = (d.bytes / (1024 * 1024)).toFixed(0);
const el = document.getElementById('cache-size-status');
if (el) el.textContent = `${mb} MB used (${d.limitMB} MB limit)`;
}).catch(() => {});
}
document.addEventListener('click', e => {
if (!e.target.closest('.settings-btn') && !e.target.closest('.settings-dropdown')) {
Expand Down Expand Up @@ -161,6 +170,7 @@ async function emptyTileCache() {
const tileCacheName = cacheNames.find(n => n.endsWith('-tiles'));
if (tileCacheName) await caches.delete(tileCacheName);
showToast(`Map cache cleared — ${d.removed} file${d.removed !== 1 ? 's' : ''} removed`, 'success');
updateCacheSizeStatus();
} else {
showToast('Failed to clear map cache', 'error');
}
Expand Down
20 changes: 20 additions & 0 deletions serve.py
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,8 @@ def do_GET(self):
try:
if self.path == "/api/data":
self._serve_data()
elif self.path == "/api/tiles/size":
self._tile_cache_size()
elif self.path.startswith("/api/tiles/proxy?"):
self._proxy_tile()
elif self.path.startswith("/api/video/download?"):
Expand Down Expand Up @@ -632,6 +634,24 @@ def _video_abort(self):
self.end_headers()
self.wfile.write(b'{"ok":true}')

def _tile_cache_size(self):
total = 0
count = 0
if os.path.isdir(TILES_DIR):
for root, _, fnames in os.walk(TILES_DIR):
for fn in fnames:
if fn.startswith('.'):
continue
try:
total += os.stat(os.path.join(root, fn)).st_size
count += 1
except OSError:
pass
self.send_response(200)
self.send_header('Content-Type', 'application/json')
self.end_headers()
self.wfile.write(json.dumps({'bytes': total, 'files': count, 'limitMB': MAX_TILES_MB}).encode())

def _clear_tile_cache(self):
"""Wipe the entire matrix-tiles directory and recreate it empty for a clean state."""
try:
Expand Down
Loading