Skip to content

fix(data-connections): target GCS, align DataProvider with storage_type - #452

Closed
alukach wants to merge 2 commits into
mainfrom
worktree-gcp-to-gcs
Closed

fix(data-connections): target GCS, align DataProvider with storage_type#452
alukach wants to merge 2 commits into
mainfrom
worktree-gcp-to-gcs

Conversation

@alukach

@alukach alukach commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Problem

Creating a GCP data connection persisted details.provider: "gcp", but the backend (data.source.coop) addresses Google Cloud Storage as "gcs". The same latent mismatch existed for Azure ("az" vs "azure").

Root cause: DataProvider (connection config) and ProductMirror.storage_type (a per-mirror field) were two vocabularies bridged by a hand-written STORAGE_TYPE_BY_PROVIDER map duplicated in products.ts and product-mirrors.ts.

Changes

1. Align DataProvider values to the backend vocabulary

enum DataProvider { S3 = "s3", Azure = "azure" /* was "az" */, GCS = "gcs" /* was GCP="gcp" */ }

Both translation maps deleted. Member GCPGCS, schema/type Gcp*Gcs*. Auth type stays gcp_workload_identity (WIF is a GCP-platform feature, not storage). Form label → "Google Cloud Storage".

2. Remove ProductMirror.storage_type entirely

Nothing branched on it — the proxy keys off details.provider, and the only source.coop consumer was a display label. It was a denormalized copy of the provider. Dropped the field; the mirror "Type" label now derives from the connection's provider. This dissolves the "two vocabularies" problem rather than just aligning it, and removes both provider→storage_type casts.

3. Migration (legacy az/gcp connection rows)

DynamoDB reads bypass Zod, so normalizeConnection in the data-connections client remaps "az"→"azure" / "gcp"→"gcs" on fetchById/listAll. Marked ponytail: — drop after backfill. Removing storage_type needs no migration: ProductMirrorSchema is a plain z.object, so old records still carrying the field parse fine (unknown key stripped).

Backend compatibility (verified against data.source.coop)

  • S3 unchanged. Azure accepts both "az" and "azure" → compatible during rollout. GCS: old "gcp" never worked (hit the error arm); "gcs" is accepted. So this is the fix, not a regression.
  • ⚠️ Separate prerequisite for GCS to actually connect: the data proxy must enable the multistore gcp cargo feature — and that currently fails to compile for wasm32-unknown-unknown (object_store's GCS credential signing uses ring::SystemRandom, unsupported on that target). Tracked separately; not in this PR.

Verification

tsc --noEmit clean · jest affected suites 116/116 · lint clean on changed files.

🤖 Generated with Claude Code

The GCP data connection persisted `provider: "gcp"`, but the backend
addresses Google Cloud Storage as "gcs" — the same latent mismatch existed
for Azure ("az" vs "azure"). Root cause: `DataProvider` and a mirror's
`storage_type` were two vocabularies bridged by a hand-written
`STORAGE_TYPE_BY_PROVIDER` map duplicated in products.ts and
product-mirrors.ts.

Detangle: align `DataProvider`'s values to the backend storage vocabulary
(`Azure="azure"`, `GCS="gcs"`), so a mirror's `storage_type` is just the
connection's provider. Both translation maps deleted; `storage_type =
details.provider` directly.

- Rename member GCP→GCS and schema/type Gcp*→Gcs* (auth type stays
  `gcp_workload_identity` — WIF is a GCP-platform feature, not storage).
- Read-shim in the data-connections DynamoDB client normalizes legacy rows
  on read ("az"→"azure", "gcp"→"gcs"); mirrors already store the correct
  storage_type via the old map, so only connection rows need it.
- Tests for the shim and for the DataProvider ⊆ storage_type invariant that
  now backs the cast.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 22, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
source-cooperative Ready Ready Preview, Comment Jul 22, 2026 5:41pm

Request Review

@claude

claude Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Claude finished @alukach's task in 1m 34s —— View job


Claude's review

  • Gather context (PR body, changed files list)
  • Review diff for correctness bugs, security issues, best-practice problems
  • Post findings

✅ No blocking issues — safe to merge.

  • The DataProvider rename ("az"→"azure", "gcp"→"gcs") is applied consistently across the codebase — no stray references to DataProvider.GCP, "gcp", or "az" remain outside the legacy-shim map (src/lib/clients/database/data-connections.ts:20-23) and its own doc comment.
  • normalizeConnection (src/lib/clients/database/data-connections.ts:25-30) is correctly wired into both fetchById and listAll, the only two raw-DynamoDB read paths for connections, and is well covered by data-connections.test.ts. update() doesn't need the shim since it always writes back a full canonical details object supplied by the caller.
  • ProductMirrorsManager.tsx:236-240 swaps mirror.storage_type for connectionInfo[...].provider — consistent with storage_type being fully dropped from ProductMirrorSchema (src/types/product.ts). If a mirror's underlying connection was deleted, connectionInfo[...] is undefined and "Type" renders blank, matching the existing degrade-gracefully pattern already used for "Bucket" on the same line — not a regression.
  • src/components/features/products/object-browser/ObjectSummary.tsx:31 correctly updates the literal comparison from "az" to "azure"; no branch exists for GCS there, but that gap pre-dates this PR (there was no "gcp" branch either) and is out of scope.
  • Schema/type renames (GcpDataConnectionSchemaGcsDataConnectionSchema, GcpDataConnectionGcsDataConnection) and their OpenAPI tags are consistent; gcp_workload_identity auth type name is correctly left unchanged since it names a GCP platform feature, not the storage backend.
  • Test updates align with the source changes (legacy-shim tests, dropped storage_type assertions, renamed GCS test cases).

Nothing branched on `storage_type`: the data proxy keys off the
connection's `details.provider`, and the only source.coop consumer was a
display label. It was a denormalized copy of the provider, bridged by a
cast. Remove the field and derive the label from the connection instead.

- Drop `storage_type` from ProductMirrorSchema; stop writing it in
  createProduct / addProductMirror (removes both provider→storage_type
  casts and the DataProvider ⊆ storage_type invariant test they needed).
- Mirror "Type" label now reads the provider from connectionInfo (sourced
  from the connection's details.provider on the edit page).
- No migration: ProductMirrorSchema is a plain z.object, so existing
  records that still carry storage_type parse fine (unknown key stripped).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alukach

alukach commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Split into two focused PRs:

Closing this combined PR in favor of those.

@alukach alukach closed this Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant