fix(data-connections): target GCS, align DataProvider with storage_type - #452
Closed
alukach wants to merge 2 commits into
Closed
fix(data-connections): target GCS, align DataProvider with storage_type#452alukach wants to merge 2 commits into
alukach wants to merge 2 commits into
Conversation
The GCP data connection persisted `provider: "gcp"`, but the backend
addresses Google Cloud Storage as "gcs" — the same latent mismatch existed
for Azure ("az" vs "azure"). Root cause: `DataProvider` and a mirror's
`storage_type` were two vocabularies bridged by a hand-written
`STORAGE_TYPE_BY_PROVIDER` map duplicated in products.ts and
product-mirrors.ts.
Detangle: align `DataProvider`'s values to the backend storage vocabulary
(`Azure="azure"`, `GCS="gcs"`), so a mirror's `storage_type` is just the
connection's provider. Both translation maps deleted; `storage_type =
details.provider` directly.
- Rename member GCP→GCS and schema/type Gcp*→Gcs* (auth type stays
`gcp_workload_identity` — WIF is a GCP-platform feature, not storage).
- Read-shim in the data-connections DynamoDB client normalizes legacy rows
on read ("az"→"azure", "gcp"→"gcs"); mirrors already store the correct
storage_type via the old map, so only connection rows need it.
- Tests for the shim and for the DataProvider ⊆ storage_type invariant that
now backs the cast.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Claude finished @alukach's task in 1m 34s —— View job Claude's review
✅ No blocking issues — safe to merge.
|
Nothing branched on `storage_type`: the data proxy keys off the connection's `details.provider`, and the only source.coop consumer was a display label. It was a denormalized copy of the provider, bridged by a cast. Remove the field and derive the label from the connection instead. - Drop `storage_type` from ProductMirrorSchema; stop writing it in createProduct / addProductMirror (removes both provider→storage_type casts and the DataProvider ⊆ storage_type invariant test they needed). - Mirror "Type" label now reads the provider from connectionInfo (sourced from the connection's details.provider on the edit page). - No migration: ProductMirrorSchema is a plain z.object, so existing records that still carry storage_type parse fine (unknown key stripped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Split into two focused PRs:
Closing this combined PR in favor of those. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Creating a GCP data connection persisted
details.provider: "gcp", but the backend (data.source.coop) addresses Google Cloud Storage as"gcs". The same latent mismatch existed for Azure ("az"vs"azure").Root cause:
DataProvider(connection config) andProductMirror.storage_type(a per-mirror field) were two vocabularies bridged by a hand-writtenSTORAGE_TYPE_BY_PROVIDERmap duplicated inproducts.tsandproduct-mirrors.ts.Changes
1. Align
DataProvidervalues to the backend vocabularyBoth translation maps deleted. Member
GCP→GCS, schema/typeGcp*→Gcs*. Auth type staysgcp_workload_identity(WIF is a GCP-platform feature, not storage). Form label → "Google Cloud Storage".2. Remove
ProductMirror.storage_typeentirelyNothing branched on it — the proxy keys off
details.provider, and the only source.coop consumer was a display label. It was a denormalized copy of the provider. Dropped the field; the mirror "Type" label now derives from the connection's provider. This dissolves the "two vocabularies" problem rather than just aligning it, and removes both provider→storage_type casts.3. Migration (legacy
az/gcpconnection rows)DynamoDB reads bypass Zod, so
normalizeConnectionin the data-connections client remaps"az"→"azure"/"gcp"→"gcs"onfetchById/listAll. Markedponytail:— drop after backfill. Removingstorage_typeneeds no migration:ProductMirrorSchemais a plainz.object, so old records still carrying the field parse fine (unknown key stripped).Backend compatibility (verified against data.source.coop)
"az"and"azure"→ compatible during rollout. GCS: old"gcp"never worked (hit the error arm);"gcs"is accepted. So this is the fix, not a regression.gcpcargo feature — and that currently fails to compile forwasm32-unknown-unknown(object_store's GCS credential signing usesring::SystemRandom, unsupported on that target). Tracked separately; not in this PR.Verification
tsc --noEmitclean ·jestaffected suites 116/116 · lint clean on changed files.🤖 Generated with Claude Code