Skip to content

fix(data-connections): align provider values to the backend vocabulary - #455

Merged
alukach merged 3 commits into
mainfrom
align-provider-values
Jul 22, 2026
Merged

fix(data-connections): align provider values to the backend vocabulary#455
alukach merged 3 commits into
mainfrom
align-provider-values

Conversation

@alukach

@alukach alukach commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Problem

Creating a GCP data connection persisted details.provider: "gcp", but the data proxy (data.source.coop) reads details.provider directly and only accepts "gcs"/"gs" for Google — never "gcp". The same latent mismatch existed for Azure ("az" vs the accepted "azure"). The provider→storage_type map already emitted the right values, so mirrors were fine — only the stored provider string was wrong.

Change — rename DataProvider values to what the backend accepts

enum DataProvider { S3 = "s3", Azure = "azure" /* was "az" */, GCS = "gcs" /* was GCP="gcp" */ }
  • Member GCPGCS, schema/type Gcp*Gcs*. Auth type stays gcp_workload_identity (WIF is a GCP-platform feature, not storage). Provider labels spelled out in the form.
  • Maps kept — the now-redundant provider→storage_type translation is left in place here and removed in the stacked follow-up (refactor(mirrors): drop denormalized ProductMirror.storage_type #456).

Migration (legacy az/gcp rows)

A one-time script — scripts/migrate-provider-values.ts — converts persisted details.provider values ("az"→"azure", "gcp"→"gcs"), matching the repo's existing migration convention (positional table arg, DynamoDB scan + guarded UpdateCommand, progress logging). Dry-run by default; --apply to write. Re-runnable (canonical rows are skipped). Run it (dry-run, then --apply) at/before deploy:

npx tsx scripts/migrate-provider-values.ts sc-prod-data-connections          # preview
npx tsx scripts/migrate-provider-values.ts sc-prod-data-connections --apply   # write

(Earlier revisions used a runtime read-shim; replaced with this script so the DB read path carries no legacy-value handling.)

Backend compatibility (verified against data.source.coop)

S3 unchanged. Azure accepts both "az" and "azure" → compatible during rollout. GCS: old "gcp" never worked (hit the error arm); "gcs" is accepted. Pairs with data.source.coop#191 (enables the GCS backend).

Verification

tsc clean · jest affected suites green · lint clean · fixture provider fixed (azazure) so module-load parsing passes.

🤖 Generated with Claude Code

Creating a GCP data connection persisted `details.provider: "gcp"`, but the
data proxy reads `details.provider` directly and only accepts "gcs"/"gs"
for Google (never "gcp"); the same latent mismatch existed for Azure ("az"
vs the accepted "azure"). The provider→storage_type map already emitted the
right values, so mirrors were fine — only the stored provider string was
wrong.

Rename the DataProvider values to what the backend accepts: Azure "az"→
"azure", GCP→GCS "gcp"→"gcs". Member GCP→GCS, schema/type Gcp*→Gcs*. Auth
type stays `gcp_workload_identity` (WIF is a GCP-platform feature, not
storage). Provider labels spelled out in the form.

Legacy rows: reads bypass Zod, so `normalizeConnection` in the
data-connections DynamoDB client remaps "az"→"azure" / "gcp"→"gcs" on
fetchById/listAll. Marked ponytail; drop after backfill.

The now-redundant provider→storage_type translation is left in place here
and removed in the follow-up that drops ProductMirror.storage_type.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 22, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
source-cooperative Ready Ready Preview, Comment Jul 22, 2026 10:29pm

Request Review

@claude

claude Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Claude finished @alukach's task in 1m 51s —— View job


Reviewing PR #455

  • Gather context and fetch diff
  • Analyze changed files for correctness/security/best-practice issues
  • Post final review

✅ No blocking issues — safe to merge.

The rename is applied consistently: enum member (GCPGCS), schema/type names, form labels, badge map, auth-type map, and all associated tests. storage_type in src/types/product.ts:12 already used "azure"/"gcs", so the two vocabularies now line up with no residual translation needed. No leftover references to the old "gcp"/"az" literals or GcpDataConnection naming remain outside the migration script/docs (verified via repo-wide grep).

One thing worth double-checking operationally rather than a code bug:

  • src/lib/clients/database/data-connections.ts (commit b42b9bd) removed the normalizeConnection read-time shim in favor of the standalone scripts/migrate-provider-values.ts. fetchById/listAll now return DynamoDB rows as-is with no Zod validation or aliasing (return (result.Items?.[0] as DataConnection) ?? null;). Any row still holding a legacy "az"/"gcp" provider value will silently fail the Record<DataProvider, ...> lookups that assume canonical keys — e.g.:

    • src/lib/actions/product-mirrors.ts:107-117storageTypeByProvider[connection.details.provider] returns undefined for "az"/"gcp", producing a mirror with an undefined storage_type.
    • src/lib/actions/products.ts:22-29 — same pattern for STORAGE_TYPE_BY_PROVIDER.
    • src/components/features/data-connections/DataConnectionForm.tsx:52-54authTypesByProvider lookup misses, breaking the auth-type selector for an existing legacy connection loaded into the edit form.

    Since the backend currently still accepts "az" for Azure (per the PR description), an existing, working Azure connection would only break if this code deploys before the migration script is run (or the script fails/is skipped) — there's no runtime fallback left to catch that ordering mistake. This is clearly intentional (documented in the script header and the DataProvider doc comment), so it's not a defect in the diff itself — just flagging that the safety net that used to exist unconditionally now depends entirely on the migration being run first. Worth confirming the deploy runbook/CI enforces that ordering, or adding a smoke check post-deploy.

Everything else — the migration script's dry-run default, ConditionExpression guard against concurrent writes, and re-run safety — looks correct.

The azure-data-connection fixture still carried the pre-rename provider
value; AzureDataConnectionSchema now requires "azure", and utils.mock.ts
eagerly parses this fixture at module load, so every test importing it (and
local dev seeding via init-local.ts) threw. Caught in CI review.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alukach

alukach commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Fixed in de9b172fixtures/data-connections.json azure connection provider azazure. Good catch; utils.mock.ts parses this fixture at module load so it took down every importing suite. Verified authz.test.ts (45) and the full fixture-consuming suites pass. Propagated the same fix into the stacked #456.

The remaining DropdownSection.integration.test.tsx failure is pre-existing on main and unrelated to this PR (doesn't touch that component).

@alukach
alukach marked this pull request as ready for review July 22, 2026 22:14
Replace the runtime normalizeConnection read-shim (LEGACY_PROVIDER_ALIASES)
with a one-time migration script that converts persisted provider values
"az"->"azure" / "gcp"->"gcs". Keeps the DB read path free of legacy-value
handling; the conversion is done once at deploy instead of on every read.

- Remove normalizeConnection + its test; fetchById/listAll return rows as-is.
- Add scripts/migrate-provider-values.ts, matching the existing migration
  convention (positional table arg, DynamoDB scan + guarded UpdateCommand,
  progress logging). Dry-run by default; --apply to write. Re-runnable
  (canonical rows are skipped).

Run the migration (dry-run, then --apply) before deploying the value rename.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alukach

alukach commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Replaced the runtime normalizeConnection read-shim with a one-time migration script (scripts/migrate-provider-values.ts, dry-run by default, --apply to write) per request — the DB read path no longer carries legacy-value handling. Propagated into the stacked #456.

@alukach
alukach merged commit 06adc4c into main Jul 22, 2026
7 checks passed
@alukach
alukach deleted the align-provider-values branch July 22, 2026 22:32
alukach pushed a commit that referenced this pull request Jul 29, 2026
🤖 I have created a release *beep* *boop*
---


##
[1.4.3](v1.4.2...v1.4.3)
(2026-07-27)


### Bug Fixes

* **data-connections:** align provider values to the backend vocabulary
([#455](#455))
([06adc4c](06adc4c))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: source-release-bot[bot] <265100246+source-release-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant