private links: model links as rows with controller-observed status#3063
Draft
jshearer wants to merge 1 commit into
Draft
private links: model links as rows with controller-observed status#3063jshearer wants to merge 1 commit into
jshearer wants to merge 1 commit into
Conversation
21cd0a4 to
2a55fa9
Compare
2a55fa9 to
be57daf
Compare
The flat `private_links` array plus separate `*_link_endpoints` output columns gave no reliable way to tie an endpoint back to its link and persisted no per-link status. Promote each link to a `data_plane_private_links` row with a stable id and controller-owned status, so the API reads status instead of reconstructing it from an opaque, externally-owned shape. * New table + migration: per-link `config`, generated `service_identity`, controller-owned `status`/`details`/`error`, unique on `(data_plane_id, service_identity)`; a trigger wakes the controller and projects rows back into `private_links` during the transition. * dpc records `pending`/`provisioned` by matching provisioned endpoints to links on `(provider, service_identity)`, and keeps reading desired links from the projected `private_links` column so it has no deploy-ordering dependency on the agent-api cutover. * agent-api exposes `privateLinks` as rows and replaces the wholesale mutation with per-link add/update/remove; `failed` awaits a follow-up est-dry-dock change.
be57daf to
eb61a36
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reworks private networking so each configured private link is a first-class row with a system-assigned id and a data-plane-controller-owned status, replacing the flat
data_planes.private_linksJSON array paired with the separate*_link_endpointsoutput columns. This is the flow-repo and database side; the est-dry-dock change that turns on realfailedstatus is a tracked follow-up, and this PR stands on its own without it.This is the base of a stacked pair: #3070 sits on top and cuts the controller over to reading the table directly.
Background
The original private networking API exposed the raw primitives we already had: the
private_linkscolumn as editable desired config, and the three*_link_endpointscolumns as the controller's output. Building a useful UI on top surfaced two problems.service_name/service_attachment, a value that lives in an opaque blob whose shape est-dry-dock owns. The controller knows the exact input-to-output correspondence at apply time but discards it before anything reaches Supabase, so every consumer reconstructs downstream a correspondence that was unambiguous at the source. If that shape ever drifts, the match silently breaks and a healthy link reads as unprovisioned.The fix is to give each link a stable identity and persist its observed status alongside its desired config, owned by the controller, so the API reads status rather than reverse-engineering it. The desired config stays user-owned; the observed status is controller-owned, the usual reconcile split.
What changed
data_plane_private_linkstable. Each row has a system id, the parentdata_plane_id,provider, the polymorphicconfig, a generatedservice_identity, and controller-ownedstatus/details/error/observed_at, unique on(data_plane_id, service_identity). RLS and grants mirrordata_planes. One trigger wakes the parent's controller task on any change viainternal.send_to_task(prompt convergence the column path never had) and, during the transition, projects rows back into theprivate_linkscolumn so the not-yet-cut-over controller keeps working. The migration backfills one row per existing array element.(provider, service_identity): present means provisioned with the endpoint stored asdetails, absent means pending (the temporary bridge until est-dry-dock echoes the id). It keeps reading desired links from the projectedprivate_linkscolumn, so it has no deploy-ordering dependency on the agent-api cutover; moving its read onto the table is the stacked cutover (private-networking: controller reads links from the table, retire the projection #3070).privateLinksnow returns a row object{ id, provider, config, status, details, error, observedAt }read from the table. The wholesaleupdateDataPlanePrivateLinksmutation is replaced with per-linkaddDataPlanePrivateLink/updateDataPlanePrivateLink/removeDataPlanePrivateLink, authorized byModifyDataPlanePrivateNetworkingon the owning data plane, with the duplicate-identity case reported clearly. The config union is renamedPrivateLinkConfigin the schema; the new row type takes thePrivateLinkname.Rollout
Stacked so each step is independently safe to roll back, which matters given the prior private-links rollback from deploy ordering.
private_linkscolumn, so the API and controller binaries have no ordering dependency on each other.private_linksand*_link_endpointscolumns, recreate thedata_planes_overviewview once, and remove the now-redundant endpoint resolvers.Status semantics
Today the controller records
pendingandprovisionedby matching existing endpoint outputs.failedwith a reason needs est-dry-dock to emit a per-link result addressed by the id and to handle per-link errors so one bad link does not wedge the converge; the status columns are already shaped for it. That is a separate est-dry-dock change.Testing
service_identity()andprovider().privateLinks(AWS provisioned with details, Azure and GCP pending), plus add/update/remove and authorization tests including the duplicate-identity guard. Full suite green..sqlxoffline cache.Follow-ups (not in this PR)
(provider, service_identity)matching to id-addressed and populates realfailed/error.