Apply calico-node before Typha during upgrades to prevent cluster-wide NotReady by AdheipSingh · Pull Request #5110 · tigera/operator

AdheipSingh · 2026-07-27T18:32:52Z

Description

Type of change: bug fix

Problem: During upgrades, the operator applies the Typha Deployment and the calico-node DaemonSet in the same reconcile, so both roll simultaneously. Typha (typically 3 replicas) finishes its rollout in about a minute, while calico-node on a large cluster takes much longer. Felix is compatible with an older Typha, but an older Felix cannot sync with a newer Typha — so once all Typha replicas are on the new version, every still-old calico-node pod loses sync, fails its readiness probe after ~90s (3 × 30s), and goes NotReady. The DaemonSet controller does not count already-unavailable pods against maxUnavailable/maxSurge, so it then replaces pods across the whole cluster at once instead of honoring the configured rolling-update limits. On a ~200 node production cluster this presented as a cluster-wide wave of NotReady calico-node pods during a 3.31 → 3.32 upgrade.

Fix:

Apply the calico-node component before the remaining components.
After applying, read the calico-node DaemonSet directly from the API server rather than the cached client — immediately after the write, the informer cache can still hold the pre-update object whose status reports the previous rollout as complete, which would open the gate prematurely (this race was observed in testing: identical code passed or failed depending on whether the watch event beat the read).
Gate the Typha component on the calico-node rollout being complete (ObservedGeneration >= Generation, all pods updated and ready) using the existing component Ready() mechanism.

With this ordering, new calico-node pods connect to old Typha (the compatible direction) while rolling under the configured limits, and Typha updates last, when every Felix is already on the new version. First installs are unaffected (no DaemonSet exists, so the gate is open), and in steady state the check passes immediately because the DaemonSet generation is unchanged.

Testing: Validated on a 30-node cluster upgrading Calico v3.31.3 → v3.32.1:

With nodeUpdateStrategy: {maxSurge: 4, maxUnavailable: 0} (the affected user's configuration): calico-node Ready stayed 30/30 for the entire upgrade across every DaemonSet status transition; peak 34 concurrent pods (30 + exactly 4 surge), no limit bypass; node phase ~4 minutes; Typha updated only after all 30 nodes completed.
With the default strategy (maxUnavailable: 1): Ready never dropped below 29/30 (the single pod being replaced), Typha gated for the full ~18 minute rollout and updated within seconds of the last node becoming ready.
Without this change, the same upgrade dropped Ready to 11/30, with the DaemonSet controller replacing ~12 pods within 2 seconds.

Affected components: installation controller (core_controller.go), Typha render component.

Release Note

During upgrades, the operator now waits for the calico-node DaemonSet rollout to complete before updating Typha, preventing cluster-wide calico-node readiness disruption caused by older Felix being unable to sync with newer Typha.

For PR author

Tests for change.
If changing pkg/apis/, run make gen-files
If changing versions, run make gen-versions

For PR reviewers

A note for code reviewers - all pull requests must have the following:

Milestone set according to targeted release.
Appropriate labels:
- kind/bug if this is a bugfix.
- kind/enhancement if this is a a new feature.
- enterprise if this PR applies to Calico Enterprise only.

During upgrades, applying Typha and calico-node in the same reconcile rolls both simultaneously. Felix may be newer than Typha but not older, so once Typha completes its rollout first, still-old calico-node pods can no longer sync, go NotReady after their readiness probe window, and the DaemonSet controller stops honoring surge limits and replaces pods cluster-wide. Apply calico-node first, then gate Typha on the DaemonSet being fully rolled out. Read the DaemonSet directly from the API server since the informer cache may still hold the pre-update object immediately after the write.

CLAassistant · 2026-07-27T18:33:19Z

All committers have signed the CLA.

caseydavenport · 2026-07-29T17:53:58Z

@AdheipSingh thanks for the PR - was there a specific issue you noticed that prompted this?

We try to make the upgrade not depend on ordering regardless, so curious if we introduced an unexpected dependency in v3.32.

caseydavenport · 2026-07-29T17:27:19Z

+		return reconcile.Result{}, err
+	}
+
+	// Check whether the calico-node rollout is complete; Typha is only


Is this the right behavior for a fresh rollout?

On a brand new cluster, we don't want to wait for calico/node to roll out before deplying Typha, since typha is needed for calico/node to become ready.

I suspect this will cause a deadlock on a fresh cluster.

You were right , the previous revision deadlocked fresh installs (the gate waited for node readiness, and node needs Typha to become ready). Fixed in the latest revision in two ways: readiness is no longer part of the condition, and only the Typha Deployment update is deferred on a fresh install (no existing Deployment) everything is created immediately. Verified on a cluster: Typha was created on the first reconcile, all nodes came up, and the deferral stops once the DaemonSet is fully scheduled.

caseydavenport · 2026-07-29T17:28:42Z

+
+	// Check whether the calico-node rollout is complete; Typha is only
+	// applied once it is. Read the DaemonSet directly from the API server
+	// rather than the informer cache: immediately after the update above,


I don't think we should be doing a direct read against the API server here for perf reasons - we can use the cache if we get the existing node DS generation prior to applying the new version, I think.

Removed the direct read. The check now runs entirely from the informer cache.

caseydavenport · 2026-07-29T17:37:44Z

+	} else {
+		nodeRolledOut = nodeDS.Status.ObservedGeneration >= nodeDS.Generation &&
+			nodeDS.Status.UpdatedNumberScheduled == nodeDS.Status.DesiredNumberScheduled &&
+			nodeDS.Status.NumberReady == nodeDS.Status.DesiredNumberScheduled


NumberReady is a steady state health check rather than a rollout check. One permanently unready calico-node pod (cordoned node, wedged kubelet, unrelated crashloop) leaves this false forever, so Typha stops getting config updates entirely, not just during upgrades.

What we actually care about is that no old-version Felix is left, which is the UpdatedNumberScheduled term on its own. We should drop the NumberReady check.

Agreed, dropped. The condition is now only ObservedGeneration >= Generation && UpdatedNumberScheduled == DesiredNumberScheduled. Added a unit test pinning the "all pods updated but some unready must not block Typha" case so it can't creep back in.

caseydavenport · 2026-07-29T17:37:45Z

+		nodeRolledOut = nodeDS.Status.ObservedGeneration >= nodeDS.Generation &&
+			nodeDS.Status.UpdatedNumberScheduled == nodeDS.Status.DesiredNumberScheduled &&
+			nodeDS.Status.NumberReady == nodeDS.Status.DesiredNumberScheduled
+	}


Nothing reliably wakes us back up when the rollout finishes. The DaemonSet watch from secondaryResources() goes through createPredicateForObject, whose UpdateFunc filters on generation change, and rollout progress is status only. So the gate stays shut until the 5 minute periodic reconcile fires.

I suspect the testing here looked fast because other watches (FelixConfiguration, secrets) are noisy enough to mask it. We should return a RequeueAfter while gated, and log that we're deliberately holding Typha back.

The latest revision returns utils.StandardRetry while the update is deferred and logs "Deferring Typha Deployment update until the calico-node DaemonSet rollout completes" with the updated/desired counts on each reconcile.

caseydavenport · 2026-07-29T17:37:47Z

 		}
 	}

+	components = append(components, typhaComponent)


Moving the append down here also reorders Typha relative to the cleanup passthrough above. Probably harmless, but it doesn't look intentional - can we leave the append where it was?

caseydavenport · 2026-07-29T17:37:48Z


 func (c *typhaComponent) Ready() bool {
-	return true
+	return c.cfg.NodeRolledOut


Worth keeping in mind that CreateOrUpdateOrDelete skips the whole component when Ready() is false, not just the Deployment. So this also stops reconciling the typha Service, ServiceAccount, RBAC, PDB, cert secret and NetworkPolicy.

That's my main concern with gating rather than just ordering the two applies: any state where calico-node is unready because something on the Typha side is missing or broken now becomes unrecoverable, since the operator won't touch Typha until node is healthy.

agreed. The latest revision removes the Ready()gating entirely.

caseydavenport · 2026-07-29T17:38:26Z

 		return reconcile.Result{}, err
 	}

+	// Apply calico-node before the remaining components so that its rollout


This ordering is inverted for downgrades, where node rolls to the older Felix first while Typha is still new, which is exactly the direction the comment says doesn't work.

Wonder if we need more awareness on if this is an upgrade / downgrade / or simply a configuration change.

Correct — as written, the deferral applies to any image change, so a downgrade would also roll node first, which is the direction the skew policy doesn't support. Might consider detecting direction by comparing the resolved desired version against the deployed one and only deferring for upgrades (falling back to current behavior for downgrades and unparseable references like digests), but didn't want to add version-parsing machinery without your input. Would you prefer that in this PR, as a follow-up, or is there an existing pattern in the operator for knowing upgrade vs downgrade that we should use?

caseydavenport · 2026-07-29T17:38:28Z

+			nodeDS.Status.UpdatedNumberScheduled == nodeDS.Status.DesiredNumberScheduled &&
+			nodeDS.Status.NumberReady == nodeDS.Status.DesiredNumberScheduled
+	}
+	typhaCfg.NodeRolledOut = nodeRolledOut


This mutates typhaCfg a few hundred lines after we handed it to render.Typha() - I think we need to adjust this so that we are setting this field before passing the config to the renderer.

caseydavenport · 2026-07-29T17:57:33Z

/sem-approve

AdheipSingh · 2026-08-03T14:23:10Z

Problem: During upgrades, the operator applies the Typha Deployment and the calico-node DaemonSet in the same reconcile, so both roll simultaneously. Typha (typically 3 replicas) finishes its rollout in about a minute, while calico-node on a large cluster takes much longer. Felix is compatible with an older Typha, but an older Felix cannot sync with a newer Typha — so once all Typha replicas are on the new version, every still-old calico-node pod loses sync, fails its readiness probe after ~90s (3 × 30s), and goes NotReady. The DaemonSet controller does not count already-unavailable pods against maxUnavailable/maxSurge, so it then replaces pods across the whole cluster at once instead of honoring the configured rolling-update limits. On a ~200 node production cluster this presented as a cluster-wide wave of NotReady calico-node pods during a 3.31 → 3.32 upgrade.

Hi @caseydavenport , thanks for the review this is main reason ˆˆ for this PR. We have large nodes ( 1000 + ), and when we increase maxSurge to speed up rollouts, keeping maxUnavailable as 0, we see a large disruption. This was the reason for fixing this. We saw no issue when maxSurge is set to 1, as we increase surge we see this issue. Also this is visible when we tried to upgrade 30 nodes with maxSurge 4, maxSurge 1/2 works fine on 30 nodes.....so the disruption is proportional to no of nodes and maxSurge count.

AdheipSingh requested a review from a team as a code owner July 27, 2026 18:32

marvin-tigera added this to the v1.44.0 milestone Jul 27, 2026

marvin-tigera added docs-pr-required release-note-required labels Jul 27, 2026

caseydavenport requested changes Jul 29, 2026

View reviewed changes

AdheipSingh added 2 commits August 4, 2026 18:10

Defer only the Typha Deployment using cached pre-apply state

4ca53a1

Use utils.StandardRetry for the deferred requeue

28bb15c

Uh oh!

Conversation

AdheipSingh commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Release Note

For PR author

For PR reviewers

Uh oh!

CLAassistant commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

caseydavenport commented Jul 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AdheipSingh Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

caseydavenport commented Jul 29, 2026

Uh oh!

AdheipSingh commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AdheipSingh commented Jul 27, 2026 •

edited

Loading

CLAassistant commented Jul 27, 2026 •

edited

Loading

AdheipSingh Aug 5, 2026 •

edited

Loading