flannel migration: document removing leftover flannel iptables rules - #2768
flannel migration: document removing leftover flannel iptables rules#2768stitrace wants to merge 4 commits into
Conversation
✅ Deploy Preview for calico-docs-preview-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview succeeded!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
@stitrace Thanks for the submission. Someone will take a look at this together with projectcalico/calico#12920 |
|
Is this an alternative to projectcalico/calico#12920 ? With the automated removal in the linked PR, presumably we don't need manual steps? |
|
Not quite an alternative — they're complementary. #12920 (projectcalico/calico#12920) automates the cleanup for migrations performed with a Calico version that includes the fix, so going forward the manual steps won't be needed. However, the leftover chains don't clean themselves up: any cluster that already migrated with an earlier version still has the stale Flannel iptables rules sitting there, and anyone migrating on an older release will keep hitting this. For those users the docs are the only discoverable fix. I'd suggest trimming this PR down to a short note along the lines of: "migrations performed with Calico vX.Y+ remove these rules automatically; if you migrated with an earlier version, clean them up manually as follows" — with the version filled in once the fix ships in a release. Happy to rework the PR that way if it sounds good. |
|
Got it. Also, just a thought - for a lot of folks simply doing a rolling update of cluster nodes will be an easier way to handle this rather than scripting running these commands across a whole cluster. Might be worth taking that approach? |
|
Good point, agreed. Since flannel is already gone after the migration, nothing re-creates the chains, so a rolling reboot (or node replacement in immutable/managed node setups) fully clears them. The current text mentions rebooting only as a one-line "alternatively" at the end. I'll flip the emphasis: recommend a rolling reboot/replacement of nodes as the primary approach, and keep the script as an option for clusters where cycling every node is impractical, or when you need to restore cross-node traffic immediately without waiting for a full node roll (the SNAT breakage affects live traffic until the rules are gone). I'll push an update shortly. |
|
|
ac8072b to
c123eca
Compare
|
Thanks for this, @stitrace One suggestion on the step's shape. The rest of this guide keeps each step to one imperative line plus a code block, so the five paragraphs of lead-in here read as heavy by comparison. The two methods are the valuable part and they get a bit buried. I'd trim the exposition but keep both methods as a clear either/or: This change currently targets 3.33+. Does this need to be applied also to previous versions? |
|
Thanks — applied your restructure in 24c425a. The step is now one mechanism paragraph plus the two methods as an explicit either/or, with a single sentence of rationale each; the command block is unchanged. Net is 44 → 21 added lines. On the backport question: the page is byte-identical across I've kept this PR to One note on the failing checks: the |
|
considering that the fix will likely land in 3.33, then we should remove this change from calico/ (because the fix removes the requirement) and add the change to all the versioned directories. I suspect the build will correct itself if you fetch and rebase your PR. @stitrace |
The live-migration controller removes the flannel daemonset and deletes the flannel network devices, but it does not remove the iptables chains flannel programs (FLANNEL-POSTRTG in nat, FLANNEL-FWD in filter). These survive the migration and the FLANNEL-POSTRTG masquerade rule keeps SNAT-ing cross-node pod-to-pod traffic to the node tunnel IP, which silently breaks NetworkPolicy after migration. Add a cleanup step (with a reboot alternative) so operators can remove the leftover rules.
The controller-side fix lands in 3.33, so the manual step is only needed for clusters migrated with 3.29-3.32.
24c425a to
1019f4b
Compare
|
Done — rebased on You were right about the build: the rebase picked up ~30 commits of One edge case worth a second opinion, not a blocker: the controller fix cleans up during migration, so it helps clusters migrating on 3.33. A cluster that migrated on 3.31 and later upgrades to 3.33 still carries the leftover chains, and its operators will be reading the 3.33 page, which no longer mentions them. If you think that's worth covering, the natural home is a short note in the 3.33 release notes or upgrade guide rather than the migration page — happy to open a follow-up for that. Otherwise this is ready as-is. |
|
@caseydavenport PTAL, I think this is good to go. Anything else to add? |
4 participants
Description
The live-migration guide tells you to delete the migration controller once migration completes, but it does not mention that flannel leaves iptables rules behind on every node.
The migration controller removes the flannel daemonset and deletes the flannel network devices (
flannel.<vni>,cni0), but it does not remove the iptables chains flannel programs:FLANNEL-POSTRTG(nat) andFLANNEL-FWD(filter). These survive the migration.The masquerade rule in
FLANNEL-POSTRTGkeeps SNAT-ing cross-node pod-to-pod traffic to the node's tunnel IP. This is invisible until you useNetworkPolicy: the SNAT'd source no longer matches pod-selector rules, so Calico's default-deny drops the traffic. Symptom after an otherwise successful migration: cross-node connections to policy-selected pods silently time out, while same-node traffic keeps working.This PR adds a cleanup step after "Delete the migration controller" with an idempotent flush command (legacy + nft backends) and a reboot alternative.
Notes
calico/...); maintainers may want to backport to the versioned snapshots.Reproduction
After migrating, on any node:
nft list chain ip nat POSTROUTINGstill showsjump FLANNEL-POSTRTGwith non-zero counters; a cross-node listener sees the client's source as the sender node's tunnel IP (<block>.0) rather than the pod IP, and NetworkPolicy-selected pods become unreachable cross-node.