From 6690f06f90ac469a9a8c5fc79fe1fe12cc51ca5c Mon Sep 17 00:00:00 2001 From: Jack Carter <128555021+SunsetDrifter@users.noreply.github.com> Date: Wed, 29 Jul 2026 12:25:44 +0200 Subject: [PATCH 1/4] docs: masquerade-off HA is achievable, not impossible Both the routing-peers page and the masquerade page stated that high availability "stops working" with masquerade off. That reads as a flat impossibility and loses a legitimate use case (source IP preservation for audit, per-user database rules, or firewall policy keyed on the real source). What is actually true is narrower: NetBird's half of the failover still works, clients move to the standby peer on their own. What is missing is the return route inside the destination network following that move. State the condition instead of the impossibility, and add a short section on what meeting it takes: distinct metrics so the active peer is the same for every client, plus a virtual IP or a router-moved route so the next hop follows. Both live outside NetBird, so the section names the pattern rather than shipping a third-party config we would then own. Also: - Soften "failover is automatic and immediate". Clients re-select within seconds, which is fast but not instant, and "immediate" invites a support ticket from anyone who measures it. Phrased so it does not assert whether the client or the management side detects the loss, which this change did not verify. - Warn against pointing the return route at both peers, as two static routes or an equal-cost pair. It is the intuitive fix and it fails silently. - Note that the second interruption on recovery cannot be avoided by pinning the virtual IP to the standby, since the client's switch back is automatic. Pinning the route while the client moves back strands the return path. - The return route can live on the destination subnet's default gateway even when the gateway and the routing peer share that subnet, which is the only option when the destination host is not the customer's to configure. The page previously offered the gateway only for a different-subnet destination, and its persistent-config steps assume host access throughout. --- .../networks/how-routing-peers-work.mdx | 8 +++--- src/pages/manage/networks/masquerade.mdx | 25 +++++++++++++++++-- 2 files changed, 28 insertions(+), 5 deletions(-) diff --git a/src/pages/manage/networks/how-routing-peers-work.mdx b/src/pages/manage/networks/how-routing-peers-work.mdx index 21ca487d1..7c1ef5b11 100644 --- a/src/pages/manage/networks/how-routing-peers-work.mdx +++ b/src/pages/manage/networks/how-routing-peers-work.mdx @@ -102,9 +102,9 @@ Multiple routing peers can serve the same network or route. Behavior depends on ### Primary / failover (different metrics) -The lower-metric peer carries all traffic. The higher-metric peer is held in reserve and only takes over when the primary becomes unreachable. Failover is automatic and immediate — clients begin sending traffic through the standby as soon as the primary stops responding. When the primary comes back online, clients switch back to it immediately. Established TCP connections through the previous peer reset and applications must reconnect. +The lower-metric peer carries all traffic. The higher-metric peer is held in reserve and only takes over when the primary becomes unreachable. Failover is automatic: clients start sending traffic through the standby once the primary is seen as unreachable, normally within seconds. When the primary comes back online, clients switch back to it. Established TCP connections through the previous peer reset and applications must reconnect. -**Example.** Routing Peer A has a lower metric than Routing Peer B. When Peer A goes down, all traffic fails over to Peer B. When Peer A comes back online, all traffic switches back to Peer A immediately. +**Example.** Routing Peer A has a lower metric than Routing Peer B. When Peer A goes down, all traffic fails over to Peer B. When Peer A comes back online, all traffic switches back to Peer A. ### Latency switching (equal metrics) @@ -131,8 +131,10 @@ Turn masquerade off when: With masquerade off, you must add a return route on the destination network pointing the NetBird CIDR (default `100.64.0.0/10`) at the routing peer. +High availability then needs one more thing from you. Clients still move to the standby peer on their own, but that return route keeps pointing at the peer that just failed, so replies are dropped even though the outbound direction has recovered. The destination network has to be able to follow the failover too. See [High availability with masquerade off](/manage/networks/masquerade#high-availability-with-masquerade-off). + -Masquerade can only be turned off on Linux routing peers. High availability also stops working with masquerade off, because return traffic must flow back through one specific routing peer's LAN address — the destination network has no way to follow a failover. +Masquerade can only be turned off on Linux routing peers. ## Access control behavior diff --git a/src/pages/manage/networks/masquerade.mdx b/src/pages/manage/networks/masquerade.mdx index 54f94002c..a48ab0817 100644 --- a/src/pages/manage/networks/masquerade.mdx +++ b/src/pages/manage/networks/masquerade.mdx @@ -10,8 +10,8 @@ Masquerade is on by default. The routing peer SNATs forwarded traffic to its own ## What changes when masquerade is off - The original NetBird overlay IP is preserved end-to-end. -- The destination host (or, if it sits in a different subnet, its gateway) must have a return route for the NetBird CIDR (default `100.64.0.0/10`) pointing at the routing peer's LAN IP. -- High availability stops working — return traffic must flow back through one specific routing peer, so the destination network has no way to follow a failover. See [How Routing Peers Work — Masquerade](/manage/networks/how-routing-peers-work#masquerade). +- A return route for the NetBird CIDR (default `100.64.0.0/10`) must point at the routing peer's LAN IP. Put it on the destination host, or on that subnet's default gateway when the host is not yours to change. +- High availability takes extra work, because the return route has to follow a failover. See [High availability with masquerade off](#high-availability-with-masquerade-off). Masquerade can only be turned off on Linux routing peers. @@ -32,6 +32,10 @@ curl -X PUT https://api.netbird.io/api/networks//routers/ The rest of this page covers the return-route prerequisite the destination network needs once masquerade is off. + +The steps below add the route on the destination host itself. If that host is not yours to configure, put the same route on the subnet's default gateway instead. This works even when the gateway and the routing peer sit on the same subnet: replies leave the host, reach the gateway, and are sent back out of the interface they arrived on. + + ## Inputs to substitute The examples below use placeholders. Swap in: @@ -164,6 +168,23 @@ sudo tcpdump -ni "src net 100.64.0.0/10 and port " The source IP should fall inside your account's `/16` (e.g. `100.121.x.x`), not the routing peer's LAN IP. +## High availability with masquerade off + +The return route points at one peer, so it has to follow whichever routing peer is currently active. NetBird moves clients to the standby on its own, but it cannot change a route inside your network: left alone, the route keeps pointing at the peer that failed and replies are dropped. + +Two steps: + +1. **Give the routing peers different metrics**, so the active peer is the same one for every client. With equal metrics each client picks its own by latency, and then no single next hop is correct for all of them. See [High availability](/manage/networks/how-routing-peers-work#high-availability). +2. **Make the route's next hop follow the active peer.** Give the peers a shared virtual IP with VRRP (`keepalived` on Linux, which needs them on the same subnet) and point the route at it, or have your router move the route on a failed health check. Tie that health check to the tunnel, not to the host answering pings. + + +Do not point the return route at both peers at once, as two static routes or an equal-cost pair. Replies reaching the peer a client is not using are silently discarded, so both ends look healthy while traffic fails. + + +Established connections reset on failover, as they do with masquerade on. Recovery causes a second brief interruption, because clients return to the primary faster than the route does, and you cannot avoid it by pinning the virtual IP to the standby: the client's switch back is automatic. + +If you do not need source IP visibility, leaving masquerade on gives you high availability with none of this to operate, which is why it is the default. + ## Security considerations From 51fb1ad41cdd1f0f399a63830a472ce671a2d54b Mon Sep 17 00:00:00 2001 From: Jack Carter <128555021+SunsetDrifter@users.noreply.github.com> Date: Wed, 29 Jul 2026 14:21:31 +0200 Subject: [PATCH 2/4] docs: health check must cross the path, not test the peer A routing peer can stay enrolled and report healthy, with its WireGuard interface present, while being unable to forward from the tunnel to the destination subnet: a failed LAN interface, or a firewall change. Nothing fails over in that state. NetBird sees a reachable peer, and a health check that only looks for the interface sees a healthy one, so the return route stays pointed at a peer that forwards nothing until someone intervenes. Say that the check has to reach a destination through the tunnel. --- src/pages/manage/networks/masquerade.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/manage/networks/masquerade.mdx b/src/pages/manage/networks/masquerade.mdx index a48ab0817..7f1574925 100644 --- a/src/pages/manage/networks/masquerade.mdx +++ b/src/pages/manage/networks/masquerade.mdx @@ -175,7 +175,7 @@ The return route points at one peer, so it has to follow whichever routing peer Two steps: 1. **Give the routing peers different metrics**, so the active peer is the same one for every client. With equal metrics each client picks its own by latency, and then no single next hop is correct for all of them. See [High availability](/manage/networks/how-routing-peers-work#high-availability). -2. **Make the route's next hop follow the active peer.** Give the peers a shared virtual IP with VRRP (`keepalived` on Linux, which needs them on the same subnet) and point the route at it, or have your router move the route on a failed health check. Tie that health check to the tunnel, not to the host answering pings. +2. **Make the route's next hop follow the active peer.** Give the peers a shared virtual IP with VRRP (`keepalived` on Linux, which needs them on the same subnet) and point the route at it, or have your router move the route on a failed health check. Make that check reach a destination through the tunnel rather than testing the peer itself: a routing peer whose agent is running and whose WireGuard interface is present can still be unable to forward, and neither NetBird nor an interface check notices. Do not point the return route at both peers at once, as two static routes or an equal-cost pair. Replies reaching the peer a client is not using are silently discarded, so both ends look healthy while traffic fails. From c1308513c5be232a0096db88511ae3f8e2302242 Mon Sep 17 00:00:00 2001 From: Jack Carter <128555021+SunsetDrifter@users.noreply.github.com> Date: Thu, 30 Jul 2026 09:54:32 +0200 Subject: [PATCH 3/4] docs: held connections stall across a failover, they do not reset MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured on a routing-peer pair with masquerade off: a held TCP connection survives the switch. Four failovers, graceful shutdown and hard kill, all showed the same signature — one round trip with multi-second latency, then the same connection continuing with monotonic sequence numbers. No resets, and no RST on the wire in any of them. The reason is a property of masquerade being off: the addresses on the connection do not change when another peer takes over, so the flow is portable between peers, and the routed ACL accepts on addresses rather than on connection state, so the standby forwards mid-stream packets it never saw a handshake for. Note this is specific to masquerade being off. The general high-availability section still says established connections reset, which is the masquerade-on default, where the standby would translate to a different source address. Also trims the mechanism guidance to name the pattern rather than lean on any one implementation, since which mechanism is right may change. --- src/pages/manage/networks/masquerade.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/pages/manage/networks/masquerade.mdx b/src/pages/manage/networks/masquerade.mdx index 7f1574925..79361f66c 100644 --- a/src/pages/manage/networks/masquerade.mdx +++ b/src/pages/manage/networks/masquerade.mdx @@ -175,13 +175,13 @@ The return route points at one peer, so it has to follow whichever routing peer Two steps: 1. **Give the routing peers different metrics**, so the active peer is the same one for every client. With equal metrics each client picks its own by latency, and then no single next hop is correct for all of them. See [High availability](/manage/networks/how-routing-peers-work#high-availability). -2. **Make the route's next hop follow the active peer.** Give the peers a shared virtual IP with VRRP (`keepalived` on Linux, which needs them on the same subnet) and point the route at it, or have your router move the route on a failed health check. Make that check reach a destination through the tunnel rather than testing the peer itself: a routing peer whose agent is running and whose WireGuard interface is present can still be unable to forward, and neither NetBird nor an interface check notices. +2. **Make the route's next hop follow the active peer.** Either a shared virtual IP across the peers (VRRP, using `keepalived` on Linux, which needs them on the same subnet), or a router that moves the route when a health check fails. The mechanism lives outside NetBird. Whichever you choose, make its health check reach a destination through the tunnel rather than testing the peer itself: a routing peer whose agent is running and whose WireGuard interface is present can still be unable to forward, and neither NetBird nor an interface check notices. Do not point the return route at both peers at once, as two static routes or an equal-cost pair. Replies reaching the peer a client is not using are silently discarded, so both ends look healthy while traffic fails. -Established connections reset on failover, as they do with masquerade on. Recovery causes a second brief interruption, because clients return to the primary faster than the route does, and you cannot avoid it by pinning the virtual IP to the standby: the client's switch back is automatic. +Established TCP connections are not broken by the switch. With masquerade off, the addresses on a connection do not change when a different peer takes over, so a held connection stalls for the length of the failover and then continues, though applications with short timeouts may still give up on their own. New connections fail until both sides have converged. Recovery then costs a second brief interruption, because clients return to the primary faster than the route does, and you cannot avoid that by pinning the virtual IP to the standby: the client's switch back is automatic. If you do not need source IP visibility, leaving masquerade on gives you high availability with none of this to operate, which is why it is the default. From b2ca60db32f7000e07ae8304bf689aa8929e4ae6 Mon Sep 17 00:00:00 2001 From: Jack Carter <128555021+SunsetDrifter@users.noreply.github.com> Date: Thu, 30 Jul 2026 10:26:01 +0200 Subject: [PATCH 4/4] docs: scope the reset claim to masquerade on, fix leftover contradiction Review of the branch turned up three problems. The Networks overview still said disabling masquerade 'breaks high availability', one click from the page arguing it does not. Reworded to say it makes high availability something you arrange rather than something you get. Neither reset/stall sentence named its masquerade scope, so a reader with masquerade off met both and they flatly disagreed. The general failover section now scopes its reset claim to masquerade on, names the reason (the standby translates to a different source address), and links onward. 'Stalls for the length of the failover' understated it. Measured stalls were 6 to 13 seconds against a 6 second failover, because the sender waits for its next retransmit after the path returns. Says so, with the range, and keeps the claim hedged to what was observed rather than asserting connections are never broken. --- src/pages/manage/networks/how-routing-peers-work.mdx | 2 +- src/pages/manage/networks/index.mdx | 2 +- src/pages/manage/networks/masquerade.mdx | 4 +++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/src/pages/manage/networks/how-routing-peers-work.mdx b/src/pages/manage/networks/how-routing-peers-work.mdx index 7c1ef5b11..4b48c3e16 100644 --- a/src/pages/manage/networks/how-routing-peers-work.mdx +++ b/src/pages/manage/networks/how-routing-peers-work.mdx @@ -102,7 +102,7 @@ Multiple routing peers can serve the same network or route. Behavior depends on ### Primary / failover (different metrics) -The lower-metric peer carries all traffic. The higher-metric peer is held in reserve and only takes over when the primary becomes unreachable. Failover is automatic: clients start sending traffic through the standby once the primary is seen as unreachable, normally within seconds. When the primary comes back online, clients switch back to it. Established TCP connections through the previous peer reset and applications must reconnect. +The lower-metric peer carries all traffic. The higher-metric peer is held in reserve and only takes over when the primary becomes unreachable. Failover is automatic: clients start sending traffic through the standby once the primary is seen as unreachable, normally within seconds. When the primary comes back online, clients switch back to it. With masquerade on, the default, established TCP connections through the previous peer reset and applications must reconnect, because the standby translates them to a different source address. With masquerade off they are not translated, and behave differently: see [High availability with masquerade off](/manage/networks/masquerade#high-availability-with-masquerade-off). **Example.** Routing Peer A has a lower metric than Routing Peer B. When Peer A goes down, all traffic fails over to Peer B. When Peer A comes back online, all traffic switches back to Peer A. diff --git a/src/pages/manage/networks/index.mdx b/src/pages/manage/networks/index.mdx index 58e68e71f..eef84250e 100644 --- a/src/pages/manage/networks/index.mdx +++ b/src/pages/manage/networks/index.mdx @@ -167,7 +167,7 @@ Before you depend on a Network in production, work through these: - **High availability.** Add more than one routing peer to the same Network for redundancy. Added individually, each peer gets its own metric: a lower metric is the primary and a higher one the failover, while equal metrics balance traffic by latency. Added as a group, the peers share one metric, so they balance by latency only and can't act as primary and failover. Keep highly available peers in different failure domains. See [High availability](/manage/networks/how-routing-peers-work#high-availability). - **Monitoring.** Enable the **Routing Peer Disconnected** event in [Notifications](/manage/settings/notifications) to get alerted by email, webhook, or Slack when a routing peer goes offline. -- **Masquerade.** On by default and the simplest option. Turn it off only when you need source-IP visibility, and only on Linux routing peers, as that's the only platform where it can be disabled. Note that disabling it breaks high availability and requires a return route. See [Masquerade](/manage/networks/masquerade). +- **Masquerade.** On by default and the simplest option. Turn it off only when you need source-IP visibility, and only on Linux routing peers, as that's the only platform where it can be disabled. Disabling it requires a return route in the destination network, and makes high availability something you have to arrange rather than something you get. See [Masquerade](/manage/networks/masquerade). - **Internal DNS.** Domain resources resolve on the routing peer, so it must be able to resolve the name. If it already can, nothing more is needed; if it can't, distribute a nameserver to the routing peer's group. See [Internal DNS Servers](/manage/dns/internal-dns-servers). - **Access to the routing peer itself.** If users also need services on the routing peer host (SSH, a dashboard), add a separate peer-to-peer [access policy](/manage/access-control) for the input chain, as explained above. diff --git a/src/pages/manage/networks/masquerade.mdx b/src/pages/manage/networks/masquerade.mdx index 79361f66c..27bf653a7 100644 --- a/src/pages/manage/networks/masquerade.mdx +++ b/src/pages/manage/networks/masquerade.mdx @@ -181,7 +181,9 @@ Two steps: Do not point the return route at both peers at once, as two static routes or an equal-cost pair. Replies reaching the peer a client is not using are silently discarded, so both ends look healthy while traffic fails. -Established TCP connections are not broken by the switch. With masquerade off, the addresses on a connection do not change when a different peer takes over, so a held connection stalls for the length of the failover and then continues, though applications with short timeouts may still give up on their own. New connections fail until both sides have converged. Recovery then costs a second brief interruption, because clients return to the primary faster than the route does, and you cannot avoid that by pinning the virtual IP to the standby: the client's switch back is automatic. +Established TCP connections behave better here than with masquerade on. Because nothing translates them, their addresses do not change when a different peer takes over, so a held connection can stall through the failover and then continue rather than resetting. Expect the stall to outlast the failover itself, since the sender waits for its next retransmit before trying again: in our testing a 6 second failover stalled held connections for 6 to 13 seconds. Applications with short timeouts will still give up on their own. + +New connections fail until both sides have converged. Recovery then costs a second brief interruption, because clients return to the primary faster than the route does, and you cannot avoid that by pinning the virtual IP to the standby: the client's switch back is automatic. If you do not need source IP visibility, leaving masquerade on gives you high availability with none of this to operate, which is why it is the default.