MTU and fragmentation over IPsec

Tunnel is up and small pings work, but large transfers, HTTPS, and SSH hang — encapsulation overhead and PMTUD blackholes, and how to fix them.

The tunnel is established, a plain ping to the far side works, small web pages load — but a large file transfer stalls at zero bytes, ssh connects and then freezes right after the banner, and HTTPS sites hang half-loaded. This is the signature of an MTU / fragmentation problem, and it is entirely separate from negotiation or routing. The tunnel is fine; the packets are simply too big to survive encapsulation.

Throughout, the VPNGeek side is 10.20.0.0/16 and the remote LAN is 192.168.1.0/24.

Why encapsulation shrinks your usable payload

Ethernet's default MTU is 1500 bytes. IPsec wraps every packet in extra headers — a new outer IP header, the ESP header and trailer, an initialization vector and integrity check, and, when NAT-Traversal is active, an additional UDP header. Depending on cipher and mode this overhead runs to roughly 60–120 bytes.

So a full-size 1500-byte inner packet becomes ~1560–1620 bytes on the wire, which exceeds the physical link's 1500-byte MTU. The router must now either fragment it or drop it — and that is where things break.

Note. Small packets fit inside the reduced budget, so pings and DNS work perfectly. It is the large, full-MTU packets — bulk TCP data, TLS handshakes with big certificates, SSH after the initial exchange — that overflow and disappear. "Small works, large hangs" is the tell-tale MTU symptom.

The PMTUD blackhole

TCP is supposed to discover the right size automatically via Path MTU Discovery. A router that cannot forward an oversized packet with the Don't-Fragment bit set returns an ICMP "fragmentation needed" (type 3, code 4) message telling the sender to use a smaller MTU. The sender shrinks its packets and everything recovers.

The problem: many firewalls and security groups filter all ICMP. When that ICMP message is dropped, the sender never learns to shrink. It keeps retransmitting the same oversized packet, which keeps being discarded — a silent blackhole. The connection establishes (small SYN packets) and then dies the instant real data flows.

Warning. Do not blanket-drop ICMP on the tunnel path. Blocking ICMP type 3 code 4 ("fragmentation needed") is what turns a benign MTU shortfall into a hard blackhole where large transfers hang forever with no error. Permit that message even if you rate-limit the rest of ICMP.

Step 1: prove it is MTU with sized pings

Send pings of increasing payload with fragmentation forbidden (-M do sets the Don't-Fragment bit). The point where they start failing is your working payload size. Remember the ping payload excludes 28 bytes of IP + ICMP header.

# DF-bit set: these are dropped rather than fragmented if too big
ping -M do -s 1472 -c 3 192.168.1.10 # 1472 + 28 = 1500, full MTU
ping -M do -s 1400 -c 3 192.168.1.10
ping -M do -s 1300 -c 3 192.168.1.10

# A "Frag needed and DF set (mtu = 1438)" reply names the exact MTU.
# Silence/100% loss on large sizes but success on small = blackhole.

# Binary-search the largest payload that still succeeds, e.g. 1372.
# Path MTU = working payload + 28 -> 1372 + 28 = 1400

If 1472 fails but 1372 succeeds, your usable path MTU through the tunnel is around 1400. That number drives the two fixes below.

Fix 1: lower the tunnel interface MTU

Set the tunnel/VTI interface MTU below the physical link minus the encapsulation overhead. 1400 is a safe, widely-used value; drop to 1360 if NAT-Traversal (the extra UDP header) is in play.

# Set the route-based tunnel interface MTU
ip link set dev vti0 mtu 1400

# Persist it in the connection (strongSwan / ipsec.conf)
# conn net-net
# ...
# mtu=1400 # if your setup uses a VTI/XFRM iface

# Verify it took
ip link show vti0 | grep -o 'mtu [0-9]*'

Lowering the interface MTU makes the kernel itself fragment or size packets correctly before encryption, so it no longer depends on a possibly-blocked ICMP message. This helps UDP and any non-TCP traffic too.

Fix 2: clamp TCP MSS (the reliable one)

The most robust fix is MSS clamping. The gateway rewrites the Maximum Segment Size in every passing TCP SYN so both endpoints agree, up front, to send segments that fit the tunnel — no ICMP and no PMTUD required. Set MSS to path MTU minus 40 (20 IP + 20 TCP), so a 1400 MTU gives MSS 1360.

# Clamp automatically to the discovered path MTU (preferred)
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
-j TCPMSS --clamp-mss-to-pmtu

# Or pin an explicit value when PMTUD can't be trusted at all
# (MSS = tunnel MTU - 40)
iptables -t mangle -A FORWARD -o vti0 -p tcp --tcp-flags SYN,RST SYN \
-j TCPMSS --set-mss 1360

# Confirm the SYN's MSS is being rewritten
tcpdump -ni vti0 'tcp[tcpflags] & tcp-syn != 0' -v | grep -i mss

Note. MSS clamping only helps TCP — it does nothing for UDP, GRE, or ICMP. For a complete fix, combine it with a lowered tunnel MTU (Fix 1) so non-TCP traffic is handled too. In practice, doing both is the belt-and-braces approach that ends MTU tickets for good.

Verify the fix

After clamping and lowering the MTU, the large transfer should complete and SSH should stop freezing. Re-run the sized ping and a real bulk transfer to confirm full-size traffic now survives the tunnel end to end.

# Full-MTU DF ping should now either succeed or return a clear
# "frag needed" (mtu=...) instead of silently timing out
ping -M do -s 1372 -c 3 192.168.1.10

# A real bulk transfer should run to completion, not stall at 0
scp bigfile.iso user@192.168.1.10:/tmp/

# Watch the tunnel iface: large inner packets should now flow
tcpdump -ni vti0 host 192.168.1.10

Summary

IPsec eats 60–120 bytes of every packet, so full-MTU traffic no longer fits the wire. When ICMP is filtered, PMTUD cannot recover and large flows blackhole while small ones work. Probe with ping -M do -s to find the real path MTU, lower the tunnel interface MTU to about 1400 (1360 with NAT-T), and clamp TCP MSS so every session negotiates a size that survives encapsulation. Apply both fixes and permit ICMP "fragmentation needed" — that combination resolves virtually every over-IPsec MTU problem.

Did this guide get you connected?

← PREVIOUS Any firewall: the universal IKEv2 parameters for a VPNGeek tunnel NEXT → Cisco IOS: site-to-site IKEv2 tunnel to a VPNGeek gateway