TCP MSS Clamping

TCP Maximum Segment Size, or MSS, specifies the maximum TCP payload an endpoint is willing to receive in a segment. Each endpoint advertises its own MSS during connection establishment. MSS differs from Maximum Transmission Unit, or MTU: an IP MTU includes IP and transport headers, whereas MSS describes TCP data only. For a 1,500-byte IP MTU, the usual advertised MSS is 1,460 bytes for IPv4 or 1,440 bytes for IPv6, assuming fixed-size headers. Senders must also account for additional IP or TCP options when constructing packets.

TCP MSS clamping reduces an advertised MSS when it exceeds a configured or calculated ceiling. A network device rewrites the MSS option in SYN packets, including SYN-ACK packets, and updates the TCP checksum. The receiving endpoint then limits the payload size it sends toward the endpoint whose advertisement was modified. Because the advertisements control opposite traffic directions, engineers must ensure that the relevant handshake packets are processed in both directions.

Cloud networks frequently connect paths with different MTUs. An instance might support jumbo frames inside a VPC, while traffic to an on-premises system crosses an IPsec tunnel with a smaller effective inner-packet MTU. Encapsulation consumes space, so the physical interface MTU alone does not establish the safe size of the original packet. If the verified inner IP MTU is 1,400 bytes, an IPv4 MSS ceiling of 1,360 bytes is a typical starting point; the corresponding IPv6 value is 1,340 bytes with fixed-size headers. Calculate the effective MTU from the actual tunnel configuration rather than assuming one value works for every VPN.

A common failure pattern is that TCP connects successfully, but a TLS handshake, file transfer, or large application response stalls. Small handshake packets fit through the path, while larger data packets are repeatedly dropped. Normally, Path MTU Discovery uses ICMP feedback to help the sender reduce packet size. For IPv4, this includes Destination Unreachable messages indicating fragmentation is needed; IPv6 uses Packet Too Big messages. Filtering this feedback can create a path MTU black hole. MSS clamping can restore TCP connectivity without depending on that feedback for normal TCP data sizing, but it does not repair the underlying ICMP filtering.

Investigate the whole path before changing settings. Compare instance, tunnel, and gateway MTUs; test progressively larger packets with appropriate fragmentation controls; and capture the TCP handshake and subsequent retransmissions. In a VPC, flow logs can help identify traffic patterns and policy rejects, but they do not expose the MSS option or packet payloads. Use host packet captures or traffic mirroring to inspect those details. Host captures can also show apparently oversized packets because segmentation offload occurs after the capture point, so interpret captures in context.

Apply clamping at a controlled forwarding point, such as a customer gateway or virtual firewall, using either a tested fixed ceiling or a path-MTU-derived setting. Automatic calculations depend on the device's routing and MTU information and can be misleading on asymmetric paths. Reestablish test connections after changing the rule because existing connections have already exchanged MSS advertisements. Confirm that large transfers succeed in both directions and that the rule does not increase a smaller MSS already advertised by an endpoint.

Clamping is not fragmentation, does not change interface MTUs, and does not affect UDP or QUIC. An unnecessarily low ceiling increases packet count and processing overhead for the same application workload. Prefer correct MTU configuration and functioning Path MTU Discovery, using MSS clamping as a targeted safeguard where TCP traffic crosses a smaller-MTU boundary.