MTU/MSS troubleshooting

Recently I noticed that my Windows 10 machines weren’t receiving updates. When I connected to VPN, or a different wi-fi network, the updates began to work. Eventually I realized that the updates failed only when IPv6 was enabled.

Since I did a bit of work to track it down, I decided to document what I did.

Table of contents:

Before beginning anything else, it might be useful to describe my network setup. Unfortunately my ISP doesn’t (yet) offer native IPv6 support. I use the Hurricane Electric Free IPv6 Tunnel Broker service.

Here is a rough diagram of how things look. (I’m aware that I’ve mixed-and-matched protocols somewhat; it’s meant to be a logical diagram.)

  +--------------+
  |   Internet   |
  +--------------+
         ||
        IPv6
         ||
+--------------------+
| Hurricane Electric |
+--------------------+
         ||
     GIF tunnel <--------- MTU: 1444
         ||
   +------------+
   |    ISP     |
   +------------+
         ||
       PPPoE <------------ MTU: 1492/1460*
         ||
  +--------------+
  |  My network  |
  +--------------+
         ||
      Ethernet <---------- MTU: 1500
         ||
  +--------------+
  |  Windows 10  |
  +--------------+

* The DSL router is set to 1492, but reports 1460,
  presumably configured at the remote (ISP) side.

I’ve specifically noted the MTUs of each link because this information will be relevant later.

Windows Update reported “Pending download” for all of the available updates:

And it would stay in that state indefinitely.

The first thing I did was crack open Wireshark. The symptoms I saw made me think of MTU/MSS-related issues right away:

• A connection is established…
• …the request is made…
• …but the response isn’t received.
• A short while later, the connection is terminated.

Back in the early 2000s, my ISP introduced DSL service, which used PPPoE for authentication; the experience of troubleshooting that – back when DSL was ‘new’ and the Internet then was not what it is today in 2018 – is still seared into my memory.

To be clear, my MTU isn’t at fault per se; IPv6 is supposed to be able to handle MTUs less than the ‘usual’ 1500 bytes via path MTU discovery (PMTUD). Just like ~18 years ago, this is a reminder that not everyone follows standards on the Internet.

Let‘s see what Windows Update has to say about this. I went to check out C:\Windows\WindowsUpdate.log, only to find it had been replaced by something in PowerShell. I ran Get-WindowsUpdateLog, then checked the resulting log file:

2018/11/10 16:25:42.9831383 6396  12664 SLS             Retrieving SLS response from server using ETAG OkOyh3sm/2urccZ0ssmcTjpc8XNaVqdEacCoJa+JHy0=_1440"..."
2018/11/10 16:25:42.9834774 6396  12664 SLS             Making request with URL HTTPS://sls.update.microsoft.com/SLS/{9482F4B4-E343-43B6-B170-9A65BC822C77}/x64/10.0.17134.254/0?CH=933&L=en-US&P=&PT=0x30&WUA=10.0.17134.254&MK=LENOVO&MD=3444B9U
2018/11/10 16:35:27.3583349 10288 9580  ComApi          * START *   EvaluateHardwareCapabilities
2018/11/10 16:35:29.3199214 10288 9580  ComApi          * END *   EvaluateHardwareCapabilities 00000000
2018/11/10 16:35:48.9759932 6396  12664 Misc            *FAILED* [80072EE2] Send request
2018/11/10 16:35:48.9760040 6396  12664 Misc            *FAILED* [80072EE2] WinHttp: SendRequestToServerForFileInformation (retrying with default proxy)
2018/11/10 16:37:48.9773975 6396  12664 Misc            *FAILED* [80072EE2] Send request
2018/11/10 16:37:48.9774404 6396  12664 Misc            *FAILED* [80072EE2] Library download error. Will retry. Retry Counter:0
[...snip...]

This is convenient, because it gives the URL that’s failing.

Trying that URL found in WindowsUpdate.log on my IPv6-enabled Linux box:

fission@nucleus[pts/11]:~% curl --insecure -6v 'https://sls.update.microsoft.com/SLS/{9482F4B4-E343-43B6-B170-9A65BC822C77}/x64/10.0.17134.254/0?CH=933&L=en-US&P=&PT=0x30&WUA=10.0.17134.254&MK=LENOVO&MD=3444B9U'
*   Trying 2a01:111:f307:1790::f001:7a5...
* Connected to sls.update.microsoft.com (2a01:111:f307:1790::f001:7a5) port 443 (#0)
* found 151 certificates in /etc/ssl/certs/ca-certificates.crt
* found 648 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* gnutls_handshake() failed: Error in the pull function.
* Closing connection 0
curl: (35) gnutls_handshake() failed: Error in the pull function.

This is the same behaviour seen on the Windows 10 machine: a connect, but no data returned.

Works in IPv4, though:

fission@nucleus[pts/11]:~% curl --insecure -4 -o /dev/null 'https://sls.update.microsoft.com/SLS/{9482F4B4-E343-43B6-B170-9A65BC822C77}/x64/10.0.17134.254/0?CH=933&L=en-US&P=&PT=0x30&WUA=10.0.17134.254&MK=LENOVO&MD=3444B9U'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 22167  100 22167    0     0  59906      0 --:--:-- --:--:-- --:--:-- 60073

Note the number of bytes returned (~21 K) – definitely more than would fit in one TCP packet.

What does this look like in Wireshark?

Packet #9 is interesting; it says “[TCP Previous segment not captured]”. I actually missed this clue the first time through.

Probably there was a bigger packet sent before that one, I suppose part of the TLS “Server Hello” that should be happening (but isn’t). That’s why Wireshark is reporting a missing segment – that packet was likely too big for the tunnel MTU.

Since the remote end never receives the ICMPv6 “packet too big” message, nor an ACK for the first packet it sent, it eventually gives up waiting and resets the connection.

Note the MSS from the remote side: 1400. This is 40 bytes lower than the local side; already this makes me a bit suspicious.

I took a lot for granted about my Internet connections, since I hadn’t experienced problems before. Most of this section is just verifying that the things I believe to be true.

For the life of me, I can’t remember why I set the HE GIF tunnel MTU to 1444 bytes. In any event, let’s see if the tunnel really is capable of sending a 1444-byte IPv6 packet.

ICMP ‘ping’ is probably the easiest way I can think of to check this. The data length chosen should be:

data length = MTU − ICMP packet length − IP packet length

In my case, that’s:

data length = 1444 − 8 − 40 = 1396

Back to the Linux server to test this. The -M do part sets the DF (Don’t Fragment) bit.

fission@nucleus[pts/11]:~% ping6 -M do -c 1 -s 1396 mirrors.pdx.kernel.org
PING mirrors.pdx.kernel.org(mirrors.pdx.kernel.org) 1396 data bytes
1404 bytes from mirrors.pdx.kernel.org: icmp_seq=1 ttl=57 time=25.3 ms
[...]

Success! If I try one more byte, it should fail:

fission@nucleus[pts/11]:~% ping6 -M do -c 1 -s 1397 mirrors.pdx.kernel.org
PING mirrors.pdx.kernel.org(mirrors.pdx.kernel.org) 1397 data bytes
From mirrors.pdx.kernel.org icmp_seq=1 Packet too big: mtu=1444
[...]

Perfect! It even tells me the MTU right there.

This also works the other way around:

[ec2-user@ip-10-0-0-43 ~]$ ping6 -M do -c 1 -s 1396 nucleus.ldx.ca
PING nucleus.ldx.ca(nucleus.ldx.ca (2001:470:eb96:2::2)) 1396 data bytes
1404 bytes from nucleus.ldx.ca (2001:470:eb96:2::2): icmp_seq=1 ttl=35 time=189 ms
[...]
[ec2-user@ip-10-0-0-43 ~]$ ping6 -M do -c 1 -s 1397 nucleus.ldx.ca
PING nucleus.ldx.ca(nucleus.ldx.ca (2001:470:eb96:2::2)) 1397 data bytes
From tserv1.sea1.he.net (2001:470:0:9b::2) icmp_seq=1 Packet too big: mtu=1444
[...]

Notice that the ICMPv6 “packet too big” is coming from the tunnel endpoint.

Let’s see if anyone is monkeying around with the MSS value in IPv6 TCP connections. To do this, I spun up an IPv6-enabled host in AWS for the remote side, so I could packet capture on both ends.

The command I ran on the client (local) side:

fission@nucleus[pts/11]:~% curl -I 'http://[2a05:d018:859:9f00:fb11:b53b:d674:2439]'
HTTP/1.1 403 Forbidden
Date: Wed, 14 Nov 2018 07:56:42 GMT
Server: Apache/2.4.34 ()
[...]

The packet capture from the local (client) side:

23:56:42.524388 IP6 2001:470:eb96:2::2.35221 > 2a05:d018:859:9f00:fb11:b53b:d674:2439.80: Flags [S], seq 895111178, win 28800, options [mss 1440,sackOK,TS val 359807423 ecr 0,nop,wscale 7], length 0
23:56:42.707159 IP6 2a05:d018:859:9f00:fb11:b53b:d674:2439.80 > 2001:470:eb96:2::2.35221: Flags [S.], seq 1704057741, ack 895111179, win 26787, options [mss 1440,sackOK,TS val 3793742308 ecr 359807423,nop,wscale 7], length 0

And from the remote (AWS) side:

07:56:42.641083 IP6 2001:470:eb96:2::2.35221 > 2a05:d018:859:9f00:fb11:b53b:d674:2439.80: Flags [S], seq 895111178, win 28800, options [mss 1440,sackOK,TS val 359807423 ecr 0,nop,wscale 7], length 0
07:56:42.641112 IP6 2a05:d018:859:9f00:fb11:b53b:d674:2439.80 > 2001:470:eb96:2::2.35221: Flags [S.], seq 1704057741, ack 895111179, win 26787, options [mss 8941,sackOK,TS val 3793742308 ecr 359807423,nop,wscale 7], length 0

Holy jumbo frames, Batman! So, yes, there is some MSS clamping going on somewhere, but it’s clamped only to the ‘standard’ value of 1440 (1500 MTU − 40-byte IPv6 header − 20-byte TCP header). This is obviously not low enough to go through my IPv6 tunnel with an MTU of 1444.

Given my configuration of an IPv6 MTU of 1444, if I cause a remote server to send more than about 1.4 K, it will need to do PMTUD, since my MSS isn’t ‘correct’ for the tunnel. Fetching the default Apache index page should do the trick:

fission@nucleus[pts/11]:~% curl -o /dev/null 'http://[2a05:d018:859:9f00:fb11:b53b:d674:2439]'                      
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3630  100  3630    0     0   6717      0 --:--:-- --:--:-- --:--:--  6709

The local (client) packet capture (important bits highlighted; IP addresses replaced for readability):

00:13:20.383768 IP6 client.37783 > awshost.80: Flags [S], seq 3916962842, win 28800, options [mss 1440,sackOK,TS val 360056888 ecr 0,nop,wscale 7], length 0
00:13:20.569823 IP6 awshost.80 > client.37783: Flags [S.], seq 3533963009, ack 3916962843, win 26787, options [mss 1440,sackOK,TS val 3794740167 ecr 360056888,nop,wscale 7], length 0
00:13:20.569881 IP6 client.37783 > awshost.80: Flags [.], ack 1, win 225, options [nop,nop,TS val 360056935 ecr 3794740167], length 0
00:13:20.569960 IP6 client.37783 > awshost.80: Flags [P.], seq 1:105, ack 1, win 225, options [nop,nop,TS val 360056935 ecr 3794740167], length 104: HTTP: GET / HTTP/1.1
00:13:20.755458 IP6 awshost.80 > client.37783: Flags [.], ack 105, win 210, options [nop,nop,TS val 3794740353 ecr 360056935], length 0
00:13:20.761947 IP6 awshost.80 > client.37783: Flags [P.], seq 2857:3915, ack 105, win 210, options [nop,nop,TS val 3794740354 ecr 360056935], length 1058: HTTP
00:13:20.761957 IP6 client.37783 > awshost.80: Flags [.], ack 1, win 242, options [nop,nop,TS val 360056983 ecr 3794740353,nop,nop,sack 1 {2857:3915}], length 0
00:13:20.923726 IP6 awshost.80 > client.37783: Flags [P.], seq 2745:3915, ack 105, win 210, options [nop,nop,TS val 3794740354 ecr 360056935], length 1170: HTTP
00:13:20.923739 IP6 client.37783 > awshost.80: Flags [.], ack 1, win 264, options [nop,nop,TS val 360057023 ecr 3794740353,nop,nop,sack 2 {2857:3915}{2745:3915}], length 0
00:13:20.923746 IP6 awshost.80 > client.37783: Flags [.], seq 1:1373, ack 105, win 210, options [nop,nop,TS val 3794740354 ecr 360056935], length 1372: HTTP: HTTP/1.1 403 Forbidden
00:13:20.923755 IP6 client.37783 > awshost.80: Flags [.], ack 1373, win 287, options [nop,nop,TS val 360057023 ecr 3794740354,nop,nop,sack 1 {2745:3915}], length 0
00:13:20.923889 IP6 awshost.80 > client.37783: Flags [.], seq 1373:2745, ack 105, win 210, options [nop,nop,TS val 3794740354 ecr 360056935], length 1372: HTTP
00:13:20.923894 IP6 client.37783 > awshost.80: Flags [.], ack 3915, win 309, options [nop,nop,TS val 360057023 ecr 3794740354], length 0
00:13:20.924126 IP6 client.37783 > awshost.80: Flags [F.], seq 105, ack 3915, win 309, options [nop,nop,TS val 360057023 ecr 3794740354], length 0
00:13:21.109162 IP6 awshost.80 > client.37783: Flags [F.], seq 3915, ack 106, win 210, options [nop,nop,TS val 3794740707 ecr 360057023], length 0
00:13:21.109185 IP6 client.37783 > awshost.80: Flags [.], ack 3916, win 309, options [nop,nop,TS val 360057069 ecr 3794740707], length 0

The remote (server) packet capture (important bits highlighted; IP addresses replaced for readability):

08:13:20.504327 IP6 client.37783 > awshost.http: Flags [S], seq 3916962842, win 28800, options [mss 1440,sackOK,TS val 360056888 ecr 0,nop,wscale 7], length 0
08:13:20.504364 IP6 awshost.http > client.37783: Flags [S.], seq 3533963009, ack 3916962843, win 26787, options [mss 8941,sackOK,TS val 3794740167 ecr 360056888,nop,wscale 7], length 0
08:13:20.690332 IP6 client.37783 > awshost.http: Flags [.], ack 1, win 225, options [nop,nop,TS val 360056935 ecr 3794740167], length 0
08:13:20.690342 IP6 client.37783 > awshost.http: Flags [P.], seq 1:105, ack 1, win 225, options [nop,nop,TS val 360056935 ecr 3794740167], length 104: HTTP: GET / HTTP/1.1
08:13:20.690362 IP6 awshost.http > client.37783: Flags [.], ack 105, win 210, options [nop,nop,TS val 3794740353 ecr 360056935], length 0
08:13:20.690654 IP6 awshost.http > client.37783: Flags [.], seq 1:1429, ack 105, win 210, options [nop,nop,TS val 3794740354 ecr 360056935], length 1428: HTTP: HTTP/1.1 403 Forbidden
08:13:20.690662 IP6 awshost.http > client.37783: Flags [.], seq 1429:2857, ack 105, win 210, options [nop,nop,TS val 3794740354 ecr 360056935], length 1428: HTTP
08:13:20.690667 IP6 awshost.http > client.37783: Flags [P.], seq 2857:3915, ack 105, win 210, options [nop,nop,TS val 3794740354 ecr 360056935], length 1058: HTTP
08:13:20.857632 IP6 2001:470:0:9b::2 > awshost: ICMP6, packet too big, mtu 1444, length 1240
08:13:20.857662 IP6 awshost.http > client.37783: Flags [.], seq 1:1373, ack 105, win 210, options [nop,nop,TS val 3794740354 ecr 360056935], length 1372: HTTP: HTTP/1.1 403 Forbidden
08:13:20.857664 IP6 awshost.http > client.37783: Flags [.], seq 1373:2745, ack 105, win 210, options [nop,nop,TS val 3794740354 ecr 360056935], length 1372: HTTP
08:13:20.857669 IP6 awshost.http > client.37783: Flags [P.], seq 2745:3915, ack 105, win 210, options [nop,nop,TS val 3794740354 ecr 360056935], length 1170: HTTP
08:13:20.860396 IP6 2001:470:0:9b::2 > awshost: ICMP6, packet too big, mtu 1444, length 1240
08:13:20.882134 IP6 client.37783 > awshost.http: Flags [.], ack 1, win 242, options [nop,nop,TS val 360056983 ecr 3794740353,nop,nop,sack 1 {2857:3915}], length 0
08:13:21.044163 IP6 client.37783 > awshost.http: Flags [.], ack 1, win 264, options [nop,nop,TS val 360057023 ecr 3794740353,nop,nop,sack 2 {2857:3915}{2745:3915}], length 0
08:13:21.044175 IP6 client.37783 > awshost.http: Flags [.], ack 1373, win 287, options [nop,nop,TS val 360057023 ecr 3794740354,nop,nop,sack 1 {2745:3915}], length 0
08:13:21.044179 IP6 client.37783 > awshost.http: Flags [.], ack 3915, win 309, options [nop,nop,TS val 360057023 ecr 3794740354], length 0
08:13:21.044182 IP6 client.37783 > awshost.http: Flags [F.], seq 105, ack 3915, win 309, options [nop,nop,TS val 360057023 ecr 3794740354], length 0
08:13:21.044239 IP6 awshost.http > client.37783: Flags [F.], seq 3915, ack 106, win 210, options [nop,nop,TS val 3794740707 ecr 360057023], length 0
08:13:21.229463 IP6 client.37783 > awshost.http: Flags [.], ack 3916, win 309, options [nop,nop,TS val 360057069 ecr 3794740707], length 0

Here one can see PMTUD in action:

1. The web server sent three packets, two of length 1428, and one of 1058; only the packet of length 1058 made it through.
2. In response to the oversized packets, 2001:470:0:9b::2 (tserv1.sea1.he.net, the tunnel endpoint) interjected with an ICMPv6 packet, giving the client’s MTU to the web server. The client never actually saw this, or knew it was happening.
3. The web server then sent three new packets, two each of length 1372, and one of 1170 – now fitting into the client’s MTU.

This works in reverse, too; I won’t show the full capture here again, but the important bit is:

01:00:36.856242 IP6 2001:470:eb96:2::1 > 2001:470:eb96:2::2: ICMP6, packet too big, mtu 1444, length 1240

Here my local router (the local end of the tunnel) is advising my local web server to send less data. (Come to think of it, this would happen for practically every IPv6 connection where any significant amount of data is sent…)

One note if you’re running tests like these: the MTU value for a remote system can be cached. So it’s worth checking that using ip route get to address and letting the cache expire before re-running a test:

[ec2-user@ip-10-0-0-43 ~]$ ip route get to 2001:470:eb96:2::2
2001:470:eb96:2::2 from :: via fe80::f3:28ff:fe6c:f9ac dev eth0 src 2a05:d018:859:9f00:fb11:b53b:d674:2439 metric 0 
    cache expires 165sec mtu 1444 hoplimit 64 pref medium

Clearly, the server sls.update.microsoft.com [2a01:111:f307:1790::f001:7a5] isn’t being a nice Internet citizen:

[ec2-user@ip-10-0-0-43 ~]$ ping6 -M do -c 1 -s 1300 2a01:111:f307:1790::f001:7a5
PING 2a01:111:f307:1790::f001:7a5(2a01:111:f307:1790::f001:7a5) 1300 data bytes

--- 2a01:111:f307:1790::f001:7a5 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

If ICMPv6 inbound is blocked, probably ICMPv6 outbound is being blocked, too. Thus the “packet too big” messages are likely being filtered, which breaks PMTUD.

RFC 1981 says:

IPv6 nodes SHOULD implement Path MTU Discovery in order to discover and take advantage of paths with PMTU greater than the IPv6 minimum link MTU [IPv6-SPEC] … Nodes not implementing Path MTU Discovery use the IPv6 minimum link MTU defined in [IPv6-SPEC] as the maximum packet size.

If you expect everyone on the Internet to do what they should, you’ll be waiting a long time, I fear. And there’s no magic tech support at Microsoft to assist with this.

There are a few possible solutions, listed here in order from technically most superior to worst (imo).

(Maybe you work for Microsoft and you’re reading this? Unlikely, but…)

The only true ‘solution’, in my view, is for people to configure their Internet-facing servers properly. That means Microsoft either needs to start allowing ICMPv6 traffic for PMTUD; or they need to set that server’s MTU to 1280. If you know of a way to get this message through to someone at Microsoft who (a) cares, and (b) can do something about it, please do!

Having your local router ‘fix’ the MSS by lowering it to match the tunnel MTU is probably the cleanest workaround that doesn’t involve haranguing Microsoft. This has been a classic workaround for many years, especially with PPPoE tunnels (eg, for DSL). Support for IPv6 TCP MSS clamping varies, however, so if you implement this, check that it’s actually effective.

For example, using pfSense, the dialog box asks you to enter a value from which 40 will be subtracted. So for an MTU of 1444, to get an IPv6 TCP MSS of 1384, I have to enter 1424 (to get 60 less than MTU). Evidently this feature was created only with IPv4 in mind…

In the end, this is the approach I took.

One workaround is to just block the offending IPv6 host or network at your edge firewall, thus forcing the client to fall back to IPv4. You could also block in other places – DNS, perhaps, or maybe a hosts file (eww).

Disabling IPv6 is, to me, a last resort, and more of a kludge than even a workaround.

As illustrated in the diagram above, my IPv6 data is carried over an IPv4 GIF tunnel, which itself is atop a PPPoE link. Clearly the MTU I chose for the HE IPv6 tunnel (1444) is small enough to fit within my IPv4 PPPoE tunnel, otherwise the IPv6 MTU test wouldn’t have worked. But I still have two questions:

1. What is my IPv4 PPPoE MTU, really?
2. How high could I set the IPv6 tunnel MTU?

To answer the first question, I did some ping tests with specific data lengths, but this time in IPv4:

% ping -M do -c 1 -s 1472 137.82.1.1 
PING 137.82.1.1 (137.82.1.1) 1472(1500) bytes of data.
From 192.234.197.241 icmp_seq=1 Frag needed and DF set (mtu = 1460)
[...]

Although the admin GUI of my DSL router says the PPPoE MTU is 1492, a ping test causes the DSL router to report it to be 1460. And yet, a 1464-byte packet was sent successfully via the same link (1404-byte ICMPv6 + 40-byte IPv6 + 20-byte IPv4 = 1464). Something is odd…

My first thought is that these supposedly ‘oversized’ packets were simply being fragmented. It would be hard for me to tell, since I don’t have control over either end of the PPPoE tunnel – both ends belong to my ISP.

Looking at the capture of the IPv6 ping test, Wireshark shows that the inbound traffic from the HE tunnel has the DF bit set:

Thus there shouldn’t be any fragmentation. And the total packet size, 1464 bytes, exceeds the MTU reported by the DSL router (1460).

If the MTU of the PPPoE link is really 1492 (as the router GUI says), then could the IPv6 tunnel MTU be 1472?

I set the HE tunnel MTU to be 1472, then ran several ping tests with varying packet sizes. What I found is that a 1396-byte ICMP data payload was the largest that didn’t result in IPv6 fragmentation on the reply. That is (in order):

20 byte IPv4 header + 40 byte IPv6 header + 8 byte ICMPv6 header + 1396 byte ICMP data = 1464

I’m not totally sure why I got four bytes for free, but it turns out that 1444 is, in fact, the highest MTU the tunnel will accept (without fragmentation).

In further TCP testing, the ‘true’ limit turned out to be 1440 (which is divisible by 8, fwiw), and I ended up using that for the MTU.

---

Feel free to contact me with any questions, comments, or feedback.