Slow Network Speed over WAN

When copying files between two Windows 2003 SP2 Servers across our WAN, I
can't get more than about 3Mbps for a single copy operation. When running
multiple copies simultaneously, each one gets about 3Mbps up to the maximum
bandwidth of the connection (on both a 1GB and a 100MB connection).

My servers are in a data center on each coast of the US (one on the West
Coast, the other on the East Coast), with about 100ms of latency. I have
observed the performance limits with both a drag-and-drop from Windows
Explorer and from Robocopy. When using iPerf with the default settings I see
the same performance, but when running multiple simultaneous tests or
increasing the TCP Window size I can get nearly my full speed.

When I attempt to make any changes to the registry (Tcp1323Opts=3 and
GlobalMaxTcpWindowSize={various values tried} in
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Servic es\Tcpip\Parameters) I get
no performance increase.

I've also tried adding DefaultSendWindow to
[HKEY_LOCAL_MACHINE\System\CurrentControlSet\Servic es\AFD\Parameters] with no
luck.

Dear Customer,

Thank you for your post. This is Neo and I will be assisting you in this
post.

From your description, I understand that:

When files are copied through WAN connection between two Windows Server
2003 SP2 Servers, the speed of one single session is limited to 3Mbps. When
multiple sessions are running, the total speed can reach the maximum
bandwidth of the WAN connection, however, the speed of each single session
is still limited to 3Mbps. Any changes made to the registry don't help.

If there is any misunderstanding, please let me know.

This issue might be caused by WAN connection bandwidth limitation.

In order to narrow down this issue, please help confirm the following
questions with me. (Let's call the West Coast location as local site, the
other as remote site.)

1. Please describe the detailed Network Topology.

2. Is there any routing/NAT devices between your two Windows Servers? If
yes, please check whether there are any bandwidth limitation policies set
on them.

3. Please tell me the way how these two Servers communicate with each
other. (Though VPN or WAN connection directly.)

4. Please test the copy speed between two Windows Servers in the same local
LAN and check whether the issue still occur.

5. Please choose a different Windows Server located in remote side and test
the copy speed between local site's Windows Server and remote site's
Windows Server via WAN connection.

6. Please try to download some big document from the internet on local
site's Windows Server and test the download speed.

After you finish the above tests, please tell me all the detailed result.

Thank you for your time and I look forward to hearing from you.

Sincerely,
Neo Zhu,
Microsoft Online Support
Microsoft Global Technical Support Center

Get Secure! - www.microsoft.com/security
================================================== ===
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
================================================== ===
This posting is provided "AS IS" with no warranties, and confers no rights.

1. West Coast Site and Remote Site are on Gigabit Ethernet LAN with Gigabit
to the public internet and are interconnected to each other by private IP
transport (Routed Gigabit). Both are connected directly to the external
router, bypassing the Firewall (Public IP addresses entered directly on the
Windows NIC). They each also have a second interface/NIC on our private LAN
(10.x.x.x IP range).
2. Each site has a Fibre-connected gigabit router, no rate limit set.
3. The servers can communicate with each other by either their public
interface or their private interface. The private interface uses the private
IP transport (effectively a site-to-site VPN), the Public interface uses the
internet. Both have the same performance.
4. Copies from the West Coast server to another in the same site happen at
the expected speed (20%-40% utilization of the max interface speed), the same
is seen at the remote site.
5. Any server in the West Coast site has the same performance limitations
when communicating with any server in the remote site.
6. When using some of the available network speed performance test
websites, I’ve gotten speeds of:
Download Speed: 201673 kbps (25209.1 KB/sec transfer rate)
Upload Speed: 18124 kbps (2265.5 KB/sec transfer rate)
Those speeds are representative of either site (when testing against a test
site geographically close).

Additional information: When I test using iPerf, I can see 30%-40% NIC
utilization when I adjust up the TCP window size. I can also see this
performance when using multiple simultaneous copy operations (either FTP or
Robocopy).

"Jian-Ping Zhu [MSFT]" wrote:
> 1. Please describe the detailed Network Topology.
> 2. Is there any routing/NAT devices between your two Windows Servers? If
> yes, please check whether there are any bandwidth limitation policies set
> on them.
> 3. Please tell me the way how these two Servers communicate with each
> other. (Though VPN or WAN connection directly.)
> 4. Please test the copy speed between two Windows Servers in the same local
> LAN and check whether the issue still occur.
> 5. Please choose a different Windows Server located in remote side and test
> the copy speed between local site's Windows Server and remote site's
> Windows Server via WAN connection.
> 6. Please try to download some big document from the internet on local
> site's Windows Server and test the download speed.

Dear Customer,

Thank you for your reply.

I performed further analysis and I noticed:

1. If copy between two servers in the same site, the issue does not occur.

2. The issue occurs with several different computers as long as the two
related computers are in different sites.

Therefore, I think the issue is not due to a Windows based setting or
limitation. Instead, it should be very likely a network related limitation.
To efficiently resolve the issue, I strongly recommend you contact the ISP
and the router manufacturer for more investigation, to see if there are
some known issues or settings on the router or on the ISP side.

Thanks.

Sincerely,
Neo Zhu,
Microsoft Online Support
Microsoft Global Technical Support Center

Get Secure! - www.microsoft.com/security
================================================== ===
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
================================================== ===
This posting is provided "AS IS" with no warranties, and confers no rights.

Thanks for your insight.

I performed further analysis and I noticed:
1. When running iPerf (on the two Microsoft Windows servers), after
modifying the TCP Window Size option, performance tests show over 100 times
faster than standard Windows copies.

2. These tests are being performed across the WAN, so they appear to
dispute your theory that it is a router/firewall/ISP limitation.

Can you explain why the WAN is able to perform as expected, but native
Windows copies cannot?

Thanks.

Sincerely,
David.

"Jian-Ping Zhu [MSFT]" wrote:

> Dear Customer,
>
> Thank you for your reply.
>
> I performed further analysis and I noticed:
>
> 1. If copy between two servers in the same site, the issue does not occur.
>
> 2. The issue occurs with several different computers as long as the two
> related computers are in different sites.
>
> Therefore, I think the issue is not due to a Windows based setting or
> limitation. Instead, it should be very likely a network related limitation.
> To efficiently resolve the issue, I strongly recommend you contact the ISP
> and the router manufacturer for more investigation, to see if there are
> some known issues or settings on the router or on the ISP side.
>
> Thanks.
>
> Sincerely,
> Neo Zhu,
> Microsoft Online Support
> Microsoft Global Technical Support Center
>
> Get Secure! - www.microsoft.com/security
> ================================================== ===
> When responding to posts, please "Reply to Group" via your newsreader so
> that others may learn and benefit from your issue.
> ================================================== ===
> This posting is provided "AS IS" with no warranties, and confers no rights.
>
>

Hi David,

Have a look to make sure you are not fragmenting encrypted packets on either
end of the tunnel. If the packets are put into the VPN or otherwise
encrypted, they'll grow, potentially pushing them past your MTU. This means
that they'll fragment as they are shipped into the WAN. This is further
complicated by Windowing as it allows more packets through before it
validates making the reassembly processor intensive. You can lose half of
your bandwidth ore more due to retransmits and the routing of nearly empty
fragments. This theorem is supported by the fact that wrenching up the
window size makes the problem less visible.

You could attack this by dropping the MTU on the servers or by forcing the
encryption after fragmentation. The latter option is done on your router (or
switch if you're running one of the big ones for IP distribution -- sup 720
or the like).

Finally, you might also look at your QoS -- it is possible that you are
being bumped in the transfer or aren't maintaining priority. Check this
both locally and as you enter the IP cloud.

Oh, and looking at SMB signing might also be worth while -- you can see a
30% overhead there.
--
Ryan Hanisco
MCSE, MCTS: SQL 2005, Project+
http://www.techsterity.com
Chicago, IL

Remember: Marking helpful answers helps everyone find the info they need
quickly.


"David" wrote:

> 1. West Coast Site and Remote Site are on Gigabit Ethernet LAN with Gigabit
> to the public Internet and are interconnected to each other by private IP
> transport (Routed Gigabit). Both are connected directly to the external
> router, bypassing the Firewall (Public IP addresses entered directly on the
> Windows NIC). They each also have a second interface/NIC on our private LAN
> (10.x.x.x IP range).
> 2. Each site has a Fibre-connected gigabit router, no rate limit set.
> 3. The servers can communicate with each other by either their public
> interface or their private interface. The private interface uses the private
> IP transport (effectively a site-to-site VPN), the Public interface uses the
> internet. Both have the same performance.
> 4. Copies from the West Coast server to another in the same site happen at
> the expected speed (20%-40% utilization of the max interface speed), the same
> is seen at the remote site.
> 5. Any server in the West Coast site has the same performance limitations
> when communicating with any server in the remote site.
> 6. When using some of the available network speed performance test
> websites, I’ve gotten speeds of:
> Download Speed: 201673 kbps (25209.1 KB/sec transfer rate)
> Upload Speed: 18124 kbps (2265.5 KB/sec transfer rate)
> Those speeds are representative of either site (when testing against a test
> site geographically close).
>
> Additional information: When I test using iPerf, I can see 30%-40% NIC
> utilization when I adjust up the TCP window size. I can also see this
> performance when using multiple simultaneous copy operations (either FTP or
> Robocopy).
>
> "Jian-Ping Zhu [MSFT]" wrote:
> > 1. Please describe the detailed Network Topology.
> > 2. Is there any routing/NAT devices between your two Windows Servers? If
> > yes, please check whether there are any bandwidth limitation policies set
> > on them.
> > 3. Please tell me the way how these two Servers communicate with each
> > other. (Though VPN or WAN connection directly.)
> > 4. Please test the copy speed between two Windows Servers in the same local
> > LAN and check whether the issue still occur.
> > 5. Please choose a different Windows Server located in remote side and test
> > the copy speed between local site's Windows Server and remote site's
> > Windows Server via WAN connection.
> > 6. Please try to download some big document from the internet on local
> > site's Windows Server and test the download speed.
>

Dear David,

Thank you for your reply.

I didn't know IPERF tool before, and after reading your e-mail I had a look
at it. As far as I understand it has a server side service and a client
side component. It looks like it measures network performance by sending
raw data over TCP connection with several options such as changing TCP
window size, changing amount of data to be sent etc. IPERF uses streaming
protocol and it does not rely on the SMB (Server Message Block)
application-layer protocol which is used when performing native Windows
copies.

I'd like to confirm whether there are any bandwidth limitation policies of
specific sessions or QoS policies enabled on your Router or ISP side.
Therefore, I still recommend you contact to the ISP and the Router
manufacturer to confirm this first. This will ensure that we troubleshoot
the issue in the most efficiently way and I appreciate your time.

However, at the same time, I will also keep researching this issue for you.

If I have made any progress, I'll inform you as soon as possible.

Thanks.

Sincerely,
Neo Zhu,
Microsoft Online Support
Microsoft Global Technical Support Center

Get Secure! - www.microsoft.com/security
================================================== ===
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
================================================== ===
This posting is provided "AS IS" with no warranties, and confers no rights.

Thanks. I've forwarded off your response (and this thread) to my ISP. They
have said there is no rate limit on my interface, and they are checking
further for anything on their end..

"Jian-Ping Zhu [MSFT]" wrote:

> Dear David,
>
> Thank you for your reply.
>
> I didn't know IPERF tool before, and after reading your e-mail I had a look
> at it. As far as I understand it has a server side service and a client
> side component. It looks like it measures network performance by sending
> raw data over TCP connection with several options such as changing TCP
> window size, changing amount of data to be sent etc. IPERF uses streaming
> protocol and it does not rely on the SMB (Server Message Block)
> application-layer protocol which is used when performing native Windows
> copies.
>
> I'd like to confirm whether there are any bandwidth limitation policies of
> specific sessions or QoS policies enabled on your Router or ISP side.
> Therefore, I still recommend you contact to the ISP and the Router
> manufacturer to confirm this first. This will ensure that we troubleshoot
> the issue in the most efficiently way and I appreciate your time.
>
> However, at the same time, I will also keep researching this issue for you.
>
> If I have made any progress, I'll inform you as soon as possible.
>
> Thanks.
>
> Sincerely,
> Neo Zhu,
> Microsoft Online Support
> Microsoft Global Technical Support Center
>
> Get Secure! - www.microsoft.com/security
> ================================================== ===
> When responding to posts, please "Reply to Group" via your newsreader so
> that others may learn and benefit from your issue.
> ================================================== ===
> This posting is provided "AS IS" with no warranties, and confers no rights.
>
>

Ryan - thanks for your time on this. I have checked for MTU, and it remains
unfragmented to 1472 (I'm testing the network across the unencrypted WAN, not
through a VPN tunnel).

I haven't looked at SMB yet because of the huge disparity on performance.
At less than 1% network utilization I don't think that the 30% SMB overhead
is causing the problem.

My ISP is saying there is no limitation or QOS, and so I'm further at a loss.

"Ryan Hanisco" wrote:

> Hi David,
>
> Have a look to make sure you are not fragmenting encrypted packets on either
> end of the tunnel. If the packets are put into the VPN or otherwise
> encrypted, they'll grow, potentially pushing them past your MTU. This means
> that they'll fragment as they are shipped into the WAN. This is further
> complicated by Windowing as it allows more packets through before it
> validates making the reassembly processor intensive. You can lose half of
> your bandwidth ore more due to retransmits and the routing of nearly empty
> fragments. This theorem is supported by the fact that wrenching up the
> window size makes the problem less visible.
>
> You could attack this by dropping the MTU on the servers or by forcing the
> encryption after fragmentation. The latter option is done on your router (or
> switch if you're running one of the big ones for IP distribution -- sup 720
> or the like).
>
> Finally, you might also look at your QoS -- it is possible that you are
> being bumped in the transfer or aren't maintaining priority. Check this
> both locally and as you enter the IP cloud.
>
> Oh, and looking at SMB signing might also be worth while -- you can see a
> 30% overhead there.
> --
> Ryan Hanisco
> MCSE, MCTS: SQL 2005, Project+
> http://www.techsterity.com
> Chicago, IL
>
> Remember: Marking helpful answers helps everyone find the info they need
> quickly.
>
>
> "David" wrote:
>
> > 1. West Coast Site and Remote Site are on Gigabit Ethernet LAN with Gigabit
> > to the public Internet and are interconnected to each other by private IP
> > transport (Routed Gigabit). Both are connected directly to the external
> > router, bypassing the Firewall (Public IP addresses entered directly on the
> > Windows NIC). They each also have a second interface/NIC on our private LAN
> > (10.x.x.x IP range).
> > 2. Each site has a Fibre-connected gigabit router, no rate limit set.
> > 3. The servers can communicate with each other by either their public
> > interface or their private interface. The private interface uses the private
> > IP transport (effectively a site-to-site VPN), the Public interface uses the
> > internet. Both have the same performance.
> > 4. Copies from the West Coast server to another in the same site happen at
> > the expected speed (20%-40% utilization of the max interface speed), the same
> > is seen at the remote site.
> > 5. Any server in the West Coast site has the same performance limitations
> > when communicating with any server in the remote site.
> > 6. When using some of the available network speed performance test
> > websites, I’ve gotten speeds of:
> > Download Speed: 201673 kbps (25209.1 KB/sec transfer rate)
> > Upload Speed: 18124 kbps (2265.5 KB/sec transfer rate)
> > Those speeds are representative of either site (when testing against a test
> > site geographically close).
> >
> > Additional information: When I test using iPerf, I can see 30%-40% NIC
> > utilization when I adjust up the TCP window size. I can also see this
> > performance when using multiple simultaneous copy operations (either FTP or
> > Robocopy).
> >
> > "Jian-Ping Zhu [MSFT]" wrote:
> > > 1. Please describe the detailed Network Topology.
> > > 2. Is there any routing/NAT devices between your two Windows Servers? If
> > > yes, please check whether there are any bandwidth limitation policies set
> > > on them.
> > > 3. Please tell me the way how these two Servers communicate with each
> > > other. (Though VPN or WAN connection directly.)
> > > 4. Please test the copy speed between two Windows Servers in the same local
> > > LAN and check whether the issue still occur.
> > > 5. Please choose a different Windows Server located in remote side and test
> > > the copy speed between local site's Windows Server and remote site's
> > > Windows Server via WAN connection.
> > > 6. Please try to download some big document from the internet on local
> > > site's Windows Server and test the download speed.
> >

Dear David,

I spent several hours investigating this issue and I'd like to share you
some progress I have made up till now.

According to your description, you have two large-bandwidth and
long-latency WAN links on two sites.

As we know, long latency is a challenging circumstance for the efficient
operation of most network protocols, in particular those
connection-oriented transport protocols such as TCP which rely on periodic
acknowledgement of receipt. Maximum throughput can be calculated as the
size of the payload that the sender can place on the wire before having to
wait for confirmation multiplied by the number of times per second that
acknowledgement is received and the next lot of payload can be sent. While
decreasing link latency tends to be impractical or not cost-effective in
most situations, the payload size is a function of protocol configuration.

To improve throughput, enabling the TCP receive window scaling options is
recommended, so that the effective maximum receive window size becomes a
multiple of the advertised window. Also, by enabling TCP time stamps
through the same registry bitmask value ('Tcp1323Opts'), will help improve
effective use of the available bandwidth on this link. This could explain
why after you modify the TCP Window Size option in IPERF tool, performance
tests show much faster than before.

I also notice that you have already tried to change registry to change
windows scaling options and timestamp settings but with no help.

After further research, I find out that there is an inherent limitation of
the SMB protocol used by Windows Explorer and the copy command. SMB has an
architectural limitation in the form of a 64KB buffer size limit which
cannot be overcome through the use of TCP window scaling. Therefore, SMB
tends to perform poorly over high latency WAN links.
Fortunately, this limitation no longer exists in SMB 2.0 which is used on
Windows Vista Operating System and Windows Server 2008.

Having said above, I recommend you not use applications which rely on the
SMB protocol to send large amounts of data across the long latency WAN
links.
You might use FTP as the protocol is faster and has the ability to continue
a download were interrupted without having to start the entire file over
again.
Upgrading client OS to Windows Vista and servers OS to Longhorn will
address the problem in the long term.

I hope this helps.

Thanks.

Sincerely,
Neo Zhu,
Microsoft Online Support
Microsoft Global Technical Support Center

Get Secure! - www.microsoft.com/security
================================================== ===
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
================================================== ===
This posting is provided "AS IS" with no warranties, and confers no rights.