Windows 2003 Large File Copy Performance Drops Off

All,

I've got a strange problem trying to copy large files. All weekend I've
been trying to solve this problem and can't seem to find a newsgroup posting
or KB article on point. After some trial and error I've narrowed the problem
down to copying a single 5-8 gig file (it doens't matter which one). For
the first ~3 Gigs (or maybe its first 3-4 minutes) I get 20-30% NIC
utilization, then when the magic threshold is hit the speed drops off to
about 0.5%. The drop is sudden and consistent.

Some other background:
- Source is Windows 2003 Standard
- Desitnation is Windows 2003 x64 Enterprise
- Each server has a gigabit NIC
- Links seem to be 1Gps full duplex
- Both servers have latest NIC drivers
- Both servers have SP2 & up to date on fixes
- Tried multiple Network Cables with the same results
- Tried multiple gigabit switches (nothing fancy, just unmanaged switches)
with the same results
- isolated servers on their own switch with same result
- servers are doing nothing other than the file copy
- disk activity seems to be minimal
- No anti virus running
- Servers are just in workgroup

- Problem crosses problem doesn't is experienced using Windows Explorer,
Copy, Robocopy, and even FTP
- Using mapped drive or UNC is no different
- Some have mentioned large file copies chewing up available ram, then
slowing --- not the case here. Machines are using between 200-400 Megs of
ram but have 2 gigs (source) and 8 gigs (destination) installed
- duplex mismatch -- doesn't seem to be the case here otherwise i wouldn't
be consistently getting ~250Mbs for the first 5 minutes.



Thanks in advance if you can provide any help. I've been tearing my hair
out for the past few days trying to track this down. I've got a close to a
terabyte to transfer and speeds like that of a 56k modem is kind of a
problem.

Thanks
-eric

Try installing this on both machines & restart server afterwards:
http://support.microsoft.com/default.aspx/kb/948496

Regards,

Andrew


"Eric Barr" <(E-Mail Removed)> wrote in message
news:9BFB2E29-6844-4B3F-84C9-(E-Mail Removed)...
> All,
>
> I've got a strange problem trying to copy large files. All weekend I've
> been trying to solve this problem and can't seem to find a newsgroup
> posting or KB article on point. After some trial and error I've narrowed
> the problem down to copying a single 5-8 gig file (it doens't matter which
> one). For the first ~3 Gigs (or maybe its first 3-4 minutes) I get
> 20-30% NIC utilization, then when the magic threshold is hit the speed
> drops off to about 0.5%. The drop is sudden and consistent.
>
> Some other background:
> - Source is Windows 2003 Standard
> - Desitnation is Windows 2003 x64 Enterprise
> - Each server has a gigabit NIC
> - Links seem to be 1Gps full duplex
> - Both servers have latest NIC drivers
> - Both servers have SP2 & up to date on fixes
> - Tried multiple Network Cables with the same results
> - Tried multiple gigabit switches (nothing fancy, just unmanaged switches)
> with the same results
> - isolated servers on their own switch with same result
> - servers are doing nothing other than the file copy
> - disk activity seems to be minimal
> - No anti virus running
> - Servers are just in workgroup
>
> - Problem crosses problem doesn't is experienced using Windows Explorer,
> Copy, Robocopy, and even FTP
> - Using mapped drive or UNC is no different
> - Some have mentioned large file copies chewing up available ram, then
> slowing --- not the case here. Machines are using between 200-400 Megs
> of ram but have 2 gigs (source) and 8 gigs (destination) installed
> - duplex mismatch -- doesn't seem to be the case here otherwise i wouldn't
> be consistently getting ~250Mbs for the first 5 minutes.
>
>
>
> Thanks in advance if you can provide any help. I've been tearing my hair
> out for the past few days trying to track this down. I've got a close to
> a terabyte to transfer and speeds like that of a 56k modem is kind of a
> problem.
>
> Thanks
> -eric

Andrew,

Thanks for the idea, but the hotfix didn't help. I applied to both servers
and still experience the same behavior.

Thanks,
-eric

"Andrew Lomakin" <(E-Mail Removed)> wrote in message
news:108D48EB-5E75-4A63-B355-(E-Mail Removed)...
> Try installing this on both machines & restart server afterwards:
> http://support.microsoft.com/default.aspx/kb/948496
>
> Regards,
>
> Andrew
>
>
> "Eric Barr" <(E-Mail Removed)> wrote in message
> news:9BFB2E29-6844-4B3F-84C9-(E-Mail Removed)...
>> All,
>>
>> I've got a strange problem trying to copy large files. All weekend I've
>> been trying to solve this problem and can't seem to find a newsgroup
>> posting or KB article on point. After some trial and error I've narrowed
>> the problem down to copying a single 5-8 gig file (it doens't matter
>> which one). For the first ~3 Gigs (or maybe its first 3-4 minutes) I
>> get 20-30% NIC utilization, then when the magic threshold is hit the
>> speed drops off to about 0.5%. The drop is sudden and consistent.
>>
>> Some other background:
>> - Source is Windows 2003 Standard
>> - Desitnation is Windows 2003 x64 Enterprise
>> - Each server has a gigabit NIC
>> - Links seem to be 1Gps full duplex
>> - Both servers have latest NIC drivers
>> - Both servers have SP2 & up to date on fixes
>> - Tried multiple Network Cables with the same results
>> - Tried multiple gigabit switches (nothing fancy, just unmanaged
>> switches) with the same results
>> - isolated servers on their own switch with same result
>> - servers are doing nothing other than the file copy
>> - disk activity seems to be minimal
>> - No anti virus running
>> - Servers are just in workgroup
>>
>> - Problem crosses problem doesn't is experienced using Windows Explorer,
>> Copy, Robocopy, and even FTP
>> - Using mapped drive or UNC is no different
>> - Some have mentioned large file copies chewing up available ram, then
>> slowing --- not the case here. Machines are using between 200-400 Megs
>> of ram but have 2 gigs (source) and 8 gigs (destination) installed
>> - duplex mismatch -- doesn't seem to be the case here otherwise i
>> wouldn't be consistently getting ~250Mbs for the first 5 minutes.
>>
>>
>>
>> Thanks in advance if you can provide any help. I've been tearing my hair
>> out for the past few days trying to track this down. I've got a close to
>> a terabyte to transfer and speeds like that of a 56k modem is kind of a
>> problem.
>>
>> Thanks
>> -eric
>

Wow Eric, that's a strange one!

Have you tried copying from one of the server to another system (maybe
your laptop), to see if either one might be causing it? (I'm guessing
it's Nic or software related, and possibly only affects one of the servers.

I'm not sure if you have ever heard of the small tool 'netio.exe' but in
cases such as these it might come in helpful in getting closer to the
source of the problem.

NetIO can be found here, no installation needed.

http://www.ars.de/ars/ars.nsf/docs/netio

regards,

Paul


Eric Barr wrote:
> All,
>
> I've got a strange problem trying to copy large files. All weekend I've
> been trying to solve this problem and can't seem to find a newsgroup
> posting or KB article on point. After some trial and error I've narrowed
> the problem down to copying a single 5-8 gig file (it doens't matter
> which one). For the first ~3 Gigs (or maybe its first 3-4 minutes) I
> get 20-30% NIC utilization, then when the magic threshold is hit the
> speed drops off to about 0.5%. The drop is sudden and consistent.
>
> Some other background:
> - Source is Windows 2003 Standard
> - Desitnation is Windows 2003 x64 Enterprise
> - Each server has a gigabit NIC
> - Links seem to be 1Gps full duplex
> - Both servers have latest NIC drivers
> - Both servers have SP2 & up to date on fixes
> - Tried multiple Network Cables with the same results
> - Tried multiple gigabit switches (nothing fancy, just unmanaged
> switches) with the same results
> - isolated servers on their own switch with same result
> - servers are doing nothing other than the file copy
> - disk activity seems to be minimal
> - No anti virus running
> - Servers are just in workgroup
>
> - Problem crosses problem doesn't is experienced using Windows
> Explorer, Copy, Robocopy, and even FTP
> - Using mapped drive or UNC is no different
> - Some have mentioned large file copies chewing up available ram, then
> slowing --- not the case here. Machines are using between 200-400 Megs
> of ram but have 2 gigs (source) and 8 gigs (destination) installed
> - duplex mismatch -- doesn't seem to be the case here otherwise i
> wouldn't be consistently getting ~250Mbs for the first 5 minutes.
>
>
>
> Thanks in advance if you can provide any help. I've been tearing my
> hair out for the past few days trying to track this down. I've got a
> close to a terabyte to transfer and speeds like that of a 56k modem is
> kind of a problem.
>
> Thanks
> -eric

Eric,

Additionally slow file copy may be caused by low memory condition, that you
won't see in the task manager.

To properly troubleshoot if that's a low mem condition, do this.

On the source server, open performance monitor, and add the following
counters from the memory object:
`Paged pool bytes`, `Non Paged pool bytes`
Also add `current disk queue lenght` counter from `physical disk` object,
and select the instance of the physical disk from which you are copying file
from.

Then start copying process and see how high do these counters go, post
results...

Regards,

Andrew



"Eric Barr" <(E-Mail Removed)> wrote in message
news:0C338CB1-B410-423E-80C5-(E-Mail Removed)...
> Andrew,
>
> Thanks for the idea, but the hotfix didn't help. I applied to both
> servers and still experience the same behavior.
>
> Thanks,
> -eric
>
> "Andrew Lomakin" <(E-Mail Removed)> wrote in message
> news:108D48EB-5E75-4A63-B355-(E-Mail Removed)...
>> Try installing this on both machines & restart server afterwards:
>> http://support.microsoft.com/default.aspx/kb/948496
>>
>> Regards,
>>
>> Andrew
>>
>>
>> "Eric Barr" <(E-Mail Removed)> wrote in message
>> news:9BFB2E29-6844-4B3F-84C9-(E-Mail Removed)...
>>> All,
>>>
>>> I've got a strange problem trying to copy large files. All weekend I've
>>> been trying to solve this problem and can't seem to find a newsgroup
>>> posting or KB article on point. After some trial and error I've narrowed
>>> the problem down to copying a single 5-8 gig file (it doens't matter
>>> which one). For the first ~3 Gigs (or maybe its first 3-4 minutes) I
>>> get 20-30% NIC utilization, then when the magic threshold is hit the
>>> speed drops off to about 0.5%. The drop is sudden and consistent.
>>>
>>> Some other background:
>>> - Source is Windows 2003 Standard
>>> - Desitnation is Windows 2003 x64 Enterprise
>>> - Each server has a gigabit NIC
>>> - Links seem to be 1Gps full duplex
>>> - Both servers have latest NIC drivers
>>> - Both servers have SP2 & up to date on fixes
>>> - Tried multiple Network Cables with the same results
>>> - Tried multiple gigabit switches (nothing fancy, just unmanaged
>>> switches) with the same results
>>> - isolated servers on their own switch with same result
>>> - servers are doing nothing other than the file copy
>>> - disk activity seems to be minimal
>>> - No anti virus running
>>> - Servers are just in workgroup
>>>
>>> - Problem crosses problem doesn't is experienced using Windows
>>> Explorer, Copy, Robocopy, and even FTP
>>> - Using mapped drive or UNC is no different
>>> - Some have mentioned large file copies chewing up available ram, then
>>> slowing --- not the case here. Machines are using between 200-400 Megs
>>> of ram but have 2 gigs (source) and 8 gigs (destination) installed
>>> - duplex mismatch -- doesn't seem to be the case here otherwise i
>>> wouldn't be consistently getting ~250Mbs for the first 5 minutes.
>>>
>>>
>>>
>>> Thanks in advance if you can provide any help. I've been tearing my
>>> hair out for the past few days trying to track this down. I've got a
>>> close to a terabyte to transfer and speeds like that of a 56k modem is
>>> kind of a problem.
>>>
>>> Thanks
>>> -eric
>>
>

Andre & Paul,

I ran some tests based on your suggestions and here is what I came up with.

First, I copied files to and from my vista laptop (with a gigabit NIC) to
the x64 server. Speeds overall were a bit slower (not surprising) but ...
- Copying from x64 server to laptop -- copy speed dropped off to a
negligible speed after ~4 gigs
- Copying from laptop to the x64 server --there was NO DROP OFF. One
interesting thing I noted was the copy speed was faster and fluctuated much
less fluctuation than I've ever seen before. Vista must have done some sort
of buffering and smoothed out the transmission (24MBs up versus 17MBs down)

Second I copied files to and from the laptop to the x32 server.
- Copying from x32 server to laptop -- copy speed did not drop off on a 6.5
gig file
- Copying from laptop to the x32 server -- At first he copy was fast &
smooth as above. Then, it dropped but it was earlier than usual (~1 gig in)
and the NIC utilization was in the 3-5% range, rathern than 0.5%. Even more
interesting, after about 5 min of reduced speed copy, the transfer rate went
back up to the 25% range for about 1/2 a gig, then dropped down again.


Third, Copying from x32 server to x64 server I ran Performance Monitor.
- Pre Copy (x64 - desitnation)
- Pool Paged Bytes ~79.7 million
- Pool Nonpaged Bytes ~49.2 million
- Current Disk Queue (System Disk) ~0
- Current Disk Queue (Data Disk) ~0

- During High Transfer Rate (x64 destination)
- Pool Paged Bytes ~ 105.8 million
- Pool Nonpaged Bytes ~ 49.1 million
- Current Disk Queue (System Disk) ~68
- Current Disk Queue (Data Disk) ~0

- After Transfer Rate Falls
- Pool Paged Bytes ~ 105.8 million
- Pool Nonpaged Bytes ~49.1 million
- Current Disk Queue (System Disk) ~0
- Current Disk Queue (Data Disk) ~0


- Pre Copy (x32 - source)
- Pool Paged Bytes ~23.0 million
- Pool Nonpaged Bytes ~22.2 million
- Current Disk Queue (Same Disk) ~0

- During High Transfer Rate (x32 - source)
- Pool Paged Bytes ~ 22.7 million
- Pool Nonpaged Bytes ~ 22.2 million
- Current Disk Queue (Same Disk) ~1

- After Transfer Rate Falls (x32 - source)
- Pool Paged Bytes ~ 22.7 million
- Pool Nonpaged Bytes ~ 22.2 million
- Current Disk Queue (Same Disk) ~0


Unfortunately, none of this tells me anything. The disk queue staying high
on the destination machine even though the input from the network has
collapsed is strange. Does this give you guys any ideas?

Pools look OK.

Weird is the destination high disk queue length...were you copying to the C
drive, or to the data disk?

Source/target server full specs please? (include network cards)

Regards,

Andrew


"Eric Barr" <(E-Mail Removed)> wrote in message
news:34F99188-7080-4DB6-9257-(E-Mail Removed)...
> Andre & Paul,
>
> I ran some tests based on your suggestions and here is what I came up
> with.
>
> First, I copied files to and from my vista laptop (with a gigabit NIC) to
> the x64 server. Speeds overall were a bit slower (not surprising) but ...
> - Copying from x64 server to laptop -- copy speed dropped off to a
> negligible speed after ~4 gigs
> - Copying from laptop to the x64 server --there was NO DROP OFF. One
> interesting thing I noted was the copy speed was faster and fluctuated
> much less fluctuation than I've ever seen before. Vista must have done
> some sort of buffering and smoothed out the transmission (24MBs up versus
> 17MBs down)
>
> Second I copied files to and from the laptop to the x32 server.
> - Copying from x32 server to laptop -- copy speed did not drop off on a
> 6.5 gig file
> - Copying from laptop to the x32 server -- At first he copy was fast &
> smooth as above. Then, it dropped but it was earlier than usual (~1 gig
> in) and the NIC utilization was in the 3-5% range, rathern than 0.5%.
> Even more interesting, after about 5 min of reduced speed copy, the
> transfer rate went back up to the 25% range for about 1/2 a gig, then
> dropped down again.
>
>
> Third, Copying from x32 server to x64 server I ran Performance Monitor.
> - Pre Copy (x64 - desitnation)
> - Pool Paged Bytes ~79.7 million
> - Pool Nonpaged Bytes ~49.2 million
> - Current Disk Queue (System Disk) ~0
> - Current Disk Queue (Data Disk) ~0
>
> - During High Transfer Rate (x64 destination)
> - Pool Paged Bytes ~ 105.8 million
> - Pool Nonpaged Bytes ~ 49.1 million
> - Current Disk Queue (System Disk) ~68
> - Current Disk Queue (Data Disk) ~0
>
> - After Transfer Rate Falls
> - Pool Paged Bytes ~ 105.8 million
> - Pool Nonpaged Bytes ~49.1 million
> - Current Disk Queue (System Disk) ~0
> - Current Disk Queue (Data Disk) ~0
>
>
> - Pre Copy (x32 - source)
> - Pool Paged Bytes ~23.0 million
> - Pool Nonpaged Bytes ~22.2 million
> - Current Disk Queue (Same Disk) ~0
>
> - During High Transfer Rate (x32 - source)
> - Pool Paged Bytes ~ 22.7 million
> - Pool Nonpaged Bytes ~ 22.2 million
> - Current Disk Queue (Same Disk) ~1
>
> - After Transfer Rate Falls (x32 - source)
> - Pool Paged Bytes ~ 22.7 million
> - Pool Nonpaged Bytes ~ 22.2 million
> - Current Disk Queue (Same Disk) ~0
>
>
> Unfortunately, none of this tells me anything. The disk queue staying
> high on the destination machine even though the input from the network has
> collapsed is strange. Does this give you guys any ideas?
>

Andrew,

Thanks again for taking an interest in this problem. On both servers I was
copying to/from the data drive. On the source machine it is a RAID5 stack
(4 x 500 Gig SATA II) with two partitions. On the destination (x64) there
are 3 x 750 Gig in a RAID5 stack for the data drive, plus a separate
phyiscal drive for the OS.

Specs -- don't get too excited, they are SOHO type boxes

Source (x32)
ASUS A8N-SLI Premium 939 NVIDIA nForce4 SLI ATX AMD Motherboard
AMD Athlon 64 X2 4200+ Manchester 2.2 GHz
2x1Ggig DDR 438
4 x 500 Gig Segate Barracude 7200.9 SATA 2.0Gb/s
Drives are plugged into the motherboard raid (crappy I know) and set as
a single RAID 5 stack with 2 partitions - one for the OS one for datda

Destination (x32)
ASUS P5N-D LGA 775 NVIDIA nForce 750i SLI ATX Intel Motherboard
3ware 9650SE-4LPML PCI Express Lanes: 4 SATA II Controller Card (w/
Battery Backup)
Intel Core 2 Quad Q6600 Kentsfield 2.4GHz
4 x 2 Gig DDR2 800
3 x Seagate Barracuda 750GB 7200 RPM SATA 3.0Gb/s
1x IDE Drive
The three 750 Gig drives are bound together in a RAID 5 stack (the data
drive) and connected to the 3ware controller. The IDE drive is for the OS.

Both machines have nVidia gigabit network ports on the Motherboard. The
source also has a marvell gigabit NIC (its disabled, but i tried that port
as well, with the current one disabled).

Hope that helps.
-eric

I've sorted out what was happening. My RAID card was to close to one of
the hard drives. After a short period of activity the hard drive heat up
enough to make the RAID controller's battery uncomfortable with the heat.
As a result, the RAID controller disabled write caching and slowed the
drives down to reduce heat. To fix the problem I moved the one hard drive
mounted next to the controller and poof the problem disappeared. Now I'm
getting in the 200-300 Mbs consistently across a 100 Gig copy. Now all I
need to do is tune the TCP stack to get that up in the 500 Mbs range.

Sorry for any time/effort anyone wasted and thanks for all the help.

Regards,
-eric


"Eric Barr" <(E-Mail Removed)> wrote in message
news:9BFB2E29-6844-4B3F-84C9-(E-Mail Removed)...
> All,
>
> I've got a strange problem trying to copy large files. All weekend I've
> been trying to solve this problem and can't seem to find a newsgroup
> posting or KB article on point. After some trial and error I've narrowed
> the problem down to copying a single 5-8 gig file (it doens't matter which
> one). For the first ~3 Gigs (or maybe its first 3-4 minutes) I get
> 20-30% NIC utilization, then when the magic threshold is hit the speed
> drops off to about 0.5%. The drop is sudden and consistent.
>
> Some other background:
> - Source is Windows 2003 Standard
> - Desitnation is Windows 2003 x64 Enterprise
> - Each server has a gigabit NIC
> - Links seem to be 1Gps full duplex
> - Both servers have latest NIC drivers
> - Both servers have SP2 & up to date on fixes
> - Tried multiple Network Cables with the same results
> - Tried multiple gigabit switches (nothing fancy, just unmanaged switches)
> with the same results
> - isolated servers on their own switch with same result
> - servers are doing nothing other than the file copy
> - disk activity seems to be minimal
> - No anti virus running
> - Servers are just in workgroup
>
> - Problem crosses problem doesn't is experienced using Windows Explorer,
> Copy, Robocopy, and even FTP
> - Using mapped drive or UNC is no different
> - Some have mentioned large file copies chewing up available ram, then
> slowing --- not the case here. Machines are using between 200-400 Megs
> of ram but have 2 gigs (source) and 8 gigs (destination) installed
> - duplex mismatch -- doesn't seem to be the case here otherwise i wouldn't
> be consistently getting ~250Mbs for the first 5 minutes.
>
>
>
> Thanks in advance if you can provide any help. I've been tearing my hair
> out for the past few days trying to track this down. I've got a close to
> a terabyte to transfer and speeds like that of a 56k modem is kind of a
> problem.
>
> Thanks
> -eric

Good to see a solution

Hard drive heating up within minutes, causing flaws around half of the
progress...weird.

RAID Cache...yeah...vital


"Eric Barr" <(E-Mail Removed)> wrote in message
news:6D2DE7D8-0790-457D-AFD6-(E-Mail Removed)...
> I've sorted out what was happening. My RAID card was to close to one of
> the hard drives. After a short period of activity the hard drive heat up
> enough to make the RAID controller's battery uncomfortable with the heat.
> As a result, the RAID controller disabled write caching and slowed the
> drives down to reduce heat. To fix the problem I moved the one hard
> drive mounted next to the controller and poof the problem disappeared.
> Now I'm getting in the 200-300 Mbs consistently across a 100 Gig copy.
> Now all I need to do is tune the TCP stack to get that up in the 500 Mbs
> range.
>
> Sorry for any time/effort anyone wasted and thanks for all the help.
>
> Regards,
> -eric
>
>
> "Eric Barr" <(E-Mail Removed)> wrote in message
> news:9BFB2E29-6844-4B3F-84C9-(E-Mail Removed)...
>> All,
>>
>> I've got a strange problem trying to copy large files. All weekend I've
>> been trying to solve this problem and can't seem to find a newsgroup
>> posting or KB article on point. After some trial and error I've narrowed
>> the problem down to copying a single 5-8 gig file (it doens't matter
>> which one). For the first ~3 Gigs (or maybe its first 3-4 minutes) I
>> get 20-30% NIC utilization, then when the magic threshold is hit the
>> speed drops off to about 0.5%. The drop is sudden and consistent.
>>
>> Some other background:
>> - Source is Windows 2003 Standard
>> - Desitnation is Windows 2003 x64 Enterprise
>> - Each server has a gigabit NIC
>> - Links seem to be 1Gps full duplex
>> - Both servers have latest NIC drivers
>> - Both servers have SP2 & up to date on fixes
>> - Tried multiple Network Cables with the same results
>> - Tried multiple gigabit switches (nothing fancy, just unmanaged
>> switches) with the same results
>> - isolated servers on their own switch with same result
>> - servers are doing nothing other than the file copy
>> - disk activity seems to be minimal
>> - No anti virus running
>> - Servers are just in workgroup
>>
>> - Problem crosses problem doesn't is experienced using Windows Explorer,
>> Copy, Robocopy, and even FTP
>> - Using mapped drive or UNC is no different
>> - Some have mentioned large file copies chewing up available ram, then
>> slowing --- not the case here. Machines are using between 200-400 Megs
>> of ram but have 2 gigs (source) and 8 gigs (destination) installed
>> - duplex mismatch -- doesn't seem to be the case here otherwise i
>> wouldn't be consistently getting ~250Mbs for the first 5 minutes.
>>
>>
>>
>> Thanks in advance if you can provide any help. I've been tearing my hair
>> out for the past few days trying to track this down. I've got a close to
>> a terabyte to transfer and speeds like that of a 56k modem is kind of a
>> problem.
>>
>> Thanks
>> -eric
>