55Mb/sec read but only 1mb/sec write via gigabit

When the server raid volume is mounted as a network drive by a client workstation the read rate was 55Mbytes/sec and the write rate was 1Mbyte/sec. More detailed notes explain below

Overview

Essentially the job is to tune the Microsoft windows networking parameters to improve performance for files that are 100mb to 2 GB rather than 50k size email files

The customer has a file server running windows XP pro and 2 client workstations running win XP pro. The problem they have is that the client systems have very slow read and write rates across their gigabit network. Some onsite benchmark tests have shown that the problem is in 2 parts.
1. The raid card on their server is too slow and
2. something about the network link is too slow.

The raid card problem can be improved by changing the raid card and raid configuration. We have established the differences in raid card performance by a series of benchmark tests run under Linux using the IO benchmark tool called Iozone. The IO benchmark tool we used can be downloaded from www.iozone.org

The read and write performance across the gigabit network is where the biggest anomaly in performance is observed. On the server at the customer site the read rate for 1Gb was 270Mbytes/sec and write was 14Mbyte/sec. When the server raid volume is mounted as a network drive by a client workstation the read rate was 55Mbytes/sec and the write rate was 1Mbyte/sec. The read rate demonstrates that the gigabit networking setup is capable of 55mb/s. The local write speed for the raid is adequate and 14mb/s but the clients are unable to achieve better than 1Mbyte/sec across the network. It would seem reasonable to be able to get a write speed of 10Mb/sec under these conditions.

The iozone test: was done by downloading the windows version from the web site. Run the default install. Copy the executable to the directory to be tested. Start the ms-dos prompt and cd to the test directory. Use the following command to run iozon
Iozone –s 1024m –r 1024 –i 0 –i 1 –t

Customer Expectations
1) Recommend a hardware configuration to provide a good write performance
2) Recommend registry settings for winXPpro to support the hardware write performanc
3) Recommend registry setting for win2003 server to support the hardware write performance if winXPpro is not capable of it
4) Recommend a linux configuration to support the hardware write performance if winXPpro or win2003 server is not capable of it
5) Provide a test system on the customer site so they can confirm the solution meets their requirements

Test Process:

A similar hardware configuration was setup
• Server with a 4 channel ATA RAID 5, gigabit Ethernet and running windows X
• An unmanaged gigabit switch and two cat6 ethernet cables
• Client PC Pentium 4, with gigabit Ethernet and running windows XP
• Setup as an isolated network when testing

An improved hardware configuration was later setup
• Server enhanced to an 8 ATA channel raid 5 for some tests, used 64k block
• Server enhanced to an 8 ATA channel raid 10 for some tests, used 1Mb blocks
• Server changed to windows 2003 server and linux dual boot
• Client PC changed to winXP and linux dual boot
• Server grade gigabit card added to server

The purpose of the raid 10 configuration was to provide a very fast write speed that would not limit the write speed of the operating system on the server. A server grade gigabit Ethernet card was added to reduce any limitation of network performance by that

The guide from Microsoft and various other sources of information on the internet indicate that
1) Increasing TcpWindowSize on the client system is likely to improve performance on a good network. Also, GlobalTcpWindowSize
2) Adjusting MTU on the server side can help on a poor network by reducing the MTU but on a good network 1500 should be used.
3) It may be that win XP pro does not support the required adjustments on the file server side so window 2003 server may be required to support the changes to the TCP/IP parameters. So, the file server was setup to boot win XP pro and win 2003 server. The idea being to tune the file server using win2003serv and the client machine with win XP pro. Then see if the optimized parameters for win2003 server will work with win XP pro on the server
4) Microsoft guide on how to optimize a windows 2003 server. This includes a section on file serving http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/networking/tcpip03.mspx
5) Increase TcpAckFrequency from 2 to 13 for gigabit ethernet
6) Guide from Syskonnect for windows 2000. http://www.syskonnect.com/syskonnect/support/info/htm/lanline_e.pdf

Registry settings:
EnablePMTUDiscovery=1
GlobalTcpWindowSize=1073741823 (tried 64kb)
MTU=1500
SackOpts=1
Tcp1323Opts=1
TcpAckFrequency=13
TcpWindowSize=1073741823 (tried 64kb)



Test results (windows only):


The following test was run locally on a hardware raid10 filesystem on win2003

Iozone: Performance Test of File I/O
Version $Revision: 3.217 $
Compiled for 32 bit mode.
Build: Windows

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million,
Jean-Marc Zucconi, Jeff Blomberg,
Erik Habbinga, Kris Strecker.

Run began: Sat Apr 3 15:48:26 2004

File size set to 1048576 KB
Record Size 1024 KB
Command line used: iozone -s 1024m -r 1024 -i 0 -i 1 -t 1
Output is in Kbytes/sec
Time Resolution = 0.000000 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 1 process
Each process writes a 1048576 Kbyte file in 1024 Kbyte records

Children see throughput for 1 initial writers = 9741.57 KB/sec
Parent sees throughput for 1 initial writers = 7047.21 KB/sec
Min throughput per process = 9741.57 KB/sec
Max throughput per process = 9741.57 KB/sec
Avg throughput per process = 9741.57 KB/sec
Min xfer = 1048576.00 KB

Children see throughput for 1 rewriters = 18550.82 KB/sec
Parent sees throughput for 1 rewriters = 11090.36 KB/sec
Min throughput per process = 18550.82 KB/sec
Max throughput per process = 18550.82 KB/sec
Avg throughput per process = 18550.82 KB/sec
Min xfer = 1048576.00 KB

Children see throughput for 1 readers = 100356.28 KB/sec
Parent sees throughput for 1 readers = 100168.72 KB/sec
Min throughput per process = 100356.28 KB/sec
Max throughput per process = 100356.28 KB/sec
Avg throughput per process = 100356.28 KB/sec
Min xfer = 1048576.00 KB

Children see throughput for 1 re-readers = 102444.33 KB/sec
Parent sees throughput for 1 re-readers = 102251.18 KB/sec
Min throughput per process = 102444.33 KB/sec
Max throughput per process = 102444.33 KB/sec
Avg throughput per process = 102444.33 KB/sec
Min xfer = 1048576.00 KB



iozone test complete.


XP client accessing w2003 server via gigabit ethernet.

Iozone: Performance Test of File I/O
Version $Revision: 3.217 $
Compiled for 32 bit mode.
Build: Windows

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million,
Jean-Marc Zucconi, Jeff Blomberg,
Erik Habbinga, Kris Strecker.

Run began: Thu Apr 1 20:01:10 2004

File size set to 1048576 KB
Record Size 1024 KB
Command line used: iozone -s 1024m -r 1024 -i i0 -i 1 -t 1
Output is in Kbytes/sec
Time Resolution = 0.000003 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 1 process
Each process writes a 1048576 Kbyte file in 1024 Kbyte records

Children see throughput for 1 initial writers = 816.38 KB/sec
Parent sees throughput for 1 initial writers = 816.26 KB/sec
Min throughput per process = 816.38 KB/sec
Max throughput per process = 816.38 KB/sec
Avg throughput per process = 816.38 KB/sec
Min xfer = 1048576.00 KB

Children see throughput for 1 rewriters = 2111.24 KB/sec
Parent sees throughput for 1 rewriters = 2109.98 KB/sec
Min throughput per process = 2111.24 KB/sec
Max throughput per process = 2111.24 KB/sec
Avg throughput per process = 2111.24 KB/sec
Min xfer = 1048576.00 KB

Children see throughput for 1 readers = 47182.87 KB/sec
Parent sees throughput for 1 readers = 47172.76 KB/sec
Min throughput per process = 47182.87 KB/sec
Max throughput per process = 47182.87 KB/sec
Avg throughput per process = 47182.87 KB/sec
Min xfer = 1048576.00 KB

Children see throughput for 1 re-readers = 47177.30 KB/sec
Parent sees throughput for 1 re-readers = 47166.67 KB/sec
Min throughput per process = 47177.30 KB/sec
Max throughput per process = 47177.30 KB/sec
Avg throughput per process = 47177.30 KB/sec
Min xfer = 1048576.00 KB



iozone test complete.

Tried a quick 5 thread test next with nbench instead of iozone
This is the xp client accessing via gigabit Ethernet to the win2003 server running raid10.

Disk Performance, MBytes/sec
File size: 100.0 MBytes

thread: 0 1 2 3 4
write: 1.62 4.01 3.25 3.76 3.50
read: 52.83 2.07 2.18 2.07 2.10



5 thread test from xp client to w2003server raid 10 – this is similar to running 5 ftp transfers simultaneously.

Iozone: Performance Test of File I/O
Version $Revision: 3.217 $
Compiled for 32 bit mode.
Build: Windows

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million,
Jean-Marc Zucconi, Jeff Blomberg,
Erik Habbinga, Kris Strecker.

Run began: Mon Apr 5 13:51:27 2004

File size set to 1048576 KB
Record Size 1024 KB
Command line used: iozone -s 1024m -r 1024 -i 0 -i 1 -t 5
Output is in Kbytes/sec
Time Resolution = 0.000000 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 1 process
Each process writes a 1048576 Kbyte file in 1024 Kbyte records

Children see throughput for 1 initial writers = 2805.57 KB/sec
Parent sees throughput for 1 initial writers = 2804.21 KB/sec
Min throughput per process = 560.57 KB/sec
Max throughput per process = 561.57 KB/sec
Avg throughput per process = 561.57 KB/sec
Min xfer = 1047552.00 KB

Children see throughput for 1 rewriters = 4601.82 KB/sec
Parent sees throughput for 1 rewriters = 4598.36 KB/sec
Min throughput per process = 919.82 KB/sec
Max throughput per process = 922.82 KB/sec
Avg throughput per process = 920.82 KB/sec
Min xfer = 1045504.00 KB

Children see throughput for 1 readers = 35300.28 KB/sec
Parent sees throughput for 1 readers = 35288.72 KB/sec
Min throughput per process = 6854.28 KB/sec
Max throughput per process = 7270.28 KB/sec
Avg throughput per process = 7060.28 KB/sec
Min xfer = 989184.00 KB

Children see throughput for 1 re-readers = 37875.33 KB/sec
Parent sees throughput for 1 re-readers = 37858.18 KB/sec
Min throughput per process = 7280.33 KB/sec
Max throughput per process = 7658.33 KB/sec
Avg throughput per process = 7575.33 KB/sec
Min xfer = 997376.00 KB



iozone test complete.


Nbench tests : went to a different client site and ran some tests.
Nbench was tried on a dual xeon server with an adaptec 2120S 128Mb scsi raid controller, five 144Gb SCSI disks in raid5 configuration, with server running win2003.
Local write was 4.4Mbyte/sec read 40Mbyte/sec
Test across gigabit network from a P4 running winXPpro write 1.8Mb/sec read 40

The above test showed a low write performance on the raid and an even worse performance across the network.