nfs speed improvements, anyone?

hi,
i am using nfs v3, a switch, and 100mbit ethernet cards.
if i take a large testfile (lets say 500MB) and copy it using MC, i get an
average speed of ~ 3.5MB/sec. IMHO this is a little slow for 100mbit, isnt
it?

i am mounting like this:
192.168.0.254:/mnt/data /mnt/data nfs rsize=32768,wsize=32768 0 0

btw.. playing with rsize and wsize doesnt matter much
the server has async set, with or without async doesnt matter much either

any hints appreciated, thx,
dieter

"Dieter Soltau" <(E-Mail Removed)> wrote in message
news:cevog9$reu$06$(E-Mail Removed)...
> hi,
> i am using nfs v3, a switch, and 100mbit ethernet cards.
> if i take a large testfile (lets say 500MB) and copy it using MC, i get an
> average speed of ~ 3.5MB/sec. IMHO this is a little slow for 100mbit, isnt
> it?
>
> i am mounting like this:
> 192.168.0.254:/mnt/data /mnt/data nfs rsize=32768,wsize=32768 0
0
>
> btw.. playing with rsize and wsize doesnt matter much
> the server has async set, with or without async doesnt matter much either
>
> any hints appreciated, thx,
> dieter
>

Dieter,

You might want to have a look at http://nfs.sourceforge.net
There is a nice performance section that goes through many
interesting details.

Enjoy,
Mangled&Munged

thx, specially this part
http://nfs.sourceforge.net/nfs-howto/performance.html
looks interessting!

anyway, i am just wondering what speeds others in here get out using 100mbit
cards.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
NotDashEscaped: You need GnuPG to verify this message

In comp.os.linux.networking Dieter Soltau <(E-Mail Removed)> suggested:
> thx, specially this part
> http://nfs.sourceforge.net/nfs-howto/performance.html
> looks interessting!

> anyway, i am just wondering what speeds others in here get out using 100mbit
> cards.

10 MB/sec NFS doesn't seem to be a problem with quality NIC. To
rule out other problems, I'd try ftp/(scp only with really fast CPUs)
at first, to rule out problems with your NICs/cabling/etc. With
quality equipment and good cabling, you can easily max out 100 MB
(FD) at around 12.5 MB/sec using ftp and even scp with fast
machines.

--
Michael Heiming (GPG-Key ID: 0xEDD27B94)
mail: echo (E-Mail Removed) | perl -pe 'y/a-z/n-za-m/'
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBE7EYAkPEju3Se5QRAk0HAJ9I+psxZAm1wvBb6Ii8I3 PmsoeU1wCgo2Cf
gTKnIZm+ZKuUA8wiKaUv/tg=
=beVS
-----END PGP SIGNATURE-----

On Fri, 06 Aug 2004 13:05:15 +0200, Dieter Soltau wrote:
> i am using nfs v3, a switch, and 100mbit ethernet cards.
> if i take a large testfile (lets say 500MB) and copy it using MC, i get an
> average speed of ~ 3.5MB/sec. IMHO this is a little slow for 100mbit, isnt
> it?
>
> i am mounting like this:
> 192.168.0.254:/mnt/data /mnt/data nfs rsize=32768,wsize=32768 0 0
>
> btw.. playing with rsize and wsize doesnt matter much
> the server has async set, with or without async doesnt matter much either

That might not be atypical. BTW, are you both reading and writing from
that same NFS mount? So, is your aggregate speed 2x 3.5MBps = 7MBps?

I recall from 10Mbps ethernet days one of the rules of thumb that was used
was that you could only achieve about 30% or 40% of the bandwidth on a
real network with many MAC devices. As you pushed harder/farther you would
start to have more collisions and more backoff/retransmits which would
make the collisions more probable: i.e. an "avalanche kind of breakdown.
With sparse networks (2 stations?) you were able to do quite a bit better,
but still nowhere close to 100%. In this regard Token Ring
(contentionless) were better since you could push them up to almost 100%
bandwidth utilization. In practice, it doesn't seem to matter much. I
don't know if the achievable bandwidth is appreciably different for
100Mbps ethernet? I'll have to follow up on the other posts.

FWIW, I just ran "bonnie -s 2000" in my home directory. I'm on a SuSE 9.1
Linux 800MHz P3 machine and the NFS server is a Sun Ultra2 Solaris8. I
don't think the CPUs or architectures matter much. These are my results:


$ bonnie -s 2000
Bonnie 1.4: File './Bonnie.14723', size: 2097152000, volumes: 1
Writing with putc()... done: 3654 kB/s 31.3 %CPU
Rewriting... done: 1088 kB/s 58.1 %CPU
Writing intelligently... done: 5161 kB/s 6.0 %CPU
Reading with getc()... done: 6657 kB/s 83.4 %CPU
Reading intelligently... done: 9011 kB/s 12.6 %CPU
Seeker 2...Seeker 3...Seeker 1...start 'em...done...done...done...
---Sequential Output (nosync)--- ---Sequential Input-- --Rnd Seek-
-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --04k (03)-
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
hppav1 1*2000 3654 31.3 5161 6.0 1088 58.1 6657 83.4 9011 12.6 87.6 4.5


I suppose I might be able to tweak them somewhat, but I'm not concerned. I
use my NFS for reliability and flexibility: e.g. get my own home
directory automounted everywhere, whichever O/S I happen to be running.

If you're reading/writing large working/temporary files, keep them in a
local /tmp file. Otherwise, plan on about 5MBps. What are you trying to do?

p.s. The performance of NFS and (esp.) ftp is not only limited by the
network bandwidth but also by the response time for acknowledge packets,
etc., which have to make the round trip, before another bunch of real data
is sent. This can introduce "dead time" into the transfer. See some stuff
on satellite links, etc., where they had to increase the sliding window!

--
Juhan Leemet
Logicognosis, Inc.

Dieter Soltau <(E-Mail Removed)> wrote in message news:<cevog9$reu$06$(E-Mail Removed)>...
> hi,
> i am using nfs v3, a switch, and 100mbit ethernet cards.
> if i take a large testfile (lets say 500MB) and copy it using MC, i get an
> average speed of ~ 3.5MB/sec. IMHO this is a little slow for 100mbit, isnt
> it?

You _should_ be able to do better, but you would be surprised how
common this performance level is in the real world (as opposed to the
lab or test net).

> i am mounting like this:
> 192.168.0.254:/mnt/data /mnt/data nfs rsize=32768,wsize=32768 0 0

If everything (server and clients) runs Linux you might want to double
check how the automounter is configured (maps, nsswitch and NIS
configs) just to eliminate these as a source of problems. How does
the transfer _rate_ compare with, say, a 50MB file transfer?

> btw.. playing with rsize and wsize doesnt matter much
> the server has async set, with or without async doesnt matter much either

async won't effect the client's read request -- it's for client write
requests. BTW, for comparison, what _is_ the write performance?

> any hints appreciated, thx,
> dieter

postmaster pointed to the usual starting place -- or specifically:
http://nfs.sourceforge.net/nfs-howto/performance.html

You must establish some baseline read/write I/O for both the server
and the client as you can imagine local I/O will limit the wire
transfer speed at _some_ point.

Confirm your server's configuration, since with v3 it has precedence
over mount options issued at the client. Check
security/authentication settings. Check whether you're using TCP or
UDP transport. Think of _anything_ that may effect request
processing. And remember, NFS relies on RPC facilities.

Sniff the wire. I like Ethereal. This is meant to confirm that the
network itself is not the bottleneck (ie., is operating below par).
Eg., does the window size get up to 32k quickly and stay there? What
other traffic is on the wire? Broadcasts? Keep-alives? Arp? Are
the machines on the same subnet? Must the traffic be carried on a
"shared" backbone? Are packets fragmenting? Any ifconfig stats
pointing to hardware problems? What does nfsstat report re: a large
file copy? A large file write? Netstat?

For large file transfers, network cards are often swamped, especially
if there are many other interrupts to be processed. This is most
noticable when the server is using a "desktop" nic rather than a
server grade nic. You may or may not be able to tune performance by
adjusting nic configuration params. If not, you _may_ actually get
better performance with a _smaller_ r/wsize -- ie., a size that
matches the nics capability to the max without _over_taxing_ it. Also
note, that with such a large tranfer, the client will have to flush to
_it's_ disk efficiently so that both disk interrupts and nic
interrupts are processed in a "steady" stream.

This should get you started. Chances are that you have several things
that are "degrading" performance and you'll just have to chase them
down one by one. Sorry there is no cookbook recipe when it comes to
improving the performance of network services (or discovering the
source of a problem).

I would do a quick and dirty test of the local I/O performance of the
server and client, then sniff the wire for clues, run ifconfig,
netstat, and nfsstat for ideas on adjusting network params (or locate
a network based bottleneck / problem).

Double check that NFS server and clients are up to date _and_
confirmed to work well together (not all clients work well with all
servers and vice versa).

I would probably like to compare NFS performance with a local net FTP
and/or HTTP server if they were readily available.

hth,
prg
email above disabled

whoa - lots of hints, thx a lot!

Juhan Leemet wrote:

> FWIW, I just ran "bonnie -s 2000" in my home directory.
how could i forgot about this nice and handy tool, thx! yes, i have my
homedirectory mounted as well.

OK, thx a lot, will investigate this weekend!

"Dieter Soltau" <(E-Mail Removed)> wrote in message
news:cf08c2$g63$02$(E-Mail Removed)...
> thx, specially this part
> http://nfs.sourceforge.net/nfs-howto/performance.html
> looks interessting!
>
> anyway, i am just wondering what speeds others in here get out using
100mbit
> cards.
>

Dieter,

On most days I get around 10 Mbytes/sec on 100mbit cards.
The trick is making sure that:

1. The server has sufficient disk bandwidth. (multiple spindles)
fast disk (able to sustain 10 Mbytes/sec), or striped disks.
2. The server is 2.6 ish ( Bug fixes in 2.6 help NFS)
3. The client is 2.6 ish (Bug fixes in 2.6 help NFS)
4. The client uses NFS mount options: -o nfsvers=3, rsize=32768,
wsize=32768
5. The LAN connection from the client to the server is
via a 100mbit switch (no routers, no hops)

Enjoy,
Mangled&Munged

"Dieter Soltau" <(E-Mail Removed)> wrote in message
news:cf0v0f$bn5$06$(E-Mail Removed)...
> whoa - lots of hints, thx a lot!
>

Google is your friend.

Tons of information on tuning Linux can be found the Internet-tho' not all
of it is applicable for one reason or another. I have just spend a couple
of
weeks (on and off) wading through it and testng a Fedora Core2 build w/
the w/ 2.6.7 kernel. Even the scheduler we are using.

There are so many variables involved. What CPU. How much memory.
Is TCP NFS turned on in the kernel (32K will most likely not work w/o
it). What is the underlying file system. What ethernet card, how is it
connected (you will get better performance by auto negotiating than
by forcing). What disks are you using and how are they connected?
And so on and so on.

There has been some really good advice on this thread. Take a look at
the TROND patches at nfs.sourceforge.net

--
Angel R. Rivera aka Wolf
----------------------------------------------------------------
Please post all reponses to UseNet. All email cheerfully and automagically
routed to Dave Null