NFS on GbE tunning

Any experience or hints to share for NFS server/client
configuration running linux on Gigabit Ethernet?

I've the following testing environment:

Intel P4 w/Intel Gigabit LAN (CSA)/RedHat 9 as NFS client
AMD Athlon w/Broadcom 5705/Fedora Core 1 as NFS server
Each machine has 512MB RAM each and uses ext3 filesystems.

I tried to copy the root filesystem from the NFS client to
the server, but the performance is terrible, something
300kBytes to 600kBytes per second (as shown by iostat from
both server/client).

I'd also tried rsize=8192, wsize=8192 option when mounting
the nfs on the client, but the performance is more or less
the same.

Compare with using samba 3.0 on the server and Win2k on the
client, I got around 15MBytes to 20MBytes per second.

So, any suggestion to tune up the NFS performance?

Stephen Wong @ Hong Kong

In article <(E-Mail Removed) domain>, Stephen SM WONG wrote:
>
> Intel P4 w/Intel Gigabit LAN (CSA)/RedHat 9 as NFS client
> AMD Athlon w/Broadcom 5705/Fedora Core 1 as NFS server
> Each machine has 512MB RAM each and uses ext3 filesystems.
>
> I tried to copy the root filesystem from the NFS client to
> the server, but the performance is terrible, something
> 300kBytes to 600kBytes per second (as shown by iostat from
> both server/client).

Something *bad* is happening.

> I'd also tried rsize=8192, wsize=8192 option when mounting
> the nfs on the client, but the performance is more or less
> the same.

The NFS HOWTO at <http://nfs.sourceforge.net/nfs-howto/> has a whole
chapter on tuning.

The first thing I'd do is determing the overall network health
between those two machines -- get NFS out of the equation. Use netperf
or ttcp or another network testing tool. To me, this sounds like a
sick network.

--
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Hi Joshua,

Thank you for your hints. But I'm quite sure the network is
working properly at gigabit/full duplex speed. For your
information, actually, the 2 machines are sitting side by
side and connected by a short plain CAT5 straight cable.

The (NFS) server side runs Fedora Core 1, samba 3.0 and
nfsd.

The client side runs RH9 / Win2k, when samba is used (client
is Win2k), I got file read/write performance of up to
20MBytes per second, well, converted to around 200Mbps on
the net, but I think the bottleneck may be well on the
filesystem/harddisk throughput.

I rebooted the client with RH9, using Intel e1000 kernel
module on 2.4.22 kernel and nfs mount the same remote
filesystem as similar to when samba is used on Win2k, and do
the file copy. I used something like `tar cf - . | ( cd
nfs_mountpoint ; tar xf - )` to copy the root filesystem on
the RH9, but the performance is very bad.

I'll have a look at the nfs tunning howto, but I don't know
how many parameters have to be reconsidered when the network
speed is higher than the disk throughput. That's not the
case in the last 10 years, the disk was faster.

Stephen Wong @ Hong Kong

On Wed, 14 Apr 2004, Joshua Baker-LePain wrote:

> In article <(E-Mail Removed) domain>, Stephen SM WONG wrote:
> >
> > Intel P4 w/Intel Gigabit LAN (CSA)/RedHat 9 as NFS client
> > AMD Athlon w/Broadcom 5705/Fedora Core 1 as NFS server
> > Each machine has 512MB RAM each and uses ext3 filesystems.
> >
> > I tried to copy the root filesystem from the NFS client to
> > the server, but the performance is terrible, something
> > 300kBytes to 600kBytes per second (as shown by iostat from
> > both server/client).
>
> Something *bad* is happening.
>
> > I'd also tried rsize=8192, wsize=8192 option when mounting
> > the nfs on the client, but the performance is more or less
> > the same.
>
> The NFS HOWTO at <http://nfs.sourceforge.net/nfs-howto/> has a whole
> chapter on tuning.
>
> The first thing I'd do is determing the overall network health
> between those two machines -- get NFS out of the equation. Use netperf
> or ttcp or another network testing tool. To me, this sounds like a
> sick network.
>
> --
> Joshua Baker-LePain
> Department of Biomedical Engineering
> Duke University
>

In article <(E-Mail Removed) domain>, Stephen SM WONG wrote:
>
> Thank you for your hints. But I'm quite sure the network is
> working properly at gigabit/full duplex speed. For your
> information, actually, the 2 machines are sitting side by
> side and connected by a short plain CAT5 straight cable.

Well, that's really no guarantee. Stuff like netcat, netperf,
and ttcp can still debug problems in drivers, e.g.. Oh and CAT5e is
recommended for gigabit over copper.

> I rebooted the client with RH9, using Intel e1000 kernel
> module on 2.4.22 kernel and nfs mount the same remote
> filesystem as similar to when samba is used on Win2k, and do
> the file copy. I used something like `tar cf - . | ( cd
> nfs_mountpoint ; tar xf - )` to copy the root filesystem on
> the RH9, but the performance is very bad.

Try (on the client) 'tar cO /nfs/mountpoint | cat > /dev/null'.
That way only one disk (the on the server) is in the equation.

--
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

In article <(E-Mail Removed)>,
Joshua Baker-LePain <(E-Mail Removed)> wrote:
> In article <(E-Mail Removed) domain>, Stephen SM WONG wrote:
> >
> > Thank you for your hints. But I'm quite sure the network is
> > working properly at gigabit/full duplex speed. For your
> > information, actually, the 2 machines are sitting side by
> > side and connected by a short plain CAT5 straight cable.
>
> Well, that's really no guarantee. Stuff like netcat, netperf,
> and ttcp can still debug problems in drivers, e.g.. Oh and CAT5e is
> recommended for gigabit over copper.
>
> > I rebooted the client with RH9, using Intel e1000 kernel
> > module on 2.4.22 kernel and nfs mount the same remote
> > filesystem as similar to when samba is used on Win2k, and do
> > the file copy. I used something like `tar cf - . | ( cd
> > nfs_mountpoint ; tar xf - )` to copy the root filesystem on
> > the RH9, but the performance is very bad.
>
> Try (on the client) 'tar cO /nfs/mountpoint | cat > /dev/null'.
> That way only one disk (the on the server) is in the equation.

I think that'll attempt to write to the tape drive (besides the UUOC). Do
you mean

tar cf /dev/null /nfs/mountpoint

?

--
-eben (E-Mail Removed)m home.tampabay.rr.com/hactar
SAGITTARIUS: All your friends are laughing behind your back... kill
them. Take down all those naked pictures of Ernest Borgnine you've got
hanging in your den. -- Weird Al, _Your Horoscope for Today_

In article <c5mm5m$mde$(E-Mail Removed)>, Hactar wrote:
> In article <(E-Mail Removed)>,
> Joshua Baker-LePain <(E-Mail Removed)> wrote:
>>
>> Try (on the client) 'tar cO /nfs/mountpoint | cat > /dev/null'.
>> That way only one disk (the on the server) is in the equation.
>
> I think that'll attempt to write to the tape drive (besides the UUOC). Do
> you mean
>
> tar cf /dev/null /nfs/mountpoint

No. The 'O' option of tar tells it to put the output on stdout. The
pipe then takes stdout and cats it to /dev/null.

The reason not to do 'tar cf /dev/null' is that when tar sees that /dev/null
is the output, it doesn't bother actually reading the bits off the disk
(since it knows you're just throwing them out anyway). All it does is
stat all the files it *would* have read if you were doing a "real" tar.
This is useful, e.g., if you want to test your exclusion list or get
a size estimate (this is how amanda uses tar to estimate backup sizes).
It's *not* useful if you want to actually test read speed from the disk
(or mount point).

--
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

In article <(E-Mail Removed)>,
Joshua Baker-LePain <(E-Mail Removed)> wrote:
> In article <c5mm5m$mde$(E-Mail Removed)>, Hactar wrote:
> > In article <(E-Mail Removed)>,
> > Joshua Baker-LePain <(E-Mail Removed)> wrote:
> >>
> >> Try (on the client) 'tar cO /nfs/mountpoint | cat > /dev/null'.
> >> That way only one disk (the on the server) is in the equation.
> >
> > I think that'll attempt to write to the tape drive (besides the UUOC). Do
> > you mean
> >
> > tar cf /dev/null /nfs/mountpoint
>
> No. The 'O' option of tar tells it to put the output on stdout. The
> pipe then takes stdout and cats it to /dev/null.

Ah, of course. I had misread the oh as a zero. Bad choice of font, I guess.

> The reason not to do 'tar cf /dev/null' is that when tar sees that /dev/null
> is the output, it doesn't bother actually reading the bits off the disk
> (since it knows you're just throwing them out anyway). All it does is
> stat all the files it *would* have read if you were doing a "real" tar.
> This is useful, e.g., if you want to test your exclusion list or get
> a size estimate (this is how amanda uses tar to estimate backup sizes).
> It's *not* useful if you want to actually test read speed from the disk
> (or mount point).

Then wouldn't it be simpler just to do

cat file > /dev/null

?

--
-eben (E-Mail Removed)m home.tampabay.rr.com/hactar

Every normal man must be tempted at times to spit upon his hands,
hoist the black flag, and begin slitting throats. -- H.L. Mencken

In article <c5p17h$nqo$(E-Mail Removed)>, Hactar wrote:
>
> Then wouldn't it be simpler just to do
>
> cat file > /dev/null

It depends on what you're trying to test. Streaming a single file is easier
than streaming lots of files. If you just want raw speed, sure, cat
a single file. But 'tarring' is (sometimes) a lot closer to actual usage.

Also, I like to test with at least 2X RAM data, and I don't often have
2+GB files just laying about.

--
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

"Hactar" <(E-Mail Removed)> wrote in message
news:c5p17h$nqo$(E-Mail Removed)...
> In article <(E-Mail Removed)>,
> Joshua Baker-LePain <(E-Mail Removed)> wrote:
> > In article <c5mm5m$mde$(E-Mail Removed)>, Hactar wrote:
> > > In article <(E-Mail Removed)>,
> > > Joshua Baker-LePain <(E-Mail Removed)> wrote:
> > >>
> > >> Try (on the client) 'tar cO /nfs/mountpoint | cat > /dev/null'.
> > >> That way only one disk (the on the server) is in the equation.
> > >
> > > I think that'll attempt to write to the tape drive (besides the UUOC).
Do
> > > you mean
> > >
> > > tar cf /dev/null /nfs/mountpoint
> >
> > No. The 'O' option of tar tells it to put the output on stdout. The
> > pipe then takes stdout and cats it to /dev/null.
>
> Ah, of course. I had misread the oh as a zero. Bad choice of font, I
guess.
>
> > The reason not to do 'tar cf /dev/null' is that when tar sees that
/dev/null
> > is the output, it doesn't bother actually reading the bits off the disk
> > (since it knows you're just throwing them out anyway). All it does is
> > stat all the files it *would* have read if you were doing a "real" tar.
> > This is useful, e.g., if you want to test your exclusion list or get
> > a size estimate (this is how amanda uses tar to estimate backup sizes).
> > It's *not* useful if you want to actually test read speed from the disk
> > (or mount point).
>
> Then wouldn't it be simpler just to do
>
> cat file > /dev/null
>
> ?
>
> --
> -eben (E-Mail Removed)m home.tampabay.rr.com/hactar
>
> Every normal man must be tempted at times to spit upon his hands,
> hoist the black flag, and begin slitting throats. -- H.L. Mencken

I would just run Iozone on the NFS client. Then one could measure
read and write performance. Also, if you use the -c flag, and you are
on NFS V3, you'll get the close() included in the measurment. If you
use a file size (-s ###) that is smaller than the NFS V3 client cache,
then
you'll be measuring the client side cache performance. If you use a file
size that is bigger than the amount of RAM in the client, then you'll
be measuring the network and the server's I/O subsystem performance.
If you use a file size that is larger than the amount of RAM in both
the client and the server, then you be measuring the I/O subsystem
on the server. (if its slower than the network)

See: http://www.iozone.org

Have fun,
Mangled&Munged