root over nfs - performance problems

hello,

we have a computer pool with 25 clients. These clients are diskless
clients (they have a disk, but only for swapping). The complete
filesystem is mounted via nfs.

The clients' OS is SuSE Linux 9.1

The performance is no problem if there are 5 or less clients running.
With 10 clients the respond time is unacceptable. Booting takes very
long, starting applications is very slow.

I've watched the network traffic with 'iptraf'. A client starting a big
application like eclipse or openoffice uses a bandwidth of about 1200
Kbit/s, which could also be handled by a 10Mbit Ethernet card.

Here some words about our hardware infrastructure:
The server is connect with a GigaBit Ethernet to a GigaBit switch. This
switch has a GigaBit uplink to the computer pool.
Each client is connected with a 100 MBit Ethernet to the switch to which
the GigaBit uplink of the server is connected to.

The server is running with SCSI devices with RAID 1.

The server's operating System is SuSE 8.2 (Kernel 2.4).

I've given no special mount options (rsize, wsize, etc). The server is
configured to export it's filesystem asynchronously (option "async" in
/etc/exports).

What do you think, where the bottle neck is? Is it the network or the
server?

What options are there to increase the performance?


Any advice is appreciated !

Thanks

Boris

Boris Glawe wrote:
> hello,
>

<...snip...>

> The server is running with SCSI devices with RAID 1.
>
> The server's operating System is SuSE 8.2 (Kernel 2.4).
>
> I've given no special mount options (rsize, wsize, etc). The server is
> configured to export it's filesystem asynchronously (option "async" in
> /etc/exports).
>
> What do you think, where the bottle neck is? Is it the network or the
> server?
>
> What options are there to increase the performance?
>
>
> Any advice is appreciated !
>
> Thanks
>
> Boris

make sure the kernel is up to date.

make sure all NICs are set to correct speed on switch and host.
autonegotiate is bad.

you can use a larger rsize/wsize on the clients, like 8192.

increase the nfs daemons on the server by starting more nfsd processes,
much more.

>
>
> make sure the kernel is up to date.

It's the most current kernel for the Suse Linux 8.2 distribution. It's a 2.4
kernel though.

>
> make sure all NICs are set to correct speed on switch and host.
> autonegotiate is bad.

The NICs are running at the highest possible speed.

>
> you can use a larger rsize/wsize on the clients, like 8192.

Is this really a good idea? The packages size is bigger then, which causes even
more traffic!?
Since the clients don't really consume much bandwidth (about 1200 KBits/s only
while starting big apps) I assume, that the problem is more the repsonse time
and not the amount of actually transported data!?

>
> increase the nfs daemons on the server by starting more nfsd processes,
> much more.

How does nfsd behave: Does it start new nfsd processes anyway as soon as there
are more then 4 clients connected or do these 4 nfs processes handle all the
traffic? In the latter case more processes is usefull...

I could also find out many of questions on my own. But testing a new
configuration is a lot of work for me, since I cannot use 8 clients at the same
time as 8 users would do. In addition the computer pool is in a different
building. So I need your help, where to start with the search.

thanks Boris

I've got a general question in addition:

Do you expect the network, which I've described above, to be fast enough to
handle the load caused by 25 "root over nfs" -clients?

Or is it an illusion to think that this network works with a satisfying performance?

thanks Boris

Boris Glawe wrote:
>
>>
>>
>> make sure the kernel is up to date.
>
>
> It's the most current kernel for the Suse Linux 8.2 distribution. It's a
> 2.4 kernel though.
>
>>
>> make sure all NICs are set to correct speed on switch and host.
>> autonegotiate is bad.
>
>
> The NICs are running at the highest possible speed.
>
>>
>> you can use a larger rsize/wsize on the clients, like 8192.
>
>
> Is this really a good idea? The packages size is bigger then, which
> causes even more traffic!?
> Since the clients don't really consume much bandwidth (about 1200
> KBits/s only while starting big apps) I assume, that the problem is
> more the repsonse time and not the amount of actually transported data!?
>

those parameters mean how much data to shove in one block. nfsv3 would
be better. if the NICs can handle large blocks then usually nfs is
faster with larger chunk sizes. YMMV.

only 1.2MB/s? that's slow for fast ethernet esp if you say when they
start big apps? are the clients' NICs 100meg or 10meg?

>>
>> increase the nfs daemons on the server by starting more nfsd
>> processes, much more.
>
>
> How does nfsd behave: Does it start new nfsd processes anyway as soon as
> there are more then 4 clients connected or do these 4 nfs processes
> handle all the traffic? In the latter case more processes is usefull...
>
> I could also find out many of questions on my own. But testing a new
> configuration is a lot of work for me, since I cannot use 8 clients at
> the same time as 8 users would do. In addition the computer pool is in a
> different building. So I need your help, where to start with the search.
>
> thanks Boris

i dont use SuSe but in RedHat it was as simple as increasing the
RPCNFSDCOUNT in /etc/init.d/nfs, so you may have to hack away in the
startup file or it may be defined in one of the parameters file in
/etc/sysconfig. i have mine set to 32, and there are 32 nfsd processes
running.

the daemons get started all at once and never die. so if you only have
4 processes those 4 processes handle all the load. i could be wrong
with the 2.6 kernels tho?

usually in most UNIXen the more nsfd processes you have running to match
the number of clients the better the response.

Boris Glawe wrote:
> I've got a general question in addition:
>
> Do you expect the network, which I've described above, to be fast enough
> to handle the load caused by 25 "root over nfs" -clients?
>
> Or is it an illusion to think that this network works with a satisfying
> performance?
>
> thanks Boris

if you have a gigabit link going into the switch for the nfs server and
your clients are "statically" set for thier speed on both ends, then i
should think you have plenty.

i'm assuming everything is on one switch (not a hub).

you can also do some OS tuning if everything else fails... that gets
into a whole diff subject.

>>
>> Is this really a good idea? The packages size is bigger then, which
>> causes even more traffic!?
>> Since the clients don't really consume much bandwidth (about 1200
>> KBits/s only while starting big apps) I assume, that the problem is
>> more the repsonse time and not the amount of actually transported data!?
>>
>
> those parameters mean how much data to shove in one block. nfsv3 would
> be better. if the NICs can handle large blocks then usually nfs is
> faster with larger chunk sizes. YMMV.

ok, I specified "rsize=8192,wsize=8192" with the mount command.

>
> only 1.2MB/s? that's slow for fast ethernet esp if you say when they
> start big apps? are the clients' NICs 100meg or 10meg?

All clients have 100 MBit NICs.
The response time is almost the same as if I started the application locally,
while the client uses these 1200KBits/s traffic. I meassured this number with
only one booted client.
I assume, that the client doesn't need anymore bandwidth, as it's spending a lot
of time with execution, before it requests another binary or library. Maybe I am
wrong with this, but I cannot imagine, that a network like ours has a thoughput
like an ADSL connection :-)

>>
>> How does nfsd behave: Does it start new nfsd processes anyway as soon
>> as there are more then 4 clients connected or do these 4 nfs processes
>> handle all the traffic? In the latter case more processes is usefull...
>>


> i dont use SuSe but in RedHat it was as simple as increasing the
> RPCNFSDCOUNT in /etc/init.d/nfs, so you may have to hack away in the
> startup file or it may be defined in one of the parameters file in
> /etc/sysconfig. i have mine set to 32, and there are 32 nfsd processes
> running.
>
> the daemons get started all at once and never die. so if you only have
> 4 processes those 4 processes handle all the load. i could be wrong
> with the 2.6 kernels tho?


Ok, this will hopefully help. I've increased the number of nfsd threads to 32.
This would also be the explanation why 4-5 clients work without any performance
loss and 10 clients are hardly usable. I'll give it a try.


>
> usually in most UNIXen the more nsfd processes you have running to match
> the number of clients the better the response.


Thanks,

I'll be at my university on thursday again. I can then report , whether it worked !!

greets Boris