NFS server performace issue

Hi,
We are having some rather strange issues with NFS and SuSe 9.0 (kernel
2.4.21). We have the kernel NFS server setup with 24 threads on a file
server that is used in a 174 node beowulf cluster. The file server has a
single AMD opteron 2 gig's ram, and a 1 TB 3ware raid array. When doing
rpc's from the storage server to nodes nfs seems to stop responding or
respond very slowly. Network bandwidth is not being exhausted. Could this be
thread contention on the server? Or would jumbo frames help this situation.
I would welcome any experiences with a setup such as this. Thanks.

-=>Adam<=-




Viper

In article <(E-Mail Removed)>, Viper wrote:
> Hi,
> We are having some rather strange issues with NFS and SuSe 9.0 (kernel
> 2.4.21). We have the kernel NFS server setup with 24 threads on a file
> server that is used in a 174 node beowulf cluster. The file server has a

Err, do you see the problem here? 24 threads for 174 clients? I have a
couple of 2TB systems serving only about 20 nodes, and I've got 256
threads on each of those. There's very little penalty in cranking up
the number of nfsd threads.

> single AMD opteron 2 gig's ram, and a 1 TB 3ware raid array. When doing
> rpc's from the storage server to nodes nfs seems to stop responding or
> respond very slowly. Network bandwidth is not being exhausted. Could this be
> thread contention on the server? Or would jumbo frames help this situation.
> I would welcome any experiences with a setup such as this. Thanks.

Look at the output of 'cat /proc/net/rpc/nfsd', specifically the line
starting with "th", e.g.:

th 256 846 2641.140 1033.620 444.270 196.350 123.210 54.990 36.530 27.830 19.010 32.410

The first number is the number of threads. The second number is the number
of times all the threads have been busy and the server has head to deny
I/O requests. After that, each number is the amount of time X% of the
threads have been busy simultaneously, with X being 10, 20, 30... That
server got bumped from 128 to 256 threads without being rebooted, so
the stats didn't get reset. The 846 was there before I moved to 256,
and hasn't increased since. Ideally, you want the number to be zero as
well as the last few numbers.

--
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University