unable to start nfsd; nfssvc "Address already in use"

hi,
running 2.6.18.2 and had a problem with starting NFS; portmap was
running fine but when rpc.nfsd was launched, it stopped immediately and
syslog showed

nfssvc "Address already in use"

there were no other nfsd runnning and even restarting the portmap,
mountd, rquotad and also un/re-loading the nfsd/sunrpc/lockd/exportfs
kernel modules werent able to resolve this. only way to get nfsd to
start was to reboot the box.

is this a kernel bug? it has only happened once.

using 2.6.18.2 kernel and nfs-utils-1.0.10
thanks
-ray

On 14 Nov 2006, in the Usenet newsgroup comp.os.linux.misc, in article
<(E-Mail Removed) .com>,
(E-Mail Removed) wrote:

>running 2.6.18.2 and had a problem with starting NFS; portmap was
>running fine but when rpc.nfsd was launched, it stopped immediately and
>syslog showed
>
> nfssvc "Address already in use"
>
>there were no other nfsd runnning and even restarting the portmap,
>mountd, rquotad and also un/re-loading the nfsd/sunrpc/lockd/exportfs
>kernel modules werent able to resolve this. only way to get nfsd to
>start was to reboot the box.

A little late now, but:

netstat -anptu
fuser -vn PROTOCOL PORT_NUMBER
lsof -i :PORT_NUMBER

where PROTOCOL is 'tcp' or 'udp' as appropriate, and 'PORT_NUMBER' is the
port number in question. You had some other application that was using
the port in question. As only one service can have exclusive use of a port,
you got the "Address already in use" message telling you why _this_ process
couldn't start, but not who/what that other process was. When you rebooted,
you killed that process, perhaps causing it to move to some other port - not
enough details to say for sure.

>is this a kernel bug? it has only happened once.

Nope.

Old guy

looks like a bug to me and not related to alredy running nfsd etc;
strace -f -x -o /tmp/nfsd.trace rpc.nfsd

1929 open("/proc/fs/nfsd/threads", O_WRONLY|O_LARGEFILE) = 4
1929 write(4, "8\n", 2) = -1 EADDRINUSE (Address
already in use)

which this error is then logged via syslog (this what causes the nfssvc
error i posted previously) -- if you are unfamiliar with this, it is
saying that nfsd opened the file /proc/fs/nfsd/threads succesffully but
then the next call to write 2 bytes to it causes the EADDRINUSE

anyone got any suggestions? full strace output below



1929 execve("/usr/sbin/rpc.nfsd", ["rpc.nfsd", "8"], [/* 23 vars
*/]) = 0
1929 uname({sys="Linux", node="doc", ...}) = 0
1929 brk(0) = 0x804b000
1929 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or
directory)
1929 open("/etc/ld.so.cache", O_RDONLY) = 3
1929 fstat64(3, {st_mode=S_IFREG|0644, st_size=40336, ...}) = 0
1929 mmap2(NULL, 40336, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7faa000
1929 close(3) = 0
1929 open("/lib/libc.so.6", O_RDONLY) = 3
1929 read(3,
"\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\ x00\x00"..., 512) =
512
1929 fstat64(3, {st_mode=S_IFREG|0755, st_size=1423161, ...}) = 0
1929 mmap2(0x41686000, 1137756, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x41686000
1929 mmap2(0x41796000, 16384, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x110) = 0x41796000
1929 mmap2(0x4179a000, 7260, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4179a000
1929 close(3) = 0
1929 mmap2(NULL, 4096, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fa9000
1929 mprotect(0x41796000, 8192, PROT_READ) = 0
1929 mprotect(0x41682000, 4096, PROT_READ) = 0
1929 set_thread_area({entry_number:-1 -> 6, base_addr:0xb7fa9a90,
limit:1048575, seg_32bit:1, contents:0, read_exec_only:0,
limit_in_pages:1, seg_not_present:0, useable:1}) = 0
1929 munmap(0xb7faa000, 40336) = 0
1929 brk(0) = 0x804b000
1929 brk(0x806c000) = 0x806c000
1929 open("/etc/nsswitch.conf", O_RDONLY) = 3
1929 fstat64(3, {st_mode=S_IFREG|0644, st_size=1686, ...}) = 0
1929 mmap2(NULL, 4096, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fb3000
1929 read(3, "#\n# /etc/nsswitch.conf\n#\n# An ex"..., 4096) = 1686
1929 read(3, "", 4096) = 0
1929 close(3) = 0
1929 munmap(0xb7fb3000, 4096) = 0
1929 open("/etc/ld.so.cache", O_RDONLY) = 3
1929 fstat64(3, {st_mode=S_IFREG|0644, st_size=40336, ...}) = 0
1929 mmap2(NULL, 40336, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7faa000
1929 close(3) = 0
1929 open("/lib/libnss_files.so.2", O_RDONLY) = 3
1929 read(3,
"\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\ x00\x00"..., 512) =
512
1929 fstat64(3, {st_mode=S_IFREG|0755, st_size=43568, ...}) = 0
1929 mmap2(NULL, 37512, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7f9f000
1929 mmap2(0xb7fa7000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7) = 0xb7fa7000
1929 close(3) = 0
1929 mprotect(0xb7fa7000, 4096, PROT_READ) = 0
1929 munmap(0xb7faa000, 40336) = 0
1929 open("/etc/services", O_RDONLY) = 3
1929 fcntl64(3, F_GETFD) = 0
1929 fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
1929 fstat64(3, {st_mode=S_IFREG|0644, st_size=19936, ...}) = 0
1929 mmap2(NULL, 4096, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fb3000
1929 read(3, "# /etc/services:\n# $Id: services"..., 4096) = 4096
1929 read(3, "TBIOS Name Service\nnetbios-ns\t13"..., 4096) = 4096
1929 read(3, "rcs\t\t994/udp\npop3s\t\t995/tcp\t\t\t\t#"..., 4096) =
4096
1929 close(3) = 0
1929 munmap(0xb7fb3000, 4096) = 0
1929 chdir("/var/lib/nfs") = 0
1929 open("/dev/null", O_RDWR|O_LARGEFILE) = 3
1929 dup2(3, 0) = 0
1929 dup2(3, 1) = 1
1929 dup2(3, 2) = 2
1929 open("/proc/self/fd",
O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 4
1929 fstat64(4, {st_mode=S_IFDIR|0500, st_size=0, ...}) = 0
1929 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
1929 getdents64(4, /* 7 entries */, 1024) = 168
1929 getdents64(4, /* 0 entries */, 1024) = 0
1929 close(4) = 0
1929 open("/proc/fs/nfsd/versions", O_WRONLY|O_LARGEFILE) = 4
1929 write(4, "+2 +3 +4 \n", 10) = 10
1929 close(4) = 0
1929 open("/proc/fs/nfsd/portlist", O_RDONLY|O_LARGEFILE) = -1
ENOENT (No such file or directory)
1929 open("/proc/fs/nfsd/threads", O_WRONLY|O_LARGEFILE) = 4
1929 write(4, "8\n", 2) = -1 EADDRINUSE (Address
already in use)
1929 close(4) = 0
1929 time([1163703611]) = 1163703611
1929 open("/etc/localtime", O_RDONLY) = 4
1929 fstat64(4, {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
1929 fstat64(4, {st_mode=S_IFREG|0644, st_size=1323, ...}) = 0
1929 mmap2(NULL, 4096, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fb3000
1929 read(4,
"\x54\x5a\x69\x66\x00\x00\x00\x00\x00\x00\x00\x00\ x00\x00"..., 4096) =
1323
1929 close(4) = 0
1929 munmap(0xb7fb3000, 4096) = 0
1929 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323,
....}) = 0
1929 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323,
....}) = 0
1929 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1323,
....}) = 0
1929 getpid() = 1929
1929 socket(PF_UNIX, SOCK_DGRAM, 0) = 4
1929 fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
1929 connect(4, {sa_family=AF_UNIX, path="/dev/log"}, 16) = 0
1929 send(4, "<27>Nov 16 19:00:11 nfsd[1929]: "..., 62,
MSG_NOSIGNAL) = 62
1929 close(4) = 0
1929 exit_group(1) = ?

On Thursday 16 November 2006 19:09, (E-Mail Removed)
<(E-Mail Removed)> wrote a load of stuff in
comp.os.linux.misc without giving any context and without quoting the
stuff to which he was replying:

> looks like a bug to me and not related to alredy running nfsd etc;

Given that you include no context, WTF are you on about?

Now comes the "canned" speech about what's wrong with GG posting.
It's offered as a learning tool, to help you get more, and better
help, without upsetting old grouches, like me, in the process. Read
it in that spirit. If you detect a note of anger, or frustration,
it's not with you, but rather with GG, for their crappy implementation
of their interface. Â*;-)


OK, let's start with the basics: Do you know what Usenet is? Do you
know what a News Group is? How about a news server? Click on this
link for the answers.

Â* Â*http://en.wikipedia.org/wiki/Usenet

Google unfortunately decided not to use the standards of quoting on
Usenet. A quick fix can be read on the following page:

Â* Â*http://www.safalra.com/special/googlegroupsreply/

If you want to get better help and learn more, please read on.

I sympathize with your problems, and am more than willing to help you
solve them. To do so you should follow the correct quoting principles
and good Netiquette explained in this FAQ available at;

Â* Â*http://www.plainfaqs.org/linux/

While you are at it, read the whole document and the links provided
in it as well. Lots of good things there.

Due to the bad quoting habits and the amount of drivel coming from
postings done via Google Groups, numerous people have resorted in
kill filing (not reading or responding to) all postings done with
Google.

This means fewer knowledgeable people that are able, or willing, to
help you. Proper quoting, and good netiquette, will keep more from
doing the same.

It's not your fault you got sucked into GG's bad habits. It will be
your fault, if you don't act to remedy the situation. Remember this:
it's in your best interest to keep these knowledgeable people happy,
by following a few simple rules. If you don't, you will be the only
loser.

To get the best possible result from your postings, it is better to
use a newsreader or Usenet client. Linux has many dedicated news
readers, as well as browsers with built in news readers.

If you're running SUSE Linux, look on your menu, under Internet -->
News, and you'll see which news readers you have installed. If you're
running something else, look for a good newsreader. Don't use an
email program.

Find a Usenet server that you can post to. Your ISP probably has one.
There are also free ones out there, that Google will help you locate.
Type in "Usenet free servers". better yet, use this:

Â* Â*http://tinyurl.com/c45wm

The one your provider has is most likely the better, faster and
easier choice. Set up the news reader of your choice, download the
groups list, type linux in the search box, then subscribe to
alt.os.linux. and alt.os.linux.<your distro> as well as any others
that interest you; comp.os.linux.<whatever> may also prove helpful.

When I see correct quoting from you, instead of what Google has been
forcing you to do, I'll be right there to greet you, and help get you
going. Until then:

I am out of here.


--
Robert HULL

Archival or publication of this article on any part of thisishull.net
is without consent and is in direct breach of the Data Protection Act

Robert Hull wrote:
> On Thursday 16 November 2006 19:09, (E-Mail Removed)
> Given that you include no context, WTF are you on about?

genuine problem as describe, if you havent got a solution or clue, how
about this: "suck the **** up you pointless fool"

(E-Mail Removed) wrote:
> looks like a bug to me and not related to alredy running nfsd etc;
> strace -f -x -o /tmp/nfsd.trace rpc.nfsd
>
> 1929 open("/proc/fs/nfsd/threads", O_WRONLY|O_LARGEFILE) = 4
> 1929 write(4, "8\n", 2) = -1 EADDRINUSE (Address
> already in use)
>
> which this error is then logged via syslog (this what causes the nfssvc
> error i posted previously) -- if you are unfamiliar with this, it is
> saying that nfsd opened the file /proc/fs/nfsd/threads succesffully but
> then the next call to write 2 bytes to it causes the EADDRINUSE

with a few pointers from the nfs developers, i managed to get some sort
of solution.

running "netstat -anptu | grep 2049" as root when the problem occurred
again, i could see that the statd had taken that port (this is the port
that the nfsd wants to come up on)

udp 0 0 0.0.0.0:2049 0.0.0.0:*
954/rpc.statd

this seems to have come about due to the nfs (starts
quotad,nfsd,mountd) and nfslock (starts lockd,statd) services running
and competing for ports -- the FC2 setup didnt seem to have defined a
port for statd and it just then took 2049

since there are no std port for stat i just chose 2 fairly random
values, put them into the FC2's sysconfig area

#/etc/sysconfig/nfs
STATD_PORT=2090
STATD_OUTGOING_PORT=2091

hope this helps others that have had the same problem

On 18 Nov 2006, in the Usenet newsgroup comp.os.linux.misc, in article
<(E-Mail Removed) .com>,
(E-Mail Removed) wrote:

>with a few pointers from the nfs developers, i managed to get some sort
>of solution.
>
>running "netstat -anptu | grep 2049" as root when the problem occurred
>again, i could see that the statd had taken that port (this is the port
>that the nfsd wants to come up on)

OK - that explains the "Address already in use" message.

>this seems to have come about due to the nfs (starts
>quotad,nfsd,mountd) and nfslock (starts lockd,statd) services running
>and competing for ports -- the FC2 setup didnt seem to have defined a
>port for statd and it just then took 2049

In your original post, you stated:

]using 2.6.18.2 kernel and nfs-utils-1.0.10

and those version numbers don't appear in Fedora (FC2 came out of box with
2.6.5.1, and had several updates of 2.6.10, ending with 2.6.10-2.3.legacy_FC2
in February 2006 - and last I checked late last month, the latest nfs-utils
Fedora was offering was 1.0.8-7 in the newly released FC6). Do you know if
this problem occurred with the original kernel/applications?

Old guy

Moe Trin wrote:

> In your original post, you stated:
>
> ]using 2.6.18.2 kernel and nfs-utils-1.0.10
>
> and those version numbers don't appear in Fedora (FC2 came out of box with
> 2.6.5.1, and had several updates of 2.6.10, ending with 2.6.10-2.3.legacy_FC2
> in February 2006 - and last I checked late last month, the latest nfs-utils
> Fedora was offering was 1.0.8-7 in the newly released FC6). Do you know if
> this problem occurred with the original kernel/applications?
>
> Old guy

dont know about the original FC2 kernel or the utils -- i've had to
upgrade the kernel for other reasons (problems with one of my usb
devices for instance)

either way, it seems like something that isnt quite right but at least
we have a work around. a few places do suggest that this is due to
statd not having a standards defined port so when the process starts,
it asks portmapper to give it a set of ports to use; unfortunately for
me, portmapper gave back the same ones that nfsd wants/needs to use

On 18 Nov 2006, in the Usenet newsgroup comp.os.linux.misc, in article
<(E-Mail Removed) .com>, Ray wrote:

>Moe Trin wrote:

>> Do you know if this problem occurred with the original kernel/applications?

>dont know about the original FC2 kernel or the utils -- i've had to
>upgrade the kernel for other reasons (problems with one of my usb
>devices for instance)

That always happens with older distributions. I suspect something went
wrong when you upgraded, as Red Hat/Fedora has been supplying NFS for
over ten years. I wouldn't think you'd be the first person in ten years
to run NFS and statd.

>either way, it seems like something that isnt quite right but at least
>we have a work around. a few places do suggest that this is due to
>statd not having a standards defined port so when the process starts,
>it asks portmapper to give it a set of ports to use; unfortunately for
>me, portmapper gave back the same ones that nfsd wants/needs to use

Asking portmapper is standard. Could this be a problem with yours?
FC2 came with portmap-4.0-59, and I don't believe that it was ever
updated. The most recent version supplied with FC6 is 4.0-65.2.2.1.

Old guy