rcp (Redhat 8) fails with "Address already in use"

Hello,

I'm getting "Address already in use" error from the rcp program; why
would it return that? Here's what happens:

rcp is called from C code via execvp() like so (3 args):
/bin/sh -c "rcp /path/to/file host:/path/to/file"

The above command exits with status 1 after printing
"/path/to/file: Address already in use"
on STDERR.


Unfortunately I have no direct access to the machine where this happens,
only to log files and the source code (of rcp & the program that calls
it). I do have access to another machine with supposedly similar setup
(Redhat 8), but I can't reproduce this problem.

I'm using Redhat 8 w/netkit-rsh-0.17. The user accounts on local &
remote hosts match.

I can't imagine why RCP would ever get EADDRINUSE, doesn't this error
happen on listen() and such, which I don't think is called from rcp at all?

I've looked through the RCP sources and AFAICT the only times it prints
a message like this ("rcp: PATH: ERRNO") are after the calls to 'read'
'open' 'stat' 'opendir' and 'write'. The man pages for these functions
don't mention EADDRINUSE. The numeric value for EADDRINUSE is not used
for any other error code (verified in /usr/include/asm/errno.h).

What the hell is going on?!

Any help would be much appreciated,
D.

Davlet Panech wrote:

> I'm getting "Address already in use" error from the rcp program; why
> would it return that? Here's what happens:
>
> rcp is called from C code via execvp() like so (3 args):
> /bin/sh -c "rcp /path/to/file host:/path/to/file"


I'm confused. Is the code:

execl("/bin/sh", "/bin/sh", "-c", "rcp /path/to/file
host:/path/to/file", NULL);
or
execl("rcp", "rcp", "/path/to/file", "host:/path/to/file", NULL);


(I changed execvp to execl to clarify the example)

Bill Pursell wrote:
> Davlet Panech wrote:
>> rcp is called from C code via execvp() like so (3 args):
>> /bin/sh -c "rcp /path/to/file host:/path/to/file"
>
> I'm confused. Is the code:
>
> execl("/bin/sh", "/bin/sh", "-c", "rcp /path/to/file
> host:/path/to/file", NULL);

Yes like this, through exec* via the shell. BTW the local/remote file
names are the same, so I can't tell whether the error occurs on the
remote host or locally.

D.

Davlet Panech wrote:
> Bill Pursell wrote:
>> I'm confused. Is the code:
>>
>> execl("/bin/sh", "/bin/sh", "-c", "rcp /path/to/file
>> host:/path/to/file", NULL);
>
> Yes like this, through exec* via the shell. BTW the local/remote file
> names are the same, so I can't tell whether the error occurs on the
> remote host or locally.
>
> D.

Just to clarify, the relevant code looks something like this:

char *args[4];
....
args[0] = "sh";
args[1] = "-c";
args[2] = "rcp /path/to/file host:/path/to/file";
args[3] = NULL;
res = execvp (args[0], args);
....

Except args[2] is actually constructed dynamically from pieces w/malloc().

There are no other "sh" or "rcp" programs in the PATH besides the ones I
expect.

D.

Davlet Panech <(E-Mail Removed)> writes:

>Bill Pursell wrote:
>> Davlet Panech wrote:
>>> rcp is called from C code via execvp() like so (3 args):
>>> /bin/sh -c "rcp /path/to/file host:/path/to/file"
>>
>> I'm confused. Is the code:
>>
>> execl("/bin/sh", "/bin/sh", "-c", "rcp /path/to/file
>> host:/path/to/file", NULL);

>Yes like this, through exec* via the shell. BTW the local/remote file
>names are the same, so I can't tell whether the error occurs on the
>remote host or locally.

Well, change them.


>D.

Unruh wrote:
> Davlet Panech <(E-Mail Removed)> writes:
>
>> Yes like this, through exec* via the shell. BTW the local/remote file
>> names are the same, so I can't tell whether the error occurs on the
>> remote host or locally.
>
> Well, change them.
>
Like I said I have no access to the machine where that happens. I'd have
to go through a chain of computer-illiterate users in another country
(remotely via email/telephone) to install an update, try it out (which
involves 2 machines => twice the pain), send the results back to me,
etc; it'll take forever. Besides "rcp" seemed to work fine from the
shell (attached to a terminal that is), but this is second-hand
information... dammit.

Everything works fine in "controlled environment".

Anyway, I was just hoping somebody has seen the EADDRINUSE thing in a
similar context before.

D.

Davlet Panech <(E-Mail Removed)> writes:

>Unruh wrote:
>> Davlet Panech <(E-Mail Removed)> writes:
>>
>>> Yes like this, through exec* via the shell. BTW the local/remote file
>>> names are the same, so I can't tell whether the error occurs on the
>>> remote host or locally.
>>
>> Well, change them.
>>
>Like I said I have no access to the machine where that happens. I'd have

So change the location on the local machine where the stuff is stored.
You sais that you cannot tell because the local and remote file names are
the same. I said change them so they are not the same.

What in the world are you doing using rcp internationally? That comes close
to being criminally irresponsible these days. Use scp or some other secure
system.


>to go through a chain of computer-illiterate users in another country
>(remotely via email/telephone) to install an update, try it out (which
>involves 2 machines => twice the pain), send the results back to me,
>etc; it'll take forever. Besides "rcp" seemed to work fine from the
>shell (attached to a terminal that is), but this is second-hand
>information... dammit.

??? It is taking forever now. It does not work at all. If you do not want
to debug it, then don't.



>Everything works fine in "controlled environment".

>Anyway, I was just hoping somebody has seen the EADDRINUSE thing in a
>similar context before.

>D.

Unruh wrote:
> Davlet Panech <(E-Mail Removed)> writes:
>
>> Unruh wrote:
>> Like I said I have no access to the machine where that happens. I'd have
>
> So change the location on the local machine where the stuff is stored.
> You sais that you cannot tell because the local and remote file names are
> the same. I said change them so they are not the same.
>
> What in the world are you doing using rcp internationally? That comes close
> to being criminally irresponsible these days. Use scp or some other secure
> system.

I'm not using rcp internationally. A remote site in another country has
a private LAN with software that uses rcp on computers on *that* LAN, to
which I have no access. I have a similarly (apparently, not similar
enough though) configured network here at work, but I can't reproduce
this problem. I mean I tried, but it works fine here, no errors,
nothing. The only trace of the problem are log files I got from the
customer + source code.

Yes I know this is a pretty ridiculous way of doing this kind of tech
support, rcp probably shouldn't have been used in the first place, tech
support should have been given some sort of remote access to the
customer's site, blah-blah-blah, but the way things are done here is
*not* up to me; I do what I have to.

D.

Davlet Panech <(E-Mail Removed)> writes:

>Unruh wrote:
>> Davlet Panech <(E-Mail Removed)> writes:
>>
>>> Unruh wrote:
>>> Like I said I have no access to the machine where that happens. I'd have
>>
>> So change the location on the local machine where the stuff is stored.
>> You sais that you cannot tell because the local and remote file names are
>> the same. I said change them so they are not the same.
>>
>> What in the world are you doing using rcp internationally? That comes close
>> to being criminally irresponsible these days. Use scp or some other secure
>> system.

>I'm not using rcp internationally. A remote site in another country has
>a private LAN with software that uses rcp on computers on *that* LAN, to
>which I have no access. I have a similarly (apparently, not similar
>enough though) configured network here at work, but I can't reproduce
>this problem. I mean I tried, but it works fine here, no errors,
>nothing. The only trace of the problem are log files I got from the
>customer + source code.

>Yes I know this is a pretty ridiculous way of doing this kind of tech
>support, rcp probably shouldn't have been used in the first place, tech
>support should have been given some sort of remote access to the
>customer's site, blah-blah-blah, but the way things are done here is
>*not* up to me; I do what I have to.

Ah, OK.

I would still advise the customer to switch to scp and ssh, rather than
rcp.

Does the problem only happen during running of that script/wrapper, or from the
command line as well?
It sounds to me like somehow the command is misinterpreting something as
the remote host.

I guess this procedure has the advantage that it multiplies the support
costs by a factor of 10 or 100 as you have to spend more time figuring out
what is going on. (Or did you enter into a fixed price contract.)

You might also have them run an strace on the program to see what is
actually being handed off in the exec call.



>D.

rcp is restricted to a subset of the "priviledged" port range. I
think it is limited to 512 to 1024 but I'm not entirely certain. If,
bewteen two IP addresses, one tries to run more than
512/lengthof(TIME_WAIT) rcp's (or rexec's etc) per second then there
will be an attempt to reuse a four-tuple of local/remote IP,
local/remote port number while that four-tuple (ie TCP connection
name) is still in TIME_WAIT. IIRC this can result in an EADDRINUSE
error on the client side.

You can check for this by looking at netstat -an output and seeing how
many TIME_WAITs there are for the remote port for rexec/rcp (iirc) on
the client system.

While it is indeed possible for a TCP connection to transition from
TIME_WAIT to established, this depends on the Initial Sequence Numbers
(ISN) for the new connection being in the "right range" relative to
where the connection left-off before. With everyone wanting fully
randomized ISNs the chances of that happening become rather small.

The possible fixes for this, in no particular order:

*) reduce the rate of rcp's - perhaps archive several files together
via some mechanism and transfer the archive

*) increase the number of IP addresses involved

*) use a file copy mechanism that isn't restricted to the priviledged
port range

I would not suggest messing with the length of TIME_WAIT, nor would I
suggest trying to assasinate the TIME_WAIT states. They are there as
part of TCP's protections against data corruption.

rick jones
--
Process shall set you free from the need for rational thought.
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...