ghost tcp/udp LISTEN ports

After installing a linux box, divx, i came across some weird open
ghost ports :

[divx:root]~)# netstat -ltunp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:32769 0.0.0.0:* LISTEN -

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 5745/sshd

tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 5914/0

udp 0 0 0.0.0.0:32768 0.0.0.0:* -

udp 0 0 0.0.0.0:800 0.0.0.0:* -

[divx:root]~)# lsof -i TCP:32769
[divx:root]~)# lsof -i UDP:32768
[divx:root]~)# lsof -i UDP:800
[divx:root]~)# lsof -i TCP:22
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
sshd 5745 root 3u IPv4 40365 TCP *:ssh (LISTEN)
sshd 5914 root 3u IPv4 40619 TCP divx.stokkie.net:ssh->jackson.stokkie.net:32913 (
ESTABLISHED)
[divx:root]~)# lsof -i TCP:6010
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
sshd 5914 root 6u IPv4 40638 TCP divx.stokkie.net:x11-ssh-offset (LISTEN)
[divx:root]~)#

What are these open ports which lsof reports nothing about? The TCP/32769
is for real :

[divx:root]~)# telnet divx 32769
Trying 127.0.0.1...
Connected to divx.stokkie.net (127.0.0.1).
Escape character is '^]'.
HELLO?
Connection closed by foreign host.
[divx:root]~)#

Anyone?
--
Robert M. Stockmann - RHCE
Network Engineer - UNIX/Linux Specialist
crashrecovery.org (E-Mail Removed)

On Sat, 31 Mar 2007 03:20:07 +0200, Robert M. Stockmann wrote:

> After installing a linux box, divx, i came across some weird open
> ghost ports :
>
> [divx:root]~)# netstat -ltunp
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
>
> tcp 0 0 0.0.0.0:32769 0.0.0.0:* LISTEN -
>
> tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 5745/sshd
>
> tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 5914/0
>
> udp 0 0 0.0.0.0:32768 0.0.0.0:* -
>
> udp 0 0 0.0.0.0:800 0.0.0.0:* -
>
> [divx:root]~)# lsof -i TCP:32769
> [divx:root]~)# lsof -i UDP:32768
> [divx:root]~)# lsof -i UDP:800
> [divx:root]~)# lsof -i TCP:22
> COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
> sshd 5745 root 3u IPv4 40365 TCP *:ssh (LISTEN)
> sshd 5914 root 3u IPv4 40619 TCP divx.stokkie.net:ssh->jackson.stokkie.net:32913 (
> ESTABLISHED)
> [divx:root]~)# lsof -i TCP:6010
> COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
> sshd 5914 root 6u IPv4 40638 TCP divx.stokkie.net:x11-ssh-offset (LISTEN)
> [divx:root]~)#
>
> What are these open ports which lsof reports nothing about? The TCP/32769
> is for real :
>
> [divx:root]~)# telnet divx 32769
> Trying 127.0.0.1...
> Connected to divx.stokkie.net (127.0.0.1).
> Escape character is '^]'.
> HELLO?
> Connection closed by foreign host.
> [divx:root]~)#
>
> Anyone?

Well the problem turns out to have been a kernel bug :

[bigpapa:root]~)# netstat -ltunp | grep "-"
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
[bigpapa:root]~)# uname -r
2.4.32
[bigpapa:root]~)#

[hubble:root]~)# netstat -ltunp | grep "-"
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
udp 0 0 0.0.0.0:2049 0.0.0.0:* -
udp 0 0 0.0.0.0:32772 0.0.0.0:* -
[hubble:root]~)# uname -r
2.4.26
[hubble:root]~)#

It seems that with kernel 2.4.30 or higher, the ghost LISTEN ports are
gone. i have a two boxes which run 2.6.7 and 2.6.12 :

[wikiwork:root]~)# netstat -ltunp | grep " - "
tcp 0 0 0.0.0.0:32769 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN -
udp 0 0 0.0.0.0:32768 0.0.0.0:* -
udp 0 0 0.0.0.0:2049 0.0.0.0:* -
[wikiwork:root]~)# uname -r
2.6.12
[wikiwork:root]~)#

[jackson:root]~)# netstat -ltunp | grep " - "
tcp 0 0 0.0.0.0:32768 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN -
udp 0 0 0.0.0.0:2049 0.0.0.0:* -
udp 0 0 0.0.0.0:32770 0.0.0.0:* -
udp 0 0 0.0.0.0:799 0.0.0.0:* -
[jackson:root]~)# uname -r
2.6.7
[jackson:root]~)#

which 2.6.xx kernel and solves the above bug?

--
Robert M. Stockmann - RHCE
Network Engineer - UNIX/Linux Specialist
crashrecovery.org (E-Mail Removed)

On Sat, 31 Mar 2007 03:50:47 +0200, Robert M. Stockmann wrote:

> [wikiwork:root]~)# netstat -ltunp | grep " - "
> tcp 0 0 0.0.0.0:32769 0.0.0.0:* LISTEN -
> tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN -
> udp 0 0 0.0.0.0:32768 0.0.0.0:* -
> udp 0 0 0.0.0.0:2049 0.0.0.0:* -
> [wikiwork:root]~)# uname -r
> 2.6.12
> [wikiwork:root]~)#
>
> [jackson:root]~)# netstat -ltunp | grep " - "
> tcp 0 0 0.0.0.0:32768 0.0.0.0:* LISTEN -
> tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN -
> udp 0 0 0.0.0.0:2049 0.0.0.0:* -
> udp 0 0 0.0.0.0:32770 0.0.0.0:* -
> udp 0 0 0.0.0.0:799 0.0.0.0:* -
> [jackson:root]~)# uname -r
> 2.6.7
> [jackson:root]~)#
>
> which 2.6.xx kernel and solves the above bug?

[jackson:root]~)# netstat -ltunp | grep " - "
tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:60269 0.0.0.0:* LISTEN -
udp 0 0 0.0.0.0:2049 0.0.0.0:* -
udp 0 0 0.0.0.0:32770 0.0.0.0:* -
[jackson:root]~)# uname -r
2.6.17
[jackson:root]~)#

well...is this a kernel bug? or a kernel config bug?
--
Robert M. Stockmann - RHCE
Network Engineer - UNIX/Linux Specialist
crashrecovery.org (E-Mail Removed)

On Sat, 31 Mar 2007 07:50:20 +0200, Robert M. Stockmann wrote:

> On Sat, 31 Mar 2007 03:50:47 +0200, Robert M. Stockmann wrote:
>
>> [wikiwork:root]~)# netstat -ltunp | grep " - " tcp 0 0
>> 0.0.0.0:32769 0.0.0.0:* LISTEN -
>> tcp 0 0 0.0.0.0:2049 0.0.0.0:*
>> LISTEN - udp 0 0 0.0.0.0:32768
>> 0.0.0.0:* - udp 0 0
>> 0.0.0.0:2049 0.0.0.0:* -
>> [wikiwork:root]~)# uname -r
>> 2.6.12
>> [wikiwork:root]~)#
>>
>> [jackson:root]~)# netstat -ltunp | grep " - " tcp 0 0
>> 0.0.0.0:32768 0.0.0.0:* LISTEN - tcp
>> 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN -
>> udp 0 0 0.0.0.0:2049 0.0.0.0:*
>> - udp 0 0 0.0.0.0:32770 0.0.0.0:*
>> - udp 0 0 0.0.0.0:799 0.0.0.0:*
>> - [jackson:root]~)# uname -r
>> 2.6.7
>> [jackson:root]~)#
>>
>> which 2.6.xx kernel and solves the above bug?
>
> [jackson:root]~)# netstat -ltunp | grep " - " tcp 0 0
> 0.0.0.0:2049 0.0.0.0:* LISTEN - tcp
> 0 0 0.0.0.0:60269 0.0.0.0:*
> LISTEN - udp 0 0 0.0.0.0:2049 0.0.0.0:*
> - udp 0 0 0.0.0.0:32770
> 0.0.0.0:* - [jackson:root]~)# uname -r
> 2.6.17
> [jackson:root]~)#
>
> well...is this a kernel bug? or a kernel config bug?

Many of those look like the sort of port numbers RPC services would use,
and 2049 is NFS. What does "rpcinfo -p" have to say on the affected
machines?

Regards, Ian

On Sat, 31 Mar 2007 16:06:20 +0100, Ian Northeast wrote:

> On Sat, 31 Mar 2007 07:50:20 +0200, Robert M. Stockmann wrote:
>
>> On Sat, 31 Mar 2007 03:50:47 +0200, Robert M. Stockmann wrote:
>>
>>> [wikiwork:root]~)# netstat -ltunp | grep " - " tcp 0 0
>>> 0.0.0.0:32769 0.0.0.0:* LISTEN -
>>> tcp 0 0 0.0.0.0:2049 0.0.0.0:*
>>> LISTEN - udp 0 0 0.0.0.0:32768
>>> 0.0.0.0:* - udp 0 0
>>> 0.0.0.0:2049 0.0.0.0:* -
>>> [wikiwork:root]~)# uname -r
>>> 2.6.12
>>> [wikiwork:root]~)#
>>>
>>> [jackson:root]~)# netstat -ltunp | grep " - " tcp 0 0
>>> 0.0.0.0:32768 0.0.0.0:* LISTEN - tcp 0
>>> 0 0.0.0.0:2049 0.0.0.0:* LISTEN - udp
>>> 0 0 0.0.0.0:2049 0.0.0.0:*
>>> - udp 0 0 0.0.0.0:32770 0.0.0.0:*
>>> - udp 0 0 0.0.0.0:799 0.0.0.0:*
>>> - [jackson:root]~)# uname -r
>>> 2.6.7
>>> [jackson:root]~)#
>>>
>>> which 2.6.xx kernel and solves the above bug?
>>
>> [jackson:root]~)# netstat -ltunp | grep " - " tcp 0 0
>> 0.0.0.0:2049 0.0.0.0:* LISTEN -
>> tcp
>> 0 0 0.0.0.0:60269 0.0.0.0:*
>> LISTEN - udp 0 0 0.0.0.0:2049 0.0.0.0:*
>> - udp 0 0 0.0.0.0:32770
>> 0.0.0.0:* - [jackson:root]~)# uname -r
>> 2.6.17
>> [jackson:root]~)#
>>
>> well...is this a kernel bug? or a kernel config bug?
>
> Many of those look like the sort of port numbers RPC services would use,
> and 2049 is NFS. What does "rpcinfo -p" have to say on the affected
> machines?

It's also worth getting the latest source for lsof from
ftp://vic.cc.purdue.edu/pub/tools/unix/lsof and building it on the machine
you are running it on. Lsof is very sensitive to differences between the
build and execution environments and can report incorrect results if they
don't match closely enough, especially IME in the network area. So if
you've upgraded things it could be getting it wrong.

Regards, Ian