Has my router failed?

Gday all, as my first post here I hope this isnt too open a question:

I have a modem router with 4 port switch. Ive not changed the config on
it, it is identical to when my server was working fine. (Server is a
Fedora Core 5 (was 4) setup with latest apache/php/mysql)

I recently had a drive failure and installed core 5 instead. Apache is
working fine from the inside of the network but from outside it has,
all of a sudden, stopped accepting connections on any port (SSH, HTTP
etc).

I figured the firewall was wrongly configured so i flushed it to empty.
No luck.I disabled SELinux. Rebooted the router/modem. No luck.
Re-configured the port forwarding. No change. Scanning the ports with
one of the many websites that do it free show port 80 open and unsecure
and all others except FTP closed and unsecure.

I'm at a loss to be honest. For now ive just blocked all 'net access to
the server but im really frustrated!

Any diagnostic pointers/tutorials that might help me identify it,
without the expense of a new router/modem?

Thanks!
J

On 3 Oct 2006, in the Usenet newsgroup comp.os.linux.networking, in article
<(E-Mail Removed). com>,
(E-Mail Removed) wrote:

>I have a modem router with 4 port switch. Ive not changed the config on
>it, it is identical to when my server was working fine. (Server is a
>Fedora Core 5 (was 4) setup with latest apache/php/mysql)

OK, I think

>I recently had a drive failure and installed core 5 instead. Apache is
>working fine from the inside of the network but from outside it has,
>all of a sudden, stopped accepting connections on any port (SSH, HTTP
>etc).

"working fine from the inside" - meaning another system connected to the
four-port switch, and not from the FC5 server (difference - network via
Ethernet verses using loopback). Verify this using 'netstat -tupan' and
see that the "Local Address" is either "0.0.0.0" (listening on all network
interfaces) or "192.168.1.1" (or whatever the address of the eth0 interface
is), and not "127.0.0.1" (I'm only listening to myself on the loopback).

What error message do you get on the "outside" computer? What shows up
in the logs on the server (or better, what does a network sniffer like
tcpdump show)?

>I figured the firewall was wrongly configured so i flushed it to empty.
>No luck.I disabled SELinux. Rebooted the router/modem. No luck.
>Re-configured the port forwarding. No change.

If you know how to configure your modem/router, that should rule it out
as a problem source.

>Scanning the ports with one of the many websites that do it free show
>port 80 open and unsecure and all others except FTP closed and unsecure.

Sounds like that incompetent idiot Steve Gibson with his "Shields Up"
site. A port that is "closed" is secure, because there is nothing there
to exploit. That useless wanker likes to promote "stealth" (firewall
set to "DROP" rather than "REJECT") because he thinks that "hides" your
computer. All it really shows is that he has no concept of how IP works,
and specifically the ICMP Type 3 Code 1 (Host Unreachable) message sent
by the upstream router when your host really doesn't exist. A traceroute
would show this, but he's to busy selling crap to notice this.

>I'm at a loss to be honest. For now ive just blocked all 'net access to
>the server but im really frustrated!

Blocked all on the router. Use another system locally and see what
happens when you try to connect from the same local network. Is
everything working as expected? Then repeat from "outside".

It is _possible_ that your ISP added filters by coincidence at the same
time you replaced your O/S (though I doubt this), and one way to test this
is to use a packet sniffer on the local AND remote systems and see what
packets are going where. A tool that I would use is an application that
does a traceroute but using TCP packets rather than ICMP or UDP.

[compton ~]$ whatis hping2 tcptraceroute
hping2 (8) - send (almost) arbitrary TCP/IP packets to network hosts
tcptraceroute (8) - A traceroute implementation using TCP packets
[compton ~]$

The regular 'traceroute' or the crippled windoze TRACERT would not be
useful here.

Old guy

Moe Trin wrote:
> On 3 Oct 2006, in the Usenet newsgroup comp.os.linux.networking, in article
<snip very helpful info>
> Old guy

Wow, thats just about the best explanation and help Ive ever been given
on any subject! Thanks! I'll give everything you suggest a try when i
get back to the server tonight. tcpdump revealed nothing that stood
out to me - I'll copy results over later if i still cant figure it out,
gave that a try, but i didnt check the loopback setup i must admit.

Thanks again, I'll be back!
James

Moe Trin wrote:
> On 3 Oct 2006, in the Usenet newsgroup comp.os.linux.networking, in article
> <(E-Mail Removed). com>,
> (E-Mail Removed) wrote:
> OK, I think

Basically I was trying to imply exactly the same setup as i always had,
software-wise, obviously a config mistake somewhere.
>
> "working fine from the inside" - meaning another system connected to the
> four-port switch, and not from the FC5 server (difference - network via
> Ethernet verses using loopback). Verify this using 'netstat -tupan' and
> see that the "Local Address" is either "0.0.0.0" (listening on all network
> interfaces) or "192.168.1.1" (or whatever the address of the eth0 interface
> is), and not "127.0.0.1" (I'm only listening to myself on the loopback).
>

>From the server (10.0.0.S) i can access the website. From any other
10.0.0.X i can also access it. From 138.253.X.X I cant.
netstat -tupan | grep httpd > /results.txt gives (hope this comes out
OK,trimmed it a bit):

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:*
LISTEN 1615/portmap
tcp 0 0 127.0.0.1:50000 0.0.0.0:*
LISTEN 1803/hpiod
tcp 0 0 127.0.0.1:50002 0.0.0.0:*
LISTEN 1808/python
tcp 0 0 0.0.0.0:55795 0.0.0.0:*
LISTEN 1634/rpc.statd
tcp 0 0 0.0.0.0:631 0.0.0.0:*
LISTEN 1819/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:*
LISTEN 2017/sendmail: acce
tcp 0 0 0.0.0.0:445 0.0.0.0:*
LISTEN 2106/smbd
tcp 0 0 10.0.0.1:139 10.0.0.10:2090
ESTABLISHED 10384/smbd
tcp 0 0 :::993 :::*
LISTEN 10116/dovecot
tcp 0 0 :::995 :::*
LISTEN 10116/dovecot
tcp 0 0 :::110 :::*
LISTEN 10116/dovecot
tcp 0 0 :::143 :::*
LISTEN 10116/dovecot
tcp 0 0 :::80 :::*
LISTEN 2048/httpd
tcp 0 0 :::22 :::*
LISTEN 1850/sshd
tcp 0 0 :::631 :::*
LISTEN 1819/cupsd
tcp 0 0 :::443 :::*
LISTEN 2048/httpd
udp 0 0 10.0.0.255:123 0.0.0.0:*
1867/ntpd
udp 0 0 10.0.0.1:123 0.0.0.0:*
1867/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0:*
1867/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:*
1867/ntpd

So httpd is listening on ::::80

I think this is the key, as the problems im having occur with dovecot,
SSH and http (and printing but i hadnt realised that until that showed
up and i tested it)

Wheres my fault? Have i specified something incorrectly in the
configs?? I'll check through and see if i have any more news by the
time i hear back from you. Thanks for your help so far!!

On 5 Oct 2006, in the Usenet newsgroup comp.os.linux.networking, in article
<(E-Mail Removed). com>,
(E-Mail Removed) wrote:

>Moe Trin wrote:

>> OK, I think
>
>Basically I was trying to imply exactly the same setup as i always had,
>software-wise, obviously a config mistake somewhere.

Yeah, but it's not on the router.

>From the server (10.0.0.S) i can access the website. From any other
>10.0.0.X i can also access it. From 138.253.X.X I cant.
>netstat -tupan | grep httpd > /results.txt gives (hope this comes out
>OK,trimmed it a bit):

Well, your 'grep' failed for some reason, but that's good. I'll trim it a
little more (got rid of the Recv-Q Send-Q columns). I can see a problem.

>Active Internet connections (servers and established)
>Proto Local Address Foreign Address State PID/Program name
>tcp 0.0.0.0:111 0.0.0.0:* LISTEN 1615/portmap

Do you need portmap? Most people don't.

>tcp 127.0.0.1:50000 0.0.0.0:* LISTEN 1803/hpiod
>tcp 127.0.0.1:50002 0.0.0.0:* LISTEN 1808/python
>tcp 0.0.0.0:55795 0.0.0.0:* LISTEN 1634/rpc.statd

Same thing for rpc.statd Most people I work with don't need this. The
other problem is that both are accepting connections from the world. This
probably isn't a security problem for you, as your router probably isn't
forwarding packets to this port.

>tcp 0.0.0.0:631 0.0.0.0:* LISTEN 1819/cupsd

Again, I don't like to let everyone on the Internet use my printer, but
maybe I'm just not sociable. ;-)

>tcp 127.0.0.1:25 0.0.0.0:* LISTEN 2017/sendmail: acce
>tcp 0.0.0.0:445 0.0.0.0:* LISTEN 2106/smbd

It would be nicer to restrict this to 10.0.0.1, but that's just me. Actually
I don't even have any windoze boxes, so this wouldn't be a problem anyway.

>tcp 10.0.0.1:139 10.0.0.10:2090 ESTABLISHED 10384/smbd

Samba - looks fine. But here starts the problems.

>tcp :::993 :::* LISTEN 10116/dovecot
>tcp :::995 :::* LISTEN 10116/dovecot
>tcp :::110 :::* LISTEN 10116/dovecot
>tcp :::143 :::* LISTEN 10116/dovecot
>tcp :::80 :::* LISTEN 2048/httpd
>tcp :::22 :::* LISTEN 1850/sshd
>tcp :::631 :::* LISTEN 1819/cupsd
>tcp :::443 :::* LISTEN 2048/httpd

Notice the difference in the address formats. This is IPv6, not the IPv4
with the addresses of four "dotted quads" like "10.20.30.40". Without
seeing what output of '/sbin/ifconfig -eth0', I can't tell what address
you are listening to, but it's likely something that vaguely looks like
fe80::211:2fff:fe68:ee9a.

>udp 10.0.0.255:123 0.0.0.0:* 1867/ntpd
>udp 10.0.0.1:123 0.0.0.0:* 1867/ntpd
>udp 127.0.0.1:123 0.0.0.0:* 1867/ntpd
>udp 0.0.0.0:123 0.0.0.0:* 1867/ntpd

Network time protocol listening four different ways on IPv4.

>So httpd is listening on ::::80

which is IPv6, instead of listening on 0.0.0.0:80 - that's your problem.

>I think this is the key, as the problems im having occur with dovecot,
>SSH and http (and printing but i hadnt realised that until that showed
>up and i tested it)

Yes, this is the key. I suspect your ISP isn't giving you an IPv6
connection (even though GB has a bunch of allocations), never mind that
your router _PROBABLY_ isn't aware of IPv6. This might also explain your
external test - your router is forwarding the packets to 10.0.0.1:80, but
there is nobody listening to that port. If the ICMP Type 3 Code 3 "nobody
home" error message from your server is getting blocked/dropped someplace,
the remote thinks the server is there, but not responding.

>Wheres my fault? Have i specified something incorrectly in the
>configs?? I'll check through and see if i have any more news by the
>time i hear back from you. Thanks for your help so far!!

Either, you are blocking IPv4 on the server (you said the firewall wasn't
doing that), or you have not told the daemons to listen to IPv4 instead of
(or in addition to) IPv6.

Hope this puts you on the right track.

Old guy

Hello,

Moe Trin a écrit :
>
>>tcp :::80 :::* LISTEN 2048/httpd
>>tcp :::443 :::* LISTEN 2048/httpd
>
> Notice the difference in the address formats. This is IPv6, not the IPv4
> with the addresses of four "dotted quads" like "10.20.30.40". Without
> seeing what output of '/sbin/ifconfig -eth0', I can't tell what address
> you are listening to,

"::" is the short representation for 0:0:0:0:0:0:0:0, the IPv6
equivalent of IPv4 0.0.0.0 which means ANY address.
So these IPv6 sockets are listening on ANY local IPv6 address, but also
probably on ANY local IPv4 address because the default in Linux
(/proc/sys/net/ipv6/bindv6only=0) is that IPv6 sockets can be used for
IPv4 communications with IPv4-mapped IPv6 addresses. Then the IPv4
address 10.20.30.40 will are seen as ::ffff:10.20.30.40 by the socket
and the application, which may disrupt ACLs or other address-based
functions such as virtual hosts.

This behaviour can be changed at system level by setting
/proc/sys/net/ipv6/bindv6only to 0, so new IPv6 sockets can only send
and listen to genuine IPv6 communications.

Other options are :
1) At system level, disable IPv6 (disable the ipv6 module or build a
kernel without IPv6 support).
2) At application level, set the "listen" option (or whatever name it
has) to 0.0.0.0 or the desired local IPv4 address(es).

For instance, in /etc/ssh/sshd_config :

# listen only on ANY IPv4 addresses, not IPv6.
#ListenAddress ::
ListenAddress 0.0.0.0

> but it's likely something that vaguely looks like
> fe80::211:2fff:fe68:ee9a.

No, this is a link-layer IPv6 address that TCP sockets cannot bind to.

[Supersedes]

Hello,

Moe Trin a écrit :
>
>>tcp :::80 :::* LISTEN 2048/httpd
>>tcp :::443 :::* LISTEN 2048/httpd
>
> Notice the difference in the address formats. This is IPv6, not the IPv4
> with the addresses of four "dotted quads" like "10.20.30.40". Without
> seeing what output of '/sbin/ifconfig -eth0', I can't tell what address
> you are listening to,

"::" is the short representation for 0:0:0:0:0:0:0:0, the IPv6
equivalent of IPv4 0.0.0.0 which means ANY address.
So these IPv6 sockets are listening on ANY local IPv6 address, but also
probably on ANY local IPv4 address because the default in Linux
(/proc/sys/net/ipv6/bindv6only=0) is that IPv6 sockets can be used for
IPv4 communications with IPv4-mapped IPv6 addresses. Then the IPv4
address 10.20.30.40 will are seen as ::ffff:10.20.30.40 by the socket
and the application, which may disrupt ACLs or other address-based
functions such as virtual hosts.

This behaviour can be changed at system level by setting
/proc/sys/net/ipv6/bindv6only to 1, so new IPv6 sockets can only send
and listen to genuine IPv6 communications.

Other options are :
1) At system level, disable IPv6 (disable the ipv6 module or build a
kernel without IPv6 support).
2) At application level, set the "listen" option (or whatever name it
has) to 0.0.0.0 or the desired local IPv4 address(es).

For instance, in /etc/ssh/sshd_config :

# listen only on ANY IPv4 addresses, not IPv6.
#ListenAddress ::
ListenAddress 0.0.0.0

> but it's likely something that vaguely looks like
> fe80::211:2fff:fe68:ee9a.

No, this is a link-layer IPv6 address that TCP sockets cannot bind to.

Moe Trin wrote:
> Well, your 'grep' failed for some reason, but that's good. I'll trim it a
> little more (got rid of the Recv-Q Send-Q columns). I can see a problem.

:-) I decided to show the full results and forgot to alter the post. It
was late, many coffees, stress...!

> Do you need portmap? Most people don't.

No, I'd re-enabled all services I could remember disabling "just in
case".

> Same thing for rpc.statd Most people I work with don't need this. The
> other problem is that both are accepting connections from the world. This
> probably isn't a security problem for you, as your router probably isn't
> forwarding packets to this port.

Noted, thanks.

> >tcp 0.0.0.0:631 0.0.0.0:* LISTEN 1819/cupsd
>
> Again, I don't like to let everyone on the Internet use my printer, but
> maybe I'm just not sociable. ;-)

LOL <embarrassed> oops, well spotted. Mind you, ive had no unusual
printouts recently lol.
>
> >tcp :::993 :::* LISTEN 10116/dovecot
> >tcp :::995 :::* LISTEN 10116/dovecot
> >tcp :::110 :::* LISTEN 10116/dovecot
> >tcp :::143 :::* LISTEN 10116/dovecot
> >tcp :::80 :::* LISTEN 2048/httpd
> >tcp :::22 :::* LISTEN 1850/sshd
> >tcp :::631 :::* LISTEN 1819/cupsd
> >tcp :::443 :::* LISTEN 2048/httpd
>
> Notice the difference in the address formats. This is IPv6, not the IPv4
> with the addresses of four "dotted quads" like "10.20.30.40".

Ahhh now i understand.

> seeing what output of '/sbin/ifconfig -eth0', I can't tell what address
> you are listening to, but it's likely something that vaguely looks like
> fe80::211:2fff:fe68:ee9a.

I can find that out later

> Either, you are blocking IPv4 on the server (you said the firewall wasn't
> doing that), or you have not told the daemons to listen to IPv4 instead of
> (or in addition to) IPv6.
>
> Hope this puts you on the right track.
>
> Old guy

Certainly does give me a clearer view of whats going on. Its really
handy to have a step through like that - something I can never find
online. Generally all i need is a pointer at what things *mean* and i
can figure out the logic behind them and fix it. Networking seems an
exception to this rule but im learning thanks to people like yourself
and many nights pulling my hair out!

Thanks

Pascal Hambourg wrote:
> [Supersedes]
>
> "::" is the short representation for 0:0:0:0:0:0:0:0, the IPv6
> equivalent of IPv4 0.0.0.0 which means ANY address.
> So these IPv6 sockets are listening on ANY local IPv6 address, but also
> probably on ANY local IPv4 address because the default in Linux
> (/proc/sys/net/ipv6/bindv6only=0) is that IPv6 sockets can be used for
> IPv4 communications with IPv4-mapped IPv6 addresses. Then the IPv4
> address 10.20.30.40 will are seen as ::ffff:10.20.30.40 by the socket
> and the application, which may disrupt ACLs or other address-based
> functions such as virtual hosts.

Ahah, im starting to get it - its filtering into my brain!

> This behaviour can be changed at system level by setting
> /proc/sys/net/ipv6/bindv6only to 1, so new IPv6 sockets can only send
> and listen to genuine IPv6 communications.

I'll check that tonight.

>
> Other options are :
> 1) At system level, disable IPv6 (disable the ipv6 module or build a
> kernel without IPv6 support).

I had an IPv6 capable kernel before but im not sure on its config so
I'll start with the first suggestion as a test.

> 2) At application level, set the "listen" option (or whatever name it
> has) to 0.0.0.0 or the desired local IPv4 address(es)......
> ....ListenAddress 0.0.0.0

Right, I'll double check what format its in. I know the httpd.conf has
just " :80 " for its listen address, maybe the source of confusion but
it worked on the last system.

> > but it's likely something that vaguely looks like
> > fe80::211:2fff:fe68:ee9a.
>
> No, this is a link-layer IPv6 address that TCP sockets cannot bind to.

:s whoosh (over my head for now, I'll keep reading lol)

Thanks for the overview and putting the time in for me, im sure it'll
help me figure it out. I'll keep you all posted when i try tonight!

J
J

(E-Mail Removed) a écrit :
>
>>This behaviour can be changed at system level by setting
>>/proc/sys/net/ipv6/bindv6only to 1, so new IPv6 sockets can only send
>>and listen to genuine IPv6 communications.

However, existing IPv6 sockets remain unaffected, so you need to
reload/restart the apps.


>>2) At application level, set the "listen" option (or whatever name it
>>has) to 0.0.0.0 or the desired local IPv4 address(es)......
>>....ListenAddress 0.0.0.0

And, I forgot to mention : disable listening on :: (IPv6 ANY), because
with bindv6only=0, a TCP IPv6 socket listening on :: (actually :: and
0.0.0.0) prevents a TCP IPv4 socket from listening on the same port. So,
if there can be multiple "listen" options (e.g. sshd), make sure there
is none with ::. If there is a special option to enable listening on
IPv6, disable it (e.g. listen-on-v6 for bind9).

> Right, I'll double check what format its in. I know the httpd.conf has
> just " :80 " for its listen address, maybe the source of confusion but
> it worked on the last system.

Maybe the apache version included in the previous system did not have
IPv6 capabilities. However it *should* work anyway. Proof is you can
reach the web server from the localhost and local network, and I don't
think your LAN uses IPv6 - or you would know. The difference is you
would see source addresses written in a weird form (actually
IPv4-mapped) in the logs.

>>>but it's likely something that vaguely looks like
>>>fe80::211:2fff:fe68:ee9a.
>>
>>No, this is a link-layer IPv6 address that TCP sockets cannot bind to.
>
> :s whoosh (over my head for now, I'll keep reading lol)

You do not need to bother about that. IPv6 link-local addresses
(fe80::/10) are used only for low level services such as some ICMPv6
signalling including router solicitation and discovery and the mechanism
that replaces ARP in IPv6. They cannot be used for higher level services
such as TCP communications.