Could an ICMP Redirect have disconnected my server?

First, my questions, then explanation. Any help would be appreciated.

Does Linux (2.4.x) act on ICMP Redirect packets by default?
If so, can an ICMP Redirect override a static default route?
If so, does a routing table entry from an ICMP Redirect time out?

I have this Linux server that went mostly off-line suddenly today,
disconnecting a number of database users and such. The server is on an
intranet, private static IP address and one default route to a internal
router. (The only odd thing is that there are multiple logical subnets on
the same physical subnet.) When it dropped all those connections, it was
still reachable from, and could still reach, systems with the same subnet
number. Unfortunately, I didn't realize that at the time - I found two
systems that could still reach it, but I didn't make the subnet connection.
So I didn't check the routing table until later. About 90 minutes after it
dropped off, it came back up; nobody did anything to it - it just started
taking to the network normally again.

Trying to figure out what happened, I was wondering if a 'rogue' ICMP
redirect could cause this. Is this possible?

ljb wrote:
> First, my questions, then explanation. Any help would be appreciated.
>
> Does Linux (2.4.x) act on ICMP Redirect packets by default?
> If so, can an ICMP Redirect override a static default route?
> If so, does a routing table entry from an ICMP Redirect time out?
>
> I have this Linux server that went mostly off-line suddenly today,
> disconnecting a number of database users and such. The server is on an
> intranet, private static IP address and one default route to a internal
> router. (The only odd thing is that there are multiple logical subnets on
> the same physical subnet.) When it dropped all those connections, it was
> still reachable from, and could still reach, systems with the same subnet
> number. Unfortunately, I didn't realize that at the time - I found two
> systems that could still reach it, but I didn't make the subnet connection.
> So I didn't check the routing table until later. About 90 minutes after it
> dropped off, it came back up; nobody did anything to it - it just started
> taking to the network normally again.
>
> Trying to figure out what happened, I was wondering if a 'rogue' ICMP
> redirect could cause this. Is this possible?

Hi,

From my personnal LAB experience I did a few weeks ago, I found out
that my Linux box (FC6) did not accept ICMP Redirect by default (My
WinXP did tought).

And there is a timeout on learned routes via ICMP redirect, it is 10
minutes.

From the testing I did in lab, forging packets, I was only able to send
ICMP REDIRECTS for HOSTs only, not complete subnets (but that is from
limited experience, maybe it feasable, but I was not able to). If you
prefer, I could only send ICMP REDIRECT for routes with a /32 Mask, a host.

Hope this helps in anyway.

PS: If you wanna try to reproduce the problem, try using linux Excalubur
Packet forger, Works good, nice study tool.

On Thu, 30 Aug 2007, in the Usenet newsgroup comp.os.linux.networking, in
article <(E-Mail Removed)>, ljb wrote:

>Does Linux (2.4.x) act on ICMP Redirect packets by default?

Under limited conditions, yes - but you seem to have a strange network
layout which might confuse things. 2.4.x? Which 2.4.x? 2.4.31.2 is the
latest (released about two weeks ago).

>If so, can an ICMP Redirect override a static default route?
>If so, does a routing table entry from an ICMP Redirect time out?

1122 Requirements for Internet Hosts - Communication Layers. R.
Braden, Ed.. October 1989. (Format: TXT=295992 bytes) (Updated by
RFC1349, RFC4379) (Also STD0003) (Status: STANDARD)

1812 Requirements for IP Version 4 Routers. F. Baker, Ed.. June 1995.
(Format: TXT=415740 bytes) (Obsoletes RFC1716, RFC1009) (Updated by
RFC2644) (Status: PROPOSED STANDARD)

A quick scan of those two, I don't see a timeout.

>I have this Linux server that went mostly off-line suddenly today,
>disconnecting a number of database users and such. The server is on an
>intranet, private static IP address and one default route to a internal
>router. (The only odd thing is that there are multiple logical subnets on
>the same physical subnet.)

Usually, having logical subnets on the same physical wire is a bad idea.

>When it dropped all those connections, it was still reachable from, and
>could still reach, systems with the same subnet number.

Logical or physical?

>Unfortunately, I didn't realize that at the time - I found two systems
>that could still reach it, but I didn't make the subnet connection. So I
>didn't check the routing table until later.

Is syslogd configured to log any routing information?

>About 90 minutes after it dropped off, it came back up; nobody did
>anything to it - it just started taking to the network normally again.

Assuming you are not using a routing daemon like routed or gated, a
redirect should stick until networking is restarted (clearing and
reloading the routing table). The only way to change the table would
be an ICMP Redirect (which a distribution-standard kernel should be
ignoring except under very limited conditions).

>Trying to figure out what happened, I was wondering if a 'rogue' ICMP
>redirect could cause this. Is this possible?

Depends on your network layout. Not very likely, but not impossible.

Old guy

(E-Mail Removed) wrote:
>...
>> Trying to figure out what happened, I was wondering if a 'rogue' ICMP
>> redirect could cause this. Is this possible?
>
> Hi,
>
> From my personnal LAB experience I did a few weeks ago, I found out
> that my Linux box (FC6) did not accept ICMP Redirect by default (My
> WinXP did tought).
>
> And there is a timeout on learned routes via ICMP redirect, it is 10
> minutes.
>
> From the testing I did in lab, forging packets, I was only able to send
> ICMP REDIRECTS for HOSTs only, not complete subnets (but that is from
> limited experience, maybe it feasable, but I was not able to). If you
> prefer, I could only send ICMP REDIRECT for routes with a /32 Mask, a host.
>
> Hope this helps in anyway.
>
> PS: If you wanna try to reproduce the problem, try using linux Excalubur
> Packet forger, Works good, nice study tool.

Thanks, it does help. I don't have a setup to do this sort of testing.
It would be scary to think that an ICMP packet could override the default
route. But I'm no closer to figuring out what happened, either.

(E-Mail Removed) wrote:
> On Thu, 30 Aug 2007, in the Usenet newsgroup comp.os.linux.networking, in
> article <(E-Mail Removed)>, ljb wrote:
>
>>Does Linux (2.4.x) act on ICMP Redirect packets by default?
>
> Under limited conditions, yes - but you seem to have a strange network
> layout which might confuse things. 2.4.x? Which 2.4.x? 2.4.31.2 is the
> latest (released about two weeks ago).

It's 2.4.31 (Slackware 10.2), planning to upgrade to 2.6.21.5 (Slackware 12)
as soon as I can get some downtime.

>>If so, can an ICMP Redirect override a static default route?
>>If so, does a routing table entry from an ICMP Redirect time out?
>
> 1122 Requirements for Internet Hosts - Communication Layers. R.
> Braden, Ed.. October 1989. (Format: TXT=295992 bytes) (Updated by
> RFC1349, RFC4379) (Also STD0003) (Status: STANDARD)
>
> 1812 Requirements for IP Version 4 Routers. F. Baker, Ed.. June 1995.
> (Format: TXT=415740 bytes) (Obsoletes RFC1716, RFC1009) (Updated by
> RFC2644) (Status: PROPOSED STANDARD)
>
> A quick scan of those two, I don't see a timeout.
>
>>I have this Linux server that went mostly off-line suddenly today,
>>disconnecting a number of database users and such. The server is on an
>>intranet, private static IP address and one default route to a internal
>>router. (The only odd thing is that there are multiple logical subnets on
>>the same physical subnet.)
>
> Usually, having logical subnets on the same physical wire is a bad idea.

I agree, but we're stuck with it. (I think it started way back when they
decided some boxes couldn't handle different subnet mask sizes.)

>>When it dropped all those connections, it was still reachable from, and
>>could still reach, systems with the same subnet number.
>
> Logical or physical?

Logical, sorry. Server is e.g. 10.1.2.3 subnet /24, only systems on
10.1.2.0/24 could talk to it. Other systems on the same physical subnet
(connected to the same Cisco Catalyst switch), but a different subnet
number, could not.

>>Unfortunately, I didn't realize that at the time - I found two systems
>>that could still reach it, but I didn't make the subnet connection. So I
>>didn't check the routing table until later.
>
> Is syslogd configured to log any routing information?

I don't think so. It doesn't act as a router (no forwarding, one network
interface). Unless the kernel logs routing messages, I'm not sure what else
would do that.

>>About 90 minutes after it dropped off, it came back up; nobody did
>>anything to it - it just started taking to the network normally again.
>
> Assuming you are not using a routing daemon like routed or gated, a
> redirect should stick until networking is restarted (clearing and
> reloading the routing table). The only way to change the table would
> be an ICMP Redirect (which a distribution-standard kernel should be
> ignoring except under very limited conditions).
>
>>Trying to figure out what happened, I was wondering if a 'rogue' ICMP
>>redirect could cause this. Is this possible?
>
> Depends on your network layout. Not very likely, but not impossible.

No routing daemon. The other person said he saw a 10 minute timeout, but
only host redirects worked, not network. Which makes sense.

Either way, I don't think this was the cause. The networking group, and
Active Directory group (this is a Windows shop) also have no idea and say
they didn't do anything. Oh well.

On Fri, 31 Aug 2007, in the Usenet newsgroup comp.os.linux.networking, in
article <(E-Mail Removed)>, ljb wrote:

>(E-Mail Removed) wrote:

>> Usually, having logical subnets on the same physical wire is a bad idea.
>
>I agree, but we're stuck with it. (I think it started way back when they
>decided some boxes couldn't handle different subnet mask sizes.)

Makes for a fun with the routing tables for everyone else.

>Logical, sorry. Server is e.g. 10.1.2.3 subnet /24, only systems on
>10.1.2.0/24 could talk to it. Other systems on the same physical subnet
>(connected to the same Cisco Catalyst switch), but a different subnet
>number, could not.

Could systems on "that" subnet talk to other systems on "this" subnet?
Obviously the problem winds down to "both ends need to know how to see
the other side" in order to talk.

>> Is syslogd configured to log any routing information?
>
>I don't think so. It doesn't act as a router (no forwarding, one network
>interface). Unless the kernel logs routing messages, I'm not sure what
>else would do that.

I was more hoping that the applications wrote something. Oh well.

>> Depends on your network layout. Not very likely, but not impossible.
>
>No routing daemon. The other person said he saw a 10 minute timeout,
>but only host redirects worked, not network. Which makes sense.

This was a problem recognized long ago. See those RFCs I mentioned.
There isn't supposed to be a timeout. If you've screwed up the routing
setup such that the router or what-ever is correcting you, the idea
is not to stop using the "corrected" routing data just because things
timed out. On the other hand, when something else changes and then
some host/router could send the Redirect to fix things. However this
is SO EASY to abuse, most network setups default to "ignore". The
assumption is that whoever set up the routing should know what they
are doing, and if they've made a mistake, it's better to learn about
it ("it's not working!!! - WHY") than to have some silly helper cover
up the mistake. (Example - ZeroConf when the MCSE so screwed up the
DHCP server that even windoze can't get an IP.)

>Either way, I don't think this was the cause. The networking group,
>and Active Directory group (this is a Windows shop) also have no idea
>and say they didn't do anything. Oh well.

OF COURSE NOT!!! We wouldn't do nothin' ;-)

Old guy

(E-Mail Removed) wrote:
>>> Usually, having logical subnets on the same physical wire is a bad idea.
>>
>>I agree, but we're stuck with it. (I think it started way back when they
>>decided some boxes couldn't handle different subnet mask sizes.)
>
> Makes for a fun with the routing tables for everyone else.

It's just a great big tree. Every system has only default route, and there
are several largish Cisco routers that connect everything. So nobody has
non-trivial routing tables except those main router boxes.

>>Logical, sorry. Server is e.g. 10.1.2.3 subnet /24, only systems on
>>10.1.2.0/24 could talk to it. Other systems on the same physical subnet
>>(connected to the same Cisco Catalyst switch), but a different subnet
>>number, could not.
>
> Could systems on "that" subnet talk to other systems on "this" subnet?
> Obviously the problem winds down to "both ends need to know how to see
> the other side" in order to talk.

Yes, other systems worked fine except that they couldn't see my server if
they had a different subnet number.
If "A" was the server with the trouble, same subnet number as "B", different
subnet number from "C", then "A" <-> "B" and "B" <-> "C" worked, but "A"
<-> "C" did not.

On Sat, 01 Sep 2007, in the Usenet newsgroup comp.os.linux.networking, in
article <(E-Mail Removed)>, ljb wrote:

>(E-Mail Removed) wrote:

>> Makes for a fun with the routing tables for everyone else.
>
>It's just a great big tree. Every system has only default route, and there
>are several largish Cisco routers that connect everything. So nobody has
>non-trivial routing tables except those main router boxes.

That is one of the problems - I'll bet the Ciscos were fun to convince
that everything was local.

>> Could systems on "that" subnet talk to other systems on "this" subnet?

>Yes, other systems worked fine except that they couldn't see my server if
>they had a different subnet number.
>If "A" was the server with the trouble, same subnet number as "B", different
>subnet number from "C", then "A" <-> "B" and "B" <-> "C" worked, but "A"
><-> "C" did not.

That pretty much says it was the server that lost it, although it might
still be the Cisco (I don't use Catalysts). I suspect that the only way
you're going to find this is to wait until it happens again, and then
look at the routing table and arp cache on the server.

Old guy