Router generates INVALID icmp packets

Hello everybody,

I am witnessing a strange behavior of our gateway/router
and cannot figure out what's going on.

The router (Slackware 10.0, bare.i) is running a simple
iptables script which, besides other rules, includes the
following rules (often suggested in docs/scripts available
in the Internet):

iptables -A OUTPUT -p icmp -m state --state INVALID -j LOG \
**--log-prefix*"Fired*INVALID*ICMP:*"
iptables -A OUTPUT -p icmp -m state --state INVALID -j DROP

Then the situation looks as follows.

Suppose I run traceroute on a LAN machine:

darkstar$ traceroute some.host
1************
2***the_first.host.after.the_router***0.894ms*.....
.....

At the same time I observe "1 * * *" at darkstar,
the router logs three times: Fired INVALID ICMP: ...
with the packets being of type 11 (time exceeded).

Thus I don't understand why these packets are INVALID
but not RELATED, and which process or sysctl variable
is responsible for this.

Any ideas will be highly appreciated!

Mikhail

Mikhail Zotov wrote:
> Hello everybody,
>
> I am witnessing a strange behavior of our gateway/router
> and cannot figure out what's going on.
>
> The router (Slackware 10.0, bare.i) is running a simple
> iptables script which, besides other rules, includes the
> following rules (often suggested in docs/scripts available
> in the Internet):
>
> iptables -A OUTPUT -p icmp -m state --state INVALID -j LOG \
> --log-prefix "Fired INVALID ICMP: "
> iptables -A OUTPUT -p icmp -m state --state INVALID -j DROP
>
> Then the situation looks as follows.
>
> Suppose I run traceroute on a LAN machine:
>
> darkstar$ traceroute some.host
> 1 * * *
> 2 the_first.host.after.the_router 0.894ms .....
> ....
>
> At the same time I observe "1 * * *" at darkstar,
> the router logs three times: Fired INVALID ICMP: ...
> with the packets being of type 11 (time exceeded).
>
> Thus I don't understand why these packets are INVALID
> but not RELATED, and which process or sysctl variable
> is responsible for this.
>
> Any ideas will be highly appreciated!

ICMP is designed to work this way:
http://www.networksorcery.com/enp/protocol/icmp.htm
http://www.networksorcery.com/enp/pr...icmp/msg11.htm

And traceroute was designed to use the error code returned:
$ man traceroute
[q]
Traceroute utilizes the IP protocol `time to live' field and
attempts to elicit an ICMP TIME_EXCEEDED response from each gateway
along the path to some host.
[eq]

The packets sent are _not_ part of a single stream, thus _not_ related.

hth,
prg

prg wrote on Tuesday 15 March 2005 17:40:
> Mikhail Zotov wrote:
....
>> Suppose I run traceroute on a LAN machine:
>>
>> darkstar$ traceroute some.host
>> 1 * * *
>> 2 the_first.host.after.the_router 0.894ms .....
>> ....
>>
>> At the same time I observe "1 * * *" at darkstar,
>> the router logs three times: Fired INVALID ICMP: ...
>> with the packets being of type 11 (time exceeded).
>>
>> Thus I don't understand why these packets are INVALID
>> but not RELATED, and which process or sysctl variable
>> is responsible for this.
>
> ICMP is designed to work this way:
> http://www.networksorcery.com/enp/protocol/icmp.htm
> http://www.networksorcery.com/enp/pr...icmp/msg11.htm
>
> And traceroute was designed to use the error code returned:
> $ man traceroute
> [q]
> Traceroute utilizes the IP protocol `time to live' field and
> attempts to elicit an ICMP TIME_EXCEEDED response from each gateway
> along the path to some host.
> [eq]
>
> The packets sent are _not_ part of a single stream, thus _not_ related.

Thank you *very much* for your really eye-opening reply and
very helpful links.

Still, I feel even more surprised now. As far as I understand,
in order to avoid the situation described above, I should enable
the following `logic' on the router:

ICMP packets of type 11 and state `INVALID' should be allowed
to go out from the interior network interface in case forwarding
of outgoing traceroute (tracert) is permitted.

On the other hand, it seems that the majority of iptables scripts
available in the Internet deny INVALID packets in OUTPUT chain
before anything is actually done, see, e.g., scripts generated
by http://easyfwgen.morizot.net/ or this article:
http://linuxgazette.net/103/odonovan.html
Thus, people who use these scripts or follow receipts provided in
the above article should face exactly the same situation as I do.
:-/

In fact, I have found only one script that allows ICMP packets
of type 11 in OUTPUT chain regardless of their state:
http://linux.fan.lt/fwall (I don't count scripts that allow
everything in OUTPUT chain.)

Thus, the situation seems to be a bit strange.

Anyway, thanks a lot for the answer!

Regards,
Mikhail

Mikhail Zotov wrote:
> prg wrote on Tuesday 15 March 2005 17:40:
> > Mikhail Zotov wrote:
> ...
> >> Suppose I run traceroute on a LAN machine:
> >>
> >> darkstar$ traceroute some.host
> >> 1 * * *
> >> 2 the_first.host.after.the_router 0.894ms .....
> >> ....
> >>
> >> At the same time I observe "1 * * *" at darkstar,
> >> the router logs three times: Fired INVALID ICMP: ...
> >> with the packets being of type 11 (time exceeded).
> >>
> >> Thus I don't understand why these packets are INVALID
> >> but not RELATED, and which process or sysctl variable
> >> is responsible for this.
> >
> > ICMP is designed to work this way:
> > http://www.networksorcery.com/enp/protocol/icmp.htm
> > http://www.networksorcery.com/enp/pr...icmp/msg11.htm
> >
> > And traceroute was designed to use the error code returned:
> > $ man traceroute
> > [q]
> > Traceroute utilizes the IP protocol `time to live' field and
> > attempts to elicit an ICMP TIME_EXCEEDED response from each gateway
> > along the path to some host.
> > [eq]
> >
> > The packets sent are _not_ part of a single stream, thus _not_
related.
>
> Thank you *very much* for your really eye-opening reply and
> very helpful links.
>
> Still, I feel even more surprised now. As far as I understand,
> in order to avoid the situation described above, I should enable
> the following `logic' on the router:
>
> ICMP packets of type 11 and state `INVALID' should be allowed
> to go out from the interior network interface in case forwarding
> of outgoing traceroute (tracert) is permitted.
>
> On the other hand, it seems that the majority of iptables scripts
> available in the Internet deny INVALID packets in OUTPUT chain
> before anything is actually done, see, e.g., scripts generated
> by http://easyfwgen.morizot.net/ or this article:
> http://linuxgazette.net/103/odonovan.html
> Thus, people who use these scripts or follow receipts provided in
> the above article should face exactly the same situation as I do.
> :-/
>
> In fact, I have found only one script that allows ICMP packets
> of type 11 in OUTPUT chain regardless of their state:
> http://linux.fan.lt/fwall (I don't count scripts that allow
> everything in OUTPUT chain.)
>
> Thus, the situation seems to be a bit strange.

No time now to offer any explanation -- appointments to keep ;-)

Go here to find all kinds of info on iptables:
http://www.linuxguruz.com/iptables/

and here for a pretty good/complete tutorial:
http://iptables-tutorial.frozentux.n...-tutorial.html

There _are_ reasons to consider the traceroute packets _entering_ a
private network segment as invalid or simply unwelcome.

Also realize that ICMP was not designed _for_ traceroute but for other
purposes and traceroute simply takes advantage of the protocol to
gather info about each hop encountered in getting to a particular IP
(by manipulating the TTL value of each packet sent and decoding the
returned ICMP error).

I'll try to check back later.

regards,
prg

prg wrote on Wednesday 16 March 2005 00:33:
>
> Mikhail Zotov wrote:
>> prg wrote on Tuesday 15 March 2005 17:40:
>> > Mikhail Zotov wrote:
>> ...
>> >> Suppose I run traceroute on a LAN machine:
>> >>
>> >> darkstar$ traceroute some.host
>> >> 1 * * *
>> >> 2 the_first.host.after.the_router 0.894ms .....
>> >> ....
>> > ICMP is designed to work this way:
>> > http://www.networksorcery.com/enp/protocol/icmp.htm
>> > http://www.networksorcery.com/enp/pr...icmp/msg11.htm
>> >
....
>> > The packets sent are _not_ part of a single stream, thus _not_
> related.
>>
>> Thank you *very much* for your really eye-opening reply and
>> very helpful links.
>>
>> Still, I feel even more surprised now. As far as I understand,
>> in order to avoid the situation described above, I should enable
>> the following `logic' on the router:
>>
>> ICMP packets of type 11 and state `INVALID' should be allowed
>> to go out from the interior network interface in case forwarding
>> of outgoing traceroute (tracert) is permitted.
....
>> In fact, I have found only one script that allows ICMP packets
>> of type 11 in OUTPUT chain regardless of their state:
>> http://linux.fan.lt/fwall (I don't count scripts that allow
>> everything in OUTPUT chain.)
>>
>> Thus, the situation seems to be a bit strange.
>
> No time now to offer any explanation -- appointments to keep ;-)

Don't worry and take your time! I am trying to solve this problem
for already two or three weeks now. Have already tried two other
NGs and the netfilter mailing list. Thus, there's no hurry. :-)

> Go here to find all kinds of info on iptables:
> http://www.linuxguruz.com/iptables/
>
> and here for a pretty good/complete tutorial:
> http://iptables-tutorial.frozentux.n...-tutorial.html

Oscar Andreasson trusts his hosts on output. At least, in the
tutorial. :-) Will check linuxguruz though. I have studied
their docs before writing the script of my own but I have probably
missed something.

[snip]
> I'll try to check back later.

Thank you!

Regards,
Mikhail

Mikhail Zotov wrote:
> prg wrote on Wednesday 16 March 2005 00:33:
> >
> > Mikhail Zotov wrote:
> >> prg wrote on Tuesday 15 March 2005 17:40:
> >> > Mikhail Zotov wrote:
> >> ...
> >> >> Suppose I run traceroute on a LAN machine:
> >> >>
> >> >> darkstar$ traceroute some.host
> >> >> 1 * * *
> >> >> 2 the_first.host.after.the_router 0.894ms .....
> >> >> ....
> >> > ICMP is designed to work this way:
> >> > http://www.networksorcery.com/enp/protocol/icmp.htm
> >> > http://www.networksorcery.com/enp/pr...icmp/msg11.htm
> >> >
> ...
> >> > The packets sent are _not_ part of a single stream, thus _not_
> > related.
> >>
> >> Thank you *very much* for your really eye-opening reply and
> >> very helpful links.
> >>
> >> Still, I feel even more surprised now. As far as I understand,
> >> in order to avoid the situation described above, I should enable
> >> the following `logic' on the router:
> >>
> >> ICMP packets of type 11 and state `INVALID' should be allowed
> >> to go out from the interior network interface in case forwarding
> >> of outgoing traceroute (tracert) is permitted.
> ...
> >> In fact, I have found only one script that allows ICMP packets
> >> of type 11 in OUTPUT chain regardless of their state:
> >> http://linux.fan.lt/fwall (I don't count scripts that allow
> >> everything in OUTPUT chain.)
> >>
> >> Thus, the situation seems to be a bit strange.
> >
> > No time now to offer any explanation -- appointments to keep ;-)
>
> Don't worry and take your time! I am trying to solve this problem
> for already two or three weeks now. Have already tried two other
> NGs and the netfilter mailing list. Thus, there's no hurry. :-)
>
> > Go here to find all kinds of info on iptables:
> > http://www.linuxguruz.com/iptables/
> >
> > and here for a pretty good/complete tutorial:
> > http://iptables-tutorial.frozentux.n...-tutorial.html
>
> Oscar Andreasson trusts his hosts on output. At least, in the
> tutorial. :-) Will check linuxguruz though. I have studied
> their docs before writing the script of my own but I have probably
> missed something.
>
> [snip]
> > I'll try to check back later.
>
> Thank you!

I can't give you a pre-determined list of what ICMP traffic to
allow/block and no one else can either :-(

The main thing people deal with is what "probes" to allow inside, eg.,
ping and some of the "dated" ICMP query messages. Simply dropping
pings at the border router, eg., deprives anyone (including you) from
confirming reachability to your network. This also involves a POV
regarding dropping packets silently or retuning "Administratively
Prohibited" messages (ie., DENY or REJECT in netfilter rules).

The next and hardest thing to deal with are the errors that application
protocols or hosts may legitimately (and informatively) return when
network conditions/routing drops in the crapper. These may be the only
clues at hand to tell what's wrong. These would be locally generated
packets sent back to the "other end". You probably will allow most of
this to go out or handle it at the border router. It may also depend
on the mix of hosts, servers, and subnets behind your firewall.
Private lan hosts may be mostly blocked while DMZ servers are allowed
to return some (or even all) error messages.

And some ICMP traffic is highly useful in some settings, eg., redirects
that routers emit to send hosts along the correct route path.

For added "protection" when allowing some ICMP traffic you can limit
the burst rate of processed traffic -- most required/legitmate traffic
is a single returned packet.

ICMP messages (codes) are very compact which adds to the complexity of
understanding what you you may want to allow/deny. Deciphering the
code numbers is tedious. Plus, it's hard to understand sometimnes when
the messages may be generated.

Sniffing traffic designed to elicit ICMP messages can help a lot in
understanding what is going on, but it remains a difficult protocol to
"master" since it is used in so many different ways.

The article you referenced is a pretty good intro and the rules you
posted are "common" practice. Beware that traceroute's _default_
behavior is _not_ a good example of ICMP's normal usage, so don't base
your impressions on what you observe using traceroute. It can be used
with some options that do make it a useful, though limited, "testing"
tool.

Here are my links that I had on hand to help understand ICMP and how
netfilter deals with ICMP traffic:
( http://www.sns.ias.edu/~jns/security...conntrack.html
)
this one especially useful

( http://www.sns.ias.edu/~jns/security...les/index.html )
( http://www.cymru.com/Documents/icmp-messages.html ) < Cisco IOS
( http://www.faqs.org/rfcs/rfc792.html )

Also a copy of Stevens' "TCP-IP Illustrated Vol. 1" is _highly_
recommended. Chapter 6 deals with ICMP and it is discussed elsewhere
when appropriate.

Note that the terms used by netfilter do _not_ coincide with the use of
those terms in TCP/IP. And the use with regard to ICMP, viz., INVALID
state, can be especially misleading -- simply means there is no
"matching" state table entry for that packet and should not be taken to
mean "illegitimate" or "dangerous".

hth,
prg

prg wrote on Thursday 17 March 2005 08:25:
> Mikhail Zotov wrote:
>> prg wrote on Wednesday 16 March 2005 00:33:
>> >
>> > Mikhail Zotov wrote:
>> >> prg wrote on Tuesday 15 March 2005 17:40:
>> >> > Mikhail Zotov wrote:
>> >> ...
>> >> >> Suppose I run traceroute on a LAN machine:
>> >> >>
>> >> >> darkstar$ traceroute some.host
>> >> >> 1 * * *
>> >> >> 2 the_first.host.after.the_router 0.894ms .....
>> >> >> ....
> The main thing people deal with is what "probes" to allow inside, eg.,
> ping and some of the "dated" ICMP query messages. Simply dropping
> pings at the border router, eg., deprives anyone (including you) from
> confirming reachability to your network.

Not in our case because we are running a DNS and a mail server.
Thus, say, "telnet mail.server 25" can be used for this purpose.
(Admittedly, the result will be misleading in case the whole LAN
is up but the mail server is down. Sometimes, this happens. :-))

I am a bit afraid of allowing pings to come in from everywhere
because it has been reported numerous times that windoops machines
(yeah, we have some :-() can be infected by certain worms via
specially crafted echo requests.

> This also involves a POV
> regarding dropping packets silently or retuning "Administratively
> Prohibited" messages (ie., DENY or REJECT in netfilter rules).
>
> The next and hardest thing to deal with are the errors that application
> protocols or hosts may legitimately (and informatively) return when
> network conditions/routing drops in the crapper. These may be the only
> clues at hand to tell what's wrong. These would be locally generated
> packets sent back to the "other end". You probably will allow most of
> this to go out or handle it at the border router. It may also depend
> on the mix of hosts, servers, and subnets behind your firewall.
> Private lan hosts may be mostly blocked while DMZ servers are allowed
> to return some (or even all) error messages.

OK, I see.

>
> And some ICMP traffic is highly useful in some settings, eg., redirects
> that routers emit to send hosts along the correct route path.
>
> For added "protection" when allowing some ICMP traffic you can limit
> the burst rate of processed traffic -- most required/legitmate traffic
> is a single returned packet.

Thanks for the idea! I'll implement it.
(I limit the rate of allowed incoming pings but haven't even thought about
limiting the rate of outgoing packets.)

[snip]
> The article you referenced is a pretty good intro and the rules you
> posted are "common" practice. Beware that traceroute's _default_
> behavior is _not_ a good example of ICMP's normal usage, so don't base
> your impressions on what you observe using traceroute. It can be used
> with some options that do make it a useful, though limited, "testing"
> tool.

Yes, I see now.

> Here are my links that I had on hand to help understand ICMP and how
> netfilter deals with ICMP traffic:
> ( http://www.sns.ias.edu/~jns/security...conntrack.html
> )
> this one especially useful
>
> ( http://www.sns.ias.edu/~jns/security...les/index.html )
> ( http://www.cymru.com/Documents/icmp-messages.html ) < Cisco IOS
> ( http://www.faqs.org/rfcs/rfc792.html )
>
> Also a copy of Stevens' "TCP-IP Illustrated Vol. 1" is _highly_
> recommended. Chapter 6 deals with ICMP and it is discussed elsewhere
> when appropriate.

Thanks a lot! I studied Stephens' materials earlier but now I realize
that I overlooked his very clear explanation of a number of ICMP-related
issues. And it seems that I really need to find Stevens' book.

> Note that the terms used by netfilter do _not_ coincide with the use of
> those terms in TCP/IP. And the use with regard to ICMP, viz., INVALID
> state, can be especially misleading -- simply means there is no
> "matching" state table entry for that packet and should not be taken to
> mean "illegitimate" or "dangerous".

Yes, this was probably the crucial point of my `problem'. I was nervous
about INVALID packets but didn't understand that they may look pretty
legitimate from the kernel point of view.

> hth,
> prg

Yes, you have really helped me a lot and I am *very* thankful!
(BTW, my script is available at http://slackfire.narod.ru/
It is far from the desired level still but I work to make it better.
The script and the README file include acknowledgments to people
who helped me to improve the script. Please, let me know if you
would prefer to be acknowledged in other way than "prg". I don't
think that there will be a lot of people who see this but some will. :-)
To contact me, please use the address given in the header with
"blah" excluded or the address provided at the web page.)

Once again, thanks a lot for the enlightening correspondence!

Regards,
Mikhail