Problems with IP forwarding

Hi all,

I am running Fedora Core 3 as a file server and gateway [to our
firewall, and beyond to the web] and am having trouble with IP
forwarding. The gateway machine is "beaker" and the firewall "bunsen".
I can not access the firewall from the internal network other than from
the gateway, which is directly connected to the firewall. I presume
this is an IP forwarding issue. I have IP forwarding enabled from
/etc/sysctl.conf:

[root@beaker ipv4]# cat /etc/sysctl.conf |grep -i forward
# Controls IP packet forwarding
net.ipv4.ip_forward = 1

But alas, forwarding doesn't seem to work [and this is driving me
crazy!!] Any pointers to debugging/fixing this problem greatfully
received!

Below I have included some system information. The [not-working]
gateway has 2 ethernet cards: eth0 (to firewall) eth1 (to internal
network).

Regards,
Trevor.

[root@beaker ipv4]# uname -rm
2.6.10 i686
[root@beaker ipv4]# pwd
/proc/sys/net/ipv4
[root@beaker ipv4]# for i in `find . |grep forward |grep -v mc`; do
echo $i; cat $i; done
../conf/eth1/forwarding
1
../conf/eth0/forwarding
1
../conf/lo/forwarding
1
../conf/default/forwarding
1
../conf/all/forwarding
1
../ip_forward
1
[root@beaker ipv4]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:08:A1:37:4A:12
inet addr:203.1.78.65 Bcast:203.1.78.127
Mask:255.255.255.192
inet6 addr: fe80::208:a1ff:fe37:4a12/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:23285 errors:0 dropped:0 overruns:0 frame:0
TX packets:14782 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:20801657 (19.8 MiB) TX bytes:2228196 (2.1 MiB)
Interrupt:5 Base address:0xd800

eth1 Link encap:Ethernet HWaddr 00:08:A1:37:4E1
inet addr:203.1.78.3 Bcast:203.1.78.63 Mask:255.255.255.192
inet6 addr: fe80::208:a1ff:fe37:4ed1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:31745 errors:0 dropped:0 overruns:0 frame:0
TX packets:40667 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4897869 (4.6 MiB) TX bytes:45747859 (43.6 MiB)
Interrupt:5 Base address:0xd400

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:6684 errors:0 dropped:0 overruns:0 frame:0
TX packets:6684 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:827462 (808.0 KiB) TX bytes:827462 (808.0 KiB)

[root@beaker ipv4]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
203.1.78.64 * 255.255.255.192 U 0 0 0
eth0
203.1.78.0 * 255.255.255.192 U 0 0 0
eth1
169.254.0.0 * 255.255.0.0 U 0 0 0
eth1
default bunsen.reptechn 0.0.0.0 UG 0 0 0
eth0
[root@beaker ipv4]# dmesg | grep -i ip
Calibrating delay loop... 5275.64 BogoMIPS (lpj=2637824)
agpgart: Detected an Intel 865 Chipset.
io scheduler anticipatory registered
ICH5: chipset revision 2
elevator: using anticipatory as default io scheduler
IP: routing cache hash table of 1024 buckets, 32Kbytes
Initializing IPsec netlink socket
ip_tables: (C) 2000-2002 Netfilter core team
ip_tables: (C) 2000-2002 Netfilter core team
ip_tables: (C) 2000-2002 Netfilter core team
ip_tables: (C) 2000-2002 Netfilter core team
IPv6 over IPv4 tunneling driver
ip_conntrack version 2.1 (4025 buckets, 32200 max) - 356 bytes per
conntrack
eth0: no IPv6 routers present
eth1: no IPv6 routers present
[root@beaker ipv4]#

(E-Mail Removed) wrote:
> Hi all,
>
> I am running Fedora Core 3 as a file server and gateway [to our
> firewall, and beyond to the web] and am having trouble with IP
> forwarding. The gateway machine is "beaker" and the firewall
"bunsen".
> I can not access the firewall from the internal network other than
from
> the gateway, which is directly connected to the firewall. I presume
> this is an IP forwarding issue. I have IP forwarding enabled from
> /etc/sysctl.conf:
>
> [root@beaker ipv4]# cat /etc/sysctl.conf |grep -i forward
> # Controls IP packet forwarding
> net.ipv4.ip_forward = 1
>
> But alas, forwarding doesn't seem to work [and this is driving me
> crazy!!] Any pointers to debugging/fixing this problem greatfully
> received!
>
> Below I have included some system information. The [not-working]
> gateway has 2 ethernet cards: eth0 (to firewall) eth1 (to internal
> network).
>
> Regards,
> Trevor.
>
> [root@beaker ipv4]# uname -rm
> 2.6.10 i686
> [root@beaker ipv4]# pwd
> /proc/sys/net/ipv4
> [root@beaker ipv4]# for i in `find . |grep forward |grep -v mc`; do
> echo $i; cat $i; done
> ./conf/eth1/forwarding
> 1
> ./conf/eth0/forwarding
> 1
> ./conf/lo/forwarding
> 1
> ./conf/default/forwarding
> 1
> ./conf/all/forwarding
> 1
> ./ip_forward
> 1
> [root@beaker ipv4]# ifconfig
> eth0 Link encap:Ethernet HWaddr 00:08:A1:37:4A:12
> inet addr:203.1.78.65 Bcast:203.1.78.127
> Mask:255.255.255.192
> inet6 addr: fe80::208:a1ff:fe37:4a12/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:23285 errors:0 dropped:0 overruns:0 frame:0
> TX packets:14782 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:20801657 (19.8 MiB) TX bytes:2228196 (2.1 MiB)
> Interrupt:5 Base address:0xd800
>
> eth1 Link encap:Ethernet HWaddr 00:08:A1:37:4E1
> inet addr:203.1.78.3 Bcast:203.1.78.63
Mask:255.255.255.192
> inet6 addr: fe80::208:a1ff:fe37:4ed1/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:31745 errors:0 dropped:0 overruns:0 frame:0
> TX packets:40667 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:4897869 (4.6 MiB) TX bytes:45747859 (43.6 MiB)
> Interrupt:5 Base address:0xd400
>
> lo Link encap:Local Loopback
> inet addr:127.0.0.1 Mask:255.0.0.0
> inet6 addr: ::1/128 Scope:Host
> UP LOOPBACK RUNNING MTU:16436 Metric:1
> RX packets:6684 errors:0 dropped:0 overruns:0 frame:0
> TX packets:6684 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:827462 (808.0 KiB) TX bytes:827462 (808.0 KiB)
>
> [root@beaker ipv4]# route
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref
Use
> Iface
> 203.1.78.64 * 255.255.255.192 U 0 0
0
> eth0
> 203.1.78.0 * 255.255.255.192 U 0 0
0
> eth1
> 169.254.0.0 * 255.255.0.0 U 0 0
0
> eth1
> default bunsen.reptechn 0.0.0.0 UG 0 0
0
> eth0
[snip]

Forwarding seems configured correctly.

Double check that firewall is not running on GW/beaker.

When using ping or traceroute, be sure to use IPs and not host names to
confirm connectivity. Leave name resolution out of the mix.

I'm not sure why you are running public IPs into/through your lan. As
is, the GW/beaker is fully exposed to the internet except for the
absolute, unbreakable configuration of your FW/bunsen. Hmmm...

Are you sure you want/need a public IP on the lan side nic of
GW/beaker. If so, you will need a router at the other end or hosts
will need public IPs on the 203.1.78.0 subnet.

Your current subnet scheme looks like this:
netmask (/26):
111111111111111111111111 11 000000 -> 255.255.255.192
eth1: GW (beaker) to lan (switch? router? hosts?)
110010110000000101001110 00 000011 -> 203.1.78.3

110010110000000101001110 00 000000 -> 203.1.78.0 -> net
110010110000000101001110 00 000011 -> 203.1.78.3 -> eth1
110010110000000101001110 00 111111 -> 203.1.78.63 -> bcast

eth0: GW (beaker) to FW (bunsen)
110010110000000101001110 01 000001 -> 203.1.78.65

110010110000000101001110 01 000000 -> 203.1.78.64 -> net
110010110000000101001110 01 000001 -> 203.1.78.65 -> eth0
110010110000000101001110 01 111111 -> 203.1.78.127 -> bcast

unused(?) subnets:
110010110000000101001110 10 000000 -> 203.1.78.128 -> net
110010110000000101001110 10 111111 -> 203.1.78.191 -> bcast

110010110000000101001110 11 000000 -> 203.1.78.192 -> net
110010110000000101001110 11 111111 -> 203.1.78.255 -> bcast

eth1:203.1.78.3 -> bc=203.1.78.63
11001011 -> 203
00000001 -> 1
01001110 -> 78
00000011 -> 3

eth0:203.1.78.65 -> bc=203.1.78.127
11001011 -> 203
00000001 -> 1
01001110 -> 78
01000001 -> 65

This is workable, but I think it's probably not what you want.

Most people would have the lan side nic of FW/bunsen with a private IP,
both nics on GW/beaker with private IPs, lan hosts with private IPs,
and FW/bunsen providing NAT for all traffic out to internet, possible
firewall/rules on GW/beaker (but _no_ NAT here).

Without knowing more about your FW-GW-lan layout as it exists and what
you intend/want it is difficult to know for sure how to proceed.
Namely, the question is, "How are the lan machines configured?"
Present layout, afaik from info provided, requires that hosts have IPs
on the 203.1.78.0 subnet. Is that what you want/have set up?

hth,
prg
email above disabled

You are indeed correct. We own a class C internet address,
203.1.78.0-203.1.78.255, and the LAN has been setup so that all our
machines have an address in this range. I have only been here a month
or so, so I don't quite understand why it has been setup this way. It
may change in time.

Even so, I don't see how the choice of LAN IP's could influence the
problem I am seeing, i.e. my GW not forwarding traffic? Am I missing
something?

Regards,
Trevor.

(E-Mail Removed) wrote:
> You are indeed correct. We own a class C internet address,
> 203.1.78.0-203.1.78.255, and the LAN has been setup so that all our
> machines have an address in this range. I have only been here a month
> or so, so I don't quite understand why it has been setup this way. It
> may change in time.
>
> Even so, I don't see how the choice of LAN IP's could influence the
> problem I am seeing, i.e. my GW not forwarding traffic? Am I missing
> something?

Well, once ip_forward is turned on, the problems always lie in
misconfiguration. Question is, "Where?"

Did not note in previous post that loopback is not entered in GW/beaker
route table. You need a loopback interface entered. It can cause
difficulty with ICMP packets like arp, among others, without it.

Still no joy?

Select one lan machine and work with it till you get connectivity. On
it check/post:
$ /sbin/ifconfig -a
$ /sbin/route -n
Make sure the netmask is correct and that IP address is on the
203.1.78.0 subnet. The other available subnets will not work in the
lan. Configure loopback. Make sure there is a route entry for net
203.1.78.0 and GW/default route set to 203.1.78.3.

On GW/beaker turn off the firewall -- # /etc/init.d/iptables -stop
Add loopback interface to route table. Confirm ip_forward=1 is in
effect. On it check/post:
$ /sbin/ifconfig -a
$ /sbin/route -n
Confirm that route table still looks good -- with loopback added.

On FW/bunsen turn off firewall if possible. If you can't turn off the
firewall, make sure your rules are not dropping/rejecting ICMP traffic
or dropping pings or anything else that might interfere -- can't be
more specific since I don't know your rule setup/policies. It's always
easiest and _much_ less error prone just to turn it off if you can. On
it check/post:
$ /sbin/ifconfig -a
$ /sbin/route -n
Confirm loopback configured and is present in route table. Make sure
that route table net entries exist for _both_ 203.1.78.64 and
203.1.78.0. This assumes that it will have a default route/GW on the
ISP interface. You may need to add an explicit GW entry for 203.1.78.0
on the GW/beaker interface

Once the configs look OK -- ie., no obvious mistakes, etc. -- it's time
to systematically ping from one end to the other. From your chosen lan
host:

$ ping 127.0.0.1
$ ping hos.tIP.add.res <- host's IP
$ ping 203.1.78.3 <- GW's lan nic (default route)
$ ping 203.1.78.65 <- GW's FW nic
$ ping 203.1.78.?? <- FW's GW nic
$ ping 203.1.78.?? <- FW's ISP nic
$ ping 203.1.78.?? <- FW's default route

You can also try traceroute as it uses UDP packets -- in case something
is dropping the pings.

No joy?

Same procedure starting from FW end working toward the lan host using
the GW/beaker nic's address.
$ ping 127.0.0.1
$ ping hos.tIP.add.res <- FW host's IP
$ ping 203.1.78.65 <- GW's FW nic
$ ping 203.1.78.3 <- GW's lan nic
$ ping 203.1.78.?? <- lan host's nic

And last but not least, work from both nics' IP addresses on GW/beaker
to the FW's ISP nic IP and the lan host's IP. Use $ ping
-I[dev/address] target.

All I can think to do for now. I always, very boringly, follow the
same routine when locating a connectivity/networking problem. I always
know what I'm expecting and successes as well as failures are valuable
clues. If still no joy, I might try a few "hunches" but I would plan
on getting the sniffer out real soon.

good luck,
prg
email above disabled

Hi,

Fisrtly, thank you very much for taking the time to look at this. Much
appreciated!

Here's our setup (note: prepend 203.1.78 to all IPs, so [.3] is really
[203.1.78.3])


Router[.195]<------>[.225]bunsen[.66]<------->[.65]beaker[.3]<----->
LAN

I can ping, say, www.google.com from beaker, so I know there's no
firewall problems on bunsen, or connectivity problems from beaker to
outside world. I have configured IP tables on beaker to accept
everything, and log it. IP tables on bunsen (firewall) is restrictive,
but allows pings out.

If I do a ping from a LAN machine to, again say www.google.com, I can
see entries in the IP tables log saying that beaker accepted an ICMP
from the LAN machine, destined for 64.233.187.99 (google). So I know
packets from the LAN are getting to beaker, and accepted, but they then
seemingly disappear. I can also successfully ping beaker from any LAN
machine.

I am in the process of recompiling my 2.6.10 kernel to turn on kprobe
support, and use the netpktlog module to trace packets through the
kernel to see what happens.

Here's some config info:

BUNSEN (Firewall)
-----------------

[root@bunsen root]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
203.1.78.3 203.1.78.65 255.255.255.255 UGH 0 0 0
eth0
203.1.78.224 0.0.0.0 255.255.255.224 U 0 0 0
eth1
203.1.78.64 0.0.0.0 255.255.255.192 U 0 0 0
eth0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0
lo
0.0.0.0 203.1.78.226 0.0.0.0 UG 0 0 0
eth1
[root@bunsen root]# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:40:F4:37:19:74
inet addr:203.1.78.66 Bcast:203.1.78.127
Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:220256 errors:0 dropped:0 overruns:0 frame:0
TX packets:371635 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:107598396 (102.6 Mb) TX bytes:167642771 (159.8 Mb)
Interrupt:9 Base address:0x9000

eth1 Link encap:Ethernet HWaddr 00:40:F4:37:273
inet addr:203.1.78.225 Bcast:203.1.78.255
Mask:255.255.255.224
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:298815 errors:0 dropped:0 overruns:0 frame:0
TX packets:236096 errors:0 dropped:0 overruns:0 carrier:0
collisions:1261 txqueuelen:100
RX bytes:93226591 (88.9 Mb) TX bytes:44706495 (42.6 Mb)
Interrupt:5 Base address:0xb000

eth2 Link encap:Ethernet HWaddr 00:40:F4:37:26:65
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:9 Base address:0xd000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:341 errors:0 dropped:0 overruns:0 frame:0
TX packets:341 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2210460 (2.1 Mb) TX bytes:2210460 (2.1 Mb)

BEAKER (Gateway)
----------------

[root@beaker ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
203.1.78.64 0.0.0.0 255.255.255.192 U 0 0 0
eth0
203.1.78.0 0.0.0.0 255.255.255.192 U 0 0 0
eth1
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0
lo
0.0.0.0 203.1.78.66 0.0.0.0 UG 0 0 0
eth0
[root@beaker ~]# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:08:A1:37:4A:12
inet addr:203.1.78.65 Bcast:203.1.78.127
Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:27155 errors:0 dropped:0 overruns:0 frame:0
TX packets:16068 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:12498309 (11.9 MiB) TX bytes:2473762 (2.3 MiB)
Interrupt:5 Base address:0xd800

eth1 Link encap:Ethernet HWaddr 00:08:A1:37:4E1
inet addr:203.1.78.3 Bcast:203.1.78.63 Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:304222 errors:0 dropped:0 overruns:0 frame:0
TX packets:387860 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:70791184 (67.5 MiB) TX bytes:446707968 (426.0 MiB)
Interrupt:5 Base address:0xd400

eth2 Link encap:Ethernet HWaddr 00:0C:6E:78:0E:28
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:5163 errors:0 dropped:0 overruns:0 frame:0
TX packets:5163 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1610106 (1.5 MiB) TX bytes:1610106 (1.5 MiB)

(E-Mail Removed) wrote:
> Hi,
>
> Fisrtly, thank you very much for taking the time to look at this.
Much
> appreciated!
>
> Here's our setup (note: prepend 203.1.78 to all IPs, so [.3] is
really
> [203.1.78.3])
>
>
> Router[.195]<------>[.225]bunsen[.66]<------->[.65]beaker[.3]<----->
> LAN
>
> I can ping, say, www.google.com from beaker, so I know there's no
> firewall problems on bunsen, or connectivity problems from beaker to
> outside world. I have configured IP tables on beaker to accept
> everything, and log it. IP tables on bunsen (firewall) is
restrictive,
> but allows pings out.
>
> If I do a ping from a LAN machine to, again say www.google.com, I can
> see entries in the IP tables log saying that beaker accepted an ICMP
> from the LAN machine, destined for 64.233.187.99 (google). So I know
> packets from the LAN are getting to beaker, and accepted, but they
then
> seemingly disappear. I can also successfully ping beaker from any LAN
> machine.

Pinging from beaker is not conclusive. The packets _from_ beaker
travel a different netfilter path than packets received on an
interface. Can your lan boxes ping bunsen? Can they ping beaker's
203.1.78.65 address (though that too is inconclusive of anything)?
What does a traceroute to bunsen's GW provide? Where does it stop?

> I am in the process of recompiling my 2.6.10 kernel to turn on kprobe
> support, and use the netpktlog module to trace packets through the
> kernel to see what happens.

Unless the kernel you're using now is a home grown compile, I seriously
doubt that a new effort will provide a fix. RH/FC have included all
the routing capabilities in their "stock distro" kernels for years now.
And googling turned up nothing suspicious about routing code.

> Here's some config info:
>
> BUNSEN (Firewall)
> -----------------
>
> [root@bunsen root]# route -n
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref
Use
> Iface
> 203.1.78.3 203.1.78.65 255.255.255.255 UGH 0 0
0
> eth0

This is the only config that looks off. Is this an artifact of NATing
on beaker? Otherwise, you should have a network route here rather than
a host route. Ie., 203.1.78.0 (255.255.255.192) rather than 203.1.78.3
(255.255.255.255). That way any packet destined for 203.1.78.0/26 will
have a route via beaker.
# route add -net 203.1.78.0 netmask 255.255.255.192 gw 203.1.78.65
added after 203.1.78.64 I usually add all my GW routes at the end
after all the networks have been entered.

> 203.1.78.224 0.0.0.0 255.255.255.224 U 0 0
0
> eth1
> 203.1.78.64 0.0.0.0 255.255.255.192 U 0 0
0
> eth0
> 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0
0
> lo
> 0.0.0.0 203.1.78.226 0.0.0.0 UG 0 0
0
> eth1

It is tedious but you really need to ping and note responses (or none)
in the sequence I posted. The idea is to find which box is the
culprit, then what about that box is set up wrong. You confirm the
culprit by pinging from the other end of the path, ie., from bunsen to
a lan box.

With your network setup, it may not be possible/reasonable to turn off
the firewall/filtering rules, but you will save much effort if you can
confirm that routing works correctly when they are disengaged. Then
you know for sure where the problem is. Otherwise, only very close
scrutiny and study will likely reveal the key.

If the machines are new to you, you really ought to document how they
are set up, including route tables, IPs, and firewall/filter rules of
both bunsen and beaker. Also check that no one has engaged policy
routing:
$ ip route show ??? < you will look for additional tables
http://linux-ip.net/html/routing-tables.html

$ ip rule show
looking for policy rules

These last two are not likely, but you never know. I would say they
are more likely in a network set up such as yours than most others.
And if you don't know they are there, they will eat your lunch

After double checking all the route tables, interface configs, and
firewall/filter rules, if no solution presents itself, it is time to
sniff the wire. The only other alternatives are hunches and
duplicating the path from bunsen<->beaker<->lan on 3 test boxes. Ugly
....

Logs just can't capture enough data of the right sort to be definitive
as seeing the decoded wire packets/frames. They _may_ reveal an
internal problem that a sniffer can't see, but you usually need a
sniffer to give you a clue where/what to look for in the logs.

Here are some links you may find handy to have -- turn around on google
and the chances of failed/erased delivery from my end are too great
(earlier post was eaten alive by google).

RH9 Reference Guide for all the networking files and their locations:
http://www.redhat.com/docs/manuals/linux/

Routing/filtering on Linux:
http://open-source.arkoon.net/kernel/kernel_net.png
http://linux-ip.net/html/routing-selection.html
http://www.netfilter.org/documentati...g-HOWTO-6.html
http://linux-ip.net/html/routing-tables.html
http://linux-ip.net/html/routing-rpdb.html
http://linux-ip.net/html/tools-ip-route.html

BTW, how _are_ you using your network if no packets can get out of the
lan?

hth,
prg
email above disabled

prg wrote:
> Pinging from beaker is not conclusive. The packets _from_ beaker
> travel a different netfilter path than packets received on an
> interface. Can your lan boxes ping bunsen? Can they ping beaker's
> 203.1.78.65 address (though that too is inconclusive of anything)?
> What does a traceroute to bunsen's GW provide? Where does it stop?

I cannot ping bunsen from any LAN machine, but I can ping bunsen from
beaker. I can ping any of beaker's interface from any LAN machine. So:

PINGS:

piggy[.12]----->[.3]beaker (WORKS)
piggy[.12]----->[.65]beaker (WORKS)
beaker[.65]---->[.66]bunsen (WORKS)
piggy[.12]----->[.3]beaker[.65]---->[.66]bunsen (FAILS)
bunsen[.66]---->[.65]beaker[.3]---->[.12]piggy (FAILS)

> > I am in the process of recompiling my 2.6.10 kernel to turn on
kprobe
> > support, and use the netpktlog module to trace packets through the
> > kernel to see what happens.
>
> Unless the kernel you're using now is a home grown compile, I
seriously
> doubt that a new effort will provide a fix. RH/FC have included all
> the routing capabilities in their "stock distro" kernels for years
now.
> And googling turned up nothing suspicious about routing code.

I wasn't looking for a fix by compiling the kernel, just some debugging
infomation to see what is happening to these packets that are not
getting through beaker.

> > Here's some config info:
> >
> > BUNSEN (Firewall)
> > -----------------
> >
> > [root@bunsen root]# route -n
> > Kernel IP routing table
> > Destination Gateway Genmask Flags Metric Ref
> Use
> > Iface
> > 203.1.78.3 203.1.78.65 255.255.255.255 UGH 0 0
> 0
> > eth0
>
> This is the only config that looks off. Is this an artifact of
NATing
> on beaker? Otherwise, you should have a network route here rather
than
> a host route. Ie., 203.1.78.0 (255.255.255.192) rather than
203.1.78.3
> (255.255.255.255). That way any packet destined for 203.1.78.0/26
will
> have a route via beaker.
> # route add -net 203.1.78.0 netmask 255.255.255.192 gw 203.1.78.65
> added after 203.1.78.64 I usually add all my GW routes at the end
> after all the networks have been entered.

I agree. I think what I am going to do is to split things into several
several networks, all in the 10.X.X.X IP space. Allocate 10.0.0.0/24 to
the LAN, 10.0.1.0/24 to the Firewall/Gateway. All of this sub-netting,
and using our class C internet IP addresses doesn't make a lot of
sense.

> > 203.1.78.224 0.0.0.0 255.255.255.224 U 0 0
> 0
> > eth1
> > 203.1.78.64 0.0.0.0 255.255.255.192 U 0 0
> 0
> > eth0
> > 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0
> 0
> > lo
> > 0.0.0.0 203.1.78.226 0.0.0.0 UG 0 0
> 0
> > eth1
>
> It is tedious but you really need to ping and note responses (or
none)
> in the sequence I posted. The idea is to find which box is the
> culprit, then what about that box is set up wrong. You confirm the
> culprit by pinging from the other end of the path, ie., from bunsen
to
> a lan box.

I have done this. See above.

> With your network setup, it may not be possible/reasonable to turn
off
> the firewall/filtering rules, but you will save much effort if you
can
> confirm that routing works correctly when they are disengaged. Then
> you know for sure where the problem is. Otherwise, only very close
> scrutiny and study will likely reveal the key.

I have even tried turning off our external firewall, very briefly, to
see if it's the culprit, but it wasn't.

<snip>
>
> BTW, how _are_ you using your network if no packets can get out of
the
> lan?

All of our internal servers live on beaker (mail, samba, etc..) which
is accessible from the LAN. And, at the moment, we have a proxy server
set up on beaker for using web, ftp, etc. So it works at the moment.
But it is quite restrictive.

Again, thanks for your ongoing help.

Trevor.

>
> piggy[.12]----->[.3]beaker (WORKS)
> piggy[.12]----->[.65]beaker (WORKS)
> beaker[.65]---->[.66]bunsen (WORKS)
> piggy[.12]----->[.3]beaker[.65]---->[.66]bunsen (FAILS)
> bunsen[.66]---->[.65]beaker[.3]---->[.12]piggy (FAILS)
>
of
> the
>> lan?
>
> All of our internal servers live on beaker (mail, samba, etc..) which
> is accessible from the LAN. And, at the moment, we have a proxy server
> set up on beaker for using web, ftp, etc. So it works at the moment.
> But it is quite restrictive.
>
> Again, thanks for your ongoing help.
>
> Trevor.
>

Do a
route add -net 203.1.78.0 netmask 255.255.255.192 gw 203.1.78.65
and delete the host route you have on bunsen, beaker is forwarding
but bunsen doesn't know how to return packets.

JC

Thanks JC! I was so convinced that the problem was with IP forwarding
on beaker, that I didn't see the obvious.

Cheers,
Trevor.

(E-Mail Removed) wrote:
> Thanks JC! I was so convinced that the problem was with IP forwarding
> on beaker, that I didn't see the obvious.
>
> Cheers,
> Trevor.

See you wee reading my posts closely ;-)

Well, somebody was.
[q]
This is the only config that looks off. Is this an artifact of NATing
on beaker? Otherwise, you should have a network route here rather
than
a host route. Ie., 203.1.78.0 (255.255.255.192) rather than
203.1.78.3
(255.255.255.255). That way any packet destined for 203.1.78.0/26
will
have a route via beaker.
# route add -net 203.1.78.0 netmask 255.255.255.192 gw 203.1.78.65
added after 203.1.78.64 I usually add all my GW routes at the end
after all the networks have been entered.
[eq]