Two routers, same configuration, different result

At work we have several Linux boxes running the most up-to-date
version of debian etch. Our ISP has provided us with 3 public IP
addresses, all coming in on the same wire.

We plan to retire the box acting as our network router and merge
router functionality onto one of the other boxes. Both have two
physical ethernet interfaces, one going to the outside world and one
going to the internal network. As far as I can tell, I have set the
same configurations between the two boxes by just copying /etc files.
It partially works. The new router lets internal machines get IP
addresses via DHCP and lets the internal machines see the outside
world just fine (as did the old router).

Pings from the internal network to the router's internal address work.
Pings from the internal network to any of the router's public
addresses work.
Pings from the external network to the router's "main" public address
work.
Pings from the external network to either of the other two public
addresses do not. They worked on the previous router (which, as far
as I can tell, was configured the same way).
Traffic other than pings behaves the same way as pings.

When I run tcpdump on the new router, it shows all of the incoming
packets, even the ones that are not being properly responded to. So
all packets are getting to the box, but the packets destined for the
second and third public addresses are getting dropped on the floor for
some reason.

At first I thought that the box didn't think that the second and third
addresses were supposed to be local. However, then it seems like
pinging those addresses from an internal machine would fail in the
same way that they are failing from an external one.

Then I thought that perhaps the iptable settings were doing something
bad, so I flushed my iptables completely. (iptables -F ; iptables -F -
t nat ; iptables -F -t mangle). No change. Well, routing stopped, of
course, but ping behavior remained the same. Working from inside,
broken from outside except for the main public address.

Here's my /etc/network/interfaces (with public IP numbers purged):

auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
pre-up iptables-restore < /etc/iptables-save
hwaddress ether 00:17:31:52:F7:3A
address 192.168.1.1
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255

auto eth1
iface eth1 inet static
hwaddress ether 00:09:5B:BC:C8:FD
address [primaryAddress]
netmask [primaryAddressMask]
network [networkAddress]
gateway [gatewayAddress]
up ip addr add dev $IFACE [secondAddress]/29
up ip addr add dev $IFACE [thirdAddress]/29
down ip addr flush dev $IFACE

The results of "ip route list" and "ip addr list" appear to be
correct. I have also confirmed that the output of those two commands
is identical on the new router to what it was on the old router.

I have compared relevant config files and even the relevant sysctl
settings that I know about and they all appear to be the same on the
new router as the old one. I'm really confused here. Not only do I
not know what to do, I've run out of things to investigate. Any
pointers would be greatly appreciated.

Jeff Morrow

On Wed, 06 Jun 2007, in the Usenet newsgroup comp.os.linux.networking, in
article <(E-Mail Removed). com>,
(E-Mail Removed) wrote:

>The new router lets internal machines get IP addresses via DHCP and
>lets the internal machines see the outside world just fine (as did
>the old router).

"see the outside world just fine" means what exactly? Can you (for
example) connect to the ftp server at ibiblio.org or rtfm.mit.edu?

>Pings from the internal network to the router's internal address work.
>Pings from the internal network to any of the router's public
>addresses work.

This only means that you can talk to the router. The "interface" does
not reply, but rather the _kernel_ does, and it replies through the
interface that will reach you. You are seeing the "inside" of the
front door, and thinking you can actually see the front sidewalk/street..
Until you can talk to some OTHER host on the "outside", you haven't
proven that the outside exists.

>Pings from the external network to the router's "main" public address
>work.
>Pings from the external network to either of the other two public
>addresses do not. They worked on the previous router (which, as far
>as I can tell, was configured the same way).

but obviously it isn't - the question is why.

>Traffic other than pings behaves the same way as pings.

But at the top, you say the internal system can "see" the outside world.
Explain.

>So all packets are getting to the box, but the packets destined for the
>second and third public addresses are getting dropped on the floor for
>some reason.

On both boxes - run '/sbin/iptables -L' and see what rules are in place.
Another thing to check is the contents of '/proc/sys/net/ipv4/ip_forward'

>The results of "ip route list" and "ip addr list" appear to be
>correct. I have also confirmed that the output of those two commands
>is identical on the new router to what it was on the old router.

So it's not likely to be a routing/address issue.

>I have compared relevant config files and even the relevant sysctl
>settings that I know about and they all appear to be the same on the
>new router as the old one. I'm really confused here. Not only do I
>not know what to do, I've run out of things to investigate. Any
>pointers would be greatly appreciated.

Compare the resulting rules - does the 'iptables -L' outputs match
exactly? What about the IP forwarding? If all else fails. follow the
boot scripts and see where networking is being set up. Yes, there is
something different - but I can't tell from your description.

Old guy