arping confusion multi-homed private networks

Here's the layout:

Server A supports several devices (single-board computers booting from
the server). These are DHCP/BOOTP over the servers eth1 device, based
at 192.168.2.1, into a subnet 192.168.2.0/24.

Server B does the exact same thing over his eth1, based at
192.168.2.1, also into a subnet 192.168.2.0/24.

The idea is that seach server provides a standalone environment and
isolates the eth1 network and everything is fine. The eth0 and eth1
ports are not routed to each other internally.

Now, the two machines are added to a network, 192.168.1.0/24 using
their respective eth0 portsm say A DHCPs for its eth0 at 192.168.1.10
and B receives 192.168.1.11.

The problem arises when arping happens in the ifup script (RHEL5),
when it seems that eth0 of the first machine to boot responds to
arping that yes, 192.168.2.1 is in use when the second machine boots,
preventing the isolated network from starting properly even though
there should be no connections between these subnets.

Sure, I could make A and B private networks .2.x and .3.x, but then I
have to maintain different configurations and I see no reason I should
have to do so.

Anyone that can offer any insights as to how to prevent this "false"
arping response? More to the point, why does eth0 respond to the
arping when that is not really in eth0's realm?

On Feb 17, 5:39 pm, percy.m...@gmail.com wrote:
> Here's the layout:
>
> Server A supports several devices (single-board computers booting from
> the server). These are DHCP/BOOTP over the servers eth1 device, based
> at 192.168.2.1, into a subnet 192.168.2.0/24.
>
> Server B does the exact same thing over his eth1, based at
> 192.168.2.1, also into a subnet 192.168.2.0/24.
>
> The idea is that seach server provides a standalone environment and
> isolates the eth1 network and everything is fine. The eth0 and eth1
> ports are not routed to each other internally.
>
> Now, the two machines are added to a network, 192.168.1.0/24 using
> their respective eth0 portsm say A DHCPs for its eth0 at 192.168.1.10
> and B receives 192.168.1.11.
>
> The problem arises when arping happens in the ifup script (RHEL5),
> when it seems that eth0 of the first machine to boot responds to
> arping that yes, 192.168.2.1 is in use when the second machine boots,
> preventing the isolated network from starting properly even though
> there should be no connections between these subnets.
>
> Sure, I could make A and B private networks .2.x and .3.x, but then I
> have to maintain different configurations and I see no reason I should
> have to do so.
>
> Anyone that can offer any insights as to how to prevent this "false"
> arping response? More to the point, why does eth0 respond to the
> arping when that is not really in eth0's realm?

Found a solution based on hints in
http://kb.linuxvirtualserver.org/wik...to_disable_ARP

I simply did
sudo sh -c "echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore"
on each machine.

1 in this control file means "reply only if the target IP address is
local address configured on the incoming interface"

On Feb 17, 3:04 pm, percy.m...@gmail.com wrote:
> On Feb 17, 5:39 pm, percy.m...@gmail.com wrote:
>
>
>
> > Here's the layout:
>
> > Server A supports several devices (single-board computers booting from
> > the server). These are DHCP/BOOTP over the servers eth1 device, based
> > at 192.168.2.1, into a subnet 192.168.2.0/24.
>
> > Server B does the exact same thing over his eth1, based at
> > 192.168.2.1, also into a subnet 192.168.2.0/24.
>
> > The idea is that seach server provides a standalone environment and
> > isolates the eth1 network and everything is fine. The eth0 and eth1
> > ports are not routed to each other internally.
>
> > Now, the two machines are added to a network, 192.168.1.0/24 using
> > their respective eth0 portsm say A DHCPs for its eth0 at 192.168.1.10
> > and B receives 192.168.1.11.
>
> > The problem arises when arping happens in the ifup script (RHEL5),
> > when it seems that eth0 of the first machine to boot responds to
> > arping that yes, 192.168.2.1 is in use when the second machine boots,
> > preventing the isolated network from starting properly even though
> > there should be no connections between these subnets.
>
> > Sure, I could make A and B private networks .2.x and .3.x, but then I
> > have to maintain different configurations and I see no reason I should
> > have to do so.
>
> > Anyone that can offer any insights as to how to prevent this "false"
> > arping response? More to the point, why does eth0 respond to the
> > arping when that is not really in eth0's realm?
>
> Found a solution based on hints inhttp://kb.linuxvirtualserver.org/wiki/Using_arp_announce/arp_ignore_t...
>
> I simply did
> sudo sh -c "echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore"
> on each machine.
>
> 1 in this control file means "reply only if the target IP address is
> local address configured on the incoming interface"

This is bad, don't do this. IP addresses belong to machines, not to
interface. You have two machines that each believe they are
192.168.2.1, and you have connected them to each other. If you must do
this, you need to properly configure NAT on the other interface.

DS

> > > Server A supports several devices (single-board computers booting from
> > > the server). These are DHCP/BOOTP over the servers eth1 device, based
> > > at 192.168.2.1, into a subnet 192.168.2.0/24.

> > > Server B does the exact same thing over his eth1, based at
> > > 192.168.2.1, also into a subnet 192.168.2.0/24.

> > > Now, the two machines are added to a network, 192.168.1.0/24 using
> > > their respective eth0 portsm say A DHCPs for its eth0 at 192.168.1.10
> > > and B receives 192.168.1.11.
>
> This is bad, don't do this. IP addresses belong to machines, not to
> interface. You have two machines that each believe they are
> 192.168.2.1, and you have connected them to each other. If you must do
> this, you need to properly configure NAT on the other interface.

I don't know how to setup an IP address without telling it which
interface(s) to run on.

But the networks are not connected. Each of the two machines is multi-
homed, with one private (hidden) network and one shared network
connecting the two servers. The hidden network is never to be accessed
from anywhere, except perhaps from the indivdual server, it only
exists to provide NFS filesystem to diskless computers. No forwarding
takes place between the two networks -- if I need iptables or
something in there providing NAT, fine, but I think NAT should not be
needed. The only thing causing me problems was the arp response.

(diskless special-purpose machines) 192.168.222.x ---- 192.168.222.1
[eth1:SERVERA:eth0] ------+ 192.168.1.yy (dhcp)

|
connection would not normally
exist, except in lab for testing-->
[switch]

|
(diskless special-purpose machines) 192.168.222.x ---- 192.168.222.1
[eth1:SERVERB:eth0] ------+ 192.168.1.zz (dhcp)

Traffic from the eth1 networks should *never* *ever* under any
circumstances be sent over the eth0 network. That's why no
forwarding. If I ran iptables, there would be no connections between
the two sides. Why then does arp have to respond? -- it doesn't and
it seems like a good thing to me that the option exists to have it
*not*.

percy.m...@gmail.com wrote:

> I don't know how to setup an IP address without telling it which
> interface(s) to run on.

This is a quirk in the way IP addresses are setup and configured. It
doesn't change the logical relationship that IP addresses belong to
machines, not interfaces.

> But the networks are not connected. Each of the two machines is multi-
> homed, with one private (hidden) network and one shared network
> connecting the two servers. The hidden network is never to be accessed
> from anywhere, except perhaps from the indivdual server, it only
> exists to provide NFS filesystem to diskless computers. No forwarding
> takes place between the two networks -- if I need iptables or
> something in there providing NAT, fine, but I think NAT should not be
> needed. The only thing causing me problems was the arp response.

You are missing the point. The point is that you have two machines
with the same IP address connected to each other.

> Traffic from the eth1 networks should *never* *ever* under any
> circumstances be sent over the eth0 network. That's why no
> forwarding. If I ran iptables, there would be no connections between
> the two sides. Why then does arp have to respond? -- it doesn't and
> it seems like a good thing to me that the option exists to have it
> *not*.

You seem to have a misunderstanding about the relationships between IP
packets and the networks they travel over. An IP packet takes whatever
network best brings it to its destination based on routing tables and
so on. But an IP packet doesn't belong to the network it just happens
to be travelling over that instant.

Packets that happen to take the eth0 network are still packets to and
from the same machine as packets on other networks.

What would happen if one of the machines decided to contact the other
machine using, as a source address, an address the other machine also
had assigned to it?

DS

> You are missing the point. The point is that you have two machines
> with the same IP address connected to each other.
>

Two firewalls connect to each other on the public side (eth0) and both
happen to decide to use 192.168.1.1 on the private side (eth1) do not
cause this problem. It's not a problem with having the same IP address
on two machines, it has to do with not having full separation between
the different networks. I had thought that by not having forwarding
enabled, there was no need to concern myself with the arp tables.

> What would happen if one of the machines decided to contact the other
> machine using, as a source address, an address the other machine also
> had assigned to it?

Not gonna spontaneously decide to do so. Traffic to the private
network will be sent to that interface, using that interface's primary
address, because it fits the netmask. Same for the public side. The
arps would not be routed through. I figure with forwarding turn off,
there's no routing either. Except for arp, it seems.

Basic statement of my problem I guess: I assume that forwarding off
implies routing is off, so should behave much like having the firewall
at full-stop between the interfaces.

Thanks for the comments.

I also foreget sometimes that Linux uses a weak-end system model with
IPs assigned to the machine as a whole, as you pointed out, rather
than a strong-end system model, where the IP is specifically
associated with an interface.

I normally would have configured iptables and not seen this issue at
all. Soemtimes I forget the dirty details that get swept under the rug
with a few iptables rules.

On Feb 19, 12:16 pm, percy.m...@gmail.com wrote:

> > You are missing the point. The point is that you have two machines
> > with the same IP address connected to each other.

> Two firewalls connect to each other on the public side (eth0) and both
> happen to decide to use 192.168.1.1 on the private side (eth1) do not
> cause this problem.

Right, that's because (we hope) they're properly configured to NAT/
hide the 192.168.1.x addresses. This configuration includes hiding its
own address.

> It's not a problem with having the same IP address
> on two machines, it has to do with not having full separation between
> the different networks. I had thought that by not having forwarding
> enabled, there was no need to concern myself with the arp tables.

Full separation requires something akin to NAT.

> > What would happen if one of the machines decided to contact the other
> > machine using, as a source address, an address the other machine also
> > had assigned to it?

> Not gonna spontaneously decide to do so. Traffic to the private
> network will be sent to that interface, using that interface's primary
> address, because it fits the netmask. Same for the public side. The
> arps would not be routed through. I figure with forwarding turn off,
> there's no routing either. Except for arp, it seems.

This is one of those "fix everything that happens to break"
approaches. It's not a good idea, because you never know when
something else will decide to break. Fix it so that it isn't broken is
the correct approach.

> Basic statement of my problem I guess: I assume that forwarding off
> implies routing is off, so should behave much like having the firewall
> at full-stop between the interfaces.

That's not the problem. The problem has nothing to do with routing.
The problem is that you have two machines with the same IP address
connected to each other. You need to properly configure NAT so that
the machine *cannot* source a packet with a source address onto a
network on which that source address is invalid.

*If* source address X is not valid on network Y, then packets with
source address X must not be sent on network Y. You can stop this with
firewalling, NAT, or some other method. But ARP is just a symptom of
the problem.

DS

> > 1 in this control file means "reply only if the target IP address
> > is local address configured on the incoming interface"

> This is bad, don't do this. IP addresses belong to machines

In the run-out-to-its-limits interpretation of a weak end system
model, yes.

I still find linux's default here troublesome. Perhaps it is simply
that I often want to "know" in/out which interface traffic will flow,
even when I have multiple interfaces connected to the same switch(es).

And while it isn't exactly the same, it often reminds me of the old
Sun behaviour where they would use the same MAC on each interface.

rick jones
--
The computing industry isn't as much a game of "Follow The Leader" as
it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose."
- Rick Jones
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

(E-Mail Removed) wrote:
> Traffic from the eth1 networks should *never* *ever* under any
> circumstances be sent over the eth0 network. That's why no
> forwarding. If I ran iptables, there would be no connections between
> the two sides. Why then does arp have to respond? -- it doesn't and
> it seems like a good thing to me that the option exists to have it
> *not*.

If the "wrong" interface is responding to ARP requests, it implies
that the wrong interface is connected to the same link-level broadcast
domain as the "right" interface. So, ipforwarding enabled or not
(that being up at layer3), unless you have vlans (which would have
precluded the wrong interface from seeing the ARP requests in the
first place) you have a situation where traffic for both IP subnets
are on the same wire, visible to anyone at layer2.

rick jones
--
denial, anger, bargaining, depression, acceptance, rebirth...
where do you want to be today?
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...