ARP and two NICS

Hi folks,

Let me start out by saying that the computer in question is running RH
2.6.10-1.770_FC3 and that the system has two NICS. Here is where
things start to get strange :-).

One NIC (eth1, 192.168.1.10) is used for standard networking to the
world at large and all is fine and dandy. The second NIC (eth0,
10.1.1.2) is used as a proprietary network. Let me define what I mean
by proprietary. This NIC communicates with some h/w designed
internally by the company I work for. These h/w modules only
communicate using the link layer, that is no IP layer and they do not
comprehend any ethernet frame other then our own proprietary protocol.
Note that the only reason I assigned an IP address to the eth0 NIC is
because it seemed to be the only way I could get linux to
install/activate the network driver.

OK now for the problem... Originally this machine only had a single
NIC, connected to this proprietary network and everything worked fine.
Recently we decided to connect a number of these systems (computers
with h/w modules) together and to the WWW. Fine... we just add a
second NIC right? Well all of a sudden our proprietary network starts
failing. After a little debug I found that ARP requests that are
occuring on the new network (eth1) are being routed/forwarded/bridged
(not sure of proper term here) to the original net (eth0) and causing
our internal protocol to get confused.

Now for the question... Is there any way I can keep the ARP requests
and any other possible traffic from eth1 off of our proprietary network
eth0? I started reading a lot of documentation on the
ifconfig/route/ip commands and routing in general and frankly I am
overwhelmed by the volume of data and most of it is directed at making
them communicate and not the other way around. My knowledge of routing
etc. is limited at best and the boss is pushing for rapid resolution
(isn't that always the case :-) ).

I checked around for some FAQs but did not come up with anything.
Could someone please gently :-) apply a data filter and direct me to a
FAQ or other DOC that might have what I'm looking for (or if you care
to expound, be my guest :-) ).

Thanks in advance,
Richard

(E-Mail Removed) wrote:
> Hi folks,
>
> Let me start out by saying that the computer in question is running
RH
> 2.6.10-1.770_FC3 and that the system has two NICS. Here is where
> things start to get strange :-).

OK, so you're running FC3 with a 2.6.10 kernel.

> One NIC (eth1, 192.168.1.10) is used for standard networking to the
> world at large and all is fine and dandy. ...

eth1 on 192.168.1.0 net works OK.

> ... The second NIC (eth0,
> 10.1.1.2) is used as a proprietary network. ...

eth0 on 10.x.x.0 net (ie., what's the netmask? /24?).

> ... Let me define what I mean
> by proprietary. This NIC communicates with some h/w designed
> internally by the company I work for. These h/w modules only
> communicate using the link layer, that is no IP layer and they do not
> comprehend any ethernet frame other then our own proprietary
protocol.
> Note that the only reason I assigned an IP address to the eth0 NIC is
> because it seemed to be the only way I could get linux to
> install/activate the network driver.

So, you have some "data" encapsulted in an ethernet frame of a
proprietary nature/format. Ethernet frame is stripped and sent to ...?

> OK now for the problem... Originally this machine only had a single
> NIC, connected to this proprietary network and everything worked
fine.

Ie., eth0 on the 10.x.x.0 net _was_ working OK. The ethernet frame was
properly stripped and sent on to where it needed to go and was
processed correctly.

> Recently we decided to connect a number of these systems (computers
> with h/w modules) together and to the WWW. Fine... we just add a
> second NIC right? ...

Ie., eth1 was added to previously working box that was connected to
10.x.x.0 via eth0. Now, you have a box with two nics on two
nets/segments, eth0 on 10.x.x.0 and eth1 on 192.168.1.0. Have you
turned on ip_forward (ie., /proc/sys/net/ipv4/ip_forward = 1)? This is
needed to "forward" packets from one nic to another one inside the box.

> ... Well all of a sudden our proprietary network starts
> failing. After a little debug I found that ARP requests that are
> occuring on the new network (eth1) are being routed/forwarded/bridged
> (not sure of proper term here) to the original net (eth0) and causing
> our internal protocol to get confused.

Hmmm ... Something here does not make sense. ARPs are broadcast
requests (initially) and will _not_ be forwarded inside the box
_unless_ you installed the bridging modules (which you would be very
aware of), configured the box to do proxy_arp (which I highly doubt),
or you failed to set ip_forward=1 (?). This last is something of a
guess, since I've never installed multiple nics on different networks
_without_ setting ip_forward=1

Mental gymnastics while I try to imagine what would occur if you did
not set ip_forward=1....

Deep in the ip stack, IPs are not _really_ associated with a nic
(despite what configuring a nic would lead you to believe.) A Linux
box will gladly answer that it can accept/process packets for any _IP_
configured on the box.

ARPs are "who has x.x.x.x" IP to MAC translations, but the MAC is only
usable on the directly connected segment (without proxy_arp). Which IP
requests are causing troubles? All ARPs? Just queries for 10.x.x.0?
Which MAC is returned? Any? Where/how are subsequent packets flowing?

At this stage, your propritary code should not be involved at all.
This is kernel/nic/route table config stuff. Now, if the ARPs are
answered with the "incorrect" MAC, then ... ;(

> Now for the question... Is there any way I can keep the ARP requests
> and any other possible traffic from eth1 off of our proprietary
network
> eth0?

The point is that ARP's should _not_ be getting there in the first
place. You should not have to do anything to "stop" broadcasts from
being forwarded (inside the box) from one nic to another. With
ip_forward=0, the two nics, on two networks should be "automagically"
separated, wholly ingnorant of the other's existence.

Except for the Linux habit of saying, "I know about that IP, try this
MAC." Could it be claiming knowledge of 10.x.x.0, but giving out the
eth1 MAC? Time for a packet sniffer -- tcpdump or ethereal.

> ... I started reading a lot of documentation on the
> ifconfig/route/ip commands and routing in general and frankly I am
> overwhelmed by the volume of data and most of it is directed at
making
> them communicate and not the other way around. My knowledge of
routing
> etc. is limited at best and the boss is pushing for rapid resolution
> (isn't that always the case :-) ).

So, you're confused. Join the club It _is_ a lot of picky, picky,
picky stuff to try to get under one hat (red or otherwise.

> I checked around for some FAQs but did not come up with anything.
> Could someone please gently :-) apply a data filter and direct me to
a
> FAQ or other DOC that might have what I'm looking for (or if you care
> to expound, be my guest :-) ).

I'm taking a break from "family crisis (sort-of)" so I'm probably
suffering a major, obvious brain fart just now.

I'm not at all clear what _specific_ symptoms you may be seeing when
you say, "it stopped working right".

Can you post:
$ /sbin/ifconfig -a
$ /sbin/route -n

An ascii art layout of your network (at least a normal vs proprietary
side of a host) might be nice too.

Some info on packet flow on the wire would be nice.

When you say "hardware module" do you mean you have written a driver
for an "attached" piece of hardware? How does it interface with the
nic driver? The kernel? Are you sure (hoping to avoid finding ;out?)
that your "driver" code may be boinking in the kernel or nic drivers?

Obviously, it's pretty tough for us to diagnose any problems with the
"proprietary" part of your network, but ARPs arriving on eth1 should
not, I think, be causing problems. Your box should simply be
responding "Network unreachable" if I understand your setup.

The possible problem I can see is the Linux "blabber-mouth" habit
combined with/part of the automatic insertion of route table entries
for each configured nic. Like I said, probably having a major brain
fart

For some ARP problems/discussions you can try here:
http://linux-ip.net/html/ether-arp.h...rp-suppression
[whole page/doc very handy to have around]

May help to "turn it off" if that's what's needed.

hth,
prg

In article <(E-Mail Removed) .com>,
(E-Mail Removed) <(E-Mail Removed)> wrote:
>Hi folks,
>
>Let me start out by saying that the computer in question is running RH
>2.6.10-1.770_FC3 and that the system has two NICS. Here is where
>things start to get strange :-).
>
>One NIC (eth1, 192.168.1.10) is used for standard networking to the
>world at large and all is fine and dandy. The second NIC (eth0,
>10.1.1.2) is used as a proprietary network. Let me define what I mean
>by proprietary. This NIC communicates with some h/w designed
>internally by the company I work for. These h/w modules only
>communicate using the link layer, that is no IP layer and they do not
>comprehend any ethernet frame other then our own proprietary protocol.
>Note that the only reason I assigned an IP address to the eth0 NIC is
>because it seemed to be the only way I could get linux to
>install/activate the network driver.
>
>OK now for the problem... Originally this machine only had a single
>NIC, connected to this proprietary network and everything worked fine.
>Recently we decided to connect a number of these systems (computers
>with h/w modules) together and to the WWW. Fine... we just add a
>second NIC right? Well all of a sudden our proprietary network starts
>failing. After a little debug I found that ARP requests that are
>occuring on the new network (eth1) are being routed/forwarded/bridged
>(not sure of proper term here) to the original net (eth0) and causing
>our internal protocol to get confused.
>
>Now for the question... Is there any way I can keep the ARP requests
>and any other possible traffic from eth1 off of our proprietary network
>eth0? I started reading a lot of documentation on the
>ifconfig/route/ip commands and routing in general and frankly I am
>overwhelmed by the volume of data and most of it is directed at making
>them communicate and not the other way around. My knowledge of routing
>etc. is limited at best and the boss is pushing for rapid resolution
>(isn't that always the case :-) ).
>
>I checked around for some FAQs but did not come up with anything.
>Could someone please gently :-) apply a data filter and direct me to a
>FAQ or other DOC that might have what I'm looking for (or if you care
>to expound, be my guest :-) ).
>
>Thanks in advance,
>Richard

Try this:
echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter
echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_filter
echo 1 > /proc/sys/net/ipv4/conf/eth1/arp_filter

HTH
johna

(E-Mail Removed) <(E-Mail Removed)> wrote:

> One NIC (eth1, 192.168.1.10) is used for standard networking to the
> world at large and all is fine and dandy. The second NIC (eth0,
> 10.1.1.2) is used as a proprietary network.
....
>
> OK now for the problem... Originally this machine only had a single
> NIC, connected to this proprietary network and everything worked fine.
> Recently we decided to connect a number of these systems (computers
> with h/w modules) together and to the WWW. Fine... we just add a
> second NIC right? Well all of a sudden our proprietary network starts
> failing. After a little debug I found that ARP requests that are
> occuring on the new network (eth1) are being routed/forwarded/bridged
> (not sure of proper term here) to the original net (eth0) and causing
> our internal protocol to get confused.

You have two ethernet segments and you do *not* have
both NICs connected to the *same* ethernet switch or
hub, right? If the linux is is a host with two interfaces
(multihomed host) or a router, you should not see the
ARP request from network 1 on network 0.

The output of ifconfig and "route -n" together with a
little ASCII art figure depicting your network would
be nice.

Ciao, Horst
--
»When pings go wrong (It hurts me too)« E.Clapton/E.James/P.Tscharn

My thanks to all who replyed. Sorry for any confusion...

We can count this as closed. I found the problem. It was the fault of
the application that we use to communicate to the proprietary h/w. The
person that coded the original routine did not 'bind' the socket to the
interface in question. Because the socket was opened as
"socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL))", w/o binding, the
"recv" call was receiving frames from both interfaces not just the one
I was expecting.

Thanks again,
Richard

Shit thats bad... Does eth0 have an IP address? Why would it get ARP
otherwise... dump your ifconfig here? try ifconfig eth0 0.0.0.0...
iptables only works on the ip layer, doesnt it?... Hmmm.. I'm no expert
at this...