strange routing problem

I have a strange routing problem with one of my hosts in a private
subnet. Basically, I cannot ping or connect to this host (ip
10.0.0.10) from my router or from a Mac (OS X) on the same subnet
unless I first set up a the arp cache manually, connect to the host
and ping the other hosts. (Once the route is set up on the Mac for
example, I can ping the "invisible" host from there, of course).

Here is the netstat output from the three machines (slightly
obfuscated):

router:
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window
irtt Iface
10.0.0.0 * 255.255.255.0 U 0 0
0 eth1
192.168.1.0 * 255.255.255.0 U 0 0
0 eth1
99.99.99.0 * 255.255.255.0 U 0 0 0
eth2
169.254.0.0 * 255.255.0.0 U 0 0
0 eth2
default 99.99.99.1 0.0.0.0 UG 0 0 0
eth2

"invisible" host (10.0.0.10):
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window
irtt Iface
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0
0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0
0 eth0
0.0.0.0 10.0.0.1 0.0.0.0 UG 0 0
0 eth0

mac (10.0.0.200 via wireless DHCP at 10.0.0.254):
Destination Gateway Flags Refs Use Netif
Expire
default 10.0.0.254 UGSc 14 4 en1
10/24 link#5 UCS 3 0 en1
10.0.0.1 0:40:f4:cf:96:32 UHLW 1 218 en1
1142
10.0.0.10 0:c0:49:fa:3c:49 UHLS 1 495 en1
10.0.0.200 127.0.0.1 UHS 2 1715 lo0
10.0.0.254 0:3:93:1e:d6:47 UHLW 13 9 en1
222
10.0.0.255 link#5 UHLWb 1 36 en1
127 127.0.0.1 UCS 0 0 lo0
127.0.0.1 127.0.0.1 UH 10 280557 lo0
169.254 link#5 UCS 0 0 en1


To set up the arp cache on the Mac, I use '/usr/sbin/arp -s 10.0.0.10
00:C0:49:FA:3C:49'

Any input would be greatly appreciated!

cheers,
h.

On Sat, 11 Aug 2007, in the Usenet newsgroup comp.os.linux.networking, in
article <(E-Mail Removed). com>,
(E-Mail Removed) wrote:

>I have a strange routing problem with one of my hosts in a private
>subnet. Basically, I cannot ping or connect to this host (ip
>10.0.0.10) from my router or from a Mac (OS X) on the same subnet
>unless I first set up a the arp cache manually, connect to the host
>and ping the other hosts. (Once the route is set up on the Mac for
>example, I can ping the "invisible" host from there, of course).

Why isn't ARP working? What does a packet sniffer (on _BOTH_ the
invisible host and the Mac) show when the arp-cache is not set
manually and you try to connect? The fact that packets can be
exchanged once the Mac knows about the hardware address suggests
something is screwed up there, but there's not enough details of
how the network is laid out.

>router:
>Kernel IP routing table
>Destination Gateway Genmask Flags MSS Window irtt Iface
>10.0.0.0 * 255.255.255.0 U 0 0 0 eth1
>192.168.1.0 * 255.255.255.0 U 0 0 0 eth1
>99.99.99.0 * 255.255.255.0 U 0 0 0 eth2
>169.254.0.0 * 255.255.0.0 U 0 0 0 eth2
>default 99.99.99.1 0.0.0.0 UG 0 0 0 eth2

1. What is the primary address on eth1? You have routes to both
10.0.0.0/24 and 192.168.1.0/24 and if the other hosts don't know what
the address is, things can get confusing (when 10.0.0.11 tries to figure
out how to reply to 192.168.1.11 or vice-versa).
2. Do you _REALLY_ want a ZeroConf address (169.254.0.0/16) on a public
address interface?

>"invisible" host (10.0.0.10):
>Kernel IP routing table
>Destination Gateway Genmask Flags MSS Window irtt Iface
>10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
>169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
>0.0.0.0 10.0.0.1 0.0.0.0 UG 0 0 0 eth0

OK

>mac (10.0.0.200 via wireless DHCP at 10.0.0.254):
>Destination Gateway Flags Refs Use Netif Expire
>default 10.0.0.254 UGSc 14 4 en1
>10/24 link#5 UCS 3 0 en1
>10.0.0.1 0:40:f4:cf:96:32 UHLW 1 218 en1 1142
>10.0.0.10 0:c0:49:fa:3c:49 UHLS 1 495 en1
>10.0.0.200 127.0.0.1 UHS 2 1715 lo0
>10.0.0.254 0:3:93:1e:d6:47 UHLW 13 9 en1 222
>10.0.0.255 link#5 UHLWb 1 36 en1
>127 127.0.0.1 UCS 0 0 lo0
>127.0.0.1 127.0.0.1 UH 10 280557 lo0
>169.254 link#5 UCS 0 0 en1

Hmmm, are you saying that this is '10.0.0.200' and to reach anything
you need to us a bridge at 10.0.0.254 (which from the Ethers seems to
be something else from Apple)? 0:40:f4 is someone called "Cameo
Communications" from Nashua, NH - but other than that, I've never
heard of them. What is it?

>To set up the arp cache on the Mac, I use '/usr/sbin/arp -s 10.0.0.10
>00:C0:49:FA:3C:49'

That looks correct, but why is this command needed? If 10.0.0.254 is
a bridge, what is it's configuration like?

Old guy

On Aug 12, 2:47 pm, ibupro...@painkiller.example.tld (Moe Trin) wrote:
> On Sat, 11 Aug 2007, in the Usenet newsgroup comp.os.linux.networking, in
> article <1186864716.490851.58...@l22g2000prc.googlegroups. com>,
>
> Why isn't ARP working?

I guess, that is what I am trying to find out Thanks for the input
so far, Old guy! Much appreciated!

> What does a packet sniffer (on _BOTH_ the
> invisible host and the Mac) show when the arp-cache is not set
> manually and you try to connect? The fact that packets can be
> exchanged once the Mac knows about the hardware address suggests
> something is screwed up there, but there's not enough details of
> how the network is laid out.

Maybe I should have left the Mac out of the picture as it just
complicates the matter unnecessarily. The issue also happens on the
router.

Alas, I was not yet able to run a package sniffer while 10.0.0.10 was
invisible to either the router or the Mac. When I tried just now, I
could not ping 10.0.0.10 from 10.0.0.1 ("Destination Host
Unreachable"), so I ssh'd to the Mac (10.0.0.200) to log into
10.0.0.10 from there. When I did (without having to set the arp entry,
btw. I assume this was still cached from the nightly backup script
that runs on the Mac and connects to 10.0.0.10 by first setting the
arp entry.), 10.0.0.10 all of a sudden was reachable from 10.0.0.1. I
guess I can wait until the ARP entry times out again...

> >router:
> >Kernel IP routing table
> >Destination Gateway Genmask Flags MSS Window irtt Iface
> >10.0.0.0 * 255.255.255.0 U 0 0 0 eth1
> >192.168.1.0 * 255.255.255.0 U 0 0 0 eth1
> >99.99.99.0 * 255.255.255.0 U 0 0 0 eth2
> >169.254.0.0 * 255.255.0.0 U 0 0 0 eth2
> >default 99.99.99.1 0.0.0.0 UG 0 0 0 eth2
>
> 1. What is the primary address on eth1? You have routes to both
> 10.0.0.0/24 and 192.168.1.0/24 and if the other hosts don't know what
> the address is, things can get confusing (when 10.0.0.11 tries to figure
> out how to reply to 192.168.1.11 or vice-versa).

I do not know how to set the primary address on a device. For now, I
have disabled the 192.168.1.0 subnet but it did not make any
difference. 10.0.0.10 remained invisible

> 2. Do you _REALLY_ want a ZeroConf address (169.254.0.0/16) on a public
> address interface?

No, I didn't. This was left over from a system upgrade over the
weekend that installed the avahi zeroconf demon. I have meanwhile
disabled it and deleted the route. This made no difference again.

> >mac (10.0.0.200 via wireless DHCP at 10.0.0.254):
> >Destination Gateway Flags Refs Use Netif Expire
> >default 10.0.0.254 UGSc 14 4 en1
> >10/24 link#5 UCS 3 0 en1
> >10.0.0.1 0:40:f4:cf:96:32 UHLW 1 218 en1 1142
> >10.0.0.10 0:c0:49:fa:3c:49 UHLS 1 495 en1
> >10.0.0.200 127.0.0.1 UHS 2 1715 lo0
> >10.0.0.254 0:3:93:1e:d6:47 UHLW 13 9 en1 222
> >10.0.0.255 link#5 UHLWb 1 36 en1
> >127 127.0.0.1 UCS 0 0 lo0
> >127.0.0.1 127.0.0.1 UH 10 280557 lo0
> >169.254 link#5 UCS 0 0 en1
>
> Hmmm, are you saying that this is '10.0.0.200' and to reach anything
> you need to us a bridge at 10.0.0.254 (which from the Ethers seems to
> be something else from Apple)? 0:40:f4 is someone called "Cameo
> Communications" from Nashua, NH - but other than that, I've never
> heard of them. What is it?

yes, the Mac is 10.0.0.200 and is connected via a wireless router
(10.0.0.254), which is an old version Apple Airport. 10.0.0.254 is
connected to the hub that also connects 10.0.0.1 and 10.0.0.10. Again,
the Mac connection is probably just adding some confusion here and I
don't suspect it to be the problem, since 10.0.0.10 is also invisible
from 10.0.0.1.

> >To set up the arp cache on the Mac, I use '/usr/sbin/arp -s 10.0.0.10
> >00:C0:49:FA:3C:49'
>
> That looks correct, but why is this command needed? If 10.0.0.254 is
> a bridge, what is it's configuration like?

10.0.0.254 uses 10.0.0.1 as gateway (255.255.255.0 netmask - could
this be a problem?). It hands out IP addresses in the 10.0.0.201 to
10.0.0.252 range via DHCP. The Mac has 10.0.0.200 manually set.

let me know what other information might be useful! Again, thanks for
any input!!

On Mon, 13 Aug 2007, in the Usenet newsgroup comp.os.linux.networking, in
article <(E-Mail Removed) .com>, Helge Weissig
wrote:

>(Moe Trin) wrote:

>> What does a packet sniffer (on _BOTH_ the invisible host and the
>> Mac) show

>Maybe I should have left the Mac out of the picture as it just
>complicates the matter unnecessarily. The issue also happens on the
>router.

There is a '-arp' option to most ifconfig commands. Could that have
accidentally been set? On all hosts that you can, run the command
/sbin/ifconfig and see if anything interesting shows

[example ~]$ /sbin/ifconfig -eth0
eth0 Link encap:10Mbps Ethernet HWaddr 00:20:AF:C7:3F:F2
inet addr:192.168.1.117 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

Note the 'BROADCAST' flag showing here.

>Alas, I was not yet able to run a package sniffer while 10.0.0.10 was
>invisible to either the router or the Mac. When I tried just now, I
>could not ping 10.0.0.10 from 10.0.0.1 ("Destination Host
>Unreachable"),

The "Host Unreachable" message means that the kernel knows _how_ to
send packets to that destination (not a routing issue), but was unable
to establish a connection - almost always because ARP failed.

>When I did (without having to set the arp entry, btw. I assume this
>was still cached from the nightly backup script that runs on the Mac

1122 Requirements for Internet Hosts - Communication Layers. R.
Braden, Ed.. October 1989. (Format: TXT=295992 bytes) (Updated by
RFC1349, RFC4379) (Also STD0003) (Status: STANDARD)

Section 2.3.2 requires unused ARP entries to time out, and suggests
that this take one minute to occur. Obviously, manually set entries
(arp -s) don't normally time out.

>and connects to 10.0.0.10 by first setting the arp entry.), 10.0.0.10
>all of a sudden was reachable from 10.0.0.1. I guess I can wait until
>the ARP entry times out again...

The packet sniffer is going to be invaluable here.

>> 1. What is the primary address on eth1? You have routes to both
>> 10.0.0.0/24 and 192.168.1.0/24 and if the other hosts don't know
>> what the address is, things can get confusing (when 10.0.0.11 tries
>> to figure out how to reply to 192.168.1.11 or vice-versa).
>
>I do not know how to set the primary address on a device. For now, I
>have disabled the 192.168.1.0 subnet but it did not make any
>difference. 10.0.0.10 remained invisible

If you ran /sbin/ifconfig, this would show the address. The "second"
network might show up as an alias entry (eth0:1) if it existed, but
you don't need one to simply ADD a second route to the routing table.

>> 2. Do you _REALLY_ want a ZeroConf address (169.254.0.0/16) on a
>> public address interface?
>
>No, I didn't. This was left over from a system upgrade over the
>weekend that installed the avahi zeroconf demon. I have meanwhile
>disabled it and deleted the route. This made no difference again.

If you read the Apple proposed draft RFC for avahi (actually
multicastDNS draft-cheshire-dnsext-multicastdns-06.txt), as well as the
RFC (RFC3927) for ZeroConf, both documents contain rather specific
warnings about security issues.

>yes, the Mac is 10.0.0.200 and is connected via a wireless router
>(10.0.0.254), which is an old version Apple Airport. 10.0.0.254 is
>connected to the hub that also connects 10.0.0.1 and 10.0.0.10.

I'm not familiar with the Apple Airport, but assume because the
addresses are on the same (sub)network, that it's operating as a
transparent bridge. Thus, anything on the 10.0.0.1 side of the LAN
is also visible on the 10.0.0.200 of the link.

And you are saying this is a _hub_ not a switch (that could also be
a source of the problem).

>Again, the Mac connection is probably just adding some confusion here
>and I don't suspect it to be the problem, since 10.0.0.10 is also
>invisible from 10.0.0.1.

That's why we need that packet sniffer. ;-)

>10.0.0.254 uses 10.0.0.1 as gateway (255.255.255.0 netmask - could
>this be a problem?). It hands out IP addresses in the 10.0.0.201 to
>10.0.0.252 range via DHCP. The Mac has 10.0.0.200 manually set.

I don't _think_ that's the problem.

Old guy

On Aug 13, 5:47 pm, ibupro...@painkiller.example.tld (Moe Trin) wrote:
> There is a '-arp' option to most ifconfig commands. Could that have
> accidentally been set? On all hosts that you can, run the command
> /sbin/ifconfig and see if anything interesting shows

The ifconfig output does not mention anything about arp but lists
BROADCAST on all machines:

10.0.0.1:
eth1 Link encap:Ethernet HWaddr 00:40:F4:CF:96:32
inet addr:10.0.0.1 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::240:f4ff:fecf:9632/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:466123 errors:0 dropped:0 overruns:0 frame:0
TX packets:629651 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:72688926 (69.3 MiB) TX bytes:394200103 (375.9 MiB)
Interrupt:18 Base address:0x2800

eth2 Link encap:Ethernet HWaddr 00:04:75:77:93:5D
inet addr:99.99.99.99 Bcast:99.99.99.255 Mask:
255.255.255.0
inet6 addr: fe80::204:75ff:fe77:935d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2017441 errors:0 dropped:0 overruns:25 frame:0
TX packets:2161135 errors:0 dropped:0 overruns:0 carrier:0
collisions:8588 txqueuelen:1000
RX bytes:638719397 (609.1 MiB) TX bytes:1027669976 (980.0
MiB)
Interrupt:20 Base address:0x8c00

10.0.0.200:
en1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULT ICAST> mtu 1500
inet6 fe80::230:65ff:fe1f:ed0a%en1 prefixlen 64 scopeid 0x5
inet 10.0.0.200 netmask 0xffffff00 broadcast 10.0.0.255
ether 00:30:65:1f:ed:0a
media: autoselect status: active
supported media: autoselect

10.0.0.10:
eth0 Link encap:Ethernet HWaddr 00:C0:49:FA:3C:49
inet addr:10.0.0.10 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::2c0:49ff:fefa:3c49/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1495872 errors:0 dropped:0 overruns:0 frame:0
TX packets:749861 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1644692076 (1.5 GiB) TX bytes:54482059 (51.9 MiB)
Interrupt:10 Base address:0x8400

> >and connects to 10.0.0.10 by first setting the arp entry.), 10.0.0.10
> >all of a sudden was reachable from 10.0.0.1. I guess I can wait until
> >the ARP entry times out again...
>
> The packet sniffer is going to be invaluable here.

tcpdump shows that all machines on the net see the arp requests,
including 10.0.0.10, although it does not seem to be answering them.
arp lists the route to 10.0.0.10 as incomplete once a connection
attempt was made:

? (10.0.0.10) at (incomplete) on en1 [ethernet]

let me know if you need more info!

On Thu, 16 Aug 2007, in the Usenet newsgroup comp.os.linux.networking, in
article <(E-Mail Removed) .com>, Helge Weissig
wrote:

>(Moe Trin) wrote:

>> There is a '-arp' option to most ifconfig commands. Could that have
>> accidentally been set? On all hosts that you can, run the command
>> /sbin/ifconfig and see if anything interesting shows
>
>The ifconfig output does not mention anything about arp but lists
>BROADCAST on all machines:

That looks fine

>>>and connects to 10.0.0.10 by first setting the arp entry.), 10.0.0.10
>>>all of a sudden was reachable from 10.0.0.1. I guess I can wait until
>>>the ARP entry times out again...
>>
>> The packet sniffer is going to be invaluable here.
>
>tcpdump shows that all machines on the net see the arp requests,
>including 10.0.0.10, although it does not seem to be answering them.

That's the problem. Now, why is this occurring. You don't mention what
distribution or kernel this is, but I don't believe there is a method
other than the 'ifconfig -arp' to ignore ARP packets. You sat that
10.0.0.10 can see the requests - I'm assuming that to mean that you
have run tcpdump on this host. Is this a 'custom' kernel you compiled,
or is it a stock kernel from some distribution?

>arp lists the route to 10.0.0.10 as incomplete once a connection
>attempt was made:
>
>? (10.0.0.10) at (incomplete) on en1 [ethernet]

That's not so much a route as a lookup table of MAC verses IP (and name,
but the question mark means that's unknown too). I was going to suggest
looking at the interrupt counts (cat /proc/interrupts) on 10.0.0.10
before and after a 'ping' is attempted to see if the card is passing
the received stuff up the stack.

At this same time, does the "other hosts" show up in the arp cache on
10.0.0.10 when that other host tries to contact 10.0.0.10? (Receiving
an arp _request_ should set the arp cache on the target - "it's arping
me, so it probably will want to talk to me - I guess I should remember
it's MAC/IP pairing").

Old guy

Hello,

Moe Trin wrote :
>
> I don't believe there is a method
> other than the 'ifconfig -arp' to ignore ARP packets.

- arptables filtering
- ebtables filtering (on a bridge)
- /proc/sys/net/ipv4/conf/*/arp_ignore

On Aug 17, 6:14 pm, ibupro...@painkiller.example.tld (Moe Trin) wrote:
> That's the problem. Now, why is this occurring. You don't mention what
> distribution or kernel this is, but I don't believe there is a method
> other than the 'ifconfig -arp' to ignore ARP packets. You sat that
> 10.0.0.10 can see the requests - I'm assuming that to mean that you
> have run tcpdump on this host. Is this a 'custom' kernel you compiled,
> or is it a stock kernel from some distribution?

ahhh... the kernel!!! Updated last night to FC7 and the problem has
gone away. I had not touched or bothered with this box for a while
since I was just using it as NAS basically. Way back when, when FC4
was the latest and greatest, I seem to have had problems with the
standard kernel support for the GigaBit card I had installed, so I
switched to the netdev kernels (the last one in use was
2.6.16-1.2069_FC4.netdev). At least that's what I remember now.

Thanks again for all the pointers, at least I learned a bit more about
networking!