Strange Multi-homed Traceroute/Ping failure for some IPs on someroutes

Dear all,

I have a very strange problem in that some of my IPs in my static
block dont seem to be routed to me, some work and some dont. I have a
leased line on 83.111.160.6 (/30 subnet, gw is 83.111.160.5), and my
ISP route an
additional block 83.111.196.56/29 (83.111.196.57 to 83.111.196.62
useable) over the link.

I have a Debian box, and the routed block IP's are setup as aliases. I
have setup the box
to accept ssh and ping for each IP alias.

/etc/network/interfaces auto eth3
iface eth3 inet static
address 83.111.160.6
netmask 255.255.255.252
up ip addr add 83.111.196.57/29 brd 83.111.196.63 dev eth3 label
eth3:0
up ip addr add 83.111.196.58/29 brd 83.111.196.63 dev eth3 label
eth3:1
up ip addr add 83.111.196.59/29 brd 83.111.196.63 dev eth3 label
eth3:2
up ip addr add 83.111.196.60/29 brd 83.111.196.63 dev eth3 label
eth3:3
up ip addr add 83.111.196.61/29 brd 83.111.196.63 dev eth3 label
eth3:4
up ip addr add 83.111.196.62/29 brd 83.111.196.63 dev eth3 label
eth3:5

And here is a snippet from the Shorewall rules config (but i am
positive this isnt a Shorewall issue):

Ping/ACCEPT net $FW
Ping/ACCEPT net $FW:83.111.196.57
Ping/ACCEPT net $FW:83.111.196.58
Ping/ACCEPT net $FW:83.111.196.59
Ping/ACCEPT net $FW:83.111.196.60
Ping/ACCEPT net $FW:83.111.196.61
Ping/ACCEPT net $FW:83.111.196.62

I can ping 83.111.160.6 fine everywhere from any host on the internet,
but I can't ping
all of the routed IP addresses from external hosts. Some IPs work and
some don't. With
Shorewall set to reject icmp and ssh, some of the connection attempts
to IPs that work
are listed as being dropped, but traffic doesn't even seem to hit the
others at all and
no entries are made. This is a multi-ISP configuration with two
providers, however I am
99.999% sure this isn't a Shorewall issue at all for reasons I will
explain below.

Siteuptime.com shows some of its sites able to connect to IPs within
the routed block and
others unable (US sites ok, London failed). I also have a number of
traceroutes from
network-tools.com which I attach to this mail. Some of the IPs within
the routed block
don?t seem to be hitting the firewall at all and are routed off into
space (from reject
logs or lack activity on the ISP ethernet to fibre converter data
transfer LEDs). This
isn't a ping issue either, SSH, SMTP etc do not work on the broken
IPs.

Now here is the strangest thing, I have a couple of servers in the UK
and they have dual
interfaces. On one of the boxes, ping fails from one interface, but
works when ping is
initiated on another, to the same destination host.

**** TRACE FROM MY UK SERVERS ****

[root@stripe ~]# ping 83.111.196.59 -I 85.234.115.64 PING
83.111.196.59 (83.111.196.59)
from 85.234.115.64 : 56(84) bytes of data.
--- 83.111.196.59 ping statistics
--- 4 packets transmitted, 0 received, 100% packet loss, time 3002ms

[root@stripe ~]# ping 83.111.196.60 -I 85.234.115.64 PING
83.111.196.60 (83.111.196.60)
from 85.234.115.64 : 56(84) bytes of data.
64 bytes from 83.111.196.60: icmp_seq=1 ttl=56 time=159 ms 64 bytes
from 83.111.196.60:
icmp_seq=2 ttl=56 time=159 ms
--- 83.111.196.60 ping statistics
--- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt
min/avg/max/mdev =
159.024/159.221/159.418/0.197 ms

[root@stripe ~]# ping 83.111.196.61 -I 85.234.115.64 PING
83.111.196.61 (83.111.196.61)
from 85.234.115.64 : 56(84) bytes of data.
64 bytes from 83.111.196.61: icmp_seq=1 ttl=54 time=148 ms 64 bytes
from 83.111.196.61:
icmp_seq=2 ttl=54 time=148 ms
--- 83.111.196.61 ping statistics
--- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt
min/avg/max/mdev =
148.549/148.615/148.681/0.066 ms

[root@stripe ~]# ping 83.111.196.62 -I 85.234.115.64 PING
83.111.196.62 (83.111.196.62)
from 85.234.115.64 : 56(84) bytes of data.
--- 83.111.196.62 ping statistics
--- 3 packets transmitted, 0 received, 100% packet loss, time 2000ms

[root@stripe ~]# ping 83.111.196.59 -I 85.234.115.115 PING
83.111.196.59 (83.111.196.59)
from 85.234.115.115 : 56(84) bytes of data.
64 bytes from 83.111.196.59: icmp_seq=1 ttl=57 time=149 ms 64 bytes
from 83.111.196.59:
icmp_seq=2 ttl=57 time=158 ms
--- 83.111.196.59 ping statistics
--- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt
min/avg/max/mdev =
149.200/153.985/158.771/4.801 ms

[root@stripe ~]# ping 83.111.196.60 -I 85.234.115.115 PING
83.111.196.60 (83.111.196.60)
from 85.234.115.115 : 56(84) bytes of data.
--- 83.111.196.60 ping statistics
--- 4 packets transmitted, 0 received, 100% packet loss, time 2999ms

[root@stripe ~]# ping 83.111.196.61 -I 85.234.115.115 PING
83.111.196.61 (83.111.196.61)
from 85.234.115.115 : 56(84) bytes of data.
--- 83.111.196.61 ping statistics
--- 3 packets transmitted, 0 received, 100% packet loss, time 2000ms

[root@stripe ~]# ping 83.111.196.62 -I 85.234.115.115 PING
83.111.196.62 (83.111.196.62)
from 85.234.115.115 : 56(84) bytes of data.
64 bytes from 83.111.196.62: icmp_seq=1 ttl=56 time=168 ms 64 bytes
from 83.111.196.62:
icmp_seq=2 ttl=56 time=178 ms
--- 83.111.196.62 ping statistics
--- 2 packets transmitted, 2 received, 0% packet loss, time 1000ms rtt
min/avg/max/mdev =
168.441/173.542/178.644/5.118 ms

Sending from Stripe using interface 85.234.115.64, my IPs
83.111.196.60 and 83.111.196.61
are ok, but .59 and .62 fail. Strangely, sending from Stripe using
interface
85.234.115.115 the opposite is true, .59 and .62 are ok but .60 and .
61 fail! My other
servers fail connecting to .59 and .62.

I have also attatched some more traceroutes from network-tools.com at
the end of this mail showing failures from their servers to .60 and .
61.

I would greatly appreciate any pointers on this issue, I have already
contacted my ISP
and they fail to believe that something is wrong. It would be most
appreciated if others
could let me know if they can contact the above IP addresses, and give
any insight as to what could be the problem.

Many thanks in advance,

Chris


************** NETWORK-TOOLS.COM TRACEROUTE ***************
Ping 83.111.196.59
Timed out
Timed out
Timed out

TraceRoute to 83.111.196.59
Hop (ms) (ms) (ms) IP Address Host name
1 9 16 18 72.249.0.65 -
2 35 25 20 64.129.174.181 64-129-174-181.static.twtelecom.net
3 51 47 45 66.192.242.253 -
4 Timed out Timed out Timed out -
5 280 270 268 195.229.1.186 -
6 263 265 276 194.170.0.154 -
7 Timed out Timed out Timed out -
8 295 318 301 83.111.206.182 -
9 Timed out Timed out Timed out -
10 Timed out Timed out Timed out -

Ping 83.111.196.60
Round trip time to 83.111.196.60: 272 ms
Round trip time to 83.111.196.60: 267 ms
Round trip time to 83.111.196.60: 270 ms
Round trip time to 83.111.196.60: 274 ms
Round trip time to 83.111.196.60: 263 ms
Round trip time to 83.111.196.60: 271 ms
Round trip time to 83.111.196.60: 273 ms
Round trip time to 83.111.196.60: 272 ms
Round trip time to 83.111.196.60: 263 ms
Round trip time to 83.111.196.60: 267 ms
Average time over 10 pings: 269.2 ms

TraceRoute to 83.111.196.60
Hop (ms) (ms) (ms) IP Address Host name
1 12 11 5 72.249.0.65 -
2 11 24 23 64.129.174.181 64-129-174-181.static.twtelecom.net
3 49 46 59 66.192.242.253 -
4 Timed out Timed out Timed out -
5 275 270 272 195.229.1.186 -
6 264 263 263 194.170.0.158 -
7 Timed out Timed out Timed out -
8 274 264 269 83.111.196.60 -

Ping 83.111.196.61
Round trip time to 83.111.196.61: 277 ms
Round trip time to 83.111.196.61: 286 ms
Round trip time to 83.111.196.61: 279 ms
Round trip time to 83.111.196.61: 285 ms
Round trip time to 83.111.196.61: 269 ms
Round trip time to 83.111.196.61: 265 ms
Timed out
Round trip time to 83.111.196.61: 268 ms
Round trip time to 83.111.196.61: 263 ms
Round trip time to 83.111.196.61: 275 ms
Average time over 10 pings: 246.7 ms

TraceRoute to 83.111.196.61
Hop (ms) (ms) (ms) IP Address Host name
1 11 19 22 72.249.0.65 -
2 15 22 16 64.129.174.181 64-129-174-181.static.twtelecom.net
3 60 50 46 66.192.242.253 -
4 Timed out Timed out Timed out -
5 273 284 282 195.229.1.186 -
6 264 263 268 194.170.0.158 -
7 Timed out Timed out Timed out -
8 274 272 271 83.111.196.61 -

Ping 83.111.196.62
Timed out
Timed out
Timed out

TraceRoute to 83.111.196.62
Hop (ms) (ms) (ms) IP Address Host name
1 7 5 5 72.249.0.65 -
2 7 7 12 64.129.174.181 64-129-174-181.static.twtelecom.net
3 62 51 51 66.192.242.253 -
4 Timed out Timed out Timed out -
5 292 272 275 195.229.1.186 -
6 271 272 262 194.170.0.158 -
7 Timed out Timed out Timed out -
8 291 281 279 83.111.206.182 -
9 Timed out Timed out Timed out -
10 Timed out Timed out Timed out -
11 Timed out Timed out Timed out -
12 Timed out Timed out Timed out -


And using the following tool (http://icfamon.dl.ac.uk/cgi-bin/
traceroute.pl) the traceroute seemed to timeout:

traceroute from 193.62.127.224 (icfamon.dl.ac.uk) to 83.111.196.60

traceroute to 83.111.196.60 (83.111.196.60), 30 hops max, 38 byte
packets
1 alan5 (193.62.127.129) 3.099 ms 0.518 ms 0.475 ms
2 gw-fw (193.63.74.131) 0.247 ms 0.210 ms 0.215 ms
3 c-pop (193.63.74.226) 1.033 ms 2.677 ms 0.733 ms
4 193.62.116.18 (193.62.116.18) 1.229 ms 1.119 ms 1.115 ms
5 so-0-1-0.warr-sbr1.ja.net (146.97.42.169) 1.720 ms 1.750 ms
1.736 ms
6 so-3-0-0.lond-sbr3.ja.net (146.97.33.18) 6.720 ms 6.716 ms
6.720 ms
7 195.219.100.17 (195.219.100.17) 7.121 ms 6.825 ms 6.806 ms
8 if-13-0-0-3.mcore3.LDN-London.teleglobe.net (195.219.195.21)
25.400 ms 205.769 ms 8.021 ms
9 Vlan62.icore1.LDN-London.teleglobe.net (195.219.83.1) 15.335 ms
17.154 ms 18.018 ms
10 linx.lon.seabone.net (195.66.224.153) 7.699 ms 8.043 ms 7.779
ms
11 customer-side-etisalat-4-pal9.pal.seabone.net (213.144.181.170)
219.954 ms 217.635 ms 220.075 ms
12 195.229.1.194 (195.229.1.194) 235.358 ms 230.137 ms 230.371 ms
13 194.170.0.158 (194.170.0.158) 293.654 ms 300.541 ms
194.170.0.154 (194.170.0.154) 222.685 ms
14 195.229.245.142 (195.229.245.142) 229.187 ms 221.517 ms 221.703
ms
15 83.111.206.182 (83.111.206.182) 311.977 ms 309.222 ms 311.551
ms

Hi, further to my last post, i tried traceroute from my ISP at home
(this is the same supplier as my leased line) and .60 is working for
me from here to the office (in the same town):

C:\Users\Chris>tracert -d 83.111.196.57

Tracing route to 83.111.196.57 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.1.3
2 4 ms 1 ms 1 ms 192.168.1.1
3 13 ms 13 ms 11 ms 195.229.244.25
4 12 ms 12 ms 12 ms 195.229.245.158
5 13 ms 12 ms 13 ms 83.111.196.57

Trace complete.

C:\Users\Chris>tracert -d 83.111.196.58

Tracing route to 83.111.196.58 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.1.3
2 2 ms 1 ms 2 ms 192.168.1.1
3 13 ms 12 ms 13 ms 195.229.244.25
4 12 ms 13 ms 12 ms 195.229.245.142
5 22 ms 23 ms 22 ms 83.111.206.182
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 ^C
C:\Users\Chris>tracert -d 83.111.196.59

Tracing route to 83.111.196.59 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.1.3
2 2 ms 1 ms 1 ms 192.168.1.1
3 12 ms 13 ms 12 ms 195.229.244.25
4 11 ms 11 ms 12 ms 195.229.245.158
5 24 ms 22 ms 23 ms 83.111.206.182
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * ^C
C:\Users\Chris>tracert -d 83.111.196.60

Tracing route to 83.111.196.60 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.1.3
2 2 ms 1 ms 1 ms 192.168.1.1
3 16 ms 14 ms 13 ms 195.229.244.25
4 12 ms 11 ms 11 ms 195.229.245.142
5 13 ms 13 ms 13 ms 83.111.196.60

Trace complete.

C:\Users\Chris>tracert -d 83.111.196.61

Tracing route to 83.111.196.61 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.1.3
2 2 ms 1 ms 1 ms 192.168.1.1
3 13 ms 13 ms 18 ms 195.229.244.25
4 11 ms 11 ms 11 ms 195.229.245.158
5 13 ms 12 ms 13 ms 83.111.196.61

Trace complete.

C:\Users\Chris>tracert -d 83.111.196.62

Tracing route to 83.111.196.62 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.1.3
2 5 ms 4 ms 4 ms 192.168.1.1
3 14 ms 13 ms 15 ms 195.229.244.25
4 13 ms 13 ms 12 ms 195.229.245.142
5 23 ms 23 ms 23 ms 83.111.206.182
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * ^C
C:\Users\Chris>


Any ideas? My ISP dont want to believe its their fault.