tracert from A to B dies just before reaching B -- and vice versa?

I administer two hosts at different providers, 69.72.177.140
(www.peacefire.org) and 67.43.158.218 (www.saltfudge.com). For some
reason they are unable to ping each other or ssh to each other (even
though each host is pingable from apparently everywhere else). As
recently as a few days ago, I was able to ssh from one to the other so
I have no idea what went wrong.

I did a tracert from 69.72.177.140 to 67.43.158.218, and the
traceroute died just before reaching 67.43.158.218. I took this to
mean that the problem was somewhere on the network hosting
67.43.158.218 (perhaps a router blocking incoming connections from
69.72.177.140 for some unknown reason), and reported the problem to
them:

peacefire:/var/www/html# tracert 67.43.158.218
traceroute to 67.43.158.218 (67.43.158.218), 30 hops max, 40 byte
packets
1 jvpsunx01.jvds.com (69.72.218.114) 0.180 ms 0.044 ms 0.034 ms
2 (208.116.63.249) 0.309 ms 0.351 ms 0.390 ms
3 po4.ar1.ewr1.us.nlayer.net (69.31.95.81) 0.852 ms 0.916 ms 0.945 ms
4 xe-2-0-0.cr1.nyc3.us.nlayer.net (69.31.95.181) 0.673 ms 0.655 ms
0.666 ms
5 10ge5-4.fr1.lga.llnw.net (198.32.160.134) 0.737 ms 0.818 ms 0.874 ms
6 tge1-2.fr4.ord.llnw.net (69.28.171.193) 28.542 ms 28.597 ms 28.627
ms
7 ve6.fr3.ord.llnw.net (69.28.172.41) 26.592 ms 26.572 ms 26.637 ms
8 tge1-3.fr4.sjc.llnw.net (69.28.171.66) 71.060 ms 71.030 ms 71.208 ms
9 ve5.fr3.sjc.llnw.net (69.28.171.209) 71.045 ms 70.988 ms 71.051 ms
10 tge1-1.fr4.lax.llnw.net (69.28.171.117) 81.984 ms 82.108 ms 82.156
ms
11 ve6.fr3.lax.llnw.net (69.28.171.205) 71.788 ms 71.996 ms 71.910 ms
12 gig2-8-1000m.cr1.lax1.vpls.net (69.28.144.174) 74.410 ms 74.202 ms
74.223 ms
13 VLAN3001.BR7.SNA1.VPLS.NET (66.186.32.238) 75.231 ms 85.870 ms
85.825 ms
14 * * *


However, the network admins for 67.43.158.218 replied that they tried
doing the reverse -- doing traceroute from 67.43.158.218 back to
69.72.177.140 -- and it showed that the traceroute got almost the
entire distance, but died just before reaching 69.72.177.140:

traceroute to 69.72.177.140 (69.72.177.140), 30 hops max, 38 byte
packets
1 CUSTOMER.VPLS.NET (67.43.158.217) 0.353 ms 0.229 ms 0.210 ms
2 VLAN3001.BR1.LAX1.VPLS.NET (66.186.32.237) 11.060 ms 0.993 ms 0.939
ms
3 VLAN2093.BR2.LAX2.VPLS.NET (74.222.141.22) 1.078 ms 0.998 ms 0.967
ms
4 xe-5-0-0-104.lax22.ip.tiscali.net (77.67.68.97) 1.271 ms 1.161 ms
1.216 ms
5 ge-6-7.car3.LosAngeles1.Level3.net (4.68.111.193) 1.214 ms 1.231 ms
ge-6-3.car3.LosAngeles1.level3.net (4.68.111.177) 5.878 ms
6 vlan99.csw4.LosAngeles1.Level3.net (4.68.20.254) 2.680 ms 1.412 ms
13.384 ms
7 ae-93-93.ebr3.LosAngeles1.Level3.net (4.69.137.45) 6.492 ms 13.486
ms 1.757 ms
8 ae-4.ebr4.Washington1.Level3.net (4.69.132.82) 72.859 ms 68.072 ms
71.274 ms
9 ae-74-74.csw2.Washington1.Level3.net (4.69.134.182) 75.666 ms 67.224
ms 72.386 ms
10 ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 63.010 ms
63.223 ms 63.410 ms
11 ae-4-4.ebr2.Newark1.Level3.net (4.69.132.102) 82.781 ms 85.681 ms
89.871 ms
12 ae-24-52.car4.Newark1.Level3.net (4.68.99.40) 80.261 ms
ae-24-54.car4.Newark1.Level3.net (4.68.99.104) 79.923 ms
ae-24-56.car4.Newark1.Level3.net (4.68.99.168) 79.700 ms
13 WBS-CONNECT.car4.Newark1.Level3.net (4.71.148.26) 75.001 ms 75.034
ms 75.305 ms
14 208.116.63.252 (208.116.63.252) 96.761 ms 80.652 ms 76.418 ms
15 jvpsunx01.jvds.com (69.72.218.114) 75.373 ms 75.117 ms 75.202 ms
16 * * *

This, to me, seems extremely weird. How can the traceroute work
*almost* all the way, but not quite, in both directions? If there
were a firewall on one network that was preventing the traceroute from
completing in one direction, wouldn't it also stop the traceroute in
the other direction before it even got out the door?

Even without knowing what IS causing this, what is a plausible set of
conditions that COULD be causing it?

-Bennett

Hello,

Bennett Haselton a écrit :
>
> This, to me, seems extremely weird. How can the traceroute work
> *almost* all the way, but not quite, in both directions? If there
> were a firewall on one network that was preventing the traceroute from
> completing in one direction, wouldn't it also stop the traceroute in
> the other direction before it even got out the door?

Not if the filtering is only on incoming packets, not outgoing.
E.g. if the firewall on B drops incoming packets from A.
When tracerouting from A to B, UDP packets from A to B are dropped by B
so B does not reply with the expected ICMP "port unreachable".
When tracerouting from B to A, ICMP "port unreachable" packets from A to
B are dropped by B.

Bennett Haselton <(E-Mail Removed)> writes:

>I administer two hosts at different providers, 69.72.177.140
>(www.peacefire.org) and 67.43.158.218 (www.saltfudge.com). For some
>reason they are unable to ping each other or ssh to each other (even
>though each host is pingable from apparently everywhere else). As
>recently as a few days ago, I was able to ssh from one to the other so
>I have no idea what went wrong.

Could be a firewall has recently been erected-- firewalls very very often
throw away pings ( and traceroute packets).


>I did a tracert from 69.72.177.140 to 67.43.158.218, and the
>traceroute died just before reaching 67.43.158.218. I took this to


So where did it die? That is the purpose of traceroute to find out which
location is the last that responds.

>mean that the problem was somewhere on the network hosting
>67.43.158.218 (perhaps a router blocking incoming connections from
>69.72.177.140 for some unknown reason), and reported the problem to
>them:

>peacefire:/var/www/html# tracert 67.43.158.218
>traceroute to 67.43.158.218 (67.43.158.218), 30 hops max, 40 byte
>packets
>1 jvpsunx01.jvds.com (69.72.218.114) 0.180 ms 0.044 ms 0.034 ms
>2 (208.116.63.249) 0.309 ms 0.351 ms 0.390 ms
>3 po4.ar1.ewr1.us.nlayer.net (69.31.95.81) 0.852 ms 0.916 ms 0.945 ms
>4 xe-2-0-0.cr1.nyc3.us.nlayer.net (69.31.95.181) 0.673 ms 0.655 ms
>0.666 ms
>5 10ge5-4.fr1.lga.llnw.net (198.32.160.134) 0.737 ms 0.818 ms 0.874 ms
>6 tge1-2.fr4.ord.llnw.net (69.28.171.193) 28.542 ms 28.597 ms 28.627
>ms
>7 ve6.fr3.ord.llnw.net (69.28.172.41) 26.592 ms 26.572 ms 26.637 ms
>8 tge1-3.fr4.sjc.llnw.net (69.28.171.66) 71.060 ms 71.030 ms 71.208 ms
>9 ve5.fr3.sjc.llnw.net (69.28.171.209) 71.045 ms 70.988 ms 71.051 ms
>10 tge1-1.fr4.lax.llnw.net (69.28.171.117) 81.984 ms 82.108 ms 82.156
>ms
>11 ve6.fr3.lax.llnw.net (69.28.171.205) 71.788 ms 71.996 ms 71.910 ms
>12 gig2-8-1000m.cr1.lax1.vpls.net (69.28.144.174) 74.410 ms 74.202 ms
>74.223 ms
>13 VLAN3001.BR7.SNA1.VPLS.NET (66.186.32.238) 75.231 ms 85.870 ms
>85.825 ms
>14 * * *


>However, the network admins for 67.43.158.218 replied that they tried
>doing the reverse -- doing traceroute from 67.43.158.218 back to
>69.72.177.140 -- and it showed that the traceroute got almost the
>entire distance, but died just before reaching 69.72.177.140:

>traceroute to 69.72.177.140 (69.72.177.140), 30 hops max, 38 byte
>packets
>1 CUSTOMER.VPLS.NET (67.43.158.217) 0.353 ms 0.229 ms 0.210 ms
>2 VLAN3001.BR1.LAX1.VPLS.NET (66.186.32.237) 11.060 ms 0.993 ms 0.939
>ms
>3 VLAN2093.BR2.LAX2.VPLS.NET (74.222.141.22) 1.078 ms 0.998 ms 0.967
>ms
>4 xe-5-0-0-104.lax22.ip.tiscali.net (77.67.68.97) 1.271 ms 1.161 ms
>1.216 ms
>5 ge-6-7.car3.LosAngeles1.Level3.net (4.68.111.193) 1.214 ms 1.231 ms
>ge-6-3.car3.LosAngeles1.level3.net (4.68.111.177) 5.878 ms
>6 vlan99.csw4.LosAngeles1.Level3.net (4.68.20.254) 2.680 ms 1.412 ms
>13.384 ms
>7 ae-93-93.ebr3.LosAngeles1.Level3.net (4.69.137.45) 6.492 ms 13.486
>ms 1.757 ms
>8 ae-4.ebr4.Washington1.Level3.net (4.69.132.82) 72.859 ms 68.072 ms
>71.274 ms
>9 ae-74-74.csw2.Washington1.Level3.net (4.69.134.182) 75.666 ms 67.224
>ms 72.386 ms
>10 ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 63.010 ms
>63.223 ms 63.410 ms
>11 ae-4-4.ebr2.Newark1.Level3.net (4.69.132.102) 82.781 ms 85.681 ms
>89.871 ms
>12 ae-24-52.car4.Newark1.Level3.net (4.68.99.40) 80.261 ms
>ae-24-54.car4.Newark1.Level3.net (4.68.99.104) 79.923 ms
>ae-24-56.car4.Newark1.Level3.net (4.68.99.168) 79.700 ms
>13 WBS-CONNECT.car4.Newark1.Level3.net (4.71.148.26) 75.001 ms 75.034
>ms 75.305 ms
>14 208.116.63.252 (208.116.63.252) 96.761 ms 80.652 ms 76.418 ms
>15 jvpsunx01.jvds.com (69.72.218.114) 75.373 ms 75.117 ms 75.202 ms
>16 * * *

>This, to me, seems extremely weird. How can the traceroute work
>*almost* all the way, but not quite, in both directions? If there
>were a firewall on one network that was preventing the traceroute from
>completing in one direction, wouldn't it also stop the traceroute in
>the other direction before it even got out the door?

>Even without knowing what IS causing this, what is a plausible set of
>conditions that COULD be causing it?

It is your own machine that seems to be causing the problem. If you had a
firewall throwing away all pings from the other machine, nothing would get
through eitehr way. What the evidence is is that one of the machines is
refusing stuff from the other.



>-Bennett

Moe Trin <(E-Mail Removed)> wrote:
> Read the man page for tcpdump.

And when you are done with that, read the manpage for traceroute

> It works by the originator sending out packets (ICMP Type 8 or UDP
> to ports ~34334+) with a time to live of "1". The router that
> receives such a packet decrements the TTL count, sees that it is now
> zero, drops the packet, and MAY send back an ICMP Type 11 Code 0
> "Time Exceeded" error. The sender increments the TTL and repeats -
> with the "next hop" discovering the TTL is zero, lather rinse,
> repeat. When the originating TTL is large enough to reach the
> destination, that host sends back an ICMP Type 0 (echo reply to a
> ICMP type 8), or ICMP Type 3 Code 3 (port unreachable).

Could there be a device in the middle somewhere putting a cap on TTL
values? I guess if it did, that should manifest as the same node
appearing over and over again at "the end" of the built-up route
though...

rick jones
--
firebug n, the idiot who tosses a lit cigarette out his car window
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...