Prev: Open Source Programs (SysAdmin and NetAdmin)
Next: installing several FTP servers on single Ubuntu machine
From: unruh on 22 Mar 2010 22:22
I will try guessing below.


On 2010-03-22, David Schwartz <davids(a)webmaster.com> wrote:
> I have a mystery on my hands. I believe I've checked all the obvious
> things and it's quite baffling. The symptom is that a machine
> sometimes does not reply to pings (or any other traffic) addressed to
> an aliased loopback interface. If I log into the machine and force it
> to, say, ping the address that can't reach it (using the aliased
> loopback addressas a source), then magically everything works. For a
> while, anyway, then it breaks again.
>
> Here's the setup in more detail: The machine has interfaces in two /24
> networks, each of which ultimately connects to the same Internet
> gateway. I'll use made up IP addresses:
>
> There are two routers, 1.1.1.1/24 and 1.1.2.1/24. They each handle
> their respective /24s. The machine has an interface on each of these
> LANs, say 1.1.1.5/24 and 1.1.2.5/24. Both LANs run OSPF, and the
> machine has an lo:0 interface numbered 1.1.3.5/32. The machine picks
> up a default route from each router over each LAN and installs both

You cannot have more than one default route. It will confuse the hell
out of the thing and the routers.

> default routes. Each router, in turn, re-advertises its link to
> 1.1.3.5/32, making the machine reachable through the LAN and the
> Internet.


>
> So here's the symptom: A machine (on a remote network) can ping
> 1.1.1.5 and 1.1.2.5, but pings to 1.1.3.5 get no replies. Routing
> looks correct. Analyzing 'tcpdump's on both physical interfaces so

So show us the routing table on that remote machine and on the local
machine.

> 'ping's being received on the 1.1.1.5 interface as they should be, but
> the machine does not appear to be sending replies out either physical
> interface.
>
> The second I ping out from that machine (using 1.1.3.5 as the source)

How do you ping out from a source? The routing tables of your machine
determine what it thinks the source is surely.

> to the machine that can't reach it, the routing works perfectly -- for
> awhile anyway.

So it is setting up its arp tables and never then bothers to try to
route by IP address. It looks up the MAC of that IP address and uses
that to get to your machine, I assume.


>
> The machine has a default route out both interfaces to each router at
> all times (received by OSPF using Quagga). The route to 1.1.3.5

That is just wrong.

> appears to be being advertised correctly across both networks. I've
> made sure rp_filter is off on all interfaces.
>
> default proto zebra metric 1
> nexthop via 1.1.1.1 dev eth0 weight 1
> nexthop via 1.1.2.1 dev eth1 weight 1

What are the routing tables-- route -n

>
> 1.1.1.0/24 dev eth0 proto kernel scope link src 1.1.1.5
> 1.1.2.0/24 dev eth1 proto kernel scope link src 1.1.2.5
>
>
> eth0 Link encap:Ethernet HWaddr <removed>
> inet addr:1.1.1.5 Bcast:1.1.1.255 Mask:255.255.255.0
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>
> eth1 Link encap:Ethernet HWaddr <removed>
> inet addr:1.1.2.5 Bcast:1.1.2.55 Mask:255.255.255.0
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>
> lo Link encap:Local Loopback
> inet addr:127.0.0.1 Mask:255.0.0.0
> UP LOOPBACK RUNNING MTU:16436 Metric:1
>
> lo:0 Link encap:Local Loopback
> inet addr:1.1.3.5 Mask:255.255.255.255
> UP LOOPBACK RUNNING MTU:16436 Metric:1
>
> # Controls IP packet forwarding
> net.ipv4.ip_forward = 1
> # Controls source route verification
> net.ipv4.conf.default.rp_filter = 0
>
> Anyone have a clue what I'm missing?
>
> DS
From: David Schwartz on 22 Mar 2010 23:39
On Mar 22, 7:22 pm, unruh <un...(a)wormhole.physics.ubc.ca> wrote:

> You cannot have more than one default route. It will confuse the hell
> out of the thing and the routers.

Huh? Of course you can. In fact, my case is the canonical case where
you have two default routes. You have a machine on two LANs, both part
of the same WAN. Each LAN has a router with a full table. The machine
can have default routes out both routers.

> So show us the routing table on that remote machine and on the local
> machine.

It's what you'd expect. One router has a route to 1.1.3.5 through
1.1.1.5 and the other has one through 1.1.2.5. The machine has default
routes through 1.1.1.1 and 1.1.2.1 and the obvious network routes.

> > 'ping's being received on the 1.1.1.5 interface as they should be, but
> > the machine does not appear to be sending replies out either physical
> > interface.

> > The second I ping out from that machine (using 1.1.3.5 as the source)

> How do you ping out from a source? The routing tables of your machine
> determine what it thinks the source is surely.

Right, but I override that with 'ping -I' or 'traceroute -s' to force
the machine to source packets with a particular address.

> > to the machine that can't reach it, the routing works perfectly -- for
> > awhile anyway.

> So it is setting up its arp tables and never then bothers to try to
> route by IP address. It looks up the MAC of that IP address and uses
> that to get to your machine, I assume.

No, it can't use ARP to do that since the IP address is not resident
on any Ethernet LAN. It cannot ARP on 1.1.1.0/24 for 1.1.3.5, not can
it ARP on 1.1.2.0/24 for 1.1.3.5. It has to route it to 1.1.1.5 or
1.1.2.5 (either of which it can ARP on the appropriate LAN).

> > The machine has a default route out both interfaces to each router at
> > all times (received by OSPF using Quagga). The route to 1.1.3.5

> That is just wrong.

How so? This is very standard multipath routing and works perfectly in
many installations. This is the first time I've had an issue with it.
(Which I think I've now traced to one router that sometimes stop
responding to ARP requests. We'll see. I've changed it, but enough
time hasn't passed to be sure the problem is gone.)

> > appears to be being advertised correctly across both networks. I've
> > made sure rp_filter is off on all interfaces.
>
> > default  proto zebra  metric 1
> >         nexthop via 1.1.1.1  dev eth0 weight 1
> >         nexthop via 1.1.2.1  dev eth1 weight 1
>
> What are the routing tables-- route -n

What you'd expect. Interface routes for 1.1.1.0/24 and 1.1.2.0/24 out
each interface and two default routes, one to each router (received by
OSPF). Each router has a route to 1.1.3.5/32 through the machine's
interface on its LAN.

DS
From: unruh on 23 Mar 2010 00:27
On 2010-03-23, David Schwartz <davids(a)webmaster.com> wrote:
> On Mar 22, 7:22?pm, unruh <un...(a)wormhole.physics.ubc.ca> wrote:
>
....
>>
>> What are the routing tables-- route -n
>
> What you'd expect. Interface routes for 1.1.1.0/24 and 1.1.2.0/24 out
> each interface and two default routes, one to each router (received by
> OSPF). Each router has a route to 1.1.3.5/32 through the machine's
> interface on its LAN.

Uh, when someone is trying to help you and asks for some information,
like the output to route -n, you might want to supply it. You were the
one who asked for help, not me. You are the one that is confused and
does not understand what is happening, but you refuse to give
information to those who might be able to help.


>
> DS
From: Pascal Hambourg on 23 Mar 2010 03:57
unruh a �crit :
>
> Uh, when someone is trying to help you and asks for some information,
> like the output to route -n, you might want to supply it.

What information would that add ? David already provided the routing
table output from "ip route", and AFAIK the old "route -n" command does
not handle multipath routes (BTW the box doesn't have multiple default
routes but one single multipath default route, which is fine).

After reading completely David's original post (sorry for not doing so
at first, but I was in a hurry) my second thought was that this issue
may involve the routing cache of the box or a router, or maybe an
out-of-memory condition that prevents a device to build the required ARP
or routing cache entry.
From: David Schwartz on 23 Mar 2010 09:56
On Mar 23, 12:57 am, Pascal Hambourg <boite-a-s...(a)plouf.fr.eu.org>
wrote:

> unruh wrote:

>> Uh, when someone is trying to help you and asks for some information,
>> like the output to route -n, you might want to supply it.

That wouldn't work because 'route -n' doesn't support multipath
routes. I provided the routing table. (And, as I said, it's bog
standard. Two network routes, a multipath default route. All routes
are known working.)

Compare:

From route -n:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
0.0.0.0 1.1.1.2 0.0.0.0 UG 1000
0 0 eth0
1.1.1.0 0.0.0.0 255.255.255.0 U 0
0 0 eth0
1.1.2.0 0.0.0.0 255.255.255.0 U 0
0 0 eth1

From 'ip route':
1.1.1.0/24 dev eth0 proto kernel scope link src 1.1.1.5
1.1.2.0/24 dev eth1 proto kernel scope link src 1.1.2.5
default proto zebra metric 1000
nexthop via 1.1.1.1 dev eth0 weight 1
nexthop via 1.1.2.1 dev eth1 weight 1

See how much information is missing from the 'route -n' output?

You don't have the level of routing knowledge needed to help me solve
the problem. But I'm still happy to help you understand what I'm
doing. I assure you, it's perfectly routine multipath routing. The
multipath route installs itself -- the routing process on the machine
receives two equal-cost default routes, one from each router. The idea
is that either router, either LAN cable, either switch, or either
network card can fail, and the machine is still reachable on its
loopback alias address.

From Quagga's 'show ip route':
O>* 0.0.0.0/0 [110/1000] via 1.1.1.1, eth0, 12:33:19
* via 1.1.1.2, eth1, 12:33:19

> After reading completely David's original post (sorry for not doing so
> at first, but I was in a hurry) my second thought was that this issue
> may involve the routing cache of the box or a router, or maybe an
> out-of-memory condition that prevents a device to build the required ARP
> or routing cache entry.

The box is under almost no load and has plenty of memory. The fault
condition persists until something is done to fix it (such as forcing
packets out sources from the loopback address). So I don't think it's
a transient condition on the machine.

It appears that for an unknown reason, one of the routers was not
responding to ARP requests. Although I can't 100% confirm that it
wasn't that the box wasn't sending them for some reason or wasn't
processing the replies. That router was a very old router running
ancient software (a 5509 Sup IIIG RSM), and replacing it with a newer
router seems to have solved the problem. I'll know in a day or so for
sure if the problem doesn't come back. If it does, I'll try to analyze
what ARP requests are going out and coming back and post an update.

DS
First  |  Prev  | 
Pages: 1 2
Prev: Open Source Programs (SysAdmin and NetAdmin)
Next: installing several FTP servers on single Ubuntu machine