Building and configuring reliable linux routers?

I have a linux machine routing between two vlans in our telecom lab at
school. One VLAN is the school's network the other vlan is the lab's
network. The main function of the router is to limit the possiblity of
anything in the lab bringing down the school lab while still allowing
internet access. I've had it setup and running for a few weeks now but
a bad cable knocked the internet out to the lab...

So now I'm looking at ways to increase the redundancy...

My first idea was to configure a second machine, just like the first
and have it automaticly assume the first machines IP if the first was
down. Each machine has 2 ethernet interfaces one on each vlan. Then I
realized I could setup vlan tagging and only use one interface.

Now I'm starting to think I won't need a second machine. Can I bridge
the two ethernet interfaces together and still run vlan tagging? Then
use spanning tree protocol(STP) to only enable one interface at a time?

Right now I've got 3 machines in the rack to play with. Each have two
ethernet interfaces and an ATM oc3 card. I don't have an ATM switch so
right now I just have two of the machines connected back to back. One
of the machines also has a digium T1 card in it.

We have 3 Allied telesyn multilayer switches to work with, 2 24 port
and a 48 port AT-8700XL.

We also have 2 Cisco 2900 switches, 2 827 routers, 4 2500 routers, 7
2600 routers. All the cisco stuff is kinda tied up for labs, there is
another $20k of new cisco gear on order.

The coures being taught in this lab are a joke, the program is missing
an OS and introduction to LANs course so they really have no idea what
they are doing. They jump right into the middle of routing, vlans and
advanced switching. I must admit I didn't get much outa the course, and
I have a background in this stuff.

(E-Mail Removed) wrote:
> I have a linux machine routing between two vlans in our telecom lab at
> school. One VLAN is the school's network the other vlan is the lab's
> network. The main function of the router is to limit the possiblity of
> anything in the lab bringing down the school lab while still allowing
> internet access. I've had it setup and running for a few weeks now but
> a bad cable knocked the internet out to the lab...
>
> So now I'm looking at ways to increase the redundancy...

OK, redundancy where? The Linux router is connected to the school vlan
and to the lab vlan via two separate interfaces, right? Whatever else
it does or does not do, it provides the routing between the two vlans.
The packets coming into the router from the vlans have had their
tagging removed, ie., they are just "regular" ethernet frames. Is
there a third interface or does one of these two (school?) interfaces
provide the upstream route to the internet?

> My first idea was to configure a second machine, just like the first
> and have it automaticly assume the first machines IP if the first was
> down. ...

Are you talking about a _redundant_ Linux router?

> ... Each machine has 2 ethernet interfaces one on each vlan. Then I
> realized I could setup vlan tagging and only use one interface.

You've kinda lost me here. If the Linux box is acting as a _router_
the downstream vlan switches will be feeding it ethernet frames, not
vlan tagged frames. Are you trunking?

> Now I'm starting to think I won't need a second machine. Can I bridge
> the two ethernet interfaces together and still run vlan tagging? ...

If I understand you, ie., two distinct vlans connected to Linux router,
then, no, you can't bridge or switch between the vlans. While
"virtual" they are still treated as separate _lan_ subnets, which is
why you need a _router_ to forward traffic between them.

The only way to have a physical link carry traffic from multiple vlans
is with trunking of some sort. Even when moving traffic on a single
vlan, the tagging only remains within the switch fabric. All frames
sent to non-native (or not trunked) devices will be "plain" ethernet
frames.

> ... Then
> use spanning tree protocol(STP) to only enable one interface at a time?

Has nothing to do with anything. See below.

> Right now I've got 3 machines in the rack to play with. Each have two
> ethernet interfaces and an ATM oc3 card. ...

Are the ethernet interfaces connected to vlan switches? Does the third
interface provide the upstream connection to the internet in some
fashion?

> ... I don't have an ATM switch so
> right now I just have two of the machines connected back to back. One
> of the machines also has a digium T1 card in it.

If you want redundancy at the link level you will have to run multiple
interfaces on the "redundant" capable device. Presumably you have in
mind failover for specific scenarios since the way you set up such
automagic failover depends on the scenario. Where can you accept a
"failure point"? From which direction? Connected to what kind of
device? I'm sensing that you don't need a high availability solution,
just failover to maintain a working pathway.

Someone else today had a question about link level failover and I could
not place the exact doc needed, but some quick snooping has rectified
this brain lapse. Take a look at the bonding.txt file on your Linux
box in /usr/* (/usr/src/linux/Documentation/networking/bonding.txt most
likely) or just look here and download it:
http://sourceforge.net/project/showf...group_id=24692

Read through and "understand" the options available, then test yourself
on a lab setup. This can get a bit tricky with some of the failover w/
aggregation modes. You do want aggregation since you have the extra
link level connection, don't you? ;-)

good luck,
prg

Curently one linux box with two ethernet interfaces is routing between
the two vlans, what I'm proposing to do is enable vlan tagging on both
ports from the switch. Then each physical interface becomes 2 virtual
interfaces, so I basicly have 4 interfaces. Now I want to bridge the
two physical ports and use STP so only one is active at a time. Basicly
failover at level 2, rather than level 3. Will this work to cover stuff
like bad cables?

I also forgot to add that I want to setup some way to monitor the
status, if one of the ports goes down, it needs to be brough to
someones attention to fix the problem before the second link goes down
and the labs offline. I think SNMP is the answer here, but I'm really
having trouble getting my head around it.

I'm just doing this to learn really and to decrease the likely hood of
problems caused by bad cables and people unpluging the wrong cable at
the patch panel. I'm trying to get it somewhat robust because most of
the other students who are working in the lab don't have a clue. The
instructors are not much better and the IT department don't want
anything todo with the lab.

The end of the post really has nothing todo with my problem, just
alittle extra info on the setup.

On 23 Feb 2006 16:46:54 -0800, "(E-Mail Removed)" <(E-Mail Removed)> wrote:

> I'm trying to get it somewhat robust because most of
>the other students who are working in the lab don't have a clue.

Going by your posts today, I wonder how clue you lack?

No context on replies (bad)... You have not yet been told??

Grant.
--
.... The computer scientist, who had listened to all of this said,
"Yes, but where do you think the chaos came from?"

> No context on replies (bad)... You have not yet been told??

I forgot that this is still usenet based and dosn't show post in
threaded form and that alot of people read this as emails. I haven't
posted on usenet in years, I'm use to threaded forums. Yikes I should
have given my post a better reread before submitting, a few typos there
really changed the meaning of what I was saying. I also got off track
from what I orginally was going to post about, hense the odd subject.

Anyway what I didn't make clear in my last post is that the internet is
provided via the school lan. I missed that question when I replyed to
prg.

Anyway to simplify my question...

Can you use 802.1Q vlan tagging and 802.1D bridging with spaning tree
protocol at the same time?

Also would you bridge the two physical interfaces or each pair of
virtual interfaces? I think I just figured out the answer to my own
question...

This is what I want to do
eth0 (connected to switch port 1 /w vlan tagging and STP enabled)
eth0:0 vlan1
eth0:1 vlan2
eth1 (connected to switch port 2 /w vlan tagging and STP enabled)
eth1:0 vlan1
eth1:1 vlan2

br0 eth0:0 & eth1:0 network 10.200.1.0/24 ip 10.200.1.201 gw
10.200.1.254
br1 eth0:1 & eth1:1 network 10.20.30.0/24 ip 10.20.30.99

I'd have to bridge the virtual interfaces so that they could share the
same IP. But then would spaning tree work?

(E-Mail Removed) wrote:
>>No context on replies (bad)... You have not yet been told??
>
>
> I forgot that this is still usenet based and dosn't show post in
> threaded form and that alot of people read this as emails. I haven't
> posted on usenet in years, I'm use to threaded forums. Yikes I should
> have given my post a better reread before submitting, a few typos there
> really changed the meaning of what I was saying. I also got off track
> from what I orginally was going to post about, hense the odd subject.
>
> Anyway what I didn't make clear in my last post is that the internet is
> provided via the school lan. I missed that question when I replyed to
> prg.
>
> Anyway to simplify my question...
>
> Can you use 802.1Q vlan tagging and 802.1D bridging with spaning tree
> protocol at the same time?

yes but perhaps you could see also for channel bonding beetween your switch if
it's support and your server then work with the 2 interfaces on at the same time.

I have this setup in the same context yours

One server 2 Gi interface , one cisco
bonding with eth0 + eth1
ether-channel on cisco
10 Vlan over bonding device on linux

>
> Also would you bridge the two physical interfaces or each pair of
> virtual interfaces? I think I just figured out the answer to my own
> question...
>
> This is what I want to do
> eth0 (connected to switch port 1 /w vlan tagging and STP enabled)
> eth0:0 vlan1
> eth0:1 vlan2
> eth1 (connected to switch port 2 /w vlan tagging and STP enabled)
> eth1:0 vlan1
> eth1:1 vlan2
>
> br0 eth0:0 & eth1:0 network 10.200.1.0/24 ip 10.200.1.201 gw
> 10.200.1.254
> br1 eth0:1 & eth1:1 network 10.20.30.0/24 ip 10.20.30.99

I don't think you can do that like this
>
> I'd have to bridge the virtual interfaces so that they could share the
> same IP. But then would spaning tree work?
>

(E-Mail Removed) wrote:
> Curently one linux box with two ethernet interfaces is routing between
> the two vlans, what I'm proposing to do is enable vlan tagging on both
> ports from the switch. ...

Each interface will be "attached" to one vlan, ie., switch/route
packets for one vlan.

> ... Then each physical interface becomes 2 virtual
> interfaces, so I basicly have 4 interfaces. ...

You're losing me here, I think.

> ... Now I want to bridge the
> two physical ports and use STP so only one is active at a time. Basicly
> failover at level 2, rather than level 3. Will this work to cover stuff
> like bad cables?

Yep, I'm lost. Ascii art showing layout? Level 3? The Linux bonding
driver _is_layer_2_ aggregation/failover. With multiple configuration
strategies.

> I also forgot to add that I want to setup some way to monitor the
> status, if one of the ports goes down, it needs to be brough to
> someones attention to fix the problem before the second link goes down
> and the labs offline. I think SNMP is the answer here, but I'm really
> having trouble getting my head around it.

There are several ways to accomplish this. The way you choose will
depend on your setup, network layout, and equipment. See monitoring
below.

> I'm just doing this to learn really and to decrease the likely hood of
> problems caused by bad cables and people unpluging the wrong cable at
> the patch panel. I'm trying to get it somewhat robust because most of
> the other students who are working in the lab don't have a clue. The
> instructors are not much better and the IT department don't want
> anything todo with the lab.
>
> The end of the post really has nothing todo with my problem, just
> alittle extra info on the setup.

If you want to run "complex" vlan switch fabrics _and_ use SPT you will
need something more than vanilla STP. STP was designed for bridges and
you will need something that is _port_ aware and not just _device_
aware. Ie., what to do if a single vlan port goes down while the other
vlans on the switch are running fine? You might start with something
like this:

http://www.google.com/search?num=50&q=vlan+failover+stp

Monitoring possiblities in addition to your switch equipment
facilities:
http://net-snmp.sourceforge.net/
http://people.ee.ethz.ch/~oetiker/webtools/mrtg/
http://www.cacti.net/
http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/
http://www.nagios.org/about/

If I understood the descriptions in your other posts re: bridging
between interfaces, you cannot _bridge_ between vlans just as you
cannot bridge between subnets. Trunking just allows multiple vlans to
travel along a common path (eg. along a backbone segment). There _are_
layer 3 switches (ie., asic devices) that can provide _basic_ routing
functionality between vlans, but they ain't cheap (talking Cisco
Catalyst eg.) We won't even mention MPLS.

Many people implement failover at the distribution and/or backbone
level (Cisco speak). Especailly important in your setup, I would think
failover _with_ aggregation at the backbone (internet access plus ?)
level would be the first thing to get working well. The bonding driver
may be all you need, _if_ you Linux boxes are on the backbone or attach
directly to it.

At the next lower level (distribution) you _may_ want some failover
redundancy using STP and/or the bonding driver. It will depend on what
access you're trying to protect, the attached equipment and the
location of the Linux box(es) in the path.

BTW, IME failing cables may indicate a poor or aging installation.
Keep an eye out and start thinking where your cabling problems are
likely to "progress". If you're really lucky, the failed cable was a
"freak" accident, unique and not indicative.

good luck,
prg

prg wrote:
> (E-Mail Removed) wrote:
> > Curently one linux box with two ethernet interfaces is routing between
> > the two vlans, what I'm proposing to do is enable vlan tagging on both
> > ports from the switch. ...
>
> Each interface will be "attached" to one vlan, ie., switch/route
> packets for one vlan.
>
> > ... Then each physical interface becomes 2 virtual
> > interfaces, so I basicly have 4 interfaces. ...
>
> You're losing me here, I think.

Each physical interface will be on both vlans using vlan tagging. I
then use vconfig to create two new virtual interfaces for each physical
interface to seperate the two vlans.


> > ... Now I want to bridge the
> > two physical ports and use STP so only one is active at a time. Basicly
> > failover at level 2, rather than level 3. Will this work to cover stuff
> > like bad cables?
>
> Yep, I'm lost. Ascii art showing layout? Level 3? The Linux bonding
> driver _is_layer_2_ aggregation/failover. With multiple configuration
> strategies.
>
> > I also forgot to add that I want to setup some way to monitor the
> > status, if one of the ports goes down, it needs to be brough to
> > someones attention to fix the problem before the second link goes down
> > and the labs offline. I think SNMP is the answer here, but I'm really
> > having trouble getting my head around it.
>
> There are several ways to accomplish this. The way you choose will
> depend on your setup, network layout, and equipment. See monitoring
> below.
>
> > I'm just doing this to learn really and to decrease the likely hood of
> > problems caused by bad cables and people unpluging the wrong cable at
> > the patch panel. I'm trying to get it somewhat robust because most of
> > the other students who are working in the lab don't have a clue. The
> > instructors are not much better and the IT department don't want
> > anything todo with the lab.
> >
> > The end of the post really has nothing todo with my problem, just
> > alittle extra info on the setup.
>
> If you want to run "complex" vlan switch fabrics _and_ use SPT you will
> need something more than vanilla STP. STP was designed for bridges and
> you will need something that is _port_ aware and not just _device_
> aware. Ie., what to do if a single vlan port goes down while the other
> vlans on the switch are running fine? You might start with something
> like this:
>
> http://www.google.com/search?num=50&q=vlan+failover+stp
>
> Monitoring possiblities in addition to your switch equipment
> facilities:
> http://net-snmp.sourceforge.net/
> http://people.ee.ethz.ch/~oetiker/webtools/mrtg/
> http://www.cacti.net/
> http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/
> http://www.nagios.org/about/
>
> If I understood the descriptions in your other posts re: bridging
> between interfaces, you cannot _bridge_ between vlans just as you
> cannot bridge between subnets. Trunking just allows multiple vlans to
> travel along a common path (eg. along a backbone segment). There _are_
> layer 3 switches (ie., asic devices) that can provide _basic_ routing
> functionality between vlans, but they ain't cheap (talking Cisco
> Catalyst eg.) We won't even mention MPLS.

This is like router on a stick but using two interfaces, I'm not
bridging between the two different vlans, I'm bridging between the same
vlan on two different ports.

> Many people implement failover at the distribution and/or backbone
> level (Cisco speak). Especailly important in your setup, I would think
> failover _with_ aggregation at the backbone (internet access plus ?)
> level would be the first thing to get working well. The bonding driver
> may be all you need, _if_ you Linux boxes are on the backbone or attach
> directly to it.

I don't have any direct access to the campus backbone, there is still a
single point of failour between the switch and the school lan that I
cannot do anything about. IT department politics... Atleast if that
fails its not my fault.

> At the next lower level (distribution) you _may_ want some failover
> redundancy using STP and/or the bonding driver. It will depend on what
> access you're trying to protect, the attached equipment and the
> location of the Linux box(es) in the path.
>
> BTW, IME failing cables may indicate a poor or aging installation.
> Keep an eye out and start thinking where your cabling problems are
> likely to "progress". If you're really lucky, the failed cable was a
> "freak" accident, unique and not indicative.

This is a telecom lab, and everything is out in the open, students are
constantly repatching things and yanking on cables and what not. Most
of the students don't have the common sense to check the link lights
when something is not working.

>
> good luck,
> prg