non-Spanning Tree ways of tolerating loops

Rahul wrote:

> But if I put in routers and increase my hops I might as well stay with
> the original switched network anyways with all servers on a single
> switched subnet with no loops. If I have no loops I need no routers. I
> wanted loops to minimise hops.

If your switches are stackable, would it help to daisy chain them all so you
have a single big logical switch?

Rahul <(E-Mail Removed)> wrote:
> David Schwartz <(E-Mail Removed)> wrote in
> news:21b348f1-9caa-4acd-9dac-(E-Mail Removed):

> > Exactly how critical are latencies?

> Very critical. More critical than anything else. More critical than
> bandwidth. More critical than reliability or redundancy. Don't need
> any advanced switch magic. No QoS, no security, no link aggregation
> etc. Just fast packet switching.

Have you disabled interrupt coalescing in your NICs? Are you still
sending traffic encapsulated in IP datagrams?

> > They are essentially the same. Just make sure you get a routing
> > switch that can route at wire speed.

> If a switch can switch at wire speed does it imply that it can route
> at wire speed too?

No. Or perhaps more precisely, you should not ass-u-me it will even if
99 switching routers out of 10 do (deliberate phrasing on my part).

> >> > IMO, large scale switching is a bad idea anyway. You don't want
> >> > 1,500 hosts in the same Ethernet broadcast domain.

> What if my traffic is such that broadcasts are not a huge fraction of
> traffic?

If STP is disabled, and there is a loop, a single broadcast will ruin
your entire day.

rick jones
you still may want to ask in comp.dcom.lans.ethernet ...
--
oxymoron n, commuter in a gas-guzzling luxury SUV with an American flag
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Rick Jones <(E-Mail Removed)> wrote in news:hm6gld$uh8$2
@usenet01.boi.hp.com:

> Rahul <(E-Mail Removed)> wrote:
>> David Schwartz <(E-Mail Removed)> wrote in
>> news:21b348f1-9caa-4acd-9dac-b095174702f6
@s36g2000prh.googlegroups.com:

>> Very critical. More critical than anything else. More critical than
>> bandwidth. More critical than reliability or redundancy. Don't need
>> any advanced switch magic. No QoS, no security, no link aggregation
>> etc. Just fast packet switching.
>
> Have you disabled interrupt coalescing in your NICs?

I think I had disabled it. But let me check my test settings.


>Are you still
> sending traffic encapsulated in IP datagrams?

Sorry, I don't understand that. Is that a setting one does on the NIC's?
Or applications?

>
>> If a switch can switch at wire speed does it imply that it can route
>> at wire speed too?
>
> No. Or perhaps more precisely, you should not ass-u-me it will even if
> 99 switching routers out of 10 do (deliberate phrasing on my part).

How does one fine? Reading most "switch" documentation, I find no mention
of "routing" speed?


>> >> > IMO, large scale switching is a bad idea anyway. You don't want
>> >> > 1,500 hosts in the same Ethernet broadcast domain.
>
>> What if my traffic is such that broadcasts are not a huge fraction of
>> traffic?
>
> If STP is disabled, and there is a loop, a single broadcast will ruin
> your entire day.

Right. Agreed. But I thougt David's objection was to "large scale single
subnet switching" per se. Even with STP enabled. Trying to find out if
there is a strong reason for that other than broadcasts.

As an aside: Wouldn't it make a lot of sense to use STP on broadcast
packets alone? STP is trying to solve the loop-porblem which is specific
to broadcasts. Why use a hatchet when one can do with a scalpel?

>
> rick jones
> you still may want to ask in comp.dcom.lans.ethernet ...

Willdo. I'm posting there. Thanks for the lead!

--
Rahul

Rick Jones <(E-Mail Removed)> wrote in news:hm6gld$uh8$2
@usenet01.boi.hp.com:

> Have you disabled interrupt coalescing in your NICs? Are you still
> sending traffic encapsulated in IP datagrams?
>

In any case that's a NIC-end constant isn't it? THe number of switch hops
and the additive delay is unrelated to coalescing, right?

--
Rahul

Rahul <(E-Mail Removed)> wrote:
> Rick Jones <(E-Mail Removed)> wrote in news:hm6gld$uh8$2
> @usenet01.boi.hp.com:

> > Are you still sending traffic encapsulated in IP datagrams?

> Sorry, I don't understand that. Is that a setting one does on the NIC's?
> Or applications?

Are your applications sending and receiving Ethernet frames directly
(no IP), or are the communicating over AF_INET/AF_INET6 sockets?

> > No. Or perhaps more precisely, you should not ass-u-me it will even if
> > 99 switching routers out of 10 do (deliberate phrasing on my part).

> How does one fine? Reading most "switch" documentation, I find no
> mention of "routing" speed?

If the switch advertises the ability to route, and their public specs
don't include mention of routing speed then email to the sales/info
alias would be my next suggestion.

> Right. Agreed. But I thougt David's objection was to "large scale
> single subnet switching" per se. Even with STP enabled. Trying to
> find out if there is a strong reason for that other than broadcasts.

Given that one can probably ass-u-me a given node will generate some
number of broadcasts per second - either direct IP broadcasts, or more
likely ARP requests then as the number of nodes goes up the number of
broadcast packets goes up. Unless each additional node actually
speaks with all the other nodes, that means the proportion of traffic
a node receives which is otherwise uninteresting broadcast increases.

> As an aside: Wouldn't it make a lot of sense to use STP on broadcast
> packets alone? STP is trying to solve the loop-porblem which is
> specific to broadcasts. Why use a hatchet when one can do with a
> scalpel?

Broadcasts are not the only frames which might go out more than one
port. Until the switch knows the source port for a given MAC, frames
to that destination will be "flooded" through the fabric too, even
though they may very well be unicast frames. So, in your terms, while
a hatchet may indeed be overkill, a carving knive is still required
over a scalpel

rick jones
--
I don't interest myself in "why." I think more often in terms of
"when," sometimes "where;" always "how much." - Joubert
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Rahul <(E-Mail Removed)> wrote:
> Rick Jones <(E-Mail Removed)> wrote in news:hm6gld$uh8$2
> @usenet01.boi.hp.com:

> > Have you disabled interrupt coalescing in your NICs? Are you
> > still sending traffic encapsulated in IP datagrams?

> In any case that's a NIC-end constant isn't it? THe number of switch
> hops and the additive delay is unrelated to coalescing, right?

Yes, I was simply responding to your assertion that latency was
"everything" for your applications and perhaps indirectly suggesting
that there may be considerations beyond the hop count.

rick jones
--
No need to believe in either side, or any side. There is no cause.
There's only yourself. The belief is in your own precision. - Joubert
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Rick Jones <(E-Mail Removed)> wrote in news:hm6otb$12v$2
@usenet01.boi.hp.com:

> Yes, I was simply responding to your assertion that latency was
> "everything" for your applications and perhaps indirectly suggesting
> that there may be considerations beyond the hop count.
>

AH! Got it. Well, thanks! Yes, latency is very important. And improving
latency is a harder battle than bandwidth I've found out.

Yes, I've explored other options. THe most attractive ones are RDMA / iwrap
which try to totally work around the TCP stack and offload to the adapter.
That makes coalascance and other considerations moot. I think.

The other option is to go with Mellanox, Infiniband etc. but then those are
totally different, non ethernet options.

My goal is to try and do the best within regular ethernet.

--
Rahul

Rick Jones <(E-Mail Removed)> wrote in news:hm6oqt$12v$1
@usenet01.boi.hp.com:

> Are your applications sending and receiving Ethernet frames directly
> (no IP), or are the communicating over AF_INET/AF_INET6 sockets?

I'll try and find out! Not sure.

> If the switch advertises the ability to route, and their public specs
> don't include mention of routing speed then email to the sales/info
> alias would be my next suggestion.

"advertizes the ability to route". That's the key. I may have been
looking at the wrong switches.

>> As an aside: Wouldn't it make a lot of sense to use STP on broadcast
>> packets alone? STP is trying to solve the loop-porblem which is
>> specific to broadcasts. Why use a hatchet when one can do with a
>> scalpel?
>
> Broadcasts are not the only frames which might go out more than one
> port. Until the switch knows the source port for a given MAC, frames
> to that destination will be "flooded" through the fabric too, even
> though they may very well be unicast frames. So, in your terms, while
> a hatchet may indeed be overkill, a carving knive is still required
> over a scalpel

Ok, point taken, craving knife it will be then! Well, I think I should
explain a little more of my specific application for context:

This is a High Performance Compute cluster. So the switch topology
rremains very static for weeks or months. The MACs attached to each
switch port rarely change.

Other than the initial startup when the switch is still discovering what
MAC needs to be pushed through what port unicast packets should always go
out through specific ports. One creative way might be to startup using
STP and then disable STP during stable ops. (IFF the broadcasts could be
dealt with seperately, just fantasy.)

--
Rahul

David Schwartz <(E-Mail Removed)> wrote:
> routing will require some modifications to data in the packet while
> switching doesn't. So some ASICs may perform differently.

I ass-u-me you mean the ethernet headers will change because there
will be a new ethernet header to encapsulate the IP datagram. That
being the case, probably best to say so because "modifications to data
in the packet" will lead at least some to ass-u-me you mean it might
modify the user's data.

> > To me it's just one feature that routers will be better at: ability-to-
> > tolerate loops.

> Right. Routers can tolerate loops without a problem.

Well... the routing protocols they use strive to avoid loops, and the
TTL field of the IP datagram means that when there are loops the IP
datagram does eventually get removed from the network. I'm not sure
I'd go so far as to say that is "without a problem" - routing loops
are still badness - they preclude end-to-end connectivity.

rick jones
--
No need to believe in either side, or any side. There is no cause.
There's only yourself. The belief is in your own precision. - Joubert
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

> Are there ways other than Spanning Tree Protocol to handle loops in
> a switching network. For reasons of low latency I want to minimize
> the number of switch hops required peer-to-peer on my topology.
> The best way seems to be a mesh or looped network topology.
> Unfortunately, STP just logically removes physical loops. Are there
> ways to actually use loops to get shortest paths but do away with the
> negatives like broadcast storms?

IIUC you're talking a network of about 300 machines so you could have
e.g. 30 switches with 16-port each and connect those switches to one
32-port master switch. That results in 4-hops max between any
two machines.

The minimum number of hops between two machines is 2 if you don't
connect machines directly to each other and 3 hops if you don't have
a single switch large enough to connect all your machines. So IIUC
you'd like to try and get down to a 3-hops-max topology. I guess you
could do that with 10 switches of 64-ports each, where each switch is
connected to 32 machines and to the other 9 switches. But then indeed
the STP will get you right back to 4-hops-max.

If I were you, I'd try and figure out whether the routing performance of
such large switches is competitive with their switching performance and
then decide whether going from 4 to 3 hops is worth the hassle.


Stefan