SOS - why does an application work fine with a hub but not a switch box?

Hi,

I would highly appreciate it if someone could help me on this, or give me
some hints.

We are working on a Linux kernel driver that is compiled with the Linux
kernel's 10/100 Ethernet supporting files (pcnet32.c, ns83820.c, etc.), so
the driver running on a target machine can talk to a host machine over the
Ethernet. The target machine's 10/100 port is connected to the 10/100 port
of a host machine that runs the IO stress tests through the Ethernet
connection.

(1) When we connect the target machine with the host machine by using a
network hub, the throughput between the target and the host is around
9.5MB/sec, which has saturated the bandwidth of the 10/100 port (this is
good and what we expected).

(2) When we connect both the target and the host machines by using a switch
box, the throughput drops to 2MB/sec.

(3) When we connect both the target and the host machines directely by using
a crossover Ethernect cable, the throughput is around 1.5MB/sec.

What could be wrong with it? Do we need to modify the original Linux kernel
Ethernet supporting files, or do we need to setup something? We have tried
different switch boxes with different setups (e.g. turn on port fast, etc.),
but didn't get any improvement.

Thanks,


T.

"L" <(E-Mail Removed)> wrote in message
news:bie62v$8eu79$(E-Mail Removed)...

> (1) When we connect the target machine with the host machine by using a
> network hub, the throughput between the target and the host is around
> 9.5MB/sec, which has saturated the bandwidth of the 10/100 port (this is
> good and what we expected).
>
> (2) When we connect both the target and the host machines by using a
switch
> box, the throughput drops to 2MB/sec.
>
> (3) When we connect both the target and the host machines directely by
using
> a crossover Ethernect cable, the throughput is around 1.5MB/sec.
>
> What could be wrong with it? Do we need to modify the original Linux
kernel
> Ethernet supporting files, or do we need to setup something? We have
tried
> different switch boxes with different setups (e.g. turn on port fast,
etc.),
> but didn't get any improvement.

Try locking all endpoints to the appropriate duplex settings. This
sounds like a negotiation problem.

DS

In article <bie8hv$4lv$(E-Mail Removed)>,
David Schwartz <(E-Mail Removed)> wrote:
> Try locking all endpoints to the appropriate duplex settings. This
>sounds like a negotiation problem.

Don Becker's miidiag program might help. See
http://www.scyld.com/diag_index.html

L <(E-Mail Removed)> wrote:
> Thanks for your help, all of you. But I'm not sure if I really understand
> it.
>
> Only two devices (the target and the host) are connected to the hub or the
> switch box. No other devices are connected to the hub or the switch box
> (see the following port status. Port 2 and port 3 are connected to the
> target and the host, respectively).

Right - each host is connected not to each other, but to the hub or switch.
When one host sends a packet intended to ultimatelly reach the other host, the
ethernet card doesn't wiggle voltages at the other host, it does so at the
switch. So it is the switch that the speed and duplex must be coordinated
with.

Think of it this way. You and a friend both speak Spanish and English. If
you ask a third person who only speaks Spanish to relay a message verbally to
the second person, then you'd better give the message in Spanish, even though
your friend can speak English. It's the same principle - each packet gets
recieved by the switch, and it's the job of the switch to relay the packet to
all appropriate destinations.

> When the target and the host are connected directly by a crossover ethernet
> cable, no hub or switch box is used.

In this case, the two units are indeed directly connected, and so can
autonegotiate with each other.

> I was told most hubs used layer 1 and most switch boxes used layer 2. What
> other differences between them?

That's not really true. Layer one in this case is the raw physical cable -
you can't do much without hardware. Both of them deal with layer 2 (ethernet),
switches simply have more inteligence in dealing with them. A google search
should turn up more details.

> Any idea why switch box or direct connection slow down the throughput?
> 3GT Up Enabled Forwarding 100 Full Enabled Not
> Defined
> 4GT Up Enabled Forwarding 100 Full Enabled Not

I'm assuming that these are the ports in use. It looks like they're
currently forced to 100 full. If the hosts are set to auto, autonegotiation
will fail, and so they will come up a 100 half. The duplex mismatch will
cause abysmal performace.

Trust me, you *really* want to read http://www.scyld.com/expert/NWay.html
This will shed much light on the issues you're facing.

--
Frank Sweetser fs at wpi.edu
WPI Network Engineer

Thanks so much for the information. It helps a lot! I have learned quite a
bit these couple of days.

As all of you suggested, the performance issue we found with the switch
boxes is probably autonegotiation related. Here is more info on this issue.

Before I connect both the target and the host to the switch box, the port
status looks like this:

Port Link Admin State Rate/Duplex Flow Ctrl Comments
----------------------------------------------------------------------------
----
3GT Down Enabled Blocking (Auto ) (Auto ) Not
Defined
4GT Down Enabled Blocking (Auto ) (Auto ) Not
Defined

After I connect the target and the host to port 3 and port 4 of the switch
box, the port status changes to:

Port Link Admin State Rate/Duplex Flow Ctrl Comments
----------------------------------------------------------------------------
----
3GT Up Enabled Forwarding (100 Full) (Disabled) Not
Defined
4GT Up Enabled Forwarding (100 Full) (Enabled ) Not
Defined

I guess the Rate/Duplex (100 Full) is the result of the autonegotiation
since I didn't force it. This autonegotiated result is supposed to give me
the best performance. However, the performance drops to 3MB/sec. With a
hub, it's around 9.5MB/sec.

My questions are:
- can autonegotiation always give us the best performance?
- how can I check what speed/duplex the hub uses (can I
use an Ethernet analyzer such as Ethereal to do it)? If I
can check it, then I can force the switch box to use it.
- what does "Flow Control" do? The port status above
shows Flow Ctrl on port 3/4 is disabled/enabled.

Thanks,

T.

"Frank Sweetser" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> L <(E-Mail Removed)> wrote:
> > Thanks for your help, all of you. But I'm not sure if I really
understand
> > it.
> >
> > Only two devices (the target and the host) are connected to the hub or
the
> > switch box. No other devices are connected to the hub or the switch box
> > (see the following port status. Port 2 and port 3 are connected to the
> > target and the host, respectively).
>
> Right - each host is connected not to each other, but to the hub or
switch.
> When one host sends a packet intended to ultimatelly reach the other host,
the
> ethernet card doesn't wiggle voltages at the other host, it does so at the
> switch. So it is the switch that the speed and duplex must be coordinated
> with.
>
> Think of it this way. You and a friend both speak Spanish and English.
If
> you ask a third person who only speaks Spanish to relay a message verbally
to
> the second person, then you'd better give the message in Spanish, even
though
> your friend can speak English. It's the same principle - each packet gets
> recieved by the switch, and it's the job of the switch to relay the packet
to
> all appropriate destinations.
>
> > When the target and the host are connected directly by a crossover
ethernet
> > cable, no hub or switch box is used.
>
> In this case, the two units are indeed directly connected, and so can
> autonegotiate with each other.
>
> > I was told most hubs used layer 1 and most switch boxes used layer 2.
What
> > other differences between them?
>
> That's not really true. Layer one in this case is the raw physical
cable -
> you can't do much without hardware. Both of them deal with layer 2
(ethernet),
> switches simply have more inteligence in dealing with them. A google
search
> should turn up more details.
>
> > Any idea why switch box or direct connection slow down the throughput?
> > 3GT Up Enabled Forwarding 100 Full Enabled Not
> > Defined
> > 4GT Up Enabled Forwarding 100 Full Enabled Not
>
> I'm assuming that these are the ports in use. It looks like they're
> currently forced to 100 full. If the hosts are set to auto,
autonegotiation
> will fail, and so they will come up a 100 half. The duplex mismatch will
> cause abysmal performace.
>
> Trust me, you *really* want to read http://www.scyld.com/expert/NWay.html
> This will shed much light on the issues you're facing.
>
> --
> Frank Sweetser fs at wpi.edu
> WPI Network Engineer

L <(E-Mail Removed)> wrote:
> Thanks so much for the information. It helps a lot! I have learned quite a
> bit these couple of days.
>
> As all of you suggested, the performance issue we found with the switch
> boxes is probably autonegotiation related. Here is more info on this issue.
>
> Before I connect both the target and the host to the switch box, the port
> status looks like this:
>
> Port Link Admin State Rate/Duplex Flow Ctrl Comments
> ----------------------------------------------------------------------------
> ----
> 3GT Down Enabled Blocking (Auto ) (Auto ) Not
> Defined
> 4GT Down Enabled Blocking (Auto ) (Auto ) Not
> Defined
>
> After I connect the target and the host to port 3 and port 4 of the switch
> box, the port status changes to:
>
> Port Link Admin State Rate/Duplex Flow Ctrl Comments
> ----------------------------------------------------------------------------
> ----
> 3GT Up Enabled Forwarding (100 Full) (Disabled) Not
> Defined
> 4GT Up Enabled Forwarding (100 Full) (Enabled ) Not
> Defined

Okay, that does look like the switch is going to 100 full. Now you just need
to make sure that the hosts are both going to 100 full. The output of the
dmesg command should have some kernel messages indicating what the cards
negotiated to.

> I guess the Rate/Duplex (100 Full) is the result of the autonegotiation
> since I didn't force it. This autonegotiated result is supposed to give me
> the best performance. However, the performance drops to 3MB/sec. With a
> hub, it's around 9.5MB/sec.
>
> My questions are:
> - can autonegotiation always give us the best performance?

In theory, yes.

> - how can I check what speed/duplex the hub uses (can I

A hub will always be at half duplex, and should have some status lights to
indicate 100 vs 10.

> use an Ethernet analyzer such as Ethereal to do it)? If I

Nope. Applications like ethereal just put bits on and off of the wire, which
doesn't change for 10 or 100.

> can check it, then I can force the switch box to use it.
> - what does "Flow Control" do? The port status above
> shows Flow Ctrl on port 3/4 is disabled/enabled.

Ethernet level flow control frames, which aren't often used. TCP has it's own
flow control mechanism built in.

--
Frank Sweetser fs at wpi.edu
WPI Network Engineer