preferred way of forcing 1000BasetT, Full Duplex?

What's the preferred way of forcing Broadcom NetXtreme
BCM5703X Gigabit Ethernet Controllers and Intel Corp. 82540EM Gigabit
Ethernet Controllers to 1000BaseT, Full Duplex?

ethtool? miitool? echo to /proc? sysctl.conf? rc script?

This is on an RHEL 3 system.

Thanks!

On Wed, 13 Oct 2004 13:16:57 -0700, Dan Stromberg wrote:

>
> What's the preferred way of forcing Broadcom NetXtreme BCM5703X Gigabit
> Ethernet Controllers and Intel Corp. 82540EM Gigabit Ethernet Controllers
> to 1000BaseT, Full Duplex?

Autonegotiation.

Autonegotiation is mandatory for gigabit, forcing it is contrary to the
standard. Note that it is quite possible to autonegotiate and only offer
one option, this is what 1000BaseT only ethernet cards do. Like this one,
a Broadcom Corporation NetXtreme BCM5704S in an IBM blade server:

b02ld001:~ # ethtool eth0
Settings for eth0:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full

If a link which is supposed to run at gigabit isn't, there's something
wrong which should be investigated. If both ends are gigabit capable but
can also manage other speeds, the result of the negotiation should be
gigabit. It should always be the best both support. There is no such thing
as gigabit half duplex BTW.

Ethtool or passing parameters to the driver module at load time can be
used to force it, but this should never be necessary. I have only seen
such an option in Linux, on other systems I have used with gigabit cards
there is no option to force it, as there should not be. The Linux drivers
which offer the option are technically in the wrong.

Regards, Ian

On Wed, 13 Oct 2004 21:46:22 +0100, Ian Northeast wrote:

> On Wed, 13 Oct 2004 13:16:57 -0700, Dan Stromberg wrote:
>
>>
>> What's the preferred way of forcing Broadcom NetXtreme BCM5703X Gigabit
>> Ethernet Controllers and Intel Corp. 82540EM Gigabit Ethernet Controllers
>> to 1000BaseT, Full Duplex?
>
> Autonegotiation.
>
> Autonegotiation is mandatory for gigabit, forcing it is contrary to the
> standard. Note that it is quite possible to autonegotiate and only offer
> one option, this is what 1000BaseT only ethernet cards do. Like this one,
> a Broadcom Corporation NetXtreme BCM5704S in an IBM blade server:
>
> b02ld001:~ # ethtool eth0
> Settings for eth0:
> Supported ports: [ FIBRE ]
> Supported link modes: 1000baseT/Full
> Supports auto-negotiation: Yes
> Advertised link modes: 1000baseT/Full
> Advertised auto-negotiation: Yes
> Speed: 1000Mb/s
> Duplex: Full
>
> If a link which is supposed to run at gigabit isn't, there's something
> wrong which should be investigated. If both ends are gigabit capable but
> can also manage other speeds, the result of the negotiation should be
> gigabit. It should always be the best both support. There is no such thing
> as gigabit half duplex BTW.
>
> Ethtool or passing parameters to the driver module at load time can be
> used to force it, but this should never be necessary. I have only seen
> such an option in Linux, on other systems I have used with gigabit cards
> there is no option to force it, as there should not be. The Linux drivers
> which offer the option are technically in the wrong.
>
> Regards, Ian

Genuine thanks for providing your information.

But does Cisco's autonegotiation support still leave something to be
desired on 1000BaseT? I heard many times on some Donald Becker mailing
lists that it's best to just leave autonegotiation turned on, but then
Cisco equipment often got it wrong on 100BaseT...

I'd be overjoyed if we could forget about forcing these parameters.

Dan Stromberg wrote:
> On Wed, 13 Oct 2004 21:46:22 +0100, Ian Northeast wrote:
>
>

<...snip...>

>>If a link which is supposed to run at gigabit isn't, there's something
>>wrong which should be investigated. If both ends are gigabit capable but
>>can also manage other speeds, the result of the negotiation should be
>>gigabit. It should always be the best both support. There is no such thing
>>as gigabit half duplex BTW.
>>
>>Ethtool or passing parameters to the driver module at load time can be
>>used to force it, but this should never be necessary. I have only seen
>>such an option in Linux, on other systems I have used with gigabit cards
>>there is no option to force it, as there should not be. The Linux drivers
>>which offer the option are technically in the wrong.
>>
>>Regards, Ian
>
>
> Genuine thanks for providing your information.
>
> But does Cisco's autonegotiation support still leave something to be
> desired on 1000BaseT? I heard many times on some Donald Becker mailing
> lists that it's best to just leave autonegotiation turned on, but then
> Cisco equipment often got it wrong on 100BaseT...
>
> I'd be overjoyed if we could forget about forcing these parameters.
>

on Cisco switches you dont have many options for gigabit. you can set
the duplex but it's best to leave it as "Full" or "Auto".... and maybe
set the flow-control... but you might be better off to leave them both
as "Auto". probably useless parameters anyway!?

it is best practice to set the speed & duplex for 100baseT as
performance may suffer... and some chipsets dont do very well
negotiating and end up negotiating every packet. again, depends on the
chipset. i know some of the DEC21142/3 chipsets often got the
negotiation wrong when a Cisco switch is set at 100/Full.

most host drivers are fixed as "Auto" for gigabit, like Linux and
Digital UNIX (although you can attempt to change negotiation but makes
no difference).

On Thu, 14 Oct 2004 16:04:50 -0700, Dan Stromberg wrote:


> But does Cisco's autonegotiation support still leave something to be
> desired on 1000BaseT? I heard many times on some Donald Becker mailing
> lists that it's best to just leave autonegotiation turned on, but then
> Cisco equipment often got it wrong on 100BaseT...

Perfectly OK at 1000BaseT IME. We have plenty of AIX systems running
gigabit connected to Cisco switches, and AIX only provides for
autonegotiation for gigabit. Although the Linux drivers do offer the
option, we always use autonegotiation for gigabit with them too, and it
always works.

We always fix the 100Mb interfaces to 100BaseTX too.

Regards, Ian

Marco Benton <(E-Mail Removed)> wrote:
> it is best practice to set the speed & duplex for 100baseT as
> performance may suffer... and some chipsets dont do very well
> negotiating and end up negotiating every packet. again, depends on
> the chipset. i know some of the DEC21142/3 chipsets often got the
> negotiation wrong when a Cisco switch is set at 100/Full.

If one side is set to auto and the other full for 100BT, then by
definition they are going to be in mismatch.

How Autoneg is supposed to work:

When both sides of the link are set to autoneg, they will "negotiate"
the duplex setting and select full duplex if both sides can do
full-duplex.

If one side is hardcoded and not using autoneg, the autoneg process
will "fail" and the side trying to autoneg is required by spec to use
half-duplex mode.

If one side is using half-duplex, and the other is using full-duplex,
sorrow and woe is the usual result.

So, the following table shows what will happen given various settings
on each side:

Auto Half Full

Auto Happiness Lucky Sorrow

Half Lucky Happiness Sorrow

Full Sorrow Sorrow Happiness

Happiness means that there is a good shot of everything going well.
Lucky means that things will likely go well, but not because you did
anything correctly Sorrow means that there _will_ be a duplex
mis-match.

When there is a duplex mismatch, on the side running half-duplex you
will see various errors and probably a number of late collisions. On
the side running full-duplex you will see things like FCS errors.
Note that those errors are not necessarily conclusive, they are simply
indicators.

rick jones
--
a wide gulf separates "what if" from "if only"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to raj in cup.hp.com but NOT BOTH...

On Fri, 15 Oct 2004 17:17:20 +0000, Rick Jones wrote:


>
> So, the following table shows what will happen given various settings
> on each side:
>
> Auto Half Full
>
> Auto Happiness Lucky Sorrow
>
> Half Lucky Happiness Sorrow
>
> Full Sorrow Sorrow Happiness

This is a terrifically informative message, and yet, my boss' boss, who
claims to have copious experience with mixing autonegotiation with hosts
that have been forced to Full Duplex, says that it isn't a problem.

He's not an unreasonable guy, but I think I'm going to need a second
opinion confirming this, or perhaps a reference or something.

Anyone else out there had the same experience Rick Jones has related? Is
there a spec somewhere that confirms this? (Sorry Rick, I'm not doing
this just to put you on the spot!)

On a semi-related subject: is there a better (or simply an "additional")
newsgroup or mailing list I should pose this question on?

Thanks!

Dan Stromberg <(E-Mail Removed)> said:
>On Fri, 15 Oct 2004 17:17:20 +0000, Rick Jones wrote:
>
>> So, the following table shows what will happen given various settings
>> on each side:
>>
>> Auto Half Full
>>
>> Auto Happiness Lucky Sorrow
>>
>> Half Lucky Happiness Sorrow
>>
>> Full Sorrow Sorrow Happiness
>
>This is a terrifically informative message, and yet, my boss' boss, who
>claims to have copious experience with mixing autonegotiation with hosts
>that have been forced to Full Duplex, says that it isn't a problem.
>
>He's not an unreasonable guy, but I think I'm going to need a second
>opinion confirming this, or perhaps a reference or something.

Rick has it correct there. Unfortunately references on this seem to be hard
to come by, even though I guess this is specified in some IEEE document.

When a device is set up with autonegotiation, it will:
- send special link-level packets telling which protocol variants it supports
- listen for the said packets from the link peer

Both ends then pick protocol variant supported both by itself and the
link partner, in the order:
- 100-FD
- 100-HD
- 10-FD
- 10-HD

(so, this with 10/100 -speeds; I don't know the details for gigabit speeds)

Then there's the issue what happens when autonegotiation is disabled.
Naturally this'll mean that for the side where autoneg is disabled,
some fixed settings have been chosen. The link partner will initially
listen for the autoneg packets on the wire for a while (and also send
its own autoneg packets), but there apparently is a timeout after which
the side with flexible settings enters a hunt mode. Hunt mode will
search from the lowest speed to the highest, with half-duplex first, and
will settle for the first matching speed. So, it'll end up with matching
speed, and half-duplex. If the other end was forced to full-duplex, there
will be sorrow (unless the autonegotiating side was limited to only
negotiate with full-duplex variants of the speeds -- but then, why wasn't
it set to the matching fixed parameters if it was configured at all).

>On a semi-related subject: is there a better (or simply an "additional")
>newsgroup or mailing list I should pose this question on?

comp.dcom.lans.ethernet and/or comp.dcom.lans.misc should be good
candidates.
--
Wolf a.k.a. Juha Laiho Espoo, Finland
(GC 3.0) GIT d- s+: a C++ ULSH++++$ P++@ L+++ E- W+$@ N++ !K w !O !M V
PS(+) PE Y+ PGP(+) t- 5 !X R !tv b+ !DI D G e+ h---- r+++ y++++
"...cancel my subscription to the resurrection!" (Jim Morrison)

The way that the ethernet standard is written is that if the other end
does not respond to auto-negotiation then assume that it is 10BaseT at
half duplex. This is understandable when you consider that
auto-negotiation was introduced at a time when most ethernets were
shared media ie.: thick coaxial cable at 10Mbps therefore if the other
end of a channel did not respond to auto-negotiation than chances are
that it predates switches and fast ethernet hence the failsafe would be
10 BaseT half duplex.

With modern equipment, if you set the speed or the duplex, then the
auto-negotiation is disabled and unless the other end is similarly set
manually, the other end will assume you are an old 10BaseT half duplex
piece of sh*t and drop to 10BaseT half duplex.

This happens all the time, almost every company I've worked or consulted
for had network administrators who thought setting their switches to 100
Mbps at full duplex was a good thing only to suddenly have a lot of runt
packets and almost in every instance those network admins would blame
the system admins or the NICs the sysadmins purchased. Also, one
popular switch manufacturer whose name starts with C and ends with sco
toss in their own proprietary negotiation before the auto-negotiation so
many NICs (especially Suns) timeout and assume no response before the
C*sco gets around to responding hence 10 Mbps half duplex connections.
The sysadmins or users would complain to the network admins and the
network admins would set the ports to 100 Mbps full duplex which doesn't
help at all and then blame everyone else when in fact they just need to
turn off all the vendor specific crap that they spent five to ten times
more money getting to begin with.

One now rather famous broadband company that's been in the news
frequently since 2001 actually ran with their gigabit building to
building links at 10 Mbps half duplex for at least a year if not longer.
No wonder they went bankrupt, well that and their accounting
practices... They also kept telling me that their OC3 and OC12 links
were not ATM but were IP only among other bits of idiocy.

Regards,
John


Juha Laiho wrote:
> Dan Stromberg <(E-Mail Removed)> said:
>
>>On Fri, 15 Oct 2004 17:17:20 +0000, Rick Jones wrote:
>>
>>
>>>So, the following table shows what will happen given various settings
>>>on each side:
>>>
>>> Auto Half Full
>>>
>>> Auto Happiness Lucky Sorrow
>>>
>>> Half Lucky Happiness Sorrow
>>>
>>> Full Sorrow Sorrow Happiness
>>
>>This is a terrifically informative message, and yet, my boss' boss, who
>>claims to have copious experience with mixing autonegotiation with hosts
>>that have been forced to Full Duplex, says that it isn't a problem.
>>
>>He's not an unreasonable guy, but I think I'm going to need a second
>>opinion confirming this, or perhaps a reference or something.
>
>
> Rick has it correct there. Unfortunately references on this seem to be hard
> to come by, even though I guess this is specified in some IEEE document.
>
> When a device is set up with autonegotiation, it will:
> - send special link-level packets telling which protocol variants it supports
> - listen for the said packets from the link peer
>
> Both ends then pick protocol variant supported both by itself and the
> link partner, in the order:
> - 100-FD
> - 100-HD
> - 10-FD
> - 10-HD
>
> (so, this with 10/100 -speeds; I don't know the details for gigabit speeds)
>
> Then there's the issue what happens when autonegotiation is disabled.
> Naturally this'll mean that for the side where autoneg is disabled,
> some fixed settings have been chosen. The link partner will initially
> listen for the autoneg packets on the wire for a while (and also send
> its own autoneg packets), but there apparently is a timeout after which
> the side with flexible settings enters a hunt mode. Hunt mode will
> search from the lowest speed to the highest, with half-duplex first, and
> will settle for the first matching speed. So, it'll end up with matching
> speed, and half-duplex. If the other end was forced to full-duplex, there
> will be sorrow (unless the autonegotiating side was limited to only
> negotiate with full-duplex variants of the speeds -- but then, why wasn't
> it set to the matching fixed parameters if it was configured at all).
>
>
>>On a semi-related subject: is there a better (or simply an "additional")
>>newsgroup or mailing list I should pose this question on?
>
>
> comp.dcom.lans.ethernet and/or comp.dcom.lans.misc should be good
> candidates.

Dan Stromberg <(E-Mail Removed)> said:
>On Fri, 15 Oct 2004 17:17:20 +0000, Rick Jones wrote:
>> So, the following table shows what will happen given various settings
>> on each side:
>>
>> Auto Half Full
>>
>> Auto Happiness Lucky Sorrow
>>
>> Half Lucky Happiness Sorrow
>>
>> Full Sorrow Sorrow Happiness
>
>This is a terrifically informative message, and yet, my boss' boss, who
>claims to have copious experience with mixing autonegotiation with hosts
>that have been forced to Full Duplex, says that it isn't a problem.
>
>He's not an unreasonable guy, but I think I'm going to need a second
>opinion confirming this, or perhaps a reference or something.

Contradicting my earlier comment on information on this topic being
scarce -- I just found out that numerous IEEE 802 specifications have
been set up for public download at IEEE:
http://standards.ieee.org/getieee802/

The autonegotiation (and fallback to what is called "Parallel
Detection") are documented in clause 28 of IEEE 802.3.

The below quotes should contain the essence of this issue:

: 28.1.4 Compatibility considerations
:
: The Auto-Negotiation function is designed to be completely backwards
: compatible and interoperable with 10BASE-T compliant devices. In
: order to achieve this, a device supporting the Auto-Negotiation
: function must provide the NLP Receive Link Integrity Test function
: as defined in Figure 28.17. The Auto- Negotiation function also
: supports connection to 100BASE-TX and 100BASE-T4 devices without Auto-
: Negotiation through the Parallel Detection function. Connection to
: technologies other than 10BASE-T, 100BASE-TX, or 100BASE-T4 that do
: not incorporate Auto-Negotiation is not supported.
:
: Implementation of the Auto-Negotiation function is optional.
: [...]
:
: 28.2.3.1 Parallel detection function
:
: The Local Device detects a Link Partner that supports Auto-Negotiation
: by FLP Burst detection. The Parallel Detection function allows detection
: of Link Partners that support 100BASE-TX, 100BASE-T4, and/or 10BASE-T,
: but do not support Auto-Negotiation.
: [...]
: NOTE 1.Native 10BASE-T devices will be detected by the NLP Receive Link
: Integrity Test function, an integrated part of the Auto-Negotiation
: function. Hence, Parallel Detection for the 10BASE-T PMA is not required
: or allowed.
:
: NOTE 2.When selecting the highest common denominator through the
: Parallel Detection function, only the halfduplex mode corresponding to
: the selected PMA may automatically be detected.

The priority order of various protocol variants _when using
autonegotiation_ is documented in IEEE 802.3 Annex 28B.3:

: a) 1000BASE-T full duplex
: b) 1000BASE-T
: c) 100BASE-T2 full duplex
: d) 100BASE-TX full duplex
: e) 100BASE-T2
: f) 100BASE-T4
: g) 100BASE-TX
: h) 10BASE-T full duplex
: i) 10BASE-T

.... but, repeating once more, the above is only for resolving the
protocol to use when both ends of the link support negotiation.
With negotiation disabled, the link will end up being half-duplex;
either 100BASE-TX, 100BASE-T4 or 10BASE-T.

Overall, these relevant parts of IEEE 802.3 were fantastic reading,
fully describing how the negotiation phase works (including how
the parties know that _the other party_ has seen and accepted their
capability lists).
--
Wolf a.k.a. Juha Laiho Espoo, Finland
(GC 3.0) GIT d- s+: a C++ ULSH++++$ P++@ L+++ E- W+$@ N++ !K w !O !M V
PS(+) PE Y+ PGP(+) t- 5 !X R !tv b+ !DI D G e+ h---- r+++ y++++
"...cancel my subscription to the resurrection!" (Jim Morrison)