1000mbps connection problem between Linux and Solaris

Hello All,

I am facing this weird problem. Hope someone here will be able to help
me. Below is my situation:

One Sun Fire 480R Installed with Sun Solaris 9.0. This system has two
10/100/1000 LAN Interface.
One Intel Desktop Box Installed with RedHat Enterprise Linux 5.0. This
system has one 10/100/1000 LAN Interface as well.

The Solaris Box has 2 NIC's one with IP Address 10.10.10.8 (this is
connected to the switch and is at 100mbps full duplex) and another
with IP Address 192.168.1.2 (this is connected to the Linux Box using
a cross over Cat6 Cable and is at 100mbps full duplex). Both are ce
interfaces. ce0 and ce1.

The Linux Box has one NIC with IP Address 192.168.1.1 (this is
connected to the Solaris Box's NIC with IP Address 192.168.1.2 at
100mbps full duplex using a Cat6 cross cable.)

Since my Sun server is running short of storage space, so i have done
NFS mount on Solaris Box for one of the paritions from the Linux Box.
All is fine. The systems are able to communicate well. But these two
interfaces i.e 192.168.1.2 (on Solaris Box) and 192.168.1.1 (on linux
box) only communicate with 100mbps full duplex link, instead of
1000mbps full duplex.

To troubleshoot my problem, I have performed the following on the
Linux Box and Solaris Box:

On the Linux Box:

ethtool -s eth0 speed 1000 duplex full

i get the following on my /var/log/messages

Oct 25 16:22:06 naspoc kernel: e1000: eth0: e1000_watchdog: NIC Link
is Up 1000 Mbps Full Duplex
Oct 25 16:22:09 naspoc kernel: e1000: eth0: e1000_watchdog: NIC Link
is Down
Oct 25 16:22:12 naspoc kernel: e1000: eth0: e1000_watchdog: NIC Link
is Up 100 Mbps Full Duplex
Oct 25 16:22:12 naspoc kernel: e1000: eth0: e1000_watchdog: 10/100
speed: disabling TSO

So the system does detect 1000mbps but it falls back to 100mbps.

On the Solaris Box:

ndd -set /dev/ce instance 1
ndd -set /dev/ce adv_autoneg_cap 0
ndd -set /dev/ce adv_1000fdx_cap 1

if i do the above, Solaris box is unable to communicate with the Linux
Box. So when revert back the commands the link is active with a
100mbps full duplex connection.

Below is the output of my ethtool command on linux box:

Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: umbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes

To test the cable connectivity, i connected 2 windows boxes with this
cable and they communicated fine with each other at 1000mbps full
duplex.

Can someone help me with this problem. Is there a cable problem or the
OS problem?

I need the communication from Linux Box to the Solaris Box at 1000mbps
to work. So that the speed is good over the NFS shares.

Thanks in advance and thank you for reading my post.

Regards,
Vishal

On Oct 25, 8:56 pm, Vishal Sharma <visha...@gmail.com> wrote:

> The Solaris Box has 2 NIC's one with IP Address 10.10.10.8 (this is
> connected to the switch and is at 100mbps full duplex) and another
> with IP Address 192.168.1.2 (this is connected to the Linux Box using
> a cross over Cat6 Cable and is at 100mbps full duplex). Both are ce
> interfaces. ce0 and ce1.

Was your Cat6 crossover cable specifically intended for GigE use? Was
it wired by someone who knows how to wire a GigE crossover cable? I'll
bet you have an FE crossover cable, not a GE crossover cable.

DS

I recently wired my house for Cat6 and gigabit ethernet and learned
that this is a very tricky business. A lot of things can go wrong,
such as cables can have too sharp bends, not crimped properly, the
list goes on, and on, and on.

Also, NFS had performance problems when run over gigabit ethernet, not
giving me gigabit throughput. I just did a test. I can transfer files
with HTTP at the speed of 50 megaBYTES per second, but I can transfer
files via NFS at only 5 megabytes per second.

i

On Fri, 26 Oct 2007 08:09:32 -0500, Ignoramus7291 <(E-Mail Removed)> wrote:


>I recently wired my house for Cat6 and gigabit ethernet and learned
>that this is a very tricky business. A lot of things can go wrong,
>such as cables can have too sharp bends, not crimped properly, the
>list goes on, and on, and on.

>Also, NFS had performance problems when run over gigabit ethernet, not
>giving me gigabit throughput. I just did a test. I can transfer files
>with HTTP at the speed of 50 megaBYTES per second, but I can transfer
>files via NFS at only 5 megabytes per second.

What buffer sizes do you have on your NFS?
I'm not running jumbo frames since I have 100mbps devices on the same network,
and I get 70MB/s on my gigabit connection from my linux client to my solaris
server.

I use wsize/rsize=32K for NFS. I also noticed differences between the two
gigabit interfaces on my asus motherboard used for linux. The Marvell 88E8001
gigabit I/F does much better than the Nvidia CK804.

AZ Nomad wrote:
> On Fri, 26 Oct 2007 08:09:32 -0500, Ignoramus7291 <(E-Mail Removed)> wrote:
>
>
>> I recently wired my house for Cat6 and gigabit ethernet and learned
>> that this is a very tricky business. A lot of things can go wrong,
>> such as cables can have too sharp bends, not crimped properly, the
>> list goes on, and on, and on.
>
>> Also, NFS had performance problems when run over gigabit ethernet, not
>> giving me gigabit throughput. I just did a test. I can transfer files
>> with HTTP at the speed of 50 megaBYTES per second, but I can transfer
>> files via NFS at only 5 megabytes per second.
>
> What buffer sizes do you have on your NFS?
> I'm not running jumbo frames since I have 100mbps devices on the same network,
> and I get 70MB/s on my gigabit connection from my linux client to my solaris
> server.
>
> I use wsize/rsize=32K for NFS. I also noticed differences between the two
> gigabit interfaces on my asus motherboard used for linux. The Marvell 88E8001
> gigabit I/F does much better than the Nvidia CK804.

i agree NFS out of the box usually performs very poorly
so does SMB and tweaking buffer sizes in both servers gives the
biggest performance increase. I use 64 k buffers here
and my speed went from stock NFS (~800 kb/s to 9 mb/s - 100 mbps lan)

I managed to fix the problem. I reinstalled my NIC drivers on the
Linux Machine and things went like breeze. 1000mbps auto detected. I
am now working on tuning my NFS and Disk performance, so that IO can
be faster. Any suggestions are welcome.

Apparently the cable i used is the right one. Crossover GE. Somewhere
i had read that its suppose to be Straight for GE. But for me the
cross worked just fine.

Thanks everyone.


On Oct 26, 11:56 am, Vishal Sharma <visha...@gmail.com> wrote:
> Hello All,
>
> I am facing this weird problem. Hope someone here will be able to help
> me. Below is my situation:
>
> One Sun Fire 480R Installed with Sun Solaris 9.0. This system has two
> 10/100/1000 LAN Interface.
> One Intel Desktop Box Installed with RedHat Enterprise Linux 5.0. This
> system has one 10/100/1000 LAN Interface as well.
>
> The Solaris Box has 2 NIC's one with IP Address 10.10.10.8 (this is
> connected to the switch and is at 100mbps full duplex) and another
> with IP Address 192.168.1.2 (this is connected to the Linux Box using
> a cross over Cat6 Cable and is at 100mbps full duplex). Both are ce
> interfaces. ce0 and ce1.
>
> The Linux Box has one NIC with IP Address 192.168.1.1 (this is
> connected to the Solaris Box's NIC with IP Address 192.168.1.2 at
> 100mbps full duplex using a Cat6 cross cable.)
>
> Since my Sun server is running short of storage space, so i have done
> NFS mount on Solaris Box for one of the paritions from the Linux Box.
> All is fine. The systems are able to communicate well. But these two
> interfaces i.e 192.168.1.2 (on Solaris Box) and 192.168.1.1 (on linux
> box) only communicate with 100mbps full duplex link, instead of
> 1000mbps full duplex.
>
> To troubleshoot my problem, I have performed the following on the
> Linux Box and Solaris Box:
>
> On the Linux Box:
>
> ethtool -s eth0 speed 1000 duplex full
>
> i get the following on my /var/log/messages
>
> Oct 25 16:22:06 naspoc kernel: e1000: eth0: e1000_watchdog: NIC Link
> is Up 1000 Mbps Full Duplex
> Oct 25 16:22:09 naspoc kernel: e1000: eth0: e1000_watchdog: NIC Link
> is Down
> Oct 25 16:22:12 naspoc kernel: e1000: eth0: e1000_watchdog: NIC Link
> is Up 100 Mbps Full Duplex
> Oct 25 16:22:12 naspoc kernel: e1000: eth0: e1000_watchdog: 10/100
> speed: disabling TSO
>
> So the system does detect 1000mbps but it falls back to 100mbps.
>
> On the Solaris Box:
>
> ndd -set /dev/ce instance 1
> ndd -set /dev/ce adv_autoneg_cap 0
> ndd -set /dev/ce adv_1000fdx_cap 1
>
> if i do the above, Solaris box is unable to communicate with the Linux
> Box. So when revert back the commands the link is active with a
> 100mbps full duplex connection.
>
> Below is the output of my ethtool command on linux box:
>
> Settings for eth0:
> Supported ports: [ TP ]
> Supported link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Full
> Supports auto-negotiation: Yes
> Advertised link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Full
> Advertised auto-negotiation: Yes
> Speed: 100Mb/s
> Duplex: Full
> Port: Twisted Pair
> PHYAD: 0
> Transceiver: internal
> Auto-negotiation: on
> Supports Wake-on: umbg
> Wake-on: g
> Current message level: 0x00000007 (7)
> Link detected: yes
>
> To test the cable connectivity, i connected 2 windows boxes with this
> cable and they communicated fine with each other at 1000mbps full
> duplex.
>
> Can someone help me with this problem. Is there a cable problem or the
> OS problem?
>
> I need the communication from Linux Box to the Solaris Box at 1000mbps
> to work. So that the speed is good over the NFS shares.
>
> Thanks in advance and thank you for reading my post.
>
> Regards,
> Vishal

On 2007-10-26, AZ Nomad <(E-Mail Removed)> wrote:
> On Fri, 26 Oct 2007 08:09:32 -0500, Ignoramus7291 <(E-Mail Removed)> wrote:
>
>
>>I recently wired my house for Cat6 and gigabit ethernet and learned
>>that this is a very tricky business. A lot of things can go wrong,
>>such as cables can have too sharp bends, not crimped properly, the
>>list goes on, and on, and on.
>
>>Also, NFS had performance problems when run over gigabit ethernet, not
>>giving me gigabit throughput. I just did a test. I can transfer files
>>with HTTP at the speed of 50 megaBYTES per second, but I can transfer
>>files via NFS at only 5 megabytes per second.
>
> What buffer sizes do you have on your NFS?
> I'm not running jumbo frames since I have 100mbps devices on the same network,
> and I get 70MB/s on my gigabit connection from my linux client to my solaris
> server.


> I use wsize/rsize=32K for NFS. I also noticed differences between the two
> gigabit interfaces on my asus motherboard used for linux. The Marvell 88E8001
> gigabit I/F does much better than the Nvidia CK804.

That's what I am doing also. 32k. I have much better luck with http
speed.

i

I've probably missed some context, but a couple things:

NFS is a request/response protocol. And 32KB isn't enough data at one
time to get link-rate on GbE. So, on reads your NFS client has to be
doing read-ahead, and on writes write-behind to get maybe four or more
requests outstanding onto the wire at one time.

If using UDP, keep in mind that all 22 or so fragments of that 32KB
read/write have to make it through for any of that 32KB read/write to
be good. That means if you have a packet loss probability of p, (1-p)
is the prob of a given fragment getting there, which means that
(1-p)^22 is the probability of all the fragments getting there.

For TCP if one of the segments is lost, it only need retransmit that
segment, not the entire 32KB read/write.

rick jones
--
Process shall set you free from the need for rational thought.
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...