Seeking recommendation for fast GigEthernet card

I've a couple of projects where network performance is an issue: a NAS
and a Router. Both are actually clusters, but I'm pretty sure that
that's beside the point (though I mention it Just In Case {8^).

The NAS is is actually a "head", speaking NFS on the front and iSCSI on
the back. The router is the usual IP device.

Everything works using the built-in GigE interfaces. But I'm thinking
that I can squeeze more performance out of my boxes if I push more of the
processing out of the system and onto the network interface cards.
So...any suggestions?

The one absolute requirement in both cases is that I not lose the ability
to "speak" 802.1q to a switch. Having multiple ports would be a nice
option, though I can always use multiple cards if necessary.

The distribution is CentOS5/RHEL5.

Thanks...Andrew

On Sun, 17 Feb 2008 22:48:49 +0000, Andrew Gideon wrote:

> I've a couple of projects where network performance is an issue: a NAS
> and a Router. Both are actually clusters, but I'm pretty sure that
> that's beside the point (though I mention it Just In Case {8^).
>
> The NAS is is actually a "head", speaking NFS on the front and iSCSI on
> the back. The router is the usual IP device.
>
> Everything works using the built-in GigE interfaces. But I'm thinking
> that I can squeeze more performance out of my boxes if I push more of
> the processing out of the system and onto the network interface cards.
> So...any suggestions?
>
> The one absolute requirement in both cases is that I not lose the
> ability to "speak" 802.1q to a switch. Having multiple ports would be a
> nice option, though I can always use multiple cards if necessary.
>
> The distribution is CentOS5/RHEL5.
>
> Thanks...Andrew

What's the CPU utilization on those boxes? If you are getting near
gigabit speeds and the CPUs aren't saturated a smarter gigabit card isn't
going to help you. If you a really want more bandwidth out of the box
then get a 10G card. However that will only help you if you aren't disk
limited.

On Sun, 17 Feb 2008 17:06:29 -0600, General Schvantzkopf wrote:

>
>
> What's the CPU utilization on those boxes?

That's a legitimate question, but I'm not pushing any production through
these yet so I don't have a real answer. I'm just queuing up my
"response" in case this does become an issue. Also, since I'm planning
to buy new boxes for these, I want to be sure to be buying boxes with the
"right" bus type for the cards I may be purchasing.

> If you are getting near
> gigabit speeds and the CPUs aren't saturated a smarter gigabit card
> isn't going to help you. If you a really want more bandwidth out of the
> box then get a 10G card.

I have to admit: I'd not thought of that. I cannot envision hitting a
real bandwidth limit on the Router cluster in the near future, but that
may be a very good idea for the NAS cluster.

Any recommended 10GigE cards?

Thanks...

Andrew

Andrew Gideon <(E-Mail Removed)> writes:

> On Sun, 17 Feb 2008 17:06:29 -0600, General Schvantzkopf wrote:
>
>>
>>
>> What's the CPU utilization on those boxes?
>
> That's a legitimate question, but I'm not pushing any production through
> these yet so I don't have a real answer. I'm just queuing up my
> "response" in case this does become an issue. Also, since I'm planning
> to buy new boxes for these, I want to be sure to be buying boxes with the
> "right" bus type for the cards I may be purchasing.

I'd go for a PCI express based system. As for the NICs, Intel is
generally a safe bet.

--
Måns Rullgård
(E-Mail Removed)

On Sun, 17 Feb 2008 23:39:23 +0000, Andrew Gideon wrote:

> On Sun, 17 Feb 2008 17:06:29 -0600, General Schvantzkopf wrote:
>
>
>>
>> What's the CPU utilization on those boxes?
>
> That's a legitimate question, but I'm not pushing any production through
> these yet so I don't have a real answer. I'm just queuing up my
> "response" in case this does become an issue. Also, since I'm planning
> to buy new boxes for these, I want to be sure to be buying boxes with
> the "right" bus type for the cards I may be purchasing.
>
>> If you are getting near
>> gigabit speeds and the CPUs aren't saturated a smarter gigabit card
>> isn't going to help you. If you a really want more bandwidth out of the
>> box then get a 10G card.
>
> I have to admit: I'd not thought of that. I cannot envision hitting a
> real bandwidth limit on the Router cluster in the near future, but that
> may be a very good idea for the NAS cluster.
>
> Any recommended 10GigE cards?
>
> Thanks...
>
> Andrew

I can't help you with the 10G cards, I haven't tried any. As for 1G, the
MACs in Nvidia chipsets have IP offload engines so you can get IP
acceleration even in a desktop system, Intel MACs also have acceleration.
However a NAS box doesn't use much horsepower, any Core2 will have way
more cycles than it needs so even a crappy 1G card should be able to run
near wire speeds.

Andrew Gideon wrote:
> On Sun, 17 Feb 2008 17:06:29 -0600, General Schvantzkopf wrote:
>
>>
>> What's the CPU utilization on those boxes?
>
> That's a legitimate question, but I'm not pushing any production through
> these yet so I don't have a real answer. I'm just queuing up my
> "response" in case this does become an issue. Also, since I'm planning
> to buy new boxes for these, I want to be sure to be buying boxes with the
> "right" bus type for the cards I may be purchasing.
>
>> If you are getting near
>> gigabit speeds and the CPUs aren't saturated a smarter gigabit card
>> isn't going to help you. If you a really want more bandwidth out of the
>> box then get a 10G card.
>
> I have to admit: I'd not thought of that. I cannot envision hitting a
> real bandwidth limit on the Router cluster in the near future, but that
> may be a very good idea for the NAS cluster.
>
> Any recommended 10GigE cards?

How much money do you have? A 10GigE infrastructure could cost
$$$$$$.

With regards to onboard or other GigE, I found that most perform
close to theoretical maximums EXCEPT when using a 33Mhz bus.
This applies to SOME on board gigabit NICs... usually ones produced
more than 4-5 years ago. An example is the Compaq (HP) ML330 G3
(I think it was the G3). It is hampered by a slow bus. At
33Mhz the maximum throughput is less than 600Mbit.

Also... a VERY expensive switch will help ensure full gigabit
port to port. The cheap ones (pretty much anything less than
probably $10K USD) probably won't do.

Andrew Gideon <(E-Mail Removed)> writes:
>
>I've a couple of projects where network performance is an issue: a NAS
>and a Router. Both are actually clusters, but I'm pretty sure that
>that's beside the point (though I mention it Just In Case {8^).
>
>The NAS is is actually a "head", speaking NFS on the front and iSCSI on
>the back. The router is the usual IP device.
>
>Everything works using the built-in GigE interfaces. But I'm thinking
>that I can squeeze more performance out of my boxes if I push more of the
>processing out of the system and onto the network interface cards.
>So...any suggestions?

Some experimental data:

The setup: The NFS server is a 2.66GHz Pentium 4 with an onboard Intel
e1000 interface; the data is a directory containing 354112 KB of
files. I first warmed up the NFS server's cache by reading this data
on a machine that I did not use later in the test. Then I read the
data on various machines with:

time tar cf - <dir> |cat >/dev/null

Running this on a 3GHz Xeon 5160 machine with an Intel e1000
interface:

real 0m5.163s
user 0m0.044s
sys 0m0.364s

A 2GHz Opteron 246 machine with a Tigon3 interface:

real 0m5.226s
user 0m0.102s
sys 0m1.693s

An Opteron 270 machine running at 1GHz (clocked down) with a Tigon3 interface:

real 0m4.959s
user 0m0.080s
sys 0m1.760s

A 2.2GHz Athlon 64 X2 4400+ machine with an Nvidia (forcedeth) interface:

real 0m4.956s
user 0m0.040s
sys 0m1.196s

An Athlon 64 X2 4600+ machine running at 1GHz (clocked down) with an
Nvidia (forcedeth) interface:

real 0m5.012s
user 0m0.076s
sys 0m2.180s

All these machines run a 2.6.18 kernel, except the Opteron 246
(2.6.10) and the NFS server (2.6.13).

Looks like the e1000 produces the least CPU load, but none of these
cards produce so much CPU load that you need to worry about it if the
machine is a dedicated NFS server.

I also watched the NFS server with top during one of these jobs, and
the load stayed low.

Note also that having all the data cached on the NFS server is a
worst-case setting for CPU load; more typically the server will wait
much of the time for disk seeks to complete.

>The one absolute requirement in both cases is that I not lose the ability
>to "speak" 802.1q to a switch.

I cannot help you there. I don't know what 802.1q is.

Followups set to comp.os.linux.hardware.

- anton
--
M. Anton Ertl Some things have to be seen to be believed
(E-Mail Removed) Most things have to be believed to be seen
http://www.complang.tuwien.ac.at/anton/home.html

Andrew Gideon wrote:

>
> I've a couple of projects where network performance is an issue: a NAS
> and a Router. Both are actually clusters, but I'm pretty sure that
> that's beside the point (though I mention it Just In Case {8^).
>
> The NAS is is actually a "head", speaking NFS on the front and iSCSI on
> the back. The router is the usual IP device.
>
> Everything works using the built-in GigE interfaces. But I'm thinking
> that I can squeeze more performance out of my boxes if I push more of the
> processing out of the system and onto the network interface cards.
> So...any suggestions?
>
> The one absolute requirement in both cases is that I not lose the ability
> to "speak" 802.1q to a switch. Having multiple ports would be a nice
> option, though I can always use multiple cards if necessary.
>
> The distribution is CentOS5/RHEL5.
>
> Thanks...Andrew

This is Linux, right? :') I'd find myself some mid-priced dual or quad port
1g nics and switches that support jumboframes and bond them... Whay more
affordable that going 10g unless ofcourse your budget is really fat to
begin with.



--

Jerry McBride ((E-Mail Removed))

On Mon, 18 Feb 2008 10:09:34 +0000, Anton Ertl wrote:

> I cannot help you there. I don't know what 802.1q is.

Just an FYI: This is support for VLAN trunking. It means that the device
in question can participate in multiple LANs (ie. "broadcast domains")
without either multiple NICs or routing.

Very convenient. For example, we used to route ridiculous amounts of
traffic merely in doing backups. The backup server now has [virtual]
ports on every LAN, and no backup traffic is routed.

Linux has had support for 802.1q since at least RH9 (though that may have
required a patch; I don't recall the details).

- Andrew

On Mon, 18 Feb 2008 09:26:13 -0500, Jerry McBride wrote:


> This is Linux, right? :')

I'm afraid I don't follow your question; what else is still in use?

<grin>

> I'd find myself some mid-priced dual or quad
> port 1g nics and switches that support jumboframes and bond them... Whay
> more affordable that going 10g unless ofcourse your budget is really fat
> to begin with.

I've some money to spend on this (this is part of a project which
includes buying a building; switch replacements are the least of my
problem {8^). But I'm not adverse to keeping some money that I don't
need to spend.

The idea of bonding multiple 1GigE ports had occurred, but still leaves
me wondering: what are good NICs?

Other posts have been answering this, though.

Thanks...Andrew