Getting a very low B/W on my switched network using netperf

We have a new 10GigE network (in theory) using Cisco's Nexus5000
switches. The network topology is roughly like this:

http://dl.dropbox.com/u/118481/switch_topology.jpg

Within each switch the backplane is full bandwidth but switch-switch I am
1:4 oversubscribed.

If I do a single server-to-server test (within a switch) I do get a
bandwidth close to the theoretical 10Gbps. I also continue to see ~10Gbps
if I do a simultaneous pairwise test of all 16 pairs within a switch.
i.e. I ran netserver on 16 servers and netperf from the other 16 servers
all connected to the same switch. Then triggered the 16 instances
simulatanously (within 1sec)

-------------------------------------------------------------------------
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.1
(192.168.0.1) port 0 AF_INET : histogram : interval
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 65536 65536 10.01 9713.67
-------------------------------------------------------------------------

If I do a simulatanous test (32 netperfs on switch1 talking to 32
netservers on switch2) across switches though the bandwidth results are
horrible. Is this to be expected or am I doing something wrong?
Throughput is all over the place ranging from 68Mbps to 4Gbps (but mostly
on the lower side) My naiive calculations would have expected at least
2.5 Gbps.

Each switch has 40 ports: 32 are directly connected to servers and the
other 8 are connected to the master switches; 4 to each master switch.
The two master switches are interconnected with some sort of "Cisco
magic" that I don't fully understand.

----------------------------------------------------------------------
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.10
(192.168.0.10) port 0 AF_INET : histogram : interval
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 65536 65536 10.04 151.40
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.11
(192.168.0.11) port 0 AF_INET : histogram : interval
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 65536 65536 10.51 4258.13
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.12
(192.168.0.12) port 0 AF_INET : histogram : interval
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 65536 65536 10.35 965.62
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.13
(192.168.0.13) port 0 AF_INET : histogram : interval
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 65536 65536 10.01 3883.32
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.14
(192.168.0.14) port 0 AF_INET : histogram : interval
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 65536 65536 10.23 1914.75
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.15
(192.168.0.15) port 0 AF_INET : histogram : interval
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 65536 65536 10.02 68.32
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.16
(192.168.0.16) port 0 AF_INET : histogram : interval
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 65536 65536 10.38 1728.64

--
Rahul