Re: Performance difference between Linux and PS3

Andrew DeFaria <(E-Mail Removed)> wrote:
> [-- text/plain, encoding 7bit, charset: UTF-8, 32 lines --]

> Here's my setup:

> * QWest DSL 7 Mbps package coming from the wall to an ActionTech modem.
> * Connected to the ActionTech is a Linksys Wireless G broadband
> router (WRT54G).
> * Wired to the Linksys are two PC's running Ubuntu 9.04 (One server
> - one desktop)

> I'm also running Windows 7 in VMWare on the Ubuntu desktop where I run a
> piece of software called PlayOn. PlayOn gets video streams from the
> internet and transcodes them and broadcasts them via UPNP to the PS3.
> The videos are stuttering.

> Measuring network performance from the Ubuntu desktop using
> speedtest.net yields a pretty strong 5-6 Mbps download. Measuring
> performance from the PS3 using speedtest.net from the PS3's browser
> yields a weak 0.8 - 2.0 Mbps (sometimes I see ~ 4-5 Mbps - during that
> time PlayOn is, of course, fine).

> Questions:

> * Does this sort of measurement make sense? Are their other/better
> ways to measure this?
> * Why is my network performance so bad coming from the PS3?
> * How can I determine the bottleneck and correct it.

Might take some packet traces if you can.

Some boilerplate I trot-out - much may be preaching to the choir, but
still

Some of my checklist items when presented with assertions of poor
network performance, in no particular order:

*) Is *any one* CPU on either end of the transfer at or close to 100%
utilization? A given TCP connection cannot really take advantage
of more than the services of a single core in the system, so
average CPU utilization being low does not a priori mean things are
OK.

*) Are there TCP retransmissions being registered in netstat
statistics on the sending system? Take a snapshot of netstat -s -t
from just before the transfer, and one from just after and run it
through beforeafter from
ftp://ftp.cup.hp.com/dist/networking/tools:

netstat -s -t > before
transfer or wait 60 or so seconds if the transfer was already going
netstat -s -t > after
beforeafter before after > delta

*) Are there packet drops registered in ethtool -S statistics on
either side of the transfer? Take snapshots in a matter similar to
that with netstat.

*) Are there packet drops registered in the stats for the switch(es)
being traversed by the transfer? These would be retrieved via
switch-specific means.

*) What is the latency between the two end points. Install netperf on
both sides, start netserver on one side and on the other side run:

netperf -t TCP_RR -l 30 -H <remote>

and invert the transaction/s rate to get the RTT latency. There
are caveats involving NIC interrupt coalescing settings defaulting
in favor of throughput/CPU util over latency:

ftp://ftp.cup.hp.com/dist/networking...cy_vs_tput.txt

but when the connections are over a WAN latency is important and
may not be clouded as much by NIC settings.

This all leads into:

*) What is the *effective* TCP (or other) window size for the
connection. One limit to the performance of a TCP bulk transfer
is:

Tput <= W(eff)/RTT

The effective window size will be the lesser of:

a) the classic TCP window advertised by the receiver (the value in
the TCP header's window field shifted by the window scaling
factor exchanged during connection establishment (why one wants
to get traces including the connection establishment...)

this will depend on whether/what the receiving application has
requested via a setsockopt(SO_RCVBUF) call and the sysctl limits
set in the OS. If the application does not call
setsockopt(SO_RCVBUF) then the Linux stack will "autotune" the
advertised window based on other sysctl limits in the OS.

b) the computed congestion window on the sender - this will be
affected by the packet loss rate over the connection, hence the
interest in the netstat and ethtool stats.

c) the quantity of data to which the sending TCP can maintain a
reference while waiting for it to be ACKnowledged by the
receiver - this will be akin to the classic TCP window case
above, but on the sending side, and concerning
setsockopt(SO_SNDBUF) and sysctl settings.

d) the quantity of data the sending application is willing/able to
send at any one time before waiting for some sort of
application-level acknowledgement. FTP and rcp will just blast
all the data of the file into the socket as fast as the socket
will take it. scp has some application-layer "windowing" which
may cause it to put less data out onto the connection than TCP
might otherwise have permitted. NFS has the maximum number of
outstanding requests it will allow at one time acting as a
defacto "window" etc etc etc


--
web2.0 n, the dot.com reunion tour...
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Andrew DeFaria <(E-Mail Removed)> wrote:
> [-- text/plain, encoding 7bit, charset: UTF-8, 232 lines --]

FWIW, most of the previous mime-encoded stuff wasn't showing-up under
tin on the HP-UX 11.11 system on which I read netnews.

> Rick Jones wrote:

> Hey Rick! Long time no speak? Do you remember me?

I thought the name sounded familiar

> How's HP treating ya?

Probably a topic for an offline discussion.


> > Might take some packet traces if you can.
> I'm not much of a packet tracer...

Still, tcpdump can be useful. The most basic would be something like:

tcpdump -i <interface> -w <file> host <remote hostname>

which should then snag the first 68 bytes or so of all the packets
appearing at <interface> involving <remote hostname> one can then pass that through a command like:

tcpdump -r <file>

for very simple post processing, or feed <file> to something like
wireshark or tcptrace or the like.

> > Some boilerplate I trot-out - much may be preaching to the choir, but
> > still
> >
> > Some of my checklist items when presented with assertions of poor
> > network performance, in no particular order:
> >
> > *) Is *any one* CPU on either end of the transfer at or close to 100%
> > utilization? A given TCP connection cannot really take advantage of
> > more than the services of a single core in the system, so average CPU
> > utilization being low does not a priori mean things are OK.

> Well when loaded I saw pretty much a 40-70% CPU load on the set of
> both CPU in the Win 7 VMWare box with neither CPU peaking at

VMware... virtualization... how fun... is that CPU util being reported
on a client OS instance, or from the hypervisor/host/dom0/whatever?

> 100%. The vendor of Playon (http://mediamall.com) verified through
> their logs that their software Playon was pumping everything it
> could down the pipe to the PS3 but the problem was in the connection
> between the PC and the PS3. I have no tools that I know of to gauge
> the CPU usage of a PS3. Do you?

Nope, perhaps folks in a PS3-related group do.

> > *) Are there TCP retransmissions being registered in netstat
> > statistics on the sending system? Take a snapshot of netstat -s -t
> > from just before the transfer, and one from just after and run it
> > through beforeafter from ftp://ftp.cup.hp.com/dist/networking/tools:
> >
> > netstat -s -t > before
> > transfer or wait 60 or so seconds if the transfer was already going
> > netstat -s -t > after
> > beforeafter before after > delta
> $ beforeafter before after

> IPv4 Statistics

> Packets Received = 256
> Received Header Errors = 0
> Received Address Errors = 0
> Datagrams Forwarded = 0
> Unknown Protocols Received = 0
> Received Packets Discarded = 3
> Received Packets Delivered = 442
> Output Requests = 448
> Routing Discards = 0
> Discarded Output Packets = 2
> Output Packet No Route = 0
> Reassembly Required = 0
> Reassembly Successful = 0
> Reassembly Failures = 0
> Datagrams Successfully Fragmented = 0
> Datagrams Failing Fragmentation = 0
> Fragments Created = 0

> IPv6 Statistics

[ deleted IPv6 stats ]

> ICMPv4 Statistics

[ deleted ]

> ICMPv6 Statistics

[ deleted]

> TCP Statistics for IPv4

> Active Opens = 4
> Passive Opens = 25
> Failed Connection Attempts = 0
> Reset Connections = 1
> Current Connections = 7
> Segments Received = 250
> Segments Sent = 262
> Segments Retransmitted = 1

Not exactly a lot of interesting stats there. As this was
(ostensibly) on the receiver, I was hoping to see something about
segments received out of order. A little surprising to me that there
are as many/more outbound segments as inbound during the test.

> TCP Statistics for IPv6

[deleted]

> UDP Statistics for IPv4

> Datagrams Received = 737
> No Ports = 0
> Receive Errors = 0
> Datagrams Sent = 157

UDP datagrams? Really? I thought this was TCP test.

> UDP Statistics for IPv6

> Datagrams Received = 0
> No Ports = 0
> Receive Errors = 0
> Datagrams Sent = 7

> > *) Are there packet drops registered in ethtool -S statistics on
> > either side of the transfer? Take snapshots in a matter similar to
> > that with netstat.
> I don't have an ethtool on the Win 7 platform nor the PS3 platform.

It is an "evolving" document started in a Linux context - I need to
add "or your platform's mechanism to report link-level statistics for
the NIC(s)" or somesuch.

>From Ubuntu:

> Earth:root ethtool -S eth0
> NIC statistics:
> tx_packets: 6737223
> rx_packets: 10302786
> tx_errors: 0
> rx_errors: 0
> rx_missed: 0
> align_errors: 0
> tx_single_collisions: 0
> tx_multi_collisions: 0
> unicast: 10289711
> broadcast: 676
> multicast: 13069
> tx_aborted: 0
> tx_underrun: 0

Clean, for what that is worth.

> > *) Are there packet drops registered in the stats for the switch(es)
> > being traversed by the transfer? These would be retrieved via
> > switch-specific means.

> I have no idea what you are saying here nor how to provide you with
> info...

Packets can be dropped many places - somewhere in the cloud between
the sender and you, or at the ingress point where you are (eg the
device connecting you to your ISP) or on any switches in your own
setup, or in the end systems themselves. The broad idea is to check
the statistics as many places as you can between sender and receiver.
Naturally, when you do not have control over the sender, or the cloud
which separates you from the sender, the options become rather
limited.

> > *) What is the latency between the two end points. Install netperf on
> > both sides, start netserver on one side and on the other side run:
> >
> > netperf -t TCP_RR -l 30 -H <remote>
> >
> > and invert the transaction/s rate to get the RTT latency. There are
> > caveats involving NIC interrupt coalescing settings defaulting in
> > favor of throughput/CPU util over latency:
> >
> > ftp://ftp.cup.hp.com/dist/networking...cy_vs_tput.txt
> >
> > but when the connections are over a WAN latency is important and may
> > not be clouded as much by NIC settings.

> Before I attempt this, are you suggesting that netperf exists for
> the PS3?

No - again, an evolving document Having said that, if someone
wishes to and knows how to get something that "speaks netperf" running
on PS3, I'll be happy to help however I can.

> > This all leads into:
> >
> > *) What is the *effective* TCP (or other) window size for the
> > connection. One limit to the performance of a TCP bulk transfer is:
> >
> > Tput <= W(eff)/RTT
> >
> > The effective window size will be the lesser of:
> >
> > a) the classic TCP window advertised by the receiver (the value in the
> > TCP header's window field shifted by the window scaling factor
> > exchanged during connection establishment (why one wants to get traces
> > including the connection establishment...)
> >
> > this will depend on whether/what the receiving application has
> > requested via a setsockopt(SO_RCVBUF) call and the sysctl limits set
> > in the OS. If the application does not call setsockopt(SO_RCVBUF) then
> > the Linux stack will "autotune" the advertised window based on other
> > sysctl limits in the OS.
> >
> > b) the computed congestion window on the sender - this will be
> > affected by the packet loss rate over the connection, hence the
> > interest in the netstat and ethtool stats.
> >
> > c) the quantity of data to which the sending TCP can maintain a
> > reference while waiting for it to be ACKnowledged by the receiver -
> > this will be akin to the classic TCP window case above, but on the
> > sending side, and concerning setsockopt(SO_SNDBUF) and sysctl settings.
> >
> > d) the quantity of data the sending application is willing/able to
> > send at any one time before waiting for some sort of application-level
> > acknowledgement. FTP and rcp will just blast all the data of the file
> > into the socket as fast as the socket will take it. scp has some
> > application-layer "windowing" which may cause it to put less data out
> > onto the connection than TCP might otherwise have permitted. NFS has
> > the maximum number of outstanding requests it will allow at one time
> > acting as a defacto "window" etc etc etc

> Asided from the standard settings/registry entries that one can make
> from the Windows end of things, on the PS3 end about all I have ot offer
> is a setting on the PS3 called MTU which is set at the max of 1500.

Which is where getting some sort of packet trace showing traffic
to/from the PS3 would be helpful. If we can see the connection(s)
being established from the PS3 to the remote, we can see what sort of
window size it is offering, and also see things like gaps in the
traffic indicating packet loss somewhere along the way.

rick jones

> --
> Andrew DeFaria <http://defaria.com>
> Friends may come and go, but enemies accumulate.


--
a wide gulf separates "what if" from "if only"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...