detailed traffic statistics

Hi,

I've running the router of my network for a few years now, all is fine and I
also have nice traffic statistics per interface. Now I'd like to have more
detail, like how much bandwidth is spent on streaming internet radio, videos
from YouTube, email, VoIP etc.

Most applications use a designated port, so with a bit of iptables-ulog it's
no problem to collect statistics on that. But it doesn't tell my, what kind
of content is in it. And then there's this huge bunch of applications, which
use HTTP.

So I need to do DPI in both cases. At first I thought about using tcpdump
and a few filter rules. But maybe there's a more elegant solution for that.

Let me state again, that I don't want to collect statistics on the content
itself, but only about the bandwidth spent on different classes of content.
I.e. everything video, not matter if rtsp, mms, http or something else is
used should be protocolled as "consumed bandwidth for video".

Any ideas?


Wolfgang

In news:hjnrrj$6hd$(E-Mail Removed),
Wolfgang Draxinger <(E-Mail Removed)> typed:

....
> Let me state again, that I don't want to collect statistics on the
> content itself, but only about the bandwidth spent on different
> classes of content. I.e. everything video, not matter if rtsp, mms,
> http or something else is used should be protocolled as "consumed
> bandwidth for video".

How would a TCP/IP packet of video content be distinguishable from a TCP/IP
packet of streaming radio, or any other type of content, for that matter?

Greg Russell wrote:

> How would a TCP/IP packet of video content be distinguishable from a
> TCP/IP packet of streaming radio, or any other type of content, for that
> matter?

The same way, the application that finally proceses the stream does: By
looking at the stream descriptors/headers, telling the demuxer, what to find
in each track. Every stream container delivers this kind of data.


Wolfgang

In news:(E-Mail Removed),
Wolfgang Draxinger <(E-Mail Removed)> typed:

>> How would a TCP/IP packet of video content be distinguishable from a
>> TCP/IP packet of streaming radio, or any other type of content, for
>> that matter?
>
> The same way, the application that finally proceses the stream does:
> By looking at the stream descriptors/headers, telling the demuxer,
> what to find in each track. Every stream container delivers this kind
> of data.

Your router is going to demux every stream that passes through it, tallying
the packet totals for each type of content? That *might* work if there's
only one client on the local network, but it would very quickly overwhelm a
router for a network with an average number of clients ... it would take
more cpu power than is currently available, I'm sure.

Maybe you mean examine every packet looking for headers that describe the
stream content, then sub-tally the totals for the entire ensuing stream and
combine the totals for the entire category such as video, etc?

If so it gets back to my original question -- how are you going to recognize
those packet headers that declare any given category of stream? Where in a
TCP/IP packet structure / stream does it say "this is video" or "this is
audio" or "this is RSS"?

Jusr curious ...

Greg Russell wrote:

> Your router is going to demux every stream that passes through it,
> tallying the packet totals for each type of content?

Why demux. The only important information is, what is contained in the
stream, i.e. the stream container headers. That information is avaliable
prior to demuxing.

> That *might* work if
> there's only one client on the local network, but it would very quickly
> overwhelm a router for a network with an average number of clients ... it
> would take more cpu power than is currently available, I'm sure.

Probably not a problem here. The router runs as hypervised virtual machine
on a dual core @2.2GHz system.

> Maybe you mean examine every packet looking for headers that describe the
> stream content, then sub-tally the totals for the entire ensuing stream
> and combine the totals for the entire category such as video, etc?

This way ^^^

> If so it gets back to my original question -- how are you going to
> recognize those packet headers that declare any given category of stream?

Not by static offsets if you're thinking into that direction.

Some applications, like VoIP, SSH, etc. run over dedicated ports. So it's
sufficient to test for a small subset of magic bytes on certain ports, to
see, if it's really VoIP, SSH, etc. going over there. In the case of SSL
encrypted content it's impossible anyway (this sort of DPI would be kinda a
man-in-the-middle attack).

On protocols which may carry all sorts of content one would test on what's
in there by looking at magic bytes. Just like the "file" utility does. This
has been implemented in Wireshark, naming an example.

I could of course use Wireshark's modules for this kind of application, but
I wonder, if there's software better suited for this kind of job.

> Where in a TCP/IP packet structure / stream does it say "this is video" or
> "this is audio" or "this is RSS"?

Um, in the payload? Of course it requires to parse the stream container's
headers.


Wolfgang