tcpdump show requested web addresses

How to use tcpdump to extract addresses of web sites that users
browse, i.e. the HTTP GET requests?

Lukasz



bbla32@op.pl

On 03/05/2007 12:55 AM, (E-Mail Removed) wrote:
> How to use tcpdump to extract addresses of web sites that users
> browse, i.e. the HTTP GET requests?

IMHO, Squid running as a transparent proxy and Sarg for Squid can be a
better solution for this kind of problem domain, besides this you
additionally will get all the caching, authentication, blocking and, or
barring features of squid also.

If the above one seems fat and, or you want to repeat it yourself
specifically per your needs, the:

man tcpdump
man awk

have a lot to say; give these a try and jot down a shell script.

--
Dr Balwinder S "bsd" Dheeman Registered Linux User: #229709
Anu'z Linux@HOME Machines: #168573, 170593, 259192
Chandigarh, UT, 160062, India Gentoo, Fedora, Knoppix/FreeBSD/XP
Home: http://cto.homelinux.net/~bsd/ Visit: http://counter.li.org/

> IMHO, Squid running as a transparent proxy and Sarg for Squid can be a
> better solution for this kind of problem domain, besides this you

too heavyweight, I only want to sometimes see accessed sites at the
moment


> man tcpdump
> man awk

I had checked, this is the farthest I can go:
# tcpdump -A -i eth0 -vvv -s 500 'tcp port 80 and ip[2:2] > 40 and
tcp[tcpflags] & tcp-push != 0 and dst port 80' -f


18:56:32.608664 IP (tos 0x0, ttl 128, id 65255, offset 0, flags [DF],
proto: TCP (6), length: 1087) 192.168.0.2.leoip > 64.233.179.99.http:
P 2815965847:2815966894(1047) ack 2615566913 win 65535
E..?..@...B.....@..c.^.P..:...bAP...q...GET /groups/favorites HTTP/1.1
Accept: */*
Accept-Language: pl
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET
CLR 1.1.4322; .NET CLR 2.0.50727; Google-TR-1)
Host: groups.google.com
Connection: Keep-Alive
Cookie: GTZ=-60;
__utma=118165087.413883557.1169431526.1173112823.1 173117250.94;
__utmz=118165087.1171542698.32.2.utmccn=(organic)| utmcsr=google|
utmctr=download+jpeg+dotn
18:56:35.157940 IP (tos 0x0, ttl 128, id 65402, offset 0, flags [DF],
proto: TCP (6), length: 1240) 192.168.0.2.leoip > 64.233.179.99.http:
P 1047:2247(1200) ack 13482 win 65535
E....z@...A.....@..c.^.P..>.....P...MS..GET /groups/static/release/
g2_common-2808fcbcb36accc4345bd5927f3708e2.js HTTP/1.1
Accept: */*
Referer: http://groups.google.com/groups/favorites
Accept-Language: pl
UA-CPU: x86
Accept-Encoding: gzip, deflate
If-Modified-Since: Fri, 16 Feb 2007 22:27:45 GMT
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET
CLR 1.1.4322; .NET CLR 2.0.50727; Google-TR-1)
Host: groups.google.com
Connection: Keep-Alive
Cookie: GTZ=-60;
18:56:35.785548 IP (tos 0x0, ttl 128, id 65435, offset 0, flags [DF],
proto: TCP (6), length: 1196) 192.168.0.2.leoip > 64.233.179.99.http:
P 2247:3403(1156) ack 13630 win 65387
E.....@...A.....@..c.^.P..C^...~P..ks...GET /groups/img/envelope.gif
HTTP/1.1
Accept: */*
Referer: http://groups.google.com/groups/favorites
Accept-Language: pl
UA-CPU: x86
Accept-Encoding: gzip, deflate
If-Modified-Since: Tue, 13 Feb 2007 19:37:50 GMT
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET
CLR 1.1.4322; .NET CLR 2.0.50727; Google-TR-1)
Host: groups.google.com
Connection: Keep-Alive
Cookie: GTZ=-60; __utma=118165087.413883557.1169431526.117311
18:56:36.477664 IP (tos 0x0, ttl 128, id 65477, offset 0, flags [DF],
proto: TCP (6), length: 1205) 192.168.0.2.leoip > 64.233.179.99.http:
P 3403:4568(1165) ack 13778 win 65239
E.....@...A.....@..c.^.P..G.....P.......GET /groups/img/3nb/
groups_medium.gif HTTP/1.1
Accept: */*
Referer: http://groups.google.com/groups/favorites
Accept-Language: pl
UA-CPU: x86
Accept-Encoding: gzip, deflate
If-Modified-Since: Tue, 13 Feb 2007 19:37:48 GMT
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET
CLR 1.1.4322; .NET CLR 2.0.50727; Google-TR-1)
Host: groups.google.com
Connection: Keep-Alive
Cookie: GTZ=-60; __utma=118165087.413883557.11694315
18:56:36.609755 IP (tos 0x0, ttl 128, id 65492, offset 0, flags [DF],
proto: TCP (6), length: 1199) 192.168.0.2.ncconfig >
64.233.179.99.http: P 2844376055:2844377214(1159) ack 2477796145 win
65535
E.....@...A}....@..c.`.P......+1P.......GET /groups/img/
mygroups_lt.gif HTTP/1.1
Accept: */*
Referer: http://groups.google.com/groups/favorites
Accept-Language: pl
UA-CPU: x86
Accept-Encoding: gzip, deflate
If-Modified-Since: Tue, 13 Feb 2007 19:38:02 GMT
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET
CLR 1.1.4322; .NET CLR 2.0.50727; Google-TR-1)
Host: groups.google.com
Connection: Keep-Alive
Cookie: GTZ=-60; __utma=118165087.413883557.1169431526.117
18:56:37.092568 IP (tos 0x0, ttl 128, id 65521, offset 0, flags [DF],
proto: TCP (6), length: 1186) 192.168.0.2.leoip > 64.233.179.99.http:
P 4568:5714(1146) ack 13926 win 65091
E.....@...Am....@..c.^.P..Lo....P..C....GET /images/x2.gif HTTP/1.1
Accept: */*
Referer: http://groups.google.com/groups/favorites
Accept-Language: pl
UA-CPU: x86
Accept-Encoding: gzip, deflate
If-Modified-Since: Fri, 21 Jul 2006 18:17:14 GMT
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET
CLR 1.1.4322; .NET CLR 2.0.50727; Google-TR-1)
Host: groups.google.com
Connection: Keep-Alive
Cookie: GTZ=-60;
__utma=118165087.413883557.1169431526.1173112823.1 1731
18:56:37.130598 IP (tos 0x0, ttl 128, id 65526, offset 0, flags [DF],
proto: TCP (6), length: 802) 192.168.0.2.wilkenlistener >
66.102.9.104.http: P 367:1129(762) ack 164 win 65372
E.."..@....f....Bf h.b.PU.~m....P..\.M..GET /__utm.gif?
utmwv=1&utmn=37005535&utmcs=utf-8&utmsr=1280x1024&utmsc=32-
bit&utmul=en-us&utmje=1&utmfl=9.0&utmdt=Google
%20Groups&utmhn=groups.google.com&utmr=-&utmp=/groups/
favorites&utmac=UA-1044941-1&utmcc=__utma
%3D118165087.413883557.1169431526.1173112823.11731 17250.94%3B%2B__utmb
%3D118165087%3B%2B__utmc%3D118165087%3B%2B__utmz
%3D118165087.1171542698.32.2.utmccn%3D(organic)%7C utmcsr%3Dgoogle
%7Cutmctr%3Ddownload%2Bjpeg%2Bdotnet%7Cutmcmd%3D
18:56:37.466336 IP (tos 0x0, ttl 128, id 4, offset 0, flags [DF],
proto: TCP (6), length: 1197) 192.168.0.2.ncconfig >
64.233.179.99.http: P 1159:2316(1157) ack 149 win 65387
E.....@...AP....@..c.`.P...~..+.P..k.
...GET /groups/img/watched_y.gif HTTP/1.1
Accept: */*
Referer: http://groups.google.com/groups/favorites
Accept-Language: pl
UA-CPU: x86
Accept-Encoding: gzip, deflate
If-Modified-Since: Tue, 13 Feb 2007 19:38:04 GMT
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET
CLR 1.1.4322; .NET CLR 2.0.50727; Google-TR-1)
Host: groups.google.com
Connection: Keep-Alive
Cookie: GTZ=-60; __utma=118165087.413883557.1169431526.11731
18:56:37.597895 IP (tos 0x0, ttl 128, id 11, offset 0, flags [DF],
proto: TCP (6), length: 1199) 192.168.0.2.leoip > 64.233.179.99.http:
P 5714:6873(1159) ack 14068 win 64949
E.....@...AG....@..c.^.P..P....4P.......GET /groups/img/
threadsub_y.gif HTTP/1.1
Accept: */*
Referer: http://groups.google.com/groups/favorites
Accept-Language: pl
UA-CPU: x86
Accept-Encoding: gzip, deflate
If-Modified-Since: Tue, 13 Feb 2007 19:38:03 GMT
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET
CLR 1.1.4322; .NET CLR 2.0.50727; Google-TR-1)
Host: groups.google.com
Connection: Keep-Alive
Cookie: GTZ=-60; __utma=118165087.413883557.1169431526.117
18:56:38.161750 IP (tos 0x0, ttl 128, id 43, offset 0, flags [DF],
proto: TCP (6), length: 1198) 192.168.0.2.ncconfig >
64.233.179.99.http: P 2316:3474(1158) ack 297 win 65239
E....+@...A(....@..c.`.P......,YP.......GET /groups/img/fusion_add.gif
HTTP/1.1
Accept: */*
Referer: http://groups.google.com/groups/favorites
Accept-Language: pl
UA-CPU: x86
Accept-Encoding: gzip, deflate
If-Modified-Since: Tue, 13 Feb 2007 19:37:50 GMT
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET
CLR 1.1.4322; .NET CLR 2.0.50727; Google-TR-1)
Host: groups.google.com
Connection: Keep-Alive
Cookie: GTZ=-60; __utma=118165087.413883557.1169431526.1173
18:56:38.177656 IP (tos 0x0, ttl 128, id 52, offset 0, flags [DF],
proto: TCP (6), length: 1200) 192.168.0.2.leoip > 64.233.179.99.http:
P 6873:8033(1160) ack 14216 win 64801
E....4@...A.....@..c.^.P..Up....P..!....GET /groups/img/
corner_tleft.gif HTTP/1.1
Accept: */*
Referer: http://groups.google.com/groups/favorites
Accept-Language: pl
UA-CPU: x86
Accept-Encoding: gzip, deflate
If-Modified-Since: Tue, 13 Feb 2007 19:37:49 GMT
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET
CLR 1.1.4322; .NET CLR 2.0.50727; Google-TR-1)
Host: groups.google.com
Connection: Keep-Alive
Cookie: GTZ=-60; __utma=118165087.413883557.1169431526.11
18:56:38.762751 IP (tos 0x0, ttl 128, id 64, offset 0, flags [DF],
proto: TCP (6), length: 1201) 192.168.0.2.ncconfig >
64.233.179.99.http: P 3474:4635(1161) ack 445 win 65091
E....@@...A.....@..c.`.P......,.P..CA...GET /groups/img/
corner_tright.gif HTTP/1.1
Accept: */*
Referer: http://groups.google.com/groups/favorites
Accept-Language: pl
UA-CPU: x86
Accept-Encoding: gzip, deflate
If-Modified-Since: Tue, 13 Feb 2007 19:37:49 GMT
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET
CLR 1.1.4322; .NET CLR 2.0.50727; Google-TR-1)
Host: groups.google.com
Connection: Keep-Alive
Cookie: GTZ=-60; __utma=118165087.413883557.1169431526.1
18:56:39.046125 IP (tos 0x0, ttl 128, id 81, offset 0, flags [DF],
proto: TCP (6), length: 1197) 192.168.0.2.leoip > 64.233.179.99.http:
P 8033:9190(1157) ack 14364 win 64653
E....Q@...A.....@..c.^.P..Y....\P....=..GET /groups/img/dot_clear.gif
HTTP/1.1
Accept: */*
Referer: http://groups.google.com/groups/favorites
Accept-Language: pl
UA-CPU: x86
Accept-Encoding: gzip, deflate
If-Modified-Since: Tue, 13 Feb 2007 19:37:50 GMT
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET
CLR 1.1.4322; .NET CLR 2.0.50727; Google-TR-1)
Host: groups.google.com
Connection: Keep-Alive
Cookie: GTZ=-60; __utma=118165087.413883557.1169431526.11731


but I've no idea how to extract source IP and GET's host and file with
awk