Is a HTTP header a clear text?

Hi
I hope this is the right group or if not please suggest a more
suitable group =). Basically we are developing a packet sniffer to
check the http header. The problem is that in one box we can get some
clear text for the http header, but in another box, using the same
program, Linux version and page that is accessed, we only see encoded/
encrypted?? values. The security enhanced linux for both computers is
disabled, but still the same result (different values). We searched
for http header descriptions and found some of the keywords but
unfortunately that is not what we see when we capture the packets. We
see special characters. What is the best way using C in parsing/
retrieving the headers? We looked upon the wget and curl source code
but it is too complicated for us. We want to know the concept on how
to retrieve those header values. Hoping for a favorable response.
Thanks in advanced!

On Oct 18, 3:34 pm, Sonny <smani...@gmail.com> wrote:
> Hi
> I hope this is the right group or if not please suggest a more
> suitable group =). Basically we are developing a packet sniffer to
> check the http header. The problem is that in one box we can get some
> clear text for the http header, but in another box, using the same
> program, Linux version and page that is accessed, we only see encoded/
> encrypted?? values. The security enhanced linux for both computers is
> disabled, but still the same result (different values). We searched
> for http header descriptions and found some of the keywords but
> unfortunately that is not what we see when we capture the packets. We
> see special characters. What is the best way using C in parsing/
> retrieving the headers? We looked upon the wget and curl source code
> but it is too complicated for us. We want to know the concept on how
> to retrieve those header values. Hoping for a favorable response.
> Thanks in advanced!

HTTP header is clear text regardless of OS.

HTTPS header is encrypted (nothing you can do about it, it's how it
was designed. Trying to intercept HTTPS communications will raise a
warning on either the client program or the server or both so the user
will know you're sniffing. Further, some client software will flat out
refuse intercepted data)

HTTP forwarded through a tunnel such as VPN or SSH is encrypted. In
this case you need to know where to capture packets. Capture them at
the tunnel interface instead of eth0.

Sonny wrote:
> Hi
> I hope this is the right group or if not please suggest a more
> suitable group =). Basically we are developing a packet sniffer to
> check the http header. The problem is that in one box we can get some
> clear text for the http header, but in another box, using the same
> program, Linux version and page that is accessed, we only see encoded/
> encrypted?? values. The security enhanced linux for both computers is
> disabled, but still the same result (different values). We searched
> for http header descriptions and found some of the keywords but
> unfortunately that is not what we see when we capture the packets. We
> see special characters. What is the best way using C in parsing/
> retrieving the headers? We looked upon the wget and curl source code
> but it is too complicated for us. We want to know the concept on how
> to retrieve those header values. Hoping for a favorable response.
> Thanks in advanced!
>
HTTP headers are defined in RFC2616. They are clear text containing
mainly ASCII characters.

Robert

On Oct 18, 3:34 am, Sonny <smani...@gmail.com> wrote:
> Hi
> I hope this is the right group or if not please suggest a more
> suitable group =). Basically we are developing a packet sniffer to
> check the http header. The problem is that in one box we can get some
> clear text for the http header, but in another box, using the same
> program, Linux version and page that is accessed, we only see encoded/
> encrypted?? values.
[snip]

I can't be entirely certain, but the HTTP 1.1 RFC seems to allow the
server to use one of the negotiated compression techniques on outbound
traffic. Your "encoded/encrypted" data may just be GZIPped or BZIP2ed
text data.

Other possibilities:
HTTP through a VPN will be encrypted
HTTPS will be encrypted

On Oct 19, 2:32 am, Lew Pitcher <lpitc...@teksavvy.com> wrote:
> On Oct 18, 3:34 am, Sonny <smani...@gmail.com> wrote:> Hi
> > I hope this is the right group or if not please suggest a more
> > suitable group =). Basically we are developing a packet sniffer to
> > check the http header. The problem is that in one box we can get some
> > clear text for the http header, but in another box, using the same
> > program, Linux version and page that is accessed, we only see encoded/
> > encrypted?? values.
>
> [snip]
>
> I can't be entirely certain, but the HTTP 1.1 RFC seems to allow the
> server to use one of the negotiated compression techniques on outbound
> traffic. Your "encoded/encrypted" data may just be GZIPped or BZIP2ed
> text data.
>
> Other possibilities:
> HTTP through a VPN will be encrypted
> HTTPS will be encrypted

hmm, so it might be GZIPped/BZIP2ed. so how would i parse that if that
is the case? is there a general way to do it. btw, thanks for the info.

On 2007-10-19, Sonny <(E-Mail Removed)> wrote:
>
> hmm, so it might be GZIPped/BZIP2ed. so how would i parse that if that
> is the case? is there a general way to do it. btw, thanks for the info.

Sure, pipe the data portion of the packets (in the correct order)
through gzip or bzip2. If you control the web server, you can turn off
compression if you want to view the data in cleartext instead of
uncompressing it first.

--keith

--
kkeller-(E-Mail Removed)
(try just my userid to email me)
AOLSFAQ=http://www.therockgarden.ca/aolsfaq.txt
see X- headers for PGP signature information

On Oct 19, 11:09 am, Keith Keller <kkeller-use...@wombat.san-
francisco.ca.us> wrote:
> On 2007-10-19, Sonny <smani...@gmail.com> wrote:
>
>
>
> > hmm, so it might be GZIPped/BZIP2ed. so how would i parse that if that
> > is the case? is there a general way to do it. btw, thanks for the info.
>
> Sure, pipe the data portion of the packets (in the correct order)
> through gzip or bzip2. If you control the web server, you can turn off
> compression if you want to view the data in cleartext instead of
> uncompressing it first.
>
> --keith
>
> --
> kkeller-use...@wombat.san-francisco.ca.us
> (try just my userid to email me)
> AOLSFAQ=http://www.therockgarden.ca/aolsfaq.txt
> see X- headers for PGP signature information

Ok, thanks, i'll try to do that. Btw, here is the code i'm using which
is found in http://everything2.com/index.pl?node_id=1459992. Actually
we're just building a prototype for proof of concept. I'm printing the
payload at the bottom, and on some computers, its encoded/compressed
data we're seeing. What's the best way, let's say, to get the value of
host?

Code:
#include <pcap.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <netinet/ether.h>
#include <netinet/ip.h>
#include <netinet/tcp.h>
#include <sys/socket.h>

char ethlen, iplen, tcplen;
void handler(char *, const struct pcap_pkthdr *, const u_char *);

int main(int argc, char **argv)
{
int buffsize = 65535;
int promisc = 1;
int timeout = 1000;

char pcap_err[PCAP_ERRBUF_SIZE];
u_char buffer[255];
char *dev;
struct in_addr net, mask;
pcap_t *pcap_nic;
struct bpf_program filter;

ethlen = sizeof(struct ether_header);
iplen = sizeof(struct iphdr);
tcplen = sizeof(struct tcphdr);

printf("ethlen: %i\niplen: %i\ntcplen: %i\n", ethlen, iplen,
tcplen);

if(!(dev = pcap_lookupdev(pcap_err)))
{
perror(pcap_err);
exit(-1);
}

printf("Dev: %s\n\n", dev);

if((pcap_nic = pcap_open_live(dev, buffsize, promisc, timeout,
pcap_err))== NULL)
{
perror(pcap_err);
exit(-1);
}

if(pcap_lookupnet(dev, &net.s_addr, &mask.s_addr, pcap_err) == -1)
{
perror(pcap_err);
exit(-1);
}
printf("net: %s\tmask: %s\n\n", inet_ntoa(net),
inet_ntoa(mask));

if(pcap_compile(pcap_nic, &filter, "tcp src port 80", 0,
net.s_addr) == -1)
{
perror(pcap_err);
exit(-1);
}

if(pcap_setfilter(pcap_nic, &filter) == -1)
{
perror(pcap_err);
exit(-1);
}

pcap_loop(pcap_nic, -1, (pcap_handler)handler, buffer);
return 0;
}

void handler(char *usr, const struct pcap_pkthdr *header, const u_char
*pkt)
{
struct ether_header *ethheader;
struct iphdr *ipheader;
struct tcphdr *tcpheader;
struct in_addr source, dest;
char *payload;

ethheader = (struct ether_header *)pkt;
ipheader = (struct iphdr *)(pkt + ethlen);
tcpheader = (struct tcphdr *)(pkt + ethlen + iplen);

payload = (char*) (pkt + ethlen + iplen + tcplen);

if(tcpheader->syn && tcpheader->ack)
{
source.s_addr = ipheader->saddr;
dest.s_addr = ipheader->daddr;

printf("From: %s \t%i\t", inet_ntoa(source), ntohs(tcpheader-
>source));
printf("To: %s \t%i\n", inet_ntoa(dest), ntohs(tcpheader-
>dest));
printf("Length: %i", ntohs(ipheader->tot_len));
printf("\n");
printf("Flags: ");
if(tcpheader->urg)
printf("URG");
if(tcpheader->ack)
printf("ACK ");
if(tcpheader->psh)
printf("PSH ");
if(tcpheader->rst)
printf("RST ");
if(tcpheader->syn)
printf("SYN ");
if(tcpheader->fin)
printf("FIN ");
printf("\n\n");
}
printf("Payload: %s\n", payload);
return;
}

On 2007-10-19, Sonny <(E-Mail Removed)> wrote:
> On Oct 19, 11:09 am, Keith Keller <kkeller-use...@wombat.san-
> francisco.ca.us> wrote:
>>
>> Sure, pipe the data portion of the packets (in the correct order)
>> through gzip or bzip2. If you control the web server, you can turn off
>> compression if you want to view the data in cleartext instead of
>> uncompressing it first.
>
> Ok, thanks, i'll try to do that. Btw, here is the code i'm using which
> is found in http://everything2.com/index.pl?node_id=1459992. Actually
> we're just building a prototype for proof of concept. I'm printing the
> payload at the bottom, and on some computers, its encoded/compressed
> data we're seeing. What's the best way, let's say, to get the value of
> host?

Do what I suggested. Since you're writing code, just include the gzip
or bzip2 libraries. As I wrote, if you control the webserver, you can
turn off compression, which will eliminate a debugging point while
you're still developing your code; later you can add compression to test
that part of the code. (If you control the client, you can probably
turn off compression there, too.)

--keith

--
kkeller-(E-Mail Removed)
(try just my userid to email me)
AOLSFAQ=http://www.therockgarden.ca/aolsfaq.txt
see X- headers for PGP signature information

On Oct 19, 12:12 pm, Keith Keller <kkeller-use...@wombat.san-
francisco.ca.us> wrote:
> On 2007-10-19, Sonny <smani...@gmail.com> wrote:
>
> > On Oct 19, 11:09 am, Keith Keller <kkeller-use...@wombat.san-
> > francisco.ca.us> wrote:
>
> >> Sure, pipe the data portion of the packets (in the correct order)
> >> through gzip or bzip2. If you control the web server, you can turn off
> >> compression if you want to view the data in cleartext instead of
> >> uncompressing it first.
>
> > Ok, thanks, i'll try to do that. Btw, here is the code i'm using which
> > is found inhttp://everything2.com/index.pl?node_id=1459992. Actually
> > we're just building a prototype for proof of concept. I'm printing the
> > payload at the bottom, and on some computers, its encoded/compressed
> > data we're seeing. What's the best way, let's say, to get the value of
> > host?
>
> Do what I suggested. Since you're writing code, just include the gzip
> or bzip2 libraries. As I wrote, if you control the webserver, you can
> turn off compression, which will eliminate a debugging point while
> you're still developing your code; later you can add compression to test
> that part of the code. (If you control the client, you can probably
> turn off compression there, too.)
>
> --keith
>
> --
> kkeller-use...@wombat.san-francisco.ca.us
> (try just my userid to email me)
> AOLSFAQ=http://www.therockgarden.ca/aolsfaq.txt
> see X- headers for PGP signature information

we tried that but still the same.

On Oct 19, 5:48 pm, Sonny <smani...@gmail.com> wrote:
> On Oct 19, 12:12 pm, Keith Keller <kkeller-use...@wombat.san-
>
>
>
> francisco.ca.us> wrote:
> > On 2007-10-19, Sonny <smani...@gmail.com> wrote:
>
> > > On Oct 19, 11:09 am, Keith Keller <kkeller-use...@wombat.san-
> > > francisco.ca.us> wrote:
>
> > >> Sure, pipe the data portion of the packets (in the correct order)
> > >> through gzip or bzip2. If you control the web server, you can turn off
> > >> compression if you want to view the data in cleartext instead of
> > >> uncompressing it first.
>
> > > Ok, thanks, i'll try to do that. Btw, here is the code i'm using which
> > > is found inhttp://everything2.com/index.pl?node_id=1459992. Actually
> > > we're just building a prototype for proof of concept. I'm printing the
> > > payload at the bottom, and on some computers, its encoded/compressed
> > > data we're seeing. What's the best way, let's say, to get the value of
> > > host?
>
> > Do what I suggested. Since you're writing code, just include the gzip
> > or bzip2 libraries. As I wrote, if you control the webserver, you can
> > turn off compression, which will eliminate a debugging point while
> > you're still developing your code; later you can add compression to test
> > that part of the code. (If you control the client, you can probably
> > turn off compression there, too.)
>
> > --keith
>
> > --
> > kkeller-use...@wombat.san-francisco.ca.us
> > (try just my userid to email me)
> > AOLSFAQ=http://www.therockgarden.ca/aolsfaq.txt
> > see X- headers for PGP signature information
>
> we tried that but still the same.

To be perfectly clear. HTTP header is ALWAYS cleartext (unless wrapped
in SSL). It CAN NEVER be bzip or gzip. Only HTTP DATA is allowed to be
compressed (you need the header to know if the data is compressed or
not).

So, if you're not seeing cleartext it's either secured (SSL) or
tunneled somehow (VPN) or you're just reading the wrong part of the
packet.