Apache Logs DNS Root server IP Addresses only

This issue started happening after upgrading a server from a single
processor to an 8 cpu monster. The Apache logs (both access and error)
contain only ROOT DNS server IP addresses for all virtual and non
hosts.

eg:
168.137.203.9 - - [23/Aug/2007:20:26:14 -0700] "GET {URL} HTTP/1.1
" 200 308 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; S
V1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR
1.1.4322)"
168.137.203.9 - - [23/Aug/2007:20:26:21 -0700] "GET {URL} HTTP/1.1"
304 - "-" "
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
128.1.0.0 - - [23/Aug/2007:20:26:24 -0700] "GET {URL} HTTP/1.1" 200
5162 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5)
Gecko/20070321 Netscape/8.1.3"

Three different people all associated with what I believe to be a root
DNS server. Almost 25K people visit the site each day, it isn't
possible for all of them to be originating from 3 IP Addresses
especially considering the sites are geared towards a younger audience
than H&R Block. :-)

The Apache (2.0.59refork) conf where the virtualhosts are defined:

NameVirtualHost *:80
<VirtualHost *:80>
ServerName tidal.gdofwr.com # resolves to a real IPv4 address
DocumentRoot /www/htdocs/tidal-main/
# Used without rotatelogs produces the same results
CustomLog "|/www/apache/sbin/rotatelogs /www/log/tidal-main/access-
%Y_%m_%d.log 1990M" combined
ErrorLog log/tidal-main/error.log
<Directory /www/htdocs/tidal-main>
Options -Indexes
</Directory>
</VirtualHost>

<VirtualHost *:80>
ServerName fireball.gdofwr.com # resolves to a real IPv4 address
DocumentRoot /www/htdocs/fireball-main/
CustomLog "|/www/apache/sbin/rotatelogs /www/log/fireball-main/
access-%Y_%m_%d.log 1990M" combined
ErrorLog log/fireball-main/error.log
<Directory /www/htdocs/fireball-main>
Options -Indexes
</Directory>
</VirtualHost>

Other services running include: bind 9.2.4-24, vsftpd 2.0.1-5, MySQL
5.1, nagios, iptables +apf, sendmail, SVN (compiled with neon),
BerkeleyDB 4.4 and xinetd. The OS is CentOS4 using Kernel version
2.6.9-55.0.2.ELsmp

Bind seemed a likely culprit and I turned off using my own DNS to
using the web providers DNS but the problem persisted. The only other
likely issue I can think of might be something to do with a rule with
iptables / forwarding, but after turning off the firewall the problem
still existed.

I've searched through the Apache mailing list archive for a solution /
cause, and then searched through the archives on this group and
several others.

Why is Apache logging root DNS IP addresses instead of logging the
user' incoming IP address? I'm sure it's something dead simple I'm
missing, but if anyone can assist it would be immensely appreciated.

The Apache server is "hand built" from a script (APR 0.9.14):

#! /bin/sh
#
# Created by configure

"./configure" \
"--enable-layout=Blackhole" \
"--disable-ipv6" \
"--enable-ssl" \
"--enable-deflate" \
"--enable-mime-magic" \
"--enable-static-htpasswd" \
"--enable-static-rotatelogs" \
"--enable-static-logresolve" \
"--enable-ext-filter" \
"--enable-rewrite" \
"--enable-dav" \
"--enable-so" \
"--with-apr=/usr/local/apr/bin/apr-config" \
"--with-apr-util=/usr/local/apr/bin/apu-config" \
"--with-berkeley-db=/usr/local/BerkeleyDB.4.4/" \
"--enable-suexec" \
"--with-mpm=prefork" \
"--enable-modules=MOST" \

Thank you.

Is this a common problem and I'm just not able to see the problem
above, or has no one ever witnessed anything like it? I know it may be
a bit above the experience level of this groups issues but I thought
I'd take a shot at it. Maybe I posted in the wrong group, does anyone
know another that might fit better?

Cheers

On Tue, 28 Aug 2007 17:47:23 -0700, Sentine| wrote:

> Is this a common problem and I'm just not able to see the problem above,

It is not a problem. That is the default - change it if you like and
slow down the server's performance.

On Aug 28, 9:08 pm, Dave Uhring <daveuhr...@yahoo.com> wrote:
> On Tue, 28 Aug 2007 17:47:23 -0700, Sentine| wrote:
> > Is this a common problem and I'm just not able to see the problem above,
>
> It is not a problem. That is the default - change it if you like and
> slow down the server's performance.


I want to change it but I don't know how to do so. And so far everyone
I've asked about it doesn't have an answer to correct Apache to show
the real user IP address instead of a Root DNS IP for every user.
Quite bizarre.

On Thu, 06 Sep 2007 21:11:39 -0700, Sentine| wrote:

> On Aug 28, 9:08 pm, Dave Uhring <daveuhr...@yahoo.com> wrote:
>> On Tue, 28 Aug 2007 17:47:23 -0700, Sentine| wrote:

>> It is not a problem. That is the default - change it if you like and
>> slow down the server's performance.
>
> I want to change it but I don't know how to do so. And so far everyone
> I've asked about it doesn't have an answer to correct Apache to show the
> real user IP address instead of a Root DNS IP for every user. Quite
> bizarre.

The IP addresses you posted originally are *not* those of any of the root
DNS servers. You posted only 2 discrete addresses, not three, and one of
them is from H&R Block, the other a bogus network address assigned to BBN
Communications.

It is most likely that those hosts are worm infected Microsfot shitware
seeking to infect your http server. H&R Block's PCs are just as
vulnerable to that crap as the system you used to post your articles.

As for the ability of Apache to report the names of those hosts, forget
about it:

$ host 168.137.203.9
Host 9.203.137.168.in-addr.arpa not found: 3(NXDOMAIN)

$ host 128.1.0.0
Host 0.0.1.128.in-addr.arpa not found: 3(NXDOMAIN)

If you are really serious about degrading your server and needlessly
increasing traffic on the Internet then set

HostnameLookups On

On Sep 7, 2:56 am, Dave Uhring <daveuhr...@yahoo.com> wrote:
> On Thu, 06 Sep 2007 21:11:39 -0700, Sentine| wrote:
> > On Aug 28, 9:08 pm, Dave Uhring <daveuhr...@yahoo.com> wrote:
> >> On Tue, 28 Aug 2007 17:47:23 -0700, Sentine| wrote:
> >> It is not a problem. That is the default - change it if you like and
> >> slow down the server's performance.
>
> > I want to change it but I don't know how to do so. And so far everyone
> > I've asked about it doesn't have an answer to correct Apache to show the
> > real user IP address instead of a Root DNS IP for every user. Quite
> > bizarre.
>
> The IP addresses you posted originally are *not* those of any of the root
> DNS servers. You posted only 2 discrete addresses, not three, and one of
> them is from H&R Block, the other a bogus network address assigned to BBN
> Communications.
>
> It is most likely that those hosts are worm infected Microsfot shitware
> seeking to infect your http server. H&R Block's PCs are just as
> vulnerable to that crap as the system you used to post your articles.
>
> As for the ability of Apache to report the names of those hosts, forget
> about it:
>
> $ host 168.137.203.9
> Host 9.203.137.168.in-addr.arpa not found: 3(NXDOMAIN)
>
> $ host 128.1.0.0
> Host 0.0.1.128.in-addr.arpa not found: 3(NXDOMAIN)
>
> If you are really serious about degrading your server and needlessly
> increasing traffic on the Internet then set
>
> HostnameLookups On

If I connect to the HTTP server my IP address becomes 128.1.0.0 ..

On Fri, 07 Sep 2007 06:50:35 -0700, Sentine| wrote:

> If I connect to the HTTP server my IP address becomes 128.1.0.0 ..

That is still not an address used by any of the root DNS servers.

What address is reported by the server when you connect using a different
protocol such as sshd or telnet?

Do you have any other hosts on the same network as that web server? If
so do they report 128.1.0.0 or 74.99.88.227?

On Sep 7, 10:41 am, Dave Uhring <daveuhr...@yahoo.com> wrote:
> On Fri, 07 Sep 2007 06:50:35 -0700, Sentine| wrote:
> > If I connect to the HTTP server my IP address becomes 128.1.0.0 ..
>
> That is still not an address used by any of the root DNS servers.
>
> What address is reported by the server when you connect using a different
> protocol such as sshd or telnet?
>
> Do you have any other hosts on the same network as that web server? If
> so do they report 128.1.0.0 or 74.99.88.227?

All other hosts on the same network report the correct IP address
regardless of protocol used.

It only seems to be an issue with Apache on this one host; sshd,
telnet, ftp all report the correct IP address.
I compiled the latest version of Apache 2.2.x with the same result,
and flipped back to 1.3.x, but still the same. It's like Apache is
linking to a bogus resolv file or something.

Very much appreciate the assistance.

Thank you.

On Sat, 08 Sep 2007 09:48:04 -0700, Sentine| wrote:

> All other hosts on the same network report the correct IP address
> regardless of protocol used.
>
> It only seems to be an issue with Apache on this one host; sshd, telnet,
> ftp all report the correct IP address. I compiled the latest version of
> Apache 2.2.x with the same result, and flipped back to 1.3.x, but still
> the same. It's like Apache is linking to a bogus resolv file or
> something.

If the daemons using other protocols record the correct connection
address it is unlikely that /etc/resolv.conf is incorrect.

You might try replacing your httpd.conf with the default file, making
whatever specific local configurations are required although I cannot
imagine how httpd could rewrite connecting IP addresses.

Check your firewall rules, specially if you are redirecting port 80
packets. Is something there capable of rewriting the IP addresses?