No space

My DHCP/Web/NetReg system fell down today because of lack of space. I
had to delete the kernel source to bring it back. What is odd about
this is that this is a 9 gig partition and the files on the filesystem
cannot account but for about 2.

netreg / # df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda3 9397724 9074352 323372 97% /
none 127128 0 127128 0% /dev/shm


netreg / # du -s /*
4145 /bin
0 /boot
513 /chroot
0 /dev
1557 /etc
1085 /home
11944 /lib
0 /mnt
0 /opt
261583 /proc
7990 /root
5288 /sbin
0 /sys
0 /tmp
892569 /usr
360022 /var

What are possible answers/steps to find out what caused this? I am
going to attempt a reboot and see if the space comes back, but I am
worried now that it could happen again.

Noah Roberts wrote:
> My DHCP/Web/NetReg system fell down today because of lack of space.
I
> had to delete the kernel source to bring it back. What is odd about
> this is that this is a 9 gig partition and the files on the
filesystem
> cannot account but for about 2.

A reboot 'solved' the issue. My guess is that something running had
opened files that were deleted...a sort of filesystem memory leak. Is
there any way to track these down?

"Noah Roberts" <(E-Mail Removed)> writes:

>My DHCP/Web/NetReg system fell down today because of lack of space. I
>had to delete the kernel source to bring it back. What is odd about
>this is that this is a 9 gig partition and the files on the filesystem
>cannot account but for about 2.

>netreg / # df
>Filesystem 1K-blocks Used Available Use% Mounted on
>/dev/hda3 9397724 9074352 323372 97% /
>none 127128 0 127128 0% /dev/shm


>netreg / # du -s /*
>4145 /bin
>0 /boot
>513 /chroot
>0 /dev
>1557 /etc
>1085 /home
>11944 /lib
>0 /mnt
>0 /opt
>261583 /proc

/proc takes up no space at all. It is a set of pointers into the kernel
memeory.

>7990 /root
>5288 /sbin
>0 /sys
>0 /tmp
>892569 /usr
>360022 /var

>What are possible answers/steps to find out what caused this? I am
>going to attempt a reboot and see if the space comes back, but I am
>worried now that it could happen again.

They could also be hidden files. Ie files which you erased (rm
/var/log/messages) but were still open at the time, so the inode is left
open, and programs could still dump stuff into them and fill them up.
That would be my suspicion.

Bill Unruh wrote:

> They could also be hidden files. Ie files which you erased (rm
> /var/log/messages) but were still open at the time, so the inode is left
> open, and programs could still dump stuff into them and fill them up.
> That would be my suspicion.
>

Yes, that is mine as well. I have seen it happen but at that time I
knew what caused it.

How do you find these files or, more to the point, the programs that
have them open?

Noah Roberts <(E-Mail Removed)> wrote:
> A reboot 'solved' the issue. My guess is that something running had
> opened files that were deleted...a sort of filesystem memory leak.

It's not a leak, it's that way by design. A file can be opened and then
deleted from the filesystem directory structure. This file is then
available only to the process(es) that have it open - it's otherwise
anonymous. As soon as the file has been closed, its space is freed up
and made available for reuse.


> Is there any way to track these down?

lsof might show you processes with anonymous files

Also, don't forget that "du -s /*" will ignore any dot files in /
Chris

Noah Roberts wrote:
> Bill Unruh wrote:
>
> > They could also be hidden files. Ie files which you erased (rm
> > /var/log/messages) but were still open at the time, so the inode is
left
> > open, and programs could still dump stuff into them and fill them
up.
> > That would be my suspicion.
> >
>
> Yes, that is mine as well. I have seen it happen but at that time I
> knew what caused it.
>
> How do you find these files or, more to the point, the programs that
> have them open?

I believe I have found the issue. It was indeed the log files. They
are getting huge because DHCP is logging every damn thing that happens,
which is a LOT. Also the setup for apache purposfully causes errors
all the time so error_log is filling up rappidly. When I deleted these
files I left the programs like apache and syslog running.

I must look into how to stop DHCP from sending syslogs and how to
delete a log without killing the process that is logging to it. It
must be possible because logrotate does something similar.

"Noah Roberts" <(E-Mail Removed)> writes:


>Noah Roberts wrote:
>> Bill Unruh wrote:
>>
>> > They could also be hidden files. Ie files which you erased (rm
>> > /var/log/messages) but were still open at the time, so the inode is
>left
>> > open, and programs could still dump stuff into them and fill them
>up.
>> > That would be my suspicion.
>> >
>>
>> Yes, that is mine as well. I have seen it happen but at that time I
>> knew what caused it.
>>
>> How do you find these files or, more to the point, the programs that
>> have them open?

>I believe I have found the issue. It was indeed the log files. They
>are getting huge because DHCP is logging every damn thing that happens,
>which is a LOT. Also the setup for apache purposfully causes errors
>all the time so error_log is filling up rappidly. When I deleted these
>files I left the programs like apache and syslog running.

>I must look into how to stop DHCP from sending syslogs and how to
>delete a log without killing the process that is logging to it. It
>must be possible because logrotate does something similar.

It is syslog that does the logging. dhcp and apache just send stuff to
syslog which does the logging.

>/var/log/messages
killall -1 syslogd
The former empties messages, the latter tells syslogd to reread its config
file and to reopen its logging files.

Bill Unruh wrote:
> "Noah Roberts" <(E-Mail Removed)> writes:

> >I must look into how to stop DHCP from sending syslogs and how to
> >delete a log without killing the process that is logging to it. It
> >must be possible because logrotate does something similar.
>
> It is syslog that does the logging. dhcp and apache just send stuff
to
> syslog which does the logging.
>
> >/var/log/messages
> killall -1 syslogd
> The former empties messages, the latter tells syslogd to reread its
config
> file and to reopen its logging files.

My system uses metalog. There is no /var/log/messages. There is a
multi-file system and DHCP messages seem to go to "everything". I
changed the log-level of that file and I am hoping that keeps the DHCP
messages out. So far it is working.

I also realized that logrotate is not running or installed, so I am
putting that in. I should also be able to lower the log rate of
apache...and those two things hopefully stop this from occuring again.
I am a temp and they do not know Linux at all :P I don't think they
could fix it should it happen again without me here...

"Noah Roberts" <(E-Mail Removed)> writes:


>Bill Unruh wrote:
>> "Noah Roberts" <(E-Mail Removed)> writes:

>> >I must look into how to stop DHCP from sending syslogs and how to
>> >delete a log without killing the process that is logging to it. It
>> >must be possible because logrotate does something similar.
>>
>> It is syslog that does the logging. dhcp and apache just send stuff
>to
>> syslog which does the logging.
>>
>> >/var/log/messages
>> killall -1 syslogd
>> The former empties messages, the latter tells syslogd to reread its
>config
>> file and to reopen its logging files.

>My system uses metalog. There is no /var/log/messages. There is a
>multi-file system and DHCP messages seem to go to "everything". I
>changed the log-level of that file and I am hoping that keeps the DHCP
>messages out. So far it is working.

>I also realized that logrotate is not running or installed, so I am
>putting that in. I should also be able to lower the log rate of
>apache...and those two things hopefully stop this from occuring again.
>I am a temp and they do not know Linux at all :P I don't think they
>could fix it should it happen again without me here...


I do not believe that dhcp or apache could fill up the log files to that
extent. YOu might get a few MB (they would put in say 3 lines into the log,
which is about 300 characters. Even at 1 per second, that is only 10MB per
day, and I strongly doubt eitehr does one per second.

Bill Unruh wrote:
> "Noah Roberts" <(E-Mail Removed)> writes:

> >My system uses metalog. There is no /var/log/messages. There is a
> >multi-file system and DHCP messages seem to go to "everything". I
> >changed the log-level of that file and I am hoping that keeps the
DHCP
> >messages out. So far it is working.
>
> I do not believe that dhcp or apache could fill up the log files to
that
> extent. YOu might get a few MB (they would put in say 3 lines into
the log,
> which is about 300 characters. Even at 1 per second, that is only
10MB per
> day, and I strongly doubt eitehr does one per second.

netreg root # ls -l /var/log/everything/
total 584
-rw-r--r-- 1 root root 82754 Jan 26 10:22 current
-rw-r--r-- 1 root root 100000 Jan 26 01:51 log-2005-01-26-02:51:32
-rw-r--r-- 1 root root 100015 Jan 26 03:01 log-2005-01-26-04:01:20
-rw-r--r-- 1 root root 100042 Jan 26 04:22 log-2005-01-26-05:22:54
-rw-r--r-- 1 root root 100036 Jan 26 05:55 log-2005-01-26-06:55:58
-rw-r--r-- 1 root root 100054 Jan 26 07:27 log-2005-01-26-08:27:56

That is 10 Megs every 1.5 hours or so. That adds up pretty damn fast I
think.

In looking at the contents of those files, almost all of it is DHCP
such as:
Jan 26 07:27:54 [dhcpd] DHCPREQUEST for 10.4.254.158 from
00:07:e9:6d:aa:1f (Mike18) via eth0
Jan 26 07:27:54 [dhcpd] DHCPACK on 10.4.254.158 to 00:07:e9:6d:aa:1f
(Mike18) via eth0

In fact:

netreg root # grep DHCP /var/log/everything/log-2005-01-26-08\:27\:56 |
wc
971 11722 90772
netreg root # cat /var/log/everything/log-2005-01-26-08\:27\:56 | wc
1094 12646 100054

so around 92% of the 10M generated in 1.5 hours (see above) is DHCP
info. That is pretty excessive...and surprising, yes.

Part of the issue may be that dhcp is constantly being restarted by
NetReg. To understand why read about NetReg...in short it is adding
host entries to dhcpd.conf and dhcpd requires a restart to read its
conf.

This machine is serving DHCP for about 200 Windows computers that are
in varying level of disarray...College Dorms...

So I lowered the 'everything' log level to more important messages and
set up logrotate to deal with apache's logs (metalog does its own).
That should fix the issue...