Networking Forums

Networking Forums > Computer Networking > Linux Networking > dhcpd dies every morning

Reply
Thread Tools Display Modes

dhcpd dies every morning

 
 
Clemens von Musil
Guest
Posts: n/a

 
      10-19-2004, 08:26 AM
Hi all,

I run dhcpd on my fileserver and find, that dhcpd
dies every morning between 5 and 6 AM.

I wrote a little script (see below) to restart
dhcpd whenever needed and run it every minute by
cron. The script fails to restart the server. Log
output below.
The timestamps in that log seem to have different
localisations - not really a problem but strange.

The command "service dhcpd start" or
"/etc/init.d/dhcpd strt" works fine and gives
peace until next morning.

syslog shows nothing special, a snip is below. The
only thing, I discovered are the entries

- Oct 19 05:49:47 dsasrv dhcpd: nss_ldap:
reconnecting to LDAP server...
- Oct 19 05:49:47 dsasrv dhcpd: nss_ldap:
reconnected to LDAP server after 1 attempt(s)

They appear every morning at dhcpd's diing times.
I don't know exactly, what they mean, but they
don't sound terrible in my ears.

That's all, I know. Do you can point me a
direction to search the problem?

Thanks in advance,
Clemens

-------------

The script/logs:

--------

1) The script, that sould recover the dhcpd server:

#!/bin/bash
PID="/var/run/dhcpd.pid";
LOGFILE="/var/log/services.log";

function reincarnate() {
service dhcpd start>/dev/null;
#TMP=`ps -e | grep $(cat $PID) | wc -l`;
if [ `ps -e | grep $(cat $PID) | wc -l`
== "0" ]; then
echo "$(date +%c) : DHCP died,
reincarnation FAILED.">>$LOGFILE;
echo "Critical error: Failed to
reincarnate DHCPD!";
else
echo "$(date +%c) : DHCP died,
reincarnation SUCCEEDED.">>$LOGFILE;
echo "Reincarnation suceeded.";
fi

}

if [ -f $PID ]; then
TMP=`ps -e | grep $(cat $PID) | wc -l`;
if [ $TMP == "0" ]; then
echo "DHCPD died, reincarnate...";
reincarnate;
else
echo "DHCPD is running.";
fi
else
echo "No lockfile found. Assuming service
is down and reincarnate...";
reincarnate;
exit;
fi

----------

2)
The output, script no 1 produced this morning.
First to entries are result of a test run
yesterday. The last entry was generated after
restarting dhcpd and "kill"ing its process:

Mo 18 Okt 2004 14:55:20 CEST : DHCP died,
reincarnation SUCCEEDED.
Mo 18 Okt 2004 14:55:41 CEST : DHCP died,
reincarnation SUCCEEDED.
Tue Oct 19 05:50:00 2004 : DHCP died,
reincarnation FAILED.
Tue Oct 19 05:51:00 2004 : DHCP died,
reincarnation FAILED.

<...>

Tue Oct 19 08:55:00 2004 : DHCP died,
reincarnation FAILED.
Di 19 Okt 2004 08:55:05 CEST : DHCP died,
reincarnation SUCCEEDED.

----------

3) The syslog around the diing time (seems to be
between 05:49:47 and 06:00:00):

Oct 19 05:49:00 dsasrv CROND[12904]: (root) CMD
(/usr/local/sbin/check_dhcpd)
Oct 19 05:49:00 dsasrv postfix/pickup[12417]:
9F6B8A3AF0: uid=0 from=<root>
Oct 19 05:49:00 dsasrv postfix/cleanup[11808]:
9F6B8A3AF0:
message-id=<(E-Mail Removed)>
Oct 19 05:49:00 dsasrv postfix/qmgr[1437]:
9F6B8A3AF0: from=<root@dsanet>, size=445, nrcpt=1
(queue active)
Oct 19 05:49:00 dsasrv postfix/local[11810]:
9F6B8A3AF0: to=<root@dsanet>, orig_to=<root>,
relay=local, delay=0, status=sent (ma
ilbox)

Oct 19 05:49:47 dsasrv dhcpd: Wrote 0 deleted host
decls to leases file.
Oct 19 05:49:47 dsasrv dhcpd: Wrote 0 new dynamic
host decls to leases file.
Oct 19 05:49:47 dsasrv dhcpd: Wrote 16 leases to
leases file.
Oct 19 05:49:47 dsasrv dhcpd: nss_ldap:
reconnecting to LDAP server...
Oct 19 05:49:47 dsasrv dhcpd: nss_ldap:
reconnected to LDAP server after 1 attempt(s)

Oct 19 05:50:00 dsasrv CROND[12914]: (root) CMD
(/usr/local/sbin/check_dhcpd)
Oct 19 05:50:00 dsasrv postfix/pickup[12417]:
BE02BA3AF0: uid=0 from=<root>
Oct 19 05:50:00 dsasrv postfix/cleanup[11808]:
BE02BA3AF0:
message-id=<(E-Mail Removed)>
Oct 19 05:50:00 dsasrv postfix/qmgr[1437]:
BE02BA3AF0: from=<root@dsanet>, size=565, nrcpt=1
(queue active)
Oct 19 05:50:00 dsasrv postfix/local[11810]:
BE02BA3AF0: to=<root@dsanet>, orig_to=<root>,
relay=local, delay=0, status=sent (ma
ilbox)
 
Reply With Quote
 
 
 
 
Davide Bianchi
Guest
Posts: n/a

 
      10-19-2004, 08:42 AM
On 2004-10-19, Clemens von Musil <(E-Mail Removed)> wrote:
> I run dhcpd on my fileserver and find, that dhcpd
> dies every morning between 5 and 6 AM.


And no idea why? Is it terminated by some log rotation process?
What's in the log?

> - Oct 19 05:49:47 dsasrv dhcpd: nss_ldap:
> reconnecting to LDAP server...
> - Oct 19 05:49:47 dsasrv dhcpd: nss_ldap:
> reconnected to LDAP server after 1 attempt(s)


Youd dhcpd uses ldap?

Davide

--
Some Windows were made to be broken.
 
Reply With Quote
 
Clemens von Musil
Guest
Posts: n/a

 
      10-19-2004, 10:20 AM
Hi Davide,

thanks a lot for your logrotation idea. I just
tested logrotate-managed logile/service,
especially those with kill/restart scripts. I
didn't manage to provoke a dhcpd-death.

Unfortunately, I deleted the actual syslog as
sideeffect of my testing (I'm so stupid...). But I
remember very well, that along the last death,
there was nothing but to postfix entries.

cron sends a logfile entry for my check_dhcpd
script every minute. At 05:49:00, dhcpd was up and
running. At 05:50:00, dhcpd was dead. The only log
entries were those, produced by postfix while
sending the mentioned cron-messages and the two
nss_ldap entries at 05:49:47.

That's, why I asusme, that dhcpd died between
05:49:47 and 05:50:00.
Other logfiles show no special event at or around
05:49:47.

I don't know, what dhcpd does with ldap. The only
thing, I could imgine, is a lookup for other
hosts. I configured nss to share hostinformation
by ldap.

From /etc/nssswitch.conf:
hosts: files ldap nisplus nis dns wins

All I am sure of are big ?'s ...

do have any other idea?
Thank you,
Clemens


Davide Bianchi wrote:
> On 2004-10-19, Clemens von Musil <(E-Mail Removed)> wrote:
>
>>I run dhcpd on my fileserver and find, that dhcpd
>>dies every morning between 5 and 6 AM.

>
>
> And no idea why? Is it terminated by some log rotation process?
> What's in the log?
>
>
>>- Oct 19 05:49:47 dsasrv dhcpd: nss_ldap:
>>reconnecting to LDAP server...
>>- Oct 19 05:49:47 dsasrv dhcpd: nss_ldap:
>>reconnected to LDAP server after 1 attempt(s)

>
>
> Youd dhcpd uses ldap?
>
> Davide
>

 
Reply With Quote
 
Davide Bianchi
Guest
Posts: n/a

 
      10-19-2004, 10:35 AM
On 2004-10-19, Clemens von Musil <(E-Mail Removed)> wrote:
> I don't know, what dhcpd does with ldap. The only thing, I could
> imgine, is a lookup for other hosts.


Maybe the problem is a failure in LDAP that makes dhcpd crash.
If dhcp dies every day at the same time maybe is worth to check
what runs at that time, also, see if your dhcpd has a 'verbose'
log feature or a debug options so you can see some information.

Davide

--
Q: How many Microsoft Programmers does it take to screw in a lightbulb?
A: It cannot be done. You will need to upgrade your house.
-- Geoff Johnson
 
Reply With Quote
 
Clemens von Musil
Guest
Posts: n/a

 
      10-19-2004, 12:34 PM
Hello Davide

> Maybe the problem is a failure in LDAP that makes dhcpd crash.


LDAP produces (or gets) some heavy traffic in the
morning hours. I used to find out, for what all
the traffic should be for, but failed..
At loglevel 128, openldap produces more than 600
megabyte logdata in less than one hour..
Maybe the problem.

I can't reproduce the behaviour, but I will study
the situation tomorrow morning - and hopefully
gonna have something to post here.

Thank you so long,
Clemens
 
Reply With Quote
 
 
 
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
VM outage this morning Graham. Broadband 1 12-16-2010 09:26 PM
Zen very very slow this morning Ken Broadband 9 02-28-2007 08:14 PM
PLUS NET SITES DOWN THIS MORNING Gel Broadband 3 08-05-2005 04:54 PM
PLUS NET SITES DOWN THIS MORNING Gel Broadband 2 08-05-2005 10:44 AM
Upgraded this morning (Plus.net) Bales1983 (Lee Bales) Broadband 8 05-23-2005 01:25 PM



1 2 3 4 5 6 7 8 9 10 11