On a system with frees/swan 2.05 and linux-2.4.25 (Slackware 9.0
based) that worked for a couple of months with two ipsec tunnels.
Suddenly the pluto starts to take lots of resources up to 98% and more
which makes the system no longer available for network access, however
the tunnels do come up when we start ipsec.
On the dns are a some lookups for TXT and KEY records but not enough
to bring down a system (in the order of 10 to 20 lookup) The process
that grabs all the cpu is pluto, while until monday it worked fine.
On friday there was a subdomain added to our dns which is hosted by
the isp and the subdomain is hosted on our own server. But we don't
use any name from that subdomain
we get this type of message in our log
Sep 6 18:31:53 firy1 pluto[693]: ERROR: pfkey write() of
SADB_X_ADDFLOW message
3397 for flow %pass failed. Errno 12: Cannot allocate memory
Sep 6 18:31:53 firy1 pluto[693]: | 02 0e 00 0b 17 00 00 00 45 0d
00 00 b5
02 00 00
in the /var/log/messages we get this
Sep 7 16:59:10 firy1 kernel: klips_debug

fkey_acquire: sending up
acquire mess
age for satype=3(ESP) to socket=0pc67429b4 failed with error=-12.
Sep 7 16:59:41 firy1 last message repeated 4283 times
Sep 7 17:00:42 firy1 last message repeated 8966 times
Sep 7 17:01:43 firy1 last message repeated 8876 times
Sep 7 17:02:08 firy1 last message repeated 3665 times
does anybody has a suggestion what we can do to solve this or what can
cause this, since it worked for several months and now if we start
ipsec in less than 3 minutes the system just gets so loaded that it
even can't shutdown cleanly, a kill -9 of the pluto works, but leave
the ipsec unusable, since we can't restart it anymore.
We don't think upgrading to openswan will solve this, since it worked
before, but as a desparate attempt we can try.
Thanks in advance if somebody has a suggestion.