Error 768 - failure to encrypt

I'm setting up a Windows 2003 Server as a VPN server for L2TP/IPSec. If a
take some random XP PC and hook it to the server with a crossover cable I
can get an L2TP tunnel going just fine. But if I take my laptop (also XP)
and do the same thing with exactly the same settings I get:

Error 768: The connection attempt failed because of failure to encrypt data.

I'm using a preshared key instead of a certificate and I know that I don't
have a typo (it's pretty hard to mistype "1").

Now for the kicker - I have Microsoft Virtual PC installed on said laptop.
From both 98 and XP virtual sessions I CAN connect to the server, so it's
not anything intrinsic to the NIC or the cabling. And the laptop itself can
connect just fine if I revert to PPTP/MPPE.

I don't want to start rolling this out if I'm going to have seemingly random
problems with some machines. Anyone have any thoughts/suggestions?




Todd hobdey

A new twist that undoubtedly related - I'm trying to follow a Microsoft
troubleshooting article and when I launch the IP Security Monitor snap-in, I
get "The IPSec server is unavailable or incompatible with the IPSec
monitor". The only information on that message is that I'm running it on
Windows 2000, which I'm not.


"Todd hobdey" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> I'm setting up a Windows 2003 Server as a VPN server for L2TP/IPSec. If a
> take some random XP PC and hook it to the server with a crossover cable I
> can get an L2TP tunnel going just fine. But if I take my laptop (also XP)
> and do the same thing with exactly the same settings I get:
>
> Error 768: The connection attempt failed because of failure to encrypt
data.
>
> I'm using a preshared key instead of a certificate and I know that I don't
> have a typo (it's pretty hard to mistype "1").
>
> Now for the kicker - I have Microsoft Virtual PC installed on said laptop.
> From both 98 and XP virtual sessions I CAN connect to the server, so it's
> not anything intrinsic to the NIC or the cabling. And the laptop itself
can
> connect just fine if I revert to PPTP/MPPE.
>
> I don't want to start rolling this out if I'm going to have seemingly
random
> problems with some machines. Anyone have any thoughts/suggestions?
>
>

Are you sure the IPSec Services is running on your machine?

--
Kadir

(E-Mail Removed) [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

"Todd hobdey" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> A new twist that undoubtedly related - I'm trying to follow a Microsoft
> troubleshooting article and when I launch the IP Security Monitor snap-in,
I
> get "The IPSec server is unavailable or incompatible with the IPSec
> monitor". The only information on that message is that I'm running it on
> Windows 2000, which I'm not.
>
>
> "Todd hobdey" <(E-Mail Removed)> wrote in message
> news:(E-Mail Removed)...
> > I'm setting up a Windows 2003 Server as a VPN server for L2TP/IPSec. If
a
> > take some random XP PC and hook it to the server with a crossover cable
I
> > can get an L2TP tunnel going just fine. But if I take my laptop (also
XP)
> > and do the same thing with exactly the same settings I get:
> >
> > Error 768: The connection attempt failed because of failure to encrypt
> data.
> >
> > I'm using a preshared key instead of a certificate and I know that I
don't
> > have a typo (it's pretty hard to mistype "1").
> >
> > Now for the kicker - I have Microsoft Virtual PC installed on said
laptop.
> > From both 98 and XP virtual sessions I CAN connect to the server, so
it's
> > not anything intrinsic to the NIC or the cabling. And the laptop itself
> can
> > connect just fine if I revert to PPTP/MPPE.
> >
> > I don't want to start rolling this out if I'm going to have seemingly
> random
> > problems with some machines. Anyone have any thoughts/suggestions?
> >
> >
>
>

Yes it is. I ran "netdiag /test:ipsec /v /debug" and everything came back
from that okay, but I still can't launch the MMC snap-in.


"Kadirvel C Vanniarajan [MSFT]" <(E-Mail Removed)> wrote in
message news:%(E-Mail Removed)...
> Are you sure the IPSec Services is running on your machine?
>
> --
> Kadir
>
> (E-Mail Removed) [MSFT]
> This posting is provided "AS IS" with no warranties, and confers no
rights.
>
> "Todd hobdey" <(E-Mail Removed)> wrote in message
> news:(E-Mail Removed)...
> > A new twist that undoubtedly related - I'm trying to follow a Microsoft
> > troubleshooting article and when I launch the IP Security Monitor
snap-in,
> I
> > get "The IPSec server is unavailable or incompatible with the IPSec
> > monitor". The only information on that message is that I'm running it
on
> > Windows 2000, which I'm not.
> >
> >
> > "Todd hobdey" <(E-Mail Removed)> wrote in message
> > news:(E-Mail Removed)...
> > > I'm setting up a Windows 2003 Server as a VPN server for L2TP/IPSec.
If
> a
> > > take some random XP PC and hook it to the server with a crossover
cable
> I
> > > can get an L2TP tunnel going just fine. But if I take my laptop (also
> XP)
> > > and do the same thing with exactly the same settings I get:
> > >
> > > Error 768: The connection attempt failed because of failure to encrypt
> > data.
> > >
> > > I'm using a preshared key instead of a certificate and I know that I
> don't
> > > have a typo (it's pretty hard to mistype "1").
> > >
> > > Now for the kicker - I have Microsoft Virtual PC installed on said
> laptop.
> > > From both 98 and XP virtual sessions I CAN connect to the server, so
> it's
> > > not anything intrinsic to the NIC or the cabling. And the laptop
itself
> > can
> > > connect just fine if I revert to PPTP/MPPE.
> > >
> > > I don't want to start rolling this out if I'm going to have seemingly
> > random
> > > problems with some machines. Anyone have any thoughts/suggestions?
> > >
> > >
> >
> >
>
>

You might want to check the following details:
1. Check if there is a binary mismatch. Ipsecnp.dll, Oakley.dll,
winipsec.dll and ipsecsvc.dll must be from the same build. The MMC error
suggested that there may be mismatch between winipsec.dll and ipsecsvc.dll.

2. Is policy agent actually up and running? (sc query policyagent).
The netdiag test on W2k3 will not indicate if ipsec modules are initiliazed
and running.

If these are not going to help, enable the RAS tracing (using netsh ras set
trace * enable) and also enable oakley logging (details in
http://support.microsoft.com/default...b;en-us;257225) get the logs
where the IPSec Monitor is failing. The ras logs will be found under
%windir%\tracing and the oakley logs can be found under %windir%\debug.


--
Kadir

(E-Mail Removed) [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

"Todd hobdey" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> Yes it is. I ran "netdiag /test:ipsec /v /debug" and everything came back
> from that okay, but I still can't launch the MMC snap-in.
>
>
> "Kadirvel C Vanniarajan [MSFT]" <(E-Mail Removed)> wrote in
> message news:%(E-Mail Removed)...
> > Are you sure the IPSec Services is running on your machine?
> >
> > --
> > Kadir
> >
> > (E-Mail Removed) [MSFT]
> > This posting is provided "AS IS" with no warranties, and confers no
> rights.
> >
> > "Todd hobdey" <(E-Mail Removed)> wrote in message
> > news:(E-Mail Removed)...
> > > A new twist that undoubtedly related - I'm trying to follow a
Microsoft
> > > troubleshooting article and when I launch the IP Security Monitor
> snap-in,
> > I
> > > get "The IPSec server is unavailable or incompatible with the IPSec
> > > monitor". The only information on that message is that I'm running it
> on
> > > Windows 2000, which I'm not.
> > >
> > >
> > > "Todd hobdey" <(E-Mail Removed)> wrote in message
> > > news:(E-Mail Removed)...
> > > > I'm setting up a Windows 2003 Server as a VPN server for L2TP/IPSec.
> If
> > a
> > > > take some random XP PC and hook it to the server with a crossover
> cable
> > I
> > > > can get an L2TP tunnel going just fine. But if I take my laptop
(also
> > XP)
> > > > and do the same thing with exactly the same settings I get:
> > > >
> > > > Error 768: The connection attempt failed because of failure to
encrypt
> > > data.
> > > >
> > > > I'm using a preshared key instead of a certificate and I know that I
> > don't
> > > > have a typo (it's pretty hard to mistype "1").
> > > >
> > > > Now for the kicker - I have Microsoft Virtual PC installed on said
> > laptop.
> > > > From both 98 and XP virtual sessions I CAN connect to the server, so
> > it's
> > > > not anything intrinsic to the NIC or the cabling. And the laptop
> itself
> > > can
> > > > connect just fine if I revert to PPTP/MPPE.
> > > >
> > > > I don't want to start rolling this out if I'm going to have
seemingly
> > > random
> > > > problems with some machines. Anyone have any thoughts/suggestions?
> > > >
> > > >
> > >
> > >
> >
> >
>
>

When I posted this, the file versions were:
winipsec.dll 5.1.2600.0
ipsecsnp.dll 5.1.2600.0
oakley.dll 5.1.2600.1106

"sc query policyagent" shows the service running.

I came across article 818043 relating to NAT-T and applied it thinking that
might have some bearing on the issue. It did, in that it changed the
problem, but I still have a problem.

After the update, the file versions on all of the files are 5.1.2600.1240.
When I try to launch the IP Security snap-in, I now get "The binding handle
is invalid", instead of the original error.

The rasapi32.log shows:

[1260] 09:15:42: RasEnumConnectionsW
[1260] 09:15:57: RasEnumConnectionsW

The tapisrv.log shows

[1332] 09:15:39:380: [INFO ] Service control code=4

And the only entry in the Oakley log is from startup, and that is:

1-19: 07:00:06:20:1f4 Obtain WSARecvMsg pointer failed 10045
1-19: 07:00:06:20:1f4 Obtain WSARecvMsg pointer failed 0
1-19: 07:00:06:20:1f4 Startup failed. Shutting down
1-19: 07:00:06:20:1f4 Initialization failed. Exiting
1-19: 07:00:06:20:1f4 Wait is done
1-19: 07:00:06:20:1f4 Before send_deletes
1-19: 07:00:06:20:1f4 AFter send_deletes
1-19: 07:00:06:20:1f4 Begin Wait. isadb_clean_socket 0
1-19: 07:00:06:20:1f4 End Wait. isadb_clean_socket
1-19: 07:00:06:20:1f4 Begin Wait. isadb_kill_old 0
1-19: 07:00:06:20:1f4 End Wait. isadb_kill_old
1-19: 07:00:06:20:1f4 Begin Wait. ActiveRpcCalls 0
1-19: 07:00:06:20:1f4 End Wait. ActiveRpcCalls




"Kadirvel C Vanniarajan [MSFT]" <(E-Mail Removed)> wrote in
message news:(E-Mail Removed)...
> You might want to check the following details:
> 1. Check if there is a binary mismatch. Ipsecnp.dll, Oakley.dll,
> winipsec.dll and ipsecsvc.dll must be from the same build. The MMC error
> suggested that there may be mismatch between winipsec.dll and
ipsecsvc.dll.
>
> 2. Is policy agent actually up and running? (sc query policyagent).
> The netdiag test on W2k3 will not indicate if ipsec modules are
initiliazed
> and running.
>
> If these are not going to help, enable the RAS tracing (using netsh ras
set
> trace * enable) and also enable oakley logging (details in
> http://support.microsoft.com/default...b;en-us;257225) get the
logs
> where the IPSec Monitor is failing. The ras logs will be found under
> %windir%\tracing and the oakley logs can be found under %windir%\debug.
>
>
> --
> Kadir
>
> (E-Mail Removed) [MSFT]
> This posting is provided "AS IS" with no warranties, and confers no
rights.
>
> "Todd hobdey" <(E-Mail Removed)> wrote in message
> news:(E-Mail Removed)...
> > Yes it is. I ran "netdiag /test:ipsec /v /debug" and everything came
back
> > from that okay, but I still can't launch the MMC snap-in.
> >
> >
> > "Kadirvel C Vanniarajan [MSFT]" <(E-Mail Removed)> wrote in
> > message news:%(E-Mail Removed)...
> > > Are you sure the IPSec Services is running on your machine?
> > >
> > > --
> > > Kadir
> > >
> > > (E-Mail Removed) [MSFT]
> > > This posting is provided "AS IS" with no warranties, and confers no
> > rights.
> > >
> > > "Todd hobdey" <(E-Mail Removed)> wrote in message
> > > news:(E-Mail Removed)...
> > > > A new twist that undoubtedly related - I'm trying to follow a
> Microsoft
> > > > troubleshooting article and when I launch the IP Security Monitor
> > snap-in,
> > > I
> > > > get "The IPSec server is unavailable or incompatible with the IPSec
> > > > monitor". The only information on that message is that I'm running
it
> > on
> > > > Windows 2000, which I'm not.
> > > >
> > > >
> > > > "Todd hobdey" <(E-Mail Removed)> wrote in message
> > > > news:(E-Mail Removed)...
> > > > > I'm setting up a Windows 2003 Server as a VPN server for
L2TP/IPSec.
> > If
> > > a
> > > > > take some random XP PC and hook it to the server with a crossover
> > cable
> > > I
> > > > > can get an L2TP tunnel going just fine. But if I take my laptop
> (also
> > > XP)
> > > > > and do the same thing with exactly the same settings I get:
> > > > >
> > > > > Error 768: The connection attempt failed because of failure to
> encrypt
> > > > data.
> > > > >
> > > > > I'm using a preshared key instead of a certificate and I know that
I
> > > don't
> > > > > have a typo (it's pretty hard to mistype "1").
> > > > >
> > > > > Now for the kicker - I have Microsoft Virtual PC installed on said
> > > laptop.
> > > > > From both 98 and XP virtual sessions I CAN connect to the server,
so
> > > it's
> > > > > not anything intrinsic to the NIC or the cabling. And the laptop
> > itself
> > > > can
> > > > > connect just fine if I revert to PPTP/MPPE.
> > > > >
> > > > > I don't want to start rolling this out if I'm going to have
> seemingly
> > > > random
> > > > > problems with some machines. Anyone have any
thoughts/suggestions?
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

Todd,

Can you check if there are any event logs indicating a file version change?
Also, from the KB article, can you check if the file versions that are
present on your machine matches that of those present in the QFE package?

There still seems to be an issue with the fileversions.

--
Kadir

(E-Mail Removed) [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

"Todd hobdey" <(E-Mail Removed)> wrote in message
news:%(E-Mail Removed)...
> When I posted this, the file versions were:
> winipsec.dll 5.1.2600.0
> ipsecsnp.dll 5.1.2600.0
> oakley.dll 5.1.2600.1106
>
> "sc query policyagent" shows the service running.
>
> I came across article 818043 relating to NAT-T and applied it thinking
that
> might have some bearing on the issue. It did, in that it changed the
> problem, but I still have a problem.
>
> After the update, the file versions on all of the files are 5.1.2600.1240.
> When I try to launch the IP Security snap-in, I now get "The binding
handle
> is invalid", instead of the original error.
>
> The rasapi32.log shows:
>
> [1260] 09:15:42: RasEnumConnectionsW
> [1260] 09:15:57: RasEnumConnectionsW
>
> The tapisrv.log shows
>
> [1332] 09:15:39:380: [INFO ] Service control code=4
>
> And the only entry in the Oakley log is from startup, and that is:
>
> 1-19: 07:00:06:20:1f4 Obtain WSARecvMsg pointer failed 10045
> 1-19: 07:00:06:20:1f4 Obtain WSARecvMsg pointer failed 0
> 1-19: 07:00:06:20:1f4 Startup failed. Shutting down
> 1-19: 07:00:06:20:1f4 Initialization failed. Exiting
> 1-19: 07:00:06:20:1f4 Wait is done
> 1-19: 07:00:06:20:1f4 Before send_deletes
> 1-19: 07:00:06:20:1f4 AFter send_deletes
> 1-19: 07:00:06:20:1f4 Begin Wait. isadb_clean_socket 0
> 1-19: 07:00:06:20:1f4 End Wait. isadb_clean_socket
> 1-19: 07:00:06:20:1f4 Begin Wait. isadb_kill_old 0
> 1-19: 07:00:06:20:1f4 End Wait. isadb_kill_old
> 1-19: 07:00:06:20:1f4 Begin Wait. ActiveRpcCalls 0
> 1-19: 07:00:06:20:1f4 End Wait. ActiveRpcCalls
>
>
>
>
> "Kadirvel C Vanniarajan [MSFT]" <(E-Mail Removed)> wrote in
> message news:(E-Mail Removed)...
> > You might want to check the following details:
> > 1. Check if there is a binary mismatch. Ipsecnp.dll, Oakley.dll,
> > winipsec.dll and ipsecsvc.dll must be from the same build. The MMC
error
> > suggested that there may be mismatch between winipsec.dll and
> ipsecsvc.dll.
> >
> > 2. Is policy agent actually up and running? (sc query
policyagent).
> > The netdiag test on W2k3 will not indicate if ipsec modules are
> initiliazed
> > and running.
> >
> > If these are not going to help, enable the RAS tracing (using netsh ras
> set
> > trace * enable) and also enable oakley logging (details in
> > http://support.microsoft.com/default...b;en-us;257225) get the
> logs
> > where the IPSec Monitor is failing. The ras logs will be found under
> > %windir%\tracing and the oakley logs can be found under %windir%\debug.
> >
> >
> > --
> > Kadir
> >
> > (E-Mail Removed) [MSFT]
> > This posting is provided "AS IS" with no warranties, and confers no
> rights.
> >
> > "Todd hobdey" <(E-Mail Removed)> wrote in message
> > news:(E-Mail Removed)...
> > > Yes it is. I ran "netdiag /test:ipsec /v /debug" and everything came
> back
> > > from that okay, but I still can't launch the MMC snap-in.
> > >
> > >
> > > "Kadirvel C Vanniarajan [MSFT]" <(E-Mail Removed)> wrote
in
> > > message news:%(E-Mail Removed)...
> > > > Are you sure the IPSec Services is running on your machine?
> > > >
> > > > --
> > > > Kadir
> > > >
> > > > (E-Mail Removed) [MSFT]
> > > > This posting is provided "AS IS" with no warranties, and confers no
> > > rights.
> > > >
> > > > "Todd hobdey" <(E-Mail Removed)> wrote in message
> > > > news:(E-Mail Removed)...
> > > > > A new twist that undoubtedly related - I'm trying to follow a
> > Microsoft
> > > > > troubleshooting article and when I launch the IP Security Monitor
> > > snap-in,
> > > > I
> > > > > get "The IPSec server is unavailable or incompatible with the
IPSec
> > > > > monitor". The only information on that message is that I'm
running
> it
> > > on
> > > > > Windows 2000, which I'm not.
> > > > >
> > > > >
> > > > > "Todd hobdey" <(E-Mail Removed)> wrote in message
> > > > > news:(E-Mail Removed)...
> > > > > > I'm setting up a Windows 2003 Server as a VPN server for
> L2TP/IPSec.
> > > If
> > > > a
> > > > > > take some random XP PC and hook it to the server with a
crossover
> > > cable
> > > > I
> > > > > > can get an L2TP tunnel going just fine. But if I take my laptop
> > (also
> > > > XP)
> > > > > > and do the same thing with exactly the same settings I get:
> > > > > >
> > > > > > Error 768: The connection attempt failed because of failure to
> > encrypt
> > > > > data.
> > > > > >
> > > > > > I'm using a preshared key instead of a certificate and I know
that
> I
> > > > don't
> > > > > > have a typo (it's pretty hard to mistype "1").
> > > > > >
> > > > > > Now for the kicker - I have Microsoft Virtual PC installed on
said
> > > > laptop.
> > > > > > From both 98 and XP virtual sessions I CAN connect to the
server,
> so
> > > > it's
> > > > > > not anything intrinsic to the NIC or the cabling. And the
laptop
> > > itself
> > > > > can
> > > > > > connect just fine if I revert to PPTP/MPPE.
> > > > > >
> > > > > > I don't want to start rolling this out if I'm going to have
> > seemingly
> > > > > random
> > > > > > problems with some machines. Anyone have any
> thoughts/suggestions?
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>