Inconsistent Remote Desktop through VPN.

Remote Desktop works great on LAN, but when through VPN
RD is very inconsistent.
During the same VPN instance I may be able to connect,
disconnect, and then receive the following error message
when I try to reconnect, "The Client could not connect to
the remote computer. Remote connection might not be
enabled or the computer might be too busy to accept new
connections. It is also possible that network problems
are preventing your connection." This error message also
applies to the first attempt at connection then later the
server has no problems. It is very "hit and miss" and
not limited to anyone server.