TCP connect problem.

I have the following situation:
My server does a listen/accept pair for an incoming connection.
My client does a connect to the port on the server and in most cases a
TCP link is established.

However, every now and again the server will dissapear at about the same
time as the client is calling 'connect'. What I then see is:
* if I do a 'netstat -ap' on the client machine, I don't see the socket
the client made to get the connection. Presumably it's been cleaned up
* The connect statement never returns.

As you may guess, the socket doing the 'connect' is non-blocking. I
assumed that it would either fail if the other end wasn't listening, or
timeout or similar.

I'm wondering if it's the fact that the initial TCP handshake has begun,
and then the Server has torn down it's end that may be causing this?

Under what conditions will connect not return at all?

Thanks