FIN_WAIT2 not working

I have a strange reaction in FIN_WAIT2 in my embedded linux box. Once the
box enters FIN_WAIT2, an arriving tcp window update causes a RST to be sent
back. This causses an error on the peer. After verifying using
"netstat -a", I know the box is in FIN_WAIT2 for the correct time period.
I've also confirmed the value in the "tcp window update" variable as 60.
Has anyone else seen this and if so how do you correct the situation?

Thanks.

Gary R. Garrison <(E-Mail Removed)> wrote:
> I have a strange reaction in FIN_WAIT2 in my embedded linux box.

Running which linux kernel?

> Once the box enters FIN_WAIT2, an arriving tcp window update causes
> a RST to be sent back. This causses an error on the peer. After
> verifying using "netstat -a", I know the box is in FIN_WAIT2 for the
> correct time period.

Unless there is a platform-specific timeout being set, there is no one
"correct" time period for a connection to be in FIN_WAIT_2. Assuming
there is still a local socket reference to the endpoint it is still a
perfectly valid receive-only connection state.

> I've also confirmed the value in the "tcp window update" variable as
> 60.

?

> Has anyone else seen this and if so how do you correct the
> situation?

FIN_WAIT_2 means that side has sent a FINished segment and had it
ACKed by the remote. The implication is that all the data up to the
FIN segment has been ACKed as well.

Given that a FIN means that side will be sending no more data, there
really isn't any point in sending a window update to the side which
has sent you a FIN. Still, there also isn't much point in that side
getting bent out of shape by receipt of a window update. Are you
certain that the only things different in the segment carrying the
window update is the update of the window? Posting a trace of the
exchange, including the FIN and ACK of same might be helpful.

I suspect that unless the TCP RFC's have something concrete to say
about it you are in a grey area. The one side is not being
conservative in what it sends by sending a window update after
receiving a FIN, and the other is not being liberal in what it accepts
by having a window update trigger a RST.

rick jones
--
Process shall set you free from the need for rational thought.
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

I'm using an embedded Linux version 2.6.25-rc8.

The variable I meant to mention as being 60 was
"proc/sys/net/ipv4/tcp_fin_timeout". I understand what FIN_WAIT_2 means and
agree that one side is not being conservative in what it sends by sending a
window update after receiving a FIN, and the other is not being liberal in
what it accepts by having a window update trigger a RST.

All communications between the Linux machine and MSWindows machines (WinXP
pro, Vista and WinXP embedded) where the Linux machine transmits information
then calls Close() result in tcp window updates arriving from the Windows
machines after the Linux machine has sent it's FIN.

Communications between Windows machines also show tcp window updates
arriving from other Windows machines after we've sent them FIN.

The problem is the Linux machine sending the RST. Since this newsgroup
keeps rejecting everything I send with a ".pcap" file attached heres a topo
of the end of a capture :

the linux machine : 192.168.250.1
WinXP embedded machine 192.168.250.2

No. Time RT Source Destination
Protocol Info
304 0.178455 0.000246 192.168.250.1 192.168.250.2 TCP
52911 > 11000 [ACK] Seq=204430 Ack=1 Win=5840 Len=1448 TSV=736118
TSER=206788
305 0.178472 0.000017 192.168.250.2 192.168.250.1 TCP
11000 > 52911 [ACK] Seq=1 Ack=204430 Win=2755 Len=0 TSV=206788 TSER=736118
306 0.261309 0.082837 192.168.250.2 192.168.250.1 TCP
11000 > 52911 [ACK] Seq=1 Ack=205878 Win=14339 Len=0 TSV=206789 TSER=736118
307 0.261759 0.000450 192.168.250.1 192.168.250.2 TCP
52911 > 11000 [ACK] Seq=205878 Ack=1 Win=5840 Len=1448 TSV=736127
TSER=206789
308 0.261881 0.000122 192.168.250.1 192.168.250.2 TCP
52911 > 11000 [ACK] Seq=207326 Ack=1 Win=5840 Len=1448 TSV=736127
TSER=206789
309 0.262002 0.000121 192.168.250.1 192.168.250.2 TCP
52911 > 11000 [FIN, PSH, ACK] Seq=208774 Ack=1 Win=5840 Len=1256 TSV=736127
TSER=206789
310 0.262013 0.000011 192.168.250.2 192.168.250.1 TCP
11000 > 52911 [ACK] Seq=1 Ack=208774 Win=11443 Len=0 TSV=206789 TSER=736127
311 0.262102 0.000089 192.168.250.2 192.168.250.1 TCP
11000 > 52911 [ACK] Seq=1 Ack=210031 Win=10187 Len=0 TSV=206789 TSER=736127
312 0.283422 0.021320 192.168.250.2 192.168.250.1 TCP
[TCP Window Update] 11000 > 52911 [ACK] Seq=1 Ack=210031 Win=61383 Len=0
TSV=206789 TSER=736127
313 0.283635 0.000213 192.168.250.1 192.168.250.2 TCP
52911 > 11000 [RST] Seq=210031 Win=0 Len=0


Linux client connects WinXP server, sends a block of 29 bytes indicating the
Linux machine is going to send 2500 "ArticleInformation" objects. The
transfer is accomplished but at the end, after the FIN by the Linux client,
the WinXP machine sends a window update then the Linux client sends a RST.
Since windows handles RST in realtime and the windows application hasn't yet
"taken" all of the incoming data, an error is seen in "read()".

Simple situation mais solution unknown.

Gary R. Garrison <(E-Mail Removed)> wrote:
> I'm using an embedded Linux version 2.6.25-rc8.

> All communications between the Linux machine and MSWindows machines
> (WinXP pro, Vista and WinXP embedded) where the Linux machine
> transmits information then calls Close() result in tcp window
> updates arriving from the Windows machines after the Linux machine
> has sent it's FIN.

Does the behaviour change if in the app on the Linux side you call
shutdown(SHUT_WR) on the socket and then hang in a timed select/poll
waiting for the socket to become readable, meanwhile on the Windows
side, you make certain that the Windows side calls close() after the
read return of zero? That sequence is what netperf does to be "sure"
that the receiver has received all the data.

Perhaps the Linux stack will be more "forgiving" of the window update
if the connection end-point isn't orhpaned after a close().

> Communications between Windows machines also show tcp window updates
> arriving from other Windows machines after we've sent them FIN.

> The problem is the Linux machine sending the RST.

Well, or the Windows system sending a window update after a FIN.

> Since this newsgroup keeps rejecting everything I send with a
> ".pcap" file attached heres a topo of the end of a capture :

> the linux machine : 192.168.250.1
> WinXP embedded machine 192.168.250.2

I've taken the liberty of converting the IPs to shorter things and
done some other "tightening-up" of the putput.

> Time Src Dst Protocol Info
> 0.178455 LNX WIN [A] Seq=204430 Ack=1 Win=5840 Len=1448
> 0.178472 WIN LNX [A] Seq=1 Ack=204430 Win=2755 Len=0

Did the application do anything to put the Windows system into an
immediate ACK mode or is this very close to the beginning of the
connection?

> 0.261309 WIN LNX [A] Seq=1 Ack=205878 Win=14339 Len=0
> 0.261759 LNX WIN [A] Seq=205878 Ack=1 Win=5840 Len=1448
> 0.261881 LNX WIN [A] Seq=207326 Ack=1 Win=5840 Len=1448
> 0.262002 LNX WIN [F, P, A] Seq=208774 Ack=1 Win=5840 Len=1256
> 0.262013 WIN LNX [A] Seq=1 Ack=208774 Win=11443 Len=0
> 0.262102 WIN LNX [A] Seq=1 Ack=210031 Win=10187 Len=0
> 0.283422 WIN LNX[TCP Window Update] [A] Seq=1 Ack=210031 Win=61383 Len=0
> 0.283635 LNX WIN [R] Seq=210031 Win=0 Len=0


> Linux client connects WinXP server, sends a block of 29 bytes
> indicating the Linux machine is going to send 2500
> "ArticleInformation" objects. The transfer is accomplished but at
> the end, after the FIN by the Linux client, the WinXP machine sends
> a window update then the Linux client sends a RST. Since windows
> handles RST in realtime and the windows application hasn't yet
> "taken" all of the incoming data, an error is seen in "read()".

> Simple situation mais solution unknown.

If all else fails and you have the ability to change both sides of the
equation, you could have the Windows side to an application-level ack
of the data on which the Linux side waits before calling close() on
the socket. It goes one step further than the shutdown() idea above
by making sure that the Windows side has no more window updates to
send - they best be piggy-backed on that app-level ack.

rick jones
--
The computing industry isn't as much a game of "Follow The Leader" as
it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose."
- Rick Jones
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

I seem to have a working solution.


> Does the behaviour change if in the app on the Linux side you call
> shutdown(SHUT_WR) on the socket and then hang in a timed select/poll
> waiting for the socket to become readable, meanwhile on the Windows
> side, you make certain that the Windows side calls close() after the
> read return of zero?

I implemented a shutdown(SHUT_WR) solution yesterday that waits for a
response before closing. This appears to have fixed the problem.


> Did the application do anything to put the Windows system into an
> immediate ACK mode or is this very close to the beginning of the
> connection?

The captured packets shown were not at the beginning of a capture - Windows
seems to have entered immediate ACK mode at some point. The first packet in
the block I sent was the 304th packet in this particular socket exchange.


> If all else fails and you have the ability to change both sides of the
> equation, you could have the Windows side to an application-level ack
> of the data on which the Linux side waits before calling close() on
> the socket. It goes one step further than the shutdown() idea above
> by making sure that the Windows side has no more window updates to
> send - they best be piggy-backed on that app-level ack.

The problem actually involves multiple existing systems using multiple OSes.
A certain amount of existing code exists in a windows utility program that's
been distributed for years. I really don't want to have to modify and
redistribute that just because a new Linux platform doesn't handle "close()"
correctly.

In any case, as mentioned at the beginning of this - I have a working
solution. Your suggestion would have led me to it if I hadn't found it
about the same time you did Rick. The reference I had found on internet
implemented a shutdown as you suggest, followed by a loop using select for a
certain timeout period to watch input. I had trouble with that loop and
instead found a way using the tcp_info structure. Heres a look at the code
:

int Close_Socket(int socketd)
{
int vl_Status;
int InfoLen;
volatile struct tcp_info MyInfo;

if (0 != shutdown(socketd, SHUT_WR))
{ // can't shutdown write so shutdown all
if (0 != close(socketd))
{
return (INVALID_SOCKET);
}
return (SUCCESS);
}

// write has been shutdown so FIN has been sent
InfoLen = sizeof(MyInfo);
do
{
vl_Status = getsockopt( socketd, SOL_TCP, TCP_INFO, (void
*)&MyInfo, &InfoLen );
}while ((0 == vl_Status) &&
((TCP_FIN_WAIT1 == MyInfo.tcpi_state) ||
(TCP_FIN_WAIT2 == MyInfo.tcpi_state)) );

if (0 != close(socketd))
{
return (INVALID_SOCKET);
}

return (SUCCESS);
}

Gary R. Garrison <(E-Mail Removed)> wrote:

> > If all else fails and you have the ability to change both sides of the
> > equation, you could have the Windows side to an application-level ack
> > of the data on which the Linux side waits before calling close() on
> > the socket. It goes one step further than the shutdown() idea above
> > by making sure that the Windows side has no more window updates to
> > send - they best be piggy-backed on that app-level ack.

> The problem actually involves multiple existing systems using
> multiple OSes. A certain amount of existing code exists in a
> windows utility program that's been distributed for years. I really
> don't want to have to modify and redistribute that just because a
> new Linux platform doesn't handle "close()" correctly.

I'd probably assign blame about 60% to Linux and 40% to Windows but
then that's me. Do feel free to mention this in the "netdev" mailing
list - just be prepared for the inevitable "have you tried the latest
kernel" questions, and make sure you present the bug plainly less you
trigger a Windows-induced immune response on the list

> In any case, as mentioned at the beginning of this - I have a working
> solution. Your suggestion would have led me to it if I hadn't found it
> about the same time you did Rick. The reference I had found on internet
> implemented a shutdown as you suggest, followed by a loop using select for a
> certain timeout period to watch input. I had trouble with that loop and
> instead found a way using the tcp_info structure. Heres a look at the code
> :

> int Close_Socket(int socketd)
> {
> int vl_Status;
> int InfoLen;
> volatile struct tcp_info MyInfo;

> if (0 != shutdown(socketd, SHUT_WR))
> { // can't shutdown write so shutdown all
> if (0 != close(socketd))
> {
> return (INVALID_SOCKET);
> }
> return (SUCCESS);
> }

> // write has been shutdown so FIN has been sent
> InfoLen = sizeof(MyInfo);
> do
> {
> vl_Status = getsockopt( socketd, SOL_TCP, TCP_INFO, (void
> *)&MyInfo, &InfoLen );
> }while ((0 == vl_Status) &&
> ((TCP_FIN_WAIT1 == MyInfo.tcpi_state) ||
> (TCP_FIN_WAIT2 == MyInfo.tcpi_state)) );

You really mean to spin waiting for the state change? It could be
spinning there for quite some time if either of the FINs have to
retransmitted...

I _guess_ that is "ok" if this is the only thing really running on the
system but my spidy sense is suggesting that some day someone may copy
that code into an environment where spinning isn't really such a good
idea.

rick jones
--
Process shall set you free from the need for rational thought.
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...