Clarifiication needed on tcp_keepalive messages ?

Hi All,

Need some clarifications on TCP_keepalive . We are facing some issues
on our Prod servers related to TCP functionality .

The issue is like this for us.

We have some machines at one end sending data in real time to another
group of machines on the other end. Now due to some hardware issues on
the other end , some of the machines becomes unresponsive/crashes.
The client system which pumps data never came to know that the server
went unresponsive . The connection remains in ESTABLISHED state and
the client always tries to send data thinking that the connection is
alive because of which we are seeing backlog on client sides.

Our understanding is like this on how TCP will handle the situation.


Q 1) Since the server went down , the client will try to the
retransmit the data until it times out. What is the
behavior of TCP after the timeout? Need clarification on the following
things.
a) Will the kernel will close the established connection after
the timeout . Looks like no in our case as we
still see the connection still in ESTABLISHED state after
around more than 2 hours.
b) Are there any kernel parameters which decides that when the
client will timeout after retransmission
fails. What is the behavior of TCP after the client
retransmission timeouts.


Q 2 ) Tried with tcp_keepalive also . Default keepalive time comes
to be around 2 hrs 2 minutes , i think? . My understanding regarding
this is that the client will send some TCP probes after the keepalive
time interval and if it cannot reach the server , then the established
connection in the client side will be closed by the kernel . But I can
see that the connection still remains in established after the
tcp_keepalive
time . We waited for around 3 hrs but the connection still remains in
established state. Tried reducing the keepalive time to be around 10
minutes , but the connection remains in ESTABLISHED state in client
side after the 10 minute interval .

Where I went wrong or my understanding .Please clarify my doubts
raised above . What should we do
to resolve the problem we are seeing above . Any help will be highly
appreciated as we are going through a hard time to resolve the issue .

Thanks in Advance

On Wed, 2011-02-23, bzaman wrote:
> Hi All,
>
> Need some clarifications on TCP_keepalive . We are facing some issues
> on our Prod servers related to TCP functionality .
>
> The issue is like this for us.
>
> We have some machines at one end sending data in real time to another

"Real time" ?

> group of machines on the other end. Now due to some hardware issues on
> the other end , some of the machines becomes unresponsive/crashes.

To simplify and disambiguate, you mean one client and one server

data
C ------> S

And "unresponsive/crashes" must include "does not come up again" --
otherwise the clients would get told as soon as the server's IP stack
started working after bootup.

> The client system which pumps data never came to know that the server
> went unresponsive . The connection remains in ESTABLISHED state and
> the client always tries to send data thinking that the connection is
> alive because of which we are seeing backlog on client sides.
>
> Our understanding is like this on how TCP will handle the situation.
>
>
> Q 1) Since the server went down , the client will try to the
> retransmit the data until it times out. What is the
> behavior of TCP after the timeout? Need clarification on the following
> things.
> a) Will the kernel will close the established connection after
> the timeout . Looks like no in our case as we
> still see the connection still in ESTABLISHED state after
> around more than 2 hours.
> b) Are there any kernel parameters which decides that when the
> client will timeout after retransmission
> fails. What is the behavior of TCP after the client
> retransmission timeouts.

I can't answer that part without reading the books. I'm not sure there
*is* a "reasonable" timeout, and I doubt that TCP keepalive is the
solution. If you want to know that data from C reached S (the server
/process/, not just the IP stack), you have to make S send an
application-level ACK.

If your protocol doesn't have such ACKs, the problems are not
surprising.

> Q 2 ) Tried with tcp_keepalive also . Default keepalive time comes
> to be around 2 hrs 2 minutes , i think? . My understanding regarding
> this is that the client will send some TCP probes after the keepalive
> time interval and if it cannot reach the server , then the established
> connection in the client side will be closed by the kernel . But I can
> see that the connection still remains in established after the
> tcp_keepalive
> time . We waited for around 3 hrs but the connection still remains in
> established state. Tried reducing the keepalive time to be around 10
> minutes , but the connection remains in ESTABLISHED state in client
> side after the 10 minute interval .

Did you confirm (by snooping) that you actually had managed to enable
the keepalives?

> Where I went wrong or my understanding .Please clarify my doubts
> raised above . What should we do
> to resolve the problem we are seeing above . Any help will be highly
> appreciated as we are going through a hard time to resolve the issue .
>
> Thanks in Advance

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

On Feb 23, 4:40*pm, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
> On Wed, 2011-02-23, bzaman wrote:
> > Hi All,
>
> > Need some clarifications on TCP_keepalive . We are facing some issues
> > on our Prod servers related to TCP functionality .
>
> > The issue is like this for us.
>
> > We have some machines at one end sending data in real time to another
>
> "Real time" ?
>
> > group of machines on the other end. Now due to some hardware issues on
> > the other end , some of the machines becomes unresponsive/crashes.
>
> To simplify and disambiguate, you mean one client and one server
>
> * * * data
> * *C ------> S
>
> And "unresponsive/crashes" must include "does not come up again" --
> otherwise the clients would get told as soon as the server's IP stack
> started working after bootup.
>
>
>
> > The client system which pumps data never came to know that the server
> > went unresponsive . The connection remains in ESTABLISHED state and
> > the client always tries to send data thinking that the connection is
> > alive because of which we are seeing backlog on client sides.
>
> > Our understanding is like this on how TCP will handle the situation.
>
> > Q 1) Since *the server went down , the client will try to the
> > retransmit the data until it times out. What is the
> > behavior of TCP after the timeout? Need clarification on the following
> > things.
> > * * * * a) Will the kernel will close the established connection after
> > the timeout . Looks like no in our case as we
> > * * * * * * still see the connection still in ESTABLISHED state after
> > around more than 2 hours.
> > * * * * b) Are there any kernel parameters which decides that when the
> > client will timeout after retransmission
> > * * * * * * fails. What is the behavior of TCP after the client
> > retransmission timeouts.
>
> I can't answer that part without reading the books. I'm not sure there
> *is* a "reasonable" timeout, and I doubt that TCP keepalive is the
> solution. *If you want to know that data from C reached S (the server
> /process/, not just the IP stack), you have to make S send an
> application-level ACK.
>
> If your protocol doesn't have such ACKs, the problems are not
> surprising.
>
> > Q 2 ) Tried with *tcp_keepalive also . Default keepalive time *comes
> > to be around 2 hrs 2 minutes , i think? . * My understanding regarding
> > this is that the client will send some TCP probes after the keepalive
> > time interval and if it cannot reach the server , then the established
> > connection in the client side will be closed by the kernel . But I can
> > see that the connection still remains in established after the
> > tcp_keepalive
> > time . We waited for around 3 hrs but the connection still remains in
> > established state. Tried reducing the keepalive time to be around 10
> > minutes , but the connection remains in ESTABLISHED state in client
> > side *after the 10 minute interval .
>
> Did you confirm (by snooping) that you actually had managed to enable
> the keepalives?
>

I tried that earlier and also tried now once again to confirm whether
any tcp keepalive messages are generated.

I tuned the value of keepalive by reducing it to 600

The settings after the tuning looks like this

$ cat /proc/sys/net/ipv4/tcp_keepalive_time
600
$ cat /proc/sys/net/ipv4/tcp_keepalive_intvl
75
$ cat /proc/sys/net/ipv4/tcp_keepalive_probes
9

I noticed that after 25 minutes , no tcp keepalive messages are
generated by the client. The filter with which I was trying to sniff
the packet was
something like this..

sudo tcpdump -i eth0 host <server-host-name>

If my understanding is not wrong , then the keepalive message will be
simply a tcp syn packet so should have been sniffed by the filter ,
but I could not see
any such packets , so probably the tcp_keepalive message was not sent
by the client.

Am i on the right track . Please give your views.

If the client is actively retransmitting, it isn't "idle" and so I
would not expect TCP keepalives to be sent. Further, if the client is
indeed actively retransmitting (ie has unacked data) then I would
expect the TCP connection to time out well before the default
keepalive setting.

rick jones
--
Wisdom Teeth are impacted, people are affected by the effects of events.
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

On Wed, 2011-02-23, Rick Jones wrote:
> If the client is actively retransmitting, it isn't "idle" and so I
> would not expect TCP keepalives to be sent. Further, if the client is
> indeed actively retransmitting (ie has unacked data) then I would
> expect the TCP connection to time out well before the default
> keepalive setting.

But how long is that timeout, typically? Way too long for the
original poster, right?

To the original poster: it's easy to experiment with this, using e.g.
netcat:

% nc -l 6969
....

and in another terminal:
% od huge_file | strace nc localhost 6969

Then quickly SIGSTOP (^Z) the first nc, set up tcpdump to monitor tcp
port 6969, and go do something else. Mine has been sitting over 20
minutes by now, trying to retransmit every 2 minutes.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

Jorgen Grahn <grahn+(E-Mail Removed)> wrote:
> On Wed, 2011-02-23, Rick Jones wrote:
> > If the client is actively retransmitting, it isn't "idle" and so I
> > would not expect TCP keepalives to be sent. Further, if the client is
> > indeed actively retransmitting (ie has unacked data) then I would
> > expect the TCP connection to time out well before the default
> > keepalive setting.

> But how long is that timeout, typically?

About as long as the typical piece of string My guess is it depends
on:

net.ipv4.tcp_retries2 = 15

and the RTT between the two systems, seasoned slightly with the
minimum RTO. If the minimum RTO is say 0.5 seconds, and there will be
15 retries, with a doubling each time that could become quite long. i
forget if there is an RTO cap in the linux stack

> Way too long for the original poster, right?

If so, the thing to do there is have application level responses and
run a timer up in the application layer.

> To the original poster: it's easy to experiment with this, using e.g.
> netcat:

> % nc -l 6969
> ...

> and in another terminal:
> % od huge_file | strace nc localhost 6969

> Then quickly SIGSTOP (^Z) the first nc, set up tcpdump to monitor
> tcp port 6969, and go do something else. Mine has been sitting over
> 20 minutes by now, trying to retransmit every 2 minutes.

If you simply SIGSTOP the receiver, all that will do is cause the
receive window to fill and then you are talking about persist not
retransmission. You need to preclude TCP segments reaching the
receiver, or the receiver's ACKs reaching the sender.

rick jones

--
I don't interest myself in "why." I think more often in terms of
"when," sometimes "where;" always "how much." - Joubert
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

On Wed, 2011-02-23, Rick Jones wrote:
> Jorgen Grahn <grahn+(E-Mail Removed)> wrote:
>> On Wed, 2011-02-23, Rick Jones wrote:
>> > If the client is actively retransmitting, it isn't "idle" and so I
>> > would not expect TCP keepalives to be sent. Further, if the client is
>> > indeed actively retransmitting (ie has unacked data) then I would
>> > expect the TCP connection to time out well before the default
>> > keepalive setting.
>
>> But how long is that timeout, typically?
>
> About as long as the typical piece of string

The string analogy is usually applied to questions with no meaning or
so many input parameters that the answer can be anything -- surely not
the case here?

> My guess is it depends
> on:
>
> net.ipv4.tcp_retries2 = 15
>
> and the RTT between the two systems, seasoned slightly with the
> minimum RTO. If the minimum RTO is say 0.5 seconds, and there will be
> 15 retries, with a doubling each time that could become quite long. i
> forget if there is an RTO cap in the linux stack
>
>> Way too long for the original poster, right?
>
> If so, the thing to do there is have application level responses and
> run a timer up in the application layer.

I suspect that is what the poster needs to be told, yes. Or at least,
that there is no quick fix which doesn't involve modifying the client.

>> To the original poster: it's easy to experiment with this, using e.g.
>> netcat:
>
>> % nc -l 6969
>> ...
>
>> and in another terminal:
>> % od huge_file | strace nc localhost 6969
>
>> Then quickly SIGSTOP (^Z) the first nc, set up tcpdump to monitor
>> tcp port 6969, and go do something else. Mine has been sitting over
>> 20 minutes by now, trying to retransmit every 2 minutes.
>
> If you simply SIGSTOP the receiver, all that will do is cause the
> receive window to fill and then you are talking about persist not
> retransmission. You need to preclude TCP segments reaching the
> receiver, or the receiver's ACKs reaching the sender.

Oh, that's true. Sorry -- I didn't see the distinction.

But then it's persisting rather than retransmission the poster is
talking about. If I understand him correctly, the server *process*
locks up, but the TCP stacks work fine and so does the path between
them. That's why I chose SIGSTOP (and I also didn't want to unplug my
machine from the network ;-)

BTW, the example from my earlier posting is still running, hours
later. The client TCP is dutifully checking the window every 120
seconds.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

Jorgen Grahn <grahn+(E-Mail Removed)> wrote:
> > If you simply SIGSTOP the receiver, all that will do is cause the
> > receive window to fill and then you are talking about persist not
> > retransmission. You need to preclude TCP segments reaching the
> > receiver, or the receiver's ACKs reaching the sender.

> Oh, that's true. Sorry -- I didn't see the distinction.

> But then it's persisting rather than retransmission the poster is
> talking about. If I understand him correctly, the server *process*
> locks up, but the TCP stacks work fine and so does the path between
> them. That's why I chose SIGSTOP (and I also didn't want to unplug my
> machine from the network ;-)

Ah, I got something else from reading "the server went down" (or words
to that effect). I interpreted that as meaning the system went down.

> BTW, the example from my earlier posting is still running, hours
> later. The client TCP is dutifully checking the window every 120
> seconds.

As I would expect it to - the connection is still there, TCP still
responds. There would have to be a decision made in the stack that
the connection could be toasted by the window remaining closed for
"too long" - it would be, IIRC, something beyond the TCP specs, not
unlike, I suppose the FIN_WAIT_2 timers people insisted go into stacks
at the dawn or early morning of the web age. (Though that could be
addressed by keepalives

And so we return once again to an application-layer mechanism

rick jones
--
I don't interest myself in "why." I think more often in terms of
"when," sometimes "where;" always "how much." - Joubert
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

On Feb 23, 2:47*pm, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:

> But then it's persisting rather than retransmission the poster is
> talking about. If I understand him correctly, the server *process*
> locks up, but the TCP stacks work fine and so does the path between
> them. That's why I chose SIGSTOP (and I also didn't want to unplug my
> machine from the network ;-)

The OP said that a hardware problem caused the server to become non-
responsive. I understand that as meaning the physical machine can no
longer send any packets.

> BTW, the example from my earlier posting is still running, hours
> later. The client TCP is dutifully checking the window every 120
> seconds.

If the OP needs acknowledgments, he should write code on the server
side to send them and on the client side to abandon the connection if
it doesn't get them.

DS

On Thu, 2011-02-24, David Schwartz wrote:
> On Feb 23, 2:47*pm, Jorgen Grahn <grahn+n...@snipabacken.se> wrote:
>
>> But then it's persisting rather than retransmission the poster is
>> talking about. If I understand him correctly, the server *process*
>> locks up, but the TCP stacks work fine and so does the path between
>> them. That's why I chose SIGSTOP (and I also didn't want to unplug my
>> machine from the network ;-)
>
> The OP said that a hardware problem caused the server to become non-
> responsive. I understand that as meaning the physical machine can no
> longer send any packets.

Oops, you're right (and Rick Jones was right). Sorry -- I got
sidetracked by the vague wording into thinking about what happens if
the server reboots. How I got from there to SIGSTOP, I don't know.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .