Linux ignoring SYN packets

Hello all.

I am facing a very puzzling problem.

Scenario:
on the server I have a program listening on port 9999
a client tries to connect (via telnet) to the server on port 9999

Sometimes, establishing the connection takes a long time (one minute
or more).
What happens is that the client sends a SYN packet, the syn packet is
received by the server, but the server does not reply to the SYN
packet with a SYN/ACK (the program listening on the server does not
see the incoming connection).

After 4/5 SYN sents by the client, the servers finally answers with a
SYN/ACK, the connection is established and the program on the server
sees the connection).

I ran a packet sniffer on both client and server, and the result is
the same.

19:53:45.038667 IP (tos 0x0, ttl 58, id 27948, offset 0, flags [DF],
proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8571162
0,nop,wscale 7>

19:53:57.044368 IP (tos 0x0, ttl 58, id 27949, offset 0, flags [DF],
proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8574162
0,nop,wscale 7>

19:54:21.045412 IP (tos 0x0, ttl 58, id 27950, offset 0, flags [DF],
proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8580162
0,nop,wscale 7>

19:55:09.040188 IP (tos 0x0, ttl 58, id 27951, offset 0, flags [DF],
proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8592162
0,nop,wscale 7>

19:57:08.584133 IP (tos 0x0, ttl 58, id 64380, offset 0, flags [DF],
proto 6, length: 60) client.43425 > server.9999: S [tcp sum ok]
2090300578:2090300578(0) win 5840 <mss 1460,sackOK,timestamp 8622046
0,nop,wscale 7>

19:57:08.595922 IP (tos 0x0, ttl 58, id 64381, offset 0, flags [DF],
proto 6, length: 52) client.43425 > server.9999: . [tcp sum ok]
2090300579:2090300579(0) ack 1191201502 win 46 <nop,nop,timestamp
8622049 2164260539>

19:57:14.223948 IP (tos 0x0, ttl 58, id 64382, offset 0, flags [DF],
proto 6, length: 52) client.43425 > server.9999: F [tcp sum ok] 0:0(0)
ack 1 win 46 <nop,nop,timestamp 8623457 2164260539>

19:57:14.236287 IP (tos 0x0, ttl 58, id 64383, offset 0, flags [DF],
proto 6, length: 52) client.43425 > server.9999: . [tcp sum ok] 1:1(0)
ack 2 win 46 <nop,nop,timestamp 8623459 2164266179>


As you can see, the server ignores the first SYNs but at some point,
for no apparent reason, finally replies with a SYN/ACK.

Does anyone have any clue?

Thank you.


Informations on the server:
CentOS release 4.6 (Final)

[root@ ~]# uname -a
Linux ebrick.net 2.6.9-67.0.7.ELsmp #1 SMP Sat Mar 15 06:54:55 EDT
2008 i686 i686 i386 GNU/Linux

dmesg (partial):
divert: allocating divert_blk for eth0
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
ACPI: PCI Interrupt 0000:00:1b.0[A] -> GSI 21 (level, low) -> IRQ 193
PCI: Setting latency timer of device 0000:00:1b.0 to 64
e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex,
Flow Control: RX
e1000: eth0: e1000_watchdog_task: 10/100 speed: disabling TSO

On May 20, 10:20*am, ErMejo <erm...@gmail.com> wrote:
> Hello all.
>
> I am facing a very puzzling problem.
>
> Scenario:
> on the server I have a program listening on port 9999
> a client tries to connect (via telnet) to the server on port 9999

What's the listen backlog set to? (When you called 'listen', the
second parameter.)

What is the program doing while it's not seeing the connection? Is it
blocked in 'accept' or doing something else entirely?

DS

On May 21, 12:13*am, David Schwartz <dav...@webmaster.com> wrote:
> On May 20, 10:20*am, ErMejo <erm...@gmail.com> wrote:
>
> > Hello all.
>
> > I am facing a very puzzling problem.
>
> > Scenario:
> > on the server I have a program listening on port 9999
> > a client tries to connect (via telnet) to the server on port 9999

> What is the program doing while it's not seeing the connection? Is it
> blocked in 'accept' or doing something else entirely?

It is blocked in the 'accept' statement.

Still, this problem surfaces with any kind of program listening on the
server (it also shows on Apache HTTPD and SSH).

Bye

On May 20, 7:20*pm, ErMejo <erm...@gmail.com> wrote:
> Hello all.
>
> I am facing a very puzzling problem.
>
> Scenario:
> on the server I have a program listening on port 9999
> a client tries to connect (via telnet) to the server on port 9999
>
> Sometimes, establishing the connection takes a long time (one minute
> or more).
> What happens is that the client sends aSYNpacket, thesynpacket is
> received by the server, but the server does not reply to theSYN
> packet with aSYN/ACK (the program listening on the server does not
> see the incoming connection).
>
> After 4/5SYNsents by the client, the servers finally answers with aSYN/ACK, the connection is established and the program on the server
> sees the connection).
>
> I ran a packet sniffer on both client and server, and the result is
> the same.
>
> 19:53:45.038667 IP (tos 0x0, ttl *58, id 27948, offset 0, flags [DF],
> proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
> 3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8571162
> 0,nop,wscale 7>
>
> 19:53:57.044368 IP (tos 0x0, ttl *58, id 27949, offset 0, flags [DF],
> proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
> 3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8574162
> 0,nop,wscale 7>
>
> 19:54:21.045412 IP (tos 0x0, ttl *58, id 27950, offset 0, flags [DF],
> proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
> 3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8580162
> 0,nop,wscale 7>
>
> 19:55:09.040188 IP (tos 0x0, ttl *58, id 27951, offset 0, flags [DF],
> proto 6, length: 60) client.42884 > server.9999: S [tcp sum ok]
> 3079908730:3079908730(0) win 5840 <mss 1460,sackOK,timestamp 8592162
> 0,nop,wscale 7>
>
> 19:57:08.584133 IP (tos 0x0, ttl *58, id 64380, offset 0, flags [DF],
> proto 6, length: 60) client.43425 > server.9999: S [tcp sum ok]
> 2090300578:2090300578(0) win 5840 <mss 1460,sackOK,timestamp 8622046
> 0,nop,wscale 7>
>
> 19:57:08.595922 IP (tos 0x0, ttl *58, id 64381, offset 0, flags [DF],
> proto 6, length: 52) client.43425 > server.9999: . [tcp sum ok]
> 2090300579:2090300579(0) ack 1191201502 win 46 <nop,nop,timestamp
> 8622049 2164260539>
>
> 19:57:14.223948 IP (tos 0x0, ttl *58, id 64382, offset 0, flags [DF],
> proto 6, length: 52) client.43425 > server.9999: F [tcp sum ok] 0:0(0)
> ack 1 win 46 <nop,nop,timestamp 8623457 2164260539>
>
> 19:57:14.236287 IP (tos 0x0, ttl *58, id 64383, offset 0, flags [DF],
> proto 6, length: 52) client.43425 > server.9999: . [tcp sum ok] 1:1(0)
> ack 2 win 46 <nop,nop,timestamp 8623459 2164266179>
>
> As you can see, the server ignores the first SYNs but at some point,
> for no apparent reason, finally replies with aSYN/ACK.
>
> Does anyone have any clue?
>
> Thank you.
>
> Informations on the server:
> CentOS release 4.6 (Final)
>
> [root@ ~]# uname -aLinuxebrick.net 2.6.9-67.0.7.ELsmp #1 SMP Sat Mar 15 06:54:55 EDT
> 2008 i686 i686 i386 GNU/Linux
>
> dmesg (partial):
> divert: allocating divert_blk for eth0
> e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
> ACPI: PCI Interrupt 0000:00:1b.0[A] -> GSI 21 (level, low) -> IRQ 193
> PCI: Setting latency timer of device 0000:00:1b.0 to 64
> e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex,
> Flow Control: RX
> e1000: eth0: e1000_watchdog_task: 10/100 speed: disabling TSO

Additionally, on the client:

$ netstat -anp
tcp 0 1 client:42776 server:443 SYN_SENT

ErMejo wrote:
> Hello all.
>
> I am facing a very puzzling problem.
>
> Scenario:
> on the server I have a program listening on port 9999
> a client tries to connect (via telnet) to the server on port 9999
>
> Sometimes, establishing the connection takes a long time (one minute
> or more).
> What happens is that the client sends a SYN packet, the syn packet is
> received by the server, but the server does not reply to the SYN
> packet with a SYN/ACK (the program listening on the server does not
> see the incoming connection).
>
> After 4/5 SYN sents by the client, the servers finally answers with a
> SYN/ACK, the connection is established and the program on the server
> sees the connection).

I've seen this before and based on your later follow-up in this thread
that this affects SSH and HTTP connections too it leads me to suggest a
DNS problem.

The listening server may be trying to do a reverse lookup on the
client's IP. If the resolver can't reach the DNS servers, it will wait
for them to time out. After the time out, the server process abandons
the reverse lookup and accepts the connection (if it is configured to
allow unresolvable IP's). However, while the server is waiting for DNS
the client will very likely send more SYN packets assuming the packet
wasn't received as it wasn't responded to in a timely fashion.

Check your DNS/resolver config

Cheers,

James

Centurion <(E-Mail Removed)> wrote:

> I've seen this before and based on your later follow-up in this
> thread that this affects SSH and HTTP connections too it leads me to
> suggest a DNS problem.

> The listening server may be trying to do a reverse lookup on the
> client's IP. If the resolver can't reach the DNS servers, it will
> wait for them to time out. After the time out, the server process
> abandons the reverse lookup and accepts the connection (if it is
> configured to allow unresolvable IP's). However, while the server
> is waiting for DNS the client will very likely send more SYN packets
> assuming the packet wasn't received as it wasn't responded to in a
> timely fashion.

Does Linux actually have a mode to allow connection accept in that
fashion? I always thought it waited at least until the SYN|ACK was
sent in response to the SYN before the listening application was
notified there was anything to consider accepting. IIRC there is no
means in the standard sockets interface to do such a split accept() in
the first place?

rick jones
--
web2.0 n, the dot.com reunion tour...
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

> I've seen this before and based on your later follow-up in this thread
> that this affects SSH and HTTP connections too it leads me to suggest a
> DNS problem.

Basically, as far as I can tell, it affects any kind of program
listening on a socket on the server.

> The listening server may be trying to do a reverse lookup on the
> client's IP. *If the resolver can't reach the DNS servers, it will wait
> for them to time out. *After the time out, the server process abandons
> the reverse lookup and accepts the connection (if it is configured to
> allow unresolvable IP's). *However, while the server is waiting for DNS
> the client will very likely send moreSYNpackets assuming the packet
> wasn't received as it wasn't responded to in a timely fashion.
> Check your DNS/resolver config

Good guess, we also experienced this problem.

But it seems it is not the case now.

During the tests, I run also a:

tcpdump -s 0 -vv -i eth0 host ns1 or host ns2 or host ns3

(where ns1-3 are my name servers).
During the tests, nor the listening program nor the server does any
DNS lookup.

Bye

> DoesLinuxactually have a mode to allow connection accept in that
> fashion? *I always thought it waited at least until theSYN|ACK was
> sent in response to theSYNbefore the listening application was
> notified there was anything to consider accepting. *IIRC there is no
> means in the standard sockets interface to do such a split accept() in
> the first place?

I do not know if such a 'split accept' exists, anyway my test server
program uses a normal 'accept', i.e. it does not receive anything
untile the 3-way handshake is complete.

Bye