proftpd random upload stalls

Hi All,

I'm running a Gentoo system (1.2 GHz, 1 GB RAM) with proftpd (version
1.2.10) as an ftp server. Users can connect to the server and begin to
upload files... however at some point, the upload always stalls.
There's no specific file that always freezes or a certain time/file
size that does it. Most likely, however, the file will always stall at
the very end of the file...

The server is behind a NAT router and I'm using passive transfers. I
configured my proftpd.conf to use ports 60000-65535 for passive
transfers and have directed my router to forward ports 21 and
60000-65535 to the ftp server.

I've used tcpdump to capture packets during one of the "stalls". I
ended up setting Timeout for a stalled upload to 30 seconds for this
test... This is what I see: (Neither am I an expert on TCP/IP
communication... so this may be normal...)

During the upload, before the "stall", I see the remote system send a 2
packets of data and then its immediatly followed by an ACK packet from
my system. This goes on for a while... until about one second before
the start of the stall. I then see a "TCP Previous Segment Lost"
message from the remote system to the server, followed by a series of
seven "TCP Dup ACK" intermixed with different packets being sent from
the remote system.

That finally clears up and it goes back to the "two packets from the
remote system, followed by one sent back from the server" sequence for
another 0.5 seconds... and then after one of the ACK packets sent from
my server to the remote system, it seems as if nothing happens anymore.
Nothing more is transmitted from the remote system. 30 seconds later
(I have the stalled timeout to 30 seconds), my system sends a FIN
packet on both the passive transfer port and port 21... and the
communication is reset.


Does anyone have any suggestions for what may be causing this? If
anyone would like to help, please let me know... I can send you my
tcpdump file if you'd like to look at it.

Thanks in advance!

~Jeff

Here is some additional information from my tcp capture. I've taken
from just before the "stall" to a little afterwards... and you can see
the 30 seconds of nothing. I've removed from the file a little traffic
that was on my network from other machines... and renamed all the IPs
to CLIENT and SERVER respectively (to protect the innocent haha)

(If this isn't nicely formatted, let me know and I can send you the
source file)

Thanks!

~Jeff


No. Time Source Destination Protocol Info
17830 514.663321 CLIENT SERVER TCP 26309 >
60604 [ACK] Seq=213121 Ack=1 Win=5840 Len=1440 TSV=29337752
TSER=156200958
17831 514.663340 SERVER CLIENT TCP 60604 >
26309 [ACK] Seq=1 Ack=214561 Win=64088 Len=0 TSV=156201011
TSER=29337752
17832 514.698532 CLIENT SERVER TCP 26309 >
60604 [ACK] Seq=214561 Ack=1 Win=5840 Len=1440 TSV=29337752
TSER=156200958
17833 514.698550 SERVER CLIENT TCP 60604 >
26309 [ACK] Seq=1 Ack=216001 Win=64088 Len=0 TSV=156201020
TSER=29337752
17834 514.734139 CLIENT SERVER TCP 26309 >
60604 [ACK] Seq=216001 Ack=1 Win=5840 Len=1440 TSV=29337789
TSER=156200967
17835 514.769574 CLIENT SERVER TCP 26309 >
60604 [ACK] Seq=217441 Ack=1 Win=5840 Len=1440 TSV=29337823
TSER=156200975
17836 514.769593 SERVER CLIENT TCP 60604 >
26309 [ACK] Seq=1 Ack=218881 Win=64088 Len=0 TSV=156201037
TSER=29337789
17837 514.804991 CLIENT SERVER TCP 26309 >
60604 [ACK] Seq=218881 Ack=1 Win=5840 Len=1440 TSV=29337860
TSER=156200984
17838 514.840449 CLIENT SERVER TCP 26309 >
60604 [ACK] Seq=220321 Ack=1 Win=5840 Len=1440 TSV=29337894
TSER=156200993
17839 514.840467 SERVER CLIENT TCP 60604 >
26309 [ACK] Seq=1 Ack=221761 Win=64088 Len=0 TSV=156201055
TSER=29337860

== 30 Second Pause ==

17840 544.842751 SERVER CLIENT TCP 60604 >
26309 [FIN, ACK] Seq=1 Ack=221761 Win=64088 Len=0 TSV=156208556
TSER=29337860
17841 544.842801 SERVER CLIENT TCP ftp > 26296
[FIN, ACK] Seq=1950 Ack=1117 Win=5792 Len=0 TSV=156208556 TSER=29332616
17842 544.882467 CLIENT SERVER TCP [TCP
Previous segment lost] 26309 > 60604 [ACK] Seq=222674 Ack=2 Win=5840
Len=0 TSV=29368152 TSER=156208556
17843 544.924596 CLIENT SERVER TCP 26296 > ftp
[ACK] Seq=1117 Ack=1951 Win=5840 Len=0 TSV=29368195 TSER=156208556
17848 573.184811 CLIENT SERVER TCP [TCP
Retransmission] 26309 > 60604 [FIN, PSH, ACK] Seq=221761 Ack=2 Win=5840
Len=912 TSV=29396437 TSER=156208556
17849 573.184889 SERVER CLIENT TCP 60604 >
26309 [RST] Seq=2 Ack=2483942091 Win=0 Len=0
17853 601.623341 CLIENT SERVER TCP 26237 > 22
[ACK] Seq=3693 Ack=6735 Win=12096 Len=0 TSV=29424903 TSER=156222741
17854 601.626616 CLIENT SERVER TCP 26237 > 22
[ACK] Seq=3693 Ack=6815 Win=12096 Len=0 TSV=29424906 TSER=156222741
17859 623.867831 CLIENT SERVER FTP [TCP
Retransmission] Request: type I
17860 623.867943 SERVER CLIENT TCP ftp > 26107
[RST] Seq=2844 Ack=3483450447 Win=0 Len=0
17861 631.925263 CLIENT SERVER TCP [TCP
Retransmission] 26309 > 60604 [FIN, PSH, ACK] Seq=221761 Ack=2 Win=5840
Len=912 TSV=29455189 TSER=156208556
17862 631.925327 SERVER CLIENT TCP 60604 >
26309 [RST] Seq=2 Ack=2483942091 Win=0 Len=0
17863 634.186275 CLIENT SERVER FTP Request:
type I
17864 634.186358 SERVER CLIENT TCP ftp > 26296
[RST] Seq=1951 Ack=2631535714 Win=0 Len=0
17865 634.256116 CLIENT SERVER TCP 26319 > ftp
[SYN] Seq=0 Ack=0 Win=5840 Len=0 MSS=1452 TSV=29457537 TSER=0 WS=0
17866 634.256159 SERVER CLIENT TCP ftp > 26319
[SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=156230909
TSER=29457537 WS=2
17867 634.296160 CLIENT SERVER TCP 26319 > ftp
[ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=29457580 TSER=156230909
17868 634.308749 SERVER CLIENT TCP 33235 >
auth [SYN] Seq=0 Ack=0 Win=5840 Len=0 MSS=1460 TSV=156230922 TSER=0
WS=2
17869 634.432214 CLIENT SERVER FTP [TCP
Retransmission] Request: type I
17870 634.432268 SERVER CLIENT TCP ftp > 26296
[RST] Seq=1951 Ack=2631535714 Win=0 Len=0
17871 634.924476 CLIENT SERVER FTP [TCP
Retransmission] Request: type I
17872 634.924548 SERVER CLIENT TCP ftp > 26296
[RST] Seq=1951 Ack=2631535714 Win=0 Len=0
17873 637.305376 SERVER CLIENT TCP 33235 >
auth [SYN] Seq=0 Ack=0 Win=23360 Len=0 MSS=1460 TSV=156231672 TSER=0
WS=2
17874 643.305322 SERVER CLIENT TCP 33235 >
auth [SYN] Seq=0 Ack=0 Win=23360 Len=0 MSS=1460 TSV=156233172 TSER=0
WS=2
17875 644.313611 SERVER CLIENT FTP Response:
220 ProFTPD 1.2.10 Server (Pro
17876 644.353760 CLIENT SERVER TCP 26319 > ftp
[ACK] Seq=1 Ack=75 Win=5840 Len=0 TSV=29467640 TSER=156233424
17877 644.356350 CLIENT SERVER FTP Request:
user *******
17878 644.356375 SERVER CLIENT TCP ftp > 26319
[ACK] Seq=75 Ack=18 Win=5792 Len=0 TSV=156233434 TSER=29467641
17879 644.359504 SERVER CLIENT FTP Response:
331 Password required for ****
17880 644.400966 CLIENT SERVER FTP Request:
pass *******
17881 644.406217 SERVER CLIENT FTP Response:
230 User ****** logged in.

Jeff Plotzke wrote:
> Hi All,
>
> I'm running a Gentoo system (1.2 GHz, 1 GB RAM) with proftpd (version
> 1.2.10) as an ftp server. Users can connect to the server and begin to
> upload files... however at some point, the upload always stalls.
> There's no specific file that always freezes or a certain time/file
> size that does it. Most likely, however, the file will always stall at
> the very end of the file...
>
> The server is behind a NAT router and I'm using passive transfers. I
> configured my proftpd.conf to use ports 60000-65535 for passive
> transfers and have directed my router to forward ports 21 and
> 60000-65535 to the ftp server.
>
> I've used tcpdump to capture packets during one of the "stalls". I
> ended up setting Timeout for a stalled upload to 30 seconds for this
> test... This is what I see: (Neither am I an expert on TCP/IP
> communication... so this may be normal...)

I don't know your network infrastructure ( switch ... ) but perhaps you could
see for spanning-tree problem with 30s convergence time


>
> During the upload, before the "stall", I see the remote system send a 2
> packets of data and then its immediatly followed by an ACK packet from
> my system. This goes on for a while... until about one second before
> the start of the stall. I then see a "TCP Previous Segment Lost"
> message from the remote system to the server, followed by a series of
> seven "TCP Dup ACK" intermixed with different packets being sent from
> the remote system.
>
> That finally clears up and it goes back to the "two packets from the
> remote system, followed by one sent back from the server" sequence for
> another 0.5 seconds... and then after one of the ACK packets sent from
> my server to the remote system, it seems as if nothing happens anymore.
> Nothing more is transmitted from the remote system. 30 seconds later
> (I have the stalled timeout to 30 seconds), my system sends a FIN
> packet on both the passive transfer port and port 21... and the
> communication is reset.
>
>
> Does anyone have any suggestions for what may be causing this? If
> anyone would like to help, please let me know... I can send you my
> tcpdump file if you'd like to look at it.
>
> Thanks in advance!
>
> ~Jeff
>

Thanks for your reply.

I don't know much about "spanning tree problem" or "convergence time".


My network has a pretty simple setup to it: My main incoming internet
line is connected to a NAT router. All servers are connected directly
to that router (including the FTP server I'm having the problem with),
and also off the router are a series of switches which go to
workstations.

Does that help any? Can someone explain spanning tree problem to me?

Thanks!

~Jeff