[crossposted to comp.mail.sendmail and comp.os.linux.networking since
the bug might be in sendmail, the linux tcp stack, ssl, iptables, or
somewhere else ....]
I've managed to discover something fairly disturbing when sending mail
from a linux machine running sendmail: if the connection is encrypted
via TLS, the connection won't close cleanly (with an exchange of FIN
packets) but will terminate abruptly with a RST packet (from the
machine that originated the email). I have not observed a RST packet
from the originating machine when TLS is _not_ used.
I just tested this with someone to try to figure out what's wrong.
Here was the setup:
My end is a RHEL 3 box:
kernel-smp-2.4.21-27.0.2.EL
sendmail-8.12.11-4.RHEL3.1
Remote end is a Debian box:
kernel: 2.6.12-rc2-mm1
sendmail: 8.13.4
We confirmed that if the RedHat box initiated the message transfer, it
would be the one to issue a RST after the transaction was complete.
Similarly, if the Debian box initiated the transfer, it would issue a
RST.
Mail *is* flowing, but I find it somewhat disturbing that there are RST
packets being thrown about. My theory of what might be happening is
that the sending sendmail process terminates after sending its final
data (QUIT) to the SSL layer, rather than waiting for any final
response (221 Goodbye). Since the process is no longer there, the TCP
connection is broken, and a RST gets generated. That means the bug is
in sendmail, right? I'd like to get some confirmation about this
reasoning from others before filing a bug report, though, since my
hypothesis is little more than an educated guess.
Cheers,
Damian Menscher
--
-=#| Physics Grad Student & SysAdmin @ U Illinois Urbana-Champaign |#=-
-=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc

217)333-0038 |#=-
-=#| 4602 Beckman, VMIL/MS, Imaging Technology Group

217)244-3074 |#=-
-=#| <(E-Mail Removed)>
www.uiuc.edu/~menscher/ Fax

217)333-9819 |#=-
-=#| The above opinions are not necessarily those of my employers. |#=-