IPtables flagging packets invalid, no access

Discussion in 'Linux Networking' started by 86brown, Apr 19, 2014.

  1. 86brown

    86brown Guest

    Hi all,

    Looking to get some help on an issue I'm having, preventing me from gettinga certain new location going. New datacenter, connecting the feed they're providing me to L3 switch. Have not been able to set up a proper iptables firewall in this location (everything is blocked, no access). Only way to access is adding my IP to allow list.

    Packets are being flagged as invalid, no reply to [SYN] packets from the server. Not sure if this is whats blocking or not.

    This is what most of the blockings are saying:
    kernel: [20604.837769] Firewall: *TCP_IN Blocked* IN=venet0 OUT= MAC= SRC=*MYIP* DST=*SERVERIP* LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=15851 DF PROTO=TCP SPT=61742 DPT=21 WINDOW=8192 RES=0x00 SYN URGP=0

    Here's the invalid packet:
    kernel: [16708.550424] Firewall: *INVALID* IN=venet0 OUT= MAC= SRC=MYIP DST=SERVERIP LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=12281 DF PROTO=TCP SPT=60992 DPT=80 WINDOW=8192 RES=0x00 SYN URGP=0

    I ran a Wireshark & tcpdump simultaneously to catch client/server side and here was the result:
    Client side (my PC):
    77 4.434317000 192.168.2.244 SERVER-IP TCP 66 63866 > http [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=4 SACK_PERM=1

    Server side (server tcpdump)
    1 0.000000 MY-ISP-IP SERVER-IP TCP 68 61992 > http [SYN] Seq=0 Win=8192 Len=0 MSS=1452 WS=4 SACK_PERM=1

    So the only thing I notice is that the server-side MSS is different then what the client (my PC) sent out. Is this normal, is this what's flagging as invalid?

    Basically I'm trying to setup a cPanel server with csf firewall (which usesiptables) but as soon as its active I get no access, have to log onto VPS node, drop into via 'vzctl enter *' and shutdown iptables.

    For hardware I'm using a Nortel Baystack 5510 L3 switch, and I believe thatthe DC is using a Cisco ASA but I could be wrong.

    Any suggestions to the solution here would be greatly appreciated!! :)
     
    86brown, Apr 19, 2014
    #1
    1. Advertisements

  2. Hello,

    86brown a écrit :
    We don't know what means "INVALID" nor the reason for "TCP_IN Blocked".
    This is ruleset-dependent. Examining the output of iptables-save may
    provide clues.
    Clamping the MSS to 1452 is a rather common ISP practice to workaround
    MTU blackhole issues which may affect some DSL lines subscribers using
    PPPoE.
    Not if done properly, i.e. recalculating checksums as needed.
     
    Pascal Hambourg, Apr 20, 2014
    #2
    1. Advertisements

  3. 86brown

    86brown Guest

    Thanks for your reply :)

    Could it be due to TCP window scaling or TCP sequence number randomization on the ASA, or I've also read reference to PIX? Do you have any suggestionsof what I should ask my DC?

    Next time I go there (Tuesday), I'm going to bypass my L3 Baystack 5510 switch, connecting the feed directly to the server. Then see if the packets are still being flagged invalid and being dropped. This is a standard installand I've setup 30 some servers in a similar fashion in different DC's but not yet in this one as it's new and trying to get things prepared.

    No good if any new packet marked only [SYN] to initiate a connection is being ignored for some reason, thats the whole basis of a web server is initiating new connections to new visitors, so disabling stateful iptables rules is not an option as we want SPI to be available.
     
    86brown, Apr 20, 2014
    #3
  4. 86brown a écrit :
    This is unlikely if "adding your IP address to the allow list" allows
    you to connect to the server. I think that it is due to the iptables
    ruleset.
     
    Pascal Hambourg, Apr 20, 2014
    #4
  5. 86brown

    86brown Guest

    Yes adding my IP to the allow list does allow me through. The server receives the packet but flags it invalid, and I assume it then checks to see if I'm on the allow list whether to allow it or drop it (csf+lfd).

    The problem I have there is that even CentOS default iptables rules have this issue, if its enabled. Or at least when I switched over to OpenVZ kernel.. I had to drive to the DC to do a 'service iptables stop' and 'chkconfig iptables off' to make sure I wouldn't lose access again during my messing with it.

    I've compared the rules on a working server to non-working server, and evencopied the configuration of csf to the non-working server and there are nochanges.

    This is not happening on my other servers in other DC's (only diff here is I'm using a Layer 3 switch, other DC's I'm on a Layer 2). I'm not sure if its the DC traffic or once it goes past my L3 switch, but something is causing iptables to flag those packets, which I would assume to be the root cause of the issue.

    Sure, I could disable the statefulness but this is not an ideal solution, especially for our clients that may want these types of features. Really need to find out why these are coming in that way causing such an issue.
     
    86brown, Apr 20, 2014
    #5
  6. 86brown a écrit :
    You keep talking about invalid packets, but you wrote in your first post
    that most firewall log messages were flagged "TCP_IN Blocked", not
    "INVALID". So what's the bigger problem ?

    Besides, we don't know what do these messages mean exactly. You need to
    check which rules produce them.
    Is it "INVALID" per the connection tracking ? AFAIK a TCP SYN packet
    cannot have the INVALID conntrack state, unless a connection with the
    same characteristics already exists (and the packet would be discarded
    by the TCP layer anyway), or the conntrack table is full. If "INVALID"
    meant "malformed" (e.g. bad checksum), then the packet would also be
    discarded by the TCP/IP stack regardless of iptables.
     
    Pascal Hambourg, Apr 20, 2014
    #6
  7. 86brown

    86brown Guest

    I honestly am not very much sure what's going on either, hence the reason for my postings here!

    All I know, is that the ports are opened, but I can't get access to my server with iptables running, unless I explicitly add a rule for my IP (in other words add it in CSF's allow list). CSF takes control of all that and it is working perfectly on my other CentOS 6.5 box in a different datacenter. *Invalid* is only one of the things I saw, a lot of them say TCP_IN or UDP blocked but I believe that is after the initial packet, upon TCP retransmission, not sure. In Wireshark, there is no response [ACK] to the initial packet [SYN] requesting to start a TCP session.

    Are default iptables rules on CentOS 6 to allow SSH connection by default? Because if so, even that's not working.

    Either way, why would the packet be blocked if the port is opened?? Clearlyits hitting the firewall, with the logs I'm getting. I guess the question is, what can make iptables think it needs to block a connection to an opened port?? Would it not be malformed or invalid packets of some kind? This iswhy I keep referring to that, because it's the only oddity I've seen so far.

    Any tips or suggestions on how to find out the problem and furthermore solution to the issue would be great! I can go and remove the problem rules from iptables, but that's just a band-aid and is not ideal to the resolution of the root cause of the issue. There must be a good reason that invalid/malformed packets are blocked, so me allowing them through purposely I'm sure is not an action plan.
     
    86brown, Apr 21, 2014
    #7
  8. 86brown

    86brown Guest

    Could it possibly be due to the fact that I've not configured the L3 switchany further? Have not tagged/untagged any ports on the VLANS or the ISP port, should I?
     
    86brown, Apr 21, 2014
    #8
  9. 86brown a écrit :
    No. VLAN tagging is a layer 2 function, iptables does not care about it.
     
    Pascal Hambourg, Apr 22, 2014
    #9
  10. 86brown

    86brown Guest

    Ok thank you, I'll just bypass the switch tomorrow and see if it still happens. If it does, I have no idea what I'll ask my colo though.

    Thanks!
     
    86brown, Apr 22, 2014
    #10
  11. 86brown

    86brown Guest

    Turns out issue was related to recent change in OpenVZ:
    http://git.openvz.org/?p=vzctl;a=commit;h=9b8afa654945acc6d3bd782f622aaf9c54e4e87b
     
    86brown, Apr 23, 2014
    #11
  12. Pascal Hambourg, Apr 23, 2014
    #12
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.