wpad.dat attack on Linux Apache server

Discussion in 'Linux Networking' started by Sandman, May 24, 2013.

  1. Sandman

    Sandman Guest

    I don't actually know if it's an attack.

    My Apache server was slow - symptoms being slow responses (after PHP had
    processed and sent to client) and dropped connections, looked in the log
    files, found 12GB of accesses to /wpad.dat on my catch-all vhost.

    I got thousands of requests per minute from hundreds of different hosts,
    and a sample of these showed that they all seemed like legit end-user
    hosts, not a tor proxy botnet at least.

    I googled some, and found that wpad is some form of auto-discover proxy
    settings. Problem is, it should be done to the local network. So if my
    machine is on the "example.com" network, my browser will send a request
    to "wpad.example.com" to find proxy settings. not
    "wpad.remoteinternetsite.com".

    So, maybe it is an attack after all?

    I added a wpad.dat file to the server, with this content:

    function FindProxyForURL(url, host) { return "PROXY 127.0.0.1:445"; }

    Which just tells these clients to look for a proxy on localhost. Nothing
    changed. Thousands of thousands of requests.

    In /server-status for Apache, my queue is filled with /wpad.dat requests
    with the "K" status (Keep-Alive), so that sounds like why it's slow.

    Ok, so I'll block it. Blocking it in apache seemd stupid, it would still
    process the requests, so to iptables:

    iptables -I INPUT -p tcp --dport 80 -m string --to 70 --algo bm \
    --string "GET /wpad.dat" -j REJECT --reject-with tcp-reset

    Yes, I know this analyzes *every* request, and wastes CPU cycles, I may
    get around to chain this into a seperate iptables chain and only act on
    correct parts. In about an hour, this has blocked 45k requests, or about
    750 per minute

    Either way, the requests are gone, the log file is clean(er) but the
    server is still slow and still drops connections.

    So, troubleshooting some more. CPU is at 0.8% usage, Memory is 80% free.
    Asking my hosting ISP, my bandwidth capacity is at 0.62%

    ethtool -S eth0 shows no errors (but a shitload of packets, of course)

    Someone suggested that it could have something to do with my using a
    wildcard DNS. So, my server hosts some 100+ virtual hosts. All my
    clients are told to use a CNAME pointer for their servers. So:

    www.client.com -> CNAME -> cluster.mydomain.com -> A -> 123.123.123.123

    Which means that every visitor to my sites has their web browser first
    look up www.client.com to find cluster.mydomain.com which in turn points
    to my IP.

    No, the mydomain.com had a wildcard setting, so if and when they would
    access "wpad.mydomain.com" my DNS would point that to
    cluster.mydomain.com and then that wold point to the IP. So supposedly,
    all the request could channel to my server this way.

    I have now removed wildcard for mydomain.com, and also added a wpad host
    for all my domains that points to 127.0.0.1. I'm waiting to see that
    propagate and see if it makes any difference. It hasn't so far.


    Do any of you guys have any ideas what this might be? Or rather - how
    do I trouble shoot this some more?

    I have:

    Slow transfer speeds on apache
    Super fast on other ports (SFTP for instance)
    Thousands of requests per minute that are now being blocked
    Super low CPU usage
    Super low RAM usage
    No reported ethernet errors
     
    Sandman, May 24, 2013
    #1
    1. Advertisements

  2. While this will send a TCP reset to the misbehaving client, AIUI it will
    not send anything to your Apache, which by this point will have an open
    TCP connection and will be awaiting the start of the HTTP request. It
    will presumably continue waiting up to some timeout. You should be able
    to use netstat to confirm or refute this.
     
    Richard Kettlewell, May 24, 2013
    #2
    1. Advertisements

  3. Sandman

    Sandman Guest

    Thank you for your reply. I thought "reject" just rejected the request
    and nothing came to Apache?

    Using /server-status I see a huge difference in active connections, but
    I do see a tremendous amount of "Total accesses" which doesn't seem to
    correlate to the actual number of "valid" HTTP requests.

    Current Time: Friday, 24-May-2013 11:39:38 CEST
    Restart Time: Friday, 24-May-2013 11:22:51 CEST
    Parent Server Generation: 0
    Server uptime: 16 minutes 47 seconds
    Total accesses: 14027 - Total Traffic: 144.8 MB
    CPU Usage: u124.78 s5.91 cu136.95 cs0 - 26.6% CPU load
    13.9 requests/sec - 147.2 kB/second - 10.6 kB/request
    29 requests currently being processed, 24 idle workers

    ...._.KK.KK..........._.._...K._.R..._...K_.._...K......._R..K_.
    K_KKKK.._K_KW_WK_K__KK_....K_K_K___K_KK._.......................
    ................................................................
    ................................................................


    As you can see - almost a thousand accesses per minute. Yet, my child
    processes are free to serve new processes.

    "netstat -lap" shows about 400 lines of this:

    tcp 0 0 www.mydomain.com:www c-83-233-215-17.c:49686 SYN_RECV -
    tcp 0 0 www.mydomain.com:www h-5-200.a327.priv:50165 SYN_RECV -
    tcp 0 0 www.mydomain.com:www c-62-220-189-209.:50627 SYN_RECV -
    tcp 0 0 www.mydomain.com:www 238.77.85.212.bah:58190 SYN_RECV -
    tcp 0 0 www.mydomain.com:www c-89-160-22-176.c:57315 SYN_RECV -

    So yes, something is still knocking on that door - could this be bogging
    down my server?

    I have also a number of lines with TIME_WAIT status, suggesting that
    some queue is full here... Right?
     
    Sandman, May 24, 2013
    #3
  4. You’re rejecting a packet that is part of an already-established TCP
    connection. iptables cannot go back in time and prevent the TCP
    connection from being established in the first place.
     
    Richard Kettlewell, May 24, 2013
    #4
  5. Sandman

    Sandman Guest

    You’re rejecting a packet that is part of an already-established TCP
    connection. iptables cannot go back in time and prevent the TCP
    connection from being established in the first place.[/QUOTE]

    Yes, like I said - I thought nothing came through to Apache. But looking
    at server-status, it seems it does anyway?
     
    Sandman, May 24, 2013
    #5
  6. I don’t know how to put it any more clearly; I give up.
     
    Richard Kettlewell, May 24, 2013
    #6
  7. Sandman

    Sandman Guest

    I don’t know how to put it any more clearly; I give up.[/QUOTE]

    No, please don't. Maybe I am misunderstanding you? I am not trying to
    argue with you.

    I thought that rejecting the TCP request in iptables blocked the request
    from ever reaching the httpd process. Obviously it isn't blocked from
    the *machine*, and I apologize if you thought that was what I meant.

    Mind you, I don't get any HTTP requests in Apache, but it does increment
    the requests number in a rate faster than the normal requests I see.

    You are free to call me stupid and ignorant about iptables/httpd here,
    of course, but I would still very much like to solve my problem even so
    :)
     
    Sandman, May 24, 2013
    #7
  8. Sandman

    Chris Davies Guest

    The TCP sequence goes like this:

    1. Remote sends SYN to Webserver
    2. Webserver sends SYN/ACK to Remote
    3. Remote sends ACK to Webserver
    --connection now established--
    4. Remote sends "GET / HTTP/1.0 [..etc..]" to Webserver
    5. Webserver sends ACK to Remote
    6. Webserver sends the HTTP response to Remote
    7. Remote sends ACK
    8. Connection gets reused (from #4) or closed (FIN - FIN/ACK)

    Often #3 and #4 are merged, and potentially #5 and #6 could be, too. Item
    #6 might be spread across several packets, in which case the Remote will
    send an ACK (#7) for each packet.

    Your iptables rule matches #4, but by this stage the Webserver has already
    got a connection established from the Remote, and possibly even an Apache
    child ready to serve it.

    Chris
     
    Chris Davies, May 24, 2013
    #8
  9. Sandman

    Sandman Guest

    The TCP sequence goes like this:

    1. Remote sends SYN to Webserver
    2. Webserver sends SYN/ACK to Remote
    3. Remote sends ACK to Webserver
    --connection now established--
    4. Remote sends "GET / HTTP/1.0 [..etc..]" to Webserver
    5. Webserver sends ACK to Remote
    6. Webserver sends the HTTP response to Remote
    7. Remote sends ACK
    8. Connection gets reused (from #4) or closed (FIN - FIN/ACK)

    Often #3 and #4 are merged, and potentially #5 and #6 could be, too. Item
    #6 might be spread across several packets, in which case the Remote will
    send an ACK (#7) for each packet.

    Your iptables rule matches #4, but by this stage the Webserver has already
    got a connection established from the Remote, and possibly even an Apache
    child ready to serve it.[/QUOTE]

    Ok, thank you for the explanation, I had that backwards.

    So, where would I start at for finding out who has done a /wpad.dat
    request and then add them to a firewall IP block list? Maybe that's the
    best route to go?
     
    Sandman, May 24, 2013
    #9
  10. Sandman

    Sandman Guest

    Yes, I have seen this thread, and they are talking about wildcard DNS
    being the culprit - I have yet to understand how this applies to me?

    I mean - I *DO* use wildcard DNS for all domains that I have a DNS for.
    The DNS server is on the machine that is currently being flooded on port
    80. Ok. So the DNS is "ns1.mydomain.com" (for example).

    So, for my clients - whose web pages I host on this machine, I tell
    *them* to point their subdomains (I.e. www.client.com) to the domainname
    "cluster.mydomain.com" as a CNAME record.

    Now, when the visitor types in www.client.com into the web browser,
    their DNS says that that resolvs to cluster.mydomain.com - which in turn
    has an A record for an IP number. So in the end, they surf to
    cluster.mydomain.com asking for www.client.com

    This works very good and has for more than a year.

    Now, mydomain.com har a wildcard DNS, meaning that
    "lkjkljklj.mydomain.com" points to "cluster.mydomain.com" and then to
    the IP.

    According to that thread, Internet Explorer and/or Windows makes
    assumptions about where to look for "wpad.dat", a javascript file that
    aims to provide the browser/hte OS info about proxy servers.

    So, Windows/IE asks for "http://wpad.client.com:80/wpad.dat" (as far as
    I know) and that's where the problem is.

    Now, to counter this:

    1. I have removed wildcard DNS on mydomain.com
    propagation may take a while though

    2. I am actively pointing wpad.mydomain.com to
    127.0.0.1, also waiting for the TTL there.

    3. I am trying to use iptables to block these accesses

    The open questions seem to be several, which the forum thread doesn't
    seem to have an answer for:

    1. Why would thousands of clients per minute all over Sweden ask for
    a wpad.dat file on *my* machine? According to the standard, they
    should be asking for it on wpad.*client.com*, not wpad.mydomain.com

    2. And why the *excessive* amount of traffic. several hundreds of IPS
    make up thousands of requests per minute, meaning that one IP makes
    several requests often.

    3. Blocking these IP-number, would I also be blocking their normal
    traffic to the server? Meaning, are these flooding some form of
    colleteral traffic from normal surfing?

    Thanks for all your replies, guys. This is a huge problem for me right
    now...
     
    Sandman, May 24, 2013
    #10
  11. Sandman

    Sandman Guest

    85.24.167.69 MY_IP - [24/May/2013:14:24:33 +0200] "GET /wpad.dat HTTP/1.1" 200 70 "-" "-"
    83.233.16.50 MY_IP - [24/May/2013:14:24:33 +0200] "GET /wpad.dat HTTP/1.1" 200 70 "-" "-"
    88.83.39.117 cluster.mydomain.com - [24/May/2013:14:24:34 +0200] "GET /wpad.dat HTTP/1.1" 200 70 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Win64; Trident/6.0)"
    88.83.39.117 cluster.mydomain.com - [24/May/2013:14:24:34 +0200] "GET /wpad.dat HTTP/1.1" 200 70 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Win64; Trident/6.0)"
    88.83.39.117 cluster.mydomain.com - [24/May/2013:14:24:34 +0200] "GET /wpad.dat HTTP/1.1" 200 70 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Win64; Trident/6.0)"

    Above are five sample rows (from thousands) on the server.

    The log format I'm using is putting the vhost in the second column, so you see that the two first requests are to my IP, and the second three are to cluster.mydomain.com <- The hostname where my clients point their CNAME subdomains

    The ones that access the IP has no agent string (consistently) and the ones that access cluster.mydomain.com does.

    I also have these:

    46.59.81.183 wpad - [24/May/2013:14:24:34 +0200] "GET /wpad.dat HTTP/1.1" 200 70 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.94 Safari/537.36"


    I.e. accesses to the vhost "wpad", which is odd - becuase that's not the name of a vhost, nor is it a qualified domain name unless you're on a local network (right?) Is that a clue that some network out there thinks I am part of their local network?

    And, as you can see - one IP above made three identical requests (which it got a 200 reponse to) three times in one second. I have more lika that with fvive, six or seven conescutive times and so on.


    How do I block this?? :)
     
    Sandman, May 24, 2013
    #11
  12. All systems shipped these days are configured to search for a
    automatic proxy configuration from "http://wpad/wpad.dat"

    Because of your use of wildcard DNS *everyone* who starts their
    webbrowser will find wpad.their.domain then look for it on
    your webserver.

    Your ip filtering rules block your customers; it also makes their
    web experience interesting (it will take some time before the system
    figures out that there is no wpad.dat and will then connect to
    the internet directly)
    Well, you shouldn't have done that.
    But you're serving their domains too, right?

    Casper
     
    Casper H.S. Dik, May 24, 2013
    #12
  13. Sandman

    Sandman Guest

    All systems shipped these days are configured to search for a
    automatic proxy configuration from "http://wpad/wpad.dat"[/QUOTE]

    All? I thought it was Windows thing.
    Why? I mean - when they go to wpad.their.domain, why would they end up
    with the IP of my server, or the CNAME of cluster.mydomain.com

    That's the part I just can't understand.
    But the people surfing to my web server wouldn't be asking my server,
    any more than microsoft.com, for information about their own networks
    proxy settings, surely?
    Fair enough, but I still don't know how that messed this up. I just
    can't wrap my head around it.
    No. Only my own domains. Their IT managers have set up their subdomains
    (i.e. www.) to point to cluster.mydomain.com which points to my IP
     
    Sandman, May 24, 2013
    #13
  14. Perhaps quoting some of the domain names involved would clarify matters.
     
    Richard Kettlewell, May 24, 2013
    #14
  15. Sandman

    Sandman Guest

    Wildcard DNS is asking for issues unless you fully understand all the
    ramifications. Best not to use it unless you really really need it and
    fully understand it.[/QUOTE]

    Fair enough.
    Yes, that's how wildcards works - but not only do I not understand why
    thousands of hosts from all over the swedish internet would start to
    request wpad.* on my server, some of the up to thirty times per second -
    per host!

    I am also not hosting any of their domains, so why would would they ever
    come to me to ask for this?
    I did that yesterday, didn't change a single thing... :(
     
    Sandman, May 24, 2013
    #15
  16. Sandman

    Sandman Guest

    Perhaps quoting some of the domain names involved would clarify matters.[/QUOTE]

    Yeah, ok.

    So a client to me, for example http://www.stadsnat.se has their DNS set
    up as such:
    www.stadsnat.se is an alias for cluster.atlascms.se.
    cluster.atlascms.se has address 94.247.170.170

    Now, atlascms.se WAS a wildcard DNS, but isn't any longer.

    Even so, the requests I get look largely like this:

    94.254.41.78 cluster.atlascms.se - [24/May/2013:16:25:24 +0200] "GET
    /wpad.dat HTTP/1.1" 200 70 "-" "Mozilla/5.0 (compatible; MSIE 10.0;
    Win32; Trident/6.0)"

    I.e. a request to that domain name, not to a wpad subdomain. So the
    wildcard DNS thing doesn't seem to even apply... Or am I mistaken?
     
    Sandman, May 24, 2013
    #16
  17. I agree; I think the wildcard was probably a red herring, and the longer
    it gets since you removed it, the more certain that is (although it’s
    worth remembering that not all DNS clients honor TTLs correctly).

    The request doesn’t seem consistent with the way that wpad searching is
    described as working, but of course it may be that there’s more to the
    implementation that the various descriptions online imply.

    Do you have any idea how many distinct addresses are involved? Are they
    in fact _all_ Swedish IP addresses or are any of them from further
    afield? Can you tell whether any are associated with any of your
    customers (e.g. if you keep logs of where they upload from, do any of
    the oddly behaving clients appear there)?

    Have you recently annoyed anyone who might have sufficiently poor
    judgement to launch a DDoS attack?
     
    Richard Kettlewell, May 24, 2013
    #17
  18. So when someone looked up wpad.stadsnet.se it was mapped
    to cluster.atlascms.se? That, I think, is the root
    of your problem.
    I wouldn't be too sure about that. Note that wpad/wpad.dat is looked
    with a different algorithm then ordinary websites because it needs
    to sidestep the proxies and such.

    Casper
     
    Casper H.S. Dik, May 24, 2013
    #18
  19. Sandman

    Sandman Guest

    So when someone looked up wpad.stadsnet.se it was mapped
    to cluster.atlascms.se? That, I think, is the root
    of your problem.[/QUOTE]

    No, that's the thing - "stadsnat.se" is not a domain I am
    administering. It's one of my clients domains. They wouldn't wildcard
    DNS and send ALL requests to me - only web requests (so www would
    point to me).

    I only know of one client that has wildcard:ed their DNS to me,
    actually. Maybe I should tell them to exempt wpad...
     
    Sandman, May 24, 2013
    #19
  20. Sandman

    Sandman Guest

    I agree; I think the wildcard was probably a red herring, and the longer
    it gets since you removed it, the more certain that is (although it’s
    worth remembering that not all DNS clients honor TTLs correctly).
    Indeed.

    The request doesn’t seem consistent with the way that wpad searching is
    described as working, but of course it may be that there’s more to the
    implementation that the various descriptions online imply.[/QUOTE]

    Or this seemingly benign request is used to stage a flood attack
    against me or my clients. Since google can't find any more serious
    attacks, especially not current one (there is that one forum post), I
    am starting to wonder why this is.
    I now have a cronjob that reads the access_log file for wpad.dat
    requests and then add them to a blacklist and to iptables. It has been
    in effect for maybe two hours and the list is 4000 IP's long. 4000
    seemingly normal swedish IP's from normal swedish ISP's. All
    bombarding me with millions of wpad.dat requests.

    Some IP's send 30-40 requests per second in a burst.

    With 4000 in two hours, I'm guessing that tomorrow morning it will be
    over 10000, and then using iptables becomes increasingly stupid.
    I have made samples now and then - all have been swedish IP's
    according to various online ip -> location functions.
    Even so, I don't have anywhere near to 4000 customers so this can't be
    due to one of my clients faulty network either. This seems like a
    targeted attack.
    I can think of only one person (from here on usenet) but he's from
    America and I doubt he has the ability to muster a botnet of
    Swedish-only clients. He has tried to flood me before, but only from a
    single IP. So no, I have to answer that I know of no one that could do
    this specifically against *me*. Maybe against one of my clients?

    Because, if they were targetting me, they would target my homepage
    (sandman.net) or some other, these attacks seem to either target the
    IP or my cluster domain name - and the cluster domain is not something
    used for anything but DNS redirection.
     
    Sandman, May 24, 2013
    #20
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.