wpad.dat attack on Linux Apache server

I don’t currently have a better theory.

From what you’ve said (and I may be wrong) it sounds like it could be
targetting your source of income. The ability to run a botnet
personally isn’t necessarily relevant, even if you’re right about that;
botnet operators rent them out.

 

If stadsnat.se is an isp, and stadsnat.se is a cname for your system,
and their customers get dhcp addresses of the form ipaddr.stadsnat.se,
then all of their customers will be going to stadsnat.se/yourssystem,
to look for the wpad info.

Workaround would require stadsnat.se to point to one of their own
servers, and have www.stadsnat.se be the cname to your system, with
there server redirecting stadsnat.se to www.stadsnat.se.

By blacklisting the ip addresses, the website for stadsnat.se will
become unreachable by their customers.

Regards, Dave Hodgins

 

That wouldn’t explain why the Host: header is cluster.atlascms.se (you
can’t reach that name using rdns, for instance).

 

....I wonder if it really is, or if the logfile fragments posted so far
are misleading. Sandman, can you capture an example request off the
wire, e.g. with:

tcpdump -nX port 80 and host <one of the problem IPs>

 

From what you’ve said (and I may be wrong) it sounds like it could be
targetting your source of income. The ability to run a botnet
personally isn’t necessarily relevant, even if you’re right about that;
botnet operators rent them out.[/QUOTE]

I didn't know that

Thanks for your comments, it's a possible scenario I suppose.

 

Did you read this article?

<https://nodpi.ORG/2013/05/09/wpad-the-internet-explorer-security-flaw-that-ex
poses-all-microsoft-users-in-the-uk/>[/QUOTE]

Not that one specifically, but I have read about the point he is
making.

Which is due to UK citizens usually having a two part top level domain
name. WPAD see's the domain "domain.co.uk" and thinks (correctly, one
might add) that "domain" is a subdomain to "co" which is the local
domain to the top level "uk". Which means that "wpad.co.uk" is a
logical assumption for this function.

This is not relevant to atlascms.se or any of swedish domains, really
(we did have the pp.se domain thing for private person, but that was a
decade ago).

The only one of my clients that have wildcard:ed their domain to me is
opennet.se, I'll have a talk to them. As far as I know, they don't
have almost 5000 users on their LAN though.

because even if what you postulate is a possible scenario, it would
under no circumstances generate hundreds of thousands of requests -
sometimes 30-40 per second from one single host. That's where the
entire "misconfigured DNS" idea falls slightly apart, don't you agree?

I mean, if I came here wondering about these wpad.dat requests I see
now and then, then that would be a logical question. But I get about
20-30 requests per second, every second. That just can't be due to a
misconfigured wildcard DNS.

Or do you think I am jumping to conclusions?

 

If stadsnat.se is an isp[/QUOTE]

They aren't actually. The domain is owned by a client of miune, the
citynet of Örebro, a swedish town. They use "www.stadsnat.se" to
promote their citynet. None of my clients are ISP's...

Just to clarify - "stadsnat.se" can never be a CNAME, the top domain
can only be pointed with an A record to an IP number. Their "www"
subdomain is, however, CNAME:d to my "cluster.atlascms.se".

Yes, all of what you say is true - if they would have been an ISP

Which, coincidentally, is exactly how it is set up right now

Yes, this has been a concern for me. I did some checking, and of all
the hundreds and thousands of IP addresses that are flooding me, I
couldn't find any that was at the same time surfing the apache server
normally.

So, these IP's seem to be normal user IP's and I am actively blocking
almost 5000 of them by now, which must include some that sooner or
later will want to surf one of the many web sites hosted by this
machine.

Blocking them in the firewall is just a measure I'm taking in order to
make my web server responsive again. It's hardly a long-term solution
(where would it stop? 10k? 100k?)

 

Certainly. I'll readily admit to not being sure whether this tells me
anything or not. It appears to be a normal (?) ACK, followed by the
HTTP request, with the host set to the IP number of the machine. and
then followed by my machine giving them the wpad.dat file as I have
set it up currently.

21:27:29.055469 IP 83.172.125.62.65391 > 94.247.170.170.80: Flags [.],
ack 1545, win 16232, length 0
0x0000: 4500 0028 2866 4000 7606 01de 53ac 7d3e E..(([email protected]}>
0x0010: 5ef7 aaaa ff6f 0050 5c93 2f59 a06e 3313 ^....o.P\./Y.n3.
0x0020: 5010 3f68 36b2 0000 0000 0000 0000 P.?h6.........
21:27:29.905887 IP 83.172.125.62.65087 > 94.247.170.170.80: Flags
[P.], seq 340:425, ack 1545, win 16328, length 85
0x0000: 4500 007d 2877 4000 7606 0178 53ac 7d3e E..}([email protected]}>
0x0010: 5ef7 aaaa fe3f 0050 9103 ca53 e213 d0bc ^....?.P...S....
0x0020: 5018 3fc8 e032 0000 4745 5420 2f77 7061 P.?..2..GET./wpa
0x0030: 642e 6461 7420 4854 5450 2f31 2e31 0d0a d.dat.HTTP/1.1..
0x0040: 436f 6e6e 6563 7469 6f6e 3a20 4b65 6570 Connection:.Keep
0x0050: 2d41 6c69 7665 0d0a 4163 6365 7074 3a20 -Alive..Accept:.
0x0060: 2a2f 2a0d 0a48 6f73 743a 2039 342e 3234 */*..Host:.94.24
0x0070: 372e 3137 302e 3137 300d 0a0d 0a 7.170.170....
21:27:29.906203 IP 94.247.170.170.80 > 83.172.125.62.65087: Flags
[P.], seq 1545:1931, ack 425, win 63, length 386
0x0000: 4500 01aa c4f4 4000 4006 99cd 5ef7 aaaa [email protected]@...^...
0x0010: 53ac 7d3e 0050 fe3f e213 d0bc 9103 caa8 S.}>.P.?........
0x0020: 5018 003f dc28 0000 4854 5450 2f31 2e31 P..?.(..HTTP/1.1
0x0030: 2032 3030 204f 4b0d 0a44 6174 653a 2046 .200.OK..Date:.F
0x0040: 7269 2c20 3234 204d 6179 2032 3031 3320 ri,.24.May.2013.
0x0050: 3139 3a32 373a 3239 2047 4d54 0d0a 5365 19:27:29.GMT..Se
0x0060: 7276 6572 3a20 4170 6163 6865 2f32 2e32 rver:.Apache/2.2
0x0070: 2e31 3620 2844 6562 6961 6e29 0d0a 4c61 .16.(Debian)..La
0x0080: 7374 2d4d 6f64 6966 6965 643a 2054 6875 st-Modified:.Thu
0x0090: 2c20 3233 204d 6179 2032 3031 3320 3231 ,.23.May.2013.21
0x00a0: 3a34 313a 3037 2047 4d54 0d0a 4554 6167 :41:07.GMT..ETag
0x00b0: 3a20 2233 3135 3430 3039 2d34 362d 3464 :."3154009-46-4d
0x00c0: 6436 3938 6133 3665 3263 3022 0d0a 4163 d698a36e2c0"..Ac
0x00d0: 6365 7074 2d52 616e 6765 733a 2062 7974 cept-Ranges:.byt
0x00e0: 6573 0d0a 436f 6e74 656e 742d 4c65 6e67 es..Content-Leng
0x00f0: 7468 3a20 3730 0d0a 4b65 6570 2d41 6c69 th:.70..Keep-Ali
0x0100: 7665 3a20 7469 6d65 6f75 743d 3135 2c20 ve:.timeout=15,.
0x0110: 6d61 783d 3335 380d 0a43 6f6e 6e65 6374 max=358..Connect
0x0120: 696f 6e3a 204b 6565 702d 416c 6976 650d ion:.Keep-Alive.
0x0130: 0a43 6f6e 7465 6e74 2d54 7970 653a 2061 .Content-Type:.a
0x0140: 7070 6c69 6361 7469 6f6e 2f78 2d6e 732d pplication/x-ns-
0x0150: 7072 6f78 792d 6175 746f 636f 6e66 6967 proxy-autoconfig
0x0160: 0d0a 0d0a 6675 6e63 7469 6f6e 2046 696e ....function.Fin
0x0170: 6450 726f 7879 466f 7255 524c 2875 726c dProxyForURL(url
0x0180: 2c20 686f 7374 2920 7b20 7265 7475 726e ,.host).{.return
0x0190: 2022 5052 4f58 5920 3132 372e 302e 302e ."PROXY.127.0.0.
0x01a0: 313a 3434 3522 3b20 7d0a 1:445";.}.




And here is the request from a host I specifically looked where
logging as the "cluster.atlascms.se" vhost:

1:30:59.331074 IP 85.24.180.196.60901 > 94.247.170.170.80: Flags [P.],
seq 2476116866:2476116951, ack 4035600534, win 16425, length 85
0x0000: 4500 007d 05e3 4000 7806 e919 5518 b4c4 E..}[email protected]
0x0010: 5ef7 aaaa ede5 0050 9396 8b82 f08a 6096 ^......P......`.
0x0020: 5018 4029 5527 0000 4745 5420 2f77 7061 [email protected])U'..GET./wpa
0x0030: 642e 6461 7420 4854 5450 2f31 2e31 0d0a d.dat.HTTP/1.1..
0x0040: 436f 6e6e 6563 7469 6f6e 3a20 4b65 6570 Connection:.Keep
0x0050: 2d41 6c69 7665 0d0a 4163 6365 7074 3a20 -Alive..Accept:.
0x0060: 2a2f 2a0d 0a48 6f73 743a 2039 342e 3234 */*..Host:.94.24
0x0070: 372e 3137 302e 3137 300d 0a0d 0a 7.170.170....

As you can see, it also requests the IP as host.

I hope this shows what you wanted to see.


I wanted to thank everyone that has participated, all comments and
theories have been very welcome!

 

It should be in your webserver log. Here's an example from mine:

192.168.130.16 - - [24/May/2013:23:40:02 +0100] "GET /proxy.pac HTTP/1.1" 200 1485 "-" "-"

and here's the log definition line for this vHost:

CustomLog "|/usr/bin/cronolog /home/www/wpad/logs/%Y/%m/%d/public-access.log" combined

If you're going to block by firewall rule I'd suggest you take a close
look at fail2ban, which dos this kind of process very well indeed.

For comparison, I've had over 4000 hits in my access log today. And this
is from a fairly lightly loaded network with just a few PCs and
servers. Windows 7 appears to be *very* noisy indeed (I think I might
need to investigate the document expiry time to see if I can persuade
W7 to cache the answer a little more often).

Chris

 

That’s consistent with the IE6 behavior (as described by Wikipedia)
which might have been copied by other software. Unfortunately, that
means we still don’t know for sure what hostname the client thought it
was connecting to.

 

$ host wpad.opennet.se
wpad.opennet.se is an alias for cluster.atlascms.se.
cluster.atlascms.se has address 94.247.170.170

Looks like a smoking gun to me...

More then one thing can be broken at once. In this case, the easiest
thing to do is to stop wpad.opennet.se pointing at you. If that fixes
it, then it’s time to speculate about why so many IP clients were
involved. If it doesn’t, move on to the next theory. Checking for a
‘wpad’ subdomain for each of your customer domains would seem like the
logical next step.

 

$ host wpad.opennet.se
wpad.opennet.se is an alias for cluster.atlascms.se.
cluster.atlascms.se has address 94.247.170.170

Looks like a smoking gun to me...[/QUOTE]

Yes, but they are still no ISP, and all requests comes from swedish
ISP's (like bahnhof.se, bredband2.se and such) and wpad.bahnhof.se
doesn't point to me.

Yes, I have found none. I keep blocking them (over night, they didn't
grow to more than about 5000 actually, I'm at 5163 right now)

 

Like I said in an earlier reply, Opennet is not an ISP they are a
communication operator, which means they own citynets, so they don't
have end customers on their opennet.se host.

But looking at the domain names of all the hosts that I am currently
blocking (5000+), there are some references to opennet, like
<random>.opennet.bredband2.se" or "<random>.karlstad.bredband2.se" anD
Karlstad is a Opennet city.

So yes, this may very well be a smoking gun as you say.

 

No, that's the thing - "stadsnat.se" is not a domain I am
administering. It's one of my clients domains. They wouldn't wildcard
DNS and send ALL requests to me - only web requests (so www would
point to me).
actually. Maybe I should tell them to exempt wpad...[/QUOTE]

[...]

But

$ host stadsnet.se
stadsnet.se has address 46.30.211.54
stadsnet.se mail is handled by 10 mx-cluster-a1.one.com.
stadsnet.se mail is handled by 10 mx-cluster-a2.one.com.

$ host wpad.stadsnet.se
wpad.stadsnet.se has address 46.30.211.54

$ host qwerty.stadsnet.se
qwerty.stadsnet.se has address 46.30.211.54

.... looks like a wildcard setting somewhere, to me.

 

It's stadsnAt.se, not stadsnEt.se.
^ ^

 

Only, you accidentally looked up "stadsnet" instead of "stadsnat"

stadsnat.se has address 94.247.170.170

That's my IP

www.stadsnat.se is an alias for cluster.atlascms.se.
cluster.atlascms.se has address 94.247.170.170

Host wpad.stadsnat.se not found: 3(NXDOMAIN)


And again - stadsnat.se is NOT an ISP.

 

Right; by filtering wpad queries from all these clients also is a
denial of service attack on all those clients, especially if you
drop the packet but don't give them an answer.

Casper

 

Sounds like a DOS nightmare. I get lots of DOS attacks. I use a fortigate 50B firewall to protect web servers. It runs on transparent mode in our network. The bottle neck of the firewall is the antivirus scanner which I disable when cpu goes above 60%. The fortigate has a profile called protect webserver which does the trick.
Also I have a script in cron that fetches a page and checks the speed of the page loading. something like time lynx --dump http://page.test/, When thetime is long I restart apache.