Networking Forums

Networking Forums > Computer Networking > Linux Networking > Handling a large number of IPs efficiently

Reply
Thread Tools Display Modes

Handling a large number of IPs efficiently

 
 
Bill Davidsen
Guest
Posts: n/a

 
      12-27-2003, 02:45 PM
I have a site which is going to want to allow access from certain site
only. A large number of sites, and not neatly grouped in CIDR blocks. A
few thousand IP addresses, in fact. Unfortunately iptables doesn't seem
to do this well, using a linear progression rather than a hash or similar.

I think I can do this using netlink socket, but before I jump in with
what looks to be an ugly solution, is there a better way I've missed? Or
a nice example of doing something similar with netlink?

--
bill davidsen <(E-Mail Removed)>
CTO TMR Associates, Inc
Doing interesting things with small computers since 1979
 
Reply With Quote
 
 
 
 
jack
Guest
Posts: n/a

 
      12-27-2003, 04:50 PM
Bill Davidsen wrote:
> I have a site which is going to want to allow access from certain site
> only. A large number of sites, and not neatly grouped in CIDR blocks. A
> few thousand IP addresses, in fact. Unfortunately iptables doesn't seem
> to do this well, using a linear progression rather than a hash or similar.


Not necessarily. In Your case, I'd simply add more chains like so:

Create a chain for all "leftmost" network addresses that are in
question, e. g. one chain "D9" for all clients where the IP address is
in 217.0.0.0/8.

In that chain "D9", create new chains the same way, e. g. a chain "D954"
for all clients from 217.84.0.0/16 and so on.

I don't know whether this is practical to You or whether it is practical
at all, but this was my first idea when I read Your term "linear",
above.


Cheers, Jack.

--
----------------------------------------------------------------------
My personal reading of the string "MicroSoft" expands to "NanoWeak"...

 
Reply With Quote
 
Bill Davidsen
Guest
Posts: n/a

 
      12-28-2003, 01:59 AM
jack wrote:
> Bill Davidsen wrote:
>
>> I have a site which is going to want to allow access from certain site
>> only. A large number of sites, and not neatly grouped in CIDR blocks.
>> A few thousand IP addresses, in fact. Unfortunately iptables doesn't
>> seem to do this well, using a linear progression rather than a hash or
>> similar.

>
>
> Not necessarily. In Your case, I'd simply add more chains like so:
>
> Create a chain for all "leftmost" network addresses that are in
> question, e. g. one chain "D9" for all clients where the IP address is
> in 217.0.0.0/8.
>
> In that chain "D9", create new chains the same way, e. g. a chain "D954"
> for all clients from 217.84.0.0/16 and so on.
>
> I don't know whether this is practical to You or whether it is practical
> at all, but this was my first idea when I read Your term "linear",
> above.


I looked at that, but I don't think it's going to be a solution. The IPs
are moderately well split, so I have about 20-40 values for the first
octet on a given day, same for 2nd octet, then up to 80 3rd octet values
for some 1st+2nd combinations. A few thousand is typical, but it can hit
10k at times, and I need to process 30-100 packets/sec for validation.
All-in-all I don't think the tree scales all that well.

Of course I could do it a bit at a time, giving max 64 rules per packet,
but I think that doesn't make it, either. Not to mention that adding and
deleting rules can happen at a high rate.

What I need is a fairly good hash or other database attack on the
problem, I'm afraid.

Thanks for the idea, but these IPs are being used for a DDoS and they
are identified and blocked in large numbers at times.

--
bill davidsen <(E-Mail Removed)>
CTO TMR Associates, Inc
Doing interesting things with small computers since 1979
 
Reply With Quote
 
Leon.
Guest
Posts: n/a

 
      12-28-2003, 06:19 AM

"Bill Davidsen" <(E-Mail Removed)> wrote in message
news:RdadneK5x8hG23OiRVn-(E-Mail Removed)...
> jack wrote:
> > Bill Davidsen wrote:
> >
> >> I have a site which is going to want to allow access from certain site
> >> only. A large number of sites, and not neatly grouped in CIDR blocks.
> >> A few thousand IP addresses, in fact. Unfortunately iptables doesn't
> >> seem to do this well, using a linear progression rather than a hash or
> >> similar.

> >
> >
> > Not necessarily. In Your case, I'd simply add more chains like so:
> >
> > Create a chain for all "leftmost" network addresses that are in
> > question, e. g. one chain "D9" for all clients where the IP address is
> > in 217.0.0.0/8.
> >
> > In that chain "D9", create new chains the same way, e. g. a chain "D954"
> > for all clients from 217.84.0.0/16 and so on.
> >
> > I don't know whether this is practical to You or whether it is practical
> > at all, but this was my first idea when I read Your term "linear",
> > above.

>
> I looked at that, but I don't think it's going to be a solution. The IPs
> are moderately well split, so I have about 20-40 values for the first
> octet on a given day, same for 2nd octet, then up to 80 3rd octet values
> for some 1st+2nd combinations. A few thousand is typical, but it can hit
> 10k at times, and I need to process 30-100 packets/sec for validation.
> All-in-all I don't think the tree scales all that well.
>
> Of course I could do it a bit at a time, giving max 64 rules per packet,
> but I think that doesn't make it, either. Not to mention that adding and
> deleting rules can happen at a high rate.
>
> What I need is a fairly good hash or other database attack on the
> problem, I'm afraid.
>
> Thanks for the idea, but these IPs are being used for a DDoS and they
> are identified and blocked in large numbers at times.


Yes hashing just on the last 8 bits this would reduce the table rules to
search through from 10k does down to
10k/254 + 8 = 40 + 8 = 48 , at the worst. thats quite a saving.

If you make the hash structure bigger, then you could do it on the last 12
bits, reducing the worst case down to about 10k/1024 + 12 =22 As thats
about 50 % in the last table and 50% in the hash tables, theres no point in
going any further ?






Define a table called 'hash' (or use the main table) , and add the rules
like this
if matches 255.255.255.0/255.255.255.128 goto table a
if matches 255.255.255.128/255.255.255.128 goto table b


Table a
if matches 255.255.255.0/255.255.255.192 goto table c
if matches 2555.255.255.64/255.255.255.192 goto table d

table b
if matches 255.255.255.128/255.255.255.192 goto table g
if matches 255.255.255.192/255.255.255.192 goto table h

Table C
if matches 255.255.255.0/255.255.255.224 goto table e
if matches 255.255.255.32/255.255.255.224 goto table f

etc






 
Reply With Quote
 
 
 
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
64-bit XP OS uses memory more efficiently than 32-bit? Spin Windows Networking 5 03-06-2008 06:56 PM
Handling 2 broadbandconnections on Linux s Linux Networking 0 10-06-2006 06:46 AM
Domain name with one large number (not IP address) Seb Broadband 7 08-29-2006 08:52 PM
How can transfer large number of files across the internet? Adrian Windows Networking 5 02-13-2006 04:38 PM
TCP FIN handling in linux kernel patrick.mcgleenon@googlemail.com Linux Networking 3 11-15-2005 12:21 AM



1 2 3 4 5 6 7 8 9 10 11