Question on single host with mutiple subnets/gateways

Hi,

My webserver connects to 3 different subnets. I would like to control the
reply traffic not to use the single default gateway but the one in the
incoming subnet. The current config is like:

subnets: subnet(1) subnet(2) subnet(3)
gateways: gateway(1) gateway(2) gateway(3)
NICs: NIC(1) NIC(2) NIC(3)

default gateway: gateway(1)
Apache server will respond all request from any NIC. but the reply traffic
will only goto NIC(1)->gateway(1)
what I'd like to do is:

request from NIC(1)->apache or other services->NIC(1)->gateway(1)
request from NIC(2)->apache or other services->NIC(2)->gateway(2)
request from NIC(3)->apache or other services->NIC(3)->gateway(3)
request from localhost such as SMTP go to default gateway

the OS is Redhat 9.0, but if it is possible, I'd like to implement to Suse
7.2 also.

Please help, thank you.

Stanoo Chang
stanoo[at]stanoo[dot]com

On Fri, 26 Dec 2003 03:37:58 +0800, Stanoo Chang <(E-Mail Removed)> wrote:
> Hi,
>
> My webserver connects to 3 different subnets. I would like to control the
> reply traffic not to use the single default gateway but the one in the
> incoming subnet. The current config is like:
>
> subnets: subnet(1) subnet(2) subnet(3)
> gateways: gateway(1) gateway(2) gateway(3)
> NICs: NIC(1) NIC(2) NIC(3)
>
> default gateway: gateway(1)
> Apache server will respond all request from any NIC. but the reply traffic
> will only goto NIC(1)->gateway(1)
> what I'd like to do is:
>
> request from NIC(1)->apache or other services->NIC(1)->gateway(1)
> request from NIC(2)->apache or other services->NIC(2)->gateway(2)
> request from NIC(3)->apache or other services->NIC(3)->gateway(3)
> request from localhost such as SMTP go to default gateway

Sounds like your routing is incorrect, but impossible to tell without
real data. If you have NIC(1), NIC(2) and NIC(3) configured with proper
netmask and network routes, traffic should not go to default gateway
unless nothing else matches. In other words, replies to traffic from
NIC(2) should go out NIC(2), likewise from/to NIC(3). If not, you may
have conflicting networks/netmasks.

> the OS is Redhat 9.0, but if it is possible, I'd like to implement to Suse
> 7.2 also.
>
> Please help, thank you.
>
> Stanoo Chang
> stanoo[at]stanoo[dot]com

--
David Efflandt - All spam ignored http://www.de-srv.com/

"David Efflandt" <(E-Mail Removed)>
:(E-Mail Removed)...
> On Fri, 26 Dec 2003 03:37:58 +0800, Stanoo Chang <(E-Mail Removed)>
wrote:
> > Hi,
> >
> > My webserver connects to 3 different subnets. I would like to control
the
> > reply traffic not to use the single default gateway but the one in the
> > incoming subnet. The current config is like:
> >
> > subnets: subnet(1) subnet(2) subnet(3)
> > gateways: gateway(1) gateway(2) gateway(3)
> > NICs: NIC(1) NIC(2) NIC(3)
> >
> > default gateway: gateway(1)
> > Apache server will respond all request from any NIC. but the reply
traffic
> > will only goto NIC(1)->gateway(1)
> > what I'd like to do is:
> >
> > request from NIC(1)->apache or other services->NIC(1)->gateway(1)
> > request from NIC(2)->apache or other services->NIC(2)->gateway(2)
> > request from NIC(3)->apache or other services->NIC(3)->gateway(3)
> > request from localhost such as SMTP go to default gateway
>
> Sounds like your routing is incorrect, but impossible to tell without
> real data. If you have NIC(1), NIC(2) and NIC(3) configured with proper
> netmask and network routes, traffic should not go to default gateway
> unless nothing else matches. In other words, replies to traffic from
> NIC(2) should go out NIC(2), likewise from/to NIC(3). If not, you may
> have conflicting networks/netmasks.
>

Sorry for my unclear description, here's the current routing table.
It only use static setup, and all three subnets go to same router for
Internet Connection.
User will request data by connecting to different INCs at the same period of
time.

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
203.65.221.200 * 255.255.255.248 U 0 0 0 eth0
203.65.221.192 * 255.255.255.248 U 0 0 0 eth1
203.65.221.184 * 255.255.255.248 U 0 0 0 eth2
127.0.0.0 * 255.0.0.0 U 0 0 0 lo
default 203.65.221.206 0.0.0.0 UG 0 0 0 eth0

should I use any dynamic routing protocol?

Thanks

Stanoo Chang

On Fri, 26 Dec 2003 13:43:57 +0800, SC <(E-Mail Removed)> wrote:
> "David Efflandt" <(E-Mail Removed)>
>:(E-Mail Removed)...
>> On Fri, 26 Dec 2003 03:37:58 +0800, Stanoo Chang <(E-Mail Removed)>
> wrote:
>> > Hi,
>> >
>> > My webserver connects to 3 different subnets. I would like to control
> the
>> > reply traffic not to use the single default gateway but the one in the
>> > incoming subnet. The current config is like:
>> >
>> > subnets: subnet(1) subnet(2) subnet(3)
>> > gateways: gateway(1) gateway(2) gateway(3)
>> > NICs: NIC(1) NIC(2) NIC(3)
>> >
>> > default gateway: gateway(1)
>> > Apache server will respond all request from any NIC. but the reply
> traffic
>> > will only goto NIC(1)->gateway(1)
>> > what I'd like to do is:
>> >
>> > request from NIC(1)->apache or other services->NIC(1)->gateway(1)
>> > request from NIC(2)->apache or other services->NIC(2)->gateway(2)
>> > request from NIC(3)->apache or other services->NIC(3)->gateway(3)
>> > request from localhost such as SMTP go to default gateway
>>
>> Sounds like your routing is incorrect, but impossible to tell without
>> real data. If you have NIC(1), NIC(2) and NIC(3) configured with proper
>> netmask and network routes, traffic should not go to default gateway
>> unless nothing else matches. In other words, replies to traffic from
>> NIC(2) should go out NIC(2), likewise from/to NIC(3). If not, you may
>> have conflicting networks/netmasks.
>>
>
> Sorry for my unclear description, here's the current routing table.
> It only use static setup, and all three subnets go to same router for
> Internet Connection.
> User will request data by connecting to different INCs at the same period of
> time.
>
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref Use
> Iface
> 203.65.221.200 * 255.255.255.248 U 0 0 0 eth0
> 203.65.221.192 * 255.255.255.248 U 0 0 0 eth1
> 203.65.221.184 * 255.255.255.248 U 0 0 0 eth2
> 127.0.0.0 * 255.0.0.0 U 0 0 0 lo
> default 203.65.221.206 0.0.0.0 UG 0 0 0 eth0
>
> should I use any dynamic routing protocol?

This looks correct "if these are separate networks", ip_forward is
enabled, and netmask/gateway is correct on all local boxes. It would
locally route 203.65.221. IPs ending with 201-206 out eth0, 193-199 out
eth1, and 185-190 out eth2. Anything other IPs would be routed to default
gateway.

However, any local boxes on eth0 would need -net routes for
203.65.221.192/255.255.255.248 and 203.65.221.184/255.255.255.248 using
your eth0 IP as gateway to reach your other local networks. If they
only have a 203.65.221.206 default gateway, then traffic for your other
local networks would go in wrong direction.

Routing example for _other_ box on eth0 assuming your Linux IP ends 201:

Destination Gateway Genmask Flags Metric Ref Use Iface
203.65.221.200 * 255.255.255.248 U 0 0 0 eth0
203.65.221.192 203.65.221.201 255.255.255.248 G 0 0 0 eth0
203.65.221.184 203.65.221.201 255.255.255.248 G 0 0 0 eth0
127.0.0.0 * 255.0.0.0 U 0 0 0 lo
default 203.65.221.206 0.0.0.0 UG 0 0 0 eth0

Not sure what order these routes would appear or Flags.

Boxes on eth1 or eth2 would not need those extra routes, just their local
network route and default gateway to Linux IP facing them.

--
David Efflandt - All spam ignored http://www.de-srv.com/

"David Efflandt" <(E-Mail Removed)> ¦b¶l¥ó
news:(E-Mail Removed) ¤¤¼¶¼g...
> On Fri, 26 Dec 2003 13:43:57 +0800, SC <(E-Mail Removed)> wrote:
> > "David Efflandt" <(E-Mail Removed)>
> >:(E-Mail Removed)...
> >> On Fri, 26 Dec 2003 03:37:58 +0800, Stanoo Chang <(E-Mail Removed)>
> > wrote:
> >> > Hi,
> >> >
> >> > My webserver connects to 3 different subnets. I would like to control
> > the
> >> > reply traffic not to use the single default gateway but the one in
the
> >> > incoming subnet. The current config is like:
> >> >
> >> > subnets: subnet(1) subnet(2) subnet(3)
> >> > gateways: gateway(1) gateway(2) gateway(3)
> >> > NICs: NIC(1) NIC(2) NIC(3)
> >> >
> >> > default gateway: gateway(1)
> >> > Apache server will respond all request from any NIC. but the reply
> > traffic
> >> > will only goto NIC(1)->gateway(1)
> >> > what I'd like to do is:
> >> >
> >> > request from NIC(1)->apache or other services->NIC(1)->gateway(1)
> >> > request from NIC(2)->apache or other services->NIC(2)->gateway(2)
> >> > request from NIC(3)->apache or other services->NIC(3)->gateway(3)
> >> > request from localhost such as SMTP go to default gateway
> >>
> >> Sounds like your routing is incorrect, but impossible to tell without
> >> real data. If you have NIC(1), NIC(2) and NIC(3) configured with
proper
> >> netmask and network routes, traffic should not go to default gateway
> >> unless nothing else matches. In other words, replies to traffic from
> >> NIC(2) should go out NIC(2), likewise from/to NIC(3). If not, you may
> >> have conflicting networks/netmasks.
> >>
> >
> > Sorry for my unclear description, here's the current routing table.
> > It only use static setup, and all three subnets go to same router for
> > Internet Connection.
> > User will request data by connecting to different INCs at the same
period of
> > time.
> >
> > Kernel IP routing table
> > Destination Gateway Genmask Flags Metric Ref Use
> > Iface
> > 203.65.221.200 * 255.255.255.248 U 0 0 0
eth0
> > 203.65.221.192 * 255.255.255.248 U 0 0 0
eth1
> > 203.65.221.184 * 255.255.255.248 U 0 0 0
eth2
> > 127.0.0.0 * 255.0.0.0 U 0 0 0
lo
> > default 203.65.221.206 0.0.0.0 UG 0 0 0
eth0
> >
> > should I use any dynamic routing protocol?
>
> This looks correct "if these are separate networks", ip_forward is
> enabled, and netmask/gateway is correct on all local boxes. It would
> locally route 203.65.221. IPs ending with 201-206 out eth0, 193-199 out
> eth1, and 185-190 out eth2. Anything other IPs would be routed to default
> gateway.
>
> However, any local boxes on eth0 would need -net routes for
> 203.65.221.192/255.255.255.248 and 203.65.221.184/255.255.255.248 using
> your eth0 IP as gateway to reach your other local networks. If they
> only have a 203.65.221.206 default gateway, then traffic for your other
> local networks would go in wrong direction.
>
> Routing example for _other_ box on eth0 assuming your Linux IP ends 201:
>
> Destination Gateway Genmask Flags Metric Ref Use
Iface
> 203.65.221.200 * 255.255.255.248 U 0 0 0 eth0
> 203.65.221.192 203.65.221.201 255.255.255.248 G 0 0 0 eth0
> 203.65.221.184 203.65.221.201 255.255.255.248 G 0 0 0 eth0
> 127.0.0.0 * 255.0.0.0 U 0 0 0 lo
> default 203.65.221.206 0.0.0.0 UG 0 0 0 eth0
>
> Not sure what order these routes would appear or Flags.
>
> Boxes on eth1 or eth2 would not need those extra routes, just their local
> network route and default gateway to Linux IP facing them.
>

Thanks, David.

I did a little study and found it is really difficult to design the routing
by using only simple routing techniques.
I noticed about iproute2 and netfilter, maybe there're some solutions in
that field.

Since the box runs and does nothing about routing, and I'd like to force
three NICs to be isolated, one way is to mark the incoming pockets either
FWMARK or QoS to make apache knows the difference of all pockets. However,
Apache is on different layer and I didn't find any module can help me to
read the mark and remark in outgoing pockets. Therefore, I can make
different tables for different marked pocket to use different default
gateway.

The second solution is more simple but I am not sure if it is doable,
because I am not familiar with the request/respond process. I believe there
must be something between the two sides, incoming and outgoing, of the
application like Apache can provide some information about the established
connection, like TCP's status (ACK, RELATE, etc). That will be possible to
alter the routing table by filtering those informations, but I didn't find
anything simple enough for me to understand yet. (except someone's HTTP
request URL match module for netfilter).

Stanoo