| Home | Register | Members | Search | Links |
![]() |
| Thread Tools | Display Modes |
|
Ken
Guest
Posts: n/a
|
I am trying to get a handle on what happens when a server fails in a DNS
Round Robin load balancing domain. Here is the scenario: 1) DNS server for site www.mydomain.com returns multiple "A" records for the FQDN. The DNS server was set up with round-robin so the "top" "A" record will change with subsequent queries. 2) Application tries to establish a connection using the IP address of the first "A" record in the returned list. 3) What happens if the server associated with the first "A" record is down (possibly because DNS server hasn't been updated to remove the failed server or a cached entry was returned by the DNS resolver)? Is the next server in the list tried automatically, or do I receive an open failure in VB? 4) Where does this alternate server "retry" access behavior occur? Is it handled automatically by the operating system stack, or is it handled automatically by the VB control, or do I have to be imbed recovery code in each application I write? I am trying to set up a number of client machines in remote sites to collect telemetry data. These machines use a variety of protocols (HTTP, FTP) to send the collected data back to our central servers. These remote site machines may sit on other peoples networks using DHCP to obtain their IP addresses. I will set up the DNS entries on the NIC's, however, to point back to my DNS servers rather than the local network's DNS server (to avoid unpredictable caching behavior at each local site). My central site DNS servers will be set up for round-robin resolution, returning the addresses of the multiple, publicly-exposed web and FTP servers on my network. The IP's returned by DNS will route traffic over multiple ISPs providing me with both load balancing and availability (fail-over). Here is how the servers will be set up: ISP1 (DSL) -------------------- Router Server1 (Network Load Balancing) Switch --------- Server2 (Network Load Balancing) Server3 (Network Load Balancing) ISP2 (Cable) ------------------- Router Network Load Balancing will be set up across the three servers to provide a pair of virtual server addresses. IIS can respond and handle the incoming requests from any one of the three servers on either virtual IP address. The pair of virtual cluster addresses will correspond to NAT'd public addresses provided by the two ISPs. The two ISP's don't talk to each other and BGP routing protocol is not implementable with them (the two ISP's were selected for their routing failover diversity, up-to-and-including the last mile). DNS will be set up to return the two IP addresses (11.1.1.1 from ISP1 and 12.2.2.2 from ISP2, for example). The NLB (Network Load Balancing) will automatically handle fail-over for any of the web servers. I can set up HSRP (Host Standby Router Protocol) on the two routers to accommodate their fail-over, as well. I am trying to figure out what will happen if one or the other ISP circuit fails. My round robin will have returned both addresses (11.1.1.1 and 12.2.2.2) which can be serviced by the either router and any of the web servers. If one of the ISP or circuits fails, I will adjust the DNS entries to eliminate the failing IP address from the round-robin. What will happen to requests being directed through the failed ISP in the interim? Will the next IP address be tried automatically, or will my program get an indication of the failure? Thanks in advance for any assistance. |
|
|
|
|
|||
|
|||
|
|
|
| |
|
Doug Sherman [MVP]
Guest
Posts: n/a
|
When a DNS client receives a response to a query, it caches the result.
Even if the IP turns out to be unreachable, the client will rely on the cached result and it not query the server again. You may be able to alter/control this behavior: http://support.microsoft.com/kb/245437/en-us Also, on the server side you need to be aware of the interplay between round robin and netmask ordering: http://support.microsoft.com/kb/842197/en-us Doug Sherman MCSE, MCSA, MCP+I, MVP "Ken" <(E-Mail Removed)> wrote in message news:(E-Mail Removed)... > I am trying to get a handle on what happens when a server fails in a DNS > Round Robin load balancing domain. Here is the scenario: > > 1) DNS server for site www.mydomain.com returns multiple "A" records for the > FQDN. The DNS server was set up with round-robin so the "top" "A" record > will change with subsequent queries. > > 2) Application tries to establish a connection using the IP address of the > first "A" record in the returned list. > > 3) What happens if the server associated with the first "A" record is down > (possibly because DNS server hasn't been updated to remove the failed server > or a cached entry was returned by the DNS resolver)? Is the next server in > the list tried automatically, or do I receive an open failure in VB? > > 4) Where does this alternate server "retry" access behavior occur? Is it > handled automatically by the operating system stack, or is it handled > automatically by the VB control, or do I have to be imbed recovery code in > each application I write? > > I am trying to set up a number of client machines in remote sites to collect > telemetry data. These machines use a variety of protocols (HTTP, FTP) to > send the collected data back to our central servers. These remote site > machines may sit on other peoples networks using DHCP to obtain their IP > addresses. I will set up the DNS entries on the NIC's, however, to point > back to my DNS servers rather than the local network's DNS server (to avoid > unpredictable caching behavior at each local site). > > My central site DNS servers will be set up for round-robin resolution, > returning the addresses of the multiple, publicly-exposed web and FTP > servers on my network. The IP's returned by DNS will route traffic over > multiple ISPs providing me with both load balancing and availability > (fail-over). Here is how the servers will be set up: > > ISP1 (DSL) -------------------- Router > > Server1 (Network Load Balancing) > Switch --------- > Server2 (Network Load Balancing) > > Server3 (Network Load Balancing) > ISP2 (Cable) ------------------- Router > > Network Load Balancing will be set up across the three servers to provide a > pair of virtual server addresses. IIS can respond and handle the incoming > requests from any one of the three servers on either virtual IP address. > The pair of virtual cluster addresses will correspond to NAT'd public > addresses provided by the two ISPs. The two ISP's don't talk to each other > and BGP routing protocol is not implementable with them (the two ISP's were > selected for their routing failover diversity, up-to-and-including the last > mile). DNS will be set up to return the two IP addresses (11.1.1.1 from > ISP1 and 12.2.2.2 from ISP2, for example). > > The NLB (Network Load Balancing) will automatically handle fail-over for any > of the web servers. I can set up HSRP (Host Standby Router Protocol) on the > two routers to accommodate their fail-over, as well. I am trying to figure > out what will happen if one or the other ISP circuit fails. My round robin > will have returned both addresses (11.1.1.1 and 12.2.2.2) which can be > serviced by the either router and any of the web servers. If one of the ISP > or circuits fails, I will adjust the DNS entries to eliminate the failing IP > address from the round-robin. What will happen to requests being directed > through the failed ISP in the interim? Will the next IP address be tried > automatically, or will my program get an indication of the failure? > > Thanks in advance for any assistance. > > |
|
|
|
|
|||
|
|||
|
Ken L
Guest
Posts: n/a
|
Doug:
Thanks for the response, but it kind of missed the mark. Basicly, what happens on the client side when it has been supplied multiple IP addresses for a FQDN by the DNS resolver? I know the RFC requires the first address to be tried (although Windows does its own thing by preferring addresses on its local subnet), but what happens when that first address fails? Does Windows try to create the session with the next address supplied by the initial DNS query? Regarding the caching, I am aware of the issues there. That is why I'll point the DNS entries on the remote system IP stacks at my central DNS servers and will set a short TTL to force them to refresh more frequently than the default for Windows (24 hours). Worst case, since we own all the remote site boxes is to turn off the local DNS caching. Can anyone point me at some Windows documentation on details of the clients handling of the DNS and connection process (I've found tons on the DNS server side, but very little from the client perspective of how multiple resource records are used. Thanks. Ken "Doug Sherman [MVP]" <(E-Mail Removed)> wrote in message news:%(E-Mail Removed)... > When a DNS client receives a response to a query, it caches the result. > Even if the IP turns out to be unreachable, the client will rely on the > cached result and it not query the server again. You may be able to > alter/control this behavior: > > http://support.microsoft.com/kb/245437/en-us > > Also, on the server side you need to be aware of the interplay between > round > robin and netmask ordering: > > http://support.microsoft.com/kb/842197/en-us > > Doug Sherman > MCSE, MCSA, MCP+I, MVP > > "Ken" <(E-Mail Removed)> wrote in message > news:(E-Mail Removed)... >> I am trying to get a handle on what happens when a server fails in a DNS >> Round Robin load balancing domain. Here is the scenario: >> >> 1) DNS server for site www.mydomain.com returns multiple "A" records for > the >> FQDN. The DNS server was set up with round-robin so the "top" "A" >> record >> will change with subsequent queries. >> >> 2) Application tries to establish a connection using the IP address of >> the >> first "A" record in the returned list. >> >> 3) What happens if the server associated with the first "A" record is >> down >> (possibly because DNS server hasn't been updated to remove the failed > server >> or a cached entry was returned by the DNS resolver)? Is the next server > in >> the list tried automatically, or do I receive an open failure in VB? >> >> 4) Where does this alternate server "retry" access behavior occur? Is it >> handled automatically by the operating system stack, or is it handled >> automatically by the VB control, or do I have to be imbed recovery code >> in >> each application I write? >> >> I am trying to set up a number of client machines in remote sites to > collect >> telemetry data. These machines use a variety of protocols (HTTP, FTP) to >> send the collected data back to our central servers. These remote site >> machines may sit on other peoples networks using DHCP to obtain their IP >> addresses. I will set up the DNS entries on the NIC's, however, to point >> back to my DNS servers rather than the local network's DNS server (to > avoid >> unpredictable caching behavior at each local site). >> >> My central site DNS servers will be set up for round-robin resolution, >> returning the addresses of the multiple, publicly-exposed web and FTP >> servers on my network. The IP's returned by DNS will route traffic over >> multiple ISPs providing me with both load balancing and availability >> (fail-over). Here is how the servers will be set up: >> >> ISP1 (DSL) -------------------- Router >> >> Server1 (Network Load Balancing) >> Switch --------- >> Server2 (Network Load Balancing) >> >> Server3 (Network Load Balancing) >> ISP2 (Cable) ------------------- Router >> >> Network Load Balancing will be set up across the three servers to provide > a >> pair of virtual server addresses. IIS can respond and handle the >> incoming >> requests from any one of the three servers on either virtual IP address. >> The pair of virtual cluster addresses will correspond to NAT'd public >> addresses provided by the two ISPs. The two ISP's don't talk to each > other >> and BGP routing protocol is not implementable with them (the two ISP's > were >> selected for their routing failover diversity, up-to-and-including the > last >> mile). DNS will be set up to return the two IP addresses (11.1.1.1 from >> ISP1 and 12.2.2.2 from ISP2, for example). >> >> The NLB (Network Load Balancing) will automatically handle fail-over for > any >> of the web servers. I can set up HSRP (Host Standby Router Protocol) on > the >> two routers to accommodate their fail-over, as well. I am trying to > figure >> out what will happen if one or the other ISP circuit fails. My round > robin >> will have returned both addresses (11.1.1.1 and 12.2.2.2) which can be >> serviced by the either router and any of the web servers. If one of the > ISP >> or circuits fails, I will adjust the DNS entries to eliminate the failing > IP >> address from the round-robin. What will happen to requests being >> directed >> through the failed ISP in the interim? Will the next IP address be tried >> automatically, or will my program get an indication of the failure? >> >> Thanks in advance for any assistance. >> >> > > |
|
|
|
|
|||
|
|||
|
Doug Sherman [MVP]
Guest
Posts: n/a
|
Don't know what else to tell you. The DNS server does not supply the client
with multiple addresses in response to a name resolution query - it supplies one. If that address fails, the client will not query the DNS server again until the cache entry is purged. Try posting your question on: microsoft.public.windows.server.dns Doug Sherman MCSE, MCSA, MCP+I, MVP "Ken L" <(E-Mail Removed)> wrote in message news:(E-Mail Removed)... > Doug: > > Thanks for the response, but it kind of missed the mark. Basicly, what > happens on the client side when it has been supplied multiple IP addresses > for a FQDN by the DNS resolver? I know the RFC requires the first address > to be tried (although Windows does its own thing by preferring addresses on > its local subnet), but what happens when that first address fails? Does > Windows try to create the session with the next address supplied by the > initial DNS query? > > Regarding the caching, I am aware of the issues there. That is why I'll > point the DNS entries on the remote system IP stacks at my central DNS > servers and will set a short TTL to force them to refresh more frequently > than the default for Windows (24 hours). Worst case, since we own all the > remote site boxes is to turn off the local DNS caching. > > Can anyone point me at some Windows documentation on details of the clients > handling of the DNS and connection process (I've found tons on the DNS > server side, but very little from the client perspective of how multiple > resource records are used. > > Thanks. > > Ken > > "Doug Sherman [MVP]" <(E-Mail Removed)> wrote in message > news:%(E-Mail Removed)... > > When a DNS client receives a response to a query, it caches the result. > > Even if the IP turns out to be unreachable, the client will rely on the > > cached result and it not query the server again. You may be able to > > alter/control this behavior: > > > > http://support.microsoft.com/kb/245437/en-us > > > > Also, on the server side you need to be aware of the interplay between > > round > > robin and netmask ordering: > > > > http://support.microsoft.com/kb/842197/en-us > > > > Doug Sherman > > MCSE, MCSA, MCP+I, MVP > > > > "Ken" <(E-Mail Removed)> wrote in message > > news:(E-Mail Removed)... > >> I am trying to get a handle on what happens when a server fails in a DNS > >> Round Robin load balancing domain. Here is the scenario: > >> > >> 1) DNS server for site www.mydomain.com returns multiple "A" records for > > the > >> FQDN. The DNS server was set up with round-robin so the "top" "A" > >> record > >> will change with subsequent queries. > >> > >> 2) Application tries to establish a connection using the IP address of > >> the > >> first "A" record in the returned list. > >> > >> 3) What happens if the server associated with the first "A" record is > >> down > >> (possibly because DNS server hasn't been updated to remove the failed > > server > >> or a cached entry was returned by the DNS resolver)? Is the next server > > in > >> the list tried automatically, or do I receive an open failure in VB? > >> > >> 4) Where does this alternate server "retry" access behavior occur? Is it > >> handled automatically by the operating system stack, or is it handled > >> automatically by the VB control, or do I have to be imbed recovery code > >> in > >> each application I write? > >> > >> I am trying to set up a number of client machines in remote sites to > > collect > >> telemetry data. These machines use a variety of protocols (HTTP, FTP) to > >> send the collected data back to our central servers. These remote site > >> machines may sit on other peoples networks using DHCP to obtain their IP > >> addresses. I will set up the DNS entries on the NIC's, however, to point > >> back to my DNS servers rather than the local network's DNS server (to > > avoid > >> unpredictable caching behavior at each local site). > >> > >> My central site DNS servers will be set up for round-robin resolution, > >> returning the addresses of the multiple, publicly-exposed web and FTP > >> servers on my network. The IP's returned by DNS will route traffic over > >> multiple ISPs providing me with both load balancing and availability > >> (fail-over). Here is how the servers will be set up: > >> > >> ISP1 (DSL) -------------------- Router > >> > >> Server1 (Network Load Balancing) > >> Switch --------- > >> Server2 (Network Load Balancing) > >> > >> Server3 (Network Load Balancing) > >> ISP2 (Cable) ------------------- Router > >> > >> Network Load Balancing will be set up across the three servers to provide > > a > >> pair of virtual server addresses. IIS can respond and handle the > >> incoming > >> requests from any one of the three servers on either virtual IP address. > >> The pair of virtual cluster addresses will correspond to NAT'd public > >> addresses provided by the two ISPs. The two ISP's don't talk to each > > other > >> and BGP routing protocol is not implementable with them (the two ISP's > > were > >> selected for their routing failover diversity, up-to-and-including the > > last > >> mile). DNS will be set up to return the two IP addresses (11.1.1.1 from > >> ISP1 and 12.2.2.2 from ISP2, for example). > >> > >> The NLB (Network Load Balancing) will automatically handle fail-over for > > any > >> of the web servers. I can set up HSRP (Host Standby Router Protocol) on > > the > >> two routers to accommodate their fail-over, as well. I am trying to > > figure > >> out what will happen if one or the other ISP circuit fails. My round > > robin > >> will have returned both addresses (11.1.1.1 and 12.2.2.2) which can be > >> serviced by the either router and any of the web servers. If one of the > > ISP > >> or circuits fails, I will adjust the DNS entries to eliminate the failing > > IP > >> address from the round-robin. What will happen to requests being > >> directed > >> through the failed ISP in the interim? Will the next IP address be tried > >> automatically, or will my program get an indication of the failure? > >> > >> Thanks in advance for any assistance. > >> > >> > > > > > > |
|
|
|
|
|||
|
|||
|
Doug Sherman [MVP]
Guest
Posts: n/a
|
Hmmmm - I just tested this with a multi record host, and you are correct.
The DNS client caches all the IPs. However, as far as I can see, the client only uses the first one for as long as these simultaneously created entries survive. Doug Sherman MCSE, MCSA, MCP+I, MVP "Doug Sherman [MVP]" <(E-Mail Removed)> wrote in message news:#(E-Mail Removed)... > Don't know what else to tell you. The DNS server does not supply the client > with multiple addresses in response to a name resolution query - it supplies > one. If that address fails, the client will not query the DNS server again > until the cache entry is purged. Try posting your question on: > > microsoft.public.windows.server.dns > > Doug Sherman > MCSE, MCSA, MCP+I, MVP > > "Ken L" <(E-Mail Removed)> wrote in message > news:(E-Mail Removed)... > > Doug: > > > > Thanks for the response, but it kind of missed the mark. Basicly, what > > happens on the client side when it has been supplied multiple IP addresses > > for a FQDN by the DNS resolver? I know the RFC requires the first address > > to be tried (although Windows does its own thing by preferring addresses > on > > its local subnet), but what happens when that first address fails? Does > > Windows try to create the session with the next address supplied by the > > initial DNS query? > > > > Regarding the caching, I am aware of the issues there. That is why I'll > > point the DNS entries on the remote system IP stacks at my central DNS > > servers and will set a short TTL to force them to refresh more frequently > > than the default for Windows (24 hours). Worst case, since we own all the > > remote site boxes is to turn off the local DNS caching. > > > > Can anyone point me at some Windows documentation on details of the > clients > > handling of the DNS and connection process (I've found tons on the DNS > > server side, but very little from the client perspective of how multiple > > resource records are used. > > > > Thanks. > > > > Ken > > > > "Doug Sherman [MVP]" <(E-Mail Removed)> wrote in > message > > news:%(E-Mail Removed)... > > > When a DNS client receives a response to a query, it caches the result. > > > Even if the IP turns out to be unreachable, the client will rely on the > > > cached result and it not query the server again. You may be able to > > > alter/control this behavior: > > > > > > http://support.microsoft.com/kb/245437/en-us > > > > > > Also, on the server side you need to be aware of the interplay between > > > round > > > robin and netmask ordering: > > > > > > http://support.microsoft.com/kb/842197/en-us > > > > > > Doug Sherman > > > MCSE, MCSA, MCP+I, MVP > > > > > > "Ken" <(E-Mail Removed)> wrote in message > > > news:(E-Mail Removed)... > > >> I am trying to get a handle on what happens when a server fails in a > DNS > > >> Round Robin load balancing domain. Here is the scenario: > > >> > > >> 1) DNS server for site www.mydomain.com returns multiple "A" records > for > > > the > > >> FQDN. The DNS server was set up with round-robin so the "top" "A" > > >> record > > >> will change with subsequent queries. > > >> > > >> 2) Application tries to establish a connection using the IP address of > > >> the > > >> first "A" record in the returned list. > > >> > > >> 3) What happens if the server associated with the first "A" record is > > >> down > > >> (possibly because DNS server hasn't been updated to remove the failed > > > server > > >> or a cached entry was returned by the DNS resolver)? Is the next > server > > > in > > >> the list tried automatically, or do I receive an open failure in VB? > > >> > > >> 4) Where does this alternate server "retry" access behavior occur? Is > it > > >> handled automatically by the operating system stack, or is it handled > > >> automatically by the VB control, or do I have to be imbed recovery code > > >> in > > >> each application I write? > > >> > > >> I am trying to set up a number of client machines in remote sites to > > > collect > > >> telemetry data. These machines use a variety of protocols (HTTP, FTP) > to > > >> send the collected data back to our central servers. These remote > site > > >> machines may sit on other peoples networks using DHCP to obtain their > IP > > >> addresses. I will set up the DNS entries on the NIC's, however, to > point > > >> back to my DNS servers rather than the local network's DNS server (to > > > avoid > > >> unpredictable caching behavior at each local site). > > >> > > >> My central site DNS servers will be set up for round-robin resolution, > > >> returning the addresses of the multiple, publicly-exposed web and FTP > > >> servers on my network. The IP's returned by DNS will route traffic > over > > >> multiple ISPs providing me with both load balancing and availability > > >> (fail-over). Here is how the servers will be set up: > > >> > > >> ISP1 (DSL) -------------------- Router > > >> > > >> Server1 (Network Load Balancing) > > >> > Switch --------- > > >> Server2 (Network Load Balancing) > > >> > > >> Server3 (Network Load Balancing) > > >> ISP2 (Cable) ------------------- Router > > >> > > >> Network Load Balancing will be set up across the three servers to > provide > > > a > > >> pair of virtual server addresses. IIS can respond and handle the > > >> incoming > > >> requests from any one of the three servers on either virtual IP > address. > > >> The pair of virtual cluster addresses will correspond to NAT'd public > > >> addresses provided by the two ISPs. The two ISP's don't talk to each > > > other > > >> and BGP routing protocol is not implementable with them (the two ISP's > > > were > > >> selected for their routing failover diversity, up-to-and-including the > > > last > > >> mile). DNS will be set up to return the two IP addresses (11.1.1.1 > from > > >> ISP1 and 12.2.2.2 from ISP2, for example). > > >> > > >> The NLB (Network Load Balancing) will automatically handle fail-over > for > > > any > > >> of the web servers. I can set up HSRP (Host Standby Router Protocol) > on > > > the > > >> two routers to accommodate their fail-over, as well. I am trying to > > > figure > > >> out what will happen if one or the other ISP circuit fails. My round > > > robin > > >> will have returned both addresses (11.1.1.1 and 12.2.2.2) which can be > > >> serviced by the either router and any of the web servers. If one of > the > > > ISP > > >> or circuits fails, I will adjust the DNS entries to eliminate the > failing > > > IP > > >> address from the round-robin. What will happen to requests being > > >> directed > > >> through the failed ISP in the interim? Will the next IP address be > tried > > >> automatically, or will my program get an indication of the failure? > > >> > > >> Thanks in advance for any assistance. > > >> > > >> > > > > > > > > > > > > |
|
|
|
|
|||
|
|||
|
Ken L
Guest
Posts: n/a
|
Doug:
That's what I had confirmed by the IPCONFIG /DISPLAYDNS command. The issue I'm trying to get resolved is what happens (and in which code) when a connection attempt fails when there are multiple IP addresses in the resolver cache. Is the connection attempt re-tried with the addtional IP addresses? If so, where is this done (by the OS, by the VB control, etc.) I don't see any way in the usual VB controls to force it to try the additional addresses in the resolver cache, if I see a connection failure. Thanks. "Doug Sherman [MVP]" <(E-Mail Removed)> wrote in message news:(E-Mail Removed)... > Hmmmm - I just tested this with a multi record host, and you are correct. > The DNS client caches all the IPs. However, as far as I can see, the > client > only uses the first one for as long as these simultaneously created > entries > survive. > > Doug Sherman > MCSE, MCSA, MCP+I, MVP > > "Doug Sherman [MVP]" <(E-Mail Removed)> wrote in > message > news:#(E-Mail Removed)... >> Don't know what else to tell you. The DNS server does not supply the > client >> with multiple addresses in response to a name resolution query - it > supplies >> one. If that address fails, the client will not query the DNS server > again >> until the cache entry is purged. Try posting your question on: >> >> microsoft.public.windows.server.dns >> >> Doug Sherman >> MCSE, MCSA, MCP+I, MVP >> >> "Ken L" <(E-Mail Removed)> wrote in message >> news:(E-Mail Removed)... >> > Doug: >> > >> > Thanks for the response, but it kind of missed the mark. Basicly, what >> > happens on the client side when it has been supplied multiple IP > addresses >> > for a FQDN by the DNS resolver? I know the RFC requires the first > address >> > to be tried (although Windows does its own thing by preferring >> > addresses >> on >> > its local subnet), but what happens when that first address fails? >> > Does >> > Windows try to create the session with the next address supplied by the >> > initial DNS query? >> > >> > Regarding the caching, I am aware of the issues there. That is why >> > I'll >> > point the DNS entries on the remote system IP stacks at my central DNS >> > servers and will set a short TTL to force them to refresh more > frequently >> > than the default for Windows (24 hours). Worst case, since we own all > the >> > remote site boxes is to turn off the local DNS caching. >> > >> > Can anyone point me at some Windows documentation on details of the >> clients >> > handling of the DNS and connection process (I've found tons on the DNS >> > server side, but very little from the client perspective of how >> > multiple >> > resource records are used. >> > >> > Thanks. >> > >> > Ken >> > >> > "Doug Sherman [MVP]" <(E-Mail Removed)> wrote in >> message >> > news:%(E-Mail Removed)... >> > > When a DNS client receives a response to a query, it caches the > result. >> > > Even if the IP turns out to be unreachable, the client will rely on > the >> > > cached result and it not query the server again. You may be able to >> > > alter/control this behavior: >> > > >> > > http://support.microsoft.com/kb/245437/en-us >> > > >> > > Also, on the server side you need to be aware of the interplay >> > > between >> > > round >> > > robin and netmask ordering: >> > > >> > > http://support.microsoft.com/kb/842197/en-us >> > > >> > > Doug Sherman >> > > MCSE, MCSA, MCP+I, MVP >> > > >> > > "Ken" <(E-Mail Removed)> wrote in message >> > > news:(E-Mail Removed)... >> > >> I am trying to get a handle on what happens when a server fails in a >> DNS >> > >> Round Robin load balancing domain. Here is the scenario: >> > >> >> > >> 1) DNS server for site www.mydomain.com returns multiple "A" records >> for >> > > the >> > >> FQDN. The DNS server was set up with round-robin so the "top" "A" >> > >> record >> > >> will change with subsequent queries. >> > >> >> > >> 2) Application tries to establish a connection using the IP address > of >> > >> the >> > >> first "A" record in the returned list. >> > >> >> > >> 3) What happens if the server associated with the first "A" record >> > >> is >> > >> down >> > >> (possibly because DNS server hasn't been updated to remove the >> > >> failed >> > > server >> > >> or a cached entry was returned by the DNS resolver)? Is the next >> server >> > > in >> > >> the list tried automatically, or do I receive an open failure in VB? >> > >> >> > >> 4) Where does this alternate server "retry" access behavior occur? > Is >> it >> > >> handled automatically by the operating system stack, or is it >> > >> handled >> > >> automatically by the VB control, or do I have to be imbed recovery > code >> > >> in >> > >> each application I write? >> > >> >> > >> I am trying to set up a number of client machines in remote sites to >> > > collect >> > >> telemetry data. These machines use a variety of protocols (HTTP, > FTP) >> to >> > >> send the collected data back to our central servers. These remote >> site >> > >> machines may sit on other peoples networks using DHCP to obtain >> > >> their >> IP >> > >> addresses. I will set up the DNS entries on the NIC's, however, to >> point >> > >> back to my DNS servers rather than the local network's DNS server >> > >> (to >> > > avoid >> > >> unpredictable caching behavior at each local site). >> > >> >> > >> My central site DNS servers will be set up for round-robin > resolution, >> > >> returning the addresses of the multiple, publicly-exposed web and >> > >> FTP >> > >> servers on my network. The IP's returned by DNS will route traffic >> over >> > >> multiple ISPs providing me with both load balancing and availability >> > >> (fail-over). Here is how the servers will be set up: >> > >> >> > >> ISP1 (DSL) -------------------- Router >> > >> >> > >> Server1 (Network Load Balancing) >> > >> >> Switch --------- >> > >> Server2 (Network Load Balancing) >> > >> >> > >> Server3 (Network Load Balancing) >> > >> ISP2 (Cable) ------------------- Router >> > >> >> > >> Network Load Balancing will be set up across the three servers to >> provide >> > > a >> > >> pair of virtual server addresses. IIS can respond and handle the >> > >> incoming >> > >> requests from any one of the three servers on either virtual IP >> address. >> > >> The pair of virtual cluster addresses will correspond to NAT'd >> > >> public >> > >> addresses provided by the two ISPs. The two ISP's don't talk to > each >> > > other >> > >> and BGP routing protocol is not implementable with them (the two > ISP's >> > > were >> > >> selected for their routing failover diversity, up-to-and-including > the >> > > last >> > >> mile). DNS will be set up to return the two IP addresses (11.1.1.1 >> from >> > >> ISP1 and 12.2.2.2 from ISP2, for example). >> > >> >> > >> The NLB (Network Load Balancing) will automatically handle fail-over >> for >> > > any >> > >> of the web servers. I can set up HSRP (Host Standby Router >> > >> Protocol) >> on >> > > the >> > >> two routers to accommodate their fail-over, as well. I am trying to >> > > figure >> > >> out what will happen if one or the other ISP circuit fails. My >> > >> round >> > > robin >> > >> will have returned both addresses (11.1.1.1 and 12.2.2.2) which can > be >> > >> serviced by the either router and any of the web servers. If one of >> the >> > > ISP >> > >> or circuits fails, I will adjust the DNS entries to eliminate the >> failing >> > > IP >> > >> address from the round-robin. What will happen to requests being >> > >> directed >> > >> through the failed ISP in the interim? Will the next IP address be >> tried >> > >> automatically, or will my program get an indication of the failure? >> > >> >> > >> Thanks in advance for any assistance. >> > >> >> > >> >> > > >> > > >> > >> > >> >> > > |
|
|
|
|
|||
|
|||
|
Doug Sherman [MVP]
Guest
Posts: n/a
|
Yeah, you were way ahead of me - sorry I wasn't aware of this. I always
thought the client only got one IP because that's the way it behaves. As far as I can tell, the client never tries any of the other addresses unless they are still alive when the first one expires. I'll try to find out whether this behavior can be modified in the OS, but so far no luck. I take it DFS is not a workable solution. If you redirect /displaydns to a text file, or better yet call a utility like nslookup to retrieve the desired host records, I would think you could write a program to read the output and sequentially or randomly try the IPs until a connection is made. But you'd have to be a better programmer than me - probably you are. I still think posting this on microsoft.public.windows.server.dns is worth a shot. Doug Sherman MCSE, MCSA, MCP+I, MVP "Ken L" <(E-Mail Removed)> wrote in message news:(E-Mail Removed)... > Doug: > > That's what I had confirmed by the IPCONFIG /DISPLAYDNS command. The issue > I'm trying to get resolved is what happens (and in which code) when a > connection attempt fails when there are multiple IP addresses in the > resolver cache. > Is the connection attempt re-tried with the addtional IP addresses? If so, > where is this done (by the OS, by the VB control, etc.) I don't see any > way in the usual VB controls to force it to try the additional addresses in > the resolver cache, if I see a connection failure. > > Thanks. > > > "Doug Sherman [MVP]" <(E-Mail Removed)> wrote in message > news:(E-Mail Removed)... > > Hmmmm - I just tested this with a multi record host, and you are correct. > > The DNS client caches all the IPs. However, as far as I can see, the > > client > > only uses the first one for as long as these simultaneously created > > entries > > survive. > > > > Doug Sherman > > MCSE, MCSA, MCP+I, MVP > > > > "Doug Sherman [MVP]" <(E-Mail Removed)> wrote in > > message > > news:#(E-Mail Removed)... > >> Don't know what else to tell you. The DNS server does not supply the > > client > >> with multiple addresses in response to a name resolution query - it > > supplies > >> one. If that address fails, the client will not query the DNS server > > again > >> until the cache entry is purged. Try posting your question on: > >> > >> microsoft.public.windows.server.dns > >> > >> Doug Sherman > >> MCSE, MCSA, MCP+I, MVP > >> > >> "Ken L" <(E-Mail Removed)> wrote in message > >> news:(E-Mail Removed)... > >> > Doug: > >> > > >> > Thanks for the response, but it kind of missed the mark. Basicly, what > >> > happens on the client side when it has been supplied multiple IP > > addresses > >> > for a FQDN by the DNS resolver? I know the RFC requires the first > > address > >> > to be tried (although Windows does its own thing by preferring > >> > addresses > >> on > >> > its local subnet), but what happens when that first address fails? > >> > Does > >> > Windows try to create the session with the next address supplied by the > >> > initial DNS query? > >> > > >> > Regarding the caching, I am aware of the issues there. That is why > >> > I'll > >> > point the DNS entries on the remote system IP stacks at my central DNS > >> > servers and will set a short TTL to force them to refresh more > > frequently > >> > than the default for Windows (24 hours). Worst case, since we own all > > the > >> > remote site boxes is to turn off the local DNS caching. > >> > > >> > Can anyone point me at some Windows documentation on details of the > >> clients > >> > handling of the DNS and connection process (I've found tons on the DNS > >> > server side, but very little from the client perspective of how > >> > multiple > >> > resource records are used. > >> > > >> > Thanks. > >> > > >> > Ken > >> > > >> > "Doug Sherman [MVP]" <(E-Mail Removed)> wrote in > >> message > >> > news:%(E-Mail Removed)... > >> > > When a DNS client receives a response to a query, it caches the > > result. > >> > > Even if the IP turns out to be unreachable, the client will rely on > > the > >> > > cached result and it not query the server again. You may be able to > >> > > alter/control this behavior: > >> > > > >> > > http://support.microsoft.com/kb/245437/en-us > >> > > > >> > > Also, on the server side you need to be aware of the interplay > >> > > between > >> > > round > >> > > robin and netmask ordering: > >> > > > >> > > http://support.microsoft.com/kb/842197/en-us > >> > > > >> > > Doug Sherman > >> > > MCSE, MCSA, MCP+I, MVP > >> > > > >> > > "Ken" <(E-Mail Removed)> wrote in message > >> > > news:(E-Mail Removed)... > >> > >> I am trying to get a handle on what happens when a server fails in a > >> DNS > >> > >> Round Robin load balancing domain. Here is the scenario: > >> > >> > >> > >> 1) DNS server for site www.mydomain.com returns multiple "A" records > >> for > >> > > the > >> > >> FQDN. The DNS server was set up with round-robin so the "top" "A" > >> > >> record > >> > >> will change with subsequent queries. > >> > >> > >> > >> 2) Application tries to establish a connection using the IP address > > of > >> > >> the > >> > >> first "A" record in the returned list. > >> > >> > >> > >> 3) What happens if the server associated with the first "A" record > >> > >> is > >> > >> down > >> > >> (possibly because DNS server hasn't been updated to remove the > >> > >> failed > >> > > server > >> > >> or a cached entry was returned by the DNS resolver)? Is the next > >> server > >> > > in > >> > >> the list tried automatically, or do I receive an open failure in VB? > >> > >> > >> > >> 4) Where does this alternate server "retry" access behavior occur? > > Is > >> it > >> > >> handled automatically by the operating system stack, or is it > >> > >> handled > >> > >> automatically by the VB control, or do I have to be imbed recovery > > code > >> > >> in > >> > >> each application I write? > >> > >> > >> > >> I am trying to set up a number of client machines in remote sites to > >> > > collect > >> > >> telemetry data. These machines use a variety of protocols (HTTP, > > FTP) > >> to > >> > >> send the collected data back to our central servers. These remote > >> site > >> > >> machines may sit on other peoples networks using DHCP to obtain > >> > >> their > >> IP > >> > >> addresses. I will set up the DNS entries on the NIC's, however, to > >> point > >> > >> back to my DNS servers rather than the local network's DNS server > >> > >> (to > >> > > avoid > >> > >> unpredictable caching behavior at each local site). > >> > >> > >> > >> My central site DNS servers will be set up for round-robin > > resolution, > >> > >> returning the addresses of the multiple, publicly-exposed web and > >> > >> FTP > >> > >> servers on my network. The IP's returned by DNS will route traffic > >> over > >> > >> multiple ISPs providing me with both load balancing and availability > >> > >> (fail-over). Here is how the servers will be set up: > >> > >> > >> > >> ISP1 (DSL) -------------------- Router > >> > >> > >> > >> Server1 (Network Load Balancing) > >> > >> > >> Switch --------- > >> > >> Server2 (Network Load Balancing) > >> > >> > >> > >> Server3 (Network Load Balancing) > >> > >> ISP2 (Cable) ------------------- Router > >> > >> > >> > >> Network Load Balancing will be set up across the three servers to > >> provide > >> > > a > >> > >> pair of virtual server addresses. IIS can respond and handle the > >> > >> incoming > >> > >> requests from any one of the three servers on either virtual IP > >> address. > >> > >> The pair of virtual cluster addresses will correspond to NAT'd > >> > >> public > >> > >> addresses provided by the two ISPs. The two ISP's don't talk to > > each > >> > > other > >> > >> and BGP routing protocol is not implementable with them (the two > > ISP's > >> > > were > >> > >> selected for their routing failover diversity, up-to-and-including > > the > >> > > last > >> > >> mile). DNS will be set up to return the two IP addresses (11.1.1.1 > >> from > >> > >> ISP1 and 12.2.2.2 from ISP2, for example). > >> > >> > >> > >> The NLB (Network Load Balancing) will automatically handle fail-over > >> for > >> > > any > >> > >> of the web servers. I can set up HSRP (Host Standby Router > >> > >> Protocol) > >> on > >> > > the > >> > >> two routers to accommodate their fail-over, as well. I am trying to > >> > > figure > >> > >> out what will happen if one or the other ISP circuit fails. My > >> > >> round > >> > > robin > >> > >> will have returned both addresses (11.1.1.1 and 12.2.2.2) which can > > be > >> > >> serviced by the either router and any of the web servers. If one of > >> the > >> > > ISP > >> > >> or circuits fails, I will adjust the DNS entries to eliminate the > >> failing > >> > > IP > >> > >> address from the round-robin. What will happen to requests being > >> > >> directed > >> > >> through the failed ISP in the interim? Will the next IP address be > >> tried > >> > >> automatically, or will my program get an indication of the failure? > >> > >> > >> > >> Thanks in advance for any assistance. > >> > >> > >> > >> > >> > > > >> > > > >> > > >> > > >> > >> > > > > > > |
|
|
|
|
|||
|
|||
|
Ken L
Guest
Posts: n/a
|
Doug:
Thanks for hanging in there with me. Regarding the suggestion on posting to server.dns, I'll give it a try. The behavior I am concerned with is from the client perspective, though. The server side behaviors are pretty well documented in the RFC's (which is how I new multiple "A" records could be sent. I even came across some code in the MS knowledge base which talked about how to handle reading the multiple records from the cache. What I am really trying to acheive is load balancing using round robin DNS. We have two ISP's servicing our center and I'd like to rotate the inbound traffic between them. This is the purpose of the round robin on the DNS servers where the "top" address returned is rotated between the various "A" record addresses that are available. My problem is that I have a large number of automated, unmanned telemetry stations sending data to us over the net. I want to balance their inbound activity in an active/active configuration. My concern is what happens when one of the ISP's fails (I have the failure of servers covered with Network Load Balancing). In that case, the resolver will have both "A" records (one going through one ISP, the second going through the other). What is the processing done by the winsock? Will it automatically handle the retry on the other "A" record or will I have include code to detect and try an alternate URL that only points to the surving route? Thanks again. Ken "Doug Sherman [MVP]" <(E-Mail Removed)> wrote in message news:%23Q%(E-Mail Removed)... > Yeah, you were way ahead of me - sorry I wasn't aware of this. I always > thought the client only got one IP because that's the way it behaves. As > far as I can tell, the client never tries any of the other addresses > unless > they are still alive when the first one expires. I'll try to find out > whether this behavior can be modified in the OS, but so far no luck. I > take > it DFS is not a workable solution. > > If you redirect /displaydns to a text file, or better yet call a utility > like nslookup to retrieve the desired host records, I would think you > could > write a program to read the output and sequentially or randomly try the > IPs > until a connection is made. But you'd have to be a better programmer than > me - probably you are. I still think posting this on > microsoft.public.windows.server.dns is worth a shot. > > Doug Sherman > MCSE, MCSA, MCP+I, MVP > > "Ken L" <(E-Mail Removed)> wrote in message > news:(E-Mail Removed)... >> Doug: >> >> That's what I had confirmed by the IPCONFIG /DISPLAYDNS command. The > issue >> I'm trying to get resolved is what happens (and in which code) when a >> connection attempt fails when there are multiple IP addresses in the >> resolver cache. >> Is the connection attempt re-tried with the addtional IP addresses? If > so, >> where is this done (by the OS, by the VB control, etc.) I don't see any >> way in the usual VB controls to force it to try the additional addresses > in >> the resolver cache, if I see a connection failure. >> >> Thanks. >> >> >> "Doug Sherman [MVP]" <(E-Mail Removed)> wrote in > message >> news:(E-Mail Removed)... >> > Hmmmm - I just tested this with a multi record host, and you are > correct. >> > The DNS client caches all the IPs. However, as far as I can see, the >> > client >> > only uses the first one for as long as these simultaneously created >> > entries >> > survive. >> > >> > Doug Sherman >> > MCSE, MCSA, MCP+I, MVP >> > >> > "Doug Sherman [MVP]" <(E-Mail Removed)> wrote in >> > message >> > news:#(E-Mail Removed)... >> >> Don't know what else to tell you. The DNS server does not supply the >> > client >> >> with multiple addresses in response to a name resolution query - it >> > supplies >> >> one. If that address fails, the client will not query the DNS server >> > again >> >> until the cache entry is purged. Try posting your question on: >> >> >> >> microsoft.public.windows.server.dns >> >> >> >> Doug Sherman >> >> MCSE, MCSA, MCP+I, MVP >> >> >> >> "Ken L" <(E-Mail Removed)> wrote in message >> >> news:(E-Mail Removed)... >> >> > Doug: >> >> > >> >> > Thanks for the response, but it kind of missed the mark. Basicly, > what >> >> > happens on the client side when it has been supplied multiple IP >> > addresses >> >> > for a FQDN by the DNS resolver? I know the RFC requires the first >> > address >> >> > to be tried (although Windows does its own thing by preferring >> >> > addresses >> >> on >> >> > its local subnet), but what happens when that first address fails? >> >> > Does >> >> > Windows try to create the session with the next address supplied by > the >> >> > initial DNS query? >> >> > >> >> > Regarding the caching, I am aware of the issues there. That is why >> >> > I'll >> >> > point the DNS entries on the remote system IP stacks at my central > DNS >> >> > servers and will set a short TTL to force them to refresh more >> > frequently >> >> > than the default for Windows (24 hours). Worst case, since we own > all >> > the >> >> > remote site boxes is to turn off the local DNS caching. >> >> > >> >> > Can anyone point me at some Windows documentation on details of the >> >> clients >> >> > handling of the DNS and connection process (I've found tons on the > DNS >> >> > server side, but very little from the client perspective of how >> >> > multiple >> >> > resource records are used. >> >> > >> >> > Thanks. >> >> > >> >> > Ken >> >> > >> >> > "Doug Sherman [MVP]" <(E-Mail Removed)> wrote in >> >> message >> >> > news:%(E-Mail Removed)... >> >> > > When a DNS client receives a response to a query, it caches the >> > result. >> >> > > Even if the IP turns out to be unreachable, the client will rely >> >> > > on >> > the >> >> > > cached result and it not query the server again. You may be able > to >> >> > > alter/control this behavior: >> >> > > >> >> > > http://support.microsoft.com/kb/245437/en-us >> >> > > >> >> > > Also, on the server side you need to be aware of the interplay >> >> > > between >> >> > > round >> >> > > robin and netmask ordering: >> >> > > >> >> > > http://support.microsoft.com/kb/842197/en-us >> >> > > >> >> > > Doug Sherman >> >> > > MCSE, MCSA, MCP+I, MVP >> >> > > >> >> > > "Ken" <(E-Mail Removed)> wrote in message >> >> > > news:(E-Mail Removed)... >> >> > >> I am trying to get a handle on what happens when a server fails >> >> > >> in > a >> >> DNS >> >> > >> Round Robin load balancing domain. Here is the scenario: >> >> > >> >> >> > >> 1) DNS server for site www.mydomain.com returns multiple "A" > records >> >> for >> >> > > the >> >> > >> FQDN. The DNS server was set up with round-robin so the "top" > "A" >> >> > >> record >> >> > >> will change with subsequent queries. >> >> > >> >> >> > >> 2) Application tries to establish a connection using the IP > address >> > of >> >> > >> the >> >> > >> first "A" record in the returned list. >> >> > >> >> >> > >> 3) What happens if the server associated with the first "A" >> >> > >> record >> >> > >> is >> >> > >> down >> >> > >> (possibly because DNS server hasn't been updated to remove the >> >> > >> failed >> >> > > server >> >> > >> or a cached entry was returned by the DNS resolver)? Is the next >> >> server >> >> > > in >> >> > >> the list tried automatically, or do I receive an open failure in > VB? >> >> > >> >> >> > >> 4) Where does this alternate server "retry" access behavior >> >> > >> occur? >> > Is >> >> it >> >> > >> handled automatically by the operating system stack, or is it >> >> > >> handled >> >> > >> automatically by the VB control, or do I have to be imbed >> >> > >> recovery >> > code >> >> > >> in >> >> > >> each application I write? >> >> > >> >> >> > >> I am trying to set up a number of client machines in remote sites > to >> >> > > collect >> >> > >> telemetry data. These machines use a variety of protocols (HTTP, >> > FTP) >> >> to >> >> > >> send the collected data back to our central servers. These > remote >> >> site >> >> > >> machines may sit on other peoples networks using DHCP to obtain >> >> > >> their >> >> IP >> >> > >> addresses. I will set up the DNS entries on the NIC's, however, > to >> >> point >> >> > >> back to my DNS servers rather than the local network's DNS server >> >> > >> (to >> >> > > avoid >> >> > >> unpredictable caching behavior at each local site). >> >> > >> >> >> > >> My central site DNS servers will be set up for round-robin >> > resolution, >> >> > >> returning the addresses of the multiple, publicly-exposed web and >> >> > >> FTP >> >> > >> servers on my network. The IP's returned by DNS will route > traffic >> >> over >> >> > >> multiple ISPs providing me with both load balancing and > availability >> >> > >> (fail-over). Here is how the servers will be set up: >> >> > >> >> >> > >> ISP1 (DSL) -------------------- Router >> >> > >> >> >> > >> Server1 (Network Load Balancing) >> >> > >> >> >> > Switch --------- >> >> > >> Server2 (Network Load Balancing) >> >> > >> >> >> > >> Server3 (Network Load Balancing) >> >> > >> ISP2 (Cable) ------------------- Router >> >> > >> >> >> > >> Network Load Balancing will be set up across the three servers to >> >> provide >> >> > > a >> >> > >> pair of virtual server addresses. IIS can respond and handle the >> >> > >> incoming >> >> > >> requests from any one of the three servers on either virtual IP >> >> address. >> >> > >> The pair of virtual cluster addresses will correspond to NAT'd >> >> > >> public >> >> > >> addresses provided by the two ISPs. The two ISP's don't talk to >> > each >> >> > > other >> >> > >> and BGP routing protocol is not implementable with them (the two >> > ISP's >> >> > > were >> >> > >> selected for their routing failover diversity, >> >> > >> up-to-and-including >> > the >> >> > > last >> >> > >> mile). DNS will be set up to return the two IP addresses > (11.1.1.1 >> >> from >> >> > >> ISP1 and 12.2.2.2 from ISP2, for example). >> >> > >> >> >> > >> The NLB (Network Load Balancing) will automatically handle > fail-over >> >> for >> >> > > any >> >> > >> of the web servers. I can set up HSRP (Host Standby Router >> >> > >> Protocol) >> >> on >> >> > > the >> >> > >> two routers to accommodate their fail-over, as well. I am trying > to >> >> > > figure >> >> > >> out what will happen if one or the other ISP circuit fails. My >> >> > >> round >> >> > > robin >> >> > >> will have returned both addresses (11.1.1.1 and 12.2.2.2) which > can >> > be >> >> > >> serviced by the either router and any of the web servers. If one > of >> >> the >> >> > > ISP >> >> > >> or circuits fails, I will adjust the DNS entries to eliminate the >> >> failing >> >> > > IP >> >> > >> address from the round-robin. What will happen to requests being >> >> > >> directed >> >> > >> through the failed ISP in the interim? Will the next IP address > be >> >> tried >> >> > >> automatically, or will my program get an indication of the > failure? >> >> > >> >> >> > >> Thanks in advance for any assistance. >> >> > >> >> >> > >> >> >> > > >> >> > > >> >> > >> >> > >> >> >> >> >> > >> > >> >> > > |
|
|
|
|
|||
|
|||
|
|
|
| |
![]() |
| Thread Tools | |
| Display Modes | |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| New server FTP failures - empty files | Dienstag W | Linux Networking | 1 | 01-10-2009 06:58 PM |
| Problem with net bios resolution and VPN and ISA server | Arch Willingham | Windows Networking | 1 | 05-23-2006 05:39 PM |
| Networking config failures dual NIC configuration Dell Server/SBS2 | =?Utf-8?B?SkRkb3Rjb20=?= | Windows Networking | 12 | 11-03-2004 01:19 PM |
| nfs strange failures (getfh, linux server, solaris client) | Chris Phillips | Linux Networking | 2 | 07-01-2004 07:25 PM |
| NetBIOS Name Resolution Failures with Wireless Network? | Tom | Windows Networking | 1 | 01-08-2004 06:28 PM |
Forum Software Powered by vBulletin®, Copyright Jelsoft Enterprises Ltd.
SEO by vBSEO 3.3.2 ©2009, Crawlability, Inc. |



Linear Mode

