Cisco lightweight APs and non-IOS DHCP for controller discovery

posted to comp.dcom.sys.cisco and alt.internet.wireless, and mailed
to The EDUCAUSE Wireless Issues mailing list, with apologies to any
who see multiple copies as a result ...

This message is long (262 lines), and I apologize in advance for that.
However, I hope that it will provide missing information for anyone
having trouble getting Cisco lightweight wireless access points to locate
their controllers, using the DHCP vendor-specific option ("option 43")
from third-party DHCP servers (such as ISC's dhcpd).

I am posting this on my last day in my current position as a wireless
network administrator (I'm moving on to a new systems-oriented job),
so I will be able to participate in any followups that will not include
further experimentation with configuration of the access points, or DHCP
attributes specific to them.

In the interest of (attempted) brevity (which I realize I failed to
accomplish!), I assume that the reader understands the sequence used
by the access points for LWAPP Discovery Protocol, and understands DHCP
and DNS. I don't claim to be an expert on any of the above (please don't
email me directly with specific questions; there are mailing lists and
netnews groups for that, populated by folks who know a lot more than I
do), but I have successfully (and finally!) gotten this to work as it
should, using ISC's dhcpd running on Linux.

I struggled with this for more than a year, continually running into
a roadblock, and falling back to using a DNS resource-record for
CISCO_LWAPP_CONTROLLER.${domain}, which is fine for a relatively small
installation (our installation isn't very small, though). Our consultants
(not from Cisco, but they would themselves consult with Cisco) were at
a loss for a proper solution to this problem, and frequently resorted
to pre-configuring access-points (allegedly on Cisco's recommendation)
with controller addresses. Again, this approach is not unreasonable for
a small installation, but is simply does not scale to larger installations
with lots of wireless access points.

We started working with lightweight access-points late summer 2007,
when we started deploying a mesh network to surround our campuses,
and recently started upgrading our (approximately 360) stand-alone IOS
access-points (a mix of 350s, 1130s, 1230s, 1240s, and recently 1250s)
to lightweight AP1250s. For controllers we have a mix of 4400-series
controllers and Wism blades.

We intended to configure our setup such that each campus would have
its own set of primary and secondary controllers, with a fail-over
to a controller normally serving another campus, and the outdoor
mesh network (AP1500 series access points) would have its own set
of controllers. For this reason, using the DNS resource-record
(CISCO_LWAPP_CONTROLLER.${domain}) was deemed to be an unsuitable
approach to having our APs find their controllers (the DNS domain is
the same across our campuses).

The APs are not all on the same network segments that the controllers
are on, so the layer 3 broadcast approach to controller discovery
isn't suitable for us. There are more access-points than is reasonable
for manual pre-configuration of each, and we are growing our wireless
network into more buildings as time goes on, so that isn't about to
become more feasible.

The Cisco lightweight access-points are supposed to be able to find
controllers based on receiving a list of controller IP addresses
from a DHCP server in the vendor-specific option, aka "option 43".
Cisco provides documentation that describes how to configure different
DHCP servers (including Cisco's own IOS DHCP server, built into some
IOS devices, Microsoft's, Sun's, and one they identify as "Linux DHCP
server", which is very likely the ISC's dhcpd, but that isn't made clear;
and others).

The specific document I refer to here is "DHCP Option 43 for Lightweight
Cisco Aironet Access Points Configuration Example" (Document ID: 97066,
© 2007 - 2008 Cisco Systems, Inc. Updated: Jul 31, 2007), supplied to
me by our local consultants, but provided to them certainly by Cisco.
This document is incorrect, and the way in which it is incorrect is
consistent with my having found that many sites seem to have encountered
similar problems as I have, trying to use DHCP Option 43 from a non
IOS DHCP server to point Cisco lightweight wireless access points to
controllers on different network segments.

What I have found suggests (and my own experiments confirm) that letting
the access-points discover controllers by layer-3 broadcasting, by a
DNS query for CISCO_LWAPP_CONTROLLER.${domain}, by pre-configuring IP
addresses, or by using an IOS DHCP server configured according to the
above document will produce the expected result, but using a third-party
DHCP server seems rather difficult. My research on this matter leads
me to believe that many other sites have encountered the same results,
but have not found the solution, (actually, it seems that some sites
have created solutions that force their DHCP servers to output the
specific binary string that makes up the option 43 with an appropriate
IP address list, converted to a HEX string, for their sites, but that
isn't how it's supposed to work, nor is it particularly as maintainable
as something that is human-readable).

We use ISC's DHCP server on a Linux system, and I really wanted to
make our lightweight wireless access points function correctly with
that DHCP server. There's no reason why they shouldn't, and in fact,
after struggling with it on and off during the past year, I finally
succeeded in making this work. The reason it didn't work immediately
is simply that the documentation provided by Cisco is incorrect.

The Cisco documentation I refer to above shows the following setup for
"Linux DHCP Server" (I added line numbers for easy reference):

Linux DHCP Server

This section contains information on how the Linux DHCP server is
configured in order to return specific information to lightweight
Cisco Aironet series APs.

1 ddns-update-style interim;
2 allow bootp;
3 # option opt-43 code 43 = text;
4 option space Cisco_LWAPP_AP;
5 option Cisco_LWAPP_AP.server-address code 43 = string;
6 subnet 192.168.247.0 netmask 255.255.255.0 {
7 authoritative;
8 option routers 192.168.247.1;
9 option subnet-mask 255.255.255.0;
10 option domain-name "cisco.com";
11 option domain-name-servers 192.168.247.2, 192.168.247.3;
12 # option opt-43 = "192.168.247.5";
13 range dynamic-bootp 192.168.247.11 192.168.247.254;
14 default-lease-time 300;
15 class "Cisco AP c1200" {
16 match if option vendor-class-identifier = "Cisco AP c1200";
17 option vendor-class-identifier "Cisco AP c1200";
18 vendor-option-space Cisco_LWAPP_AP;
19 option Cisco_LWAPP_AP.server-address f1:04:c0:a8:f7:05; }
20 class "Cisco AP c1130" {
21 match if option vendor-class-identifier = "Cisco AP c1130";
22 option vendor-class-identifier "Cisco AP c1130";
23 vendor-option-space Cisco_LWAPP_AP;
24 option Cisco_LWAPP_AP.server-address f1:04:c0:a8:f7:05; }
25 class "Cisco AP c1240" {
26 match if option vendor-class-identifier = "Cisco AP c1240";
27 option vendor-class-identifier "Cisco AP c1240";
28 vendor-option-space Cisco_LWAPP_AP;
29 option Cisco_LWAPP_AP.server-address f1:04:c0:a8:f7:05; }
30 class "Airespace 1200" {
31 match if option vendor-class-identifier = "Airespace 1200";
32 option vendor-class-identifier "Airespace 1200";
33 vendor-option-space Cisco_LWAPP_AP;
34 option Cisco_LWAPP_AP.server-address "192.168.247.5"; }
35 class "Airespace.AP1200" {
36 match if option vendor-class-identifier = "Airespace.AP1200";
37 option vendor-class-identifier "Airespace 1200";
38 vendor-option-space Cisco_LWAPP_AP;
39 option Cisco_LWAPP_AP.server-address "192.168.247.5"; }
40 }

Line 5 is incorrect. Line 3 is commented out, and I'm not sure how it
would have been used had it not been commented (hrmmm... see line 12,
also commented, but that likely wouldn't work, because there is nothing
telling the receiving access point what that value refers to; at least
at line 5, it is declared to be a "server-address", and we can reasonably
believe that this will result in the correct sequence being sent to the
access point).

The rest of the above functions as expected (although the resulting DHCP
option is also meaningless to the access points, but that isn't obvious
from the configuration itself). My current (working) configuration
streamlines some of it, but what I started with looked very much like
the above, though with a slightly different namespace. It should be
noted that the same document provides a very similar (and likely just
as incorrect) configuration for the "Sun Solaris DHCP Server" (by the
syntax, I'm ready to believe that this also is ISC's dhcpd).

After much debugging and following packet-traces to be sure that
my DHCP server was indeed sending exactly what I expected it to be
sending (I presently have our configuration to send a different list
of controllers, based on the model of access point, as identified by
the vendor-class-identifier sent back by unit during the DHCP discover
phase), I thought I was going out of my mind, because it wasn't working,
but decided that what I needed to solve this, was to know exactly what
the access-points expect to see.

We setup a test environment using a Cisco IOS DHCP server on a small
router to provide DHCP service to a test access point, and configured
the DHCP server according to the same document, which for an IOS DHCP
server explains the following:

5. Add the Option 43 line with this syntax:

option 43 hex <hexadecimal string>

The hexadecimal string in Step 4 is assembled as a sequence of
the TLV values for the Option 43 sub-option: Type + Length +
Value. Type is always the sub-option code 0xf1. Length is the
number of controller management IP addresses times 4 in hex. Value
is the IP address of the controller listed sequentially in hex.

For example, suppose there are two controllers with management
interface IP addresses, 192.168.10.5 and 192.168.10.20. The type is
0xf1. The length is 2 * 4 = 8 = 0x08. The IP addresses translate
to c0a80a05 (192.168.10.5) and c0a80a14 (192.168.10.20). When the
string is assembled, it yields f108c0a80a05c0a80a14. The Cisco
IOS command that is added to the DHCP scope is:

option 43 hex f108c0a80a05c0a80a14

The above is consistent with how the same document explains the option
43 configuration for Microsoft DHCP servers (type is 241, or 0xf1),
but not at all consistent with the example quoted above for ISC's DHCP
server on Linux (see line 5 in that example; the type code there is 43!).

Given packet traces from a configuration that works using an IOS
DHCP server, I was able to easily pinpoint where our configuration was
incorrect in the ISC dhcp server, and that's when I went back to Cisco's
document, as I was sure I'd followed it (repeatedly) and configured our
DHCP server exactly as that document said we should.

Once I understood what it was that the access points expect to see,
corrected our configuration and tested, I decided to make a point to write
this up, for the benefit of any archives that will pick up this message,
and hopefully save someone else the same grief I suffered to discover
how to configure this correctly. Our DHCP server is now configured as
follows, for Cisco lightweight wireless access points:

1 option space CiscoAP;
2 option CiscoAP.server-address code 241 = array of ip-address;
3
4 class "cisco-aps" {
5 match if substring (option vendor-class-identifier, 0, 8) = "Cisco AP";
6 vendor-option-space CiscoAP;
7
8 switch (substring (option vendor-class-identifier, 8, 8)) {
9 case " c1240":
10 option CiscoAP.server-address 192.168.125.87;
11 break;
12 case " c1250":
13 option CiscoAP.server-address 192.168.3.2,192.168.3.4;
14 break;
15 case ".c1500":
16 case ".OAP1500":
17 case ".LAP1510":
18 case ".LAP1505":
19 case " c1520":
20 option CiscoAP.server-address 192.168.7.101,192.168.83.20;
21 break;
22 default:
23 option CiscoAP.server-address 192.168.7.101,192.168.83.20;
24 break;
25 }
26
27 # 1-week default; 10-days max.
28 default-lease-time 604800;
29 max-lease-time 864000;
30 option log-servers 192.168.1.5;
31 }

Notice line 2 above: the type code is 241 (not 43!) and the data type is
an "array of IP address", not "string". I've tried to create a flexible
configuration, with respect to how the value of option 43 is selected
according to the value of option 60 (the vendor-class-identifier, sent
by the DHCP client). Obviously other criteria can be used for that
selection, and I make sure to also provide a default (ie, we've entered
"class cisco-aps" because the VCI matched "Cisco AP", but if it didn't
match any of the specific model numbers in the switch statement, we
still point it to a couple of controllers as default).

I apologize for the length of this message, but I hope that the detailed
explanation will be useful to others, and that no one else struggles to
get this function working with Cisco's lightweight wireless access points.

--
----------------------------------------------------------------------
Sylvain Robitaille

Systems and Network analyst Concordia University
Instructional & Information Technology Montreal, Quebec, Canada
----------------------------------------------------------------------