Unix Insider –
In March we introduced the Address Resolution Protocol (ARP), a half-brother to the TCP/IP stack that maps the logical IP address space into the real world of Ethernet hardware addresses. We didn't venture very far away from the safe confines of a local area network and some simple troubleshooting, which is fine for introducing the relationships of ARP to IP and various network broadcast activities, but it doesn't describe the real real world very well. If you're reading this, you've encountered a byzantine complex of hosts, routers, desktops, and networks that will lay claim to some measure of high availability, redundancy and well-ordered connectedness. To keep the packets flowing from one end to the other, no matter what secret packet vacuum networks you encounter on the way, you'll need to play games with IP addresses or MAC addresses and fool clients' ARP requests into believing your temporary version of network reality -- however cobbled together it might be.
This month we're going to complicate our simple explanation of ARP with a closer look at stupid networking tricks, starting with the interactions between the ARP family of protocols and the boot process. We'll toss some routers into the fray, and look at ways of sending packets to IP networks that aren't quite where the routers think they are, and finally we'll show you how to run multiple IP networks on the same physical network, a necessary strategy for most networking transition and migration plans. We'll sprinkle hints and pointers through this conclusion to dealing with things that go bump, ARP, or otherwise collide in the night.
Double reverse slam-dunks: Reversing the ARP protocol
As we saw in my March column, ARP is used to locate the hardware, or MAC address for a given IP address. It takes the logical, 32-bit IP addresses like 192.9.200.1 and associates with it a 48-bit physical address like 8:0:20:45:7:b, uniquely identifying the machine with the noted IP address. ARP is the workhorse for establishing IP-level connections to new, previously uncontacted hosts.
What do you do if you know your own MAC address, but need to find an IP address to use? Such is the case with diskless clients, network computers, and PCs that are configured without a name service or file of hostnames and IP addresses -- they can look up their own MAC addresses in hardware, but have to ask some network authority to supply the correct IP address. The most common suppliers of the IP address are Sun's diskless workstation boot parameter (bootparam) remote procedure call, the bootp client booting protocol, and the Dynamic Host Configuration Protocol (DHCP). DHCP is described in RFC 983, and is at the heart of many PC-LAN based TCP/IP configurations. All of these protocols use Reverse ARP (RARP) to match the MAC address to a corresponding IP address.
RARP differs from ARP in more than just the direction of the lookup; while ARP is a full-fledged part of a TCP/IP implementation, RARP is almost always handled in a user-level process on the server side. You'll see the
<font face="Courier">in.rarpd</font>server daemon opening on /dev/dlpi (in Solaris) or /dev/nit (in SunOS), extracting raw RARP broadcast frames off the network. Why bother implementing a network layer protocol in user space? The short answer is "configuration control." RARP uses the /etc/ethers file or the
<font face="Courier">ethers</font>NIS map to correlate MAC addresses to IP addresses; this is maintained by you, our gentle reader, in the same manner that IP address-to-host name mappings are noted in a hosts file. The diskless client booting protocol uses other configuration files to indicate the location of the root and /usr filesystems, and DHCP has its own set of user-level files as well. More information can be found in the manual page for
<font face="Courier">rarpd</font>.
We're taking this detour through RARP land primarily for completeness. ARP failures tend to be dramatic and affect entire network segments -- and that's when you're just using the dynamically inserted ARP entries. We'll look at further complications shortly. ARP troubles appear as slow or intermittent network connections, hideous network file access performance, and the occasional broadcast storm. RARP difficulties, on the other hand, tend to blow up one machine at a time. You'll see machines that hang during boot or fail to boot cleanly, or can't configure network interfaces correctly.
If you see messages of the form
<font face="Courier">revarp: no response for MAC address 8:0:20:a:b:c </font>
while booting your desktop, it means that the system couldn't determine a valid IP address for itself during the boot sequence. Check your hostname configuration in /etc/nodename, /etc/hostname.le0 (or /etc/hostname.hme0 if you're Ultra-inclined) and in the /etc/hosts file; if there are inconsistencies your desktop may decide to ask the network to supply the missing IP bits. More likely, you have an empty /etc/hostname.le0 file, so the host configures its le0 interface but can't figure out the IP address from the (missing) host name. What you'll see if you look in /etc/init.d/rootusr is a line of the form
<font face="Courier">ifconfig -a auto-revarp </font>
that indicates the RARP protocol should be used to configure the built-in interfaces.
Proxy votes: Hiding hideous hairballs with proxy ARP
A proxy server is simply one that "stands in" for the real McCoy, much like a stockholder casts a proxy vote at the annual meeting of a publicly traded company instead of personally attending the meeting to vote in person. Proxy ARP is a publicly available package that allows a machine to answer ARP requests for others that can't be there to answer the requests on their own behalf.
Here's an ideal example: you're supporting several remote workstations connected over PPP links to a router attached to one of your LAN campus segments. To conserve IP networks, you'd really like to keep the remote machines on the same IP network as the local ones, merely "giving away" a few IP addresses to the remote machines and their PPP end points on the router. While this is ideal from a configuration management point of view, it's a nightmare for your routers and other machines that need to reach these remote workstations. Creating host routes for the remote machines only partially solves the problem; those machines on the same IP network aren't going to be looking for a router to get to machines that are ostensibly on the same wire.
Proxy ARP fits the bill nicely -- it allows you to designate one machine to answer ARP requests for the remote machines, providing the MAC address of the router with the PPP end points on it. The router can then handle forwarding to the appropriate destination. Why wouldn't you use published ARP entries on the proxy ARP server? Publishing an ARP entry has the same effect as using proxy ARP, namely, the server answers ARP requests for the IP address in the entry, supplying the matching MAC address (not its own). However, published ARP entries fill a slot in the ARP cache, making this solution minimally scalable. Publishing ARP entries is adequate for a handful of remote machines; to support several dozen you'll do better with a proxy ARP server so that the server's ARP cache doesn't become half-full of infrequently used entries.
The big zero: Multiple IP networks on a single wire
Proxy ARP services let you spread a single IP network over several physical segments, saving the virtual real estate of your IP address space while minimizing complexity. How do you do the opposite, that is, put several logical IP networks on the same physical network segment? Better yet, why would you want to do such a thing in the first place? Think about migrating to a new IP addressing scheme, a new network topology, or merging two networks together. You're going to go through a phase where you want old and new network numbers to live together on the same wire, or where you've consolidated two or more LANs onto the same physical segment while migrating to Fast Ethernet, ATM, or an FDDI ring design. As Ethernet switches gain in popularity, the creation of virtual LANs (VLANs) often requires running more than one IP network on a series of ports switched like a single segment; rather than renumber your machines you can work around the IP routing issues with a clever combination of route definition and ARP fiddling.
There are two ways to solve this IP migration problem: supply each machine with routing information so it knows that both networks are on the same wire, or give each machine an IP address on each network so that it has explicit connectivity to each logical network. The multiple IP address per network interface trick is useful when you need to make a machine appear with several names, for example, when you're hosting Web content for more than one named host.
To merge the IP networks with the routing solution, you use a zero-cost route that indicates the target network is directly connected to the local machine. Let's assume the first IP network is 192.9.200.0, and the "new" network is 201.2.14.0. To add a route to the 201-network, you'd insert the following line in /etc/init.d/inetsvc:
<font face="Courier">huey# route add network 201.2.14.0 `cat /etc/hostname.le0` 0 </font>
Zero-cost routes imply that the local machine will send an ARP request for each machine on the target network, since that network is accessible over the local le0 interface. Normally, a host will send an ARP request using the IP address of the designated router for the network; but in this case the router is the machine itself so it does the ARP for the destination IP address. Strange-looking ARP requests for IP addresses that shouldn't be on a particular network segment may be caused by a misplaced zero-cost route.
Here's a corner case where proxy ARP once again comes to the rescue: While merging two IP networks together, you have some "old" machines that have no need to reach other systems on the same network, but have to use the single router that has an IP address on the "new" IP network. That is, router 201.2.14.100 has to handle packets from hosts 192.9.200.x, and you don't want to create zero-cost routes on every one of those machines. Using proxy ARP or a published ARP entry on one of the 201.2.14.x machines, create a "fake" IP address for the router on the 192.9.200.0 network, say, 192.9.200.100. Then insert an ARP entry using the MAC address of the router, so that ARP requests for IP address 192.9.200.100 are answered with the Ethernet address of your router, and packets will be delivered to the router from hosts on the 192.9.200.x network. Make sure those "old" machines put 192.9.200.100 in their /etc/defaultrouter files, or otherwise manually set up a route through the "fake" IP address. Note that this address won't appear in RIP packets or other routing information, so you'll have to use a route command, an entry in /etc/gateways, or a default route to point your old machines at the router of multiple personalities.
More Multiple Personalities: Virtual interfaces
When you're renumbering a network or trying to make a machine impersonate multiple personalities, you need to create virtual interfaces on top of a single physical network interface. For SunOS users, you'll need a package called
<font face="Courier">vif</font>, written by John Ioannidis while at Columbia University. Solaris has this feature built into the
<font face="Courier">ifconfig</font>command and IP implementation. Virtual interfaces use notations of
<font face="Courier">le0:1, le0:2, le0:3</font>and so on to number the virtual interfaces created on top of the
<font face="Courier">le0</font>device. Solaris 2.5 and earlier releases have a limit of 256 virtual interfaces per physical network connection; this limit is relaxed in Solaris 2.5.1/Internet supplement and later releases.
Turn on the virtual interface by adding lines like the following to a new boot script called /etc/rc2.d/S99vif:
<font face="Courier">huey# ifconfig le0:1 hostname-net1 broadcast + netmask + huey# ifconfig le0:2 hostname-net2 broadcast + netmask + </font>
There's no need to "plumb" the virtual interface because the necessary STREAMS code was pushed onto the physical interface when it was configured earlier in the boot process. Virtual interfaces simply add more IP addresses to the same underlying device. The explicit
<font face="Courier">ifconfig</font>command sets the broadcast address and netmask. Virtual interfaces aren't handled nicely by the /etc/hostname.le0 convention, as the boot code that reads those files expects to plumb, configure, and RARP as needed. Life is simpler when you're telling these little white IP lies, so stick the configuration in a separate boot file.