When you run multiple IP networks on the same physical segment, you may confuse some machines' routing tables. A typical problem is messages like "packet from unknown router" that appear when a machine receives a packet that comes from an IP address that does not appear to be on the local network. In our previous example, any machine on the 192.9.200.0 network may receive a packet from a router using IP address 201.2.14.100, but become confused when it finds that it has no direct connection to network 201.2.14.0. Adding a second IP address on the local interface, or a zero-cost route will dissipate the messages.
When you add second and consecutive IP addresses to an interface, you cause it to answer ARP requests for all of those addresses. The Solaris kernel maintains a list of IP addresses per interface and matches incoming ARP requests against this list. You'll see the same MAC address appearing in ARP caches tied to several IP addresses if you use the virtual interface mechanism, and that multiplicity could impair network management discovery tools or other scripts that rely on ARP cache browsing to build their view of the local network.
ARP is part of the kernel like the virtual memory system or the filesystem cache -- ideally you never have to deal with it or know that it's there, but when it breaks it wreaks havoc on your day to day operations. Knowing where to poke it, how to watch for trouble and trace it back will help you keep too many things from bumping into each other in the dark.
This story, "ARP networking tricks" was originally published by ITworld.