Server's down: How do I find out what's wrong?

This article is excerpted from the book DevOps Troubleshooting: Linux Server Best Practices and is reprinted by permission of publisher Pearson/Addison-Wesley Professional.

Many different problems can creep up on a network, so network troubleshooting skills become crucial for anyone responsible for servers or services on servers attached to a network. Linux provides a large set of network troubleshooting tools, and this article discusses a few common network problems along with how to use some of the tools available for Linux to track down the root cause.

Problem: Server A can't talk to server B

Probably the most common network troubleshooting scenario involves one server being unable to communicate with another server on the network. This section will use an example in which a server named dev1 can't access the web service (port 80) on a second server named web1. Any number of different problems could cause this, so we'll run step by step through tests you can perform to isolate the cause of the problem.

Normally when troubleshooting a problem like this, you might skip a few of these initial steps (such as checking the link), since tests further down the line will also rule them out. For instance, if you test and confirm that DNS works, you've proven that your host can communicate on the local network. For this example, though, we'll walk through each intermediary step to illustrate how you might test each level.

Client or server problem?

One quick test you can perform to narrow down the cause of your problem is to go to another host on the same network and try to access the server. In this example, you would find another server on the same network as dev1, such as dev2, and try to access web1. If dev2 also can't access web1, then you know the problem is more likely on web1, or on the network between dev1, dev2, and web1. If dev2 can access web1, then you know the problem is more likely on dev1. To start, let's assume that dev2 can access web1, so we will focus our troubleshooting on dev1.

Is it plugged in?

The first troubleshooting steps to perform are on the client. You first want to verify that your client's connection to the network is healthy. To do this you can use the ethtool program (installed via the ethtool package) to verify that your link is up (the Ethernet device is physically connected to the network). If you aren't sure what interface you use, run the /sbin/ifconfig command to list all the available network interfaces and their settings. So if your Ethernet device was at eth0

$ sudo ethtool eth0
Settings for eth0:
     Supported ports: [ TP ]
     Supported link modes:   10baseT/Half 10baseT/Full 
                               100baseT/Half 100baseT/Full 
                               1000baseT/Half 1000baseT/Full 
     Supports auto-negotiation: Yes
     Advertised link modes:  10baseT/Half 10baseT/Full 
                               100baseT/Half 100baseT/Full 
                               1000baseT/Half 1000baseT/Full 
     Advertised auto-negotiation: Yes
     Speed: 100Mb/s
     Duplex: Full
     Port: Twisted Pair
     PHYAD: 0
     Transceiver: internal
     Auto-negotiation: on
     Supports Wake-on: pg
     Wake-on: d
     Current message level: 0x000000ff (255)
     Link detected: yes

Here, on the final line, you can see that Link detected is set to yes, so dev1 is physically connected to the network. If this was set to no, you would need to physically inspect dev1's network connection and make sure it was connected. Since it is physically connected, you can move on.