Skip the navigation

Server's down: How do I find out what's wrong?

By Kyle Rankin
January 28, 2013 05:55 AM ET

Test the remote host locally

At this point, we have either been able to narrow the problem down to a network issue or we believe the problem is on the host itself. If we think the problem is on the host itself, we can do a few things to test whether port 80 is available.

Test for listening ports

One of the first things you should do on web1 is test whether port 80 is listening. The netstat -lnp command will list all ports that are listening along with the process that has the port open. You could just run that and parse through the output for anything that is listening on port 80, or you could use grep to show only things listening on port 80:

$ sudo netstat -lnp | grep :80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 919/apache

The first column tells you what protocol the port is using. The second and third columns are the receive and send queues (both are set to 0 here). The column you want to pay attention to is the fourth column, as it lists the local address on which the host is listening. Here the 0.0.0.0:80 tells us that the host is listening on all of its IPs for port 80 traffic. If Apache were listening only on web1's Ethernet address, you would see 10.1.2.5:80 here.

The final column will tell you which process has the port open. Here you can see that Apache is running and listening. If you do not see this in your netstat output, you need to start your Apache server.

Firewall rules

If the process is running and listening on port 80, it's possible that web1 has some sort of firewall in place. Use the iptables command to list all of your firewall rules. If your firewall is disabled, your output will look like this:

 sudo /sbin/iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Notice that the default policy is set to ACCEPT. It's possible, though, that your firewall is set to drop all packets by default, even if it doesn't list any rules. If that is the case you will see output more like the following:

$  sudo /sbin/iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination

Chain OUTPUT (policy DROP)
target     prot opt source               destination

On the other hand, if you had a firewall rule that blocked port 80, it might look like this:

$ sudo /sbin/iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
REJECT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 reject-with(
icmp-port-unreachable

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Clearly, in the latter case you would need to modify the firewall rules to allow port 80 traffic from the host.

Troubleshoot slow networks

In a way, it's easier to troubleshoot network problems when something doesn't work at all. When a host is unreachable, you can perform the troubleshooting steps discussed earlier until the host is reachable again. When the network is just slow, however, sometimes it can be a bit tricky to track down why. This section discusses a few techniques you can use to track down the cause of slow networks.

DNS issues

Although DNS is blamed more often than it should be for network problems, when DNS does have an issue, it can often result in poor network performance. For instance, if you have two DNS servers configured for a domain and the first one you try goes down, your DNS requests will wait 30 seconds before they time out and go to the secondary DNS server. Although this will definitely be noticeable when you run tools like dig or nslookup, DNS issues can cause apparent network slowdowns in some unexpected ways; this is because so many services rely on DNS to resolve hostnames to IP addresses. Such issues can even affect your network troubleshooting tools.

Ping, traceroute, route, netstat, and even iptables are great examples of network troubleshooting tools that can degrade during DNS issues. By default, all of these tools will attempt to resolve IP addresses into hostnames if they can. If there are DNS problems, however, the results from each of these commands might stall while they attempt to look up IP addresses and fail. In the case of ping or traceroute, it might seem like your ping replies are taking a long time, yet when they do finally come through, the round-trip time is relatively low. In the case of route, netstat, and iptables, the results might stall for quite some time before you get output. The system is waiting for DNS requests to time out.

Our Commenting Policies