15 Unix tricks to get your new year off to a good start

tricks nicolas alejandro
Credit: flickr / Nicolas Alejandro

The Unix command line provides numerous ways to make our work easier. Here are 15 "tricks" that I use often to make quick work of various tasks

Display the last word in every line of a file

This one is pretty easy if you know the meaning of $NF. I came across this many years ago when I was building scripts entirely in awk.

awk '{print $NF}
awk -F: '{print $NF}'

NF for awk represents the number of fields (e.g., 5), so $NF then represents the value of the rightmost field (e.g., $5).

$ cat numbers
1 2 3 4 5
one two three four five
$ awk '{print $NF}' numbers
5
five

This trick makes it easy for you to print the rightmost word in a file even when each line might contain a different number of words. If you want to print some other word relative to the end of each line, you can still use NF as an anchor for your field selection. In this command, we select the next to last field by using NF-1.

$ awk '{print $(NF-1)}' numbers
4
four

Remove the Nth line line of a file

Removing a particular line from a file is a trick that sed handles easily. Assume numbers is a file with the numbers 1 through 11, one number per line. Using the command shown below, line 7 will disappear. The "7 d" command tells sed to remove the 7th line.

$ sed -i '7 d' numbers
$ cat numbers
1
2
3
4
5
6
8
9
10
11

You can remove a sequence of lines by putting a range in place of the 7. The command sed -i '7,9 d' numbers removes lines 7 through 9.

Remove blank lines from a file

There are several ways to remove blank lines from a file. One of the easiest is to use sed. In the command below, we use ^$ to represent a blank line. We do this because ^ represents the start of a line and $ represents the end of a line, so a line with a start and an end with nothing between is blank (as opposed to a line containing a sequence of blanks).

$ cat strings
This is the last day
of your life thus far

Mr. Logic
$ sed -i '/^$/d' strings
$ cat strings
This is the last day
of your life thus far
Mr. Logic

Using a sed command like the one shown below, however, you can both remove blank lines and create a backup of the original file.

$ perl -i.bak -n -e'print if /\S/' strings
$ ls -l strings*
-rw-r----- 1 shs faculty 53 Jan  4  2015 strings
-rw-r----- 1 shs faculty 54 Jan  4 18:36 strings.bak

Notice that, with the single blank line missing, the file is one character smaller than the original.

A grep command, similar to one of the sed commands shown above, also uses the ^$ to select blanks lines and the -v to select anything BUT these lines. This command then displays the text without the blank lines but doesn't change the original file.

grep -v '^$' myfile

You can also use an awk command like the one shown below. This will also display the text and ignore the blank lines but, like the grep command shown above, it won't change the original file. This works because $0 represents all of the words on the line (i.e., the entire line). You'd have to redirect the output if you want to store it as a new file.

$ awk '$0' strings
This is the last day
of your life thus far
Mr. Logic

Find symbolic links

Finding symbolic links on a Unix system is easy because the find command has an option for finding files by type. The type for a symbolic link is "l" just like the first letter that you see in a long listing for one of these files.

$ find . -type l -ls
12763168    0 lrwxrwxrwx   1 shs      staff         6 Jan 27  2013 ./hlinks/5 -> 5beers
12763423    0 lrwxrwxrwx   1 shs      staff         4 Nov 23  2013 ./projects/tmp -> /tmp

Find symbolic links that don't point to current files

It's possible on Unix system to create a symbolic link that doesn't point to an actual file or, of course, to create a symbolic link and then remove the original file. It's also possible to create a pair of symbolic links that point to each other. To track down any of these oddities, you can use a find command that joins forces with a Perl command.

find . -type l -print | perl -nle '-e || print';

In this command, we're using find to look for symbolic links and sending the results to a perl command that determines whether the target of the link exists or not -- and prints the link's name if it doesn't.

Determine the architecture (32 or 64 bit) of your system

This one's very easy. Use the arch command and you should see something like this:

$ arch
i686

This works on Linux anyway. When I run this command on Solaris, I get an answer like "sun4".

Reverse a string

The rev command reverses a string completely -- one letter at a time. You can pipe text to it like this:

$ echo "dlroW ,olleH" | rev
Hello, World

Alternately, you can provide the rev command with a file name and it reverses the text completely.

$ rev strings
yad tsal eht si sihT
raf suht efil ruoy fo

cigoL .rM

Count how many time a word appears in a file

This is trickier than it might first appear. For one thing, a word might be part of another word -- such as "the" being part of "there". For another, a word might appear more than once on a line. Will your command count each instance or just count the line as one hit? You can use grep -c "word" filename command, but it 's basically doing the same thing as grep word filename. It will only tell you how many lines the word appears on, not how many times the word appears as we see here:

$ cat words
the the the the the then there they
$ grep -c the words
1

Alternatively, you could do something like this:

#!/bin/bash

count=0
look4=$1
file=$2

for word in `cat $file`
do
  if [ $word == "$look4" ]; then
    ((count++))
  fi
done
echo $count

In this script, the for loop has us looking at every word and only increments the count if the word exactly matches what we're looking for. The problem with scripts is that they might not be generic enough to useful very often. At leat this one lets you pass your word and file name as arguments, though you have to pass them in the correct order.

Replace all instances of a word with another word

For many such tasks, I will use a simple command like this one:

sed s/this/that/g filename

That will often work, but it will change any instance of "this" to "that" whether or not the instances of "this" are words or parts of words such as "thistle". You can get really complicated with sed commands that are particular about the context in which the words they are out to change appear. Here's an example which is a bit more complicated. It uses the \b (word boundary) marker to ensure we only change "the" to "why" when it's an entire word.

$ cat words
the the the the the then there they
$ sed -e 's/\bthe\b/why/g' words
why why why why why then there they

Note that the string "the" wasn't changed when it was part of a longer word.

If you want to change "The" as well as "the", you can add the "ignore case" (i) option to your sed command.

$ sed -e 's/\bthe\b/why/g' words
The why why why why then there they
$ sed -e 's/\bthe\b/why/gi' words
why why why why why then there they

By the way, the sed command can change all instances of a string on a line of text, just the first, or the second or the third, etc. To change only the second, for example, you would use a 2 (2nd) instead of a g (global) in the last position of you s/this/that/g command -- s/this/that/2.

NOTE: The sed command can display the modified text or it can modify the original file. The -i option tells sed to edit the file "in place" -- rather than requiring you to redirect the output to another file if you want to save it.

Make a variable available from a subshell

Subshells won't recognize variables that you set up in the shell unless you export them.

$ cat showme
#!/bin/bash

echo $something

$ something="well done"
$ ./showme

$ export something
$ ./showme
well done

Explain why a hard link can't span file systems

A lot of people seem totally confused about the nature of hard links. I sometimes try to explain them as multiple file system incarnations of single files. But why they can't reach across file system boundaries is due to their nature. They, like all files on a system look to inodes to find and define the files they represent. And inodes are specific to file systems. It's the symbolic links that are the oddballs. They can point to files in other file systems because their content is nothing but the file system path to the files they "point" to.

Judge how busy a network interface is

This can be a bit tricky. The best solution is probably to install a tool that does this for you. One thing you can do is look at the output of the ifconfig command. Along with all the addressing information, you'll see some stats that tell you how many packets have been received and transmitted. That will give you some idea how busy the interface has been, but will likely only tell you what you want to know if you also know how long those statistics have been collected -- since the system was last booted.

$ ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:16:35:69:BD:79
          inet addr:192.168.0.11  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::216:35ff:fe69:bd79/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:11577802 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11656938 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2819912413 (2.6 GiB)  TX bytes:3530480976 (3.2 GiB)
          Interrupt:217 Memory:fdef0000-fdf00000

This probably isn't all that useful unless you're comparing interfaces or the system hasn't been up for very long.

View swap space available and in use

The swapon command can list your swap partitions and files.

$ swapon -s
Filename                                Type            Size    Used    Priority
/dev/cciss/c0d0p3                       partition       4192956 41152   -1

On Solaris systems, use the swap -l command.

# swap -l
swapfile             dev  swaplo blocks   free
/dev/dsk/c0t2d0s1   136,1      16 4177904 4177904?

Determine which process is using a file

The lsof command was nothing short of an amazing when I came across it ten years or so ago. Standing for "list open files", the lsof command comes in extremely handy when you want information about your devices or to find out what a particular user is accessing at some point in time. And it has a plethora of command options:

SYNOPSIS
     lsof [ -?abChlnNOPRstUvVX ] [ -A A ] [ -c c ] [ +|-d d  ]  [
     +|-D D ] [ +|-f [cfgGn] ] [ -F [f] ] [ -g [s] ] [ -i [i] ] [
     -k k ] [ +|-L [l] ] [ -m m ] [ +|-M ] [ -o [o] ] [ -p s ]  [
     +|-r  [t]  ]  [ -S [t] ] [ -T [t] ] [ -u s ] [ +|-w ] [ -- ]
     [names]

Without options, the lsof command will list all open files (i.e., files that are being accessed by some process).

server# lsof | more
COMMAND     PID   USER   FD   TYPE        DEVICE SIZE/OFF    NODE NAME
sched         0   root  cwd   VDIR         136,0     2048       2 /
init          1   root  cwd   VDIR         136,0     2048       2 /
init          1   root  txt   VREG         136,0   553268  902896 / (/dev/dsk/c0t2d0s0)
init          1   root  txt   VREG         136,0     5008  167126 / -- libdl.so.1
init          1   root  txt   VREG         136,0   266140  167125 / -- ld.so.1
init          1   root    0u  FIFO         136,0      0t0   45285 / (/dev/dsk/c0t2d0s0)
pageout       2   root  cwd   VDIR         136,0     2048       2 /
fsflush       3   root  cwd   VDIR         136,0     2048       2 /
syseventd    61   root  cwd   VDIR         136,0     2048       2 /
syseventd    61   root  txt   VREG         136,0    23272  179817 / (/dev/dsk/c0t2d0s0)
syseventd    61   root  txt   VREG         136,0    12384  185871 / -- modules/picl_slm.so

You can also look to see what processes are using a particular file.

server :>:# lsof /bin/bash
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
monVMweb    456 root  txt   VREG  136,0   516392 896720 /./usr/bin/bash
monTracke   457 root  txt   VREG  136,0   516392 896720 /./usr/bin/bash
monProxy    616 root  txt   VREG  136,0   516392 896720 /./usr/bin/bash
bash      18172 root  txt   VREG  136,0   516392 896720 /./usr/bin/bash

With the -i (Internet) option, lsof competes with commands like netstat for reporting on network connections.

Related:
1 2 Page 1
Windows 10 annoyances and solutions
Shop Tech Products at Amazon
Notice to our Readers
We're now using social media to take your comments and feedback. Learn more about this here.