DEV.MD / Docs / UNIX Shell Quote

UNIX SHELL Quote Tutorial

Written by Bruce Barnett

Table of Contents

Copyright © 1991, 2001 Bruce Barnett and General Electric Company

All Rights reserved

Thanks to Jesse Silverman for corrections.

Last corrections: Thu Nov 22 09:19:32 EST 2007

I had planned to discuss regular expressions with you this month. I realized, however, that unless you fully understood the Unix quoting mechanism, you might become confused. Regular expressions use meta-characters. The shells also have meta-characters. Meta-characters are simply characters that have a special meaning. The problem occurs when you want to use a regular expression in a shell script. Will the shell do something special with the character? Or will it be passed unchanged to the program? The "$" character is a good example. It could be the beginning of a variable name, or it could be part of a regular expression. If you need a regular expression, you must know if any of the characters of the expression are meta-characters, and must know the right way to quote that character, so that it is passed to the program without being modified by the shell.

Here is a chart of the meta-characters the Bourne and C shell know about. I have also included several combinations of characters just to make this table more complete. There is a lot of detail on this chart.

List of Special Characters and what they mean
Character Where Meaning
csh, sh Execute command
# csh, sh, ASCII files Start a comment
csh, sh Argument separator
` csh, sh Command substitution
" csh, sh Weak Quotes
' csh, sh Strong Quotes
\ csh, sh Single Character Quote
variable sh, csh Variable
variable csh, sh Same as variable
| csh, sh Pipe character
^ sh Pipe Character
& csh, sh Run program in background
? csh, sh Match one character
* csh, sh Match any number of characters
; csh, sh Command separator
;; sh End of Case statement
~ csh Home Directory
~user csh User's Home Directory
! csh History of Commands
- Programs Start of optional argument
$# csh, sh Number of arguments to script
$* csh, sh Arguments to script
$@ sh Original arguments to script
$- sh Flags passed to shell
$? sh Status of previous command
$$ sh Process identification number
$! sh PID of last background job
&& sh Short-circuit AND
|| sh Short-circuit OR
. csh, sh Typ. filename extension
. sh Source a file and execute as command
: sh Nothing command
: sh Separates Values in environment variables
: csh Variable modifier
Character Where Meaning
[ ] csh, sh Match range of characters
[ ] sh Test
%job csh Identifies job Number
(cmd;cmd) csh. sh Runs cmd;cmd as a sub-shell
{ } csh In-line expansions
{cmd;cmd } sh Like (cmd;cmd ) without a subshell
>file csh, sh Standard output
>>file csh, sh Append to standard output
<file csh, sh Standard Input
<<word csh, sh Read until word, substitute variables
<<\word csh, sh Read until word, no substitution
<<-word sh Read until word, ignoring TABS
>>!file csh Append to file, ignore error if not there
>!file csh Output to new file, ignore error if not there
>&file csh Send standard & error output to file
<&digit sh Switch Standard Input to file
<&- sh Close Standard Input
>&digit sh Switch Standard Output to file
>&- sh Close Standard Output
digit1<&\fIdigit2\fP sh Connect digit2 to \fIdigit1\fP
digit<&- sh Close file digit
digit2>&\fIdigit1\fP sh Connect digit2 to \fIdigit1\fP
digit>&- sh Close file digit

I am not going to cover each one of these special meta-characters. I'll save that for a future column, perhaps. What is important is a solid understanding of the characters that have these special meanings. I will also discuss how you can verify the shell is interpreting the special characters, so you can pinpoint where your problem lies.

There are three different mechanisms used for quoting characters. One of them is not the use of the back quote (more properly called backslash) character "`". That character is used for command substitution:

% echo the date is `date`
The three quoting mechanisms you can use are the backslash, single quote, and double quote.

Quoting a single character with the backslash

You can prevent the shell from interpreting a character by placing a backslash ("\") in front of it. Here is a shell script that can delete any files that contain an asterisk:
echo This script removes all files that
echo contain an asterisk in the name.
echo
echo Are you sure you want to remove these files\?
rm -i *\**
The backslash was also necessary before the question mark, which is also a shell meta-character. Without it, the shell would look for all files that match the pattern "files?." If you had the files "files1" and "files2" the script would print out
Are you sure you want to remove these files1 files2

which is not what you want.

The backslash is the "strongest" method of quotation. It works when every other method fails. If you want to place text on two or more lines for readability, but the program expects one line, you need a line continuation character. Just use the backslash as the last character on the line:

% echo This could be \
a very \
long line\!
This could be a very long line!
%
This escapes or quotes the end of line character, so it no longer has a special meaning. In the above example, I also put a backslash before the exclamation point. This is necessary if you are using the C shell, which treats the "!" as a special character. If you are using some other shell, it might not be necessary.

Strong Quoting with the Single Quotes

When you need to quote several character at once, you could use several backslashes:
% echo a\ \ \ \ \ \ \ b
(There are 7 spaces between 'a' and 'b'.) This is ugly but works. It is easier to use pairs of quotation marks to indicate the start and end of the characters to be quoted:
% echo 'a b'
(The HTML ruins the formatting. Imagine that there are 7 spaces between the a and b. -Bruce) Inside the single quotes, you can include almost all meta-characters:
% echo 'What the *heck* is a $ doing here???'
What the *heck* is a $ doing here???
The above example uses asterisks, dollar signs, and question marks meta-characters. The single quotes should be used when you want the text left alone. If you are using the C shell, the "!" character may need a backslash before it. It depends on the characters next to it. If it is surrounded by spaces, you don't need to use a backslash.

Weak Quotes with the Double Quotes

Sometimes you want a weaker type of quoting: one that doesn't expand meta-characters like "*" or "?," but does expand variables and does command substitution. This can be done with the double quote characters:
% echo "Is your home directory $HOME?"
Is your home directory /home/kreskin/u0/barnett?
% echo "Your current directory is `pwd`"
Your current directory is /home/kreskin/u0/barnett
Once you learn the difference between single quotes and double quotes, you will have mastered a very useful skill. It's not hard. The single quotes are stronger than the double quotes. Got it? Okay. And the backslash is the strongest of all.

Using quotes to include spaces and characters in filenames

If you want to work with files with spaces or special characters in the filename, you may have to use quotes. For instance, if you wanted to create a file with a space in the name, you could use the following:
% cp /dev/null 'a file with spaces in the name'
Normally, the shell uses spaces to determine the end of each argument. Quoting changes that, and the above example only has two arguments. You can also use a backslash before the character. The example below will rename a file with a space in the name, changing the space to an underscore:
% mv a\ file a_file

Using the same techniques, you can deal with any character in a filename:

% mv a 'a?'
At worst, a space in a file makes it difficult to use as an argument. Other characters are very dangerous to use in a filename. In particular, using "?" and "*" in a filename is playing with fire. If you want to delete the file "a?" you may end up deleting more than the single file.

Quotes within Quotes

While having two types of quotes (three if you count the backslash) might seem confusing, in reality it provides you with several ways to solve the same problems. You can put either quotes inside the other. If you want to quote single quotes, use double quotes around it. To quote double quotes, use single quotes. Heck, it's easier to show you:
% echo "Don't do that"
Don't do that
% echo 'The quote of the day is: "TGIF"'
The quote of the day is: "TGIF"
%

Finding out if your quotes are wrong

In some cases, you may need to use the backslash when you are not sure. In other cases, a backslash will do the wrong thing. How can you find out if you are quoting things correctly? The answer: use the shell.

An easy way to check quotes is to add an "echo" before the command so you can see what is happening, or change an "ls" command into an "echo" command:

echo rcp gateway:\*.tar.Z .
rsh -n cruncher echo ls \*
rsh -n cruncher echo 'ls *'
If you want to do file redirection on a remote machine, echo isn't sufficient. The command
rsh -n cruncher echo 'ls * >/tmp/file'
sends the results of the echo to the file when you wanted the command echoed to your terminal. You need to nest the quotes:
rsh -n cruncher "echo 'ls * >/tmp/file'"
rsh -n cruncher 'echo "ls * >/tmp/file"'
rsh -n cruncher "echo 'cd newdir;ls * >>/tmp/file'"

If you are debugging a shell script, and you want to see what your script is doing, you can duplicate one of the important lines in your script and insert an "echo" in front of one of the duplicates. Doing this one or two times in a script isn't very difficult, but there are times when you want to watch every line of your script. In this case, just ask the shell to show you want is going on.

The verbose and echo variables in the C shell

The C shell has two variables that, when set, will help you follow the convoluted trail of variable and meta-character expansion. The command
set verbose
will echo every line of your script before the variables have been evaluated. The command
set echo
will display each line after the variables and meta-characters have been substituted. If you wish to turn the variables off, use unset instead of set

A convenient way to turn these variables on the first line of the script using the the "-x" option (echo)

#!/bin/csh -x

or the "-v" option (verbose):

#!/bin/csh -v
In both examples above, the .cshrc file is read at the beginning of the script. The "-f" option can skip this file. You can combine all three options if you like:
#!/bin/csh -fxv
If you want to read in the .cshrc file, and want to trace the values of these variables, capitalize the "X" and "V" variables. This turns on tracing before the .cshrc file is read:
#!/bin/csh -XV

It is not necessary to modify the program if you want to turn on the verbose or echo variables. If this is a script that you do not have the permissions to modify, you can set these variables from the command line:

% csh -x shell_script

The Bourne Shell variables

You can enable variable expansion to Bourne shell scripts the same way:
% sh -v script
% sh -x script
Inside a Bourne shell script, the syntax is different. To turn on the verbose flag, use
set -v
to turn on the echo variable, use:
set -x
If you want to turn these variables off, use a plus instead of a minus:
set +x
set +v

Including identical quotes within quotes

One problem people have is including the same quotes within quotes. Many expect the following to work:
echo "The word for today is \"TGIF\""
echo 'Don\'t quote me'
The first example works with the Bourne shell, but not the C shell. The second example doesn't work for either of them. I bet many of you programmers are confused by this. All of us are very familiar with strings in programming languages like C. This is why we get confused. The quotes turn substitution on and off. They are not used to indicate the starting and ending of a string. Consider the following set of quotes:
echo 'a'b'c'
This is broken up into three units. The first and last are quoted, and the middle is not. After quoting and substitution occurs, the three units are combined. The middle can be a variable, for instance:
echo 'a'$HOME'b'
This technique is a typical way to get a shell variable into an awk script. Here is a simple shell script that demonstrates this. Please study this, as it is important: 
#!/bin/sh
# this is a shell script that acts like a filter,
# but in only prints out one column.
# the value of the column is the argument 
# to the script
#
# uncomment the next line to see how this works
#set -x
#
# example:
#       printcol 1
#       printcol 3
# the value of the argument is $1
# Here comes the tricky part -
awk '{print $'$1'}'
# I told you!
In this example, the shell breaks up the argument to awk into three pieces: pieces:
{print $ Quoted
$1 Evaluated
} Quoted
You can uncomment the command "set -x" and try the script with the command
printcol 2 </etc/hosts
The argument to the shell script is 2, so "$1" is evaluated and returns the value "2." This makes the argument to awk the string "print {$2}" and the second column is printed out.

You must understand this when you want to have quotes within quotes. In fact, you don't want to put quotes within quotes, you want to combine or concatenate several units into one argument.

Let me rephrase that. If you want to include a single quote in an argument that starts with a single quote, you must turn off the mechanism started by the single quote, and use a different quoting method. Remember, the backslash is the strongest of all quoting mechanisms. You can quote anything with the backslash. This example quotes all three quote characters: 

% echo \'\"\\
Where the results are 
'"\
You can always use the backslash to quote a character. However, within the single quote mechanism, "\'" does not "quote the quote." The proper way to do this is as follows:
% echo 'Don' \' 't do that'
Don ' t do that
I put in a few extra spaces, so you could follow what was happening. Here it is again without the extra spaces:
% echo 'Don'\''t do that'
Don't do that
Just remember to match the quotes together when you mentally parse a shell script. This also works with double quotes:
% echo "The quote for today is "\"TGIF\"
The quote for today is "TGIF"
Or, if you want to put quotes around "TGIF:"
% echo "The quote for today is "\""TGIF"\"
The quote for today is "TGIF"
Note that this technique works for any shell using either one of the quotes.

Quoting long lines

Most Unix programs also use the backslash to escape a special characters. It is common for a Unix utility to interpret a backslash on the end of a line as a line continuation character--that is, the end of line character is quoted or escaped so the standard meaning is prevented.

The Bourne shell and C shell behave differently when you are quoting lines that continue beyond the end of line marker. The C shell will not extend a quote beyond the line unless the last character is a backslash: 

% echo "A quote \
on two lines"
A quote 
on two lines
The Bourne shell does allow quotes to extend beyond lines:
$ echo "A quote
> on two lines"
A quote
on two lines
Notice how the Bourne shell prompts you with a ">" when the quote is not closed. You can argue among yourselves which behavior is correct. I can understand the rational behind the C shell's reason for disallowing multi-line quoting by default. The earlier example:
echo 'Don\'t do that'
will generate an error if you use the C shell. However, if you use the Bourne shell, you will be given a ">" prompt and this will continue until you type in a single quote. This can be confusing for new Unix users.

I find the Bourne shell easier to use when I write multi-line quotes which are awk scripts: 

#!/bin/sh -x
#This script counts how many people
# are in the group specified as the first argument
grp=${1:?"Missing argument"}	# get group ID number
# If missing, report an error and exit.
awk -F: '
# Awk script starts here
BEGIN {
	# set total to zero 
	#   before we start
	total=0;	
}
$3 ~ /^'$grp'$/ {total++;} 
END {
	# end of file, print total
	printf("Total: %d\n", total);
}' </etc/passwd
This example uses the "$grp" shell variable right in the middle of the awk script. This is a common method of passing shell variables into the middle of an awk script.

HERE IS documents

There is another type of quote the shells support. There are called Here is documents. There are used when you need to read something from standard input, but you don't want to create a file to provide that input. There are also used to create files in a shell script. This can be done by the "<<" character, followed by a special word:
% sort >file <<EndOfSort
zygote
abacus
EndOfSort
This is very useful because variables are evaluated during this operation. Here is a way to transfer a file using ftp from a shell script:
#!/bin/sh
# Usage:
# ftpfile machine file
# set -x
SOURCE=$1
FILE=$2
BFILE=`basename $FILE`
ftp -n $SOURCE <<EndFTP
ascii
user anonymous $USER@`hostname`
get $FILE /tmp/$BFILE
EndFTP
As you can see, variables and command substitutions are done. If you want the quoted text to be left alone, put a slash in front of the name of the word:
cat >file <<\FunkyStriNG
Notice the funky string. This is done just in case the text contains the word that ends the file. It is very unlikely that I will want to put that particular combination of characters in any file. You should be warned that the C shell expects the word to be escaped. i.e. "\FunkyStriNG" while the Bourne shell does not. It expects to see "FunkyStriNG."

When to quote?

As you can see, this quoting business can cause a lot of confusion. At this point, you should know how to get the shell to recognize the exact string you want. When I cover regular expressions next month, you will be able to follow the quotes without losing your mind.