Unix Shell Scripts

You can't really do scripts in an hour but here goes.

Introduction

Shell scripts are more or less the same as other types of scripts or macros. In the basic sense, its just a list of commands. If you find yourself doing the same thing over and over or have something very complex to do and don't want to make a mistake, a script is a good way to save time and aggravation.

There are several different types of shells that more or less do the same thing. The purists will say that you never should script in anything other than bourne shell (sh). Some of the more serious criticisms against non-sh scripts are poor handling of file descriptors (e.g. stdin and stderr) and redirects. Indeed if you find yourself putting your scripts in /etc/rc, running them as cron jobs, or putting them on the web, then the purists are absolutely correct. Most of us don't need to do that so, at the risk of the wrath of the sysadmin hordes, I think you should just script in whatever shell you're most comfortable with and gets the job done. Otherwise, you can burn alot of time learning the idiosyncracies between them instead of getting the job done. Since most people at CERI have csh (or tcsh) we'll give our examples in that and, where possible, provide the equivalent sh.

To make a script you open your favorite editor (there's equal zeolotry among nerds on which is best: vi of course), type in your commands, save it, and run it. There are two ways to run it:

mwithers@smeagol> tcsh myscript.csh

or

mwithers@smeagol> myscript.csh

In the first instance we're telling the computer to run myscript using the tcsh. This is ok if I know it was written in tcsh but it will fall over if it wasn't. The second is probably better but requires two things.

1. You need to make it executable. Use:

mwithers@smeagol> chmod u+x myscript.csh

(or chmod 755 myscript.csh to be a nerd and make it executable for the group and other too).

2. You need a line at the very start of the script to launch the shell that will run the script. It should look something like this:

#!/usr/bin/tcsh -f

or in sh

#!/usr/bin/sh

Note that the actual path to tcsh or sh could be /bin and/or /usr/bin on nearly any unix system (that's been probably set up).

Following that first line, all lines that begin with a # are comments. A comment can begin in the middle of a line too and will continue until the end of that line.

Here's an example:

#!/usr/bin/tcsh -f
# this is a comment
echo "Begin myscript.csh" # this is a comment too.

Or in sh

#!/usr/bin/sh
# this is a comment
echo "Begin myscript.sh" # this is a comment too.

There are other idiosyncracies to the echo command like using it with out without quotes, more or less white space, handling control characters, etc. See the man page. Some prefer to use printf instead of echo. This works more like the printf statement in C and is a bit more useful.

Variables

There are a few very rare instances where you can write a useful script without variables. Some you can set as sort of definitions in the beginning and others you can set and reset on the fly in the script. Note that unlike compiled languages, you don't typecast variables (e.g. int, float, char). They are all strings no matter what you want (though you can still do some basic arithmetic, indexing, and counting). If you don't like that, then you should consider an different tool like perl, or C.

You set your variables equal to something using an equals symbol (surprise, surprise). In csh you also have to use "set". Here we'll define a variable and print it to the screen.

set basedir = /gaia/home/mwithers/somedir
printf "basedir = %s\n" $basedir

or in sh

basedir=/gaia/home/mwithers/somedir
printf "basedir = %s\n" $basedir

You might observe that once you set a variable, you can return its contents with the $.

You can also set a variable equal to the output of a command using back ticks (open single quote). In this example, we'll set basedir equal to the output of the shell command pwd (print working directory, aka dot).

set basedir = `pwd`
echo $basedir

or in sh

basedir=`pwd`
echo $basedir

There are several special variable types. Notable ones are environment variables which are also available to your shell scripts. Be careful with dependencies on these though since minor changes could vary the behavior of your script and you may not find out about it until long after you forgot you changed your environment (plus, it may not work for others). Other than environments there are command line variables and return variables.

The number of command line arguments typed are contained in the $#argv (csh only) or just $#. From there the strings typed in are counted from 0 to N-1 where 0 is the first thing typed, 1 the second, etc. Or, 0 is the name of the script, 1 is the first argument, 2 the second, etc. For example

echo my name is $0
echo I was given $# arguments
echo The first one is $1
echo The second is $2

or in sh

echo my name is $0
echo I was given $# arguments
echo The first one is $1
echo The second is $2

If

If your script uses command line arguments, then its good to check for them and print a usage statement if they aren't found. We can use an if statement (i.e. a conditional) to do this.

if ($# != 2) then
    echo "usage $0: "
    echo "where arg1 is this and arg2 is that"
    exit -1
else
    set this = $1
    set that = $2
    echo "processing $this and $that"
endif

or in sh

if [$# -ne 2]; then
    echo "usage $0: "
    echo "where arg1 is this and arg2 is that"
    exit -1
else
    set this = $1
    set that = $2
    echo "processing $this and $that"
fi

Here we notice some fairly signifcant syntactic differences between csh and sh. You have been warned. You can also use the else if conditional.

if ($# == 2) then
    set this = $1
    set that = $2
    echo "processing $this and $that"
else if ($# == 1) then
    set this = $1
    echo "processing with just $this"
else if ($# == 0) then
    echo "processing with neither this nor that"
else
    echo "$0 detected id10t error"
endif

or in sh

if [ $# = 2 ]; then
    this=$1
    that=$2
    echo "processing $this and $that"
elif [ $# = 1 ]; then
    this = $1
    echo "processing with just $this"
elif [ $# = 0 ]; then
    echo "processing with neither this nor that"
else
    echo "$0 detected id10t error"
fi

If you're working with a file (or directory) you can also use if to check existence, permissions, etc. In this example we use the ! to indicate a "not" and a -f to test for existence (this is really just a front end to the test shell command).

set myfile = $basedir/myfile
if( -e $myfile) then
    echo "working with $myfile"
else
    echo "Error $0: can't find $myfile"
endif

or in sh

myfile=$basedir/myfile
if [ ! -f $myfile ]; then
    echo "working with $myfile"
else
    echo "Error $0: can't find $basedir/myfile"
fi

There is alot more to if than explained here (e.g. tests for read, write, execute; -r, -w, -x respectively, etc, etc). See the man page or a more complete tutorial.

Loops

There are two primary tools for doing loops: for and while.

With for you loop over some explicit or implicit set of elements.

foreach i (1 2 3 4 5)
    echo "Working on $i"
end

or in sh

for i in 1 2 3 4 5
do
    echo "Working on $i"
done

You could also use a while along with the arithmetic operator to do the same thing.

@ N=5
@ i=0
while ($i < $N)
    echo "Working on $i of $N"
    @ i++
end

or in sh

N=5
i=0
while [ $i -lt $N ]
do
    echo "Working on $i of $N"
    i=`expr $i + 1`
done

One I find myself using frequently is looping over a bunch of files.

#!/usr/bin/tcsh -f

set basedir = /gaia/data/seisnet/
set mynet = NM
foreach yeardir (`ls $basedir/$mynet`)
    foreach evtdir (`ls $yeardir`)
        echo "do something with $evtdir"
    end
end

or in sh

#!/usr/bin/sh

basedir=/gaia/data/seisnet/
mynet=NM
for yeardir in `ls $basedir/$mynet`
do
    for evtdir in `ls $yeardir`
    do
        echo "do something with $evtdir"
    end
end

You can write even more useful scripts by taking advantage of other programs (as opposed to just shell commands). Keep in mind though that if you can do it with a shell command, it will be faster than loading another program. It makes no appreciable difference if you call an external program only once. It may make a difference if done in a loop or a nested loop. Awk and grep could be considered to have their own language but are still useful in a script.

foreach mycol (`awk '{print $1}' somefile`)
    echo "do something with my col"
end
and similarly for sh.

Other

You can prompt for input from your script by doing something like this.

echo "Would you like to exit this script now?"
set answer = $<
if ( $answer == "y" ) then
    echo "Exiting..."
    exit 0
endif

Or in sh.

echo "Would you like to exit this script now?"
read answer
if [ "$answer" = y ]
then
    echo "Exiting..."
    exit 0
fi

To check and verify the return status of a command, do something like this.

find . -name $0 -print
if ( $status != 0 ) then
    echo "find failed"
    exit 1
endif

Or in sh.

find . -name $0 -print
if [ $? -ne 0 ] ; then
    echo "find failed"
    exit 1
fi

You could do something in sac. But be careful because this slows things down considerably so depending on what you're doing, it could be time to write some C or C++. With sac it would be better to have a generic macro and call it using blackboard variables.

set sacmacro = $basedir/mymacro.sac
foreach sacfile (`ls $evtdir`)
    sac $sacmacro $arg1 $arg2
end

Or if you really need to use a program that prompts for input (and hopefully you never ever write one of those without also allowing some automated input mechanism) then, again, take care because this is even slower. But it does work. Here is a snippet from a script that runs fpfit and we'll leave the details on how to actually define the variables as an exercise.

echo running fpfit
$fpfit << END >> $logfile
hyp '$arcfile'
out '$prefix.out'
sum '$prefix.fps'
pol '$arcfile.pol'
fit '$arcfile.fit'
for 2
rep 1
fps
sto
END

There is much more you can do such as make use of arrays (what do you think $argv[0], $argv[1], $argv[$N] might be?), make use of return values, etc, etc. But hey we only had an hour. Here are some words of advice.

• Avoid hard-wires. Set variables on the fly if you can. If it can't be determined on the fly, make it an input arg. If it can't be an input arg and you really have to to make a hard-wire, set it at the beginning of the script so its at least easy to change.
• Put all your scripts in a "shell" or "scripts" directory separate from compiled binaries. Alternatively, make them end in .sh, .csh, or whatever is appropriate. (and make sure your scripts directory is in your path).
• Be aware that scripts are very useful, but not always the best tool. Switch to something like perl or C (fortran if you must) if it starts getting too long and complex.
• Read the man page.

Some decent tutorials can be found by googling shell scripting or C shell scripting.