Unlike other scripting programming languages, shell scripting is optimized for shell related tasks, e.g. reading from STDIN and saving results into files are primitives.
Pesky Whitespace
foo = bar
# Doesn't work because it's interpreted as calling the foo program with the args '=' and 'bar'
foo=bar # Now that works!
Literal Strings vs. Substitutable Strings
Single quotes are literal strings, but double quotes allow variable substitution.
foo=bar
echo "$foo" # prints bar
echo '$foo' # prints $foo
Special Variables in Shell Scripting
$0 # Name of the script
$1 # First argument to the script. Similar thing till $9
$@ # All the arguments
$# # Number of arguments
$? # Return code of the previous command
$$ # Process Identification number for the current script
!! # Entire last command, including arguments.
$_ # Last argument from the last command.
Command Substitution
Whenever you place $( CMD ) it will execute CMD, get the output of the command and substitute it in place, e.g.
for file in $(ls) # The shell calls `ls` and then iterates over those values
Process Substitution
<( CMD ) will execute CMD and place the output in a temp file and substitute the <() with that file’s name, e.g.
# This command will show differences between files in dirs foo and bar
diff <(ls foo) <(ls bar)
This is useful when commands expect values to be passed by file instead of by STDIN.
Comparison Checks
if [[ $? -ne 0 ]]; then
# ...
Try to use double brackets [[ ]] instead of simple brackets [ ] as the chance of making mistakes is lower.
However, the double brackets are not portable to sh. Single brackets are the old way of testing conditions.
Expanding Strings
convert image.{png,jpg} # will expand to...
convert image.png image.jpg
Shebang Lines
To be called from the terminal, scripts need not be written in the shell language. A she=bang is sufficient, e.g. a python script:
#!/usr/bin/env python
import sys
for arg in reversed(sys.argv[1:]):
print(arg)
When writing a shebang, use the env command to resolve to wherever the command lives in the system. Not every computer will have python living at #!/usr/local/bin/python.
Differences Between Shell Functions and Scripts
| Shell Functions | Shell Scripts |
|---|---|
| Have to be in same language as the shell | All you need is a shebang |
| Loaded once when their definition is read | Loaded every time they’re executed |
| Executed in current shell environment. Can modify environment variables, change your directory, etc. | Execute in their own process. Receive exported environment variables by value. |
As with any programming language functions are a powerful construct to achieve modularity, code reuse and clarity of shell code. Often shell scripts will include their own function definitions.
Finding Files
Unix-like systems come with find, which recursively searches for files matching some criteria.
The . means start searching from the current directory. ? and * match one or any amount of characters respectively.
The globstar, ** matches all files and zero or more directories and subdirectories. **/ matches only directories and subdirectories. See
Globs (mywiki.wooledge.org)
for more information.
# Find all directories named src
find . -name src -type d
# Find all python files that have a folder named test in their path
find . -path '**/test/**/*.py' -type f
# Find all files modified in the last day
find . -mtime -1
# Find all zip files with size in range 500k to 10M
find . -size +500k -size -10M -name '*.tar.gz'
# Find all PNG files and convert them to JPG
find . -name '*.png' -exec convert {} {.}.jpg \;
Unlike find which does a fresh search each time, locate uses a (daily) updated database. However, despite being faster, locate is indexed only on the file name and can be stale.
But most of the time, you’re more interested in the file contents. grep is your friend.
Shell History
Tried of pressing the up arrow a 1,000 times for that terminal command? Ctrl + R allows you to perform backwards search through your history!
If you start a command with a leading space it won’t be added to you shell history. If you forgot the space, edit your ~/.bash_history or ~/.zhistory file.
At the cost of being painful on the eyes. Python scripts are way more readable (and cross-platform) than shell scripts.