Bash

What are Bash scripts?

When you open up a terminal, you can type in commands like cd, ls, mkdir, mv, etc. Most of the commands that you do in the terminal have to do with file system operations: making/editing/deleting/modifying files and directories, moving around within the file system, etc. The stuff that happens when you type these commands into the screen isn't magic; the computer is running a program, called a shell, that executes the commands for you in the file system. One of the most popular shells is called Bash (Bourne-again shell), and it's available on all Linux distributions and MacOS (Windows has several free bash emulators, for example git bash). Often, especially if you're doing something to set up some system on many different machines or devices, you'll find yourself doing the same commands in your shell repeatedly, and it can be very tedious. That's where shell scripts come in (and since bash is a shell, a bash script is a shell script). Bash scripts are basically files that contain a list of commands that a bash shell should run. It's almost a sort of "meta-program" because you're running a script (a program) to a call a list of other programs (ls, mkdir, cd are all programs themselves) to do something for you.

Our bash scripts

What part do bash scripts play in Runtime? Well, we have only four scripts in our repo, but they're all really important:

• runtime: located in the root directory, this script runs commands / runs other scripts to build, run, test, format, and clean Runtime
• scripts/run.sh: runs Runtime. Assuming runtime build succeeded in making all the executables, this short script basically runs all of the main Runtime executables in the right order and "boots up" Runtime on the machine.
• scripts/test.sh: runs some or all of Runtime's automated tests. test.sh actually runs build.sh to build Runtime first with all of the latest changes, then compiles and runs each specified test, one after the other. This shell script is what puts the "automated" in "automated testing", so it's really important.
• scripts/flash.sh: this is by far the most complex of our bash scripts. Getting code onto the Arduino microcontrollers on our devices is surprisingly difficult, due to the jankiness and inconsistency with the quality of the Arduinos. The process of compiling the Arduino code and getting it onto the board is often known as "flashing", and this script attempts to automate that process as much as possible. It makes use of an Arduino-provided tool called arduino-cli to assist in the compiling and uploading, but the script tries to identify connected boards, extract the relevant information about the boards, and creates and moves files and directories around within the file system to try and ensure the compiling and uploading goes smoothly.

Bash Scripts 101

Bash (or the older sh or bsh shells) has been around for a LONG time, and as a result, has acquired a MASSIVE amount of syntax. There are always a ton of ways to do the same thing (or almost the same thing, with very subtle differences). Since we don't want to have to deal with all of those features, and because our scripts are used for very specific purposes, we've tried to limit the number of features that we use in our scripts to a small subset of all the syntax in bash, to make it simpler to understand. With that said, let's get started!

Shebang

All bash scripts begin with the line:

#!/bin/bash

This basically says that the following script should be executed by bash, and not some other shell (ksh, zsh, etc.). It's a special line, because normally a # symbol begins a comment (unless it's part of a string), but if it's the first line in the whole script, it has this very special and important meaning. In fact, it's so special that people give it a special name, the "shebang", which you might see in online documentation and tutorials.

Issuing commands

Remember, bash scripts is fundamentally about giving bash (which is a shell) a list of commands that you would otherwise have to type out manually. So, for example, if a common thing you do is cd into a folder called test and run an executable called foo, you would simply write the following script to do that for you:

#!/bin/bash

cd test
./foo

Pretty simple so far, right?

Variables

All names are taken to be global variables when followed by an =. Since spaces are used to separate arguments in bash (among other things), assignment of a variable must not have a space between the variable name and the = sign. The following declares the variable str to be equal to the empty string:

str=""

You might think that setting a second variable (let's call it str1) to the contents of str would be given by this:

str1=str

But this is wrong. Since str doesn't have an = sign after it, bash actually thinks that str is a command into the shell and will error out because it can't find that command. What we really want is to tell bash to evaluate str, and to do that, we put a $ in front of the variable name. Don't ask why :\

str1=$str

We can take it even further than this. Say you want str1 to be equal to the contents of doing a listing of the folder test, i.e. you want str1 to contain the contents of doing ls test in the terminal. Well, since $<expression> causes <expression> to be evaluated by the shell, we simply need to put parentheses around ls test and the $ will evaluate the entire expression. We can simply do the following to obtain the desired result:

str1=$(ls test)

Another common piece of manipulation of variables is appending things to path names. For example, suppose the variable path1 contains some path, say c-runtime/tests, and we want to append /integration to path1 and put it into a variable called path2. The way we do this is:

path2="$(path1)/integration"

which looks strange at first, but makes sense after staring at it for a while. The $ says to evaluate the expression in the parentheses, which in our case is the variable path1. That contains the string "c-runtime/tests". After expansion, we now have the string "c-runtime/tests/integration" on the right side which is what we want path2 to be equal to.

Lastly, in order to append something to the end of a variable, we can't use some += operator or something similar in bash, because it doesn't exist. In order to append "foo" to the end of some variable holding a string (let's call it bar), we would write this:

bar="$bar \"foo\""

A couple things to note here:

• Since we want to append literally the string "foo" to the end of whatever string is in bar, we can't simply put quotes around the word foo, since the whole thing is already in quotes. We need to tell bash to put the literal " character in, so we need to escape it with the \ character before.
• Now that we know why we need the backslashes, what this is saying is that the variable bar is now equal to the evaluation of the variable bar followed by the string "foo", which is what we want.

Control

A script wouldn't be so powerful if you couldn't control which commands get executed based on some conditions. So, the standard while loops, for loops, if statements, case statements, and functions are all in bash. while loops and for loops actually have syntax that is surprisingly similar to Python:

while read line; do
    <code>
done <<< "$(<some command that generates lines of output>)"

is a common construct to read lines from the output of the evaluation after the <<< until there are no more lines of input. So, for example if that last line read done <<< "$(cat temp.txt)", this while loop would process the contents of temp.txt line by line. Similarly:

for element in $(<some expression that has many elements>); do
    <code>
done

is a common construct to do something to each element in the expression in the $(). Of course, by putting the name element in the for loop like that, the name element is now made available globally. You can access the contents of element from anywhere within the script by doing $element.

if statements are pretty simple as well. They contain a conditional expression and will run if the conditional expression is true. Conditional expressions in bash scripts are wonky and deserve an entire section, so for now, just think of it as some expression that evaluates to pass (returns 0) or fail (returns non-zero):

if <conditional expression>; then
    <code>
fi

Lastly, case statements are useful for comparing the value of something to multiple different possible values. They look like this:

case $var in
    1)
        <code for if $var == 1>
        ;;
    2)
        <code for if $var == 2>
        ;;
    *)
        <code for if $var is anything else>
        ;;
esac

A few things to point out here:

• ;; is like a break statement, so once the $var matches with something, it only executes the code in that block and then exits the case statement immediately.
• * is a bash "wildcard" character that basically means "anything else". So that last case where we compare if $var == * will always be true if both the first and second cases fail. So, it kind of acts like a "default" case.
• Make sure to close the case statement with esac (case spelled backwards!).

Conditional expressions

Most of the time, when you want to test if some variable is equal to some value, you use double square brackets. For example:

[[ $str == "" ]]       # test if variable str contains the empty string
[[ $str == *"test"* ]] # test if variable str contains any sequence of characters, followed by "test", and then any more sequence of characters
[[ $var != 0 ]]        # test if variable var is not equal to 0
[[ -f <file_path> ]]   # test if file_path is a valid file path (that file exists)
[[ -d <dir_path> ]]    # test if dir_path is a valid path to a directory (that directory exists)

There are many, many, many caveats to conditional expressions in bash. Yes, there is meaning to single square brackets, double parentheses, and single parentheses, but for all of the things that we are using them for, the double square brackets does the job, so to make our scripts as simple as possible, we're going to stick with using only the double square brackets!

Functions

Functions have to be defined before use. This forces all scripts to have the "smaller" functions at the top, and the "larger", more complex functions near the bottom. The actual script (commands that are issued) are at the very bottom of the file. Functions in bash can take in any arbitrary number of arguments (or none at all), so there isn't a specific place to declare the arguments to a function. A typical function definition looks like this:

function example {
    <code>
}

Now, later in your code, you can write example and it will run the <code> in the function. To pass arguments to example, put more variables or expressions after the word example when you invoke the function to pass them, for example:

example "test" "foo" "bar"

calls the function example with the strings "test", "foo", and "bar" as the first, second, and third arguments, respectively. But if we don't give these arguments specific names by declaring them in the function header, how do we access them within the function? We use $1 to refer to the first argument, $2 to refer to the second argument, and so on. If that argument wasn't given, then $<number> evaluates to the empty string by default. There's also the special value $@, which refers to all the arguments to the function. So:

for arg in $@; do
    <some commands>
done

is a common construct to do <some commands> to every argument to a function. Finally, the use of $1, $2, $3, $@, etc. is not restricted to use inside of functions defined in your shell scripts! When used outside of any functions, these values refer to the first, second, third, and all command-line arguments that were passed in when you invoked your shell script itself. So, suppose you have a shell script called foo.sh that looked like this:

#!/bin/bash

echo $1
echo $2
echo $@

If you invoke foo.sh like so:

./foo.sh "hello" "world"

the output of the script will be:

hello
world
hello world

Additional Resources

Here are some additional resources that are useful for understanding bash syntax and commands:

• Intro to Bash Scripts
• String Manipulation in Bash Scripts
• [[]] vs. [] vs. (()) vs. ()