path: root/gemfeed/2021-11-21-bash-golfing.draft.gmi

# Bash golfing

```

     '\                   .  .                        |>18>>
       \              .         ' .                   |
      O>>         .                 'o                |
       \       .                                      |
       /\    .                                        |
      / /  .'                                         |
jgs^^^^^^^`^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                            Art by Joan Stark
```

> Published by Paul Buetow 2021-11-27

This blog post is about random Bash tips, tricks and wierdnesses I came across. It's a collection of smaller articles I wrote in an older (in german language) blog which I translated and refreshed with some new insights.

## TCP/IP networking

You probably know the Netcat utility, which is a swiss army knife for TCP/IP networking on the command line. But did you know that the Bash natively supports TCP/IP networking?

Have a look here how that works:

```
❯ cat < /dev/tcp/time.nist.gov/13

59536 21-11-18 08:09:16 00 0 0 153.6 UTC(NIST) *
```

The Bash treats /dev/tcp/HOST/PORT in a special way so that it actually establishing a TCP connection to HOST:PORT. The example above redirects the TCP output of the timeserver to cat and cat is printing it on standard output (stdout).

A more sophisticated example is firing up a HTTP request. Let's create a new read-write (rw) file descriptor (fd) 5, redirect the HTTP request string to it, and then read the response back:

```
❯ exec 5<>/dev/tcp/google.de/80
❯ echo -e "GET / HTTP/1.1\nhost: google.de\n\n" >&5
❯ cat <&5 | head
HTTP/1.1 301 Moved Permanently
Location: http://www.google.de/
Content-Type: text/html; charset=UTF-8
Date: Thu, 18 Nov 2021 08:27:18 GMT
Expires: Sat, 18 Dec 2021 08:27:18 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 218
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
```

You would assume that this also works with the ZSH, but it doesn't. This is one of the few things which don't work with the ZSH but in the Bash. There might be plugins you could use for ZSH to do something similar though.

## Process substitution

The idea here is, that you can read the output (stdout) of a command from a file descriptor:

```
❯ uptime # Without process substitution
 10:58:03 up 4 days, 22:08,  1 user,  load average: 0.16, 0.34, 0.41
❯ cat <(uptime) # With process substitution
 10:58:16 up 4 days, 22:08,  1 user,  load average: 0.14, 0.33, 0.41
❯ stat <(uptime)
  File: /dev/fd/63 -> pipe:[468130]
  Size: 64              Blocks: 0          IO Block: 1024   symbolic link
Device: 16h/22d Inode: 468137      Links: 1
Access: (0500/lr-x------)  Uid: ( 1001/    paul)   Gid: ( 1001/    paul)
Context: unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Access: 2021-11-20 10:59:31.482411961 +0000
Modify: 2021-11-20 10:59:31.482411961 +0000
Change: 2021-11-20 10:59:31.482411961 +0000
 Birth: -
```

This example doesn't make any sense practically speaking, but it clearly demonstrates how process substitution works. The standard output pipe of "uptime" is redirected to a anonymous file descriptor. That fd then is opened by the "cat" command as a regular file.

A useful use case is displaying the diff of two sorted files:

```
❯ echo a > /tmp/file-a.txt
❯ echo b >> /tmp/file-a.txt
❯ echo c >> /tmp/file-a.txt
❯ echo b > /tmp/file-b.txt
❯ echo a >> /tmp/file-b.txt
❯ echo c >> /tmp/file-b.txt
❯ echo X >> /tmp/file-b.txt
❯ diff -u <(sort /tmp/file-a.txt) <(sort /tmp/file-b.txt)
--- /dev/fd/63  2021-11-20 11:05:03.667713554 +0000
+++ /dev/fd/62  2021-11-20 11:05:03.667713554 +0000
@@ -1,3 +1,4 @@
 a
 b
 c
+X
❯ echo X >> /tmp/file-a.txt # Now, both files have the same content again.
❯ diff -u <(sort /tmp/file-a.txt) <(sort /tmp/file-b.txt)
❯
```

Another example is displaying the difference of two directories:

```
❯ diff -u <(ls ./dir1/ | sort) <(ls ./dir2/ | sort)
```

More (Bash golfing) examples:

```
❯ wc -l <(ls /tmp/) /etc/passwd <(env)
     24 /dev/fd/63
     49 /etc/passwd
     24 /dev/fd/62
     97 total
❯
❯ while read foo; do
  echo $foo
done < <(echo foo bar baz)
foo bar baz
❯
```

So far we only used process substitution for stdout redirection. But it also works for stdin. The following two commands result into the same outcome, but the second one is writing the tar data stream to an anonymous file descriptor which is substituted by the "bzip2" command reading the data stream from stdin and compressing it to its own stdout, which then gets redirected to a file:

```
❯ tar cjf file.tar.bz2 foo
❯ tar cjf >(bzip2 -c > file.tar.bz2) foo
```

Just think a while and see whether you understand fully what is happening here.

## Grouping

Command grouping can be quite useful for combining the output of multiple commands:

```
❯ { ls /tmp; cat /etc/passwd; env; } | wc -l
97
❯ ( ls /tmp; cat /etc/passwd; env; ) | wc -l
97
```

But wait, what is the difference between curly braces and normal braces? I assumed that the normal braces create a subprocess whereas the curly ones don't, but I was wrong:

```
❯ echo $$
62676
❯ { echo $$; }
62676
❯ ( echo $$; )
62676
```

One difference is, that the curly braces require you to end the last statement with a semicolon, whereas with the normal braces you can omit the last semicolon:

```
❯ ( env; ls ) | wc -l
27
❯ { env; ls } | wc -l
>
> ^C
```

If you know more (subtle) difference, please write me an E-Mail and let me know.

## Expansions

The Bash expansions are yet more useful (and interesting) features. Let's start with a simple example:

```
❯ echo {0..5}
0 1 2 3 4 5
❯ for i in {0..5}; do echo $i; done
0
1
2
3
4
5
```

You can also add leading 0 or expand to any number range:

```
❯ echo {00..05}
00 01 02 03 04 05
❯ echo {000..005}
000 001 002 003 004 005
❯ echo {201..205}
201 202 203 204 205
```

It also works with letters:

```
❯ echo {a..e}
a b c d e
```

Now it get's interesting. The following takes a list of words and expands so that all words are quoted:

```
❯ echo \"{These,words,are,quoted}\"
"These" "words" "are" "quoted"
```

Let's also expand to the cross product of two lists given:

```
❯ echo {one,two}\:{A,B,C}
one:A one:B one:C two:A two:B two:C
❯ echo \"{one,two}\:{A,B,C}\"
"one:A" "one:B" "one:C" "two:A" "two:B" "two:C"
```

Just because we can:

```
❯ echo Linux-{one,two,three}\:{A,B,C}-FreeBSD
Linux-one:A-FreeBSD Linux-one:B-FreeBSD Linux-one:C-FreeBSD Linux-two:A-FreeBSD Linux-two:B-FreeBSD Linux-two:C-FreeBSD Linux-three:A-FreeBSD Linux-three:B-FreeBSD Linux-three:C-FreeBSD
```

## - aka stdin and stdout placeholder

Some commands and Bash builtins use "-" as a placeholder for stdin and stdout:

```
❯ echo Hello world
Hello world
❯ echo Hello world | cat -
Hello world
❯ cat - <<ONECHEESEBURGERPLEASE
Hello world
ONECHEESEBURGERPLEASE
Hello world
❯ cat - <<< 'Hello world'
Hello world
```

Let's walk through all three examples from the above snippet:

* The first example is obvious (the Bash builtin "echo" prints its arguments to stdout).
* The second pipes "Hello world" via stdout to stdin of the "cat" command. As cat's argument is "-" it reads its data from stdin and not from a regular file named "-". So "-" has a special meaning for cat.
* The third and fourth examples are interesting as we don't use a pipe as of "|" but a so-called HERE-document and a HERE-string. But the end result is the same.

The "tar" command understands "-" too. This example tars up some local directory and sends the data to stdout (this is what "-f -" commands it to do). stdout then is piped via a SSH session to a remote tar process (running on example.org) and reads the data from stdin and extracts all the data coming from stdin (as we told tar "-f -") on the remote machine:

```
❯ tar -czf - /some/dir | ssh someuser@example.org tar -xzvf - 
```

This is yet another example of using "-", but this time using the "file" command:

```
$ head -n 1 grandmaster.sh
#!/usr/bin/env bash
$ file - < <(head -n 1 grandmaster.sh)
/dev/stdin: a /usr/bin/env bash script, ASCII text executable
```

Some more golfing:

```
$ cat -
hello
hello
^C
$ file -
#!/usr/bin/perl
/dev/stdin: Perl script text executable
```

## Alternative argument passing

This is a quite unusual way of passing arguments to a Bash script:

```
❯ cat foo.sh
#/usr/bin/env bash
declare -r USER=${USER:?Missing the username}
declare -r PASS=${PASS:?Missing the secret password for $USER}
echo $USER:$PASS
```

So what we are doing here is to pass the arguments via environment variables to the script. The script will abort with an error when there's an undefined argument.

```
❯ chmod +x foo.sh
❯ ./foo.sh
./foo.sh: line 3: USER: Missing the username
❯ USER=paul ./foo.sh
./foo.sh: line 4: PASS: Missing the secret password for paul
❯ echo $?
1
❯ USER=paul PASS=secret ./foo.sh
paul:secret
```

You have probably noticed this *strange* syntax:

```
❯ VARIABLE1=value1 VARIABLE2=value2 ./script.sh
```

That's just another way to pass environment variables to a script. You could write it as well as like this:

```
❯ export VARIABLE1=value1
❯ export VARIABLE2=value2
❯ ./script.sh
```

But the downside of it would be that the variables would also be defined in your current shell and not only in the scripts sub-process.

## : aka the null command

First of all, let's use the "help" Bash-builtin so we see what the docs are saying about the null command:

```
❯ help :
:: :
    Null command.

    No effect; the command does nothing.

    Exit Status:
    Always succeeds.
```

PS: IMHO people should use the Bash help more often. It is a very useful reference to all the Bash stuff. Too many fallback to a Google search and then land on Stack Overflow. Sadly, there's no help-builtin for the ZSH shell though. 

OK, back to the null command. What happens when you try to run it? As you can see, absolutely nothing. And its exit status is 0 (success):

```
❯ :
❯ echo $?
0
```

Why would that useful? You can use it as a placeholder in an endless while-loop:

```
❯ while : ; do date; sleep 1; done
Sun 21 Nov 12:08:31 GMT 2021
Sun 21 Nov 12:08:32 GMT 2021
Sun 21 Nov 12:08:33 GMT 2021
^C
❯
```

You can also use it as a placeholder for a function body not yet fully implemented, as an empty function would be a syntax error:

```
❯ foo () {  }
-bash: syntax error near unexpected token `}'
❯ foo () { :; }
❯ foo
❯
```

Or use it as a placeholder for not yet implemented conditional branches:

```
❯ if foo; then :; else echo bar; fi
```

Or (not recommended) as a fancy way to comment your Bash code:

```
❯ : I am a comment and have no other effect
❯ : I am a comment and result in a syntax error ()
-bash: syntax error near unexpected token `('
❯ : "I am a comment and don't result in a syntax error ()"
❯
```

As you can see in the previous example the Bash still tries to interpret some of the syntax of all text following after ":". This can be exploited (also not recommended) like this:

```
❯ declare i=0
❯ $[ i = i + 1 ]
bash: 1: command not found...
❯ : $[ i = i + 1 ]
❯ : $[ i = i + 1 ]
❯ : $[ i = i + 1 ]
❯ echo $i
4
```

For these kind of expressions it's always better to use "let" though. And you should also use $((...expression...)) instead of the old (deprecated) way $[ ...expression... ] like this example demonstrates:

```
❯ declare j=0
❯ let i=$((j + 1))
❯ let i=$((j + 1))
❯ let i=$((j + 1))
❯ let i=$((j + 1))
❯ echo $j
4
```

## (No) floating point support

I have to give a plus-point to the ZSH here. As the ZSH supports floating point calculation, whereas the Bash doesn't:

```
❯ bash -c 'echo $(( 1/10 ))'
0
❯ zsh -c 'echo $(( 1/10 ))'
0
❯ bash -c 'echo $(( 1/10.0 ))'
bash: line 1: 1/10.0 : syntax error: invalid arithmetic operator (error token is ".0 ")
❯ zsh -c 'echo $(( 1/10.0 ))'
0.10000000000000001
❯
```

It would be nice to have native floating point support for the Bash too, but you don't want to to use the shell for complicated calculations anyway. So it's fine that Bash doesn't have that I guess. 

In the Bash you would have to fallback to an external command like "bc" (the arbitrary precision calculator language):

```
❯ bc <<< 'scale=2; 1/10'
.10
```

## Redirection

Let's have a closer look at Bash redirection. As you might already know that there are 3 standard file descripors:

* 0 aka stdin (standard input)
* 1 aka stdout (standard output)
* 2 aka stderr (standard error output )

These are most certainly the ones you are using on regular basis. "/proc/self/fd" lists all file descriptors which are open by the current process (in this case: the current bash shell itself):

```
❯ ls -l /proc/self/fd/
total 0
lrwx------. 1 paul paul 64 Nov 23 09:46 0 -> /dev/pts/9
lrwx------. 1 paul paul 64 Nov 23 09:46 1 -> /dev/pts/9
lrwx------. 1 paul paul 64 Nov 23 09:46 2 -> /dev/pts/9
lr-x------. 1 paul paul 64 Nov 23 09:46 3 -> /proc/162912/fd
```

The following examples demonstrate two different ways to accomplish the same thing. The difference is that the first command is directly printing out "Foo" to stdout and the second command is explicitly redirecting stdout to its own stdout file descriptor:

```
❯ echo Foo
Foo
❯ echo Foo > /proc/self/fd/0
Foo
```

Other useful redirections are:

* Redirect stderr to stdin: "echo foo 2>&1"
* Redirect stdin to stderr: "echo foo >&2"

It is however not possible to redirect multiple times within the same command. E.g. the following won't work. You would expect stdin to be redirected to stderr and then stderr to be redirected to /dev/null. But as the example shows, Foo is still printed out:

```
❯ echo Foo 1>&2 2>/dev/null
Foo
```

This is where you can use grouping (neither of these commands will print out anything to stdout):

```
❯ { echo Foo 1>&2; } 2>/dev/null
❯ ( echo Foo 1>&2; ) 2>/dev/null
❯ { { { echo Foo 1>&2; } 2>&1; } 1>&2; } 2>/dev/null
❯ ( ( (  echo Foo 1>&2; ) 2>&1; ) 1>&2; ) 2>/dev/null
❯
```

A handy way to list all open file descriptors is to use the "lsof" command (that's not a Bash builtin), whereas $$ is the process id (pid) of the current shell process:

```
❯ lsof -a -p $$ -d0,1,2
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
bash    62676 paul    0u   CHR  136,9      0t0   12 /dev/pts/9
bash    62676 paul    1u   CHR  136,9      0t0   12 /dev/pts/9
bash    62676 paul    2u   CHR  136,9      0t0   12 /dev/pts/9
```

Let's create our own descriptor "3" for redirection to a file named "foo":

```
❯ touch foo
❯ exec 3>foo # This opens fd 3 and binds it to file foo.
❯ ls -l /proc/self/fd/3
l-wx------. 1 paul paul 64 Nov 23 10:10 \
    /proc/self/fd/3 -> /home/paul/foo
❯ cat foo
❯ echo Bratwurst >&3
❯ cat foo
Bratwurst
❯ exec 3>&- # This closes fd 3.
❯ echo Steak >&3
-bash: 3: Bad file descriptor
```

You can also override the default file descriptors as the following example script demonstrates:

```
❯ cat grandmaster.sh
#!/usr/bin/env bash

# Write a file data-file containing two lines
echo Learn You a Haskell > data-file
echo for Great Good >> data-file

# Link fd with fd 6 (saves default stdin)
exec 6<&0

# Overwrite stdin with data-file
exec < data-file

# Read the first two lines from it
declare LINE1 LINE2
read LINE1
read LINE2

# Print them
echo First line: $LINE1
echo Second line: $LINE2

# Restore default stdin and delete fd 6
exec 0<&6 6<&-
```

Let's execute it:

```
❯ chmod 750 ./grandmaster.sh
❯ ./grandmaster.sh
First line: Learn You a Haskell
Second line: for Great Good
```

## HERE

I have mentioned HERE-documents and HERE-strings already in this post. Let's do some more examples. The following "cat" receives a multi line string from stdin. In this case the input multi line string is a HERE-document. As you can see, it also interpolates variables (in this case the output of "date" running in a sub-shell).

```
❯ cat <<END
> Hello World
> It’s $(date)
> END
Hello World
It's Fri 26 Nov 08:46:52 GMT 2021
```

You can also write it this way, but that's less readable (it's good for an obfuscation contest):

```
❯ <<END cat
> Hello Universe
> It’s $(date)
> END
Hello Universe
It's Fri 26 Nov 08:47:32 GMT 2021
```

Besides of an HERE-document there is also a so-called HERE-string. Besides of...

```
❯ declare VAR=foo
❯ if echo "$VAR" | grep -q foo; then
> echo '$VAR ontains foo'
> fi
$VAR ontains foo
```

...you can use a HERE-string like that:

```
❯ if grep -q foo <<< "$VAR"; then
> echo '$VAR contains foo'
> fi
$VAR contains foo
```

Or even shorter you can do:

```
❯ grep -q foo <<< "$VAR" && echo '$VAR contains foo'
$VAR contains foo
```

You can also use a Bash regex to accomplish the same thing, but the point of the examples so far is to demonstrate HERE-{documents,strings} and not Bash regular expressions:

```
❯ if [[ "$VAR" =~ foo ]]; then echo yay; fi
yay
```

You can also use it with "read", e.g. read a sentence into an array of words:

```
❯ read a <<< ja
❯ echo $a
ja
❯ read b <<< 'NEIN!!!'
❯ echo $b
NEIN!!!
❯ dumdidumstring='Learn you a Golang for Great Good'
❯ read -a words <<< "$dumdidumstring"
❯ echo ${words[0]}
Learn
❯ echo ${words[3]}
Golang
```

The following is good for an obfuscation contest too:

```
❯ echo 'I like Perl too' > perllove.txt
❯ cat - perllove.txt <<< "$dumdidumstring"
Learn you a Golang for Great Good
I like Perl too
```

## RANDOM

Random is a special builtin variable containing a pseudo random number each time it's used.

```
❯ echo $RANDOM
11811
❯ echo $RANDOM
14997
❯ echo $RANDOM
9104
```

That's very useful if you want to randomly delay the execution of your scripts when you run it on many servers concurrently just to spread the server load (which might be caused by the script run) better.

Let's say you want to introduce a random delay of 1 minute you can do:

```
❯ cat ./calc_answer_to_ultimate_question_in_life.sh
#!/usr/bin/env bash

declare -i MAX_DELAY=60

random_delay () {
    local -i sleep_for=$((RANDOM % MAX_DELAY))
    echo "Delaying script execution for $sleep_for seconds..."
    sleep $sleep_for
    echo 'Continuing script execution...'
}

main () {
    random_delay
    # From here, do the real work. Calculating the answer to
    # the ultimate question can take billions of years....
    : ....
}

main

❯
❯ ./calc_answer_to_ultimate_question_in_life.sh
Delaying script execution for 42 seconds...
Continuing script execution...
```

## set -x and set -e and pipefile

In my opinion, -x and -e and pipefile are the most useful Bash options. Let's have a look at them one after another.

### -x

-x prints commands and their arguments as they are executed. This helps developing and debugging your Bash code:

```
❯ set -x
❯ square () { local -i num=$1; echo $((num*num)); }
❯ num=11; echo "Square of $num is $(square $num)"
+ num=11
++ square 11
++ local -i num=11
++ echo 121
+ echo 'Square of 11 is 121'
Square of 11 is 121
```

However, you need to set -x also for the sub-shell in oder to print out what is happening:

```
❯ num=12; echo "Square of $num is $(set -x;square $num)"
+ num=12
++ set -x
++ square 12
++ local -i num=12
++ echo 144
+ echo 'Square of 12 is 144'
Square of 12 is 144
❯
```

You can also use the option like this, if helped me personally a lot:

```
❯ bash -x ./half_broken_script_to_be_debugged.sh
```

Let's do that on one of the example scripts we used earlier:

```
❯ bash -x ./grandmaster.sh
+ bash -x ./grandmaster.sh
+ echo Learn You a Haskell
+ echo for Great Good
+ exec
+ exec
+ declare LINE1 LINE2
+ read LINE1
+ read LINE2
+ echo First line: Learn You a Haskell
First line: Learn You a Haskell
+ echo Second line: for Great Good
Second line: for Great Good
+ exec
❯
```

### -e

This is a very important option you want to set when you are paranoid. This means, you should always "set -e" in your scripts when you need to make absolutely sure that your script runs successfully:

```
❯ help set | grep -- -e
      -e  Exit immediately if a command exits with a non-zero status.
```

As you can see in the following example, the bash terminates after the execution of "grep" as "foo" is not matching "bar". Therefore, grep exits with 1 (unsuccessfully) and the shell aborts. And therefore, "bar" will  not be printed out anymore:

```
❯ bash -c 'set -e; echo hello; grep -q bar <<< foo; echo bar'
hello
❯ echo $?
1
```

Whereas in this example everything works as expected:

```
❯ bash -c 'set -e; echo hello; grep -q bar <<< barman; echo bar'
hello
bar
❯ echo $?
0
```

If you still want to use grep in your script but "set -e" would always make it terminate you can add it as a conditional expression instead:

```
bar
❯ bash -c 'set -e
>    if grep -q bar <<< foo; then
>        echo "matching"
>    else
>        echo "not matching"
>    fi
>    echo bar'
not matching
bar
❯ echo $?
0
❯ bash -c 'set -e
>    if grep -q bar <<< barman; then
>        echo "matching"
>    else
>        echo "not matching"
>    fi
>    echo bar'
matching
bar
❯ echo $?
0
```

Read this blog post of mine about being paranoid in real life:

=> ./2021-10-22-defensive-devops.gmi Defensive DevOps

### pipefail

## More

Have also a look at my personal Bash coding style guide:

=> ./2021-05-16-personal-bash-coding-style-guide.gmi Personal Bash coding style guide

E-Mail me your thoughts at comments@mx.buetow.org!

=> ../ Go back to the main site