path: root/gemfeed/2021-11-21-bash-golfing.draft.gmi

# Bash golfing

> Written by Paul Buetow 2021-11-21

This blog post is about some (mostly uncommon) bash tricks I came across in the past.

## TCP/IP networking

You probably know the Netcat utility, which is like a swiss army knife for TCP/IP networking on the command line. But did you know that the Bash natively supports TCP/IP networking?

To demonstrate establishing a network connection without Netcat or any other external tool from the Bash, have a look here:

```
❯ cat < /dev/tcp/time.nist.gov/13

59536 21-11-18 08:09:16 00 0 0 153.6 UTC(NIST) *
```

The Bash treats /dev/tcp/HOST/PORT in a special way so that it actually establishing a TCP connection to HOST:PORT. The example above redirects the TCP output of the timeserver given to cat and cat is printing it on standard output (stdout).

A more sophisticated example would be firing up a HTTP request. Here, we create a new read-write (rw) file descriptor (fd) 5, redirect the HTTP request string to it, and then read the response back.

```
❯ exec 5<>/dev/tcp/google.de/80
❯ echo -e "GET / HTTP/1.1\nhost: google.de\n\n" >&5
❯ cat <&5 | head
HTTP/1.1 301 Moved Permanently
Location: http://www.google.de/
Content-Type: text/html; charset=UTF-8
Date: Thu, 18 Nov 2021 08:27:18 GMT
Expires: Sat, 18 Dec 2021 08:27:18 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 218
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
```

You would assume that this also works with the ZSH, but it doesn't out of the box. Although there might be plugins you could use for ZSH to do something similar.

## Process substitution

I personally use process substitution quite frequently. The idea is, that you can read the output (stdout) of an command from a file descriptor. To demonstrate this:

```
❯ uptime
 10:58:03 up 4 days, 22:08,  1 user,  load average: 0.16, 0.34, 0.41
❯ cat <(uptime)
 10:58:16 up 4 days, 22:08,  1 user,  load average: 0.14, 0.33, 0.41
❯ stat <(uptime)
  File: /dev/fd/63 -> pipe:[468130]
  Size: 64              Blocks: 0          IO Block: 1024   symbolic link
Device: 16h/22d Inode: 468137      Links: 1
Access: (0500/lr-x------)  Uid: ( 1001/    paul)   Gid: ( 1001/    paul)
Context: unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Access: 2021-11-20 10:59:31.482411961 +0000
Modify: 2021-11-20 10:59:31.482411961 +0000
Change: 2021-11-20 10:59:31.482411961 +0000
 Birth: -
```

This example doesn't make any sense practically speaking, but it clearly demonstrates how process substitution works. The standard output pipe of "uptime" is redirected to a anonymous file descriptor. That fd then is opened by the "cat" command as a regular file.

A useful use case would be displaying the diff of two sorted files:

```
❯ echo a > /tmp/file-a.txt
❯ echo b >> /tmp/file-a.txt
❯ echo c >> /tmp/file-a.txt
❯ echo b > /tmp/file-b.txt
❯ echo a >> /tmp/file-b.txt
❯ echo c >> /tmp/file-b.txt
❯ echo X >> /tmp/file-b.txt
❯ diff -u <(sort /tmp/file-a.txt) <(sort /tmp/file-b.txt)
--- /dev/fd/63  2021-11-20 11:05:03.667713554 +0000
+++ /dev/fd/62  2021-11-20 11:05:03.667713554 +0000
@@ -1,3 +1,4 @@
 a
 b
 c
+X
❯ echo X >> /tmp/file-a.txt
❯ diff -u <(sort /tmp/file-a.txt) <(sort /tmp/file-b.txt)
❯
```

Another example would be displaying the difference of two directories like this:

```
❯ diff -u <(ls ./dir1/ | sort) <(ls ./dir2/ | sort)
```

More (Bash golfing) examples:

```
❯ wc -l <(ls /tmp/) /etc/passwd <(env)
     24 /dev/fd/63
     49 /etc/passwd
     24 /dev/fd/62
     97 total
❯
❯ while read foo; do
  echo $foo
done < <(echo foo bar baz)
foo bar baz
❯
```

So far we only used process substitution for stdout redirection. But it also works for stdin. The following two commands have the same result, but the second one is writing the tar data stream to an anonymous file descriptor which is substituted by the "bzip2" command reading the data stream from stdin and compressing it to its own stdout, which then gets redirected to a file:

```
tar cjf file.tar.bz2 foo
tar cjf >(bzip2 -c > file.tar.bz2) foo
```

## Grouping

Command grouping can be quite useful for combining the output of multiple commands like this:

```
❯ { ls /tmp; cat /etc/passwd; env; } | wc -l
97
❯ ( ls /tmp; cat /etc/passwd; env; ) | wc -l
97
```

But wait, what is the difference between curly braces and normal braces? I assumed that the normal braces create a subprocess whereas the curly ones don't, but I was wrong:

```
❯ echo $$
62676
❯ { echo $$; }
62676
❯ ( echo $$; )
62676
```

If you know the (subtle) difference, please write me an E-Mail and let me know. One difference is, that the curly braces require you to end the last statement with a semicolon, whereas with the normal braces you can omit the last semicolon:

```
❯ ( env; ls ) | wc -l
27
❯ { env; ls } | wc -l
>
> ^C
```

## Expansions

The Bash expansions are yet more useful (and interesting) features. Let's start it simple:

```
❯ echo {0..5}
0 1 2 3 4 5
❯ for i in {0..5}; do echo $i; done
0
1
2
3
4
5
```

You can also add leading 0 or expand to any number range:

```
❯ echo {00..05}
00 01 02 03 04 05
❯ echo {000..005}
000 001 002 003 004 005
❯ echo {201..205}
201 202 203 204 205
```
It also works with non-numerics:

```
❯ echo {a..e}
a b c d e
```

Now it get's interesting. The following takes a list of words and expands so that all words are quoted:

```
❯ echo \"{These,words,are,quoted}\"
"These" "words" "are" "quoted"
```

Let's also expand to the cross product of two lists given:

```
❯ echo {one,two}\:{A,B,C}
one:A one:B one:C two:A two:B two:C
❯ echo \"{one,two}\:{A,B,C}\"
"one:A" "one:B" "one:C" "two:A" "two:B" "two:C"
```

Just because I can:

```
❯ echo Linux-{one,two,three}\:{A,B,C}-FreeBSD
Linux-one:A-FreeBSD Linux-one:B-FreeBSD Linux-one:C-FreeBSD Linux-two:A-FreeBSD Linux-two:B-FreeBSD Linux-two:C-FreeBSD Linux-three:A-FreeBSD Linux-three:B-FreeBSD Linux-three:C-FreeBSD
```

## - aka stdin and stdout placeholder

Some commands and Bash builtins use - as a placeholder for stdin and stdout. Let's have a look first at the following snippet:

```
❯ echo Hello world
Hello world
❯ echo Hello world | cat -
Hello world
❯ cat - <<ONECHEESEBURGERPLEASE
Hello world
ONECHEESEBURGERPLEASE
Hello world
❯ cat - <<< 'Hello world'
Hello world
```

All examples result into the same outcome:

* The first example is obvious (the Bash builtin "echo" prints its arguments to "stdout").
* The second pipes "Hello world" via "stdout" to "stdin" of the "cat" command. As cat's argument is "-" it reads its data from "stdin" and not from a regular file named "-". So "-" has a special meaning for cat.
* The third and fourth examples are interesting as we don't use a pipe as of "|" but a so-called here-document and a here-string. But the end result is the same.

The "tar" command understands "-" too. This example tars up some local directory and sends the data to stdout (this is what "-f -" commands it to do). stdout then is piped via a SSH session to a remote tar process (on example.org) and reads the data from stdin and extracts all the files on the remote machine again:

```
❯ tar -czf - /some/dir | ssh someuser@example.org tar -xzvf - 
```

And this is yet another example of using "-", but this time using the "file" command:

```
$ head -n 1 test.sh
#!/usr/bin/env bash
$ file - < <(head -n 1 test.sh)
/dev/stdin: a /usr/bin/env bash script, ASCII text executable
```

Some more random golfing:

```
$ cat -
hello
hello
^C
$ file -
#!/usr/bin/perl
/dev/stdin: Perl script text executable
```

## Alternative argument passing

This is a quite unusual way of passing arguments to a Bash script:

```
❯ cat foo.sh
#/usr/bin/env bash
declare -r USER=${USER:?Missing the username}
declare -r PASS=${PASS:?Missing the secret password for $USER}
echo $USER:$PASS
```

So what we are doing here is to pass the arguments via environment variables to the script. The script will abort with an error when there's an argument missing.

```
❯ chmod +x foo.sh
❯ ./foo.sh
./foo.sh: line 3: USER: Missing the username
❯ USER=paul ./foo.sh
./foo.sh: line 4: PASS: Missing the secret password for paul
❯ echo $?
1
❯ USER=paul PASS=secret ./foo.sh
paul:secret
```

You have probably noticed this *strange* syntax:

```
❯ VARIABLE1=value1 VARIABLE2=value2 ./script.sh
```

That's just another way to pass environment variables to a script. You could write it as well as like this:

```
❯ export VARIABLE1=value1
❯ export VARIABLE2=value2
❯ ./script.sh
```

But the downside of it would be that the variables would also be defined in your current shell and not only in the script's sub-process.

## : aka the null command

First of all, let's use the "help" Bash-builtin so we see what the docs are saying about the null command:

```
❯ help :
:: :
    Null command.

    No effect; the command does nothing.

    Exit Status:
    Always succeeds.
```

PS: IMHO people should use the Bash help more often. It is a very useful reference to all the Bash stuff. Sadly, there's no help-builtin for the ZSH shell though.

OK, back to the null command. What happens when you try to run it? As you can see, absolutely nothing. And its exit status is 0 (succeeded executing):

```
❯ :
❯ echo $?
0
```

Why would that useful? You can use it in an endless while-loop:

```
❯ while : ; do date; sleep 1; done
Sun 21 Nov 12:08:31 GMT 2021
Sun 21 Nov 12:08:32 GMT 2021
Sun 21 Nov 12:08:33 GMT 2021
^C
❯
```

You could also use it as a placeholder for a function body not yet fully implemented, as an empty function would be a Bash syntax error:

```
❯ foo () {  }
-bash: syntax error near unexpected token `}'
❯ foo () { :; }
❯ foo
❯
```

Or use it as a placeholder for not yet implemented conditional branches:

```
❯ if foo; then :; else echo bar; fi
```

Or (not recommended) as a fancy way to comment your Bash code:
```
❯ : I am a comment and have no other effect
❯ : I am a comment and result in a syntax error ()
-bash: syntax error near unexpected token `('
❯ : "I am a comment and don't result in a syntax error ()"
❯
```

As you can see in the previous example the Bash still tries to interpret some of the syntax of all text following after ":". This can be exploited (also not recommended) like this:

```
❯ declare i=0
❯ $[ i = i + 1 ]
bash: 1: command not found...
❯ : $[ i = i + 1 ]
❯ : $[ i = i + 1 ]
❯ : $[ i = i + 1 ]
❯ echo $i
4
```

For these kind of expressions it's always better to use "let" though. And you should also use $((...expression...)) instead of the old way $[ ...expression... ] as this example demonstrates:

```
❯ declare j=0
❯ let i=$((j + 1))
❯ let i=$((j + 1))
❯ let i=$((j + 1))
❯ let i=$((j + 1))
❯ echo $j
4
```

## Redirection

Let's have a closer look at Bash redirection. As you might already know that there are 3 standard file descripors:

* 0 aka stdin (standard input)
* 1 aka stdout (standard output)
* 2 aka stderr (standard error output )

These are most certainly the ones you are using on regular basis. "/proc/self/fd" lists all file descriptors which are open by the current process (in this case: the bash):

```
❯ ls -l /proc/self/fd/
total 0
lrwx------. 1 paul paul 64 Nov 23 09:46 0 -> /dev/pts/9
lrwx------. 1 paul paul 64 Nov 23 09:46 1 -> /dev/pts/9
lrwx------. 1 paul paul 64 Nov 23 09:46 2 -> /dev/pts/9
lr-x------. 1 paul paul 64 Nov 23 09:46 3 -> /proc/162912/fd
```

And here are two different ways to accomplish the same thing. The only differene is that the first command is directly printing out to stdout and the second command is explicitly redirecting stdout to its own stdout file descriptor:

```
❯ echo Foo
Foo
❯ echo Foo > /proc/self/fd/0
Foo
```

Other useful redirections are:

* Redirect stderr to stdin: "echo foo 2>&1"
* Redirect stdin to stderr: "echo foo >&2"

It is however not possible to redirect multiple times within the same command. E.g. the following won't work. You would expect stdin to be redirected to stderr and then stderr to be redirected to /dev/null. But as the example shows, Foo is still printed out:

```
❯ echo Foo 1>&2 2>/dev/null
Foo
```

This is, as you have seen earlier in this post, where you can use grouping (neither of these commands will print out anything to stdout):

```
❯ { echo Foo 1>&2; } 2>/dev/null
❯ ( echo Foo 1>&2; ) 2>/dev/null
❯ { { { echo Foo 1>&2; } 2>&1; } 1>&2; } 2>/dev/null
❯ ( ( (  echo Foo 1>&2; ) 2>&1; ) 1>&2; ) 2>/dev/null
❯
```

A handy way to list all open file descriptors is to use the "lsof" command (that's not a Bash builtin), whereas $$ is the pid of the current shell process:

```
❯ lsof -a -p $$ -d0,1,2
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
bash    62676 paul    0u   CHR  136,9      0t0   12 /dev/pts/9
bash    62676 paul    1u   CHR  136,9      0t0   12 /dev/pts/9
bash    62676 paul    2u   CHR  136,9      0t0   12 /dev/pts/9
```

Let's create our own descriptor "3" for redirection to a file named "foo":

```
❯ touch foo
❯ exec 3>foo # This opens fd 3.
❯ ls -l /proc/self/fd/3
l-wx------. 1 paul paul 64 Nov 23 10:10 \
    /proc/self/fd/3 -> /home/paul/foo
❯ cat foo
❯ echo Bratwurst >&3
❯ cat foo
Bratwurst
❯ exec 3>&- # This closes fd 3.
❯ echo Kombucha >&3
-bash: 3: Bad file descriptor
```

You can also override the default file descriptors, like demonstrated in this script:

```
❯ cat test.sh
#!/usr/bin/env bash

# Write a file data-file containing two lines
echo Learn You a Haskell > data-file
echo for Great Good >> data-file

# Link fd with fd 6 (saves default stdin)
exec 6<&0

# Overwrite stdin with data-file
exec < data-file

# Read the first two lines from it
declare LINE1 LINE2
read LINE1
read LINE2

# Print them
echo First line: $LINE1
echo Second line: $LINE2

# Restore default stdin and delete fd 6
exec 0<&6 6<&-
```

Let's execute it:

```
❯ ./test.sh
First line: Learn You a Haskell
Second line: for Great Good
```

## Here

Wie in vielen anderen Skriptsprachen unterstützt auch die Bash sog. Here-Dokumente. Hier ein kleines
Beispiel:
pb@titania: $ cat END
> Hallo Welt
> it’s $(date)
> END
Hallo Welt
it’s Fr 13. Mai 11:07:36 CEST 2011
pb@titania: $
Neben kennt die Bash auch den Operator <. Während
sog. Here-Strings verwendet.
für Here-Dokumente reserviert ist, wird < für
So könnte man ohne einen Here-String prüfen, ob eine Variable einen bestimmte Substring enthält:
VAR=foo; if echo ” $VAR” | grep -q foo; then echo \ $VAR contains foo; fi
Und so mit Here-String:
if grep -q foo < ” $VAR”; then echo \ $VAR contains foo; fi
Wie unschwer zu erkennen ist spart man sich hier einiges an Tipparbeit (ein echo und eine Pipe weniger).
(PS: Das könnte man auch ohne grep, nämlich mit Bash Regexp überprüfen, aber dazu evtl. später mehr).

Here-Strings können auch in Kombination mit read angewandt werden:

pb@titania:
pb@titania:
pb@titania:
Learn
pb@titania:
you
pb@titania:
a
$ dumdidumstring=”Learn you a Haskell for Great Good”
$ read -a words < ” $dumdidumstring”
$ echo $ {words[0] }
$ echo $ {words[1] }
$ echo $ {words[2] }
Das -a bei read bezweckt, dass words aus dem Here-String als Array befüllt werden soll.
Mittels Here-String kann man auch eine Zeile einer Textdatei prependen:
pb@titania:/tmp $ echo for Great Good > file.txt
pb@titania:/tmp $ cat - file.txt <”Learn you a Haskell”
Learn you a Haskell
for Great Good
Das hat allerdings den Nachteil, dass man das Ergebnis zuerst in eine temporäre Datei oder Variable schreiben
muss, bevor man die Originaldatei file.txt überschreibt. Ansonsten kommt es zu einem Fehler:
pb@titania:/tmp $ cat - file.txt <”Learn you a Haskell” > file.txt
cat: file.txt: Eingabedatei ist Ausgabedatei
Natürlich wäre hierbei sed sowieso das bessere Tool der Wahl:

pb@titania:/tmp $ echo for great Good > file.txt
pb@titania:/tmp $ sed -i -e ’1i\
Learn you a Haskell’ file.txt
pb@titania:/tmp $ cat file.txt
Learn you a Haskell
for great Good
pb@titania:/tmp $

## xargs

## RANDOM

## -x and set -x

## More

Reference to my bash coding style guide.

E-Mail me your thoughts at comments@mx.buetow.org!

=> ../ Go back to the main site