ioriot

18 files changed, 838 insertions, 161 deletions

diff --git a/buetow.org.sh b/buetow.org.sh
index b2df6aa3..3e32a99b 100755
--- a/buetow.org.sh
+++ b/buetow.org.sh
@@ -4,7 +4,7 @@ declare -r ARG=$1; shift
 source buetow.org.conf
 
 declare DATE=date
-if where gdate &>/dev/null; then
+if which gdate &>/dev/null; then
     DATE=gdate
 fi
 
@@ -34,7 +34,7 @@ gemfeed::updatemainindex () {
     local -r index_gmi="$CONTENT_DIR/gemtext/index.gmi"
 
     # Remove old gemfeeds from main index
-    sed '/^=> .\/gemfeed\/[0-9]/d;' "$index_gmi" > "$index_gmi.tmp"
+    sed '/^=> .\/gemfeed\/[0-9].* - .*/d;' "$index_gmi" > "$index_gmi.tmp"
     # Add current gemfeeds to main index
     sed -n '/^=> / { s| ./| ./gemfeed/|; p; }' "$gemfeed_dir/index.gmi" >> "$index_gmi.tmp"
 
@@ -60,13 +60,13 @@ GEMFEED
         # Extract the date from the file name.
         local filename_date=$(basename "$gemfeed_dir/$gmi_file" | cut -d- -f1,2,3)
 
-        echo "=> ./$gmi_file $filename_date $title" >> "$gemfeed_dir/index.gmi.tmp"
+        echo "=> ./$gmi_file $filename_date - $title" >> "$gemfeed_dir/index.gmi.tmp"
     done
 
     mv "$gemfeed_dir/index.gmi.tmp" "$gemfeed_dir/index.gmi"
     git add "$gemfeed_dir/index.gmi"
 
-    # gemfeed::updatemainindex
+    gemfeed::updatemainindex
 }
 
 ## Atom module
@@ -382,7 +382,7 @@ html::test () {
 main::help () {
     cat <<HELPHERE
 $0's possible arguments:
-    --atom
+    --feed
     --publish
     --test
     --help
@@ -393,7 +393,7 @@ case $ARG in
     --test)
         html::test
         ;;
-    --atom)
+    --feed)
         atomfeed::generate
         ;;
     --publish)
diff --git a/content/gemtext/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.gmi b/content/gemtext/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.gmi
index 6bfad89e..e2019901 100644
--- a/content/gemtext/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.gmi
+++ b/content/gemtext/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.gmi
@@ -1,4 +1,4 @@
-> Written by Paul Buetow 2018-06-01, last updated 2021-05-08
+ Written by Paul Buetow 2018-06-01, last updated 2021-05-08
 
 => ../ Go back to the main site
 
@@ -25,6 +25,8 @@ I havn't worked on I/O Riot for some time now, but all what is written here is s
 
 I am eagerly looking forward to revamp I/O Riot so that it uses the new BPF Linux capabilities instead of Systemtap. Also, when I wrote I/O Riot initially, I didn't have any experience with the Go programming language yet and therefore I wrote it in C. Once it gets revamped I might consider using Go instead of C as it would spare me from many segmentation faults and headaches during development ;-). I might also just stick to C for plain performance reasons and just refactor the code dealing with concurrency.
 
+Pleace notice that some of the screenshots show the command "ioreplay" instead of "ioriot". That's because the name has changed after taking those.
+
 # The article
 
 With I/O Riot IT administrators can load test and optimize the I/O subsystem of Linux-based operating systems. The tool makes it possible to record I/O patterns and replay them at a later time as often as desired. This means bottlenecks can be reproduced and eradicated. 
@@ -81,9 +83,9 @@ During recording, ioriot acts as a wrapper and executes all relevant Systemtap c
 % sudo ioriot -c io.capture
 ```
 
-A Ctrl-C (SIGINT) stops recording prematurely. Otherwise, ioriot terminates itself automatically after ten minutes. Depending on the system load, the output file can grow to several gigabytes. Only metadata is logged, not the read and written data itself. When replaying later, only random data is used. Under certain circumstances, Systemtap may omit some system calls and issue warnings. This is to ensure that Systemtap does not consume too many resources.
+=> ./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure1-ioriot-io-recording.png Screenshot I/O recording
 
-=> ./2018-06-01-realistic-load-testing-with-ioriot/figure1-ioriot-io-recording.png Screenshot I/O recording
+A Ctrl-C (SIGINT) stops recording prematurely. Otherwise, ioriot terminates itself automatically after ten minutes. Depending on the system load, the output file can grow to several gigabytes. Only metadata is logged, not the read and written data itself. When replaying later, only random data is used. Under certain circumstances, Systemtap may omit some system calls and issue warnings. This is to ensure that Systemtap does not consume too many resources.
 
 ### Test preparation
 
@@ -105,9 +107,9 @@ The test will most likely want to access existing files. These are files the tes
 
 To avoid any damage to the running system, ioreplay only works in special directories. The tool creates a separate subdirectory for each file system mount point (e.g. /, /usr/local, /store/00,...) (here: /.ioriot/TESTNAME, /usr/local/.ioriot/TESTNAME, /store/00/.ioriot/TESTNAME,...). By default, the working directory of ioriot is /usr/local/ioriot/TESTNAME.
 
-You must re-initialize the environment before each run. Data from previous tests will be moved to a trash directory automatically, which can be finally deleted with "sudo ioriot -P".
+=> ./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure2-ioriot-test-preparation.png Screenshot test preparation
 
-=> ./2018-06-01-realistic-load-testing-with-ioriot/figure2-ioriot-test-preparation.png Screenshot test preparation
+You must re-initialize the environment before each run. Data from previous tests will be moved to a trash directory automatically, which can be finally deleted with "sudo ioriot -P".
 
 ### Replay
 
@@ -115,7 +117,7 @@ After initialization, you can replay the log with -r. You can use -R to initiate
 
 You can also influence the playback speed: "-s 0" is interpreted as "Playback as fast as possible" and is the default setting. With "-s 1" all operations are performed at original speed. "-s 2" would double the playback speed and "-s 0.5" would halve it.
 
-=> ./2018-06-01-realistic-load-testing-with-ioriot/figure3-ioriot-replay.png Screenshot replaying I/O
+=> ./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure3-ioriot-replay.png Screenshot replaying I/O
 
 As an initial test, for example, you could compare the two Linux I/O schedulers CFQ and Deadline and check which scheduler the test runs the fastest. They run the test separately for each scheduler. The following shell loop iterates through all attached block devices of the system and changes their I/O scheduler to the one specified in variable $new_scheduler (in this case either cfq or deadline). Subsequently, all I/O events from the io.replay protocol are played back. At the end, an output file with statistics is generated:
 
@@ -157,13 +159,13 @@ Total time: 1213.00s
 
 In any case, you should also set up a time series database, such as Graphite, where the I/O throughput can be plotted. Figures 4 and 5 show the read and write access times of both tests. The break-in makes it clear when the CFQ test ended and the deadline test was started. The reading latency of both tests is similar. Write latency is dramatically improved using the Deadline Scheduler.
 
-=> ./2018-06-01-realistic-load-testing-with-ioriot/figure4-ioriot-read-latency.png Graphite visualization of the mean read access times in ms with CFQ and Deadline Scheduler.
+=> ./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure4-ioriot-read-latency.png Graphite visualization of the mean read access times in ms with CFQ and Deadline Scheduler.
 
-=> ./2018-06-01-realistic-load-testing-with-ioriot/figure5-ioriot-write-latency.png Graphite visualization of the average write access times in ms with CFQ and Deadline Scheduler.
+=> ./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure5-ioriot-write-latency.png Graphite visualization of the average write access times in ms with CFQ and Deadline Scheduler.
 
 You should also take a look at the iostat tool. The iostat screenshot shows the output of iostat -x 10 during a test run. As you can see, a block device is fully loaded with 99% utilization, while all other block devices still have sufficient buffer. This could be an indication of poor data distribution in the storage system and is worth pursuing. It is not uncommon for I/O Riot to reveal software problems.
 
-=> ./2018-06-01-realistic-load-testing-with-ioriot/figure6-iostat.png Output of iostat. The block device sdy seems to be almost fully utilized by 99%.
+=> ./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure6-iostat.png Output of iostat. The block device sdy seems to be almost fully utilized by 99%.
 
 ## I/O Riot is Open Source
 
diff --git a/content/gemtext/gemfeed/DRAFT-buetow.org.sh-One-Bash-script-to-rule-it-all.gmi b/content/gemtext/gemfeed/DRAFT-buetow.org.sh-One-Bash-script-to-rule-it-all.gmi.draft
index be65b8b9..be65b8b9 100644
--- a/content/gemtext/gemfeed/DRAFT-buetow.org.sh-One-Bash-script-to-rule-it-all.gmi
+++ b/content/gemtext/gemfeed/DRAFT-buetow.org.sh-One-Bash-script-to-rule-it-all.gmi.draft
diff --git a/content/gemtext/gemfeed/atom.xml b/content/gemtext/gemfeed/atom.xml
index f894c98a..409ccfc3 100644
--- a/content/gemtext/gemfeed/atom.xml
+++ b/content/gemtext/gemfeed/atom.xml
@@ -1,24 +1,27 @@
 <?xml version="1.0" encoding="utf-8"?>
 <feed xmlns="http://www.w3.org/2005/Atom">
-    <updated>2021-05-07T22:55:55+01:00</updated>
+    <updated>2021-05-08T10:29:53+01:00</updated>
     <title>buetow.org feed</title>
     <subtitle>Having fun with computers!</subtitle>
     <link href="gemini://buetow.org/gemfeed/atom.xml" rel="self" />
     <link href="gemini://buetow.org/" />
     <id>gemini://buetow.org/</id>
     <entry>
-        <title>Welcome to the Geminispace</title>
+        <title></title>
         <link href="gemini://buetow.org/gemfeed/2021-04-24-welcome-to-the-geminispace.gmi" />
         <id>gemini://buetow.org/gemfeed/2021-04-24-welcome-to-the-geminispace.gmi</id>
-        <updated>2021-04-24T19:28:41+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>Have you reached this article already via Gemini? You need a special client for that, web browsers such as Firefox, Chrome, Safari etc. don't support the Gemini protocol. The Gemini address of this site (or the address of this capsule as people say in Geminispace) is: ... to read on visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>Welcome to the Geminispace</h1>
+                <pre>Written by Paul Buetow 2021-04-24, last updated 2021-04-30</pre>
+<pre>ASCII Art by Andy Hood</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Welcome to the Geminispace</h1>
 <p>Have you reached this article already via Gemini? You need a special client for that, web browsers such as Firefox, Chrome, Safari etc. don't support the Gemini protocol. The Gemini address of this site (or the address of this capsule as people say in Geminispace) is:</p>
 <a class="textlink" href="gemini://buetow.org">gemini://buetow.org</a><br />
 <p>If you however still use HTTP then you are just surfing the fallback HTML version of this capsule. In that case I suggest reading on what this is all about :-).</p>
@@ -73,18 +76,20 @@
         </content>
     </entry>
     <entry>
-        <title>DTail - The distributed log tail program</title>
+        <title></title>
         <link href="gemini://buetow.org/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi" />
         <id>gemini://buetow.org/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi</id>
-        <updated>2021-04-22T19:28:41+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>This article first appeared at the Mimecast Engineering Blog but I made it available here in my personal Gemini capsule too. ...to read on visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>DTail - The distributed log tail program</h1>
+                <pre>Written by Paul Buetow 2021-04-22, last updated 2021-04-26</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>DTail - The distributed log tail program</h1>
 <i>DTail logo image:</i><a href="https://buetow.org/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program/dtail-logo.png"><img alt="DTail logo image" title="DTail logo image" src="https://buetow.org/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program/dtail-logo.png" /></a><br />
 <p>This article first appeared at the Mimecast Engineering Blog but I made it available here in my personal Gemini capsule too.</p>
 <a class="textlink" href="https://medium.com/mimecast-engineering/dtail-the-distributed-log-tail-program-79b8087904bb">Original Mimecast Engineering Blog post at Medium</a><br />
@@ -153,18 +158,160 @@ dtail –servers serverlist.txt –files ‘/var/log/*.log’ –regex ‘(?i:er
         </content>
     </entry>
     <entry>
-        <title>Methods in C</title>
+        <title></title>
+        <link href="gemini://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.gmi" />
+        <id>gemini://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.gmi</id>
+        <updated></updated>
+        <author>
+            <name></name>
+            <email></email>
+        </author>
+        <summary></summary>
+        <content type="xhtml">
+            <div xmlns="http://www.w3.org/1999/xhtml">
+                <p> Written by Paul Buetow 2018-06-01, last updated 2021-05-08</p>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Realistic load testing with I/O Riot for Linux</h1>
+<pre>
+       .---.
+      /     \
+      \.@-@./
+      /`\_/`\
+     //  _  \\
+    | \     )|_
+   /`\_`&gt;  &lt;_/ \
+jgs\__/'---'\__/
+</pre>
+<h2>Foreword</h2>
+<p>This text first was published in the german ADMIN computer Magazine. 3 years have passed since and I decided to publish it on my blog too. </p>
+<a class="textlink" href="https://www.admin-magazin.de/Das-Heft/2018/06/Realistische-Lasttests-mit-I-O-Riot">https://www.admin-magazin.de/Das-Heft/2018/06/Realistische-Lasttests-mit-I-O-Riot</a><br />
+<p>I havn't worked on I/O Riot for some time now, but all what is written here is still valid. I am still using I/O Riot to debug I/O issues and pattern once in a while, so by all means the tool is not obsolete yet. The tool even helped to resolve a major production incident at work involving I/O.</p>
+<p>I am eagerly looking forward to revamp I/O Riot so that it uses the new BPF Linux capabilities instead of Systemtap. Also, when I wrote I/O Riot initially, I didn't have any experience with the Go programming language yet and therefore I wrote it in C. Once it gets revamped I might consider using Go instead of C as it would spare me from many segmentation faults and headaches during development ;-). I might also just stick to C for plain performance reasons and just refactor the code dealing with concurrency.</p>
+<p>Pleace notice that some of the screenshots show the command "ioreplay" instead of "ioriot". That's because the name has changed after taking those.</p>
+<h1>The article</h1>
+<p>With I/O Riot IT administrators can load test and optimize the I/O subsystem of Linux-based operating systems. The tool makes it possible to record I/O patterns and replay them at a later time as often as desired. This means bottlenecks can be reproduced and eradicated. </p>
+<p>When storing huge amounts of data, such as more than 200 billion archived emails at Mimecast, it's not only the available storage capacity that matters, but also the data throughput and latency. At the same time, operating costs must be kept as low as possible. The more systems involved, the more important it is to optimize the hardware, the operating system and the applications running on it.</p>
+<h2>Background: Existing Techniques</h2>
+<p>Conventional I/O benchmarking: Administrators usually use open source benchmarking tools like IOZone and bonnie++. Available database systems such as Redis and MySQL come with their own benchmarking tools. The common problem with these tools is that they work with prescribed artificial I/O patterns. Although this can test both sequential and randomized data access, the patterns do not correspond to what can be found on production systems.</p>
+<p>Testing by load test environment: Another option is to use a separate load test environment in which, as far as possible, a production environment with all its dependencies is simulated. However, an environment consisting of many microservices is very complex. Microservices are usually managed by different teams, which means extra coordination effort for each load test. Another challenge is to generate the load as authentically as possible so that the patterns correspond to a productive environment. Such a load test environment can only handle as many requests as its weakest link can handle. For example, load generators send many read and write requests to a frontend microservice, whereby the frontend forwards the requests to a backend microservice responsible for storing the data. If the frontend service does not process the requests efficiently enough, the backend service is not well utilized in the first place. As a rule, all microservices are clustered across many servers, which makes everything even more complicated. Under all these conditions it is very difficult to test I/O of separate backend systems. Moreover, for many small and medium-sized companies, a separate load test environment would not be feasible for cost reasons.</p>
+<p>Testing in the production environment: For these reasons, benchmarks are often carried out in the production environment. In order to derive value from this such tests are especially performed during peak hours when systems are under high load. However, testing on production systems is associated with risks and can lead to failure or loss of data without adequate protection.</p>
+<h2>Benchmarking the Email Cloud at Mimecast</h2>
+<p>For email archiving, Mimecast uses an internally developed microservice, which is operated directly on Linux-based storage systems. A storage cluster is divided into several replication volumes. Data is always replicated three times across two secure data centers. Customer data is automatically allocated to one or more volumes, depending on throughput, so that all volumes are automatically assigned the same load. Customer data is archived on conventional, but inexpensive hard disks with several terabytes of storage capacity each. I/O benchmarking proved difficult for all the reasons mentioned above. Furthermore, there are no ready-made tools for this purpose in the case of self-developed software. The service operates on many block devices simultaneously, which can make the RAID controller a bottleneck. None of the freely available benchmarking tools can test several block devices at the same time without extra effort. In addition, emails typically consist of many small files. Randomized access to many small files is particularly inefficient. In addition to many software adaptations, the hardware and operating system must also be optimized.</p>
+<p>Mimecast encourages employees to be innovative and pursue their own ideas in the form of an internal competition, Pet Project. The goal of the pet project I/O Riot was to simplify OS and hardware level I/O benchmarking. The first prototype of I/O Riot was awarded an internal roadmap prize in the spring of 2017. A few months later, I/O Riot was used to reduce write latency in the storage clusters by about 50%. The improvement was first verified by I/O replay on a test system and then successively applied to all storage systems. I/O Riot was also used to resolve a production incident related to disk I/O load.</p>
+<h2>Using I/O Riot</h2>
+<p>First, all I/O events are logged to a file on a production system with I/O Riot. It is then copied to a test system where all events are replayed in the same way. The crucial point here is that you can reproduce I/O patterns as they are found on a production system as often as you like on a test system. This results in the possibility of optimizing the set screws on the system after each run.</p>
+<h3>Installation</h3>
+<p>I/O Riot was tested under CentOS 7.2 x86_64. For compiling, the GNU C compiler and Systemtap including kernel debug information are required. Other Linux distributions are theoretically compatible but untested. First of all, you should update the systems involved as follows:</p>
+<pre>
+% sudo yum update
+</pre>
+<p>If the kernel is updated, please restart the system. The installation would be done without a restart but this would complicate the installation. The installed kernel version should always correspond to the currently running kernel. You can then install I/O Riot as follows:</p>
+<pre>
+% sudo yum install gcc git systemtap yum-utils kernel-devel-$(uname -r)
+% sudo debuginfo-install kernel-$(uname -r)
+% git clone https://github.com/mimecast/ioriot
+% cd ioriot
+% make
+% sudo make install
+% export PATH=$PATH:/opt/ioriot/bin
+</pre>
+<p>Note: It is not best practice to install any compilers on production systems. For further information please have a look at the enclosed README.md.</p>
+<h3>Recording of I/O events</h3>
+<p>All I/O events are kernel related. If a process wants to perform an I/O operation, such as opening a file, it must inform the kernel of this by a system call (short syscall). I/O Riot relies on the Systemtap tool to record I/O syscalls. Systemtap, available for all popular Linux distributions, helps you to take a look at the running kernel in productive environments, which makes it predestined to monitor all I/O-relevant Linux syscalls and log them to a file. Other tools, such as strace, are not an alternative because they slow down the system too much.</p>
+<p>During recording, ioriot acts as a wrapper and executes all relevant Systemtap commands for you. Use the following command to log all events to io.capture:</p>
+<pre>
+% sudo ioriot -c io.capture
+</pre>
+<i>Screenshot I/O recording:</i><a href="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure1-ioriot-io-recording.png"><img alt="Screenshot I/O recording" title="Screenshot I/O recording" src="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure1-ioriot-io-recording.png" /></a><br />
+<p>A Ctrl-C (SIGINT) stops recording prematurely. Otherwise, ioriot terminates itself automatically after ten minutes. Depending on the system load, the output file can grow to several gigabytes. Only metadata is logged, not the read and written data itself. When replaying later, only random data is used. Under certain circumstances, Systemtap may omit some system calls and issue warnings. This is to ensure that Systemtap does not consume too many resources.</p>
+<h3>Test preparation</h3>
+<p>Then copy io.capture to a test system. The log also contains all accesses to the pseudo file systems devfs, sysfs and procfs. This makes little sense, which is why you must first generate a cleaned and playable version io.replay from io.capture as follows:</p>
+<pre>
+% sudo ioriot -c io.capture -r io.replay -u $USER -n TESTNAME
+</pre>
+<p>The parameter -n allows you to assign a freely selectable test name. An arbitrary system user under which the test is to be played is specified via paramater -u.</p>
+<h3>Test Initialization</h3>
+<p>The test will most likely want to access existing files. These are files the test wants to read but does not create by itself. The existence of these must be ensured before the test. You can do this as follows:</p>
+<pre>
+% sudo ioriot -i io.replay
+</pre>
+<p>To avoid any damage to the running system, ioreplay only works in special directories. The tool creates a separate subdirectory for each file system mount point (e.g. /, /usr/local, /store/00,...) (here: /.ioriot/TESTNAME, /usr/local/.ioriot/TESTNAME, /store/00/.ioriot/TESTNAME,...). By default, the working directory of ioriot is /usr/local/ioriot/TESTNAME.</p>
+<i>Screenshot test preparation:</i><a href="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure2-ioriot-test-preparation.png"><img alt="Screenshot test preparation" title="Screenshot test preparation" src="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure2-ioriot-test-preparation.png" /></a><br />
+<p>You must re-initialize the environment before each run. Data from previous tests will be moved to a trash directory automatically, which can be finally deleted with "sudo ioriot -P".</p>
+<h3>Replay</h3>
+<p>After initialization, you can replay the log with -r. You can use -R to initiate both test initialization and replay in a single command and -S can be used to specify a file in which statistics are written after the test run.</p>
+<p>You can also influence the playback speed: "-s 0" is interpreted as "Playback as fast as possible" and is the default setting. With "-s 1" all operations are performed at original speed. "-s 2" would double the playback speed and "-s 0.5" would halve it.</p>
+<i>Screenshot replaying I/O:</i><a href="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure3-ioriot-replay.png"><img alt="Screenshot replaying I/O" title="Screenshot replaying I/O" src="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure3-ioriot-replay.png" /></a><br />
+<p>As an initial test, for example, you could compare the two Linux I/O schedulers CFQ and Deadline and check which scheduler the test runs the fastest. They run the test separately for each scheduler. The following shell loop iterates through all attached block devices of the system and changes their I/O scheduler to the one specified in variable $new_scheduler (in this case either cfq or deadline). Subsequently, all I/O events from the io.replay protocol are played back. At the end, an output file with statistics is generated:</p>
+<pre>
+% new_scheduler=cfq
+% for scheduler in /sys/block/*/queue/scheduler; do
+    echo $new_scheduler | sudo tee $scheduler
+done
+% sudo ioriot -R io.replay -S cfq.txt
+% new_scheduler=deadline
+% for scheduler in /sys/block/*/queue/scheduler; do
+   echo $new_scheduler | sudo tee $scheduler
+done
+% sudo ioriot -R io.replay -S deadline.txt
+</pre>
+<p>According to the results, the test could run 940 seconds faster with Deadline Scheduler:</p>
+<pre>
+% cat cfq.txt
+Num workers: 4
+hreads per worker: 128
+otal threads: 512
+Highest loadavg: 259.29
+Performed ioops: 218624596
+Average ioops/s: 101544.17
+Time ahead: 1452s
+Total time: 2153.00s
+% cat deadline.txt
+Num workers: 4
+Threads per worker: 128
+Total threads: 512
+Highest loadavg: 342.45
+Performed ioops: 218624596
+Average ioops/s: 180234.62
+Time ahead: 2392s
+Total time: 1213.00s
+</pre>
+<p>In any case, you should also set up a time series database, such as Graphite, where the I/O throughput can be plotted. Figures 4 and 5 show the read and write access times of both tests. The break-in makes it clear when the CFQ test ended and the deadline test was started. The reading latency of both tests is similar. Write latency is dramatically improved using the Deadline Scheduler.</p>
+<i>Graphite visualization of the mean read access times in ms with CFQ and Deadline Scheduler.:</i><a href="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure4-ioriot-read-latency.png"><img alt="Graphite visualization of the mean read access times in ms with CFQ and Deadline Scheduler." title="Graphite visualization of the mean read access times in ms with CFQ and Deadline Scheduler." src="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure4-ioriot-read-latency.png" /></a><br />
+<i>Graphite visualization of the average write access times in ms with CFQ and Deadline Scheduler.:</i><a href="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure5-ioriot-write-latency.png"><img alt="Graphite visualization of the average write access times in ms with CFQ and Deadline Scheduler." title="Graphite visualization of the average write access times in ms with CFQ and Deadline Scheduler." src="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure5-ioriot-write-latency.png" /></a><br />
+<p>You should also take a look at the iostat tool. The iostat screenshot shows the output of iostat -x 10 during a test run. As you can see, a block device is fully loaded with 99% utilization, while all other block devices still have sufficient buffer. This could be an indication of poor data distribution in the storage system and is worth pursuing. It is not uncommon for I/O Riot to reveal software problems.</p>
+<i>Output of iostat. The block device sdy seems to be almost fully utilized by 99%.:</i><a href="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure6-iostat.png"><img alt="Output of iostat. The block device sdy seems to be almost fully utilized by 99%." title="Output of iostat. The block device sdy seems to be almost fully utilized by 99%." src="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure6-iostat.png" /></a><br />
+<h2>I/O Riot is Open Source</h2>
+<p>The tool has already proven to be very useful and will continue to be actively developed as time and priority permits. Mimecast intends to be an ongoing contributor to Open Source. You can find I/O Riot at:</p>
+<a class="textlink" href="https://github.com/mimecast/ioriot">https://github.com/mimecast/ioriot</a><br />
+<h2>Systemtap</h2>
+<p>Systemtap is a tool for the instrumentation of the Linux kernel. The tool provides an AWK-like programming language. Programs written in it are compiled from Systemtap to C- and then into a dynamically loadable kernel module. Loaded into the kernel, the program has access to Linux internals. A Systemtap program written for I/O Riot monitors when, with which parameters, at which time, and from which process I/O syscalls take place and their return values.</p>
+<p>For example, the open syscall opens a file and returns the responsible file descriptor. The read and write syscalls can operate on a file descriptor and return the number of read or written bytes. The close syscall closes a given file descriptor. I/O Riot comes with a ready-made Systemtap program, which you have already compiled into a kernel module and installed to /opt/ioriot. In addition to open, read and close, it logs many other I/O-relevant calls.</p>
+<a class="textlink" href="https://sourceware.org/systemtap/">https://sourceware.org/systemtap/</a><br />
+<h2>More refereces</h2>
+<a class="textlink" href="http://www.iozone.org/">IOZone</a><br />
+<a class="textlink" href="https://www.coker.com.au/bonnie++/">Bonnie++</a><br />
+<a class="textlink" href="https://graphiteapp.org">Graphite</a><br />
+<a class="textlink" href="https://en.wikipedia.org/wiki/Memory-mapped_I/O">Memory mapped I/O</a><br />
+<p>E-Mail me your thoughts at comments@mx.buetow.org!</p>
+            </div>
+        </content>
+    </entry>
+    <entry>
+        <title></title>
         <link href="gemini://buetow.org/gemfeed/2016-11-20-methods-in-c.gmi" />
         <id>gemini://buetow.org/gemfeed/2016-11-20-methods-in-c.gmi</id>
-        <updated>2016-11-20T18:36:51+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>You can do some sort of object oriented programming in the C Programming Language. However, that is very limited. But also very easy and straight forward to use.. .....to read on please visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>Methods in C</h1>
+                <pre>Written by Paul Buetow 2016-11-20</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Methods in C</h1>
 <p>You can do some sort of object oriented programming in the C Programming Language. However, that is very limited. But also very easy and straight forward to use.</p>
 <h2>Example</h2>
 <p>Lets have a look at the following sample program. Basically all you have to do is to add a function pointer such as "calculate" to the definition of struct "something_s". Later, during the struct initialization, assign a function address to that function pointer:</p>
@@ -232,18 +379,20 @@ mult.calculate(mult,a,b));
         </content>
     </entry>
     <entry>
-        <title>Spinning up my own authoritative DNS servers</title>
+        <title></title>
         <link href="gemini://buetow.org/gemfeed/2016-05-22-spinning-up-my-own-authoritative-dns-servers.gmi" />
         <id>gemini://buetow.org/gemfeed/2016-05-22-spinning-up-my-own-authoritative-dns-servers.gmi</id>
-        <updated>2016-05-22T18:59:01+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>Finally, I had time to deploy my own authoritative DNS servers (master and slave) for my domains 'buetow.org' and 'buetow.zone'. My domain name provider is Schlund Technologies. They allow their customers to manually edit the DNS records (BIND files). And they also give you the opportunity to set your own authoritative DNS servers for your domains. From now I am making use of that option.. .....to read on please visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>Spinning up my own authoritative DNS servers</h1>
+                <pre>Written by Paul Buetow 2016-05-22</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Spinning up my own authoritative DNS servers</h1>
 <h2>Background</h2>
 <p>Finally, I had time to deploy my own authoritative DNS servers (master and slave) for my domains "buetow.org" and "buetow.zone". My domain name provider is Schlund Technologies. They allow their customers to manually edit the DNS records (BIND files). And they also give you the opportunity to set your own authoritative DNS servers for your domains. From now, I am making use of that option.</p>
 <a class="textlink" href="http://www.schlundtech.de">Schlund Technologies</a><br />
@@ -456,18 +605,20 @@ apply Service "dig6" {
         </content>
     </entry>
     <entry>
-        <title>Offsite backup with ZFS (Part 2)</title>
+        <title></title>
         <link href="gemini://buetow.org/gemfeed/2016-04-16-offsite-backup-with-zfs-part2.gmi" />
         <id>gemini://buetow.org/gemfeed/2016-04-16-offsite-backup-with-zfs-part2.gmi</id>
-        <updated>2016-04-16T22:43:42+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>I enhanced the procedure a bit. From now on I am having two external 2TB USB hard drives. Both are setup exactly the same way. To decrease the probability that they will not fail at about the same time both drives are of different brands. One drive is kept at the secret location. The other one is kept at home right next to my HP MicroServer. ...to read on visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>Offsite backup with ZFS (Part 2)</h1>
+                <pre>Written by Paul Buetow 2016-04-16</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Offsite backup with ZFS (Part 2)</h1>
 <pre>
  ________________
 |# :           : #|
@@ -491,18 +642,20 @@ apply Service "dig6" {
         </content>
     </entry>
     <entry>
-        <title>Offsite backup with ZFS</title>
+        <title></title>
         <link href="gemini://buetow.org/gemfeed/2016-04-03-offsite-backup-with-zfs.gmi" />
         <id>gemini://buetow.org/gemfeed/2016-04-03-offsite-backup-with-zfs.gmi</id>
-        <updated>2016-04-03T22:43:42+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>When it comes to data storage and potential data loss I am a paranoid person. It is not just due to my job  but also due to a personal experience I encountered over 10 years ago: A single drive failure and loss of all my data (pictures, music, ....). ...to read on visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>Offsite backup with ZFS</h1>
+                <pre>Written by Paul Buetow 2016-04-03</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Offsite backup with ZFS</h1>
 <pre>
  ________________
 |# :           : #|
@@ -531,18 +684,20 @@ apply Service "dig6" {
         </content>
     </entry>
     <entry>
-        <title>Perl Daemon (Service Framework)</title>
+        <title></title>
         <link href="gemini://buetow.org/gemfeed/2011-05-07-perl-daemon-service-framework.gmi" />
         <id>gemini://buetow.org/gemfeed/2011-05-07-perl-daemon-service-framework.gmi</id>
-        <updated>2011-05-07T22:26:02+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>PerlDaemon is a minimal daemon for Linux and other Unix like operating systems programmed in Perl. It is a minimal but pretty functional and fairly generic service framework. This means that it does not do anything useful other than providing a framework for starting, stopping, configuring and logging. In order to do something a module (written in Perl) bust be provided.. .....to read on please visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>Perl Daemon (Service Framework)</h1>
+                <pre>Written by Paul Buetow 2011-05-07, last updated 2021-05-07</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Perl Daemon (Service Framework)</h1>
 <pre>
    a'!   _,,_ a'!   _,,_     a'!   _,,_
      \\_/    \  \\_/    \      \\_/    \.-,
@@ -676,18 +831,29 @@ sub do ($) {
         </content>
     </entry>
     <entry>
-        <title>The Fype Programming Language</title>
+        <title></title>
         <link href="gemini://buetow.org/gemfeed/2010-05-09-the-fype-programming-language.gmi" />
         <id>gemini://buetow.org/gemfeed/2010-05-09-the-fype-programming-language.gmi</id>
-        <updated>2010-05-09T12:48:29+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>Fype is an interpreted programming language created by me for learning and fun. The interpreter is written in C. It has been tested on FreeBSD and NetBSD and may also work on other Unix like operating systems such as Linux based ones. To be honest, besides learning and fun there is really no other use case of why Fype actually exists as many other programming languages are much faster and more powerful.. .....to read on please visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>The Fype Programming Language</h1>
+                <pre>Written by Paul Buetow 2010-05-09, last updated 2021-05-05</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<pre>
+      ____                                      _        __       
+     / / _|_   _ _ __   ___    _   _  ___  __ _| |__    / _|_   _ 
+    / / |_| | | | '_ \ / _ \  | | | |/ _ \/ _` | '_ \  | |_| | | |
+ _ / /|  _| |_| | |_) |  __/  | |_| |  __/ (_| | | | |_|  _| |_| |
+(_)_/ |_|  \__, | .__/ \___|   \__, |\___|\__,_|_| |_(_)_|  \__, |
+           |___/|_|            |___/                        |___/ 
+
+</pre>
+<h1>The Fype Programming Language</h1>
 <p>Fype is an interpreted programming language created by me for learning and fun. The interpreter is written in C. It has been tested on FreeBSD and NetBSD and may also work on other Unix like operating systems such as Linux based ones. To be honest, besides learning and fun there is really no other use case of why Fype actually exists as many other programming languages are much faster and more powerful.</p>
 <p>The Fype syntax is very simple and is using a maximum look ahead of 1 and a very easy top down parsing mechanism. Fype is parsing and interpreting its code simultaneously. This means, that syntax errors are only detected during program runtime. </p>
 <p>Fype is a recursive acronym and means "Fype is For Your Program Execution" or "Fype is Free Yak Programmed for ELF". You could also say "It's not a hype - it's Fype!".</p>
@@ -1082,18 +1248,20 @@ BB
         </content>
     </entry>
     <entry>
-        <title>Perl Poetry</title>
+        <title></title>
         <link href="gemini://buetow.org/gemfeed/2008-06-26-perl-poetry.gmi" />
         <id>gemini://buetow.org/gemfeed/2008-06-26-perl-poetry.gmi</id>
-        <updated>2008-06-26T21:43:51+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>Here are some Perl Poems I wrote. They don't do anything useful when you run them but they don't produce a compiler error either. They only exists for fun and demonstrate what you can do with Perl syntax.. .....to read on please visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>Perl Poetry</h1>
+                <pre>Written by Paul Buetow 2008-06-26, last updated 2021-05-04</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Perl Poetry</h1>
 <pre>
  '\|/'                                  *
 -- * -----
diff --git a/content/gemtext/gemfeed/index.gmi b/content/gemtext/gemfeed/index.gmi
index 8f70ea34..6b35f04a 100644
--- a/content/gemtext/gemfeed/index.gmi
+++ b/content/gemtext/gemfeed/index.gmi
@@ -2,12 +2,13 @@
 
 ## Having fun with computers!
 
-=> ./2021-04-24-welcome-to-the-geminispace.gmi 2021-04-24 Welcome to the Geminispace
-=> ./2021-04-22-dtail-the-distributed-log-tail-program.gmi 2021-04-22 DTail - The distributed log tail program
-=> ./2016-11-20-methods-in-c.gmi 2016-11-20 Methods in C
-=> ./2016-05-22-spinning-up-my-own-authoritative-dns-servers.gmi 2016-05-22 Spinning up my own authoritative DNS servers
-=> ./2016-04-16-offsite-backup-with-zfs-part2.gmi 2016-04-16 Offsite backup with ZFS (Part 2)
-=> ./2016-04-03-offsite-backup-with-zfs.gmi 2016-04-03 Offsite backup with ZFS
-=> ./2011-05-07-perl-daemon-service-framework.gmi 2011-05-07 Perl Daemon (Service Framework)
-=> ./2010-05-09-the-fype-programming-language.gmi 2010-05-09 The Fype Programming Language
-=> ./2008-06-26-perl-poetry.gmi 2008-06-26 Perl Poetry
+=> ./2021-04-24-welcome-to-the-geminispace.gmi 2021-04-24 - Welcome to the Geminispace
+=> ./2021-04-22-dtail-the-distributed-log-tail-program.gmi 2021-04-22 - DTail - The distributed log tail program
+=> ./2018-06-01-realistic-load-testing-with-ioriot-for-linux.gmi 2018-06-01 - Realistic load testing with I/O Riot for Linux
+=> ./2016-11-20-methods-in-c.gmi 2016-11-20 - Methods in C
+=> ./2016-05-22-spinning-up-my-own-authoritative-dns-servers.gmi 2016-05-22 - Spinning up my own authoritative DNS servers
+=> ./2016-04-16-offsite-backup-with-zfs-part2.gmi 2016-04-16 - Offsite backup with ZFS (Part 2)
+=> ./2016-04-03-offsite-backup-with-zfs.gmi 2016-04-03 - Offsite backup with ZFS
+=> ./2011-05-07-perl-daemon-service-framework.gmi 2011-05-07 - Perl Daemon (Service Framework)
+=> ./2010-05-09-the-fype-programming-language.gmi 2010-05-09 - The Fype Programming Language
+=> ./2008-06-26-perl-poetry.gmi 2008-06-26 - Perl Poetry
diff --git a/content/gemtext/index.gmi b/content/gemtext/index.gmi
index d38033a3..3020e633 100644
--- a/content/gemtext/index.gmi
+++ b/content/gemtext/index.gmi
@@ -52,12 +52,13 @@ English is not my mother tongue. So please ignore any errors you might encounter
 
 I have switched blog software multiple times. I might be back filling some of the older articles here. So please don't wonder when suddenly very old posts appear here.
 
-=> ./gemfeed/2021-04-24-welcome-to-the-geminispace.gmi 2021-04-24 Welcome to the Geminispace
-=> ./gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi 2021-04-22 DTail - The distributed log tail program
-=> ./gemfeed/2016-11-20-methods-in-c.gmi 2016-11-20 Methods in C
-=> ./gemfeed/2016-05-22-spinning-up-my-own-authoritative-dns-servers.gmi 2016-05-22 Spinning up my own authoritative DNS servers
-=> ./gemfeed/2016-04-16-offsite-backup-with-zfs-part2.gmi 2016-04-16 Offsite backup with ZFS (Part 2)
-=> ./gemfeed/2016-04-03-offsite-backup-with-zfs.gmi 2016-04-03 Offsite backup with ZFS
-=> ./gemfeed/2011-05-07-perl-daemon-service-framework.gmi 2011-05-07 Perl Daemon (Service Framework)
-=> ./gemfeed/2010-05-09-the-fype-programming-language.gmi 2010-05-09 The Fype Programming Language
-=> ./gemfeed/2008-06-26-perl-poetry.gmi 2008-06-26 Perl Poetry
+=> ./gemfeed/2021-04-24-welcome-to-the-geminispace.gmi 2021-04-24 - Welcome to the Geminispace
+=> ./gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi 2021-04-22 - DTail - The distributed log tail program
+=> ./gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.gmi 2018-06-01 - Realistic load testing with I/O Riot for Linux
+=> ./gemfeed/2016-11-20-methods-in-c.gmi 2016-11-20 - Methods in C
+=> ./gemfeed/2016-05-22-spinning-up-my-own-authoritative-dns-servers.gmi 2016-05-22 - Spinning up my own authoritative DNS servers
+=> ./gemfeed/2016-04-16-offsite-backup-with-zfs-part2.gmi 2016-04-16 - Offsite backup with ZFS (Part 2)
+=> ./gemfeed/2016-04-03-offsite-backup-with-zfs.gmi 2016-04-03 - Offsite backup with ZFS
+=> ./gemfeed/2011-05-07-perl-daemon-service-framework.gmi 2011-05-07 - Perl Daemon (Service Framework)
+=> ./gemfeed/2010-05-09-the-fype-programming-language.gmi 2010-05-09 - The Fype Programming Language
+=> ./gemfeed/2008-06-26-perl-poetry.gmi 2008-06-26 - Perl Poetry
diff --git a/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.html b/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.html
new file mode 100644
index 00000000..741998a8
--- /dev/null
+++ b/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.html
@@ -0,0 +1,147 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+<title>Having fun with computers!</title>
+<link rel="shortcut icon" type="image/gif" href="/favicon.ico" />
+<style type="text/css">
+body { background-color: #282c34; color: #dfdfdf; margin: auto; max-width: 900px; }
+img { display: block; max-width: 70%;}
+pre { color: #ecbe7b; }
+blockquote { color: #ecbe7b; }
+a { color: #c678dd; text-decoration: none; }
+a.textlink:before { content: " > "; padding-left: 2px; }
+a:hover { color: #51afef; text-decoration: underline; }
+h1 { color: #ff6c6b; }
+h2, h3, h4, h5, h6 { color: #51afef; }
+li { color: #98be65; }
+</style>
+</head>
+<body>
+<p> Written by Paul Buetow 2018-06-01, last updated 2021-05-08</p>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Realistic load testing with I/O Riot for Linux</h1>
+<pre>
+       .---.
+      /     \
+      \.@-@./
+      /`\_/`\
+     //  _  \\
+    | \     )|_
+   /`\_`&gt;  &lt;_/ \
+jgs\__/'---'\__/
+</pre>
+<h2>Foreword</h2>
+<p>This text first was published in the german ADMIN computer Magazine. 3 years have passed since and I decided to publish it on my blog too. </p>
+<a class="textlink" href="https://www.admin-magazin.de/Das-Heft/2018/06/Realistische-Lasttests-mit-I-O-Riot">https://www.admin-magazin.de/Das-Heft/2018/06/Realistische-Lasttests-mit-I-O-Riot</a><br />
+<p>I havn't worked on I/O Riot for some time now, but all what is written here is still valid. I am still using I/O Riot to debug I/O issues and pattern once in a while, so by all means the tool is not obsolete yet. The tool even helped to resolve a major production incident at work involving I/O.</p>
+<p>I am eagerly looking forward to revamp I/O Riot so that it uses the new BPF Linux capabilities instead of Systemtap. Also, when I wrote I/O Riot initially, I didn't have any experience with the Go programming language yet and therefore I wrote it in C. Once it gets revamped I might consider using Go instead of C as it would spare me from many segmentation faults and headaches during development ;-). I might also just stick to C for plain performance reasons and just refactor the code dealing with concurrency.</p>
+<p>Pleace notice that some of the screenshots show the command "ioreplay" instead of "ioriot". That's because the name has changed after taking those.</p>
+<h1>The article</h1>
+<p>With I/O Riot IT administrators can load test and optimize the I/O subsystem of Linux-based operating systems. The tool makes it possible to record I/O patterns and replay them at a later time as often as desired. This means bottlenecks can be reproduced and eradicated. </p>
+<p>When storing huge amounts of data, such as more than 200 billion archived emails at Mimecast, it's not only the available storage capacity that matters, but also the data throughput and latency. At the same time, operating costs must be kept as low as possible. The more systems involved, the more important it is to optimize the hardware, the operating system and the applications running on it.</p>
+<h2>Background: Existing Techniques</h2>
+<p>Conventional I/O benchmarking: Administrators usually use open source benchmarking tools like IOZone and bonnie++. Available database systems such as Redis and MySQL come with their own benchmarking tools. The common problem with these tools is that they work with prescribed artificial I/O patterns. Although this can test both sequential and randomized data access, the patterns do not correspond to what can be found on production systems.</p>
+<p>Testing by load test environment: Another option is to use a separate load test environment in which, as far as possible, a production environment with all its dependencies is simulated. However, an environment consisting of many microservices is very complex. Microservices are usually managed by different teams, which means extra coordination effort for each load test. Another challenge is to generate the load as authentically as possible so that the patterns correspond to a productive environment. Such a load test environment can only handle as many requests as its weakest link can handle. For example, load generators send many read and write requests to a frontend microservice, whereby the frontend forwards the requests to a backend microservice responsible for storing the data. If the frontend service does not process the requests efficiently enough, the backend service is not well utilized in the first place. As a rule, all microservices are clustered across many servers, which makes everything even more complicated. Under all these conditions it is very difficult to test I/O of separate backend systems. Moreover, for many small and medium-sized companies, a separate load test environment would not be feasible for cost reasons.</p>
+<p>Testing in the production environment: For these reasons, benchmarks are often carried out in the production environment. In order to derive value from this such tests are especially performed during peak hours when systems are under high load. However, testing on production systems is associated with risks and can lead to failure or loss of data without adequate protection.</p>
+<h2>Benchmarking the Email Cloud at Mimecast</h2>
+<p>For email archiving, Mimecast uses an internally developed microservice, which is operated directly on Linux-based storage systems. A storage cluster is divided into several replication volumes. Data is always replicated three times across two secure data centers. Customer data is automatically allocated to one or more volumes, depending on throughput, so that all volumes are automatically assigned the same load. Customer data is archived on conventional, but inexpensive hard disks with several terabytes of storage capacity each. I/O benchmarking proved difficult for all the reasons mentioned above. Furthermore, there are no ready-made tools for this purpose in the case of self-developed software. The service operates on many block devices simultaneously, which can make the RAID controller a bottleneck. None of the freely available benchmarking tools can test several block devices at the same time without extra effort. In addition, emails typically consist of many small files. Randomized access to many small files is particularly inefficient. In addition to many software adaptations, the hardware and operating system must also be optimized.</p>
+<p>Mimecast encourages employees to be innovative and pursue their own ideas in the form of an internal competition, Pet Project. The goal of the pet project I/O Riot was to simplify OS and hardware level I/O benchmarking. The first prototype of I/O Riot was awarded an internal roadmap prize in the spring of 2017. A few months later, I/O Riot was used to reduce write latency in the storage clusters by about 50%. The improvement was first verified by I/O replay on a test system and then successively applied to all storage systems. I/O Riot was also used to resolve a production incident related to disk I/O load.</p>
+<h2>Using I/O Riot</h2>
+<p>First, all I/O events are logged to a file on a production system with I/O Riot. It is then copied to a test system where all events are replayed in the same way. The crucial point here is that you can reproduce I/O patterns as they are found on a production system as often as you like on a test system. This results in the possibility of optimizing the set screws on the system after each run.</p>
+<h3>Installation</h3>
+<p>I/O Riot was tested under CentOS 7.2 x86_64. For compiling, the GNU C compiler and Systemtap including kernel debug information are required. Other Linux distributions are theoretically compatible but untested. First of all, you should update the systems involved as follows:</p>
+<pre>
+% sudo yum update
+</pre>
+<p>If the kernel is updated, please restart the system. The installation would be done without a restart but this would complicate the installation. The installed kernel version should always correspond to the currently running kernel. You can then install I/O Riot as follows:</p>
+<pre>
+% sudo yum install gcc git systemtap yum-utils kernel-devel-$(uname -r)
+% sudo debuginfo-install kernel-$(uname -r)
+% git clone https://github.com/mimecast/ioriot
+% cd ioriot
+% make
+% sudo make install
+% export PATH=$PATH:/opt/ioriot/bin
+</pre>
+<p>Note: It is not best practice to install any compilers on production systems. For further information please have a look at the enclosed README.md.</p>
+<h3>Recording of I/O events</h3>
+<p>All I/O events are kernel related. If a process wants to perform an I/O operation, such as opening a file, it must inform the kernel of this by a system call (short syscall). I/O Riot relies on the Systemtap tool to record I/O syscalls. Systemtap, available for all popular Linux distributions, helps you to take a look at the running kernel in productive environments, which makes it predestined to monitor all I/O-relevant Linux syscalls and log them to a file. Other tools, such as strace, are not an alternative because they slow down the system too much.</p>
+<p>During recording, ioriot acts as a wrapper and executes all relevant Systemtap commands for you. Use the following command to log all events to io.capture:</p>
+<pre>
+% sudo ioriot -c io.capture
+</pre>
+<i>Screenshot I/O recording:</i><a href="./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure1-ioriot-io-recording.png"><img alt="Screenshot I/O recording" title="Screenshot I/O recording" src="./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure1-ioriot-io-recording.png" /></a><br />
+<p>A Ctrl-C (SIGINT) stops recording prematurely. Otherwise, ioriot terminates itself automatically after ten minutes. Depending on the system load, the output file can grow to several gigabytes. Only metadata is logged, not the read and written data itself. When replaying later, only random data is used. Under certain circumstances, Systemtap may omit some system calls and issue warnings. This is to ensure that Systemtap does not consume too many resources.</p>
+<h3>Test preparation</h3>
+<p>Then copy io.capture to a test system. The log also contains all accesses to the pseudo file systems devfs, sysfs and procfs. This makes little sense, which is why you must first generate a cleaned and playable version io.replay from io.capture as follows:</p>
+<pre>
+% sudo ioriot -c io.capture -r io.replay -u $USER -n TESTNAME
+</pre>
+<p>The parameter -n allows you to assign a freely selectable test name. An arbitrary system user under which the test is to be played is specified via paramater -u.</p>
+<h3>Test Initialization</h3>
+<p>The test will most likely want to access existing files. These are files the test wants to read but does not create by itself. The existence of these must be ensured before the test. You can do this as follows:</p>
+<pre>
+% sudo ioriot -i io.replay
+</pre>
+<p>To avoid any damage to the running system, ioreplay only works in special directories. The tool creates a separate subdirectory for each file system mount point (e.g. /, /usr/local, /store/00,...) (here: /.ioriot/TESTNAME, /usr/local/.ioriot/TESTNAME, /store/00/.ioriot/TESTNAME,...). By default, the working directory of ioriot is /usr/local/ioriot/TESTNAME.</p>
+<i>Screenshot test preparation:</i><a href="./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure2-ioriot-test-preparation.png"><img alt="Screenshot test preparation" title="Screenshot test preparation" src="./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure2-ioriot-test-preparation.png" /></a><br />
+<p>You must re-initialize the environment before each run. Data from previous tests will be moved to a trash directory automatically, which can be finally deleted with "sudo ioriot -P".</p>
+<h3>Replay</h3>
+<p>After initialization, you can replay the log with -r. You can use -R to initiate both test initialization and replay in a single command and -S can be used to specify a file in which statistics are written after the test run.</p>
+<p>You can also influence the playback speed: "-s 0" is interpreted as "Playback as fast as possible" and is the default setting. With "-s 1" all operations are performed at original speed. "-s 2" would double the playback speed and "-s 0.5" would halve it.</p>
+<i>Screenshot replaying I/O:</i><a href="./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure3-ioriot-replay.png"><img alt="Screenshot replaying I/O" title="Screenshot replaying I/O" src="./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure3-ioriot-replay.png" /></a><br />
+<p>As an initial test, for example, you could compare the two Linux I/O schedulers CFQ and Deadline and check which scheduler the test runs the fastest. They run the test separately for each scheduler. The following shell loop iterates through all attached block devices of the system and changes their I/O scheduler to the one specified in variable $new_scheduler (in this case either cfq or deadline). Subsequently, all I/O events from the io.replay protocol are played back. At the end, an output file with statistics is generated:</p>
+<pre>
+% new_scheduler=cfq
+% for scheduler in /sys/block/*/queue/scheduler; do
+    echo $new_scheduler | sudo tee $scheduler
+done
+% sudo ioriot -R io.replay -S cfq.txt
+% new_scheduler=deadline
+% for scheduler in /sys/block/*/queue/scheduler; do
+   echo $new_scheduler | sudo tee $scheduler
+done
+% sudo ioriot -R io.replay -S deadline.txt
+</pre>
+<p>According to the results, the test could run 940 seconds faster with Deadline Scheduler:</p>
+<pre>
+% cat cfq.txt
+Num workers: 4
+hreads per worker: 128
+otal threads: 512
+Highest loadavg: 259.29
+Performed ioops: 218624596
+Average ioops/s: 101544.17
+Time ahead: 1452s
+Total time: 2153.00s
+% cat deadline.txt
+Num workers: 4
+Threads per worker: 128
+Total threads: 512
+Highest loadavg: 342.45
+Performed ioops: 218624596
+Average ioops/s: 180234.62
+Time ahead: 2392s
+Total time: 1213.00s
+</pre>
+<p>In any case, you should also set up a time series database, such as Graphite, where the I/O throughput can be plotted. Figures 4 and 5 show the read and write access times of both tests. The break-in makes it clear when the CFQ test ended and the deadline test was started. The reading latency of both tests is similar. Write latency is dramatically improved using the Deadline Scheduler.</p>
+<i>Graphite visualization of the mean read access times in ms with CFQ and Deadline Scheduler.:</i><a href="./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure4-ioriot-read-latency.png"><img alt="Graphite visualization of the mean read access times in ms with CFQ and Deadline Scheduler." title="Graphite visualization of the mean read access times in ms with CFQ and Deadline Scheduler." src="./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure4-ioriot-read-latency.png" /></a><br />
+<i>Graphite visualization of the average write access times in ms with CFQ and Deadline Scheduler.:</i><a href="./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure5-ioriot-write-latency.png"><img alt="Graphite visualization of the average write access times in ms with CFQ and Deadline Scheduler." title="Graphite visualization of the average write access times in ms with CFQ and Deadline Scheduler." src="./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure5-ioriot-write-latency.png" /></a><br />
+<p>You should also take a look at the iostat tool. The iostat screenshot shows the output of iostat -x 10 during a test run. As you can see, a block device is fully loaded with 99% utilization, while all other block devices still have sufficient buffer. This could be an indication of poor data distribution in the storage system and is worth pursuing. It is not uncommon for I/O Riot to reveal software problems.</p>
+<i>Output of iostat. The block device sdy seems to be almost fully utilized by 99%.:</i><a href="./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure6-iostat.png"><img alt="Output of iostat. The block device sdy seems to be almost fully utilized by 99%." title="Output of iostat. The block device sdy seems to be almost fully utilized by 99%." src="./2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure6-iostat.png" /></a><br />
+<h2>I/O Riot is Open Source</h2>
+<p>The tool has already proven to be very useful and will continue to be actively developed as time and priority permits. Mimecast intends to be an ongoing contributor to Open Source. You can find I/O Riot at:</p>
+<a class="textlink" href="https://github.com/mimecast/ioriot">https://github.com/mimecast/ioriot</a><br />
+<h2>Systemtap</h2>
+<p>Systemtap is a tool for the instrumentation of the Linux kernel. The tool provides an AWK-like programming language. Programs written in it are compiled from Systemtap to C- and then into a dynamically loadable kernel module. Loaded into the kernel, the program has access to Linux internals. A Systemtap program written for I/O Riot monitors when, with which parameters, at which time, and from which process I/O syscalls take place and their return values.</p>
+<p>For example, the open syscall opens a file and returns the responsible file descriptor. The read and write syscalls can operate on a file descriptor and return the number of read or written bytes. The close syscall closes a given file descriptor. I/O Riot comes with a ready-made Systemtap program, which you have already compiled into a kernel module and installed to /opt/ioriot. In addition to open, read and close, it logs many other I/O-relevant calls.</p>
+<a class="textlink" href="https://sourceware.org/systemtap/">https://sourceware.org/systemtap/</a><br />
+<h2>More refereces</h2>
+<a class="textlink" href="http://www.iozone.org/">IOZone</a><br />
+<a class="textlink" href="https://www.coker.com.au/bonnie++/">Bonnie++</a><br />
+<a class="textlink" href="https://graphiteapp.org">Graphite</a><br />
+<a class="textlink" href="https://en.wikipedia.org/wiki/Memory-mapped_I/O">Memory mapped I/O</a><br />
+<p>E-Mail me your thoughts at comments@mx.buetow.org!</p>
+</body>
+</html>
diff --git a/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure1-ioriot-io-recording.png b/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure1-ioriot-io-recording.png
new file mode 100644
index 00000000..43ac852f
--- /dev/null
+++ b/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure1-ioriot-io-recording.png
diff --git a/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure2-ioriot-test-preparation.png b/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure2-ioriot-test-preparation.png
new file mode 100644
index 00000000..709d7490
--- /dev/null
+++ b/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure2-ioriot-test-preparation.png
diff --git a/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure3-ioriot-replay.png b/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure3-ioriot-replay.png
new file mode 100644
index 00000000..3bd66b6f
--- /dev/null
+++ b/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure3-ioriot-replay.png
diff --git a/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure4-ioriot-read-latency.png b/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure4-ioriot-read-latency.png
new file mode 100644
index 00000000..160b2305
--- /dev/null
+++ b/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure4-ioriot-read-latency.png
diff --git a/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure5-ioriot-write-latency.png b/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure5-ioriot-write-latency.png
new file mode 100644
index 00000000..e30efdbb
--- /dev/null
+++ b/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure5-ioriot-write-latency.png
diff --git a/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure6-iostat.png b/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure6-iostat.png
new file mode 100644
index 00000000..0d3fc0d8
--- /dev/null
+++ b/content/html/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure6-iostat.png
diff --git a/content/html/gemfeed/DRAFT-buetow.org.sh-One-Bash-script-to-rule-it-all.gmi.draft b/content/html/gemfeed/DRAFT-buetow.org.sh-One-Bash-script-to-rule-it-all.gmi.draft
new file mode 100644
index 00000000..be65b8b9
--- /dev/null
+++ b/content/html/gemfeed/DRAFT-buetow.org.sh-One-Bash-script-to-rule-it-all.gmi.draft
@@ -0,0 +1,183 @@
+ADD WRITTEN BY AND CREATED AT BLABLA
+
+ADD GO BACK LINK
+
+# buetow.org.sh - One Bash script to rule it all
+
+You might have read my previous blog post about entering the Geminispace.
+
+=> ./2021-04-24-welcome-to-the-geminispace Welcome to the Geminispace
+
+## Motivation
+
+Another benefit of using Gemini is that the Gemtext markup language is very easy to parse. As my site is dual hosted (Gemini+HTTP) I could in theory just write a shell script to deal with the conversion from Gemtext to HTML and not to rely on any external tools here.
+
+So I did exactly that, I wrote a Bash script which does the following:
+
+- Converts all Gemtext (*.gmi) files to HTML files
+- Generates a Gemtext atom.xml feed for my blog posts
+- Generates a HTML atom.xml feed of my blog posts
+
+I could have done all of that with a more powerful language than Bash (such as Perl, Ruby, Go...), but I didn't. The purpose of this exercise was to challenge what I can do with a "simple" Bash script and also to learn new things.
+
+```
+                                                               o .,<>., o
+                                                               |\/\/\/\/|
+                                                               '========'
+                                                               (_ SSSSSSs
+                                                               )a'`SSSSSs
+                                                              /_   SSSSSS
+                                                              .=## SSSSS
+                                                              .####  SSSSs
+                                                              ###::::SSSSS
+                                                             .;:::""""SSS
+                                                            .:;:'  . .  \\
+                                                           .::/  '     .'|
+                                                          .::( .         |
+                                                          :::)           \
+                                                          /\(            /
+                                                         /)            ( |
+                                                       .'  \  .       ./ /
+                                                    _-'    |\  .        |
+                                  _..--..   .  /"---\      | ` |      . |
+          -=====================,' _     \=(*#(7.#####()   |  `/_..   , (
+                      _.-''``';'-''-) ,.  \ '  '+/// |   .'/   \  ``-.) \
+                    ,'  _.-  ((    `-'  `._\    `` \_/_.'  )    /`-._  ) |
+                  ,'\ ,'  _.'.`:-.    \.-'                 /   <_L   )"  |
+                _/   `._,' ,')`;  `-'`'                    |     L  /    /
+               / `.   ,' ,|_/ / \                          (    <_-'     \
+               \ / `./  '  / /,' \                        /|`         `. |
+               )\   /`._   ,'`._.-\                       |)            \'
+              /  `.'    )-'.-,' )__)                      |\            `|
+             : /`. `.._(--.`':`':/ \                      ) \             \
+             |::::\     ,'/::;-))  /                      ( )`.            |
+             ||:::::  . .::':  :`-(                       |/    .          |
+             ||::::|  . :|  |==[]=:                       .        -       \
+             |||:::|  : ||  :  |  |                      /\           `     |
+ ___ ___     '|;:::|  | |'   \=[]=|                     /  \                \
+|   /_  ||``|||:::::  | ;    | |  |                     \_.'\_               `-.
+:   \_``[]--[]|::::'\_;'     )-'..`._                 .-'\``:: ` .              \
+ \___.>`''-.||:.__,'     SSt |_______`>              <_____:::.         . . \  _/
+                                                           `+a:f:......jrei'''
+```
+
+## W3C validator says all good
+#
+All generated HTML and Atom files pass the W3C validation. It is crazy that generating the Atom feed with valid XHTML content body for each blog posts was the most difficult part to implement in Bash. These formats are the reason why I decided to use Gemini as the primary protocol in the first place. However, Ironically I spent a couple of hours to get the XHTML and web Atoom feed working. To be fair, the Atom feed also works with Gemini.
+
+## Meta files for atom feed generation
+
+## Not without sed and grep and cut
+
+Soon I realised that I didn't want to go without a bit of grep and sed and cut. Regular expression matchings and simple string substitution tasks can be done in pure Bash but in my own opinion grep+sed are more powerful and easier to use (as I am used to these anyway). I managed not to use any AWK though.
+
+### Grepping
+
+I could use Bash's built-in regular expression matching engine here, but I am used to the grep pattern syntax, that's why I decided to do it this way:
+```
+if grep -E -q "$IMAGE_PATTERN" <<< "$link"; then
+    html::img "$link" "$descr"
+    return
+fi
+```
+
+### Sed-ing
+
+Sed comes in very handy for things like fixing HTML block text by replacing the lower than "<" and larger than ">" symbols with their corresponding HTML codes with one single command :
+
+```
+TODO: UPDATE SNIPPET echo "$line" | sed 's|<|\&lt;|g; s|>|\&gt;|g'
+```
+
+Sed is also useful in the following example, where the script checks whether the newly generated Atom feed file has changed compared to the previous version or not:
+
+```
+if ! diff -u <(sed 3d "$atom_file.tmp") <(sed 3d "$atom_file"); then
+    ... 
+else
+    ...
+fi
+```
+
+### Cut-ing
+
+## Bash Modules for better structure
+
+I separated the script into different section; you could call them modules. For example, all functions dealing with the Atom feed are prefixed with atomfeed::, all functions dealing with HTML are prefixed with html:: and so on.
+
+As of writing this the script has the following modules and module functions:
+
+```
+TODO: UPDATE SNIPPET
+❯ grep '::.* ()' buetow.org.sh
+assert::equals () {
+atom::meta () {
+atom::generate () {
+html::paragraph () {
+html::heading () {
+html::quote () {
+html::img () {
+html::link () {
+html::gemini2html () {
+html::generate () {
+html::test () {
+main::help () {
+```
+
+## Declaring all variables
+
+Many Bash scripts out in the wild don't have their variables declared, which leads to bad surprises as the default behaviour is that an undeclared variable is automatically a global variable once in use. So the best practise is to always declare a variable with one of the keywords "delcare", "readonly" or "local".
+
+Whole numbers can also have the option "-i", e.g. "declare -i num=52" and read only variables can be either declared via "readonly" or "rdeclare -r" or "local -r". Function local variables can also be declared with the "local" keyword. 
+
+This is an example from the Atom module, where all variables are local to the function. I also make use of the "assign-then-shift"-pattern which goes like this: "local -r var1=$1; shift; local -r var2=$1; shift". The idea is that you only use "$1" to assign function arguments to named (better readable) local function variables. You will never have to bother about "$2" or above. That's is very useful when you constantly refactor your code and remove or add function arguments. It's something what I picked up from a colleague (a purely Bash wizard) some time ago:
+
+```
+atomfeed::meta () {
+    local -r now="$1"; shift
+    local -r gmi_file_path="$1"; shift
+    ...
+}
+```
+
+## Unit tests
+
+Especially the Gemtext to HTML conversion part is an excellent use case for unit testing. There are unit tests for various Gemtext to HTML conversions (e.g. A header, paragraph, link, quote ...). My small unit test framework only consists of the test::assert() function.
+
+Forces to think creatively and to keep features fairly simple (good things)
+
+## De-facto templates
+
+## It's a static website generator
+
+Generate statically on my laptop and commit all statically generated files to fit. Can also preview locally.
+
+A lot of bash tricks
+
+## Config file
+
+## Learnings from ShellCheck
+
+ShellSheck: Not happy with all recommentations but most, e.g. read -r, quotes, etc.
+
+### While-read loops
+
+Specify -r
+
+### Warnings about variables not quoted
+
+### if cmd; then
+
+## The result(s)
+
+### Gemtext via Gemini protocol
+
+=> gemini://buetow.org gemini://buetow.org - The original Gemini capsule  
+=> gemini://buetow.org/gemfeed/ gemini://buetow.org/gemfeed/ - The Gemfeed
+=> gemini://buetow.org/gemfeed/atom.xml gemini://buetow.org/gemfeed/atom.xml - The Atom feed
+
+### XHTML via HTTP protocol
+
+=> https://buetow.org https://buetow.org - The original Gemini capsule  
+=> https://buetow.org/gemfeed/ https://buetow.org/gemfeed/ - The Gemfeed
+=> https://buetow.org/gemfeed/atom.xml https://buetow.org/gemfeed/atom.xml - The Atom feed
diff --git a/content/html/gemfeed/atom.xml b/content/html/gemfeed/atom.xml
index 6cdf5970..7a48fb34 100644
--- a/content/html/gemfeed/atom.xml
+++ b/content/html/gemfeed/atom.xml
@@ -1,24 +1,27 @@
 <?xml version="1.0" encoding="utf-8"?>
 <feed xmlns="http://www.w3.org/2005/Atom">
-    <updated>2021-05-07T22:55:55+01:00</updated>
+    <updated>2021-05-08T10:29:53+01:00</updated>
     <title>buetow.org feed</title>
     <subtitle>Having fun with computers!</subtitle>
     <link href="https://buetow.org/gemfeed/atom.xml" rel="self" />
     <link href="https://buetow.org/" />
     <id>https://buetow.org/</id>
     <entry>
-        <title>Welcome to the Geminispace</title>
+        <title></title>
         <link href="https://buetow.org/gemfeed/2021-04-24-welcome-to-the-geminispace.html" />
         <id>https://buetow.org/gemfeed/2021-04-24-welcome-to-the-geminispace.html</id>
-        <updated>2021-04-24T19:28:41+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>Have you reached this article already via Gemini? You need a special client for that, web browsers such as Firefox, Chrome, Safari etc. don't support the Gemini protocol. The Gemini address of this site (or the address of this capsule as people say in Geminispace) is: ... to read on visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>Welcome to the Geminispace</h1>
+                <pre>Written by Paul Buetow 2021-04-24, last updated 2021-04-30</pre>
+<pre>ASCII Art by Andy Hood</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Welcome to the Geminispace</h1>
 <p>Have you reached this article already via Gemini? You need a special client for that, web browsers such as Firefox, Chrome, Safari etc. don't support the Gemini protocol. The Gemini address of this site (or the address of this capsule as people say in Geminispace) is:</p>
 <a class="textlink" href="https://buetow.org">https://buetow.org</a><br />
 <p>If you however still use HTTP then you are just surfing the fallback HTML version of this capsule. In that case I suggest reading on what this is all about :-).</p>
@@ -73,18 +76,20 @@
         </content>
     </entry>
     <entry>
-        <title>DTail - The distributed log tail program</title>
+        <title></title>
         <link href="https://buetow.org/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html" />
         <id>https://buetow.org/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html</id>
-        <updated>2021-04-22T19:28:41+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>This article first appeared at the Mimecast Engineering Blog but I made it available here in my personal Gemini capsule too. ...to read on visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>DTail - The distributed log tail program</h1>
+                <pre>Written by Paul Buetow 2021-04-22, last updated 2021-04-26</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>DTail - The distributed log tail program</h1>
 <i>DTail logo image:</i><a href="https://buetow.org/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program/dtail-logo.png"><img alt="DTail logo image" title="DTail logo image" src="https://buetow.org/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program/dtail-logo.png" /></a><br />
 <p>This article first appeared at the Mimecast Engineering Blog but I made it available here in my personal Gemini capsule too.</p>
 <a class="textlink" href="https://medium.com/mimecast-engineering/dtail-the-distributed-log-tail-program-79b8087904bb">Original Mimecast Engineering Blog post at Medium</a><br />
@@ -153,18 +158,160 @@ dtail –servers serverlist.txt –files ‘/var/log/*.log’ –regex ‘(?i:er
         </content>
     </entry>
     <entry>
-        <title>Methods in C</title>
+        <title></title>
+        <link href="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.html" />
+        <id>https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.html</id>
+        <updated></updated>
+        <author>
+            <name></name>
+            <email></email>
+        </author>
+        <summary></summary>
+        <content type="xhtml">
+            <div xmlns="http://www.w3.org/1999/xhtml">
+                <p> Written by Paul Buetow 2018-06-01, last updated 2021-05-08</p>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Realistic load testing with I/O Riot for Linux</h1>
+<pre>
+       .---.
+      /     \
+      \.@-@./
+      /`\_/`\
+     //  _  \\
+    | \     )|_
+   /`\_`&gt;  &lt;_/ \
+jgs\__/'---'\__/
+</pre>
+<h2>Foreword</h2>
+<p>This text first was published in the german ADMIN computer Magazine. 3 years have passed since and I decided to publish it on my blog too. </p>
+<a class="textlink" href="https://www.admin-magazin.de/Das-Heft/2018/06/Realistische-Lasttests-mit-I-O-Riot">https://www.admin-magazin.de/Das-Heft/2018/06/Realistische-Lasttests-mit-I-O-Riot</a><br />
+<p>I havn't worked on I/O Riot for some time now, but all what is written here is still valid. I am still using I/O Riot to debug I/O issues and pattern once in a while, so by all means the tool is not obsolete yet. The tool even helped to resolve a major production incident at work involving I/O.</p>
+<p>I am eagerly looking forward to revamp I/O Riot so that it uses the new BPF Linux capabilities instead of Systemtap. Also, when I wrote I/O Riot initially, I didn't have any experience with the Go programming language yet and therefore I wrote it in C. Once it gets revamped I might consider using Go instead of C as it would spare me from many segmentation faults and headaches during development ;-). I might also just stick to C for plain performance reasons and just refactor the code dealing with concurrency.</p>
+<p>Pleace notice that some of the screenshots show the command "ioreplay" instead of "ioriot". That's because the name has changed after taking those.</p>
+<h1>The article</h1>
+<p>With I/O Riot IT administrators can load test and optimize the I/O subsystem of Linux-based operating systems. The tool makes it possible to record I/O patterns and replay them at a later time as often as desired. This means bottlenecks can be reproduced and eradicated. </p>
+<p>When storing huge amounts of data, such as more than 200 billion archived emails at Mimecast, it's not only the available storage capacity that matters, but also the data throughput and latency. At the same time, operating costs must be kept as low as possible. The more systems involved, the more important it is to optimize the hardware, the operating system and the applications running on it.</p>
+<h2>Background: Existing Techniques</h2>
+<p>Conventional I/O benchmarking: Administrators usually use open source benchmarking tools like IOZone and bonnie++. Available database systems such as Redis and MySQL come with their own benchmarking tools. The common problem with these tools is that they work with prescribed artificial I/O patterns. Although this can test both sequential and randomized data access, the patterns do not correspond to what can be found on production systems.</p>
+<p>Testing by load test environment: Another option is to use a separate load test environment in which, as far as possible, a production environment with all its dependencies is simulated. However, an environment consisting of many microservices is very complex. Microservices are usually managed by different teams, which means extra coordination effort for each load test. Another challenge is to generate the load as authentically as possible so that the patterns correspond to a productive environment. Such a load test environment can only handle as many requests as its weakest link can handle. For example, load generators send many read and write requests to a frontend microservice, whereby the frontend forwards the requests to a backend microservice responsible for storing the data. If the frontend service does not process the requests efficiently enough, the backend service is not well utilized in the first place. As a rule, all microservices are clustered across many servers, which makes everything even more complicated. Under all these conditions it is very difficult to test I/O of separate backend systems. Moreover, for many small and medium-sized companies, a separate load test environment would not be feasible for cost reasons.</p>
+<p>Testing in the production environment: For these reasons, benchmarks are often carried out in the production environment. In order to derive value from this such tests are especially performed during peak hours when systems are under high load. However, testing on production systems is associated with risks and can lead to failure or loss of data without adequate protection.</p>
+<h2>Benchmarking the Email Cloud at Mimecast</h2>
+<p>For email archiving, Mimecast uses an internally developed microservice, which is operated directly on Linux-based storage systems. A storage cluster is divided into several replication volumes. Data is always replicated three times across two secure data centers. Customer data is automatically allocated to one or more volumes, depending on throughput, so that all volumes are automatically assigned the same load. Customer data is archived on conventional, but inexpensive hard disks with several terabytes of storage capacity each. I/O benchmarking proved difficult for all the reasons mentioned above. Furthermore, there are no ready-made tools for this purpose in the case of self-developed software. The service operates on many block devices simultaneously, which can make the RAID controller a bottleneck. None of the freely available benchmarking tools can test several block devices at the same time without extra effort. In addition, emails typically consist of many small files. Randomized access to many small files is particularly inefficient. In addition to many software adaptations, the hardware and operating system must also be optimized.</p>
+<p>Mimecast encourages employees to be innovative and pursue their own ideas in the form of an internal competition, Pet Project. The goal of the pet project I/O Riot was to simplify OS and hardware level I/O benchmarking. The first prototype of I/O Riot was awarded an internal roadmap prize in the spring of 2017. A few months later, I/O Riot was used to reduce write latency in the storage clusters by about 50%. The improvement was first verified by I/O replay on a test system and then successively applied to all storage systems. I/O Riot was also used to resolve a production incident related to disk I/O load.</p>
+<h2>Using I/O Riot</h2>
+<p>First, all I/O events are logged to a file on a production system with I/O Riot. It is then copied to a test system where all events are replayed in the same way. The crucial point here is that you can reproduce I/O patterns as they are found on a production system as often as you like on a test system. This results in the possibility of optimizing the set screws on the system after each run.</p>
+<h3>Installation</h3>
+<p>I/O Riot was tested under CentOS 7.2 x86_64. For compiling, the GNU C compiler and Systemtap including kernel debug information are required. Other Linux distributions are theoretically compatible but untested. First of all, you should update the systems involved as follows:</p>
+<pre>
+% sudo yum update
+</pre>
+<p>If the kernel is updated, please restart the system. The installation would be done without a restart but this would complicate the installation. The installed kernel version should always correspond to the currently running kernel. You can then install I/O Riot as follows:</p>
+<pre>
+% sudo yum install gcc git systemtap yum-utils kernel-devel-$(uname -r)
+% sudo debuginfo-install kernel-$(uname -r)
+% git clone https://github.com/mimecast/ioriot
+% cd ioriot
+% make
+% sudo make install
+% export PATH=$PATH:/opt/ioriot/bin
+</pre>
+<p>Note: It is not best practice to install any compilers on production systems. For further information please have a look at the enclosed README.md.</p>
+<h3>Recording of I/O events</h3>
+<p>All I/O events are kernel related. If a process wants to perform an I/O operation, such as opening a file, it must inform the kernel of this by a system call (short syscall). I/O Riot relies on the Systemtap tool to record I/O syscalls. Systemtap, available for all popular Linux distributions, helps you to take a look at the running kernel in productive environments, which makes it predestined to monitor all I/O-relevant Linux syscalls and log them to a file. Other tools, such as strace, are not an alternative because they slow down the system too much.</p>
+<p>During recording, ioriot acts as a wrapper and executes all relevant Systemtap commands for you. Use the following command to log all events to io.capture:</p>
+<pre>
+% sudo ioriot -c io.capture
+</pre>
+<i>Screenshot I/O recording:</i><a href="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure1-ioriot-io-recording.png"><img alt="Screenshot I/O recording" title="Screenshot I/O recording" src="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure1-ioriot-io-recording.png" /></a><br />
+<p>A Ctrl-C (SIGINT) stops recording prematurely. Otherwise, ioriot terminates itself automatically after ten minutes. Depending on the system load, the output file can grow to several gigabytes. Only metadata is logged, not the read and written data itself. When replaying later, only random data is used. Under certain circumstances, Systemtap may omit some system calls and issue warnings. This is to ensure that Systemtap does not consume too many resources.</p>
+<h3>Test preparation</h3>
+<p>Then copy io.capture to a test system. The log also contains all accesses to the pseudo file systems devfs, sysfs and procfs. This makes little sense, which is why you must first generate a cleaned and playable version io.replay from io.capture as follows:</p>
+<pre>
+% sudo ioriot -c io.capture -r io.replay -u $USER -n TESTNAME
+</pre>
+<p>The parameter -n allows you to assign a freely selectable test name. An arbitrary system user under which the test is to be played is specified via paramater -u.</p>
+<h3>Test Initialization</h3>
+<p>The test will most likely want to access existing files. These are files the test wants to read but does not create by itself. The existence of these must be ensured before the test. You can do this as follows:</p>
+<pre>
+% sudo ioriot -i io.replay
+</pre>
+<p>To avoid any damage to the running system, ioreplay only works in special directories. The tool creates a separate subdirectory for each file system mount point (e.g. /, /usr/local, /store/00,...) (here: /.ioriot/TESTNAME, /usr/local/.ioriot/TESTNAME, /store/00/.ioriot/TESTNAME,...). By default, the working directory of ioriot is /usr/local/ioriot/TESTNAME.</p>
+<i>Screenshot test preparation:</i><a href="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure2-ioriot-test-preparation.png"><img alt="Screenshot test preparation" title="Screenshot test preparation" src="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure2-ioriot-test-preparation.png" /></a><br />
+<p>You must re-initialize the environment before each run. Data from previous tests will be moved to a trash directory automatically, which can be finally deleted with "sudo ioriot -P".</p>
+<h3>Replay</h3>
+<p>After initialization, you can replay the log with -r. You can use -R to initiate both test initialization and replay in a single command and -S can be used to specify a file in which statistics are written after the test run.</p>
+<p>You can also influence the playback speed: "-s 0" is interpreted as "Playback as fast as possible" and is the default setting. With "-s 1" all operations are performed at original speed. "-s 2" would double the playback speed and "-s 0.5" would halve it.</p>
+<i>Screenshot replaying I/O:</i><a href="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure3-ioriot-replay.png"><img alt="Screenshot replaying I/O" title="Screenshot replaying I/O" src="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure3-ioriot-replay.png" /></a><br />
+<p>As an initial test, for example, you could compare the two Linux I/O schedulers CFQ and Deadline and check which scheduler the test runs the fastest. They run the test separately for each scheduler. The following shell loop iterates through all attached block devices of the system and changes their I/O scheduler to the one specified in variable $new_scheduler (in this case either cfq or deadline). Subsequently, all I/O events from the io.replay protocol are played back. At the end, an output file with statistics is generated:</p>
+<pre>
+% new_scheduler=cfq
+% for scheduler in /sys/block/*/queue/scheduler; do
+    echo $new_scheduler | sudo tee $scheduler
+done
+% sudo ioriot -R io.replay -S cfq.txt
+% new_scheduler=deadline
+% for scheduler in /sys/block/*/queue/scheduler; do
+   echo $new_scheduler | sudo tee $scheduler
+done
+% sudo ioriot -R io.replay -S deadline.txt
+</pre>
+<p>According to the results, the test could run 940 seconds faster with Deadline Scheduler:</p>
+<pre>
+% cat cfq.txt
+Num workers: 4
+hreads per worker: 128
+otal threads: 512
+Highest loadavg: 259.29
+Performed ioops: 218624596
+Average ioops/s: 101544.17
+Time ahead: 1452s
+Total time: 2153.00s
+% cat deadline.txt
+Num workers: 4
+Threads per worker: 128
+Total threads: 512
+Highest loadavg: 342.45
+Performed ioops: 218624596
+Average ioops/s: 180234.62
+Time ahead: 2392s
+Total time: 1213.00s
+</pre>
+<p>In any case, you should also set up a time series database, such as Graphite, where the I/O throughput can be plotted. Figures 4 and 5 show the read and write access times of both tests. The break-in makes it clear when the CFQ test ended and the deadline test was started. The reading latency of both tests is similar. Write latency is dramatically improved using the Deadline Scheduler.</p>
+<i>Graphite visualization of the mean read access times in ms with CFQ and Deadline Scheduler.:</i><a href="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure4-ioriot-read-latency.png"><img alt="Graphite visualization of the mean read access times in ms with CFQ and Deadline Scheduler." title="Graphite visualization of the mean read access times in ms with CFQ and Deadline Scheduler." src="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure4-ioriot-read-latency.png" /></a><br />
+<i>Graphite visualization of the average write access times in ms with CFQ and Deadline Scheduler.:</i><a href="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure5-ioriot-write-latency.png"><img alt="Graphite visualization of the average write access times in ms with CFQ and Deadline Scheduler." title="Graphite visualization of the average write access times in ms with CFQ and Deadline Scheduler." src="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure5-ioriot-write-latency.png" /></a><br />
+<p>You should also take a look at the iostat tool. The iostat screenshot shows the output of iostat -x 10 during a test run. As you can see, a block device is fully loaded with 99% utilization, while all other block devices still have sufficient buffer. This could be an indication of poor data distribution in the storage system and is worth pursuing. It is not uncommon for I/O Riot to reveal software problems.</p>
+<i>Output of iostat. The block device sdy seems to be almost fully utilized by 99%.:</i><a href="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure6-iostat.png"><img alt="Output of iostat. The block device sdy seems to be almost fully utilized by 99%." title="Output of iostat. The block device sdy seems to be almost fully utilized by 99%." src="https://buetow.org/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux/figure6-iostat.png" /></a><br />
+<h2>I/O Riot is Open Source</h2>
+<p>The tool has already proven to be very useful and will continue to be actively developed as time and priority permits. Mimecast intends to be an ongoing contributor to Open Source. You can find I/O Riot at:</p>
+<a class="textlink" href="https://github.com/mimecast/ioriot">https://github.com/mimecast/ioriot</a><br />
+<h2>Systemtap</h2>
+<p>Systemtap is a tool for the instrumentation of the Linux kernel. The tool provides an AWK-like programming language. Programs written in it are compiled from Systemtap to C- and then into a dynamically loadable kernel module. Loaded into the kernel, the program has access to Linux internals. A Systemtap program written for I/O Riot monitors when, with which parameters, at which time, and from which process I/O syscalls take place and their return values.</p>
+<p>For example, the open syscall opens a file and returns the responsible file descriptor. The read and write syscalls can operate on a file descriptor and return the number of read or written bytes. The close syscall closes a given file descriptor. I/O Riot comes with a ready-made Systemtap program, which you have already compiled into a kernel module and installed to /opt/ioriot. In addition to open, read and close, it logs many other I/O-relevant calls.</p>
+<a class="textlink" href="https://sourceware.org/systemtap/">https://sourceware.org/systemtap/</a><br />
+<h2>More refereces</h2>
+<a class="textlink" href="http://www.iozone.org/">IOZone</a><br />
+<a class="textlink" href="https://www.coker.com.au/bonnie++/">Bonnie++</a><br />
+<a class="textlink" href="https://graphiteapp.org">Graphite</a><br />
+<a class="textlink" href="https://en.wikipedia.org/wiki/Memory-mapped_I/O">Memory mapped I/O</a><br />
+<p>E-Mail me your thoughts at comments@mx.buetow.org!</p>
+            </div>
+        </content>
+    </entry>
+    <entry>
+        <title></title>
         <link href="https://buetow.org/gemfeed/2016-11-20-methods-in-c.html" />
         <id>https://buetow.org/gemfeed/2016-11-20-methods-in-c.html</id>
-        <updated>2016-11-20T18:36:51+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>You can do some sort of object oriented programming in the C Programming Language. However, that is very limited. But also very easy and straight forward to use.. .....to read on please visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>Methods in C</h1>
+                <pre>Written by Paul Buetow 2016-11-20</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Methods in C</h1>
 <p>You can do some sort of object oriented programming in the C Programming Language. However, that is very limited. But also very easy and straight forward to use.</p>
 <h2>Example</h2>
 <p>Lets have a look at the following sample program. Basically all you have to do is to add a function pointer such as "calculate" to the definition of struct "something_s". Later, during the struct initialization, assign a function address to that function pointer:</p>
@@ -232,18 +379,20 @@ mult.calculate(mult,a,b));
         </content>
     </entry>
     <entry>
-        <title>Spinning up my own authoritative DNS servers</title>
+        <title></title>
         <link href="https://buetow.org/gemfeed/2016-05-22-spinning-up-my-own-authoritative-dns-servers.html" />
         <id>https://buetow.org/gemfeed/2016-05-22-spinning-up-my-own-authoritative-dns-servers.html</id>
-        <updated>2016-05-22T18:59:01+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>Finally, I had time to deploy my own authoritative DNS servers (master and slave) for my domains 'buetow.org' and 'buetow.zone'. My domain name provider is Schlund Technologies. They allow their customers to manually edit the DNS records (BIND files). And they also give you the opportunity to set your own authoritative DNS servers for your domains. From now I am making use of that option.. .....to read on please visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>Spinning up my own authoritative DNS servers</h1>
+                <pre>Written by Paul Buetow 2016-05-22</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Spinning up my own authoritative DNS servers</h1>
 <h2>Background</h2>
 <p>Finally, I had time to deploy my own authoritative DNS servers (master and slave) for my domains "buetow.org" and "buetow.zone". My domain name provider is Schlund Technologies. They allow their customers to manually edit the DNS records (BIND files). And they also give you the opportunity to set your own authoritative DNS servers for your domains. From now, I am making use of that option.</p>
 <a class="textlink" href="http://www.schlundtech.de">Schlund Technologies</a><br />
@@ -456,18 +605,20 @@ apply Service "dig6" {
         </content>
     </entry>
     <entry>
-        <title>Offsite backup with ZFS (Part 2)</title>
+        <title></title>
         <link href="https://buetow.org/gemfeed/2016-04-16-offsite-backup-with-zfs-part2.html" />
         <id>https://buetow.org/gemfeed/2016-04-16-offsite-backup-with-zfs-part2.html</id>
-        <updated>2016-04-16T22:43:42+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>I enhanced the procedure a bit. From now on I am having two external 2TB USB hard drives. Both are setup exactly the same way. To decrease the probability that they will not fail at about the same time both drives are of different brands. One drive is kept at the secret location. The other one is kept at home right next to my HP MicroServer. ...to read on visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>Offsite backup with ZFS (Part 2)</h1>
+                <pre>Written by Paul Buetow 2016-04-16</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Offsite backup with ZFS (Part 2)</h1>
 <pre>
  ________________
 |# :           : #|
@@ -491,18 +642,20 @@ apply Service "dig6" {
         </content>
     </entry>
     <entry>
-        <title>Offsite backup with ZFS</title>
+        <title></title>
         <link href="https://buetow.org/gemfeed/2016-04-03-offsite-backup-with-zfs.html" />
         <id>https://buetow.org/gemfeed/2016-04-03-offsite-backup-with-zfs.html</id>
-        <updated>2016-04-03T22:43:42+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>When it comes to data storage and potential data loss I am a paranoid person. It is not just due to my job  but also due to a personal experience I encountered over 10 years ago: A single drive failure and loss of all my data (pictures, music, ....). ...to read on visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>Offsite backup with ZFS</h1>
+                <pre>Written by Paul Buetow 2016-04-03</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Offsite backup with ZFS</h1>
 <pre>
  ________________
 |# :           : #|
@@ -531,18 +684,20 @@ apply Service "dig6" {
         </content>
     </entry>
     <entry>
-        <title>Perl Daemon (Service Framework)</title>
+        <title></title>
         <link href="https://buetow.org/gemfeed/2011-05-07-perl-daemon-service-framework.html" />
         <id>https://buetow.org/gemfeed/2011-05-07-perl-daemon-service-framework.html</id>
-        <updated>2011-05-07T22:26:02+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>PerlDaemon is a minimal daemon for Linux and other Unix like operating systems programmed in Perl. It is a minimal but pretty functional and fairly generic service framework. This means that it does not do anything useful other than providing a framework for starting, stopping, configuring and logging. In order to do something a module (written in Perl) bust be provided.. .....to read on please visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>Perl Daemon (Service Framework)</h1>
+                <pre>Written by Paul Buetow 2011-05-07, last updated 2021-05-07</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Perl Daemon (Service Framework)</h1>
 <pre>
    a'!   _,,_ a'!   _,,_     a'!   _,,_
      \\_/    \  \\_/    \      \\_/    \.-,
@@ -676,18 +831,29 @@ sub do ($) {
         </content>
     </entry>
     <entry>
-        <title>The Fype Programming Language</title>
+        <title></title>
         <link href="https://buetow.org/gemfeed/2010-05-09-the-fype-programming-language.html" />
         <id>https://buetow.org/gemfeed/2010-05-09-the-fype-programming-language.html</id>
-        <updated>2010-05-09T12:48:29+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>Fype is an interpreted programming language created by me for learning and fun. The interpreter is written in C. It has been tested on FreeBSD and NetBSD and may also work on other Unix like operating systems such as Linux based ones. To be honest, besides learning and fun there is really no other use case of why Fype actually exists as many other programming languages are much faster and more powerful.. .....to read on please visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>The Fype Programming Language</h1>
+                <pre>Written by Paul Buetow 2010-05-09, last updated 2021-05-05</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<pre>
+      ____                                      _        __       
+     / / _|_   _ _ __   ___    _   _  ___  __ _| |__    / _|_   _ 
+    / / |_| | | | '_ \ / _ \  | | | |/ _ \/ _` | '_ \  | |_| | | |
+ _ / /|  _| |_| | |_) |  __/  | |_| |  __/ (_| | | | |_|  _| |_| |
+(_)_/ |_|  \__, | .__/ \___|   \__, |\___|\__,_|_| |_(_)_|  \__, |
+           |___/|_|            |___/                        |___/ 
+
+</pre>
+<h1>The Fype Programming Language</h1>
 <p>Fype is an interpreted programming language created by me for learning and fun. The interpreter is written in C. It has been tested on FreeBSD and NetBSD and may also work on other Unix like operating systems such as Linux based ones. To be honest, besides learning and fun there is really no other use case of why Fype actually exists as many other programming languages are much faster and more powerful.</p>
 <p>The Fype syntax is very simple and is using a maximum look ahead of 1 and a very easy top down parsing mechanism. Fype is parsing and interpreting its code simultaneously. This means, that syntax errors are only detected during program runtime. </p>
 <p>Fype is a recursive acronym and means "Fype is For Your Program Execution" or "Fype is Free Yak Programmed for ELF". You could also say "It's not a hype - it's Fype!".</p>
@@ -1082,18 +1248,20 @@ BB
         </content>
     </entry>
     <entry>
-        <title>Perl Poetry</title>
+        <title></title>
         <link href="https://buetow.org/gemfeed/2008-06-26-perl-poetry.html" />
         <id>https://buetow.org/gemfeed/2008-06-26-perl-poetry.html</id>
-        <updated>2008-06-26T21:43:51+01:00</updated>
+        <updated></updated>
         <author>
-            <name>Paul Buetow</name>
-            <email>comments@mx.buetow.org</email>
+            <name></name>
+            <email></email>
         </author>
-        <summary>Here are some Perl Poems I wrote. They don't do anything useful when you run them but they don't produce a compiler error either. They only exists for fun and demonstrate what you can do with Perl syntax.. .....to read on please visit my site.</summary>
+        <summary></summary>
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
-                <h1>Perl Poetry</h1>
+                <pre>Written by Paul Buetow 2008-06-26, last updated 2021-05-04</pre>
+<a class="textlink" href="../">Go back to the main site</a><br />
+<h1>Perl Poetry</h1>
 <pre>
  '\|/'                                  *
 -- * -----
diff --git a/content/html/gemfeed/index.html b/content/html/gemfeed/index.html
index dd3b185d..34c6e094 100644
--- a/content/html/gemfeed/index.html
+++ b/content/html/gemfeed/index.html
@@ -20,14 +20,15 @@ li { color: #98be65; }
 <body>
 <h1>buetow.org's Gemfeed</h1>
 <h2>Having fun with computers!</h2>
-<a class="textlink" href="./2021-04-24-welcome-to-the-geminispace.html">2021-04-24 Welcome to the Geminispace</a><br />
-<a class="textlink" href="./2021-04-22-dtail-the-distributed-log-tail-program.html">2021-04-22 DTail - The distributed log tail program</a><br />
-<a class="textlink" href="./2016-11-20-methods-in-c.html">2016-11-20 Methods in C</a><br />
-<a class="textlink" href="./2016-05-22-spinning-up-my-own-authoritative-dns-servers.html">2016-05-22 Spinning up my own authoritative DNS servers</a><br />
-<a class="textlink" href="./2016-04-16-offsite-backup-with-zfs-part2.html">2016-04-16 Offsite backup with ZFS (Part 2)</a><br />
-<a class="textlink" href="./2016-04-03-offsite-backup-with-zfs.html">2016-04-03 Offsite backup with ZFS</a><br />
-<a class="textlink" href="./2011-05-07-perl-daemon-service-framework.html">2011-05-07 Perl Daemon (Service Framework)</a><br />
-<a class="textlink" href="./2010-05-09-the-fype-programming-language.html">2010-05-09 The Fype Programming Language</a><br />
-<a class="textlink" href="./2008-06-26-perl-poetry.html">2008-06-26 Perl Poetry</a><br />
+<a class="textlink" href="./2021-04-24-welcome-to-the-geminispace.html">2021-04-24 - Welcome to the Geminispace</a><br />
+<a class="textlink" href="./2021-04-22-dtail-the-distributed-log-tail-program.html">2021-04-22 - DTail - The distributed log tail program</a><br />
+<a class="textlink" href="./2018-06-01-realistic-load-testing-with-ioriot-for-linux.html">2018-06-01 - Realistic load testing with I/O Riot for Linux</a><br />
+<a class="textlink" href="./2016-11-20-methods-in-c.html">2016-11-20 - Methods in C</a><br />
+<a class="textlink" href="./2016-05-22-spinning-up-my-own-authoritative-dns-servers.html">2016-05-22 - Spinning up my own authoritative DNS servers</a><br />
+<a class="textlink" href="./2016-04-16-offsite-backup-with-zfs-part2.html">2016-04-16 - Offsite backup with ZFS (Part 2)</a><br />
+<a class="textlink" href="./2016-04-03-offsite-backup-with-zfs.html">2016-04-03 - Offsite backup with ZFS</a><br />
+<a class="textlink" href="./2011-05-07-perl-daemon-service-framework.html">2011-05-07 - Perl Daemon (Service Framework)</a><br />
+<a class="textlink" href="./2010-05-09-the-fype-programming-language.html">2010-05-09 - The Fype Programming Language</a><br />
+<a class="textlink" href="./2008-06-26-perl-poetry.html">2008-06-26 - Perl Poetry</a><br />
 </body>
 </html>
diff --git a/content/html/index.html b/content/html/index.html
index 73f8df8d..e4b4eb8f 100644
--- a/content/html/index.html
+++ b/content/html/index.html
@@ -56,14 +56,15 @@ li { color: #98be65; }
 <a class="textlink" href="./gemfeed/index.html">Subscribe to this blog's Gemfeed</a><br />
 <h3>Posts</h3>
 <p>I have switched blog software multiple times. I might be back filling some of the older articles here. So please don't wonder when suddenly very old posts appear here.</p>
-<a class="textlink" href="./gemfeed/2021-04-24-welcome-to-the-geminispace.html">2021-04-24 Welcome to the Geminispace</a><br />
-<a class="textlink" href="./gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html">2021-04-22 DTail - The distributed log tail program</a><br />
-<a class="textlink" href="./gemfeed/2016-11-20-methods-in-c.html">2016-11-20 Methods in C</a><br />
-<a class="textlink" href="./gemfeed/2016-05-22-spinning-up-my-own-authoritative-dns-servers.html">2016-05-22 Spinning up my own authoritative DNS servers</a><br />
-<a class="textlink" href="./gemfeed/2016-04-16-offsite-backup-with-zfs-part2.html">2016-04-16 Offsite backup with ZFS (Part 2)</a><br />
-<a class="textlink" href="./gemfeed/2016-04-03-offsite-backup-with-zfs.html">2016-04-03 Offsite backup with ZFS</a><br />
-<a class="textlink" href="./gemfeed/2011-05-07-perl-daemon-service-framework.html">2011-05-07 Perl Daemon (Service Framework)</a><br />
-<a class="textlink" href="./gemfeed/2010-05-09-the-fype-programming-language.html">2010-05-09 The Fype Programming Language</a><br />
-<a class="textlink" href="./gemfeed/2008-06-26-perl-poetry.html">2008-06-26 Perl Poetry</a><br />
+<a class="textlink" href="./gemfeed/2021-04-24-welcome-to-the-geminispace.html">2021-04-24 - Welcome to the Geminispace</a><br />
+<a class="textlink" href="./gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html">2021-04-22 - DTail - The distributed log tail program</a><br />
+<a class="textlink" href="./gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.html">2018-06-01 - Realistic load testing with I/O Riot for Linux</a><br />
+<a class="textlink" href="./gemfeed/2016-11-20-methods-in-c.html">2016-11-20 - Methods in C</a><br />
+<a class="textlink" href="./gemfeed/2016-05-22-spinning-up-my-own-authoritative-dns-servers.html">2016-05-22 - Spinning up my own authoritative DNS servers</a><br />
+<a class="textlink" href="./gemfeed/2016-04-16-offsite-backup-with-zfs-part2.html">2016-04-16 - Offsite backup with ZFS (Part 2)</a><br />
+<a class="textlink" href="./gemfeed/2016-04-03-offsite-backup-with-zfs.html">2016-04-03 - Offsite backup with ZFS</a><br />
+<a class="textlink" href="./gemfeed/2011-05-07-perl-daemon-service-framework.html">2011-05-07 - Perl Daemon (Service Framework)</a><br />
+<a class="textlink" href="./gemfeed/2010-05-09-the-fype-programming-language.html">2010-05-09 - The Fype Programming Language</a><br />
+<a class="textlink" href="./gemfeed/2008-06-26-perl-poetry.html">2008-06-26 - Perl Poetry</a><br />
 </body>
 </html>
diff --git a/content/meta/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.meta b/content/meta/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.meta
new file mode 100644
index 00000000..bf6c13e0
--- /dev/null
+++ b/content/meta/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.meta
@@ -0,0 +1,5 @@
+local meta_date="2018-06-01T10:16:14+01:00"
+local meta_author="Paul Buetow"
+local meta_email="comments@mx.buetow.org"
+local meta_title="Realistic load testing with I/O Riot for Linux"
+local meta_summary="This text first was published in the german ADMIN computer Magazine. 3 years have passed since and I decided to publish it on my blog too. . .....to read on please visit my site."