From 596ced56b3729c506af1542e2dd33618adb12b06 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Sun, 6 Jun 2021 09:13:20 +0100 Subject: Some tweaks --- ...5-gemtexter-one-bash-script-to-rule-it-all.html | 27 ++++++++++---------- gemfeed/atom.xml | 29 +++++++++++----------- gemfeed/index.html | 2 +- 3 files changed, 30 insertions(+), 28 deletions(-) (limited to 'gemfeed') diff --git a/gemfeed/2021-06-05-gemtexter-one-bash-script-to-rule-it-all.html b/gemfeed/2021-06-05-gemtexter-one-bash-script-to-rule-it-all.html index c65d4d24..9780251f 100644 --- a/gemfeed/2021-06-05-gemtexter-one-bash-script-to-rule-it-all.html +++ b/gemfeed/2021-06-05-gemtexter-one-bash-script-to-rule-it-all.html @@ -96,7 +96,7 @@ h2, h3 {

Motivation

Another benefit of using Gemini is that the Gemtext markup language is easy to parse. As my site is dual-hosted (Gemini+HTTP), I could, in theory, just write a shell script to deal with the conversion from Gemtext to HTML; there is no need for a full-featured programming language here. I have done a lot of Bash in the past, but I am also often revisiting old tools and techniques for refreshing and keeping the knowledge up to date here.

Motivational comic strip:

I have exactly done that - I wrote a Bash script, named gemtexter, for that. Have a look at GitHub:

I have exactly done that - I wrote a Bash script, named Gemtexter, for that:

https://github.com/snonux/gemtexter

In short, Gemtexter is a static site generator and blogging engine that uses Gemtext as its input format.

Output formats

@@ -109,10 +109,10 @@ h2, h3 {

An HTML Atom feed of my blog posts

I could have done all of that with a more robust language than Bash (such as Perl, Ruby, Go...), but I didn't. The purpose of this exercise was to challenge what I can do with a "simple" Bash script and learn new things.

Taking it as far as I should, but no further

Taking it as far as I should, but no farther

The Bash is suitable very well for small scripts and ad-hoc automation on the command line. But it is for sure not a robust programming language. Writing this blog post, Gemtexter is nearing 1000 lines of code, which is actually a pretty large Bash script.

Modularization

I modularized the code so that each core functionality has its own file in ./lib. All the modules are included (sourced) from the main Gemtexter script. For example, there is one module for HTML generation, one for Markdown generation, and so on.

I modularized the code so that each core functionality has its own file in ./lib. All the modules are included from the main Gemtexter script. For example, there is one module for HTML generation, one for Markdown generation, and so on.

 paul in uranus in gemtexter on 🌱 main
 ❯ wc -l gemtexter lib/*
@@ -127,30 +127,30 @@ paul in uranus in gemtexter on 🌱 main
      63 lib/md.source.sh
      834 total

This way, the script could grow far beyond 1000 lines of code and still be maintainable. With more features, execution speed may slowly become a problem, though. I already notice that Gemtexter doesn't produce results instantly but after a few seconds of runtime. That's not a problem yet, though.

This way, the script could grow far beyond 1000 lines of code and still be maintainable. With more features, execution speed may slowly become a problem, though. I already notice that Gemtexter doesn't produce results instantly but requires few seconds of runtime already. That's not a problem yet, though.

Bash best practises and ShellCheck

While working on Gemtexter, I also had a look at the Google Shell Style Guide and wrote a blog post on that:

Personal bash coding style guide

I followed all these best practices, and in my opinion, the result is a pretty maintainable Bash script (given that you are fluent with all the sed and grep commands I used).

ShellCheck, a shell script analysis tool written in Haskell, is run on Gemtexter together with its unit tests, ensuring that all code is still acceptable. I am pretty impressed with what ShellCheck found.

It, for example, detected "some_command | while read var; do ...; done" loops and hinted that these create a new subprocess for the while part. The result is that all variable modifications taking place in the while-subprocess won't reflect the primary Bash process. ShellSheck then recommended rewriting the loop so that no subprocess is spawned as "while read -r var; do ...; done < <(some_command)". ShellCheck also pointed out to add a "-r" to the read; otherwise, there could be an issue with backspaces in the loop data.

ShellCheck, a shell script analysis tool written in Haskell, is run on Gemtexter ensuring that all code is acceptable. I am pretty impressed with what ShellCheck found.

It, for example, detected "some_command | while read var; do ...; done" loops and hinted that these create a new subprocess for the while part. The result is that all variable modifications taking place in the while-subprocess won't reflect the primary Bash process. ShellSheck then recommended rewriting the loop so that no subprocess is spawned as "while read -r var; do ...; done < <(some_command)". ShellCheck also pointed out to add a "-r" to "read"; otherwise, there could be an issue with backspaces in the loop data.

Furthermore, ShellCheck recommended many more improvements. Declaration of unused variables and missing variable and string quotations were the most common ones. ShellSheck immensely helped to improve the robustness of the script.

https://shellcheck.net

Unit testing

There is a basic unit test module in ./lib/assert.source.sh, which is used for unit testing various functions of Gemtexter. I found this to be very beneficial for cross-platform development. For example, I noticed that some unit tests failed on macOS while everything still worked fine on my Fedora Linux laptop.

After digging a bit, I noticed that I had to install the GNU versions of the sed and grep commands on macOS and a newer version of the Bash to make all unit tests pass and it all work.

There is a basic unit test module in ./lib/assert.source.sh, which is used for unit testing. I found this to be very beneficial for cross-platform development. For example, I noticed that some unit tests failed on macOS while everything still worked fine on my Fedora Linux laptop.

After digging a bit, I noticed that I had to install the GNU versions of the sed and grep commands on macOS and a newer version of the Bash to make all unit tests pass and Gemtexter work.

It has been proven quite helpful to have unit tests in place for the HTML part already when working on the Markdown generator part. To test the Markdown part, I copied the HTML unit tests and changed the expected outcome in the assertions. This way, I could implement the Markdown generator in a test-driven way (writing the test first and afterwards the implementation).

HTML unit test example

-line='=> http://example.org Description of the link'
-assert::equals "$(generate::make_link html "$line")" \
+gemtext='=> http://example.org Description of the link'
+assert::equals "$(generate::make_link html "$gemtext")" \
     '<a class="textlink" href="http://example.org">Description of the link</a><br />'

Markdown unit test example

-line='=> http://example.org Description of the link'
-assert::equals "$(generate::make_link md "$line")" \
+gemtext='=> http://example.org Description of the link'
+assert::equals "$(generate::make_link md "$gemtext")" \
     '[Description of the link](http://example.org)  '

Handcrafted HTML styles

@@ -161,11 +161,12 @@ assert::equals "$(generate::make_link md "$line")" \

Configurability

In case someone else than me wants to use Gemtexter for his own site, it is pretty much configurable. It is possible to specify your own configuration file and your own HTML templates. Have a look at the GitHub page for examples.

Future features

I could think of the following features added to a future version of gemtexter:

I could think of the following features added to a future version of Gemtexter:

Templating of Gemtext files so that the .gmi files are generated from .gmi.tpl files. The template engine could do such things as an automatic table of contents and sitemap generation. It could also include the output of inlined shell code, e.g. a fortune quote.
Add support for more output formats, such as Groff, PDF, plain text, Gopher, etc.
External CSS file for HTML.
Improve speed by introducing parallelism and/or concurrency and/or better caching.

Conclusion

It was quite a lot of fun writing Gemtexter. It's a relatively small project, but given that I worked on that in my spare time once in a while, it kept me busy for several weeks.

diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index 4a8f2809..e1cbebb8 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,6 +1,6 @@ - 2021-06-05T19:08:44+01:00 + 2021-06-06T09:12:58+01:00 buetow.org feed Having fun with computers! @@ -66,7 +66,7 @@

Motivation

Motivational comic strip:

I have exactly done that - I wrote a Bash script, named gemtexter, for that. Have a look at GitHub:

I have exactly done that - I wrote a Bash script, named Gemtexter, for that:

https://github.com/snonux/gemtexter

In short, Gemtexter is a static site generator and blogging engine that uses Gemtext as its input format.

Output formats

@@ -79,10 +79,10 @@

An HTML Atom feed of my blog posts

Taking it as far as I should, but no further

Taking it as far as I should, but no farther

Modularization

 paul in uranus in gemtexter on 🌱 main
 ❯ wc -l gemtexter lib/*
@@ -97,30 +97,30 @@ paul in uranus in gemtexter on 🌱 main
      63 lib/md.source.sh
      834 total

This way, the script could grow far beyond 1000 lines of code and still be maintainable. With more features, execution speed may slowly become a problem, though. I already notice that Gemtexter doesn't produce results instantly but requires few seconds of runtime already. That's not a problem yet, though.

Bash best practises and ShellCheck

While working on Gemtexter, I also had a look at the Google Shell Style Guide and wrote a blog post on that:

Personal bash coding style guide

I followed all these best practices, and in my opinion, the result is a pretty maintainable Bash script (given that you are fluent with all the sed and grep commands I used).

ShellCheck, a shell script analysis tool written in Haskell, is run on Gemtexter ensuring that all code is acceptable. I am pretty impressed with what ShellCheck found.

It, for example, detected "some_command | while read var; do ...; done" loops and hinted that these create a new subprocess for the while part. The result is that all variable modifications taking place in the while-subprocess won't reflect the primary Bash process. ShellSheck then recommended rewriting the loop so that no subprocess is spawned as "while read -r var; do ...; done < <(some_command)". ShellCheck also pointed out to add a "-r" to "read"; otherwise, there could be an issue with backspaces in the loop data.

https://shellcheck.net

Unit testing

After digging a bit, I noticed that I had to install the GNU versions of the sed and grep commands on macOS and a newer version of the Bash to make all unit tests pass and it all work.

After digging a bit, I noticed that I had to install the GNU versions of the sed and grep commands on macOS and a newer version of the Bash to make all unit tests pass and Gemtexter work.

HTML unit test example

-line='=> http://example.org Description of the link'
-assert::equals "$(generate::make_link html "$line")" \
+gemtext='=> http://example.org Description of the link'
+assert::equals "$(generate::make_link html "$gemtext")" \
     '<a class="textlink" href="http://example.org">Description of the link</a><br />'

Markdown unit test example

-line='=> http://example.org Description of the link'
-assert::equals "$(generate::make_link md "$line")" \
+gemtext='=> http://example.org Description of the link'
+assert::equals "$(generate::make_link md "$gemtext")" \
     '[Description of the link](http://example.org)  '

Handcrafted HTML styles

@@ -131,11 +131,12 @@ assert::equals "$(generate::make_link md "$line")" \

Configurability

Future features

I could think of the following features added to a future version of gemtexter:

I could think of the following features added to a future version of Gemtexter:

Templating of Gemtext files so that the .html files are generated from .html.tpl files. The template engine could do such things as an automatic table of contents and sitemap generation. It could also include the output of inlined shell code, e.g. a fortune quote.
Add support for more output formats, such as Groff, PDF, plain text, Gopher, etc.
External CSS file for HTML.
Improve speed by introducing parallelism and/or concurrency and/or better caching.

Conclusion

It was quite a lot of fun writing Gemtexter. It's a relatively small project, but given that I worked on that in my spare time once in a while, it kept me busy for several weeks.

diff --git a/gemfeed/index.html b/gemfeed/index.html index 44abbd38..b9990a62 100644 --- a/gemfeed/index.html +++ b/gemfeed/index.html @@ -50,7 +50,7 @@ h2, h3 {

buetow.org's Gemfeed

Having fun with computers!

-2021-06-05 (1198 words) - Gemtexter - One Bash script to rule it all
+2021-06-05 (1191 words) - Gemtexter - One Bash script to rule it all
2021-05-16 (1717 words) - Personal Bash coding style guide
2021-04-24 (0797 words) - Welcome to the Geminispace
2021-04-22 (2117 words) - DTail - The distributed log tail program
-- cgit v1.2.3