diff options
| author | Paul Buetow <paul@buetow.org> | 2024-09-07 16:33:41 +0300 |
|---|---|---|
| committer | Paul Buetow <paul@buetow.org> | 2024-09-07 16:33:41 +0300 |
| commit | de6323a35e6df2e0016fc498d55484879faf148e (patch) | |
| tree | 07fef72bc46d33ea2697d78c6423c3b4f150c8ce /gemfeed/atom.xml | |
| parent | cf1016e2b1cccb29a4d5a15bf71c35fb50a1f5b8 (diff) | |
Update content for html
Diffstat (limited to 'gemfeed/atom.xml')
| -rw-r--r-- | gemfeed/atom.xml | 643 |
1 files changed, 104 insertions, 539 deletions
diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index 6b7c59a3..154c409c 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,12 +1,98 @@ <?xml version="1.0" encoding="utf-8"?> <feed xmlns="http://www.w3.org/2005/Atom"> - <updated>2024-09-07T16:11:00+03:00</updated> + <updated>2024-09-07T16:32:50+03:00</updated> <title>foo.zone feed</title> <subtitle>To be in the .zone!</subtitle> <link href="https://foo.zone/gemfeed/atom.xml" rel="self" /> <link href="https://foo.zone/" /> <id>https://foo.zone/</id> <entry> + <title>Site Reliability Engineering - Part 4: Onboarding for On-Call Engineers</title> + <link href="https://foo.zone/gemfeed/2024-09-07-site-reliability-engineering-part-4.html" /> + <id>https://foo.zone/gemfeed/2024-09-07-site-reliability-engineering-part-4.html</id> + <updated>2024-09-07T16:27:58+03:00</updated> + <author> + <name>Paul Buetow aka snonux</name> + <email>paul@dev.buetow.org</email> + </author> + <summary>Welcome to Part 4 of my Site Reliability Engineering (SRE) series. I'm currently working as a Site Reliability Engineer, and I’m here to share what SRE is all about in this blog series.</summary> + <content type="xhtml"> + <div xmlns="http://www.w3.org/1999/xhtml"> + <h1 style='display: inline' id='site-reliability-engineering---part-4-onboarding-for-on-call-engineers'>Site Reliability Engineering - Part 4: Onboarding for On-Call Engineers</h1><br /> +<br /> +<span class='quote'>Published at 2024-09-07T16:27:58+03:00</span><br /> +<br /> +<span>Welcome to Part 4 of my Site Reliability Engineering (SRE) series. I'm currently working as a Site Reliability Engineer, and I’m here to share what SRE is all about in this blog series.</span><br /> +<br /> +<a class='textlink' href='./2023-08-18-site-reliability-engineering-part-1.html'>2023-08-18 Site Reliability Engineering - Part 1: SRE and Organizational Culture</a><br /> +<a class='textlink' href='./2023-11-19-site-reliability-engineering-part-2.html'>2023-11-19 Site Reliability Engineering - Part 2: Operational Balance</a><br /> +<a class='textlink' href='./2024-01-09-site-reliability-engineering-part-3.html'>2024-01-09 Site Reliability Engineering - Part 3: On-Call Culture</a><br /> +<a class='textlink' href='./2024-09-07-site-reliability-engineering-part-4.html'>2024-09-07 Site Reliability Engineering - Part 4: Onboarding for On-Call Engineers (You are currently reading this)</a><br /> +<br /> +<pre> + __..._ _...__ + _..-" `Y` "-._ + \ Once upon | / + \\ a time..| // + \\\ | /// + \\\ _..---.|.---.._ /// +jgs \\`_..---.Y.---.._`// +</pre> +<br /> +<span>This time, I want to share some tips on how to onboard software engineers, QA engineers, and Site Reliability Engineers (SREs) to the primary on-call rotation. Traditionally, onboarding might take half a year (depending on the complexity of the infrastructure), but with a bit of strategy and structured sessions, we've managed to reduce it to just six weeks per person. Let's dive in!</span><br /> +<br /> +<h2 style='display: inline' id='setting-the-scene-tier-1-on-call-rotation'>Setting the Scene: Tier-1 On-Call Rotation</h2><br /> +<br /> +<span>First things first, let's talk about Tier-1. This is where the magic begins. Tier-1 covers over 80% of the common on-call cases and is the perfect breeding ground for new on-call engineers to get their feet wet. It's designed to be manageable training ground.</span><br /> +<br /> +<h3 style='display: inline' id='why-tier-1'>Why Tier-1?</h3><br /> +<br /> +<ul> +<li>Easy to Understand: Every on-call engineer should be familiar with Tier-1 tasks. </li> +<li>Training Ground: This is where engineers start their on-call career. It's purposefully kept simple so that it's not overwhelming right off the bat.</li> +<li>Runbook/recipe driven: Every alert is attached to a comprehensive runbook, making it easy for every engineer to follow.</li> +</ul><br /> +<h2 style='display: inline' id='onboarding-process-from-6-months-to-6-weeks'>Onboarding Process: From 6 Months to 6 Weeks</h2><br /> +<br /> +<span>So how did we cut down the onboarding time so drastically? Here’s the breakdown of our process:</span><br /> +<br /> +<span>Knowledge Transfer (KT) Sessions: We kicked things off with more than 10 KT sessions, complete with video recordings. These sessions are comprehensive and cover everything from the basics to some more advanced topics. The recorded sessions mean that new engineers can revisit them anytime they need a refresher.</span><br /> +<br /> +<span>Shadowing Sessions: Each new engineer undergoes two on-call week shadowing sessions. This hands-on experience is invaluable. They get to see real-time incident handling and resolution, gaining practical knowledge that's hard to get from just reading docs.</span><br /> +<br /> +<span>Comprehensive Runbooks: We created 64 runbooks (by the time writing this probably more than 100) that are composable like Lego bricks. Each runbook covers a specific scenario and guides the engineer step-by-step to resolution. Pairing these with monitoring alerts linked directly to Confluence docs, and from there to the respective runbooks, ensures every alert can be navigated with ease (well, there are always exceptions to the rule...).</span><br /> +<br /> +<span>Self-Sufficiency & Confidence Building: With all these resources at their fingertips, our on-call engineers become self-sufficient for most of the common issues they'll face (new starters can now handle around 80% of the most common issue after 6 weeks they had joined the company). This boosts their confidence and ensures they can handle Tier-1 incidents independently.</span><br /> +<br /> +<span>Documentation and Feedback Loop: Continuous improvement is key. We regularly update our documentation based on feedback from the engineers. This makes our process even more robust and user-friendly.</span><br /> +<br /> +<h2 style='display: inline' id='it-s-all-about-the-tiers'>It's All About the Tiers</h2><br /> +<br /> +<span>Let’s briefly touch on the Tier levels:</span><br /> +<br /> +<ul> +<li>Tier 1: Easy and foundational tasks. Perfect for getting new engineers started. This covers around 80% of all on-call cases we face. This is what we trained on.</li> +<li>Tier 2: Slightly more complex, requiring more background knowledge. We trained on some of the topics but not all.</li> +<li>Tier 3: Requires a good understanding of the platform/architecture. Likely needs KT sessions with domain experts.</li> +<li>Tier DE (Domain Expert): The heavy hitters. Domain experts are required for these tasks. </li> +</ul><br /> +<h3 style='display: inline' id='growing-into-higher-tiers'>Growing into Higher Tiers</h3><br /> +<br /> +<span>From Tier-1, engineers naturally grow into Tier-2 and beyond. The structured training and gradual increase in complexity help ensure a smooth transition as they gain experience and confidence. The key here is that engineers stay curous and engaged in the on-call, so that they always keep learning.</span><br /> +<br /> +<h2 style='display: inline' id='keeping-runbooks-up-to-date'>Keeping Runbooks Up to Date</h2><br /> +<br /> +<span>It is important that runbooks are not a "project to be finished"; runbooks have to be maintained and updated over time. Sections may change, new runbooks need to be added, and old ones can be deleted. So the acceptance criteria of an on-call shift would not just be reacting to alerts and incidents, but also reviewing and updating the current runbooks.</span><br /> +<br /> +<h2 style='display: inline' id='conclusion'>Conclusion</h2><br /> +<br /> +<span>By structuring the onboarding process with KT sessions, shadowing, comprehensive runbooks, and a feedback loop, we've been able to fast-track the process from six months to just six weeks. This not only prepares our engineers for the on-call rotation quicker but also ensures they're confident and capable when handling incidents.</span><br /> +<br /> +<span>If you're looking to optimize your on-call onboarding process, these strategies could be your ticket to a more efficient and effective transition. Happy on-calling!</span><br /> + </div> + </content> + </entry> + <entry> <title>Projects I support</title> <link href="https://foo.zone/gemfeed/2024-09-07-projects-i-support.html" /> <id>https://foo.zone/gemfeed/2024-09-07-projects-i-support.html</id> @@ -2473,7 +2559,7 @@ http://www.gnu.org/software/src-highlite --> </content> </entry> <entry> - <title>Site Reliability Engineering - Part 3: On-Call Culture and the Human Side</title> + <title>Site Reliability Engineering - Part 3: On-Call Culture</title> <link href="https://foo.zone/gemfeed/2024-01-09-site-reliability-engineering-part-3.html" /> <id>https://foo.zone/gemfeed/2024-01-09-site-reliability-engineering-part-3.html</id> <updated>2024-01-09T18:35:48+02:00</updated> @@ -2484,15 +2570,16 @@ http://www.gnu.org/software/src-highlite --> <summary>Welcome to Part 3 of my Site Reliability Engineering (SRE) series. I'm currently working as a Site Reliability Engineer, and I’m here to share what SRE is all about in this blog series.</summary> <content type="xhtml"> <div xmlns="http://www.w3.org/1999/xhtml"> - <h1 style='display: inline' id='site-reliability-engineering---part-3-on-call-culture-and-the-human-side'>Site Reliability Engineering - Part 3: On-Call Culture and the Human Side</h1><br /> + <h1 style='display: inline' id='site-reliability-engineering---part-3-on-call-culture'>Site Reliability Engineering - Part 3: On-Call Culture</h1><br /> <br /> <span class='quote'>Published at 2024-01-09T18:35:48+02:00</span><br /> <br /> <span>Welcome to Part 3 of my Site Reliability Engineering (SRE) series. I'm currently working as a Site Reliability Engineer, and I’m here to share what SRE is all about in this blog series.</span><br /> <br /> <a class='textlink' href='./2023-08-18-site-reliability-engineering-part-1.html'>2023-08-18 Site Reliability Engineering - Part 1: SRE and Organizational Culture</a><br /> -<a class='textlink' href='./2023-11-19-site-reliability-engineering-part-2.html'>2023-11-19 Site Reliability Engineering - Part 2: Operational Balance in SRE</a><br /> -<a class='textlink' href='./2024-01-09-site-reliability-engineering-part-3.html'>2024-01-09 Site Reliability Engineering - Part 3: On-Call Culture and the Human Side (You are currently reading this)</a><br /> +<a class='textlink' href='./2023-11-19-site-reliability-engineering-part-2.html'>2023-11-19 Site Reliability Engineering - Part 2: Operational Balance</a><br /> +<a class='textlink' href='./2024-01-09-site-reliability-engineering-part-3.html'>2024-01-09 Site Reliability Engineering - Part 3: On-Call Culture (You are currently reading this)</a><br /> +<a class='textlink' href='./2024-09-07-site-reliability-engineering-part-4.html'>2024-09-07 Site Reliability Engineering - Part 4: Onboarding for On-Call Engineers</a><br /> <br /> <pre> ..--""""----.. @@ -2520,7 +2607,7 @@ http://www.gnu.org/software/src-highlite --> </pre> <br /> -<h2 style='display: inline' id='on-call-culture-and-the-human-side-putting-well-being-first-in-the-world-of-reliability'>On-Call Culture and the Human Side: Putting Well-being First in the World of Reliability</h2><br /> +<h2 style='display: inline' id='putting-well-being-first'>Putting Well-being First</h2><br /> <br /> <span>Site Reliability Engineering is all about keeping systems reliable, but we often forget how important the human side is. A healthy on-call culture is just as crucial as any technical fix. The well-being of the engineers really matters.</span><br /> <br /> @@ -2977,7 +3064,7 @@ echo baz </content> </entry> <entry> - <title>Site Reliability Engineering - Part 2: Operational Balance in SRE</title> + <title>Site Reliability Engineering - Part 2: Operational Balance</title> <link href="https://foo.zone/gemfeed/2023-11-19-site-reliability-engineering-part-2.html" /> <id>https://foo.zone/gemfeed/2023-11-19-site-reliability-engineering-part-2.html</id> <updated>2023-11-19T00:18:18+03:00</updated> @@ -2988,15 +3075,16 @@ echo baz <summary>This is the second part of my Site Reliability Engineering (SRE) series. I am currently employed as a Site Reliability Engineer and will try to share what SRE is about in this blog series.</summary> <content type="xhtml"> <div xmlns="http://www.w3.org/1999/xhtml"> - <h1 style='display: inline' id='site-reliability-engineering---part-2-operational-balance-in-sre'>Site Reliability Engineering - Part 2: Operational Balance in SRE</h1><br /> + <h1 style='display: inline' id='site-reliability-engineering---part-2-operational-balance'>Site Reliability Engineering - Part 2: Operational Balance</h1><br /> <br /> <span class='quote'>Published at 2023-11-19T00:18:18+03:00</span><br /> <br /> <span>This is the second part of my Site Reliability Engineering (SRE) series. I am currently employed as a Site Reliability Engineer and will try to share what SRE is about in this blog series.</span><br /> <br /> <a class='textlink' href='./2023-08-18-site-reliability-engineering-part-1.html'>2023-08-18 Site Reliability Engineering - Part 1: SRE and Organizational Culture</a><br /> -<a class='textlink' href='./2023-11-19-site-reliability-engineering-part-2.html'>2023-11-19 Site Reliability Engineering - Part 2: Operational Balance in SRE (You are currently reading this)</a><br /> -<a class='textlink' href='./2024-01-09-site-reliability-engineering-part-3.html'>2024-01-09 Site Reliability Engineering - Part 3: On-Call Culture and the Human Side</a><br /> +<a class='textlink' href='./2023-11-19-site-reliability-engineering-part-2.html'>2023-11-19 Site Reliability Engineering - Part 2: Operational Balance (You are currently reading this)</a><br /> +<a class='textlink' href='./2024-01-09-site-reliability-engineering-part-3.html'>2024-01-09 Site Reliability Engineering - Part 3: On-Call Culture</a><br /> +<a class='textlink' href='./2024-09-07-site-reliability-engineering-part-4.html'>2024-09-07 Site Reliability Engineering - Part 4: Onboarding for On-Call Engineers</a><br /> <br /> <pre> ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣠⣾⣷⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ @@ -3013,7 +3101,7 @@ echo baz ⠀⠀⠀⠀⠀⠀⠴⠶⠿⠿⠿⠿⠿⠿⠿⠿⠿⠿⠿⠿⠿⠿⠶⠦⠀⠀ </pre> <br /> -<h2 style='display: inline' id='operational-balance-in-sre-striking-the-right-balance-between-reliability-and-speed'>Operational Balance in SRE: Striking the Right Balance Between Reliability and Speed</h2><br /> +<h2 style='display: inline' id='striking-the-right-balance-between-reliability-and-speed'>Striking the Right Balance Between Reliability and Speed</h2><br /> <br /> <span>Site Reliability Engineering is more than just a bunch of best practices or methods. It's a guiding light for engineering teams, helping them navigate the tricky waters of modern software development and system management.</span><br /> <span>In the world of software production, there are two big forces that often clash: the push for fast feature releases (velocity) and the need for reliable systems. Traditionally, moving faster meant more risk. SRE helps balance these opposing goals with things like error budgets and SLIs/SLOs. These tools give teams a clear way to measure how much they can push changes without hurting system health. So, the error budget becomes a balancing act, helping teams trade off between innovation and reliability.</span><br /> @@ -3032,7 +3120,7 @@ echo baz <br /> <span>Continue with the third part of this series:</span><br /> <br /> -<a class='textlink' href='./2024-01-09-site-reliability-engineering-part-3.html'>2024-01-09 Site Reliability Engineering - Part 3: On-Call Culture and the Human Side</a><br /> +<a class='textlink' href='./2024-01-09-site-reliability-engineering-part-3.html'>2024-01-09 Site Reliability Engineering - Part 3: On-Call Culture</a><br /> <br /> <span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br /> <br /> @@ -3849,8 +3937,9 @@ http://www.gnu.org/software/src-highlite --> <span>Being a Site Reliability Engineer (SRE) is like stepping into a lively, ever-evolving universe. The world of SRE mixes together different tech, a unique culture, and a whole lot of determination. It’s one of the toughest but most exciting jobs out there. There's zero chance of getting bored because there's always a fresh challenge to tackle and new technology to play around with. It's not just about the tech side of things either; it's heavily rooted in communication, collaboration, and teamwork. As someone currently working as an SRE, I’m here to break it all down for you in this blog series. Let's dive into what SRE is really all about!</span><br /> <br /> <a class='textlink' href='./2023-08-18-site-reliability-engineering-part-1.html'>2023-08-18 Site Reliability Engineering - Part 1: SRE and Organizational Culture (You are currently reading this)</a><br /> -<a class='textlink' href='./2023-11-19-site-reliability-engineering-part-2.html'>2023-11-19 Site Reliability Engineering - Part 2: Operational Balance in SRE</a><br /> -<a class='textlink' href='./2024-01-09-site-reliability-engineering-part-3.html'>2024-01-09 Site Reliability Engineering - Part 3: On-Call Culture and the Human Side</a><br /> +<a class='textlink' href='./2023-11-19-site-reliability-engineering-part-2.html'>2023-11-19 Site Reliability Engineering - Part 2: Operational Balance</a><br /> +<a class='textlink' href='./2024-01-09-site-reliability-engineering-part-3.html'>2024-01-09 Site Reliability Engineering - Part 3: On-Call Culture</a><br /> +<a class='textlink' href='./2024-09-07-site-reliability-engineering-part-4.html'>2024-09-07 Site Reliability Engineering - Part 4: Onboarding for On-Call Engineers</a><br /> <br /> <pre> ▓▓▓▓░░ @@ -3896,7 +3985,7 @@ DC on fire: <br /> <span>Continue with the second part of this series:</span><br /> <br /> -<a class='textlink' href='./2023-11-19-site-reliability-engineering-part-2.html'>2023-11-19 Site Reliability Engineering - Part 2: Operational Balance in SRE</a><br /> +<a class='textlink' href='./2023-11-19-site-reliability-engineering-part-2.html'>2023-11-19 Site Reliability Engineering - Part 2: Operational Balance</a><br /> <br /> <span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br /> <br /> @@ -9081,528 +9170,4 @@ GNU/kFreeBSD rhea.buetow.org 8.0-RELEASE-p5 FreeBSD 8.0-RELEASE-p5 #2: Sat Nov 2 </div> </content> </entry> - <entry> - <title>Bash Golf Part 2</title> - <link href="https://foo.zone/gemfeed/2022-01-01-bash-golf-part-2.html" /> - <id>https://foo.zone/gemfeed/2022-01-01-bash-golf-part-2.html</id> - <updated>2022-01-01T23:36:15+00:00</updated> - <author> - <name>Paul Buetow aka snonux</name> - <email>paul@dev.buetow.org</email> - </author> - <summary>This is the second blog post about my Bash Golf series. This series is random Bash tips, tricks and weirdnesses I came across. It's a collection of smaller articles I wrote in an older (in German language) blog, which I translated and refreshed with some new content.</summary> - <content type="xhtml"> - <div xmlns="http://www.w3.org/1999/xhtml"> - <h1 style='display: inline' id='bash-golf-part-2'>Bash Golf Part 2</h1><br /> -<br /> -<span class='quote'>Published at 2022-01-01T23:36:15+00:00; Updated at 2022-01-05</span><br /> -<br /> -<span>This is the second blog post about my Bash Golf series. This series is random Bash tips, tricks and weirdnesses I came across. It's a collection of smaller articles I wrote in an older (in German language) blog, which I translated and refreshed with some new content.</span><br /> -<br /> -<a class='textlink' href='./2021-11-29-bash-golf-part-1.html'>2021-11-29 Bash Golf Part 1</a><br /> -<a class='textlink' href='./2022-01-01-bash-golf-part-2.html'>2022-01-01 Bash Golf Part 2 (You are currently reading this)</a><br /> -<a class='textlink' href='./2023-12-10-bash-golf-part-3.html'>2023-12-10 Bash Golf Part 3</a><br /> -<br /> -<pre> - '\ '\ . . |>18>> - \ \ . ' . | - O>> O>> . 'o | - \ .\. .. . | - /\ . /\ . . | - / / . / / .' . | -jgs^^^^^^^`^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Art by Joan Stark, mod. by Paul Buetow -</pre> -<br /> -<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br /> -<br /> -<ul> -<li><a href='#bash-golf-part-2'>Bash Golf Part 2</a></li> -<li>⇢ <a href='#redirection'>Redirection</a></li> -<li>⇢ <a href='#here'>HERE</a></li> -<li>⇢ <a href='#random'>RANDOM</a></li> -<li>⇢ <a href='#set--x-and-set--e-and-pipefile'>set -x and set -e and pipefile</a></li> -<li>⇢ ⇢ <a href='#-x'>-x</a></li> -<li>⇢ ⇢ <a href='#-e'>-e</a></li> -<li>⇢ ⇢ <a href='#pipefail'>pipefail</a></li> -</ul><br /> -<h2 style='display: inline' id='redirection'>Redirection</h2><br /> -<br /> -<span>Let's have a closer look at Bash redirection. As you might already know that there are 3 standard file descriptors:</span><br /> -<br /> -<ul> -<li>0 aka stdin (standard input)</li> -<li>1 aka stdout (standard output)</li> -<li>2 aka stderr (standard error output)</li> -</ul><br /> -<span>These are most certainly the ones you are using on regular basis. "/proc/self/fd" lists all file descriptors which are open by the current process (in this case: the current Bash shell itself):</span><br /> -<br /> -<pre> -❯ ls -l /proc/self/fd/ -total 0 -lrwx------. 1 paul paul 64 Nov 23 09:46 0 -> /dev/pts/9 -lrwx------. 1 paul paul 64 Nov 23 09:46 1 -> /dev/pts/9 -lrwx------. 1 paul paul 64 Nov 23 09:46 2 -> /dev/pts/9 -lr-x------. 1 paul paul 64 Nov 23 09:46 3 -> /proc/162912/fd -</pre> -<br /> -<span>The following examples demonstrate two different ways to accomplish the same thing. The difference is that the first command is directly printing out "Foo" to stdout and the second command is explicitly redirecting stdout to its own stdout file descriptor:</span><br /> -<br /> -<pre> -❯ echo Foo -Foo -❯ echo Foo > /proc/self/fd/0 -Foo -</pre> -<br /> -<span class='quote'>Update: A reader pointed out, that the redirection should actually go to <span class='inlinecode'>/proc/self/fd/1</span> and not <span class='inlinecode'>0</span>. But apparently, either way works for this particular example. Do you know why? </span><br /> -<br /> -<span>Other useful redirections are:</span><br /> -<br /> -<ul> -<li>Redirect stderr to stdin: "echo foo 2>&1"</li> -<li>Redirect stdin to stderr: "echo foo >&2"</li> -</ul><br /> -<span>It is, however, not possible to redirect multiple times within the same command. E.g. the following won't work. You would expect stdin to be redirected to stderr and then stderr to be redirected to /dev/null. But as the example shows, Foo is still printed out:</span><br /> -<br /> -<pre> -❯ echo Foo 1>&2 2>/dev/null -Foo -</pre> -<br /> -<span class='quote'>Update: A reader sent me an email and pointed out that the order of the redirections is important. </span><br /> -<br /> -<span>As you can see, the following will not print out anything:</span><br /> -<br /> -<pre> -❯ echo Foo 2>/dev/null 1>&2 -❯ -</pre> -<br /> -<span>A good description (also pointed out by the reader) can be found here:</span><br /> -<br /> -<a class='textlink' href='https://wiki.bash-hackers.org/howto/redirection_tutorial#order_of_redirection_ie_file_2_1_vs_2_1_file'>Order of redirection</a><br /> -<br /> -<span>Ok, back to the original blog post. You can also use grouping here (neither of these commands will print out anything to stdout):</span><br /> -<br /> -<pre> -❯ { echo Foo 1>&2; } 2>/dev/null -❯ ( echo Foo 1>&2; ) 2>/dev/null -❯ { { { echo Foo 1>&2; } 2>&1; } 1>&2; } 2>/dev/null -❯ ( ( ( echo Foo 1>&2; ) 2>&1; ) 1>&2; ) 2>/dev/null -❯ -</pre> -<br /> -<span>A handy way to list all open file descriptors is to use the "lsof" command (that's not a Bash built-in), whereas $$ is the process id (pid) of the current shell process:</span><br /> -<br /> -<pre> -❯ lsof -a -p $$ -d0,1,2 -COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME -bash 62676 paul 0u CHR 136,9 0t0 12 /dev/pts/9 -bash 62676 paul 1u CHR 136,9 0t0 12 /dev/pts/9 -bash 62676 paul 2u CHR 136,9 0t0 12 /dev/pts/9 -</pre> -<br /> -<span>Let's create our own descriptor "3" for redirection to a file named "foo":</span><br /> -<br /> -<pre> -❯ touch foo -❯ exec 3>foo # This opens fd 3 and binds it to file foo. -❯ ls -l /proc/self/fd/3 -l-wx------. 1 paul paul 64 Nov 23 10:10 \ - /proc/self/fd/3 -> /home/paul/foo -❯ cat foo -❯ echo Bratwurst >&3 -❯ cat foo -Bratwurst -❯ exec 3>&- # This closes fd 3. -❯ echo Steak >&3 --bash: 3: Bad file descriptor -</pre> -<br /> -<span>You can also override the default file descriptors, as the following example script demonstrates:</span><br /> -<br /> -<pre> -❯ cat grandmaster.sh -#!/usr/bin/env bash - -# Write a file data-file containing two lines -echo Learn You a Haskell > data-file -echo for Great Good >> data-file - -# Link fd with fd 6 (saves default stdin) -exec 6<&0 - -# Overwrite stdin with data-file -exec < data-file - -# Read the first two lines from it -declare LINE1 LINE2 -read LINE1 -read LINE2 - -# Print them -echo First line: $LINE1 -echo Second line: $LINE2 - -# Restore default stdin and delete fd 6 -exec 0<&6 6<&- -</pre> -<br /> -<span>Let's execute it:</span><br /> -<br /> -<pre> -❯ chmod 750 ./grandmaster.sh -❯ ./grandmaster.sh -First line: Learn You a Haskell -Second line: for Great Good -</pre> -<br /> -<h2 style='display: inline' id='here'>HERE</h2><br /> -<br /> -<span>I have mentioned HERE-documents and HERE-strings already in this post. Let's do some more examples. The following "cat" receives a multi line string from stdin. In this case, the input multi line string is a HERE-document. As you can see, it also interpolates variables (in this case the output of "date" running in a subshell).</span><br /> -<br /> -<pre> -❯ cat <<END -> Hello World -> It’s $(date) -> END -Hello World -It's Fri 26 Nov 08:46:52 GMT 2021 -</pre> -<br /> -<span>You can also write it the following way, but that's less readable (it's good for an obfuscation contest):</span><br /> -<br /> -<pre> -❯ <<END cat -> Hello Universe -> It’s $(date) -> END -Hello Universe -It's Fri 26 Nov 08:47:32 GMT 2021 -</pre> -<br /> -<span>Besides of an HERE-document, there is also a so-called HERE-string. Besides of...</span><br /> -<br /> -<pre> -❯ declare VAR=foo -❯ if echo "$VAR" | grep -q foo; then -> echo '$VAR ontains foo' -> fi -$VAR ontains foo -</pre> -<br /> -<span>...you can use a HERE-string like that:</span><br /> -<br /> -<pre> -❯ if grep -q foo <<< "$VAR"; then -> echo '$VAR contains foo' -> fi -$VAR contains foo -</pre> -<br /> -<span>Or even shorter, you can do:</span><br /> -<br /> -<pre> -❯ grep -q foo <<< "$VAR" && echo '$VAR contains foo' -$VAR contains foo -</pre> -<br /> -<span>You can also use a Bash regex to accomplish the same thing, but the points of the examples so far were to demonstrate HERE-{documents,strings} and not Bash regular expressions:</span><br /> -<br /> -<pre> -❯ if [[ "$VAR" =~ foo ]]; then echo yay; fi -yay -</pre> -<br /> -<span>You can also use it with "read":</span><br /> -<br /> -<pre> -❯ read a <<< ja -❯ echo $a -ja -❯ read b <<< 'NEIN!!!' -❯ echo $b -NEIN!!! -❯ dumdidumstring='Learn you a Golang for Great Good' -❯ read -a words <<< "$dumdidumstring" -❯ echo ${words[0]} -Learn -❯ echo ${words[3]} -Golang -</pre> -<br /> -<span>The following is good for an obfuscation contest too:</span><br /> -<br /> -<pre> -❯ echo 'I like Perl too' > perllove.txt -❯ cat - perllove.txt <<< "$dumdidumstring" -Learn you a Golang for Great Good -I like Perl too -</pre> -<br /> -<h2 style='display: inline' id='random'>RANDOM</h2><br /> -<br /> -<span>Random is a special built-in variable containing a different pseudo random number each time it's used.</span><br /> -<br /> -<pre> -❯ echo $RANDOM -11811 -❯ echo $RANDOM -14997 -❯ echo $RANDOM -9104 -</pre> -<br /> -<span>That's very useful if you want to randomly delay the execution of your scripts when you run it on many servers concurrently, just to spread the server load (which might be caused by the script run) better.</span><br /> -<br /> -<span>Let's say you want to introduce a random delay of 1 minute. You can accomplish it with:</span><br /> -<br /> -<pre> -❯ cat ./calc_answer_to_ultimate_question_in_life.sh -#!/usr/bin/env bash - -declare -i MAX_DELAY=60 - -random_delay () { - local -i sleep_for=$((RANDOM % MAX_DELAY)) - echo "Delaying script execution for $sleep_for seconds..." - sleep $sleep_for - echo 'Continuing script execution...' -} - -main () { - random_delay - # From here, do the real work. Calculating the answer to - # the ultimate question can take billions of years.... - : .... -} - -main - -❯ -❯ ./calc_answer_to_ultimate_question_in_life.sh -Delaying script execution for 42 seconds... -Continuing script execution... -</pre> -<br /> -<h2 style='display: inline' id='set--x-and-set--e-and-pipefile'>set -x and set -e and pipefile</h2><br /> -<br /> -<span>In my opinion, -x and -e and pipefile are the most useful Bash options. Let's have a look at them one after another.</span><br /> -<br /> -<h3 style='display: inline' id='-x'>-x</h3><br /> -<br /> -<span>-x prints commands and their arguments as they are executed. This helps to develop and debug your Bash code:</span><br /> -<br /> -<pre> -❯ set -x -❯ square () { local -i num=$1; echo $((num*num)); } -❯ num=11; echo "Square of $num is $(square $num)" -+ num=11 -++ square 11 -++ local -i num=11 -++ echo 121 -+ echo 'Square of 11 is 121' -Square of 11 is 121 -</pre> -<br /> -<span>You can also set it when calling an external script without modifying the script itself:</span><br /> -<br /> -<pre> -❯ bash -x ./half_broken_script_to_be_debugged.sh -</pre> -<br /> -<span>Let's do that on one of the example scripts we covered earlier:</span><br /> -<br /> -<pre> -❯ bash -x ./grandmaster.sh -+ bash -x ./grandmaster.sh -+ echo Learn You a Haskell -+ echo for Great Good -+ exec -+ exec -+ declare LINE1 LINE2 -+ read LINE1 -+ read LINE2 -+ echo First line: Learn You a Haskell -First line: Learn You a Haskell -+ echo Second line: for Great Good -Second line: for Great Good -+ exec -❯ -</pre> -<br /> -<h3 style='display: inline' id='-e'>-e</h3><br /> -<br /> -<span>This is a very important option you want to use when you are paranoid. This means, you should always "set -e" in your scripts when you need to make absolutely sure that your script runs successfully (with that I mean that no command should exit with an unexpected status code).</span><br /> -<br /> -<span>Ok, let's dig deeper:</span><br /> -<br /> -<pre> -❯ help set | grep -- -e - -e Exit immediately if a command exits with a non-zero status. -</pre> -<br /> -<span>As you can see in the following example, the Bash terminates after the execution of "grep" as "foo" is not matching "bar". Therefore, grep exits with 1 (unsuccessfully) and the shell aborts. And therefore, "bar" will not be printed out anymore:</span><br /> -<br /> -<pre> -❯ bash -c 'set -e; echo hello; grep -q bar <<< foo; echo bar' -hello -❯ echo $? -1 -</pre> -<br /> -<span>Whereas the outcome changes when the regex matches:</span><br /> -<br /> -<pre> -❯ bash -c 'set -e; echo hello; grep -q bar <<< barman; echo bar' -hello -bar -❯ echo $? -0 -</pre> -<br /> -<span>So does it mean that grep will always make the shell terminate whenever its exit code isn't 0? This will render "set -e" quite unusable. Frankly, there are other commands where an exit status other than 0 should not terminate the whole script abruptly. Usually, what you want is to branch your code based on the outcome (exit code) of a command:</span><br /> -<br /> -<pre> -❯ bash -c 'set -e -> grep -q bar <<< foo -> if [ $? -eq 0 ]; then -> echo "matching" -> else -> echo "not matching" -> fi' -❯ echo $? -1 -</pre> -<br /> -<span>...but the example above won't reach any of the branches and won't print out anything, as the script terminates right after grep.</span><br /> -<br /> -<span>The proper solution is to use grep as an expression in a conditional (e.g. in an if-else statement):</span><br /> -<br /> -<pre> -❯ bash -c 'set -e -> if grep -q bar <<< foo; then -> echo "matching" -> else -> echo "not matching" -> fi' -not matching -❯ echo $? -0 -❯ bash -c 'set -e -> if grep -q bar <<< barman; then -> echo "matching" -> else -> echo "not matching" -> fi' -matching -❯ echo $? -0 -</pre> -<br /> -<span>You can also temporally undo "set -e" if there is no other way:</span><br /> -<br /> -<pre> -❯ cat ./e.sh -#!/usr/bin/env bash - -set -e - -foo () { - local arg="$1"; shift - - if [ -z "$arg" ]; then - arg='You!' - fi - echo "Hello $arg" -} - -bar () { - # Temporally disable e - set +e - local arg="$1"; shift - # Enable e again. - set -e - - if [ -z "$arg" ]; then - arg='You!' - fi - echo "Hello $arg" -} - -# Will succeed -bar World -foo Universe -bar - -# Will terminate the script -foo - -❯ ./e.sh -Hello World -Hello Universe -Hello You! -</pre> -<br /> -<span>Why does calling "foo" with no arguments make the script terminate? Because as no argument was given, the "shift" won't have anything to do as the argument list $@ is empty, and therefore "shift" fails with a non-zero status.</span><br /> -<br /> -<span>Why would you want to use "shift" after function-local variable assignments? Have a look at my personal Bash coding style guide for an explanation :-):</span><br /> -<br /> -<a class='textlink' href='./2021-05-16-personal-bash-coding-style-guide.html'>./2021-05-16-personal-bash-coding-style-guide.html</a><br /> -<br /> -<h3 style='display: inline' id='pipefail'>pipefail</h3><br /> -<br /> -<span>The pipefail option makes it so that not only the exit code of the last command of the pipe counts regards its exit code but any command of the pipe:</span><br /> -<br /> -<pre> -❯ help set | grep pipefail -A 2 - pipefail the return value of a pipeline is the status of - the last command to exit with a non-zero status, - or zero if no command exited with a non-zero status -</pre> -<br /> -<span>The following greps for paul in passwd and converts all lowercase letters to uppercase letters. The exit code of the pipe is 0, as the last command of the pipe (converting from lowercase to uppercase) succeeded:</span><br /> -<br /> -<pre> -❯ grep paul /etc/passwd | tr '[a-z]' '[A-Z]' -PAUL:X:1000:1000:PAUL BUETOW:/HOME/PAUL:/BIN/BASH -❯ echo $? -0 -</pre> -<br /> -<span>Let's look at another example, where "TheRock" doesn't exist in the passwd file. However, the pipes exit status is still 0 (success). This is so because the last command ("tr" in this case) still succeeded. It is just that it didn't get any input on stdin to process:</span><br /> -<br /> -<pre> -❯ grep TheRock /etc/passwd -❯ echo $? -1 -❯ grep TheRock /etc/passwd | tr '[a-z]' '[A-Z]' -❯ echo $? -0 -</pre> -<br /> -<span>To change this behaviour, pipefile can be used. Now, the pipes exit status is 1 (fail), because the pipe contains at least one command (in this case grep) which exited with status 1:</span><br /> -<br /> -<pre> -❯ set -o pipefail -❯ grep TheRock /etc/passwd | tr '[a-z]' '[A-Z]' -❯ echo $? -1 -</pre> -<br /> -<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br /> -<br /> -<span>Other related posts are:</span><br /> -<br /> -<a class='textlink' href='./2021-05-16-personal-bash-coding-style-guide.html'>2021-05-16 Personal Bash coding style guide</a><br /> -<a class='textlink' href='./2021-06-05-gemtexter-one-bash-script-to-rule-it-all.html'>2021-06-05 Gemtexter - One Bash script to rule it all</a><br /> -<a class='textlink' href='./2021-11-29-bash-golf-part-1.html'>2021-11-29 Bash Golf Part 1</a><br /> -<a class='textlink' href='./2022-01-01-bash-golf-part-2.html'>2022-01-01 Bash Golf Part 2 (You are currently reading this)</a><br /> -<a class='textlink' href='./2023-12-10-bash-golf-part-3.html'>2023-12-10 Bash Golf Part 3</a><br /> -<br /> -<a class='textlink' href='../'>Back to the main site</a><br /> - </div> - </content> - </entry> </feed> |