From 7f614094b3b5b02eeccd6354bfe069eeeca45ee1 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Wed, 26 Apr 2023 00:15:49 +0300 Subject: new draft --- gemfeed/DRAFT-kiss-monitoring-with-gogios.gmi | 145 +++++++++++++++----------- 1 file changed, 87 insertions(+), 58 deletions(-) (limited to 'gemfeed') diff --git a/gemfeed/DRAFT-kiss-monitoring-with-gogios.gmi b/gemfeed/DRAFT-kiss-monitoring-with-gogios.gmi index 55755f0f..fa2971ec 100644 --- a/gemfeed/DRAFT-kiss-monitoring-with-gogios.gmi +++ b/gemfeed/DRAFT-kiss-monitoring-with-gogios.gmi @@ -2,17 +2,19 @@ ## Introduction -Gogios is a minimalistic and easy-to-use monitoring tool, designed specifically for small-scale self-hosted servers and virtual machines. With compatibility for the Nagios Check API, Gogios offers a simple yet effective solution for users looking to monitor a limited number of resources. In this blog post, we'll explore Gogios' features, configuration, and use cases to help you determine if it's the right monitoring solution for your needs. +Gogios is a minimalistic and easy-to-use monitoring tool I programmed in Google Go designed specifically for small-scale self-hosted servers and virtual machines. The primary purpose of Gogios is to monitor my personal server infrastructure for `foo.zone`, my MTAs, my authoritative DNS servers, my NextCloud, Wallabag and Anki sync server installations, etc. + +With compatibility with the Nagios Check API, Gogios offers a simple yet effective solution for users looking to monitor a limited number of resources. In this blog post, we'll explore Gogios' features, configuration, and use cases to help you determine if it's the proper monitoring solution for your needs, too. => https://codeberg.org/snonux/gogios ## Motivation -As a Site Reliability Engineer with experience in various monitoring solutions like Nagios, Icinga, Prometheus and OpsGenie, I found that these tools often came with a plethora of features that I didn't necessarily need for personal use. Contact groups, host groups, re-check intervals, check clustering, and the requirement of operating a DBMS and a WebUI added complexity and bloat to my monitoring setup. +As a Site Reliability Engineer with experience in monitoring solutions like Nagios, Icinga, Prometheus and OpsGenie, these tools often came with many features that I didn't necessarily need. Contact groups, host groups, re-check intervals, check clustering, and the requirement of operating a DBMS and a WebUI added complexity and bloat to my monitoring setup. -My primary goal was to have a single email address for notifications and a simple mechanism to periodically execute standard Nagios check scripts and notify me of any state changes. I wanted the most minimalistic monitoring solution possible, but I wasn't satisfied with the available options. +My primary goal was to have a single email address for notifications and a simple mechanism to periodically execute standard Nagios check scripts and notify me of any state changes. I wanted the most minimalistic monitoring solution possible but wasn't satisfied with the available options. -This led me to create Gogios, a lightweight monitoring tool tailored to my specific needs. I chose the Go programming language for this project as it not only allowed me to refresh my Go programming skills but also provided a robust platform for developing a fast and efficient monitoring tool. +This led me to create Gogios, a lightweight monitoring tool tailored to my specific needs. I chose the Go programming language for this project as it allowed me to refresh my Go programming skills and provided a robust platform for developing a fast and efficient monitoring tool. Gogios eliminates unnecessary features and focuses on simplicity, providing a no-frills monitoring solution for small-scale self-hosted servers and virtual machines. The result is a tool that is easy to configure, set up, and maintain, ensuring that monitoring your resources is as hassle-free as possible. @@ -20,9 +22,32 @@ Gogios eliminates unnecessary features and focuses on simplicity, providing a no * Compatible with Nagios Check API: Gogios leverages the widely-used Nagios Check API, allowing you to use existing Nagios plugins for monitoring various services. * Lightweight and Minimalistic: Gogios is designed to be simple and easy to set up, making it an ideal choice for users with limited monitoring requirements. -* Configurable Check Timeout and Concurrency: Gogios allows you to set a timeout for checks and configure the number of concurrent checks, offering flexibility in how you monitor your resources. +* Configurable Check Timeout and Concurrency: Gogios allows you to set a timeout for checks and configure the number of concurrent checks, offering flexibility in monitoring your resources. +* Configurable check dependency: A check can depend on another check, which enables scenarios like not executing an HTTP check when the server isn't pingable. * Email Notifications: Gogios can send email notifications regarding the status of monitored services, ensuring you stay informed about potential issues. -* CRON-based Execution: Gogios can be easily scheduled to run periodically via CRON, allowing you to automate monitoring without the need for a complex setup. +* CRON-based Execution: Gogios can be quickly scheduled to run periodically via CRON, allowing you to automate monitoring without needing a complex setup. + +## Example alert + +This is an example alert report received via E-Mail. Whereas, `[C:2 W:0 U:0 OK:51]` means that we've got two alerts in status critical, 0 warnings, 0 unknowns and 51 OKs. + +``` +Subject: GOGIOS Report [C:2 W:0 U:0 OK:51] + +This is the recent Gogios report! + +# Alerts with status changed: + +OK->CRITICAL: Check ICMP4 vulcan.buetow.org: Check command timed out +OK->CRITICAL: Check ICMP6 vulcan.buetow.org: Check command timed out + +# Unhandled alerts: + +CRITICAL: Check ICMP4 vulcan.buetow.org: Check command timed out +CRITICAL: Check ICMP6 vulcan.buetow.org: Check command timed out + +Have a nice day! +``` ## Installation @@ -30,7 +55,7 @@ Gogios eliminates unnecessary features and focuses on simplicity, providing a no To compile and install Gogios on OpenBSD, follow these steps: -``` +```shell git clone https://codeberg.org/snonux/gogios.git cd gogios go build -o gogios cmd/gogios/main.go @@ -38,41 +63,41 @@ doas cp gogios /usr/local/bin/gogios doas chmod 755 /usr/local/bin/gogios ``` -Please note, depending on your operating system (e.g. Linux based), you may want to use `sudo` instead of `doas` and also change `/usr/local/bin` to a different path. If you want to compile Gogios for OpenBSD on a Linux system without installing the Go compiler on OpenBSD, you can use cross-compilation. Follow these steps: +This document is primarily written for OpenBSD, but applying the corresponding steps to any Unix-like (e.g. Linux-based) operating system should be easy. On systems other than OpenBSD, you may always have to replace `does` with the `sudo` command and replace the `/usr/local/bin` path with `/usr/bin`. -``` +You can use cross-compilation if you want to compile Gogios for OpenBSD on a Linux system without installing the Go compiler on OpenBSD. Follow these steps: + +```shell export GOOS=openbsd export GOARCH=amd64 go build -o gogios cmd/gogios/main.go ``` -On your OpenBSD system, copy the binary to `/usr/local/bin` and set the correct permissions as described in the previous section. I personally use Rexify, the friendly configuration management system, to automate the installation. +On your OpenBSD system, copy the binary to `/usr/local/bin` and set the correct permissions as described in the previous section. All steps described here you could automate with your configuration management system of choice. I use Rexify, the friendly configuration management system, to automate the installation, but that is out of the scope of this document. + +=> https://www.rexify.org ### Setting up user, group and directories -It is best to create a dedicated system user and group for Gogios to ensure proper isolation and security. The process of creating a user and group may vary depending on the operating system you're using. Here are the steps to create the `gogios` user and group under OpenBSD: +It is best to create a dedicated system user and group for Gogios to ensure proper isolation and security. Here are the steps to create the `_gogios` user and group under OpenBSD: -``` -sudo groupadd gogios -sudo useradd -g gogios -d /nonexistent -s /sbin/nologin -r gogios +```shell +doas adduser -group _gogios -batch _gogios +doas usermod -d /var/run/gogios _gogios +doas mkdir -p /var/run/gogios +doas chown _gogios:_gogios /var/run/gogios +doas chmod 750 /var/run/gogios ``` -Please note that the process of creating a user and group might differ depending on the operating system you are using. For other operating systems, consult their documentation for creating system users and groups. - -To set up the `StateDir` correctly with the correct permissions, follow these steps: - -``` -sudo mkdir -p /var/run/gogios -sudo chown gogios:gogios /var/run/gogios -sudo chmod 750 /var/run/gogios -``` +Please note that creating a user and group might differ depending on your operating system. For other operating systems, consult their documentation for creating system users and groups. ### Installing monitoring plugins -Gogios relies on external Nagios or Icinga monitoring plugin scripts. On OpenBSD, you can install the `monitoring-plugins` package to use with Gogios. The monitoring-plugins package is a collection of monitoring plugins, similar to Nagios plugins, that can be used to monitor various services and resources: +Gogios relies on external Nagios or Icinga monitoring plugin scripts. On OpenBSD, you can install the `monitoring-plugins` package with Gogios. The monitoring-plugins package is a collection of monitoring plugins, similar to Nagios plugins, that can be used to monitor various services and resources: -``` -pkg_add monitoring-plugins +```shell +doas pkg_add monitoring-plugins +doas pkg_add nrpe # If you want to execute checks remotely via NRPE. ``` Once the installation is complete, you can find the monitoring plugins in the `/usr/local/libexec/nagios` directory, which then can be configured to be used in `gogios.json`. @@ -81,21 +106,21 @@ Once the installation is complete, you can find the monitoring plugins in the `/ ### MTA -Gogios requires a local Mail Transfer Agent (MTA) such as Postfix or OpenBSD SMTPD running on the same server where the CRON job (see about the CRON job further below) is executed. The local MTA is responsible for handling email delivery, allowing Gogios to send out email notifications for monitoring status changes. Before using Gogios, ensure that you have a properly configured MTA installed and running on your server to facilitate the sending of emails. Once the MTA is set up and functioning correctly, Gogios can leverage it to send email notifications as needed. +Gogios requires a local Mail Transfer Agent (MTA) such as Postfix or OpenBSD SMTPD running on the same server where the CRON job (see about the CRON job further below) is executed. The local MTA handles email delivery, allowing Gogios to send email notifications to monitor status changes. Before using Gogios, ensure that you have a properly configured MTA installed and running on your server to facilitate the sending of emails. Once the MTA is set up and functioning correctly, Gogios can leverage it to send email notifications. -To send an email via the command line on OpenBSD, you can use the mail command. Here's an example of how to send a test email to ensure that your email server is working correctly: +You can use the mail command to send an email via the command line on OpenBSD. Here's an example of how to send a test email to ensure that your email server is working correctly: ``` -echo "This is a test email from OpenBSD." | mail -s "Test Email" your-email@example.com +echo 'This is a test email from OpenBSD.' | mail -s 'Test Email' your-email@example.com ``` -Check the recipient's inbox to confirm the delivery of the test email. If the email is delivered successfully, it indicates that your email server is properly configured and functioning. Please check your MTA logs in case of issues. +Check the recipient's inbox to confirm the delivery of the test email. If the email is delivered successfully, it indicates that your email server is configured correctly and functioning. Please check your MTA logs in case of issues. ### Configuring Gogios To configure Gogios, create a JSON configuration file (e.g., `/etc/gogios.json`). Here's a sample configuration: -``` +```json { "EmailTo": "paul@dev.buetow.org", "EmailFrom": "gogios@buetow.org", @@ -103,13 +128,27 @@ To configure Gogios, create a JSON configuration file (e.g., `/etc/gogios.json`) "CheckConcurrency": 2, "StateDir": "/var/run/gogios", "Checks": { + "Check ICMP4 www.foo.zone": { + "Plugin": "/usr/local/libexec/nagios/check_ping", + "Args": [ "-H", "www.foo.zone", "-4", "-w", "50,10%", "-c", "100,15%" ] + }, + "Check ICMP6 www.foo.zone": { + "Plugin": "/usr/local/libexec/nagios/check_ping", + "Args": [ "-H", "www.foo.zone", "-6", "-w", "50,10%", "-c", "100,15%" ] + }, "www.foo.zone HTTP IPv4": { "Plugin": "/usr/local/libexec/nagios/check_http", - "Args": ["www.foo.zone", "-4"] + "Args": ["www.foo.zone", "-4"], + "DependsOn": ["Check ICMP4 www.foo.zone"] }, "www.foo.zone HTTP IPv6": { "Plugin": "/usr/local/libexec/nagios/check_http", - "Args": ["www.foo.zone", "-6"] + "Args": ["www.foo.zone", "-6"], + "DependsOn": ["Check ICMP6 www.foo.zone"] + } + "Check NRPE Disk Usage foo.zone": { + "Plugin": "/usr/local/libexec/nagios/check_nrpe", + "Args": ["-H", "foo.zone", "-c", "check_disk", "-p", "5666", "-4"] } } } @@ -120,54 +159,44 @@ To configure Gogios, create a JSON configuration file (e.g., `/etc/gogios.json`) * `CheckTimeoutS`: Sets the timeout for checks in seconds. * `CheckConcurrency`: Determines the number of concurrent checks that can run simultaneously. * `StateDir`: Specifies the directory where Gogios stores its persistent state in a `state.json` file. -* `Checks`: Defines a list of checks to be performed, with each check having a unique name, plugin path, and arguments. +* `Checks`: Defines a list of checks to be performed, each with a unique name, plugin path, and arguments. Adjust the configuration file according to your needs, specifying the checks you want Gogios to perform. -The `state.json` file mentioned above keeps track of the monitoring state and check results between Gogios runs, enabling Gogios to only send email notifications when there are changes in the check status. +If you want to execute checks only when another check succeeded (status OK), use `DependsOn`. In the example above, the HTTP checks won't run when the hosts aren't pingable. They will show up as `UNKNOWN` in the report. -## Running Gogios +For remote checks, use the `check_nrpe` plugin. You also need to have the NRPE server set up correctly on the target host (out of scope for this document). -Now it is time to give it a first run. On OpenBSD, do: +The `state.json` file mentioned above keeps track of the monitoring state and check results between Gogios runs, enabling Gogios only to send email notifications when there are changes in the check status. -``` -doas -u gogios /usr/local/bin/gogios -cfg /etc/gogios.json -``` +## Running Gogios -and on Linux based systems, you likely would need to run: +Now it is time to give it a first run. On OpenBSD, do: -``` -sudo -u gogios /usr/bin/gogios -cfg /etc/gogios.json +```shell +doas -u _gogios /usr/local/bin/gogios -cfg /etc/gogios.json ``` To run Gogios via CRON on OpenBSD as the `gogios` user and check all services once per minute, follow these steps: -Type `doas crontab -e -u gogios` and press Enter to open the crontab file for the `gogios` user for editing and add the following line to the crontab file: +Type `doas crontab -e -u _gogios` and press Enter to open the crontab file for the `_gogios` user for editing and add the following lines to the crontab file: ``` -* * * * * /usr/local/bin/gogios -cfg /etc/gogios.json +*/5 8-22 * * * /usr/local/bin/gogios -cfg /etc/gogios.json +0 7 * * * /usr/local/bin/gogios -renotify -cfg /etc/gogios.json ``` -On Linux, please replace `doas` with `sudo` in the commands above. You may also need to adjust the path to the `gogios` binary. - -Gogios is now configured to run every minute via CRON as the `gogios` user, and it will execute the checks and send monitoring status via email according to your configuration. By running Gogios under the gogios user's crontab, you further enhance the isolation and security of the monitoring setup. +Gogios is now configured to run every five minutes from 8 am to 10 pm via CRON as the `_gogios` user. It will execute the checks and send monitoring status whenever a check status changes via email according to your configuration. Also, Gogios will run once at 7 am every morning and re-notify all unhandled alerts as a reminder. ### High-availability To create a high-availability Gogios setup, you can install Gogios on two servers that will monitor each other using the NRPE (Nagios Remote Plugin Executor) plugin. By running Gogios in alternate cron intervals on both servers, you can ensure that even if one server goes down, the other will continue monitoring your infrastructure and sending notifications. * Install Gogios on both servers following the compilation and installation instructions provided earlier. -* Install the NRPE server and plugin on both servers. This plugin allows you to execute Nagios check scripts on remote hosts. +* Install the NRPE server (out of scope for this document) and plugin on both servers. This plugin allows you to execute Nagios check scripts on remote hosts. * Configure Gogios on both servers to monitor each other using the NRPE plugin. Add a check to the Gogios configuration file (`/etc/gogios.json`) on both servers that uses the NRPE plugin to execute a check script on the other server. For example, if you have Server A and Server B, the configuration on Server A should include a check for Server B, and vice versa. -* Set up alternate cron intervals on both servers. Configure the cron job on Server A to run Gogios at odd minutes (e.g., 1, 3, 5, ...), and on Server B to run at even minutes (e.g., 0, 2, 4, ...). This will ensure that if one server goes down, the other server will continue monitoring and sending notifications. - -## Use Cases: - -Gogios is ideal for monitoring small-scale self-hosted servers and virtual machines, particularly in cases where only a handful of resources need to be monitored. Some example use cases include: - -* Monitoring web server availability and response times for a small website. -* Ensuring that database servers are up and running for a small self-hosted application. -* Monitoring the health and status of a few virtual machines running on a single host. +* Set up alternate cron intervals on both servers. Configure the cron job on Server A to run Gogios at minutes 0, 10, 20, ..., and on Server B to run at minutes 5, 15, 25, ... This will ensure that if one server goes down, the other server will continue monitoring and sending notifications. +* Gogios doesn't support clustering. So it means when both servers are up, unhandled alerts will be notified via E-Mail twice; from each server once. That's the trade-off for simplicity. ## Conclusion: -- cgit v1.2.3