Release of DTail v1.0.0v1.0.0

10 files changed, 249 insertions, 0 deletions

diff --git a/doc/dcat.gif b/doc/dcat.gif
new file mode 100644
index 0000000..a6eae0f
--- /dev/null
+++ b/doc/dcat.gif
diff --git a/doc/dgrep.gif b/doc/dgrep.gif
new file mode 100644
index 0000000..e2f2ac6
--- /dev/null
+++ b/doc/dgrep.gif
diff --git a/doc/dmap.gif b/doc/dmap.gif
new file mode 100644
index 0000000..eb75e8d
--- /dev/null
+++ b/doc/dmap.gif
diff --git a/doc/dtail-map.gif b/doc/dtail-map.gif
new file mode 100644
index 0000000..c1a74fe
--- /dev/null
+++ b/doc/dtail-map.gif
diff --git a/doc/dtail.gif b/doc/dtail.gif
new file mode 100644
index 0000000..8f6b56b
--- /dev/null
+++ b/doc/dtail.gif
diff --git a/doc/examples.md b/doc/examples.md
new file mode 100644
index 0000000..78a9caf
--- /dev/null
+++ b/doc/examples.md
@@ -0,0 +1,67 @@
+Examples
+========
+
+This page demonstrate the basic usage of DTail. Please also see ``dtail --help`` for more available options.
+
+# How to use ``dtail``
+
+## Tailing logs
+
+The following example demonstrates how to follow logs of multiple servers at once. The server list is provided as a flat text file. The example filters all logs containing the string ``STAT``. Any other Go compatible regular expression can be used instead of ``STAT``.
+
+```shell
+workstation01 ~ % dtail --servers serverlist.txt --files "/var/log/service/*.log" --regex STAT
+```
+
+![dtail](dtail.gif "Tail example")
+
+## Aggregating logs
+
+To run ad-hoc mapreduce aggregations on newly written log lines you also must add a query. This example follows all remote log lines and prints out every 5 seconds the top 10 servers with most average free memory according to the logs. To run a mapreduce query across log lines written in the past please use the ``dmap`` command instead.
+
+```shell
+workstation01 ~ % dtail --servers serverlist.txt  \
+    --query 'select avg(memfree), $hostname from MCVMSTATS group by $hostname order by avg(memfree) limit 10 interval 5' \
+    --files '/var/log/service/*.log'
+```
+
+In order for mapreduce queries to work you have to make sure that your log format is supported by DTail. You can either use the ones which are already defined in ``mapr/logformat`` or add an extension to support a custom log format.
+
+![dtail-map](dtail-map.gif "Tail mapreduce example")
+
+# How to use ``dcat``
+
+The following example demonstrates how to cat files (display the whole content of the files) of multiple servers at once. The servers are provided as a comma separated list this time.
+
+```shell
+workstation01 ~ % dcat --servers serv-011.lan.example.org,serv-012.lan.example.org,serv-013.lan.example.org \
+    --files /etc/hostname
+```
+
+![dcat](dcat.gif "Cat example")
+
+# How to use ``dgrep``
+
+The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example we look after the swap partition in ``/etc/fstab``. We do that only on the first 20 servers from ``serverlist.txt``. ``dgrep`` is also very useful for searching log files of the past.
+
+```shell
+workstation01 ~ % dgrep --servers <(head -n 20 serverlist.txt) \
+    --files /etc/fstab \
+    --regex swap
+```
+
+![dgrep](dgrep.gif "Grep example")
+
+# How to use ``dmap``
+
+To run a mapreduce aggregation over logs written in the past the ``dmap`` command can be used. For example the following command aggregates all mapreduce fields of all the logs and calculates the average memory free grouped by day of the month, hour, minute and the server hostname. ``dmap`` will print interim results every few seconds. The final result however will be written to file ``mapreduce.csv``.
+
+```shell
+dmap --servers serv-011.lan.example.org,serv-012.lan.example.org,serv-013.lan.example.org,serv-021.lan.example.org,serv-022.lan.example.org,serv-023.lan.example.org \
+    --query 'select avg(memfree), $day, $hour, $minute, $hostname from MCVMSTATS group by $day, $hour, $minute, $hostname order by avg(memfree) limit 10 outfile mapreduce.csv' \
+    --files "/var/log/service/*.log"
+```
+
+Remember: In order for that to work you have to make sure that your log format is supported by DTail. You can either use the ones which are already defined in ``mapr/logformat`` or add an extension to support a custom log format.
+
+![dmap](dmap.gif "DMap example")
diff --git a/doc/installation.md b/doc/installation.md
new file mode 100644
index 0000000..d1ecf9f
--- /dev/null
+++ b/doc/installation.md
@@ -0,0 +1,83 @@
+DTail Installation Guide
+========================
+
+The following installation guide has been tested successfully on CentOS 7. You may need to adjust accordingly depending on the distribution you use.
+
+This guide also assumes that you know how to use ``systemd`` and how to configure a service there. If you are unsure please consult the documentation of your distribution.
+
+This guide also assumes that you know how to add a new Nagios check to your monitoring infrastructure.
+
+# Compile it
+
+Please check the [Quick Starting Guide](quickstart.md) for instructions how to compile DTail. It is recommended to automate the build process via your build pipeline (e.g. produce a deployable RPM via Jenkins). But that is out of scope of this documentation.
+
+# Install it
+
+It is recommended to automate all the installation process outlined here. You could use a configuration management system such as Puppet, Chef or Ansible. However, that relies heavily on how your infrastructure is managed and is out of scope of this documentation.
+1. The ``dserver`` binary has to be installed on all machines (server boxes) involved. A good location for the binary would be ``/usr/local/bin/dserver`` with permissions set as follows:
+
+```console
+serv-001 ~ % sudo chown root:root /usr/local/bin/dserver
+serv-001 ~ % sudo chmod 0755 /usr/local/bin/dserver
+```
+
+2. Create the ``dserver`` run user and group. The user could look like this:
+
+```console
+serv-001 ~ % id dserver
+uid=670(dserver) gid=670(dserver) groups=670(dserver)
+```
+
+3. Create the required file system structure and set the correct permissions:
+
+```console
+serv-001 ~ % sudo mkdir -p /etc/dserver /var/run/dserver
+serv-001 ~ % sudo chown -R dserver:dserver /var/run/dserver
+```
+
+4. Install the ``dtail.json`` config to ``/etc/dserver/dtail.json``. An example can be found [here](../samples/dtail.json.sample).
+
+5. It is recommended to configure DTail server as a service to ``systemd``. An example unit file for ``systemd`` can be found [here](../samples/dserver.service.sample).
+
+# Start it
+
+To start the DTail server via ``systemd`` run:
+
+```console
+serv-001 ~ % sudo systemctl start dserver
+serv-001 ~ % sudo systemctl status dserver
+● dserver.service - DTail server
+   Loaded: loaded (/etc/systemd/system/dserver.service; disabled; vendor preset: disabled)
+   Active: active (running) since Fri 2019-12-06 13:21:24 GMT; 2s ago
+   Main PID: 12296 (dserver)
+   Memory: 1.5M
+   CGroup: /dserver.slice/dserver.service
+     └─12296 /usr/local/bin/dserver -cfg /etc/dserver/dtail.json
+
+    Dec 06 13:21:24 serv-001.lan.example.org systemd[1]: Started DTail server.
+    Dec 06 13:21:24 serv-001.lan.example.org dserver[12296]: SERVER|serv-001|INFO|Launching server|server|DTail 1.0.0
+    Dec 06 13:21:24 serv-001.lan.example.org dserver[12296]: SERVER|serv-001|INFO|Creating server|DTail 1.0.0
+    Dec 06 13:21:24 serv-001.lan.example.org dserver[12296]: SERVER|serv-001|INFO|Reading private server RSA host key from file|cache/ssh_host_key
+    Dec 06 13:21:24 serv-001.lan.example.org dserver[12296]: SERVER|serv-001|INFO|Starting server
+    Dec 06 13:21:24 serv-001.lan.example.org dserver[12296]: SERVER|serv-001|INFO|Binding server|1.2.3.4:2222
+```
+
+# Register SSH public keys in DTail server
+
+The DTail server now runs as a ``systemd`` service under system user ``dserver``. The system user ``dserver`` however has no permissions to read the SSH public keys from ``/home/USER/.ssh/authorized_keys``. Therefore, no user would be able to establish a SSH session to DTail server. As an alternative path DTail server also checks for public SSH key files in ``/var/run/dserver/cache/USER.authorized_keys``.
+
+It is recommended to execute [update_key_cache.sh](../samples/update_key_cache.sh.sample) periodically to update the key cache. In case you manage your public SSH keys via Puppet you could subscribe the script to corresponding module. Or alternatively just configure a cron job to run every once in a while.
+
+# Run DTail client
+
+Now you should be able to use DTail client like outlined in the [Quick Starting Guide](quickstart.md). Also have a look at the [Examples](examples.md).
+
+# Monitor it
+
+To verify that DTail server is up and running and functioning as expected  you should configure the Nagios check [check_dserver.sh](../samples/check_dserver.sh.sample) in your monitoring system. The check has to be executed locally on the server (e.g. via NRPE). How to configure the monitoring system in detail is out of scope of this guide, as it depends on the monitoring infrastructure used.
+
+```console
+% ./check_dserver.sh
+OK: DTail SSH Server seems fine
+```
+
diff --git a/doc/logo.png b/doc/logo.png
new file mode 100644
index 0000000..38edb16
--- /dev/null
+++ b/doc/logo.png
diff --git a/doc/logo.webp b/doc/logo.webp
new file mode 100644
index 0000000..5160d5b
--- /dev/null
+++ b/doc/logo.webp
diff --git a/doc/quickstart.md b/doc/quickstart.md
new file mode 100644
index 0000000..57432c0
--- /dev/null
+++ b/doc/quickstart.md
@@ -0,0 +1,99 @@
+Quick Starting Guide
+====================
+
+This is the quick starting guide. For a more sustainable setup, involving how to create a background service via ``systemd``, recommendations about automation via Jenkins and/or Puppet and health monitoring via Nagios please also follow the [Installation Guide](installation.md).
+
+This guide assumes that you know how to generate and configure a public/private SSH key pair for secure authorization and shell access. That is out of scope of this guide. For more information please have a look at the OpenSSH documentation of your distribution.
+
+This guide also assumes that you know how to install and use a Go compiler and GNU make.
+
+# Compile it
+
+To produce all DTail binaries run ``make``:
+
+```console
+workstation01 ~/git/dtail % make
+go build
+cp -pv ./dtail ./dcat
+./dtail -> ./dcat
+cp -pv ./dtail ./dgrep
+./dtail -> ./dgrep
+cp -pv ./dtail ./dmap
+./dtail -> ./dmap
+cp -pv ./dtail ./dserver
+./dtail -> ./dserver
+```
+
+It produces the following executables:
+
+* ``dserver``: The DTail server
+* ``dtail``: Client for tailing/following log files remotely (distributed tail)
+* ``dcat``: Client for displaying whole files remotely (distributed cat)
+* ``dgrep``: Client for searching whole files files remotely using a regex (distributed grep)
+* ``dmap``: Client for executing distributed mapreduce queries (may will consume a lot of RAM and CPU)
+
+# Start DTail server
+
+Copy the ``dserver`` binary to the remote server machines of your choice (e.g. ``serv-001.lan.example.org`` and ``serv-002.lan.example.org``) and start it on each of the servers as follows:
+
+```console
+serv-001 ~ % ./dserver
+SERVER|serv-001|INFO|Launching server|server|DTail 1.0.0
+SERVER|serv-001|INFO|Creating server|DTail 1.0.0
+SERVER|serv-001|INFO|Generating private server RSA host key
+SERVER|serv-001|INFO|Starting server
+SERVER|serv-001|INFO|Binding server|0.0.0.0:2222
+```
+
+``dserver`` is now listening on TCP port 2222 and waiting for incoming connections. All SSH keys listed in ``~/.ssh/authorized_keys`` are now respected by the DTail server for authorization.
+
+# Setup DTail client
+
+## Setup SSH
+
+Make sure that your public SSH key is listed in ``~/.ssh/authorized_keys`` on all server machines involved. The private SSH key counterpart should preferably stay on your Laptop or workstation in ``~/.ssh/id_rsa`` or ``~/.ssh/id_dsa``.
+
+DTail utilises the SSH Agent for SSH authentication. This is to avoid entering the passphrase of the private SSH key over and over again when a new SSH session is initiated from the DTail client to a new DTail server. For this the private SSH key has to be registered at the SSH Agent:
+
+```console
+workstation01 ~ % ssh-add ~/.ssh/id_rsa
+Enter passphrase for ~/.ssh/id_rsa: **********
+Identity added: ~/.ssh/id_rsa (~/.ssh/id_rsa)
+```
+
+The DTail client communicates with the SSH Agent through ``~/.ssh/ssh_auth_socket`` whenever a new session to a DTail server is established.
+
+To test whether SSH is setup correctly you should be able to SSH into the servers with the OpenSSH client and your private SSH key through the SSH Agent without entering the private keys passphrase. The following assumes to have an OpenSSH server running on ``serv-001.lan.example.org`` and an OpenSSH client installed on your laptop or workstation. Please notice that DTail does not require to have an OpenSSH infrastructure set up but DTail uses by default the same public/private key file paths as OpenSSH. OpenSSH can be of a great help to verify that the SSH keys are configured correctly:
+
+```console
+workstation01 ~/git/dtail % ssh serv-001.lan.example.org
+serv-001 ~ %
+serv-001 ~ % exit
+workstation01 ~/git/dtail %
+```
+
+## Run DTail client
+
+Now it is time to connect to the DTail servers through the DTail client:
+
+```console
+workstation01 ~/git/dtail % ./bin/dtail --servers serv-001.lan.example.org,server-002.lan.example.org --files "/var/log/service/*.log"
+CLIENT|workstation01|INFO|Launching client|tail|DTail 1.0.0
+CLIENT|workstation01|INFO|Initiating base client
+CLIENT|workstation01|INFO|Added SSH Agent to list of auth methods
+CLIENT|workstation01|INFO|Deduped server list|1|1
+CLIENT|workstation01|WARN|Encountered unknown host|{serv-002.lan.example.org:2222 0xc000146450 0xc00014a2f0 [serv-002.lan.example.org]:2222 ssh-rsa AAAA....
+CLIENT|workstation01|WARN|Encountered unknown host|{serv-001.lan.example.org:2222 0xc0001ff450 0xc00ee4a2f0 [serv-001.lan.example.org]:2222 ssh-rsa AAAA....
+Encountered 2 unknown hosts: 'serv-002.lan.example.org:2222 serv-001.lan.example.org:2222'
+Do you want to trust these hosts?? (y=yes,a=all,n=no,d=details): y
+CLIENT|workstation01|INFO|Added hosts to known hosts file|~/.ssh/known_hosts
+CLIENT|workstation01|INFO|stats|connected=1/1(100%)|new=1|rate=0.20/s|throttle=0|cpus/goroutines=8/17
+CLIENT|workstation01|INFO|stats|connected=1/1(100%)|new=0|rate=0.00/s|throttle=0|cpus/goroutines=8/17
+CLIENT|workstation01|INFO|stats|connected=1/1(100%)|new=0|rate=0.00/s|throttle=0|cpus/goroutines=8/17
+CLIENT|workstation01|INFO|stats|connected=1/1(100%)|new=0|rate=0.00/s|throttle=0|cpus/goroutines=8/17
+.
+.
+.
+```
+
+Have a look [here](examples.md) for more usage examples.