diff options
| author | Paul Bütow <pbuetow@mimecast.com> | 2020-01-22 12:58:26 +0000 |
|---|---|---|
| committer | Paul Bütow <pbuetow@mimecast.com> | 2020-01-22 12:58:26 +0000 |
| commit | eee041d58738f17f97db4b4302ea77086ff8f5ac (patch) | |
| tree | 6ab18f911a01fa557ee8c1c6c1ee3304acbd4555 /doc | |
| parent | ff7ef579b887452de749eab1796b0ae455f7e875 (diff) | |
minor formatting fixes
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/examples.md | 10 | ||||
| -rw-r--r-- | doc/installation.md | 14 | ||||
| -rw-r--r-- | doc/quickstart.md | 24 |
3 files changed, 23 insertions, 25 deletions
diff --git a/doc/examples.md b/doc/examples.md index 78a9caf..959105c 100644 --- a/doc/examples.md +++ b/doc/examples.md @@ -10,7 +10,7 @@ This page demonstrate the basic usage of DTail. Please also see ``dtail --help`` The following example demonstrates how to follow logs of multiple servers at once. The server list is provided as a flat text file. The example filters all logs containing the string ``STAT``. Any other Go compatible regular expression can be used instead of ``STAT``. ```shell -workstation01 ~ % dtail --servers serverlist.txt --files "/var/log/service/*.log" --regex STAT +% dtail --servers serverlist.txt --files "/var/log/service/*.log" --regex STAT ```  @@ -20,7 +20,7 @@ workstation01 ~ % dtail --servers serverlist.txt --files "/var/log/service/*.log To run ad-hoc mapreduce aggregations on newly written log lines you also must add a query. This example follows all remote log lines and prints out every 5 seconds the top 10 servers with most average free memory according to the logs. To run a mapreduce query across log lines written in the past please use the ``dmap`` command instead. ```shell -workstation01 ~ % dtail --servers serverlist.txt \ +% dtail --servers serverlist.txt \ --query 'select avg(memfree), $hostname from MCVMSTATS group by $hostname order by avg(memfree) limit 10 interval 5' \ --files '/var/log/service/*.log' ``` @@ -34,7 +34,7 @@ In order for mapreduce queries to work you have to make sure that your log forma The following example demonstrates how to cat files (display the whole content of the files) of multiple servers at once. The servers are provided as a comma separated list this time. ```shell -workstation01 ~ % dcat --servers serv-011.lan.example.org,serv-012.lan.example.org,serv-013.lan.example.org \ +% dcat --servers serv-011.lan.example.org,serv-012.lan.example.org,serv-013.lan.example.org \ --files /etc/hostname ``` @@ -45,7 +45,7 @@ workstation01 ~ % dcat --servers serv-011.lan.example.org,serv-012.lan.example.o The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example we look after the swap partition in ``/etc/fstab``. We do that only on the first 20 servers from ``serverlist.txt``. ``dgrep`` is also very useful for searching log files of the past. ```shell -workstation01 ~ % dgrep --servers <(head -n 20 serverlist.txt) \ +% dgrep --servers <(head -n 20 serverlist.txt) \ --files /etc/fstab \ --regex swap ``` @@ -57,7 +57,7 @@ workstation01 ~ % dgrep --servers <(head -n 20 serverlist.txt) \ To run a mapreduce aggregation over logs written in the past the ``dmap`` command can be used. For example the following command aggregates all mapreduce fields of all the logs and calculates the average memory free grouped by day of the month, hour, minute and the server hostname. ``dmap`` will print interim results every few seconds. The final result however will be written to file ``mapreduce.csv``. ```shell -dmap --servers serv-011.lan.example.org,serv-012.lan.example.org,serv-013.lan.example.org,serv-021.lan.example.org,serv-022.lan.example.org,serv-023.lan.example.org \ +% dmap --servers serv-011.lan.example.org,serv-012.lan.example.org,serv-013.lan.example.org,serv-021.lan.example.org,serv-022.lan.example.org,serv-023.lan.example.org \ --query 'select avg(memfree), $day, $hour, $minute, $hostname from MCVMSTATS group by $day, $hour, $minute, $hostname order by avg(memfree) limit 10 outfile mapreduce.csv' \ --files "/var/log/service/*.log" ``` diff --git a/doc/installation.md b/doc/installation.md index e9c6d47..305eae5 100644 --- a/doc/installation.md +++ b/doc/installation.md @@ -15,22 +15,22 @@ It is recommended to automate all the installation process outlined here. You co 1. The ``dserver`` binary has to be installed on all machines (server boxes) involved. A good location for the binary would be ``/usr/local/bin/dserver`` with permissions set as follows: ```console -serv-001 ~ % sudo chown root:root /usr/local/bin/dserver -serv-001 ~ % sudo chmod 0755 /usr/local/bin/dserver +% sudo chown root:root /usr/local/bin/dserver +% sudo chmod 0755 /usr/local/bin/dserver ``` 2. Create the ``dserver`` run user and group. The user could look like this: ```console -serv-001 ~ % id dserver +% id dserver uid=670(dserver) gid=670(dserver) groups=670(dserver) ``` 3. Create the required file system structure and set the correct permissions: ```console -serv-001 ~ % sudo mkdir -p /etc/dserver /var/run/dserver -serv-001 ~ % sudo chown -R dserver:dserver /var/run/dserver +% sudo mkdir -p /etc/dserver /var/run/dserver +% sudo chown -R dserver:dserver /var/run/dserver ``` 4. Install the ``dtail.json`` config to ``/etc/dserver/dtail.json``. An example can be found [here](../samples/dtail.json.sample). @@ -42,8 +42,8 @@ serv-001 ~ % sudo chown -R dserver:dserver /var/run/dserver To start the DTail server via ``systemd`` run: ```console -serv-001 ~ % sudo systemctl start dserver -serv-001 ~ % sudo systemctl status dserver +% sudo systemctl start dserver +% sudo systemctl status dserver ● dserver.service - DTail server Loaded: loaded (/etc/systemd/system/dserver.service; disabled; vendor preset: disabled) Active: active (running) since Fri 2019-12-06 13:21:24 GMT; 2s ago diff --git a/doc/quickstart.md b/doc/quickstart.md index e7ca4b8..46f7fae 100644 --- a/doc/quickstart.md +++ b/doc/quickstart.md @@ -12,11 +12,11 @@ This guide also assumes that you know how to install and use a Go compiler and G To install all DTail binaries from github run: ```console -workstatoin01 ~ % go get github.com/mimecast/dtail/cmd/dtail -workstatoin01 ~ % go get github.com/mimecast/dtail/cmd/dcat -workstatoin01 ~ % go get github.com/mimecast/dtail/cmd/dgrep -workstatoin01 ~ % go get github.com/mimecast/dtail/cmd/dmap -workstatoin01 ~ % go get github.com/mimecast/dtail/cmd/dserver +% go get github.com/mimecast/dtail/cmd/dtail +% go get github.com/mimecast/dtail/cmd/dcat +% go get github.com/mimecast/dtail/cmd/dgrep +% go get github.com/mimecast/dtail/cmd/dmap +% go get github.com/mimecast/dtail/cmd/dserver ``` It produces the following executables in ``$GOPATH/bin``: @@ -32,7 +32,7 @@ It produces the following executables in ``$GOPATH/bin``: Copy the ``dserver`` binary to the remote server machines of your choice (e.g. ``serv-001.lan.example.org`` and ``serv-002.lan.example.org``) and start it on each of the servers as follows: ```console -serv-001 ~ % ./dserver +% ./dserver SERVER|serv-001|INFO|Launching server|server|DTail 1.0.0 SERVER|serv-001|INFO|Creating server|DTail 1.0.0 SERVER|serv-001|INFO|Generating private server RSA host key @@ -51,7 +51,7 @@ Make sure that your public SSH key is listed in ``~/.ssh/authorized_keys`` on al DTail utilises the SSH Agent for SSH authentication. This is to avoid entering the passphrase of the private SSH key over and over again when a new SSH session is initiated from the DTail client to a new DTail server. For this the private SSH key has to be registered at the SSH Agent: ```console -workstation01 ~ % ssh-add ~/.ssh/id_rsa +% ssh-add ~/.ssh/id_rsa Enter passphrase for ~/.ssh/id_rsa: ********** Identity added: ~/.ssh/id_rsa (~/.ssh/id_rsa) ``` @@ -61,10 +61,9 @@ The DTail client communicates with the SSH Agent through ``~/.ssh/ssh_auth_socke To test whether SSH is setup correctly you should be able to SSH into the servers with the OpenSSH client and your private SSH key through the SSH Agent without entering the private keys passphrase. The following assumes to have an OpenSSH server running on ``serv-001.lan.example.org`` and an OpenSSH client installed on your laptop or workstation. Please notice that DTail does not require to have an OpenSSH infrastructure set up but DTail uses by default the same public/private key file paths as OpenSSH. OpenSSH can be of a great help to verify that the SSH keys are configured correctly: ```console -workstation01 ~/git/dtail % ssh serv-001.lan.example.org -serv-001 ~ % -serv-001 ~ % exit -workstation01 ~/git/dtail % +% ssh serv-001.lan.example.org +% +% exit ``` ## Run DTail client @@ -72,7 +71,7 @@ workstation01 ~/git/dtail % Now it is time to connect to the DTail servers through the DTail client: ```console -workstation01 ~/git/dtail % ./bin/dtail --servers serv-001.lan.example.org,server-002.lan.example.org --files "/var/log/service/*.log" +% dtail --servers serv-001.lan.example.org,server-002.lan.example.org --files "/var/log/service/*.log" CLIENT|workstation01|INFO|Launching client|tail|DTail 1.0.0 CLIENT|workstation01|INFO|Initiating base client CLIENT|workstation01|INFO|Added SSH Agent to list of auth methods @@ -88,7 +87,6 @@ CLIENT|workstation01|INFO|stats|connected=1/1(100%)|new=0|rate=0.00/s|throttle=0 CLIENT|workstation01|INFO|stats|connected=1/1(100%)|new=0|rate=0.00/s|throttle=0|cpus/goroutines=8/17 . . -. ``` Have a look [here](examples.md) for more usage examples. |
