diff options
Diffstat (limited to 'gemfeed/DRAFT-dtail-usage-examples.html')
| -rw-r--r-- | gemfeed/DRAFT-dtail-usage-examples.html | 286 |
1 files changed, 0 insertions, 286 deletions
diff --git a/gemfeed/DRAFT-dtail-usage-examples.html b/gemfeed/DRAFT-dtail-usage-examples.html deleted file mode 100644 index f82a37a1..00000000 --- a/gemfeed/DRAFT-dtail-usage-examples.html +++ /dev/null @@ -1,286 +0,0 @@ -<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> -<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"> -<head> -<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> -<title>DTail usage examples</title> -<link rel="shortcut icon" type="image/gif" href="/favicon.ico" /> -<link rel="stylesheet" href="../style.css" /> -<link rel="stylesheet" href="style-override.css" /> -</head> -<body> -<h1 style='display: inline'>DTail usage examples</h1><br /> -<br /> -<span>DTail is a distributed DevOps tool for tailing, grepping, catting logs and other text files on many remote machines at once which I programmed in Go.</span><br /> -<br /> -<a class='textlink' href='https://dtail.dev'>https://dtail.dev</a><br /> -<br /> -<pre> - ,_---~~~~~----._ - _,,_,*^____ _____``*g*\"*, - ____ _____ _ _ / __/ /' ^. / \ ^@q f - | _ \_ _|_ _(_) | @f | ((@| |@)) l 0 _/ - | | | || |/ _` | | | \`/ \~____ / __ \_____/ \ - | |_| || | (_| | | | | _l__l_ I - |____/ |_|\__,_|_|_| } [______] I - ] | | | | - ] ~ ~ | - | Let's tail those logs! | - | | -</pre> -<br /> -<span>DTail consists out of a server and several client binaries. In this post, I am showcasing their use!</span><br /> -<br /> -<h2 style='display: inline'>Following logs</h2><br /> -<br /> -<span>The following example demonstrates how to follow logs of several servers at once. The server list is provided as a flat text file. The example filters all records containing the string <span class='inlinecode'>INFO</span>. Any other Go compatible regular expression can also be used instead of <span class='inlinecode'>INFO</span>.</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dtail --servers serverlist<font color="#990000">.</font>txt --grep INFO --files <font color="#FF0000">"/var/log/dserver/*.log"</font> -</pre> -<br /> -<span>Hint: you can also provide a comma separated server list, e.g.: <span class='inlinecode'>servers server1.example.org,server2.example.org:PORT,...</span></span><br /> -<br /> -<a href='./dtail-usage-examples/dtail.gif'><img alt='Tail example' title='Tail example' src='./dtail-usage-examples/dtail.gif' /></a><br /> -<br /> -<span class='quote'>Hint: You can also use the shorthand version (omitting the <span class='inlinecode'>--files</span>)</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dtail --servers serverlist<font color="#990000">.</font>txt --grep INFO <font color="#FF0000">"/var/log/dserver/*.log"</font> -</pre> -<br /> -<h2 style='display: inline'>Aggregating logs</h2><br /> -<br /> -<span>To run ad-hoc map-reduce aggregations on newly written log lines you must add a query. The following example follows all remote log lines and prints out every few seconds the result to standard output.</span><br /> -<br /> -<span class='quote'>Hint: To run a map-reduce query across log lines written in the past, please use the <span class='inlinecode'>dmap</span> command instead.</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dtail --servers serverlist<font color="#990000">.</font>txt <font color="#990000">\</font> - --files <font color="#FF0000">'/var/log/dserver/*.log'</font> <font color="#990000">\</font> - --query <font color="#FF0000">'from STATS select sum($goroutines),sum($cgocalls),</font> -<font color="#FF0000"> last($time),max(lifetimeConnections)'</font> -</pre> -<br /> -<span>Beware: For map-reduce queries to work, you have to ensure that DTail supports your log format. Check out the documentaiton of the DTail query language and the DTail log formats on the DTail homepage for more information.</span><br /> -<br /> -<a href='./dtail-usage-examples/dtail-map.gif'><img alt='Tail map-reduce example' title='Tail map-reduce example' src='./dtail-usage-examples/dtail-map.gif' /></a><br /> -<br /> -<span class='quote'>Hint: You can also use the shorthand version:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dtail --servers serverlist<font color="#990000">.</font>txt <font color="#990000">\</font> - --files <font color="#FF0000">'/var/log/dserver/*.log'</font> <font color="#990000">\</font> - <font color="#FF0000">'from STATS select sum($goroutines),sum($cgocalls),</font> -<font color="#FF0000"> last($time),max(lifetimeConnections)'</font> -</pre> -<br /> -<span>Here is another example:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dtail --servers serverlist<font color="#990000">.</font>txt <font color="#990000">\</font> - --files <font color="#FF0000">'/var/log/dserver/*.log'</font> <font color="#990000">\</font> - --query <font color="#FF0000">'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,</font> -<font color="#FF0000"> lifetimeConnections group by $hostname order by max($cgocalls)'</font> -</pre> -<br /> -<a href='./dtail-usage-examples/dtail-map2.gif'><img alt='Tail map-reduce example 2' title='Tail map-reduce example 2' src='./dtail-usage-examples/dtail-map2.gif' /></a><br /> -<br /> -<span>You can also continuously append the results to a CSV file by adding <span class='inlinecode'>outfile append filename.csv</span> to the query:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dtail --servers serverlist<font color="#990000">.</font>txt <font color="#990000">\</font> - --files <font color="#FF0000">'/var/log/dserver/*.log'</font> <font color="#990000">\</font> - --query <font color="#FF0000">'from STATS select ... outfile append result.csv'</font> -</pre> -<br /> -<h2 style='display: inline'>How to use <span class='inlinecode'>dcat</span></h2><br /> -<br /> -<span>The following example demonstrates how to cat files (display the full content of the files) on several servers at once.</span><br /> -<br /> -<span>As you can see in this example, a DTail client also creates a local log file of all received data in <span class='inlinecode'>~/log</span>. You can also use the <span class='inlinecode'>noColor</span> and <span class='inlinecode'>-plain</span> flags (this all also work with other DTail commands than <span class='inlinecode'>dcat</span>).</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dcat --servers serverlist<font color="#990000">.</font>txt --files /etc/hostname -</pre> -<br /> -<a href='./dtail-usage-examples/dcat.gif'><img alt='Cat example' title='Cat example' src='./dtail-usage-examples/dcat.gif' /></a><br /> -<br /> -<span class='quote'>Hint: You can also use the shorthand version:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dcat --servers serverlist<font color="#990000">.</font>txt /etc/hostname -</pre> -<br /> -<h2 style='display: inline'>How to use <span class='inlinecode'>dgrep</span></h2><br /> -<br /> -<span>The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after some entries in <span class='inlinecode'>/etc/passwd</span>. This time, we don't provide the server list via an file but rather via a comma separated list directly on the command line. We also explore the <span class='inlinecode'>-before</span>, <span class='inlinecode'>-after</span> and <span class='inlinecode'>-max</span> flags (see animation).</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dgrep --servers server1<font color="#990000">.</font>example<font color="#990000">.</font>org<font color="#990000">:</font><font color="#993399">2223</font> <font color="#990000">\</font> - --files /etc/passwd <font color="#990000">\</font> - --regex nologin -</pre> -<br /> -<span>Generally, <span class='inlinecode'>dgrep</span> is also a very useful way to search historic application logs for certain content.</span><br /> -<br /> -<a href='./dtail-usage-examples/dgrep.gif'><img alt='Grep example' title='Grep example' src='./dtail-usage-examples/dgrep.gif' /></a><br /> -<br /> -<span>Hint: <span class='inlinecode'>-regex</span> is an alias for <span class='inlinecode'>-grep</span>.</span><br /> -<br /> -<h2 style='display: inline'>How to use <span class='inlinecode'>dmap</span></h2><br /> -<br /> -<span>To run a map-reduce aggregation over logs written in the past, the <span class='inlinecode'>dmap</span> command can be used. The following example aggregates all map-reduce fields <span class='inlinecode'>dmap</span> will print interim results every few seconds. You can also write the result to an CSV file by adding <span class='inlinecode'>outfile result.csv</span> to the query.</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dmap --servers serverlist<font color="#990000">.</font>txt <font color="#990000">\</font> - --files <font color="#FF0000">'/var/log/dserver/*.log'</font> <font color="#990000">\</font> - --query <font color="#FF0000">'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,</font> -<font color="#FF0000"> lifetimeConnections group by $hostname order by max($cgocalls)'</font> -</pre> -<br /> -<span>Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in <span class='inlinecode'>internal/mapr/logformat</span> or add an extension to support a custom log format. The example here works out of the box though, as DTail understands its own log format already. </span><br /> -<br /> -<a href='./dtail-usage-examples/dmap.gif'><img alt='DMap example' title='DMap example' src='./dtail-usage-examples/dmap.gif' /></a><br /> -<br /> -<h2 style='display: inline'>How to use the DTail serverless mode</h2><br /> -<br /> -<span>Until now, all examples so far required to have remote server(s) to connect to. That makes sense, as after all DTail is a *distributed* tool. However, there are circumstances where you don't really need to connect to a server remotely. For example, you already have a login shell open to the server an all what you want is to run some queries directly on local log files.</span><br /> -<br /> -<span>The serverless mode does not require any <span class='inlinecode'>dserver</span> up and running and therefore there is no networking/SSH involved. </span><br /> -<br /> -<span>All commands shown so far also work in a serverless mode. All what needs to be done is to omit a server list. The DTail client then starts in serverless mode.</span><br /> -<br /> -<h3 style='display: inline'>Serverless map-reduce query</h3><br /> -<br /> -<span>The following <span class='inlinecode'>dmap</span> example is the same as the previously shown one, but the difference is that it operates on a local log file directly:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dmap --files /var/log/dserver/dserver<font color="#990000">.</font>log - --query <font color="#FF0000">'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,</font> -<font color="#FF0000"> lifetimeConnections group by $hostname order by max($cgocalls)'</font> -</pre> -<br /> -<span>As a shorthand version the following command can be used:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dmap <font color="#FF0000">'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,</font> -<font color="#FF0000">lifetimeConnections group by $hostname order by max($cgocalls)'</font> /var/log/dsever/dserver<font color="#990000">.</font>log -</pre> -<br /> -<span>You can also use a file input pipe as follows:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> cat /var/log/dserver/dserver<font color="#990000">.</font>log <font color="#990000">|</font> <font color="#990000">\</font> - dmap <font color="#FF0000">'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,</font> -<font color="#FF0000"> lifetimeConnections group by $hostname order by max($cgocalls)'</font> -</pre> -<br /> -<h3 style='display: inline'>Aggregating CSV files</h3><br /> -<br /> -<span>In essence, this works exactly like aggregating logs. All files operated on must be valid CSV files and the first line of the CSV must be the header. E.g.:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> cat example<font color="#990000">.</font>csv -name<font color="#990000">,</font>lastname<font color="#990000">,</font>age<font color="#990000">,</font>profession -Michael<font color="#990000">,</font>Jordan<font color="#990000">,</font><font color="#993399">40</font><font color="#990000">,</font>Basketball player -Michael<font color="#990000">,</font>Jackson<font color="#990000">,</font><font color="#993399">100</font><font color="#990000">,</font>Singer -Albert<font color="#990000">,</font>Einstein<font color="#990000">,</font><font color="#993399">200</font><font color="#990000">,</font>Physician -<font color="#990000">%</font> dmap --query <font color="#FF0000">'select lastname,name where age > 40 logformat csv outfile result.csv'</font> example<font color="#990000">.</font>csv -<font color="#990000">%</font> cat result<font color="#990000">.</font>csv -lastname<font color="#990000">,</font>name -Jackson<font color="#990000">,</font>Michael -Einstein<font color="#990000">,</font>Albert -</pre> -<br /> -<span>DMap can also be used to query and aggregate CSV files from remote servers.</span><br /> -<br /> -<h3 style='display: inline'>Other serverless commands</h3><br /> -<br /> -<span>The serverless mode works transparently with all other DTail commands. Here are some examples:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dtail /var/log/dserver/dserver<font color="#990000">.</font>log -</pre> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dtail --logLevel trace /var/log/dserver/dserver<font color="#990000">.</font>log -</pre> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dcat /etc/passwd -</pre> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dcat --plain /etc/passwd <font color="#990000">></font> /etc/test -<i><font color="#9A1900"># Should show no differences.</font></i> -diff /etc/test /etc/passwd -</pre> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><font color="#990000">%</font> dgrep --regex ERROR --files /var/log/dserver/dsever<font color="#990000">.</font>log -</pre> -<br /> -<p class="footer"> -Generated by <a href="https://codeberg.org/snonux/gemtexter">Gemtexter 2.1.0-release</a> | -served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> | -<a href="https://www.foo.zone/site-mirrors.html">Site Mirrors</a> -</p> -</body> -</html> |