From 76f252524871b203ac11279511ea0adcf8b1386c Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Fri, 21 Feb 2025 11:09:28 +0200 Subject: Update content for html --- gemfeed/2023-09-25-dtail-usage-examples.html | 158 +++++++++++++++------------ 1 file changed, 89 insertions(+), 69 deletions(-) (limited to 'gemfeed/2023-09-25-dtail-usage-examples.html') diff --git a/gemfeed/2023-09-25-dtail-usage-examples.html b/gemfeed/2023-09-25-dtail-usage-examples.html index 220ee174..d210c8a4 100644 --- a/gemfeed/2023-09-25-dtail-usage-examples.html +++ b/gemfeed/2023-09-25-dtail-usage-examples.html @@ -8,7 +8,10 @@ -

DTail usage examples

+ +

DTail usage examples

Published at 2023-09-25T14:57:42+03:00

@@ -32,6 +35,23 @@ | |
+

+
+

DTail usage examples
⇢ Commands
⇢ Following logs
⇢ Aggregating logs
⇢ How to use dcat
⇢ How to use dgrep
⇢ How to use dmap
⇢ How to use the DTail serverless mode
⇢ ⇢ Serverless map-reduce query
⇢ ⇢ Aggregating CSV files
⇢ ⇢ Other serverless commands

Commands

+
DTail consists out of a server and several client binaries. In this post, I am showcasing their use!

Use dmap to aggregate logs and other text files already written
dserver is the DTail server, where all the clients can connect to

Following logs

The following example demonstrates how to follow logs of several servers at once. The server list is provided as a flat text file. The example filters all records containing the string INFO. Any other Go compatible regular expression can also be used instead of INFO.

@@ -50,7 +70,7 @@ by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -

% dtail --servers serverlist.txt --grep INFO --files "/var/log/dserver/*.log"
+% dtail --servers serverlist.txt --grep INFO --files "/var/log/dserver/*.log"
 
 

 Hint: you can also provide a comma separated server list, e.g.: servers server1.example.org,server2.example.org:PORT,...

@@ -63,10 +83,10 @@ http://www.gnu.org/software/src-highlite -->
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-% dtail --servers serverlist.txt --grep INFO "/var/log/dserver/*.log"
+% dtail --servers serverlist.txt --grep INFO "/var/log/dserver/*.log"
 
 

-Aggregating logs


+Aggregating logs


 

 To run ad-hoc map-reduce aggregations on newly written log lines you must add a query. The following example follows all remote log lines and prints out every few seconds the result to standard output.

 

@@ -76,10 +96,10 @@ http://www.gnu.org/software/src-highlite -->
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-% dtail --servers serverlist.txt \
-    --files '/var/log/dserver/*.log' \
-    --query 'from STATS select sum($goroutines),sum($cgocalls),
-             last($time),max(lifetimeConnections)'
+% dtail --servers serverlist.txt \
+    --files '/var/log/dserver/*.log' \
+    --query 'from STATS select sum($goroutines),sum($cgocalls),
+             last($time),max(lifetimeConnections)'
 
 

 Beware: For map-reduce queries to work, you have to ensure that DTail supports your log format. Check out the documentaiton of the DTail query language and the DTail log formats on the DTail homepage for more information.

@@ -92,10 +112,10 @@ http://www.gnu.org/software/src-highlite -->
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-% dtail --servers serverlist.txt \
-    --files '/var/log/dserver/*.log' \
-    'from STATS select sum($goroutines),sum($cgocalls),
-     last($time),max(lifetimeConnections)'
+% dtail --servers serverlist.txt \
+    --files '/var/log/dserver/*.log' \
+    'from STATS select sum($goroutines),sum($cgocalls),
+     last($time),max(lifetimeConnections)'
 
 

 Here is another example:

@@ -104,10 +124,10 @@ http://www.gnu.org/software/src-highlite -->
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-% dtail --servers serverlist.txt \
-    --files '/var/log/dserver/*.log' \
-    --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,
-             lifetimeConnections group by $hostname order by max($cgocalls)'
+% dtail --servers serverlist.txt \
+    --files '/var/log/dserver/*.log' \
+    --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,
+             lifetimeConnections group by $hostname order by max($cgocalls)'
 
 

 

@@ -118,12 +138,12 @@ http://www.gnu.org/software/src-highlite -->
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-% dtail --servers serverlist.txt \
-    --files '/var/log/dserver/*.log' \
-    --query 'from STATS select ... outfile append result.csv'
+% dtail --servers serverlist.txt \
+    --files '/var/log/dserver/*.log' \
+    --query 'from STATS select ... outfile append result.csv'
 
 

-How to use dcat


+How to use dcat


 

 The following example demonstrates how to cat files (display the full content of the files) on several servers at once.

 

@@ -133,7 +153,7 @@ http://www.gnu.org/software/src-highlite -->
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-% dcat --servers serverlist.txt --files /etc/hostname
+% dcat --servers serverlist.txt --files /etc/hostname
 
 

 

@@ -144,10 +164,10 @@ http://www.gnu.org/software/src-highlite -->
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-% dcat --servers serverlist.txt /etc/hostname
+% dcat --servers serverlist.txt /etc/hostname
 
 

-How to use dgrep


+How to use dgrep


 

 The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after some entries in /etc/passwd. This time, we don't provide the server list via an file but rather via a comma separated list directly on the command line. We also explore the -before, -after and -max flags (see animation).

 

@@ -155,8 +175,8 @@ http://www.gnu.org/software/src-highlite -->
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-% dgrep --servers server1.example.org:2223 \
-    --files /etc/passwd \
+% dgrep --servers server1.example.org:2223 \
+    --files /etc/passwd \
     --regex nologin
 
 

@@ -166,7 +186,7 @@ http://www.gnu.org/software/src-highlite -->
 

 Hint: -regex is an alias for -grep.

 

-How to use dmap


+How to use dmap


 

 To run a map-reduce aggregation over logs written in the past, the dmap command can be used. The following example aggregates all map-reduce fields dmap will print interim results every few seconds. You can also write the result to an CSV file by adding outfile result.csv to the query.

 

@@ -174,17 +194,17 @@ http://www.gnu.org/software/src-highlite -->
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-% dmap --servers serverlist.txt \
-    --files '/var/log/dserver/*.log' \
-    --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,
-             lifetimeConnections group by $hostname order by max($cgocalls)'
+% dmap --servers serverlist.txt \
+    --files '/var/log/dserver/*.log' \
+    --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,
+             lifetimeConnections group by $hostname order by max($cgocalls)'
 
 

 Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in internal/mapr/logformat or add an extension to support a custom log format. The example here works out of the box though, as DTail understands its own log format already. 

 

 

 

-How to use the DTail serverless mode


+How to use the DTail serverless mode


 

 Until now, all examples so far required to have remote server(s) to connect to. That makes sense, as after all DTail is a *distributed* tool. However, there are circumstances where you don't really need to connect to a server remotely. For example, you already have a login shell open to the server an all what you want is to run some queries directly on local log files.

 

@@ -192,7 +212,7 @@ http://www.gnu.org/software/src-highlite -->
 

 All commands shown so far also work in a serverless mode. All what needs to be done is to omit a server list. The DTail client then starts in serverless mode.

 

-Serverless map-reduce query


+Serverless map-reduce query


 

 The following dmap example is the same as the previously shown one, but the difference is that it operates on a local log file directly:

 

@@ -200,9 +220,9 @@ http://www.gnu.org/software/src-highlite -->
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-% dmap --files /var/log/dserver/dserver.log
-    --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,
-              lifetimeConnections group by $hostname order by max($cgocalls)'
+% dmap --files /var/log/dserver/dserver.log
+    --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,
+              lifetimeConnections group by $hostname order by max($cgocalls)'
 
 

 As a shorthand version the following command can be used:

@@ -211,9 +231,9 @@ http://www.gnu.org/software/src-highlite -->
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-% dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,
-        lifetimeConnections group by $hostname order by max($cgocalls)' \
-        /var/log/dsever/dserver.log
+% dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,
+        lifetimeConnections group by $hostname order by max($cgocalls)' \
+        /var/log/dsever/dserver.log
 
 

 You can also use a file input pipe as follows:

@@ -222,12 +242,12 @@ http://www.gnu.org/software/src-highlite -->
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-% cat /var/log/dserver/dserver.log | \
-    dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,
-          lifetimeConnections group by $hostname order by max($cgocalls)'
+% cat /var/log/dserver/dserver.log | \
+    dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,
+          lifetimeConnections group by $hostname order by max($cgocalls)'
 
 

-Aggregating CSV files


+Aggregating CSV files


 

 In essence, this works exactly like aggregating logs. All files operated on must be valid CSV files and the first line of the CSV must be the header. E.g.:

 

@@ -235,21 +255,21 @@ http://www.gnu.org/software/src-highlite -->
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-% cat example.csv
-name,lastname,age,profession
-Michael,Jordan,40,Basketball player
-Michael,Jackson,100,Singer
-Albert,Einstein,200,Physician
-% dmap --query 'select lastname,name where age > 40 logformat csv outfile result.csv' example.csv
-% cat result.csv
-lastname,name
-Jackson,Michael
-Einstein,Albert
+% cat example.csv
+name,lastname,age,profession
+Michael,Jordan,40,Basketball player
+Michael,Jackson,100,Singer
+Albert,Einstein,200,Physician
+% dmap --query 'select lastname,name where age > 40 logformat csv outfile result.csv' example.csv
+% cat result.csv
+lastname,name
+Jackson,Michael
+Einstein,Albert
 
 

 DMap can also be used to query and aggregate CSV files from remote servers.

 

-Other serverless commands


+Other serverless commands


 

 The serverless mode works transparently with all other DTail commands. Here are some examples:

 

@@ -257,29 +277,29 @@ Einstein,Albert
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-% dtail /var/log/dserver/dserver.log
+% dtail /var/log/dserver/dserver.log
 
 

 
-% dtail --logLevel trace /var/log/dserver/dserver.log
+% dtail --logLevel trace /var/log/dserver/dserver.log
 
 

 
-% dcat /etc/passwd
+% dcat /etc/passwd
 
 

 
-% dcat --plain /etc/passwd > /etc/test
-# Should show no differences.
+% dcat --plain /etc/passwd > /etc/test
+# Should show no differences.
 diff /etc/test /etc/passwd 
 
 

@@ -287,36 +307,36 @@ diff /etc/test /etc/passwd
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-% dgrep --regex ERROR --files /var/log/dserver/dsever.log
+% dgrep --regex ERROR --files /var/log/dserver/dsever.log
 
 

 
-% dgrep --before 10 --after 10 --max 10 --grep ERROR /var/log/dserver/dsever.log
+% dgrep --before 10 --after 10 --max 10 --grep ERROR /var/log/dserver/dsever.log
 
 

 Use --help for more available options. Or go to the DTail page for more information! Hope you find DTail useful!

 

+E-Mail your comments to paul@nospam.buetow.org :-)

+

 Other related posts are:

 

-2021-04-22 DTail - The distributed log tail program

-2022-03-06 The release of DTail 4.0.0

-2022-10-30 Installing DTail on OpenBSD

 2023-09-25 DTail usage examples (You are currently reading this)

+2022-10-30 Installing DTail on OpenBSD

+2022-03-06 The release of DTail 4.0.0

+2021-04-22 DTail - The distributed log tail program

 

 I hope you find the tools presented in this post useful!

 

 Paul

 

-E-Mail your comments to paul@nospam.buetow.org :-)

-

 Back to the main site

 
-Generated by Gemtexter 2.1.0-release |
-served by OpenBSD/httpd(8) |
-Site Mirrors
+Generated with Gemtexter 3.0.1-develop |
+served by OpenBSD/relayd(8)+httpd(8) |
+Site Mirrors
 
 
 
-- 
cgit v1.2.3

DTail usage examples

DTail usage examples

Table of Contents

Commands

Following logs

Following logs

Aggregating logs

Aggregating logs

How to use dcat

How to use dcat

How to use dgrep

How to use dgrep

How to use dmap

How to use dmap

How to use the DTail serverless mode

How to use the DTail serverless mode

Serverless map-reduce query

Serverless map-reduce query

Aggregating CSV files

Aggregating CSV files

Other serverless commands

Other serverless commands