diff options
Diffstat (limited to 'content/html/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html')
| -rw-r--r-- | content/html/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/content/html/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html b/content/html/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html index 6c591fb6..99dcbd3c 100644 --- a/content/html/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html +++ b/content/html/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html @@ -44,6 +44,7 @@ li { color: #98be65; } <h2>The DTail family of commands</h2> <p>Following the UNIX philosophy, DTail includes multiple command-line commands each of them for a different purpose:</p> <ul> +<li>dserver: The DTail server, the only binary required to be installed on the servers involved.</li> <li>dtail: The distributed log tail client for following log files.</li> <li>dcat: The distributed cat client for concatenating and displaying text files.</li> <li>dgrep: The distributed grep client for searching text files for a regular expression pattern.</li> @@ -65,6 +66,7 @@ dtail –servers serverlist.txt –files ‘/var/log/*.log’ –regex ‘(?i:er <h2>Advanced features</h2> <p>The features listed here are out of the scope of this blog post but are worthwhile to mention:</p> <ul> +<li>Distributed map-reduce queries on stats provided in log files with dmap. dmap comes with its own SQL-like aggregation query language.</li> <li>Stats streaming with continuous map-reduce queries. The difference to normal queries is that the stats are aggregated over a specified interval only on the newly written log lines. Thus, giving a de-facto live stat view for each interval.</li> <li>Server-side scheduled queries on log files. The queries are configured in the DTail server configuration file and scheduled at certain time intervals. Results are written to CSV files. This is useful for generating daily stats from the log files without the need for an interactive client.</li> <li>Server-side stats streaming with continuous map-reduce queries. This for example can be used to periodically generate stats from the logs at a configured interval, e.g., log error counts by the minute. These then can be sent to a time-series database (e.g., Graphite) and then plotted in a Grafana dashboard.</li> @@ -73,6 +75,7 @@ dtail –servers serverlist.txt –files ‘/var/log/*.log’ –regex ‘(?i:er <h2>For the future</h2> <p>There are various features we want to see in the future.</p> <ul> +<li>A spartan mode, not printing out any extra information but the raw remote log files would be a nice feature to have. This will make it easier to post-process the data produced by the DTail client with common UNIX tools. (To some degree this is possible already, just disable the ANSI terminal colour output of the client with -noColors and pipe the output to another program).</li> <li>Tempting would be implementing the dgoawk command, a distributed version of the AWK programming language purely implemented in Go, for advanced text data stream processing capabilities. There are 3rd party libraries available implementing AWK in pure Go which could be used.</li> <li>A more complex change would be the support of federated queries. You can connect to thousands of servers from a single client running on a laptop. But does it scale to 100k of servers? Some of the servers could be used as middleware for connecting to even more servers.</li> <li>Another aspect is to extend the documentation. Especially the advanced features such as map-reduce query language and how to configure the server-side queries currently do require more documentation. For now, you can read the code, sample config files or just ask the author for that! But this will be certainly addressed in the future.</li> |