diff options
| author | Paul Buetow <paul@buetow.org> | 2024-04-30 13:14:09 +0300 |
|---|---|---|
| committer | Paul Buetow <paul@buetow.org> | 2024-04-30 13:14:09 +0300 |
| commit | 07c56086aa0c4e015c9044e333ae4001debcb28d (patch) | |
| tree | 2b2c0a3dae32313d55c6f223095612dcf0b7d779 /gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html | |
| parent | 5ad16713db1f011c08d2db602ed5b1d3294f0939 (diff) | |
Update content for html
Diffstat (limited to 'gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html')
| -rw-r--r-- | gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html | 20 |
1 files changed, 10 insertions, 10 deletions
diff --git a/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html b/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html index 68fcfcf7..48b450c0 100644 --- a/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html +++ b/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html @@ -8,7 +8,7 @@ <link rel="stylesheet" href="style-override.css" /> </head> <body> -<h1 style='display: inline'>DTail - The distributed log tail program</h1><br /> +<h1 style='display: inline' id='DTailThedistributedlogtailprogram'>DTail - The distributed log tail program</h1><br /> <br /> <span class='quote'>Published at 2021-04-22T19:28:41+01:00; Updated at 2021-04-26</span><br /> <br /> @@ -26,7 +26,7 @@ <br /> <span>Think of DTail as a distributed version of the tail program which is very useful when you have a distributed application running on many servers. DTail is an Open-Source, cross-platform, fairly easy to use, support and maintain log file analysis & statistics gathering tool designed for Engineers and Systems Administrators. It is programmed in Google Go.</span><br /> <br /> -<h2 style='display: inline'>A Mimecast Pet Project</h2><br /> +<h2 style='display: inline' id='AMimecastPetProject'>A Mimecast Pet Project</h2><br /> <br /> <span>DTail got its inspiration from public domain tools available already in this area but it is a blue sky from-scratch development which was first presented at Mimecast’s annual internal Pet Project competition (awarded with a Bronze prize). It has gained popularity since and is one of the most widely deployed DevOps tools at Mimecast (reaching nearly 10k server installations) and many engineers use it on a regular basis. The Open-Source version of DTail is available at:</span><br /> <br /> @@ -34,7 +34,7 @@ <br /> <span>Try it out — We would love any feedback. But first, read on…</span><br /> <br /> -<h2 style='display: inline'>Differentiating from log management systems</h2><br /> +<h2 style='display: inline' id='Differentiatingfromlogmanagementsystems'>Differentiating from log management systems</h2><br /> <br /> <span>Why not just use a full-blown log management system? There are various Open-Source and commercial log management solutions available on the market you could choose from (e.g. the ELK stack). Most of them store the logs in a centralized location and are fairly complex to set up and operate. Possibly they are also pretty expensive to operate if you have to buy dedicated hardware (or pay fees to your cloud provider) and have to hire support staff for it.</span><br /> <br /> @@ -44,7 +44,7 @@ <br /> <span>As a downside, you won’t be able to access any logs with DTail when the server is down. Furthermore, a server can store logs only up to a certain capacity as disks will fill up. For the purpose of ad-hoc debugging, these are not typically issues. Usually, it’s the application you want to debug and not the server. And disk space is rarely an issue for bare metal and VM-based systems these days, with sufficient space for several weeks’ worth of log storage being available. DTail also supports reading compressed logs. The currently supported compression algorithms are gzip and zstd.</span><br /> <br /> -<h2 style='display: inline'>Combining simplicity, security and efficiency</h2><br /> +<h2 style='display: inline' id='Combiningsimplicitysecurityandefficiency'>Combining simplicity, security and efficiency</h2><br /> <br /> <span>DTail also has a client component that connects to multiple servers concurrently for log files (or any other text files).</span><br /> <br /> @@ -54,7 +54,7 @@ <br /> <span>Recent log files are very likely still in the file system caches on the servers. Therefore, there tends to be a minimal I/O overhead involved.</span><br /> <br /> -<h2 style='display: inline'>The DTail family of commands</h2><br /> +<h2 style='display: inline' id='TheDTailfamilyofcommands'>The DTail family of commands</h2><br /> <br /> <span>Following the UNIX philosophy, DTail includes multiple command-line commands each of them for a different purpose:</span><br /> <br /> @@ -67,7 +67,7 @@ </ul><br /> <a href='./2021-04-22-dtail-the-distributed-log-tail-program/dgrep.gif'><img alt='DGrep sample session animated gif' title='DGrep sample session animated gif' src='./2021-04-22-dtail-the-distributed-log-tail-program/dgrep.gif' /></a><br /> <br /> -<h2 style='display: inline'>Usage example</h2><br /> +<h2 style='display: inline' id='Usageexample'>Usage example</h2><br /> <br /> <span>The use of these commands is almost self-explanatory for a person already used to the standard command line in Unix systems. One of the main goals is to make DTail easy to use. A tool that is too complicated to use under high-pressure scenarios (e.g., during an incident) can be quite detrimental.</span><br /> <br /> @@ -85,11 +85,11 @@ dtail –servers serverlist.txt –files ‘/var/log/*.log’ –regex ‘(?i:er <br /> <span>You could also provide a comma-separated list of servers as opposed to a text file. There are many more options you could use. The ones listed here are just the very basic ones. There are more instructions and usage examples on the GitHub page. Also, you can study even more of the available options via the –help switch (some real treasures might be hidden there).</span><br /> <br /> -<h2 style='display: inline'>Fitting it in</h2><br /> +<h2 style='display: inline' id='Fittingitin'>Fitting it in</h2><br /> <br /> <span>DTail integrates nicely into the user management of existing infrastructure. It follows normal system permissions and does not open new “holes” on the server which helps to keep security departments happy. The user would not have more or less file read permissions than he would have via a regular SSH login shell. There is a full SSH key, traditional UNIX permissions, and Linux ACL support. There is also a very low resource footprint involved. On average for tailing and searching log files less than 100MB RAM and less than a quarter of a CPU core per participating server are required. Complex map-reduce queries on big data sets will require more resources accordingly.</span><br /> <br /> -<h2 style='display: inline'>Advanced features</h2><br /> +<h2 style='display: inline' id='Advancedfeatures'>Advanced features</h2><br /> <br /> <span>The features listed here are out of the scope of this blog post but are worthwhile to mention:</span><br /> <br /> @@ -100,7 +100,7 @@ dtail –servers serverlist.txt –files ‘/var/log/*.log’ –regex ‘(?i:er <li>Server-side stats streaming with continuous map-reduce queries. This for example can be used to periodically generate stats from the logs at a configured interval, e.g., log error counts by the minute. These then can be sent to a time-series database (e.g., Graphite) and then plotted in a Grafana dashboard.</li> <li>Support for custom extensions. E.g., for different server discovery methods (so you don’t have to rely on plain server lists) and log file formats (so that map-reduce queries can parse more stats from the logs).</li> </ul><br /> -<h2 style='display: inline'>For the future</h2><br /> +<h2 style='display: inline' id='Forthefuture'>For the future</h2><br /> <br /> <span>There are various features we want to see in the future.</span><br /> <br /> @@ -110,7 +110,7 @@ dtail –servers serverlist.txt –files ‘/var/log/*.log’ –regex ‘(?i:er <li>A more complex change would be the support of federated queries. You can connect to thousands of servers from a single client running on a laptop. But does it scale to 100k of servers? Some of the servers could be used as middleware for connecting to even more servers.</li> <li>Another aspect is to extend the documentation. Especially the advanced features such as map-reduce query language and how to configure the server-side queries currently do require more documentation. For now, you can read the code, sample config files or just ask the author for that! But this will be certainly addressed in the future.</li> </ul><br /> -<h2 style='display: inline'>Open Source</h2><br /> +<h2 style='display: inline' id='OpenSource'>Open Source</h2><br /> <br /> <span>Mimecast highly encourages you to have a look at DTail and submit an issue for any features you would like to see. Have you found a bug? Maybe you just have a question or comment? If you want to go a step further: We would also love to see pull requests for any features or improvements. Either way, if in doubt just contact us via the DTail GitHub page.</span><br /> <br /> |