diff options
| author | Paul Buetow <git@mx.buetow.org> | 2021-06-19 11:08:15 +0100 |
|---|---|---|
| committer | Paul Buetow <git@mx.buetow.org> | 2021-06-19 11:08:15 +0100 |
| commit | 5954326f4d7bd39a8f1155fa7f2fbf94a8a2498d (patch) | |
| tree | d81346613ecbaeef3b091534f6df097b86516972 /gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html | |
| parent | 109648ca317842d8f008e9c5642acd8b3e337cf1 (diff) | |
descr as alt tags only
Diffstat (limited to 'gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html')
| -rw-r--r-- | gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html | 6 |
1 files changed, 3 insertions, 3 deletions
diff --git a/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html b/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html index 8cea6e07..63424869 100644 --- a/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html +++ b/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.html @@ -64,7 +64,7 @@ p.quote:after { <body> <h1>DTail - The distributed log tail program</h1> <p class="quote"><i>Written by Paul Buetow 2021-04-22, last updated 2021-04-26</i></p> -<i>DTail logo image:</i><a href="./2021-04-22-dtail-the-distributed-log-tail-program/title.png"><img alt="DTail logo image" title="DTail logo image" src="./2021-04-22-dtail-the-distributed-log-tail-program/title.png" /></a><br /> +<a href="./2021-04-22-dtail-the-distributed-log-tail-program/title.png"><img alt="DTail logo image" title="DTail logo image" src="./2021-04-22-dtail-the-distributed-log-tail-program/title.png" /></a><br /> <p>This article first appeared at the Mimecast Engineering Blog but I made it available here in my personal internet site too.</p> <a class="textlink" href="https://medium.com/mimecast-engineering/dtail-the-distributed-log-tail-program-79b8087904bb">Original Mimecast Engineering Blog post at Medium</a><br /> <p>Running a large cloud-based service requires monitoring the state of huge numbers of machines, a task for which many standard UNIX tools were not really designed. In this post, I will describe a simple program, DTail, that Mimecast has built and released as Open-Source, which enables us to monitor log files of many servers at once without the costly overhead of a full-blown log management system.</p> @@ -78,7 +78,7 @@ p.quote:after { <h2>Differentiating from log management systems</h2> <p>Why not just use a full-blown log management system? There are various Open-Source and commercial log management solutions available on the market you could choose from (e.g. the ELK stack). Most of them store the logs in a centralized location and are fairly complex to set up and operate. Possibly they are also pretty expensive to operate if you have to buy dedicated hardware (or pay fees to your cloud provider) and have to hire support staff for it.</p> <p>DTail does not aim to replace any of the log management tools already available but is rather an additional tool crafted especially for ad-hoc debugging and troubleshooting purposes. DTail is cheap to operate as it does not require any dedicated hardware for log storage as it operates directly on the source of the logs. It means that there is a DTail server installed on all server boxes producing logs. This decentralized comes with the direct advantages that there is no introduced delay because the logs are not shipped to a central log storage device. The reduced complexity also makes it more robust against outages. You won’t be able to troubleshoot your distributed application very well if the log management infrastructure isn’t working either.</p> -<i>DTail sample session animated gif:</i><a href="./2021-04-22-dtail-the-distributed-log-tail-program/dtail.gif"><img alt="DTail sample session animated gif" title="DTail sample session animated gif" src="./2021-04-22-dtail-the-distributed-log-tail-program/dtail.gif" /></a><br /> +<a href="./2021-04-22-dtail-the-distributed-log-tail-program/dtail.gif"><img alt="DTail sample session animated gif" title="DTail sample session animated gif" src="./2021-04-22-dtail-the-distributed-log-tail-program/dtail.gif" /></a><br /> <p>As a downside, you won’t be able to access any logs with DTail when the server is down. Furthermore, a server can store logs only up to a certain capacity as disks will fill up. For the purpose of ad-hoc debugging, these are not typically issues. Usually, it’s the application you want to debug and not the server. And disk space is rarely an issue for bare metal and VM-based systems these days, with sufficient space for several weeks’ worth of log storage being available. DTail also supports reading compressed logs. The currently supported compression algorithms are gzip and zstd.</p> <h2>Combining simplicity, security and efficiency</h2> <p>DTail also has a client component that connects to multiple servers concurrently for log files (or any other text files).</p> @@ -94,7 +94,7 @@ p.quote:after { <li>dgrep: The distributed grep client for searching text files for a regular expression pattern.</li> <li>dmap: The distributed map-reduce client for aggregating stats from log files.</li> </ul> -<i>DGrep sample session animated gif:</i><a href="./2021-04-22-dtail-the-distributed-log-tail-program/dgrep.gif"><img alt="DGrep sample session animated gif" title="DGrep sample session animated gif" src="./2021-04-22-dtail-the-distributed-log-tail-program/dgrep.gif" /></a><br /> +<a href="./2021-04-22-dtail-the-distributed-log-tail-program/dgrep.gif"><img alt="DGrep sample session animated gif" title="DGrep sample session animated gif" src="./2021-04-22-dtail-the-distributed-log-tail-program/dgrep.gif" /></a><br /> <h2>Usage example</h2> <p>The use of these commands is almost self-explanatory for a person already used to the standard command line in Unix systems. One of the main goals is to make DTail easy to use. A tool that is too complicated to use under high-pressure scenarios (e.g., during an incident) can be quite detrimental.</p> <p>The basic idea is to start one of the clients from the command line and provide a list of servers to connect to with –servers. You also must provide a path of remote (log) files via –files. If you want to process multiple files per server, you could either provide a comma-separated list of file paths or make use of file system globbing (or a combination of both).</p> |