1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Installing DTail on OpenBSD</title>
<link rel="shortcut icon" type="image/gif" href="/favicon.ico" />
<link rel="stylesheet" href="../style.css" />
<link rel="stylesheet" href="style-override.css" />
</head>
<body>
<p class="header">
<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/2022-10-30-installing-dtail-on-openbsd.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/2022-10-30-installing-dtail-on-openbsd.gmi">Gemini</a>
</p>
<h1 style='display: inline' id='installing-dtail-on-openbsd'>Installing DTail on OpenBSD</h1><br />
<br />
<span class='quote'>Published at 2022-10-30T11:03:19+02:00</span><br />
<br />
<span>This will be a quick blog post, as I am busy with my personal life now. I have relocated to a different country and am still busy arranging things. So bear with me :-)</span><br />
<br />
<span> In this post, I want to give a quick overview (or how-to) about installing DTail on OpenBSD, as the official documentation only covers Red Hat and Fedora Linux! And this blog post will also be used as my reference!</span><br />
<br />
<a class='textlink' href='https://dtail.dev'>https://dtail.dev</a><br />
<br />
<span>I am using Rexify for my OpenBSD automation. Check out the following article covering my Rex setup in a little bit more detail:</span><br />
<br />
<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>Let's Encrypt with OpenBSD and Rex</a><br />
<br />
<span>I will also mention some relevant <span class='inlinecode'>Rexfile</span> snippets in this post!</span><br />
<br />
<pre>
,_---~~~~~----._
_,,_,*^____ _____``*g*\"*,
/ __/ /' ^. / \ ^@q f
@f | | | | 0 _/
\`/ \~__((@/ __ \__((@/ \
| _l__l_ I <--- The Go Gopher
} [______] I
] | | | |
] ~ ~ |
| |
| |
| | A ;
~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~,--,-/ \---,-/|~~,~~~~~~~~~~~~~~~~~~~~~~~~~~~
_|\,'. /| /| `/|-.
\`.' /| , `;.
,'\ A A A A _ /| `.;
,/ _ A _ / _ /| ;
/\ / \ , , A / / `/|
/_| | _ \ , , ,/ \
// | |/ `.\ ,- , , ,/ ,/ \/
/ @| |@ / /' \ \ , > /| ,--.
|\_/ \_/ / | | , ,/ \ ./' __:..
| __ __ | | | .--. , > > |-' / `
,/| / ' \ | | | \ , | /
/ |<--.__,->| | | . `. > > / (
/_,' \\ ^ / \ / / `. >-- /^\ |
\\___/ \ / / \__' \ \ \/ \ |
`. |/ , , /`\ \ )
\ ' |/ , V \ / `-\
OpenBSD Puffy ---> `|/ ' V V \ \.' \_
'`-. V V \./'\
`|/-. \ / \ /,---`\ kat
/ `._____V_____V'
' '
</pre>
<br />
<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br />
<br />
<ul>
<li><a href='#installing-dtail-on-openbsd'>Installing DTail on OpenBSD</a></li>
<li>⇢ <a href='#compile-it'>Compile it</a></li>
<li>⇢ <a href='#install-it'>Install it</a></li>
<li>⇢ ⇢ <a href='#rexification'>Rexification</a></li>
<li>⇢ <a href='#configure-it'>Configure it</a></li>
<li>⇢ ⇢ <a href='#rexification'>Rexification</a></li>
<li>⇢ <a href='#update-the-key-cache-for-it'>Update the key cache for it</a></li>
<li>⇢ ⇢ <a href='#rexification'>Rexification</a></li>
<li>⇢ <a href='#start-it'>Start it</a></li>
<li>⇢ <a href='#use-it'>Use it</a></li>
<li>⇢ <a href='#conclusions'>Conclusions</a></li>
</ul><br />
<h2 style='display: inline' id='compile-it'>Compile it</h2><br />
<br />
<span>First of all, DTail needs to be downloaded and compiled. For that, <span class='inlinecode'>git</span>, <span class='inlinecode'>go</span>, and <span class='inlinecode'>gmake</span> are required:</span><br />
<br />
<pre>
$ doas pkg_add git go gmake
</pre>
<br />
<span>I am happy that the Go Programming Language is readily available in the OpenBSD packaging system. Once the dependencies got installed, clone DTail and compile it:</span><br />
<br />
<pre>
$ mkdir git
$ cd git
$ git clone https://github.com/mimecast/dtail
$ cd dtail
$ gmake
</pre>
<br />
<span>You can verify the version by running the following command:</span><br />
<br />
<pre>
$ ./dtail --version
DTail 4.1.0 Protocol 4.1 Have a lot of fun!
$ file dtail
dtail: ELF 64-bit LSB executable, x86-64, version 1
</pre>
<br />
<span>Now, there isn't any need anymore to keep <span class='inlinecode'>git</span>, <span class='inlinecode'>go</span> and <span class='inlinecode'>gmake</span>, so they can be deinstalled now:</span><br />
<br />
<pre>
$ doas pkg_delete git go gmake
</pre>
<br />
<span>One day I shall create an official OpenBSD port for DTail.</span><br />
<br />
<h2 style='display: inline' id='install-it'>Install it</h2><br />
<br />
<span>Installing the binaries is now just a matter of copying them to <span class='inlinecode'>/usr/local/bin</span> as follows:</span><br />
<br />
<pre>
$ for bin in dserver dcat dgrep dmap dtail dtailhealth; do
doas cp -p $bin /usr/local/bin/$bin
doas chown root:wheel /usr/local/bin/$bin
done
</pre>
<br />
<span>Also, we will be creating the <span class='inlinecode'>_dserver</span> service user:</span><br />
<br />
<pre>
$ doas adduser -class nologin -group _dserver -batch _dserver
$ doas usermod -d /var/run/dserver/ _dserver
</pre>
<br />
<span>The OpenBSD init script is created from scratch (not part of the official DTail project). Run the following to install the bespoke script:</span><br />
<br />
<pre>
$ cat <<'END' | doas tee /etc/rc.d/dserver
#!/bin/ksh
daemon="/usr/local/bin/dserver"
daemon_flags="-cfg /etc/dserver/dtail.json"
daemon_user="_dserver"
. /etc/rc.d/rc.subr
rc_reload=NO
rc_pre() {
install -d -o _dserver /var/log/dserver
install -d -o _dserver /var/run/dserver/cache
}
rc_cmd $1 &
END
$ doas chmod 755 /etc/rc.d/dserver
</pre>
<br />
<h3 style='display: inline' id='rexification'>Rexification</h3><br />
<br />
<span>This is the task for setting it up via Rex. Note the <span class='inlinecode'>. . . .</span>, that's a placeholder which we will fill up more and more during this blog post:</span><br />
<br />
<pre>
desc 'Setup DTail';
task 'dtail', group => 'frontends',
sub {
my $restart = FALSE;
file '/etc/rc.d/dserver':
content => template('./etc/rc.d/dserver.tpl'),
owner => 'root',
group => 'wheel',
mode => '755',
on_change => sub { $restart = TRUE };
.
.
.
.
service 'dserver' => 'restart' if $restart;
service 'dserver', ensure => 'started';
};
</pre>
<br />
<h2 style='display: inline' id='configure-it'>Configure it</h2><br />
<br />
<span>Now, DTail is fully installed but still needs to be configured. Grab the default config file from GitHub ...</span><br />
<br />
<pre>
$ doas mkdir /etc/dserver
$ curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dtail.json.examples |
doas tee /etc/dserver/dtail.json
</pre>
<br />
<span>... and then edit it and adjust <span class='inlinecode'>LogDir</span> in the <span class='inlinecode'>Common</span> section to <span class='inlinecode'>/var/log/dserver</span>. The result will look like this:</span><br />
<br />
<pre>
"Common": {
"LogDir": "/var/log/dserver",
"Logger": "Fout",
"LogRotation": "Daily",
"CacheDir": "cache",
"SSHPort": 2222,
"LogLevel": "Info"
}
</pre>
<br />
<h3 style='display: inline' id='rexification'>Rexification</h3><br />
<br />
<span>That's as simple as adding the following to the Rex task:</span><br />
<br />
<pre>
file '/etc/dserver',
ensure => 'directory';
file '/etc/dserver/dtail.json',
content => template('./etc/dserver/dtail.json.tpl'),
owner => 'root',
group => 'wheel',
mode => '755',
on_change => sub { $restart = TRUE };
</pre>
<br />
<h2 style='display: inline' id='update-the-key-cache-for-it'>Update the key cache for it</h2><br />
<br />
<span>DTail relies on SSH for secure authentication and communication. However, the system user <span class='inlinecode'>_dserver</span> has no permission to read the SSH public keys from the user's home directories, so the DTail server also checks for available public keys in an alternative path <span class='inlinecode'>/var/run/dserver/cache</span>. </span><br />
<br />
<span>The following script, populating the DTail server key cache, can be run periodically via <span class='inlinecode'>CRON</span>:</span><br />
<br />
<pre>
$ cat <<'END' | doas tee /usr/local/bin/dserver-update-key-cache.sh
#!/bin/ksh
CACHEDIR=/var/run/dserver/cache
DSERVER_USER=_dserver
DSERVER_GROUP=_dserver
echo 'Updating SSH key cache'
ls /home/ | while read remoteuser; do
keysfile=/home/$remoteuser/.ssh/authorized_keys
if [ -f $keysfile ]; then
cachefile=$CACHEDIR/$remoteuser.authorized_keys
echo "Caching $keysfile -> $cachefile"
cp $keysfile $cachefile
chown $DSERVER_USER:$DSERVER_GROUP $cachefile
chmod 600 $cachefile
fi
done
# Cleanup obsolete public SSH keys
find $CACHEDIR -name \*.authorized_keys -type f |
while read cachefile; do
remoteuser=$(basename $cachefile | cut -d. -f1)
keysfile=/home/$remoteuser/.ssh/authorized_keys
if [ ! -f $keysfile ]; then
echo 'Deleting obsolete cache file $cachefile'
rm $cachefile
fi
done
echo 'All set...'
END
$ doas chmod 500 /usr/local/bin/dserver-update-key-cache.sh
</pre>
<br />
<span>Note that the script above is a slight variation of the official DTail script. The official DTail one is a <span class='inlinecode'>bash</span> script, but on OpenBSD, there's <span class='inlinecode'>ksh</span>. I run it once daily by adding it to the <span class='inlinecode'>daily.local</span>:</span><br />
<br />
<pre>
$ echo /usr/local/bin/dserver-update-key-cache.sh | doas tee -a /etc/daily.local
/usr/local/bin/dserver-update-key-cache.sh
</pre>
<br />
<h3 style='display: inline' id='rexification'>Rexification</h3><br />
<br />
<span>That's done by adding ...</span><br />
<br />
<pre>
file '/usr/local/bin/dserver-update-key-cache.sh',
content => template('./scripts/dserver-update-key-cache.sh.tpl'),
owner => 'root',
group => 'wheel',
mode => '500';
append_if_no_such_line '/etc/daily.local', '/usr/local/bin/dserver-update-key-cache.sh';
</pre>
<br />
<span>... to the Rex task!</span><br />
<br />
<h2 style='display: inline' id='start-it'>Start it</h2><br />
<br />
<span>Now, it's time to enable and start the DTail server:</span><br />
<br />
<pre>
$ sudo rcctl enable dserver
$ sudo rcctl start dserver
$ tail -f /var/log/dserver/*.log
INFO|1022-090634|Starting scheduled job runner after 2s
INFO|1022-090634|Starting continuous job runner after 2s
INFO|1022-090644|24204|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnections=0
INFO|1022-090654|24204|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnections=0
INFO|1022-090719|Starting server|DTail 4.1.0 Protocol 4.1 Have a lot of fun!
INFO|1022-090719|Generating private server RSA host key
INFO|1022-090719|Starting server
INFO|1022-090719|Binding server|0.0.0.0:2222
INFO|1022-090719|Starting scheduled job runner after 2s
INFO|1022-090719|Starting continuous job runner after 2s
INFO|1022-090729|86050|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnections=0
INFO|1022-090739|86050|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnect
.
.
.
Ctr+C
</pre>
<br />
<span>As we don't want to wait until tomorrow, let's populate the key cache manually:</span><br />
<br />
<pre>
$ doas /usr/local/bin/dserver-update-key-cache.sh
Updating SSH key cache
Caching /home/_dserver/.ssh/authorized_keys -> /var/cache/dserver/_dserver.authorized_keys
Caching /home/admin/.ssh/authorized_keys -> /var/cache/dserver/admin.authorized_keys
Caching /home/failunderd/.ssh/authorized_keys -> /var/cache/dserver/failunderd.authorized_keys
Caching /home/git/.ssh/authorized_keys -> /var/cache/dserver/git.authorized_keys
Caching /home/paul/.ssh/authorized_keys -> /var/cache/dserver/paul.authorized_keys
Caching /home/rex/.ssh/authorized_keys -> /var/cache/dserver/rex.authorized_keys
All set...
</pre>
<br />
<h2 style='display: inline' id='use-it'>Use it</h2><br />
<br />
<span>The DTail server is now ready to serve connections. You can use any DTail commands, such as <span class='inlinecode'>dtail</span>, <span class='inlinecode'>dgrep</span>, <span class='inlinecode'>dmap</span>, <span class='inlinecode'>dcat</span>, <span class='inlinecode'>dtailhealth</span>, to do so. Checkout out all the usage examples on the official DTail page.</span><br />
<br />
<span>I have installed DTail server this way on my personal OpenBSD frontends <span class='inlinecode'>blowfish</span>, and <span class='inlinecode'>fishfinger</span>, and the following command connects as user <span class='inlinecode'>rex</span> to both machines and greps the file <span class='inlinecode'>/etc/fstab</span> for the string <span class='inlinecode'>local</span>:</span><br />
<br />
<pre>
❯ ./dgrep -user rex -servers blowfish.buetow.org,fishfinger.buetow.org --regex local /etc/fstab
CLIENT|earth|WARN|Encountered unknown host|{blowfish.buetow.org:2222 0xc0000a00f0 0xc0000a61e0 [blowfish.buetow.org]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9ZnF/LAk14SgqCzk38yENVTNfqibcluMTuKx1u53cKSp2xwHWzy0Ni5smFPpJDIQQljQEJl14ZdXvhhjp1kKHxJ79ubqRtIXBlC0PhlnP8Kd+mVLLHYpH9VO4rnaSfHE1kBjWkI7U6lLc6ks4flgAgGTS5Bb7pLAjwdWg794GWcnRh6kSUEQd3SftANqQLgCunDcP2Vc4KR9R78zBmEzXH/OPzl/ANgNA6wWO2OoKKy2VrjwVAab6FW15h3Lr6rYIw3KztpG+UMmEj5ReexIjXi/jUptdnUFWspvAmzIl6kwzzF8ExVyT9D75JRuHvmxXKKjyJRxqb8UnSh2JD4JN [23.88.35.144]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9ZnF/LAk14SgqCzk38yENVTNfqibcluMTuKx1u53cKSp2xwHWzy0Ni5smFPpJDIQQljQEJl14ZdXvhhjp1kKHxJ79ubqRtIXBlC0PhlnP8Kd+mVLLHYpH9VO4rnaSfHE1kBjWkI7U6lLc6ks4flgAgGTS5Bb7pLAjwdWg794GWcnRh6kSUEQd3SftANqQLgCunDcP2Vc4KR9R78zBmEzXH/OPzl/ANgNA6wWO2OoKKy2VrjwVAab6FW15h3Lr6rYIw3KztpG+UMmEj5ReexIjXi/jUptdnUFWspvAmzIl6kwzzF8ExVyT9D75JRuHvmxXKKjyJRxqb8UnSh2JD4JN 0xc0000a2180}
CLIENT|earth|WARN|Encountered unknown host|{fishfinger.buetow.org:2222 0xc0000a0150 0xc000460110 [fishfinger.buetow.org]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNiikdL7+tWSN0rCaw1tOd9aQgeUFgb830V9ejkyJ5h93PKLCWZSMMCtiabc1aUeUZR//rZjcPHFLuLq/YC+Y3naYtGd6j8qVrcfG8jy3gCbs4tV9SZ9qd5E24mtYqYdGlee6JN6kEWhJxFkEwPfNlG+YAr3KC8lvEAE2JdWvaZavqsqMvHZtAX3b25WCBf2HGkyLZ+d9cnimRUOt+/+353BQFCEct/2mhMVlkr4I23CY6Tsufx0vtxx25nbFdZias6wmhxaE9p3LiWXygPWGU5iZ4RSQSImQz4zyOc9rnJeP1rwGk0OWDJhdKNXuf0kIPdzMfwxv2otgY32/DJj6L [46.23.94.99]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNiikdL7+tWSN0rCaw1tOd9aQgeUFgb830V9ejkyJ5h93PKLCWZSMMCtiabc1aUeUZR//rZjcPHFLuLq/YC+Y3naYtGd6j8qVrcfG8jy3gCbs4tV9SZ9qd5E24mtYqYdGlee6JN6kEWhJxFkEwPfNlG+YAr3KC8lvEAE2JdWvaZavqsqMvHZtAX3b25WCBf2HGkyLZ+d9cnimRUOt+/+353BQFCEct/2mhMVlkr4I23CY6Tsufx0vtxx25nbFdZias6wmhxaE9p3LiWXygPWGU5iZ4RSQSImQz4zyOc9rnJeP1rwGk0OWDJhdKNXuf0kIPdzMfwxv2otgY32/DJj6L 0xc0000a2240}
Encountered 2 unknown hosts: 'blowfish.buetow.org:2222,fishfinger.buetow.org:2222'
Do you want to trust these hosts?? (y=yes,a=all,n=no,d=details): a
CLIENT|earth|INFO|STATS:STATS|cgocalls=11|cpu=8|connected=2|servers=2|connected%=100|new=2|throttle=0|goroutines=19
CLIENT|earth|INFO|Added hosts to known hosts file|/home/paul/.ssh/known_hosts
REMOTE|blowfish|100|7|fstab|31bfd9d9a6788844.h /usr/local ffs rw,wxallowed,nodev 1 2
REMOTE|fishfinger|100|7|fstab|093f510ec5c0f512.h /usr/local ffs rw,wxallowed,nodev 1 2
</pre>
<br />
<span>Running it the second time, and given that you trusted the keys the first time, it won't prompt you for the host keys anymore:</span><br />
<br />
<pre>
❯ ./dgrep -user rex -servers blowfish.buetow.org,fishfinger.buetow.org --regex local /etc/fstab
REMOTE|blowfish|100|7|fstab|31bfd9d9a6788844.h /usr/local ffs rw,wxallowed,nodev 1 2
REMOTE|fishfinger|100|7|fstab|093f510ec5c0f512.h /usr/local ffs rw,wxallowed,nodev 1 2
</pre>
<br />
<h2 style='display: inline' id='conclusions'>Conclusions</h2><br />
<br />
<span>It's a bit of manual work, but it's ok on this small scale! I shall invest time in creating an official OpenBSD port, though. That would render most of the manual steps obsolete, as outlined in this post!</span><br />
<br />
<span>Check out the following for more information:</span><br />
<br />
<a class='textlink' href='https://dtail.dev'>https://dtail.dev</a><br />
<a class='textlink' href='https://github.com/mimecast/dtail'>https://github.com/mimecast/dtail</a><br />
<a class='textlink' href='https://www.rexify.org'>https://www.rexify.org</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other related posts are:</span><br />
<br />
<a class='textlink' href='./2023-09-25-dtail-usage-examples.html'>2023-09-25 DTail usage examples</a><br />
<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD (You are currently reading this)</a><br />
<a class='textlink' href='./2022-03-06-the-release-of-dtail-4.0.0.html'>2022-03-06 The release of DTail 4.0.0</a><br />
<a class='textlink' href='./2021-04-22-dtail-the-distributed-log-tail-program.html'>2021-04-22 DTail - The distributed log tail program</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
<p class="footer">
Generated with <a href="https://codeberg.org/snonux/gemtexter">Gemtexter 3.0.1-develop</a> |
served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/relayd.8">relayd(8)</a>+<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> |
<a href="https://foo.zone/site-mirrors.html">Site Mirrors</a>
<br />
Webring: <a href="https://shring.sh/foo.zone/previous">previous</a> | <a href="https://shring.sh">shring</a> | <a href="https://shring.sh/foo.zone/next">next</a>
</p>
</body>
</html>
|
