Documenting log formats

2 files changed, 75 insertions, 48 deletions

diff --git a/doc/logformats.md b/doc/logformats.md
index a9f9865..06fff76 100644
--- a/doc/logformats.md
+++ b/doc/logformats.md
@@ -1,4 +1,72 @@
 Log Formats
 ===========
 
-TODO: document log formats.
+You may have looked at the [DTail Query Language](./querylanguage.md) and wondered how to make DTail understand your own log formats. Otherwise, DTail won't be able to extract information from your logs (e.g. extract fields and variables from your log lines to be used in the query language).
+
+You could either make your application follow the DTail default log format, or you would need to implement a custom log format in Go.
+
+## Current log formats
+
+The following log formats are currently available out of the box:
+
+* `default` - The default DTail log format.
+* `generic` - A generic log format with a very simple set of fields.
+* `generickv` - A simple log format expecting all log lines in form of `field1=value1|field2=value2|...`.
+
+For details, have a look at the implementations at `./internal/mapr/logformat/`.
+
+### Selecting a log format
+
+By default, DTail will use the `default` log format. You can override the log format with the `logformat` keyword:
+
+```shell
+% dmap --files /var/log/example.log --query 'from EXAMPLE select ....queryhere.... logformat generickv'
+```
+
+Alternatively, you can override the default log format via `MapreduceLogFormat` in the Server section of `dtail.json`.
+
+## Log format fields
+
+TODO: Difference between field and variables.
+
+## Log format variables
+
+This is the list of pre-defined variables. Please note that these vary depending on the log format used. 
+
+### Common variables:
+
+The common variables may exist in all log formats.
+
+* `$empty` - The empty string `""`
+* `$hostname` - The server FQDN
+* `$line` - The current log line
+* `$server` - Alias for `$hostname`
+* `$timeoffset` -  Offset of $timezone
+* `$timezone` -  The current time zone
+* `*` - Special placeholder 
+
+### Default log format variables:
+
+These variables may only exist when your logs are in the DTail default log format:
+
+*Date and time:*
+
+* `$hour` - The current hour in format HH
+* `$minute` - The current minute in format MM
+* `$second` - The current second in format SS.
+* `$time` - The current time in format YYYYMMDD-HHMMSS
+
+*Log level/severity:*
+
+* `$loglevel` - Alias for `$severity`
+* `$severity` - The log severity
+
+*System and Go runtime:*
+
+* `$caller` - DTail server caller of the logger
+* `$cgocalls` - Num of DTail server CGo calls
+* `$cpus` - Num of DTail server CPUs used
+* `$goroutines` - Num of DTail server Goroutines used
+* `$loadavg` - 1 min. average load average
+* `$pid` - DTail server process ID
+* `$uptime` - DTail server uptime
diff --git a/doc/querylanguage.md b/doc/querylanguage.md
index c6b9beb..96d0fd1 100644
--- a/doc/querylanguage.md
+++ b/doc/querylanguage.md
@@ -7,9 +7,7 @@ The query language allows you to run mapreduce queries on log files. This page i
 
 For this to work, DTail needs to understand your log format. DTail already understands its own log format. You can have a look at all examples of the [examples](./examples.md) page using `-query` (these would be all examples of the `dmap` command, and some examples using the `dtail` command).
 
-DTail also ships with a generic log format, which only allows you to run very basic queries. Check out the [log formats](./logformats.md) documentation for this.
-
-To implement your own log format, please also check out the [log formats](./logformats.md) documentation.
+DTail also ships with a generic log format, which only allows you to run very basic queries. Check out the [log format](./logformats.md) documentation for this. To implement your own log format, please also check out the log format documentation.
 
 ## The complete language
 
@@ -23,6 +21,7 @@ QUERY :=
     [interval SECONDS]
     [limit NUM]
     [outfile "FILENAME.csv"]
+    [logformat LOGFORMAT]
 SELECT := FIELD|AGGREGATION(FIELD)
 TABLE := The mapreduce table name, e.g. WRITE in MAPREDUCE:WRITE
 AGGREGATION := count|sum|min|max|avg|last|len
@@ -31,56 +30,16 @@ ARG := This is either
     a string: "foo bar"
     a float number: 3.14
     a bareword e.g.: responsecode
-    or a $variable (see below).
+    a field or a $variable
 OPERATOR := This is one of ...
     Floating point operators:
         == != < <= > >=
     String operators:
-        eq ne contains lacks (lacks is the opposite of contains, e.g. 
-                              "not contains")
+        eq ne contains lacks (lacks is the opposite of contains, e.g. "not contains")
 GROUPFIELD := bareword|$variable       
 ORDERFIELD := This must be a AGGREGATION(FIELD) or FIELD which was specified in
               select clause already.
+LOGFORMAT := The name of the log format implementation. It's 'default' by default.
 ```
 
-## Predefined variables
-
-This is the list of pre-defined variables. Please note that these vary depending on the log format used. 
-
-### Common variables:
-
-The common variables may exist in all log formats.
-
-* `$empty` - The empty string `""`
-* `$hostname` - The server FQDN
-* `$line` - The current log line
-* `$server` - Alias for `$hostname`
-* `$timeoffset` -  Offset of $timezone
-* `$timezone` -  The current time zone
-* `* (special placeholder)
-
-### DTail default log format:
-
-These variables may only exist when your logs are in the DTail default log format:
-
-*Date and time:*
-
-* `$hour` - The current hour in format HH
-* `$minute` - The current minute in format MM
-* `$second` - The current second in format SS.
-* `$time` - The current time in format YYYYMMDD-HHMMSS
-
-*Log level/severity:*
-
-* `$loglevel` - Alias for `$severity`
-* `$severity` - The log severity
-
-*System and Go runtime:*
-
-* `$caller` - DTail server caller of the logger
-* `$cgocalls` - Num of DTail server CGo calls
-* `$cpus` - Num of DTail server CPUs used
-* `$goroutines` - Num of DTail server Goroutines used
-* `$loadavg` - 1 min. average load average
-* `$pid` - DTail server process ID
-* `$uptime` - DTail server uptime
+Note, that the available fields and variables vary from the log format used. There is also a subtle difference between a field and a variable. Check out the [log format](./logformats.md) documentation for more information.