Using Keen

Map structured log fields

Chapter 5 of 17

Map structured fields from agent events

Agent and OTLP events carry structured fields such as the process name, the HTTP status or the log file. This chapter shows how to write rules against those fields, which operator each situation needs, and two worked examples for nginx logs.

In Admin → Sources & evidence mapping, select New mapping and Keen Agent / OTLP logs. Choose a sample event, then add field conditions. Field labels follow the JSON keys you see on the event: fields → process.name means the literal process.name key inside fields.

Evidence	Condition
All package activity	Action regex `^package\.`
Agent's own journal messages	`fields → SYSLOG_IDENTIFIER` equals `keen-agent`, or use `fields → _SYSTEMD_UNIT` equals `keen-agent.service` in a separate rule.
Installed packages	Action `package.status` and `fields → package_status` equals `installed`.
SSH session messages	`fields → SYSLOG_IDENTIFIER` equals the observed program, such as `sshd-session`. Inspect the message before choosing a more specific condition.
Cron invocations	When emitted by the parser, action `scheduled_job.started`; optionally constrain `fields → job.command`.

Going deeper: which “source” is which

The outer Keen source is keen-agent. Inside the normalised payload, source keeps the configured agent input, such as system or dpkg, and System identifies the registered agent. The original JSON is your guide to the values available to a rule.

Choose the operator each condition needs

Every condition pairs a field with an operator. Conditions join with and, so every filled condition in a rule must match, and text comparisons are case-sensitive.

Operator	Matches when
equals	The field holds exactly this text.
starts with	The field's text begins with this text.
contains	The field's text includes this text, read literally (no regex).
does not contain	The field exists, holds a non-null scalar value and lacks this text. A missing field never matches, and an empty exclusion matches nothing.
matches regex	The pattern is found within the value; see the next section.
exists	The key is present, whatever its value.
is in IP network (CIDR)	A client address falls inside a network such as `192.0.2.0/24`; see Client IP evidence conditions.

Write regex conditions

Matches regex searches within the selected value. It does not need to match the whole value, and it is case-sensitive by default. A pattern such as ^(?:gills|systemd)$ on fields → process.name matches those unit names exactly.

Enter the pattern without surrounding slashes.
A missing field or a non-scalar value does not match a regex condition.
Patterns follow Keen's shared rules: a 512-character limit, restricted constructions and bounded execution. The editor rejects a pattern that would run the matcher for too long; simplify it and try again.

Work in the focused rule editor

The editor opens as a focused view, and Back to mappings returns to the catalogue. A new rule starts with one optional row, Summary / contains. Summary refers to the event's displayed summary, so the wording there is what your text sees.

A blank Summary row adds no restriction, so a rule can be driven entirely by payload conditions.
From the Summary row you can enter text, change the operator (for example to matches regex), choose another field, or remove the row. A rule saved with a regex in Summary reopens as Summary / matches regex.
More filters holds the less-used exact and regex filters, and a collapsed regex guide sits below the editor steps.
A dotted field name stays one literal key; payload fields keep their literal JSON paths.

Preview, then save and apply to history

Use Preview to check the actual matches against collected evidence, choose the framework targets, and save. The save confirmation disappears after a few seconds; when a historical mapping job follows, a separate confirmation reports when it finishes. Existing evidence keeps its current mappings until you apply the saved rule to history from the rule catalogue.

Assess each record by what it shows. A cron invocation records a start, so assess completion with a result record. Package activity supports maintenance evidence; patch SLA coverage needs a defined population, a reporting period and an effectiveness measure.

Map nginx request evidence

A worked example: evidence of requests for a sensitive path on a web server.

Select source Keen Agent / OTLP logs and action http.request, then add:

HTTP status equals 404
Request path equals /.aws/config (or starts with /.aws/)
Log file equals /var/log/nginx/access-example.org.log.1; to include rotated and current files, use Log file starts with /var/log/nginx/access-example.org.log.

Every filled condition must match. Leave Outcome blank to include all reported outcomes.

Going deeper: what a 404 does and does not show

A 404 is evidence of the HTTP response to a request; on its own it does not prove that a security control prevented an attack. The agent parses standard nginx access-log timestamps and records the HTTP method, and query strings are removed from the normalised path.

Client IP evidence conditions

Nginx common and combined log records parsed by the agent include a Client IP, so you can scope evidence to an address or a network.

In a mapping definition, select Client IP with equals for one address, or is in IP network (CIDR) for a network such as 192.0.2.0/24 or 2001:db8::/32. Combine it with HTTP status, Request path and Log file; every condition must match.

Going deeper: where the address comes from

A Client IP condition matches events that contain the client.ip field. The address is the logged remote address; your nginx trusted-proxy configuration determines whether that represents the original client.