Administering Keen

Keen Agent installation and operation

Chapter 27 of 30

Operate Keen Agent and OTLP collectors

Keen Agent is a small Linux service that reads selected host logs and sends them to Keen as evidence. Getting one running takes three steps: register the collector in Keen, install the agent on the server, and confirm the first events arrive. Everything else in this chapter is detail for when you need it.

Diagram source (Mermaid)

flowchart TB
  L["Selected log files or journal"] --> P["Parse and redact"]
  P --> Q["Atomic local transaction: UUID, record and checkpoint"]
  Q --> T["Send OTLP JSON over verified HTTPS"]
  T --> A{"Valid scoped credential and batch?"}
  A -->|Yes| D{"UUID already stored?"}
  D -->|Same content| OK["Return success"]
  D -->|New| S["Store artifact, event and mappings; commit"]
  S --> OK
  OK --> R["Remove acknowledged records from queue"]
  A -->|Permanent rejection| H["Pause delivery; retain records"]
  D -->|Conflicting content| H
  T -->|Network error or retryable response| B["Back off and retry the same UUIDs"]
  B --> T

Open full-size diagram

The agent is a standalone Go service. It enriches records with patching, HTTP, PHP and Linux Audit context and sends OTLP/HTTP JSON to Keen, which stores the events and their redacted artifacts and applies the existing mapping rules.

Set up agent access

1. Open Admin → Keen Agents and select Register a collector. Choose the credential expiry (1-365 days, default 90).
2. Save the one-time credential somewhere safe; Keen stores only a SHA-256 hash of it.
3. Note the endpoint. Through the standard reverse proxy it is https://keen.example.org/api/v1/otlp/logs; direct API deployments use /v1/otlp/logs. TLS must terminate at a trusted reverse proxy.

For an autoscaling fleet, create an enrollment profile instead (see below). No Fernet integration key is needed for agent credentials; outbound API integration credentials keep their existing encryption mechanism.

Build and install on a Linux server

Keen Agent is maintained in its own repository. Prefer the package and installation instructions shipped with that release; the manual build below shows what a packaged installation does for you. Keep the token, config and spool during an upgrade.

1. Build with the Go version required by the repository's go.mod (dependencies are pinned in go.mod and go.sum), create the service account and install the files:

   cd keen-agent
   make build
   sudo useradd --system --home-dir /var/lib/keen-agent --shell /usr/sbin/nologin keen-agent
   sudo install -m 0755 dist/keen-agent /usr/local/bin/keen-agent
   sudo install -d -o root -g keen-agent -m 0750 /etc/keen-agent
   sudo install -o root -g keen-agent -m 0640 examples/config.yaml /etc/keen-agent/config.yaml
   sudo install -o root -g keen-agent -m 0640 /dev/null /etc/keen-agent/token
   sudoedit /etc/keen-agent/config.yaml
   sudoedit /etc/keen-agent/token
   sudo install -m 0644 packaging/keen-agent.service /etc/systemd/system/keen-agent.service
   

2. Paste the issued credential into the token file as a single line. Set the endpoint and enable only sources that exist on the server.
3. Grant the service account read access to the selected logs. For journald, where the group exists:

   sudo usermod -aG systemd-journal keen-agent
   

4. Check, enable and watch:

   sudo -u keen-agent /usr/local/bin/keen-agent -command check
   sudo systemctl daemon-reload
   sudo systemctl enable --now keen-agent
   sudo journalctl -u keen-agent -f
   sudo -u keen-agent /usr/local/bin/keen-agent -command status
   

check validates configuration, file permissions and CA loading; status reads the latest local report while the service runs. The service reports health to Keen about once a minute. Restart it after changing configuration or replacing a blocked or expired credential.

Configuration and log semantics

examples/config.yaml holds the complete configuration with commented source examples. Each source names a path pattern or journal selection and a parser. Parsing is deliberately conservative: an unrecognised format remains evidence as redacted text.

Delivery, rotation and retention

Every record receives a random UUID and is committed to the local queue together with its file offset or journal cursor in one synchronised bbolt transaction. After Keen acknowledges a batch, those records are removed. A crash before acknowledgement resends the same UUIDs, and Keen deduplicates by registered identity plus UUID while rejecting conflicting content.

The result is at-least-once delivery for records durably collected, with server-side deduplication while their Keen events remain retained. The status file and agent UI show queue bytes, record count, dropped/gap count, source errors, credential expiry and last contact; “Connected” means recent authenticated contact, so also inspect source status and queue growth.

Evidence and framework mapping

The server assigns source=keen-agent and binds system to the administrator-registered name; client-supplied host names remain descriptive attributes. The normalised payload includes source (the configured log source), action, outcome, actor, raw, fields and the stable event ID, and parsed fields are available as structured-field conditions in the rule editor.

Use Admin → Evidence definitions to create rules against these fields for your own framework and controls. Useful criteria include source=keen-agent, action=package.upgrade, action=auditd.SYSCALL or action=http.request with outcome=failure. Rules are not installed automatically for an assumed framework version, so preview them in the rule editor.

Security and privacy

The agent has no inbound listener, remote command channel, plugin loading, template execution or shell-based parser. Its only child process is /usr/bin/journalctl with fixed arguments and validated unit names, and parsing uses bounded inputs and Go regular expressions. Install the binary, configuration, CA files, log directories and token under trusted ownership.

Default redaction covers common password and token assignments, bearer tokens and authentication and cookie headers; redact adds RE2 expressions, applied before the spool, and Keen applies its own redaction and optional privacy masking again before storage. These patterns cannot guarantee removal of every secret or personal datum, so review actual source formats, minimise selected logs and add organisation-specific rules.

OTLP interoperability

The receiver implements the logs signal over OTLP/HTTP JSON at /v1/otlp/logs (or /api/v1/otlp/logs through the UI proxy), so other OpenTelemetry collectors can send to it. Standard collectors must select JSON and override their logs endpoint; Keen-specific enrichment is optional, and examples/alloy.alloy shows a configuration. Keen's OTLP endpoint accepts logs only; Prometheus metrics need a separate conversion into effectiveness-measure submissions.

Keen Agent emits these optional log attributes:

Check agent operation

keen-agent -command check
keen-agent -command status
systemctl status keen-agent
journalctl -u keen-agent --since "1 hour ago"

Run the checks as the agent service user so they use the same file permissions as the service. Confirm that its configuration, credentials and selected logs are readable, review queue size, source health and delivery counters, then open the agent inventory in Keen and inspect a received event: its timestamp, fields, retained artefact and mapped controls.

Enrollment profiles for autoscaling fleets

When machines come and go, an enrollment profile lets each new machine register itself. Open Admin → Keen Agents → Enrollment profiles and create a profile for each cluster or environment: choose the lifetime of the agent credentials, an optional enrollment expiry, an optional total enrollment limit, and optional allowed client CIDRs and fleet labels. Keen shows the bootstrap key once; store it in the fleet's secret manager.

Each enrollment creates an independent agent identity and token, and the agent inventory identifies its profile. Individual servers can continue using Register a collector.

Profile expiry and enrollment limits apply to registration; profile revocation also applies to ingestion and renewal. Revoking a compromised fleet's bootstrap access requires revoking or rotating its key, because an attacker who retains a valid reusable key can attempt new registrations.

Configure keen-agent for an autoscaling machine

Use examples/autoscale.yaml from the agent repository as a starting point. Provision the bootstrap key from your secret manager before starting the service, at /etc/keen-agent/bootstrap-key, owned by root:keen-agent with mode 0640, and configure the agent with:

endpoint: https://keen.example.org/api/v1/otlp/logs
enrollment:
  bootstrap_key_file: /etc/keen-agent/bootstrap-key
state_dir: /var/lib/keen-agent
sources:
  - name: system
    kind: journal
    parser: syslog

The enrollment name defaults to the hostname; an optional enrollment.name sets a stable machine name. Omit token_file in enrollment mode (static servers use it with an administrator-managed token), and configure ca_file when your receiver uses a private certificate authority; HTTPS verification remains enabled.

Run keen-agent -command check, then start the service: it enrolls the machine and saves its identity, token and retry state atomically in /var/lib/keen-agent/identity.json with mode 0600. After successful enrollment the bootstrap key file can be removed from that machine; renewal uses its individual agent token.

Enrollment API for provisioning clients

Provisioning tools can enrol without the agent's built-in support. Use verified HTTPS and POST to /api/v1/agents/enroll with the bootstrap key in the Authorization bearer header and JSON containing name and nonce. Generate the nonce from 32 cryptographically random bytes, encoded as unpadded URL-safe Base64 (43 characters), and persist it in a private file before sending the first request. The response contains agent and token; save them atomically with restrictive permissions and keep agent state across reboots.

Credential renewal and recovery

Enrolled agents POST to /api/v1/agents/renew using their current agent bearer token and a fresh private random nonce, before the credential expires. Persist the renewal nonce before sending the request and atomically replace the token when the response arrives; the old token immediately loses ingestion access.

Network and secret handling

Restrict the enrollment endpoint at nginx, a load balancer or a private network, and prevent direct access around that restriction. Application-level CIDR checks use Keen's trusted-proxy configuration and ignore forwarded addresses from untrusted peers. Enrollment and renewal have fail-closed rate limits backed by Valkey, so clients should honour Retry-After and use bounded retry delays.

Fetch bootstrap keys from a secret manager using the machine's workload identity where available, and keep keys, nonces and operational tokens out of command-line arguments, images and logs. Removing a local bootstrap file does not remove copies held in user-data or provisioning logs. Use separate profiles to limit a leaked key's reach.

Agent health and journal evidence

Agent heartbeats report delivery counters, queue size and source health. Review these in the agent inventory to identify collection failures or delivery backlogs, and use the agent's local status and service logs to investigate a reported problem.

Journal records include available service.name, process.name, process.pid, user.id, group.id, host.name and log.syslog.priority; original metadata remains and is redacted before queuing. Known agent failures, cron command starts and SSH authentication results receive specific actions, and unrecognised messages remain syslog.record. Inspect an event's structured fields when choosing mapping conditions.

Filter before agent delivery

Filtering on the host keeps noisy logs out of Keen altogether. Each source accepts include_when and exclude_when lists, and each condition has a field (a literal parsed field key), an operator and a value.

By default each list uses OR: a record must match any include condition if any are given, and is then discarded on any exclusion match. Set exclude_match: all under the source to require every exclude_when condition (AND); for example, client.ip in 192.0.2.0/24 AND status equals "200" discards only status 200 requests from that network. Include conditions remain OR, empty exclusion lists discard nothing, and a missing field prevents an AND exclusion.