Using Keen

Collect evidence

Chapter 3 of 17

Connect sources and collect evidence

This chapter is about getting records into Keen. It introduces the four ways evidence arrives, walks through the one sequence every source follows (connect, collect, inspect, map), and then covers the guided API ingester builder for services without a built-in adapter. Setting up sources is administrator work; understanding where evidence comes from helps everyone who reviews it.

Evidence reaches Keen by four routes, and each one produces ordinary events you can map to controls:

• Built-in adapters pull from services such as Loki, Jenkins, GitHub, Forgejo and RSS feeds.
• Custom API ingesters, built in the web interface, pull from any HTTPS/JSON service.
• Keen Agent and OTLP collectors push server logs and activity.
• Webhooks accept records pushed by other systems, with a shared secret.

Diagram source (Mermaid)

flowchart TB
  subgraph Collection["Choose a collection method"]
    A["Built-in adapters: Loki, Jenkins, RSS and others"]
    B["Guided API integrations"]
    C["Keen Agent or OTLP collector"]
    D["Authenticated webhooks"]
  end
  A --> N["Validate, normalize and redact"]
  B --> N
  C --> N
  D --> N
  N --> E["Event and provenance in PostgreSQL"]
  N --> S["Evidence artifact in local or S3 storage"]
  E --> R["Mapping rules select matching events"]
  S --> V["Artifact reference and SHA-256"]
  V --> E
  R --> F["Framework controls"]
  F --> U["Review, questions and audit samples"]

Open full-size diagram

Connect, collect, inspect, map

Whatever the source, the same four steps bring it into service. Do them with one small input first; widen later.

1. Connect. Give Keen access to the upstream service. Built-in adapters use credentials you save on the source; the API ingester builder uses reusable encrypted connections.
2. Collect. In Admin → Sources & evidence mapping → Sources, define the server or account, then add the users, organisations, repositories, feeds or queries to read. For a custom API, configure and publish an ingester in the builder.
3. Inspect. Run a small collection and open one event. Check its source, timestamp and payload look the way you expect.
4. Map. On that event choose Create mapping from this event, pick the controls and preview the rule. A rule set to All collected events (any collection) applies across every input of the source.

Each input in the Sources list offers Add mapping rule, which opens the mapping editor with that source and input already selected.

Bring a connector into service

A careful first rollout saves time later. This is the checklist administrators follow for a new adapter.

1. Choose one adapter and one small upstream input. Confirm outbound connectivity and TLS validation from the runtime, and grant a dedicated upstream identity access to that input only.
2. Set the adapter's enable flag and connection credentials in the server environment, then restart the affected services according to your deployment process.
3. Create the collection input and evidence definition in Admin → Sources & evidence mapping, and run the adapter from Admin → Imports & adapters.
4. Open the resulting event and artefact. Verify timestamp, source, system, actor, action, outcome and collection identity. A run that returns zero events may simply have found no new data.
5. Preview the definition against one positive and one negative example. Confirm the intended controls exist in the chosen framework and that other inputs cannot match by accident.
6. Enable routine scheduling and watch the next run. Note the expected lookback, the upstream quota and what a failure looks like.

Adapter-specific settings for Loki, CloudWatch Logs, GitHub, Forgejo, Jenkins, Taiga, BookStack, RSS and Google Workspace are in Administering Keen. A query or job you can run in a browser does not by itself give the collector access; grant it explicit permission to read, for example, the intended CloudWatch log groups.

Know which configuration is authoritative

Most configuration lives in the web interface and in PostgreSQL. Framework metadata and nodes, evidence definitions and managed collection documents are edited in the web editors, and the database keeps their revisions. Keep bootstrap files as recovery material, and back up the database to keep current edits.

Some settings stay server-side: specialist-adapter credentials, adapter enablement and deployment settings. The API ingester builder stores reusable connection credentials encrypted in PostgreSQL, while its destination policy and encryption key are deployment settings.

Build integrations in the web interface

Admin → Integrations is a guided builder for HTTP/JSON collectors. You add a reusable connection, choose a template or Custom JSON API, and describe the request, the records to extract, the field mappings and the schedule. Starter recipes exist for GitHub repository events, GitLab project events, Jenkins builds and BookStack page metadata. The specialist adapters stay available under Imports & adapters.

1. Connect. Provide an HTTPS base URL and the authentication: none, bearer, Basic, custom header, query parameter or OAuth client credentials. Credentials are encrypted and stay on the server; a blank credential field keeps the existing secret.
2. Request. Set a relative endpoint, GET or POST, query parameters and optional headers or body. Use read or search endpoints: a POST preview still calls the upstream service.
3. Evidence. Point at the JSON array with a pointer such as /results. Map a stable id, a timezone-aware timestamp and a summary. Actor, system, action, outcome, severity and source URL are optional. Constants and bounded transforms are available; no scripts run.
4. Schedule. Choose single response, page, offset, cursor, next-URL or Link-header pagination, with parameter names, page size and interval. Optional incremental collection sends an ISO timestamp in the configured since parameter.
5. Test. Save a draft and preview its first page (up to ten records). The preview shows normalised evidence and framework matches without storing anything or moving the cursor. You can also paste sample JSON to test field extraction without a network request.
6. Publish. Publish and schedule the draft. Later edits stay draft-only until you publish again. Configure recognition and framework targets for the collector in sources & evidence mapping.

Operate and recover integrations

Once an ingester is published it runs on its schedule, one job at a time. If a scheduled run fails it is tried again at the next interval; after three failed runs it pauses until someone resumes it. Before resuming, look at the last error, the worker and broker health, and the connection.

Evidence is stored as it arrives, and the cursor advances only after a successful full run. A retry deduplicates stable record IDs and completes missing measurements, so an interrupted run does not leave gaps or duplicates.

Collect measured effectiveness evidence

An ingester can also feed an effectiveness measure. In the builder, expand Collect effectiveness measurements and select Evidence and a measured value. Choose an existing measure, use its exact unit, and map value, period_start and period_end (as YYYY-MM-DD) alongside the usual evidence fields.

Keen compares each reported value with the measure's threshold, and every observation links back to its source event with the collector's revision and run. Before relying on the automation, inspect the measure's observations and a passing and a failing example.

Collect server evidence with Keen Agent

An administrator registers a server in Admin → Integrations → Manage agents, installs the agent with its own credential and selects the logs it may send. Examples of what arrives: package changes, PHP errors, HTTP requests and Linux Audit events.

In Events, filter by the Keen Agent source. The system is the registered server name. Each event shows the activity, result, the original redacted message and the parsed fields. Queue and source health appear in Manage agents; a growing queue or stale contact needs a look. Installation and troubleshooting are in Administering Keen.

Remove a custom API ingester or connection

In the API ingester builder, choose Delete ingester on an ingester card, or Delete connection beside a saved connection. Review the affected items and type the displayed name to confirm. Deleting a connection also removes its ingesters; deleting one ingester keeps the connection for reuse. Collected evidence stays available either way. Deletion details cover rules, mappings and run history.

Set up a source connection

Three words recur in the Sources screen. A source type is an adapter, such as Loki, GitLab or Forgejo. A connection is one endpoint or account with its credentials. Inputs choose what to collect through that connection: queries, projects, users or feed URLs.

1. Open Sources & evidence mapping → Sources and choose Define a source. Only source types enabled in the deployment are offered.
2. Enter a name, endpoint and credentials, and save.
3. Choose Add input to pick a user, organisation, repository, query or other supported input.
4. Choose Add mapping rule on the input to connect its evidence to controls.

Use Edit source to change an address or credentials. Events show which connection captured them, and deleting a connection keeps its collected evidence and recorded identity.

RSS: one source per feed

Choose Define a source → RSS, enter a name, the full feed URL and any optional authentication or headers, and save. Saving configures the feed; there is no separate input step. Then use Collect now to fetch evidence or Add mapping rule to link it to controls.

Each feed keeps its own credentials and collection progress, and credentials are sent only to the feed server's origin. For another feed, create another RSS source. File-managed sources and older sources that hold several feeds keep their configured inputs.