Administering Keen

Elasticsearch

Chapter 11 of 30

Elasticsearch

Keen collects log documents from Elasticsearch by index, alias or data stream pattern, with an optional message filter, so searches you already run can become evidence. The collector targets the Elasticsearch 8.19 and 9.x APIs.

Connect and collect

Set KEEN_ELASTICSEARCH_ENABLED=true on the API, worker and scheduler, and recreate those containers.
In Admin → Sources & evidence mapping → Sources, choose Define a source, select Elasticsearch, and enter a source name and HTTPS base URL, such as https://search.example.org:9200.
Enter either an encoded Elasticsearch API key (without the ApiKey prefix), or a username and password. Give the account read access only to the required indices, aliases or data streams.
Add an input with a distinct name and an index pattern, such as logs-*. The optional message filter uses simple-query-string syntax against the selected message field: "authentication failed" | "access denied" matches either phrase. A blank filter collects every document in the time window.
Confirm the field mappings. Defaults are @timestamp, message, host.name, user.name, event.action and event.outcome.
Choose Collect evidence now, inspect an event, and add a mapping rule for the source or its input. Structured document fields remain available to mapping conditions.

Going deeper: certificates, credentials and field requirements

HTTPS certificate and hostname verification are mandatory; private certificates require a trusted CA. Clear saved credentials for the previous authentication method when switching methods. Comma-separated index patterns are supported. The timestamp must be a single Elasticsearch date or date_nanos field with doc values enabled, and documents must have _source enabled.

Collection windows and deduplication

Scheduled collection runs every 30 minutes, and the first run begins 24 hours back. Each run processes up to four one-hour windows and overlaps the previous completed cursor by five minutes to collect delayed records; delays longer than the overlap can be missed, so an upstream ingestion timestamp helps with delayed log delivery.

Connection, index and document ID identify an event. Repeated documents are deduplicated, including across overlapping inputs on that connection, and the first collecting input supplies the event's input identity. Stored evidence represents the first collected version of a document.

Input limits and retries

Additional input setting	Range	Default
`initial_lookback_minutes`	1-43200	1440
`overlap_minutes`	0-60	5
`window_minutes`	1-1440, greater than the overlap	60
`page_size`	1-1000	500
`max_pages`	1-100	100
`windows_per_run`	1-24	4

Collection has a two-minute connection budget. Partial searches and failed windows retain their previous cursor, and already saved documents deduplicate on retry. If an input reaches its page or time limit, reduce its window or narrow its filter. Changing the pattern, filter or field mappings starts a fresh bounded backfill.

Environment-managed sources

Environment-managed collection uses KEEN_ELASTICSEARCH_BASE_URL, KEEN_ELASTICSEARCH_API_KEY or KEEN_ELASTICSEARCH_USERNAME/KEEN_ELASTICSEARCH_PASSWORD, with a queries list in config/elasticsearch.yml. The same input structure works in named connections supplied through KEEN_SOURCE_CONNECTIONS_FILE. The source type's enabled flag gates every connection.