Developing Keen
Ingestion, evidence and artefacts
Chapter 4 of 6
Ingestion, artefacts and historical mapping
Every collector, however different its upstream, ends in the same place: a normalised event with provenance, a redacted artefact with a digest, and the mapping rules that decide what the event supports. This chapter covers that shared path and the jobs that re-run rules over history.
Pull adapters and incoming webhooks normalise source data into events. ingest/common.py handles shared persistence, deduplication, redaction and artefacts; stable source and external identifiers prevent duplicate events, and collection provenance must survive normalisation so a collection-specific rule can identify its own evidence.
Keep outbound requests behind the shared checks: URL safety, TLS validation and restricted redirects. Administrator-controlled URLs can still reach dangerous network locations if SSRF protection is weakened. Webhooks must preserve secret validation, replay controls and fail-closed rate limits.
Historical mapping jobs
Saving a rule can queue historical evaluation. Rule-backfill jobs reconcile the rule's owned mappings against the saved definition, adding matches and removing obsolete links while preserving manual, imported and other supported mappings; legacy mappings with uncertain ownership remain for explicit review. Records persist progress, worker tasks process jobs, beat schedules recovery, and completion is tracked separately from the configuration transaction.
Going deeper: artefact references, remap and statistics
Artefacts have stored references and integrity metadata. In the hosted integration, S3 writes are conditional and stored references include object version information for retrieval; the runtime uses its tenant role and must not accept a reference to an arbitrary bucket, and version information must survive download.
The broader remap action is additive and can include already-mapped events, while rule-specific backfill reconciles mappings with exact rule ownership. The rule editor uses the resumable backfill workflow. Test rule tightening, concurrent edits and cancellation alongside creation. Control-evidence statistics and caches are maintained separately from event storage, so a discrepancy can arise from stale derived data as well as missing evidence; use the shared statistics service when adding an endpoint.
Ingester contracts and evidence semantics
All collectors supply an event timestamp, source, system, actor, action, outcome, severity, summary, provenance and normalised payload to the evidence persistence layer. action is the activity recorded by the producer and outcome is the producer's result label; the mapping engine compares selected conditions against those fields. Exact conditions are case-sensitive and combine with AND, regex conditions are an advanced option, and there is no universal outcome enumeration.
Diagram source (Mermaid)
flowchart TB
subgraph Collection["Choose a collection method"]
A["Built-in adapters: Loki, Jenkins, RSS and others"]
B["Guided API integrations"]
C["Keen Agent or OTLP collector"]
D["Authenticated webhooks"]
end
A --> N["Validate, normalize and redact"]
B --> N
C --> N
D --> N
N --> E["Event and provenance in PostgreSQL"]
N --> S["Evidence artifact in local or S3 storage"]
E --> R["Mapping rules select matching events"]
S --> V["Artifact reference and SHA-256"]
V --> E
R --> F["Framework controls"]
F --> U["Review, questions and audit samples"]
Adding an integration
Inspect real records, choose stable activity and result values and document their meaning, and populate a result only when the source supports the claim. Guided API integrations extract JSON-pointer values or constants and can apply explicit value translations; built-in adapters define their normalisation in code or managed collection settings; the agent's parsers define their labels in keen-agent/cmd/keen-agent/parser.go.
Going deeper: how the rule editor presents these fields
The user-facing rule editor starts with a real event. Its “What happened?” and “What was the result?” labels retain the underlying action and outcome keys, datalists show recent observed values, blank optional fields permit any value, and preview includes concrete explanations of exact-field mismatches. Extra predicates remain available for specialised rules. Event selection and mapping never execute the named action.
Connection-scoped ingestion
The app.ingest.connections module resolves named source configurations and establishes a context-local scope for adapter runs; adapter settings and input loading use this scope. Credentials remain encrypted in source_connections until the selected connection runs, and missing credentials on a named connection remain empty.
Event persistence attaches the trusted connection_id and connection_name from the execution scope; upstream payloads cannot select a connection identity, and names are snapshots retained with historical evidence. Mapping conditions support connection_id alongside source and collector, backfill reconciles previously owned mappings after scope changes, and PostgreSQL advisory locks prevent overlapping polling runs for one connection.