Keen documentation
Open Keen ↗

Developing Keen

Frameworks, rules and field conditions

Chapter 3 of 6

Frameworks are application data

Frameworks, their nodes and the mapping rules that attach evidence to them are rows in PostgreSQL, edited through the application. This chapter covers the framework editor, evidence definitions and the shared condition evaluator.

api/routes/frameworks.py exposes framework listing and upsert and the administrator node editor; the UI controller is pages/framework-editor.js embedded in admin.html. It manages framework metadata and Clause, Control and Custom nodes with descriptions, upstream URLs, ordering, scope and parent relationships.

Two representations of a clause. ControlItem identities include framework, kind and reference, and clause nodes also maintain first-class FrameworkClause records used by clause views and ControlClauseLink relationships. Keep both consistent when modifying the editor, and keep slugs and node kind/reference stable.

Going deeper: deletion, seed data and provenance

Parents must belong to the appropriate framework or category and form a valid hierarchy. Deletion checks children, clause links and mappings before allowing removal; extend dependency handling deliberately when adding another relationship type.

The release migrations 0001_release_schema.py and 0002_release_seed.py define the schema and seed the bundled catalogues. The frozen OSA snapshot supplies 96 of the 102 frameworks; Keen Assurance, DVSTF, Cyber Essentials and Risk Ledger supply four more, alongside the two ISO 9001 editions. Bundled JSON provides deterministic migration input, everyday catalogue edits use database records through the web editor, and seed logic must preserve administrator edits. Upstream publication links identify authoritative text and its usage rights.

Evidence definitions and managed configuration

An evidence definition combines a source adapter, a collection input, matching conditions and framework targets. The API implementation is api/routes/managed_configurations.py; the editor is pages/evidence-config.js within Admin → Sources & evidence mapping.

ManagedConfiguration stores named documents and their versions and ManagedConfigurationRevision stores prior documents. core/managed_configuration.py reads a database document first and falls back to a shipped input only when no database document exists; the migration chain initialises these documents so migrated installations use PostgreSQL as the working source of configuration.

Input and rule publish together. PUT /v1/admin/evidence-definitions/{rule_id} validates the adapter, input section, stable collection key, rule identity and targets. For a collection-specific definition it locks configuration documents in a consistent order, checks both submitted versions and commits the input and rule revisions in one transaction; a 409 tells the caller to reload and reconcile. Publishing a query without its matching rule is a partial update.

Going deeper: identities, catalogue and credentials

Source-wide definitions use the selected source without a collector condition; collection-specific definitions obtain their collector identity through shared mapping helpers. Source identity and collection identity are distinct, because several queries can emit events with the same adapter source. The source catalogue exposes enablement and observed activity, and the BookStack catalogue makes server-side requests using the selected connection. Input and adapter JSON panels reject secret-like content; the connection editor stores credentials encrypted in the database, and environment and deployment-file connections are also supported.

Rule parsing, matching and preview

mapping/rules.py parses the managed rules document and evaluates events. Conditions include exact source, system, actor, action, outcome, severity and label values, supported regex fields, collection identity and BookStack-specific selectors; filled conditions combine as AND, and label matching also considers values extracted from supported payload structures.

Rules contain IDs, enabled state, conditions, targets and confidence; targets identify a framework and reference, and one rule can map to several frameworks. Admin validation checks regex and target validity while the common parser is shared with runtime evaluation, so keep preview and ingestion semantics aligned when adding a field. The editor supports sampled preview and an explicit predefinition path when no events exist; suggested controls remain proposals for an administrator to accept.

Structured field conditions

The shared evaluator applies when.fields in ingestion, preview and historical mapping. All conditions are ANDed, and paths contain literal dictionary keys with no expression execution or array traversal.

when:
  source: keen-agent
  fields:
    - path: [fields, _SYSTEMD_UNIT]
      operator: equals
      value: keen-agent.service
RuleDetail
Operatorsequals, starts_with, contains, exists
KeysA key containing a dot stays one key: ["fields", "service.name"]. Missing fields fail matching.
ValuesScalars compare as text; booleans and numbers use JSON notation.
Limits20 conditions, eight keys per path, 256 characters per key and 4,096 per match value. Invalid conditions fail validation.

Heartbeat counters are optional typed fields; unknown keys remain rejected.