Keen documentation
Open Keen ↗

Developing Keen

Architecture, conventions and feature switches

Chapter 1 of 6

Architecture and repository map

Keen is a FastAPI application over PostgreSQL, with Celery workers for collection and maintenance, Valkey for sessions and queues, and a static ES-module UI. This chapter maps the repository, follows one request through it, and sets out the conventions and checks a change is expected to meet.

The API uses SQLAlchemy models and Alembic migrations over PostgreSQL. Celery worker and beat use Valkey as broker and backend; Valkey also stores sessions, authorisation versions, caches and notification messages. The UI uses HTML, ES modules, Bootstrap and D3 with shared MOSP design-system helpers.

WhereWhat lives there
services/api/app/main.pyApplication construction, middleware and startup hooks.
api/routes/HTTP adapters; __init__.py assembles the domain routers.
db/models.py, db/session.pyORM models; engine, SessionLocal and the request database dependency. Schema history is in services/api/alembic/versions.
core/config.py, core/managed_configuration.pyEnvironment settings; database-managed configuration.
security/Authentication, sessions, OIDC, roles, permissions, CSRF, rich-text sanitisation and redaction.
ingest/, mapping/Source adapters with shared persistence and SSRF helpers; collection identity and rule evaluation.
services/, worker/, storage/s3.pyDomain operations, changelogs, statistics and scheduling; the Celery application and tasks; artefact storage.
services/ui/publicHTML pages and their pages/*.js controllers. app.js wraps and re-exports the MOSP helpers.
Going deeper: the UI build

The UI Docker build installs the public design-system npm dependency at the commit pinned in package.json and its lockfile, applies the bundled compatibility patch and builds its assets. Keep the pin, lockfile and patch compatible with the application.

Follow a request through the application

  1. The external HTTPS proxy forwards to keen-ui. nginx serves static files and forwards /api/ to keen-api with that prefix removed, so a browser call to /api/v1/events reaches /v1/events in FastAPI. The OpenAPI server prefix reflects the public path.
  2. Middleware establishes request identity and applies permission and CSRF checks before the handler. Authentication uses a short-lived database session and detaches the user before passing the request onward.
  3. Pydantic payload models validate shape and restrict writable fields; the domain operation validates relationships and permissions before writing, using SQLAlchemy or bound parameters, and records semantic changes through the shared entity-changelog service.

Middleware order is reversed. Starlette runs inbound middleware in reverse registration order, so inspect the registration order when changing it. Authentication must populate the request's user before checks that rely on it.

Going deeper: sessions, body limits and what nginx does not do

Handlers use their own get_db dependency. Do not attach a middleware ORM instance to a second session or leave a transaction open across slow external calls. Request audit logs and entity changelogs serve different purposes. The request-size middleware checks the declared length; enforce streaming request-body limits at the proxy alongside it. A body parsed as JSON, whatever its declared type, holds at most 2 MiB outside /v1/admin/ (40 MiB there) and is decoded in a worker thread. An nginx page-hiding rule is not API authorisation.

Frontend conventions

Use the shared API helpers for credentials, errors and CSRF headers. Escape plain values with esc, validate external links with safeExternalHref, render rich text through the shared sanitiser, and give D3 labels their text through the DOM text API. Carry the active framework through relevant queries and links, and avoid treating global objects, such as the underlying risk scenario, as though switching framework creates another object. Keep accessible labels, status messages and keyboard interaction when adding editors.

Develop and verify a change

  1. Identify the authoritative model, route, permission, UI controller and migration for the feature.
  2. Add schema changes through Alembic and update models together. Preserve existing data and relationships.
  3. Validate payloads and enforce permissions at the route and middleware boundaries.
  4. Keep multi-record updates transactional and use version checks where another editor can race.
  5. Record semantic changes, invalidate derived data and bump authorisation versions when relevant.
  6. Update the corresponding UI and manual from the actual behaviour.
  7. Test the consequential boundaries: unauthorised access, concurrent edits, invalid relationships, replay, rollback and historical data handling.

Run the test suite

The CI workflow uses PostgreSQL 16 and Valkey 8. Install the locked API development dependencies with poetry install --with dev --no-root in keen/services/api, and the frontend test dependencies with npm ci --prefix services/ui/tests --ignore-scripts from keen. Set KEEN_TEST_DATABASE_URL to a disposable PostgreSQL database and KEEN_REDIS_URL to the test Valkey instance, then run bash tests.sh from keen. The runner checks backend behaviour, frontend DOM workflows, JavaScript syntax and coverage, and rejects skipped backend tests.

Test database behaviour on PostgreSQL. Use PostgreSQL integration tests for database-specific behaviour, including JSONB, row locking, uniqueness and transaction semantics; Keen supports PostgreSQL only, and its tests run there. For hosted changes, include negative cross-tenant identity and storage checks using separate test tenants.

Rehearse migrations and restores in a disposable test installation. Pin deployment dependencies and record the source and image versions used in a release; the UI image build installs its pinned design-system dependency.

Feature availability and cache behaviour

app/services/features.py defines stable feature keys, defaults, parent dependencies and entity mappings. GET /v1/admin/features returns configured and effective settings; PUT /v1/admin/features accepts an expected version and a changes object of known boolean settings. Both require administrator authority, and normal CSRF, authentication and request limits apply.

DomainRoute applies the shared feature dependency to matched route owners and entity parameters while existing permission checks remain authoritative. /v1/me supplies the effective feature map during bootstrap, the shared frontend feature helper filters navigation and marked elements, and page loaders skip disabled subsections; the frontend checks are presentation only. Every new route or cross-feature operation needs an availability decision and regression coverage alongside its permission inventory entry.

Going deeper: how the setting is stored and cached

The document and its revisions use the existing managed configuration tables. Saving locks the current row and checks the revision, a competing initial insert is rejected by the primary key, and successful writes commit the document and revision together and invalidate the local compiled cache. Each new guarded request reads the authoritative database revision, and a worker reloads the bounded immutable document only when its revision changes. The cache is scoped to its database bind and contains no account identity or permission decisions; database and configuration errors fail closed. This avoids a stale time-to-live window and adds one small indexed revision lookup per guarded request, however many features it checks. Public metric ingestion checks availability after its normal rate-limit and credential checks, domain operations separately guard sampling, new scope associations and scheduled work, and existing historical audit references are retained.