Administering Keen

Upgrades, backups and troubleshooting

Chapter 30 of 30

Upgrade safely

Upgrades, backups and diagnosis follow the same principle: know your rollback point before you change anything, and check one representative record of each kind afterwards.

Record the current image versions, database revision and configuration. Take a database backup and protect the corresponding artefacts, encryption material and environment secrets.
Restore that backup in a test environment and rehearse the upgrade there first.
Quiesce ingestion and writes for the window where a consistent rollback point is required.
Build or pull the reviewed release, run alembic upgrade head once, start the API/UI and worker, and start beat after checking the migrations.
Verify sign-in, a framework node, an existing risk, a document version, a mapped event and an artefact download.

Rolling back. A container rollback is safe only if the earlier application understands the migrated schema. Otherwise restore the pre-upgrade database and compatible object state into an isolated recovery environment. Do not run an Alembic downgrade against production without a reviewed data-loss analysis.

Going deeper: routine Compose commands

For routine service inspection use docker compose ps and scoped logs. docker compose stop preserves volumes; docker compose down -v removes them and is destructive, so use stop/start for a routine restart. Avoid printing the fully resolved Compose configuration into shared logs because it can contain secrets.

Backups and restore

Back up PostgreSQL, all referenced artefact storage, configuration and secrets, and the release identifiers needed to rebuild the runtime. Use an application-consistent database procedure and encrypted destinations with access controls independent of normal application access.

For the standard self-managed Compose service, a custom-format PostgreSQL dump can be created as follows. Store the output on a protected backup destination and monitor the command's exit status.

umask 077
docker compose exec -T postgres sh -c 'pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Fc' > keen-postgres.dump

Test a restore

Restore into a separate, empty recovery database with the matching artefacts and keys and the compatible application release. Disable outbound notifications and ingestion until validated, check representative records and artefact checksums, and keep production intact until the recovery copy is verified and a cutover is agreed. Record the last successful backup and the last successful restore test separately.

Going deeper: artefacts, keys and lifecycle policies

For S3, inventory object versions and retention dates; for local artefacts, include the shared artefact volume in the backup procedure. Preserve the metadata needed to resolve each stored artefact reference. Encryption is only recoverable if the corresponding keys and permissions remain available.

A retention period and a deletion policy are different. Object Lock prevents premature removal; expiry does not itself erase every retained version. Define and test lifecycle and deletion procedures, including noncurrent versions, legal holds and backup copies.

Diagnose problems in order

Symptom	Check
API fails at startup	Its logs for invalid settings, database connectivity and migration errors; that the configured database exists and the credentials match.
Worker or beat cannot connect	`KEEN_REDIS_URL` and the private service network.
UI loads but requests fail	The browser-visible error and the nginx and API logs. Browser API URLs begin `/api/v1/`; the API receives `/v1/`. Check proxy rewrites, TLS forwarding and cookie settings. A 401 concerns authentication; a 403 can concern permission or CSRF. Never resolve either by disabling the control.
OIDC fails	The exact issuer, client, callback URL, JWKS and host origin; clock accuracy; whether a prior login state was already consumed. In hosted mode, confirm the correct owner subject without exposing ID tokens or client secrets in logs.
Ingestion is disabled	The adapter flag.
Ingestion runs without evidence	Its collection items, credentials, upstream permissions, filters and cursor.
Evidence arrives but does not map	Preview the definition against that exact event; check collector identity, field values and existing framework targets.
S3 AccessDenied	The runtime principal, bucket, prefix, encryption key and any required conditional-write headers. In hosted mode a denial against another customer's resource is expected. Never add wildcard permissions.
Stale UI behaviour or counts	The deployed UI image and asset cache before changing data; the aggregation and cache refresh path for counts.
Failed mail or outgoing webhooks	The task result and destination configuration; avoid repeated notifications while diagnosing.

Routine assurance checks

Review backup success and capacity daily according to the service process. Investigate failed collections, missing metrics and expired credentials. Periodically test restoration, customer isolation, owner account recovery and alert delivery, and review unused accounts, group permissions and upstream integration privileges.

Use the entity changelog for content changes and the request log for API activity; protect those logs and minimise sensitive request data. Retain operational evidence of maintenance, incident response and restore testing, so your own hosting service can be assessed with the same discipline as its customers' systems.

Prepare your workspace

Choose the frameworks relevant to your organisation in Admin → Frameworks. Configure source connections and inputs in Sources & evidence mapping, collect evidence, and add mapping rules for the controls it supports. Then review access permissions, evidence retention and backup procedures for your installation.