Administering Keen

Custom API ingesters

Chapter 24 of 30

Build a custom API ingester

When no built-in adapter collects from a service, the API ingester builder describes the HTTP requests itself: a connection, a request, the records to extract and the fields that make them events. No code is needed.

1. Configure a custom connection with its base URL and supported authentication. Credentials are stored encrypted using KEEN_INTEGRATION_SECRET_KEY; set this key consistently for API and workers and include it in protected backups.
2. Define the request path, query parameters, supported request payload and pagination settings. Start with a bounded page size and a small collection.
3. Select the records to extract and configure their normalised event fields, including stable identity and timestamp information.
4. Test the request, inspect the resulting event, then configure its collection schedule.
5. Create an evidence mapping in Sources & evidence mapping and preview it against the collected events.

Outbound network restrictions, credential handling and request limits still apply, and a remote API's permissions and history determine the evidence available. Custom connection objects appear in the builder's credentials list; built-in sources have their own credential editor under sources & evidence mapping. Keep mapping rules focused on what the event demonstrates; the rules chapter explains revision management, YAML export and historical mapping.

Delete ingesters and connections

In the builder, choose Delete ingester on a card to remove one ingester, or Delete connection beside the saved connection to remove it and its ingesters. Connections can also be removed from sources & evidence mapping.

1. Read the impact preview: the affected ingesters, the source-specific mapping rules and the number of automatic evidence-to-control links.
2. Check that the selected item is the one you intend to remove, type its displayed name exactly, and confirm. Cancel keeps the configuration.
3. Review the refreshed list. Collected events, evidence artefacts, measurements, manual and imported mappings, and shared cross-source rules remain available.