mirror of
https://github.com/elastic/kibana.git
synced 2025-06-27 18:51:07 -04:00
[Streams] Introducing the new Streams plugin (#198713)
## Summary This PR introduces the new experimental "Streams" plugin into the Kibana project. The Streams project aims to simplify workflows around dealing with messy logs in Elasticsearch. Our current offering is either extremely opinionated with integrations or leaves the user alone with the high flexibility of Elasticsearch concepts like index templates, component templates and so on, which make it challenging to configure everything correctly for good performance and controlling search speed and cost. ### Scope of PR - Provides an API for the user to "enable" the streams framework which creates the "root" entity `logs` with all the backing Elasticsearch assets - Provides an API for the user to "fork" a stream - Provides an API for the user to "read" a stream and all of it's Elasticsearch assets. - Provides an API for the user to upsert a stream (and implicitly child streams that are mentioned) - Part of this API is placing grok and disscect processing steps as well as fields to the mapping - Implements the Stream Naming Schema (SNS) which uses dots to express the index patterns and stream IDs. Example: `logs.nginx.errors` - The APIs will fully manage the `index_template`, `component_template`, and `ingest_pipelines`. ### Out of scope - Integration tests (coming in a follow-up) ### Reviewer Notes - I haven't implemented tests beyond a unit test for converting the filter conditions to Painless. I wanted to get a PR up so we can start iterating on the interface and functionality before we invest in testing. - You might need to add `server.versioned.versionResolution: oldest` to your `config/kibana.dev.yaml` to play with the requests below in the Kibana "Dev console". ### Example API Calls Enable the root stream (and set the mapping for the internal `.streams` index) ``` POST kbn:/api/streams/_enable ``` Read the root entity "logs" ``` GET kbn:/api/streams/logs ``` Fork the "root" entity "logs" and create "logs.nginx" based on a condition ``` POST kbn:/api/streams/logs/_fork { "stream": { "id": "logs.nginx", "children": [], "processing": [], "fields": [], }, "condition": { "field": "log.logger", "operator": "eq", "value": "nginx_proxy" } } ``` Fork the entity "logs.nginx" and create "logs.nginx.errors" based on a condition ``` POST kbn:/api/streams/logs.nginx/_fork { "stream": { "id": "logs.nginx.error", "children": [], "processing": [], "fields": [], }, "condition": { "or": [ { "field": "log.level", "operator": "eq", "value": "error" }, { "field": "log.level", "operator": "eq", "value": "ERROR" } ] } } ``` Set some processing on a stream and map the generated field ``` PUT kbn:/api/streams/logs.nginx { "children": [], "processing": [ { "config": { "type": "grok", "patterns": ["^%{IP:ip} – –"], "field": "message" } } ], "fields": [ { "name": "ip", "type": "ip" } ], } } ``` Field definitions are checked for both descendants and ancestors for incompatibilities to ensure they stay additive. If children are defined in the `PUT /api/streams/<name>` API, sub-streams are created implicitly. If a stream is `PUT`, it's added to the parent as well with a condition that is never true (can be edited subsequently). `POST /api/streams/_resync` can be used to re-sync all streams from their meta data in case the Elasticsearch objects got messed up by some external change - not sure whether we want to keep that. Follow-ups * API integration tests * Check read permissions on data streams to determine whether a user is allowed to read certain streams --------- Co-authored-by: Joe Reuter <johannes.reuter@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This commit is contained in:
parent
fb71f4e027
commit
b86dc8102a
62 changed files with 2419 additions and 0 deletions
|
@ -897,6 +897,10 @@ routes, etc.
|
|||
|The stack_connectors plugin provides connector types shipped with Kibana, built on top of the framework provided in the actions plugin.
|
||||
|
||||
|
||||
|{kib-repo}blob/{branch}/x-pack/plugins/streams/README.md[streams]
|
||||
|This plugin provides an interface to manage streams
|
||||
|
||||
|
||||
|{kib-repo}blob/{branch}/x-pack/plugins/observability_solution/synthetics/README.md[synthetics]
|
||||
|The purpose of this plugin is to provide users of Heartbeat more visibility of what's happening
|
||||
in their infrastructure.
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue