kibana/x-pack/plugins/fleet/dev_docs/integrations_overview.md

Overview of Integrations

Elastic Agent integrations are mechanisms for installing assets that provide an opinionated "out of the box" experience for ingesting data from various pieces of software, third party services, and even internal Elastic products. Fleet provides an interface for installing and managing these integrations. Integrations tell Elastic Agent where and how to retrieve data, and how it should be ingested into Elasticsearch.

In terms of what an integration actually is, it's a set of mostly YML files in a particular directory structure with particular naming conventions. These conventions are captured in a specification maintained by Elastic called the package spec.

This document will detail some of the key files and fields defined by the package spec that we'll commonly deal with in Fleet, and will attempt to provide an informed mental model around integrations. We'll use Fleet's integration policy editor UI to show how integrations' configuration files are translated into the Fleet interface we present to users.

References

• https://github.com/elastic/package-spec
• https://github.com/elastic/integrations/tree/main/packages

Top level manifest.yml

🔗 Package spec reference

• Basic metadata for the integration like name, title, version, description, categories, etc
• Kibana compatibility spec
• Screenshots and icon assets

policy_templates

🔗 Package spec reference

The top level manifest.yml file defines a set of policy_templates which define a grouping of fields used to configure the integration. These policy_templates control what’s rendered in Fleet UI’s “policy editor” when creating or editing integration policies.

Most integrations will only specify a single policy_template as they don’t require this level of structure. For integrations that export many “sub-integrations,” however, they rely on defining multiple templates.

For example, Nginx defines only a single policy_template - aptly named nginx:

# https://github.com/elastic/integrations/blob/main/packages/nginx/manifest.yml
policy_templates:
  - name: nginx
    title: Nginx logs and metrics
    description: Collect logs and metrics from Nginx instances
    inputs:
    # ...

This results in a policy editor UI that looks like this:

In contrast, the AWS integration defines many policy_templates, as it allows users to configure policy values for many distinct services like EC2, S3, or RDS. See the policy_templates definition for AWS here.

The AWS policy editor has a section for each provided policy_template value, e.g.

Each policy template is also exposed as its own distinct integration, and can be installed or managed separately as if it were a first-class integration:

inputs

🔗 Package spec reference

Each policy_template entry defines a set of inputs. An input is essentially a named grouping of fields related to a particular type of data to be ingested. Each input declares a type that typically maps to a Beats input like httpjson or filestream. Elastic Agent manages Beats under the hood, so these inputs are how we can configure the individual Beats that Agent is managing.

For example, the Nginx integration defines three inputs:

1. logfile
2. httpjson
3. nginx/metrics

These inputs appear in the policy editor UI as separate fieldsets, e.g.

Each input can define a list of vars that will allow a user to configure variables via form fields in the Fleet policy editor UI.

For example, on the Nginx integration's httpjson input, there are several "input level variables" configure, which are rendered as below:

These input level variables are defined in the Nginx integration here.

It's important to note that integration also define variables at the data stream level, which we'll get into later in this doc. In the screenshot above, for example, the section beneath the "Settings" section of the httpjson input contains data stream variables for the streams configured as part of the httpjson input.

data_stream directory

Each integration includes a data_stream directory that contains configuration for, well, data streams.

Data streams are a construct in Elasticsearch designed for storing "append-only" time series data across multiple backing indices. They give Elastic Agent an easy and performant way to ingest logs and metrics data into Elasticsearch.

Inside of the data_stream directory, each data stream is defined as its own directory. For example, the Nginx integration's data_stream directory contains three data streams:

1. access
2. error
3. substatus

Elastic Agent data streams conform to a naming schema of {type}-{dataset}-{namespace}. The type of a data stream is either logs or metrics. The dataset of a datastream is controlled in most cases by an interpolation of the integration's name field and the directory name containing the data stream's manifest.yml file. The namespace value is provided by the user for the purpose of grouping or organizing data.

Each data stream directory contains a manifest.yml file that controls the configuration for the data stream, as well as Elasticsearch/Kibana assets, configuration for Agent, and more. Generally, though, the manifest.yml file is the most critical one in a given data stream directory.

In a data stream's manifest.yml file, the integration defines a list of streams. Each item in that streams list is tied to a single input, and defines its own list of vars that controls the set of form fields that appear in the policy editor UI.

For example, the Nginx integration's access data stream defines an entry in its streams list tied to the logfile input mentioned above that includes variables like paths and tags. The variables appear as form fields in the policy editor UI as below:

The "Nginx access logs" fieldset we see in the policy editor UI is defined here in the data_streams/access/manifest.yml file within the Nginx integration. It's input value is set to logfile, so it appears under the "Collect logs from Nginx instances" section in the policy editor. The title and description values control what appears on the left-hand side to describe the fieldset.

Data stream assets

Each data stream directory may contain an elasticsearch or kibana directory that contains assets (like ingest pipelines or Kibana dashboards) for a given stack component. These assets are usually YML or JSON files that Fleet passes off to the corresponding stack component to install. For example, the Nginx integration's data_stream/access directory contains an elasticsearch/ingest_pipeline directory that contains the following files:

• default.yml
• third-party.yml

These files represent two ingest pipelines that ship with the Nginx integration to provide out-of-the-box mappings, field processing, etc for Nginx data.

For more information on data streams, see Fleet's dev docs here.

agent/stream directory

Each data stream contains an agent/stream directory that includes YML files for configuring agent. These are template files that are compiled by Fleet when generating the full agent policy.

TODO: Document this in more detail

fields directory

The fields directory contains YML files that define the mappings for a given data stream. There may be any number of fields files based on the integration maintainers preferred organization for their mappings. For example, many integrations contain a base-fields.yml file for common mappings, an ecs.yml file for ECS-compliant fields, an agent.yml file for mappings specific to the actual agent process, and a fields.yml file for everything else.

The mappings defined by fields files are used by Fleet to generate an index template for each data stream. This index template composes all of the mappings and settings and ensures that all documents ingested by the data stream are indexed as configured.

docs directory

Contains the README file rendered on the integrations detail page for the integration.

img directory

Contains screenshots rendered on the integrations detail page for the integration.

kibana directory

Contains top level Kibana assets (dashboards, visualizations, Discover sessions etc) for the integration.

Relationship diagram

Reference the following diagram to understand the relationship between the various components of an integration:

erDiagram
  Integration ||--|{ PolicyTemplate : "Many policy templates"
  Integration ||--|{ DataStream : "Many data streams"

  PolicyTemplate ||--|{ Input : "Many inputs"

  DataStream ||--|{ Input : "Many inputs"

  Integration {}
  PolicyTemplate {}
  Input {}
  DataStream {}