mirror of
https://github.com/elastic/logstash.git
synced 2025-04-23 22:27:21 -04:00
d7a9dcd887
Moves Cloud info to Configuration section to make it more obvious and easier to find Expands content for using cloud id and cloud auth outside of modules Moves module-specific info into modules section
211 lines
6.8 KiB
Text
211 lines
6.8 KiB
Text
[[logstash-modules]]
|
||
== Working with Logstash Modules
|
||
|
||
Logstash modules provide a quick, end-to-end solution for ingesting data and
|
||
visualizing it with purpose-built dashboards.
|
||
|
||
These modules are available:
|
||
|
||
* <<arcsight-module>>
|
||
* <<netflow-module,Netflow Module (deprecated)>>
|
||
* <<azure-module, Microsoft Azure Module (deprecated)>>
|
||
|
||
Each module comes pre-packaged with Logstash configurations, Kibana dashboards,
|
||
and other meta files that make it easier for you to set up the Elastic Stack for
|
||
specific use cases or data sources.
|
||
|
||
You can think of modules as providing three essential functions that make it
|
||
easier for you to get started. When you run a module, it will:
|
||
|
||
. Create the Elasticsearch index.
|
||
|
||
. Set up the Kibana dashboards, including the index pattern, searches, and
|
||
visualizations required to visualize your data in Kibana.
|
||
|
||
. Run the Logstash pipeline with the configurations required to read and parse
|
||
the data.
|
||
|
||
image::static/images/logstash-module-overview.png[Logstash modules overview]
|
||
|
||
[discrete]
|
||
[[running-logstash-modules]]
|
||
=== Running modules
|
||
|
||
To run a module and set up dashboards, you specify the following options:
|
||
|
||
[source,shell]
|
||
----
|
||
bin/logstash --modules MODULE_NAME --setup [-M "CONFIG_SETTING=VALUE"]
|
||
----
|
||
|
||
|
||
//TODO: For 6.0, show how to run multiple modules
|
||
|
||
Where:
|
||
|
||
* `--modules` runs the Logstash module specified by `MODULE_NAME`.
|
||
|
||
* `-M "CONFIG_SETTING=VALUE"` is optional and overrides the specified
|
||
configuration setting. You can specify multiple overrides. Each override must
|
||
start with `-M`. See <<overriding-logstash-module-settings>> for more info.
|
||
|
||
* `--setup` creates an index pattern in Elasticsearch and imports Kibana
|
||
dashboards and visualizations. Running `--setup` is a one-time setup step. Omit
|
||
this option for subsequent runs of the module to avoid overwriting existing
|
||
Kibana dashboards.
|
||
|
||
For example, the following command runs the Netflow module with the default
|
||
settings, and sets up the netflow index pattern and dashboards:
|
||
|
||
[source,shell]
|
||
----
|
||
bin/logstash --modules netflow --setup
|
||
----
|
||
|
||
The following command runs the Netflow module and overrides the Elasticsearch
|
||
`host` setting. Here it's assumed that you've already run the setup step.
|
||
|
||
[source,shell]
|
||
----
|
||
bin/logstash --modules netflow -M "netflow.var.elasticsearch.host=es.mycloud.com"
|
||
----
|
||
|
||
|
||
[discrete]
|
||
[[configuring-logstash-modules]]
|
||
=== Configuring modules
|
||
|
||
To configure a module, you can either
|
||
<<setting-logstash-module-config,specify configuration settings>> in the
|
||
`logstash.yml` <<logstash-settings-file,settings file>>, or use command-line overrides to
|
||
<<overriding-logstash-module-settings,specify settings at the command line>>.
|
||
|
||
[discrete]
|
||
[[setting-logstash-module-config]]
|
||
==== Specify module settings in `logstash.yml`
|
||
|
||
To specify module settings in the `logstash.yml`
|
||
<<logstash-settings-file,settings file>> file, you add a module definition to
|
||
the modules array. Each module definition begins with a dash (-) and is followed
|
||
by `name: module_name` then a series of name/value pairs that specify module
|
||
settings. For example:
|
||
|
||
[source,shell]
|
||
----
|
||
modules:
|
||
- name: netflow
|
||
var.elasticsearch.hosts: "es.mycloud.com"
|
||
var.elasticsearch.username: "foo"
|
||
var.elasticsearch.password: "password"
|
||
var.kibana.host: "kb.mycloud.com"
|
||
var.kibana.username: "foo"
|
||
var.kibana.password: "password"
|
||
var.input.tcp.port: 5606
|
||
----
|
||
|
||
For a list of available module settings, see the documentation for the module.
|
||
|
||
[discrete]
|
||
[[overriding-logstash-module-settings]]
|
||
==== Specify module settings at the command line
|
||
|
||
[IMPORTANT]
|
||
.macOS Gatekeeper warnings
|
||
====
|
||
Apple's rollout of stricter notarization requirements affected the notarization
|
||
of the {version} {ls} artifacts. If macOS Catalina displays a dialog when you
|
||
first run {ls} that interrupts it, you will need to take an action to allow it
|
||
to run.
|
||
To prevent Gatekeeper checks on the {ls} files, run the following command on the
|
||
downloaded `.tar.gz` archive or the directory to which was extracted:
|
||
[source,sh]
|
||
----
|
||
xattr -d -r com.apple.quarantine <archive-or-directory>
|
||
----
|
||
For example, if the `.tar.gz` file was extracted to the default
|
||
logstash-{version} directory, the command is:
|
||
[source,sh,subs="attributes"]
|
||
----
|
||
xattr -d -r com.apple.quarantine logstash-{version}
|
||
----
|
||
Alternatively, you can add a security override if a Gatekeeper popup appears by
|
||
following the instructions in the _How to open an app that hasn’t been notarized
|
||
or is from an unidentified developer_ section of
|
||
https://support.apple.com/en-us/HT202491[Safely open apps on your Mac].
|
||
====
|
||
|
||
You can override module settings by specifying one or more configuration
|
||
overrides when you start Logstash. To specify an override, you use the `-M`
|
||
command line option:
|
||
|
||
[source,shell]
|
||
----
|
||
-M MODULE_NAME.var.PLUGINTYPE1.PLUGINNAME1.KEY1=VALUE
|
||
----
|
||
|
||
Notice that the fully-qualified setting name includes the module name.
|
||
|
||
You can specify multiple overrides. Each override must start with `-M`.
|
||
|
||
The following command runs the Netflow module and overrides both the
|
||
Elasticsearch `host` setting and the `udp.port` setting:
|
||
|
||
[source,shell]
|
||
----
|
||
bin/logstash --modules netflow -M "netflow.var.input.udp.port=3555" -M "netflow.var.elasticsearch.hosts=my-es-cloud"
|
||
----
|
||
|
||
Any settings defined in the command line are ephemeral and will not persist across
|
||
subsequent runs of Logstash. If you want to persist a configuration, you need to
|
||
set it in the `logstash.yml` <<logstash-settings-file,settings file>>.
|
||
|
||
Settings that you specify at the command line are merged with any settings
|
||
specified in the `logstash.yml` file. If an option is set in both
|
||
places, the value specified at the command line takes precedence.
|
||
|
||
[discrete]
|
||
[[module-to-cloud]]
|
||
==== Sending data to Elasticsearch Service from modules
|
||
|
||
Cloud ID and Cloud Auth can be specified in the `logstash.yml` <<logstash-settings-file,settings file>>.
|
||
They should be added separately from any module configuration settings you may have added before.
|
||
|
||
Cloud ID overwrites these settings:
|
||
----
|
||
var.elasticsearch.hosts
|
||
var.kibana.host
|
||
----
|
||
|
||
Cloud Auth overwrites these settings:
|
||
----
|
||
var.elasticsearch.username
|
||
var.elasticsearch.password
|
||
var.kibana.username
|
||
var.kibana.password
|
||
----
|
||
|
||
["source","yaml",subs="attributes"]
|
||
----
|
||
# example with a label
|
||
cloud.id: "staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy"
|
||
cloud.auth: "elastic:{pwd}"
|
||
----
|
||
|
||
["source","yaml",subs="attributes"]
|
||
----
|
||
# example without a label
|
||
cloud.id: "dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy"
|
||
cloud.auth: "elastic:{pwd}"
|
||
----
|
||
|
||
These settings can be also specified at the command line, like this:
|
||
|
||
["source","sh",subs="attributes,callouts"]
|
||
----
|
||
bin/logstash --modules netflow -M "netflow.var.input.udp.port=3555" --cloud.id <your-cloud-id> --cloud.auth <your-cloud-auth>
|
||
----
|
||
|
||
NOTE: When working with modules, use the dot notation to specify cloud.id and
|
||
cloud.auth, as indicated in the examples.
|
||
|
||
For more info on Cloud ID and Cloud Auth, see <<connecting-to-cloud>>.
|