github-mirrors/logstash
1
0
Fork 0
mirror of https://github.com/elastic/logstash.git synced 2025-04-24 06:37:19 -04:00
Code Issues Projects Releases Packages Wiki Activity

Add changes and edits for Logstash APIs

Browse source 
Remove alpha annotations for added features

Fixes #5947
This commit is contained in:
DeDe Morton 2016-09-19 11:48:53 -07:00
parent 53c48fb947
commit 46f0b1ddc1
5 changed files with 107 additions and 100 deletions
Download patch file Download diff file Expand all files Collapse all files

26
docs/static/command-line-flags.asciidoc vendored
View file

@ -6,17 +6,15 @@ Logstash has the following flags. You can use the `--help` flag to display this
Instead of specifying options at the command line, we recommend that you control Logstash execution
by specifying options in the Logstash <<logstash-settings-file,settings file>>. Using a settings file
makes it easier for you to specify mutliple options, and it provides you with a single, versionable
file that you can use to start up Logstash consistently for each run.
file that you can use to start up Logstash consistently for each run.
Any flags that you set at the command line override the corresponding settings in the Logstash
<<logstash-settings-file,settings file>>.
added[5.0.0-alpha3, Command-line flags have dots instead of dashes in their names]
<<logstash-settings-file,settings file>>.
*`--node.name NAME`*::
Specify the name of this Logstash instance. If no value is given it will default to the current
hostname.
*`-f, --path.config CONFIG_PATH`*::
Load the Logstash config from a specific file or directory. If a directory is given, all
files in that directory will be concatenated in lexicographical order and then parsed as a
@ -36,7 +34,7 @@ added[5.0.0-alpha3, Command-line flags have dots instead of dashes in their name
in parallel, execute the filter and output stages of the pipeline. If you find that events are
backing up, or that the CPU is not saturated, consider increasing this number to better utilize
machine processing power. The default is 8.
*`-b, --pipeline.batch.size SIZE`*::
Size of batches the pipeline is to work in. This option defines the maximum number of events an
individual worker thread will collect from inputs before attempting to execute its filters and outputs.
@ -48,7 +46,7 @@ added[5.0.0-alpha3, Command-line flags have dots instead of dashes in their name
When creating pipeline batches, how long to wait while polling for the next event. This option defines
how long in milliseconds to wait before dispatching an undersized batch to filters and workers.
The default is 5ms.
*`--pipeline.unsafe_shutdown`*::
Force Logstash to exit during shutdown even if there are still inflight events
in memory. By default, Logstash will refuse to quit until all received events
@ -57,8 +55,8 @@ added[5.0.0-alpha3, Command-line flags have dots instead of dashes in their name
*`--path.data PATH`*::
This should point to a writable directory. Logstash will use this directory whenever it needs to store
data. Plugins will also have access to this path. The default is the `data` directory under
Logstash home.
Logstash home.
*`-p, --path.plugins PATH`*::
A path of where to find custom plugins. This flag can be given multiple times to include
multiple paths. Plugins are expected to be in a specific directory hierarchy:
@ -81,7 +79,7 @@ added[5.0.0-alpha3, Command-line flags have dots instead of dashes in their name
Show the fully compiled configuration as a debug log message (you must also have `--log.level=debug` enabled).
WARNING: The log message will include any 'password' options passed to plugin configs as plaintext, and may result
in plaintext passwords appearing in your logs!
*`-i, --interactive SHELL`*::
Drop to shell instead of running as normal. Valid shells are "irb" and "pry".
@ -93,7 +91,7 @@ added[5.0.0-alpha3, Command-line flags have dots instead of dashes in their name
correctness with this flag. Logstash can read multiple config files from a directory. If you combine this
flag with `--log.level=debug`, Logstash will log the combined config file, annotating
each config block with the source file it came from.
*`-r, --config.reload.automatic`*::
Monitor configuration changes and reload whenever the configuration is changed.
NOTE: Use SIGHUP to manually reload the config. The default is false.
@ -107,7 +105,7 @@ added[5.0.0-alpha3, Command-line flags have dots instead of dashes in their name
*`--http.port HTTP_PORT`*::
Web API http port. This option specifies the bind port for the metrics REST endpoint. The default is 9600-9700.
This setting accepts a range of the format 9600-9700. Logstash will pick up the first available port.
*`--log.format FORMAT`*::
Specify if Logstash should write its own logs in JSON form (one event per line) or in plain text
(using Ruby's Object#inspect). The default is "plain".
@ -115,8 +113,8 @@ added[5.0.0-alpha3, Command-line flags have dots instead of dashes in their name
*`--path.settings SETTINGS_DIR`*::
Set the directory containing the `logstash.yml` <<logstash-settings-file,settings file>> as well
as the log4j logging configuration. This can also be set through the LS_SETTINGS_DIR environment variable.
The default is the `config` directory under Logstash home.
The default is the `config` directory under Logstash home.
*`-h, --help`*::
Print help

18
docs/static/logging.asciidoc vendored
View file

@ -2,11 +2,11 @@
=== Logging
Logstash emits internal logs during its operation, which are placed in `LS_HOME/logs`. The default logging level is `INFO`.
Logstash's logging framework is based on http://logging.apache.org/log4j/2.x/[Log4j2 framework], and many of its functionality
are exposed directly to users.
Logstash's logging framework is based on http://logging.apache.org/log4j/2.x/[Log4j2 framework], and much of its functionality
is exposed directly to users.
When debugging problems, particularly problems with plugins, it can be helpful to increase the logging level to `DEBUG`
to emit more verbose messages. Previously, you could only set a log level which applied to the entire Logstash product.
to emit more verbose messages. Previously, you could only set a log level that applied to the entire Logstash product.
Starting with 5.0, you can configure logging for a particular subsystem in Logstash. For example, if you are
debugging issues with Elasticsearch Output, you can increase log levels just for that component. This way
you can reduce noise due to excessive logging and focus on the problem area effectively.
@ -17,20 +17,20 @@ You can specify the log file location using `--path.logs` setting.
==== Log4j2 Configuration
Logstash ships with a `log4j2.properties` file with out of the box settings. Users can modify this file directly to change the
rotation policy, type and other https://logging.apache.org/log4j/2.x/manual/configuration.html#Loggers[log4j2 configuration].
Any change made to this file is only effective after a process restart.
Logstash ships with a `log4j2.properties` file with out-of-the-box settings. You can modify this file directly to change the
rotation policy, type, and other https://logging.apache.org/log4j/2.x/manual/configuration.html#Loggers[log4j2 configuration].
You must restart Lostash to apply any changes that you make to this file.
==== Logging APIs
You could modify the `log4j2.properties` file and restart your Logstash, but that is both tedious and leads to unnecessary
downtime. Instead, you can dynamically update logging levels through the settings API. These settings are effective
immediately and do not need a restart. To do so, take the subsystem/module you are interested in and prepend
downtime. Instead, you can dynamically update logging levels through the logging API. These settings are effective
immediately and do not need a restart. To update logging levels, take the subsystem/module you are interested in and prepend
`logger.` to it. For example:
[source,js]
--------------------------------------------------
PUT /_node/settings
PUT /_node/logging
{
"logger.logstash.outputs.elasticsearch" : "DEBUG"
}

105
docs/static/monitoring-apis.asciidoc vendored
View file

@ -32,7 +32,7 @@ Example response:
--------------------------------------------------
NOTE: By default, the monitoring API attempts to bind to `tcp:9600`. If this port is already in use by another Logstash
instance, you need to launch Logstash with the `--http.port` flag specified to bind to a different port. See
instance, you need to launch Logstash with the `--http.port` flag specified to bind to a different port. See
<<command-line-flags>> for more information.
[float]
@ -66,8 +66,6 @@ consumption. The default for the `human` flag is
[[node-info-api]]
=== Node Info API
added[5.0.0-alpha4]
experimental[]
The node info API retrieves information about the node.
@ -79,7 +77,7 @@ GET /_node/<types>
Where `<types>` is optional and specifies the types of node info you want to return.
You can limit the info that's returned by combining any of the following types in a comma-separated list: added[5.0.0-alpha5,Support for specifying a comma-separated list of types]
You can limit the info that's returned by combining any of the following types in a comma-separated list:
[horizontal]
`pipeline`::
@ -143,7 +141,7 @@ Example response:
==== JVM Info
The following request returns a JSON document that shows node-level JVM stats, such as the JVM process id, version,
VM info, and memory usage:
VM info, memory usage, and info about garbage collectors:
[source,js]
--------------------------------------------------
@ -156,18 +154,22 @@ Example response:
--------------------------------------------------
{
"jvm": {
"pid": 31580,
"pid": 8187,
"version": "1.8.0_65",
"vm_name": "Java HotSpot(TM) 64-Bit Server VM",
"vm_version": "1.8.0_65",
"vm_vendor": "Oracle Corporation",
"start_time_in_millis": 1466799661753,
"start_time_in_millis": 1474305161631,
"mem": {
"heap_init_in_bytes": 268435456,
"heap_max_in_bytes": 1037959168,
"non_heap_init_in_bytes": 2555904,
"non_heap_max_in_bytes": 0
}
},
"gc_collectors": [
"ParNew",
"ConcurrentMarkSweep"
]
}
--------------------------------------------------
@ -214,11 +216,9 @@ Example response:
[[node-stats-api]]
=== Node Stats API
added[5.0.0-beta3,Replaces the Stats Info API]
experimental[]
The node stats API retrieves runtime stats about Logstash.
The node stats API retrieves runtime stats about Logstash.
[source,js]
--------------------------------------------------
@ -227,74 +227,86 @@ GET /_node/stats/<types>
Where `<types>` is optional and specifies the types of stats you want to return.
By default, all stats are returned. You can limit the info that's returned by combining any of the following types in a comma-separated list: added[5.0.0-alpha5,Support for specifying a comma-separated list of types]
By default, all stats are returned. You can limit the info that's returned by combining any of the following types in a comma-separated list:
[horizontal]
`jvm`::
Gets JVM stats, including stats about threads. added[5.0.0-alpha3,Adds thread count]
Gets JVM stats, including stats about threads, memory usage, and garage collectors.
`process`::
Gets process stats, including stats about file descriptors, memory consumption, and CPU usage. added[5.0.0-alpha3]
Gets process stats, including stats about file descriptors, memory consumption, and CPU usage.
`mem`::
Gets memory usage stats. added[5.0.0-alpha4]
Gets memory usage stats.
`pipeline`::
Gets runtime stats about the Logstash pipeline.
==== JVM Stats
The following request returns a JSON document containing JVM stats:
The following request returns a JSON document containing JVM stats:
[source,js]
--------------------------------------------------
GET /_node/stats/jvm
--------------------------------------------------
Example response: added[5.0.0-alpha5,JVM stats now include memory stats that were previously returned by `/_node/stats/mem`]
Example response:
[source,js]
--------------------------------------------------
{
"jvm": {
"threads": {
"count": 32,
"peak_count": 33
"count": 33,
"peak_count": 34
},
"mem": {
"heap_used_in_bytes": 290715552,
"heap_used_percent": 14,
"heap_used_in_bytes": 276974824,
"heap_used_percent": 13,
"heap_committed_in_bytes": 519045120,
"heap_max_in_bytes": 2075918336,
"non_heap_used_in_bytes": 181911616,
"non_heap_committed_in_bytes": 193249280,
"non_heap_used_in_bytes": 182122272,
"non_heap_committed_in_bytes": 193372160,
"pools": {
"survivor": {
"peak_used_in_bytes": 8912896,
"used_in_bytes": 9358024,
"used_in_bytes": 11182152,
"peak_max_in_bytes": 35782656,
"max_in_bytes": 71565312,
"committed_in_bytes": 17825792
},
"old": {
"peak_used_in_bytes": 106400040,
"used_in_bytes": 164247880,
"peak_used_in_bytes": 111736080,
"used_in_bytes": 171282544,
"peak_max_in_bytes": 715849728,
"max_in_bytes": 1431699456,
"committed_in_bytes": 357957632
},
"young": {
"peak_used_in_bytes": 71630848,
"used_in_bytes": 117109648,
"used_in_bytes": 94510128,
"peak_max_in_bytes": 286326784,
"max_in_bytes": 572653568,
"committed_in_bytes": 143261696
}
}
},
"gc": {
"collectors": {
"old": {
"collection_time_in_millis": 48,
"collection_count": 2
},
"young": {
"collection_time_in_millis": 316,
"collection_count": 23
}
}
}
}
--------------------------------------------------
==== Process Stats
The following request returns a JSON document containing process stats:
The following request returns a JSON document containing process stats:
[source,js]
--------------------------------------------------
@ -321,13 +333,11 @@ Example response:
--------------------------------------------------
[[pipeline-stats]]
==== Pipeline Stats
added[5.0.0-alpha4,Stats for input stages are not yet available]
==== Pipeline Stats
The following request returns a JSON document containing pipeline stats, including the number of events that were
input, filtered, or output by the pipeline. The request also returns stats for each configured input, filter, or
output stage.
output stage, and info about whether config reload (if configured) failed or succeeded.
[source,js]
--------------------------------------------------
@ -341,6 +351,7 @@ Example response:
{
"pipeline": {
"events": {
"duration_in_millis": 7863504,
"in": 100,
"filtered": 100,
"out": 100
@ -349,9 +360,9 @@ Example response:
"inputs": [],
"filters": [
{
"id": "grok_c4900bd0-29ef-44a5-b44c-f6ffef3ddf8c",
"id": "grok_20e5cb7f7c9e712ef9750edf94aefb465e3e361b-2",
"events": {
"duration_in_millis": 43,
"duration_in_millis": 48,
"in": 100,
"out": 100
},
@ -362,9 +373,9 @@ Example response:
"name": "grok"
},
{
"id": "geoip_130740d3-cad0-4ae5-96dd-7ef8f0eb1adb",
"id": "geoip_20e5cb7f7c9e712ef9750edf94aefb465e3e361b-3",
"events": {
"duration_in_millis": 116,
"duration_in_millis": 141,
"in": 100,
"out": 100
},
@ -373,15 +384,21 @@ Example response:
],
"outputs": [
{
"id": "elasticsearch_2f22c8b5-3d63-426e-a4cf-08e48af29538",
"id": "20e5cb7f7c9e712ef9750edf94aefb465e3e361b-4",
"events": {
"duration_in_millis": 533,
"in": 100,
"out": 100
},
"name": "elasticsearch"
}
]
},
"reloads": {
"last_error": null,
"successes": 0,
"last_success_timestamp": null,
"last_failure_timestamp": null,
"failures": 0
}
}
--------------------------------------------------
@ -405,7 +422,7 @@ GET /_node/hot_threads
--------------------------------------------------
The output is a JSON document that contains a breakdown of the top hot threads for
Logstash.
Logstash.
Example response:
@ -413,7 +430,7 @@ Example response:
--------------------------------------------------
{
"hot_threads": {
"time": "2016-07-26T18:39:08-07:00",
"time": "2016-09-19T10:44:13-07:00",
"busiest_threads": 3,
"threads": [
{
@ -476,8 +493,8 @@ Example response:
The parameters allowed are:
[horizontal]
`threads`:: The number of hot threads to return. The default is 3.
`human`:: If true, returns plain text instead of JSON format. The default is false.
`threads`:: The number of hot threads to return. The default is 3.
`human`:: If true, returns plain text instead of JSON format. The default is false.
`ignore_idle_threads`:: If true, does not return idle threads. The default is true.
You can use the `?human` parameter to return the document in a human-readable format.
@ -487,12 +504,12 @@ You can use the `?human` parameter to return the document in a human-readable fo
GET /_node/hot_threads?human=true
--------------------------------------------------
Example of a human-readable response:
Example of a human-readable response:
[source,js]
--------------------------------------------------
::: {}
Hot threads at 2016-07-26T18:46:18-07:00, busiestThreads=3:
Hot threads at 2016-07-26T18:46:18-07:00, busiestThreads=3:
================================================================================
0.15 % of cpu usage by timed_waiting thread named 'LogStash::Runner'
java.lang.Object.wait(Native Method)

24
docs/static/setting-up-logstash.asciidoc vendored
View file

@ -1,7 +1,7 @@
[[setup-logstash]]
== Setting Up and Running Logstash
Before reading this section, see <<installing-logstash>> for basic installation instructions to get you started.
Before reading this section, see <<installing-logstash>> for basic installation instructions to get you started.
This section includes additional information on how to set up and run Logstash, including:
@ -16,8 +16,6 @@ This section includes additional information on how to set up and run Logstash,
This section describes the default directory structure that is created when you unpack the Logstash installation packages.
added[5.0.0-alpha3, Includes breaking changes to the Logstash directory structure]
[[zip-targz-layout]]
==== Directory Layout of `.zip` and `.tar.gz` Archives
@ -35,7 +33,7 @@ config and the logs directories so that you do not delete important data later o
| home
| Home directory of the Logstash installation.
| `{extract.path}`- Directory created by unpacking the archive
d|
d|
| bin
| Binary scripts, including `logstash` to start Logstash
@ -67,7 +65,7 @@ locations for the system:
| home
| Home directory of the Logstash installation.
| `/usr/share/logstash`
d|
d|
| bin
| Binary scripts including `logstash` to start Logstash
@ -101,26 +99,24 @@ locations for the system:
=== Logstash Configuration Files
Logstash has two types of configuration files: _pipeline configuration files_, which define the Logstash processing
pipeline, and _settings files_, which specify options that control Logstash startup and execution.
pipeline, and _settings files_, which specify options that control Logstash startup and execution.
==== Pipeline Configuration Files
You create pipeline configuration files when you define the stages of your Logstash processing pipeline. On deb and
rpm, you place the pipeline configuration files in the `/etc/logstash/conf.d` directory. Logstash tries to load all
files in the `/etc/logstash/conf.d directory`, so don't store any non-config files or backup files in this directory.
files in the `/etc/logstash/conf.d directory`, so don't store any non-config files or backup files in this directory.
See <<configuration>> for more info.
==== Settings Files
added[5.0.0-alpha3]
The settings files are already defined in the Logstash installation. Logstash includes the following settings files:
*`logstash.yml`*::
*`logstash.yml`*::
Contains Logstash configuration flags. You can set flags in this file instead of passing the flags at the command
line. Any flags that you set at the command line override the corresponding settings in the `logstash.yml` file. See <<logstash-settings-file>> for more info.
*`jvm.options`*::
*`jvm.options`*::
Contains JVM configuration flags. Specify each flag on a separate line. You can also use this file to set the locale
for Logstash.
*`startup.options` (Linux)*::
@ -135,10 +131,8 @@ The settings files are already defined in the Logstash installation. Logstash in
[[running-logstash]]
=== Running Logstash as a Service on Debian or RPM
added[5.0.0-alpha3]
Logstash is not started automatically after installation. How to start and stop Logstash depends on whether your system
uses systemd, upstart, or SysV.
uses systemd, upstart, or SysV.
[[running-logstash-systemd]]
==== Running Logstash by Using Systemd
@ -173,7 +167,7 @@ For systems that use SysV, you can start Logstash with:
sudo /etc/init.d/logstash start
-------------------------------------------
The auto-generated configuration file for SysV systems is `/etc/init.d/logstash`.
The auto-generated configuration file for SysV systems is `/etc/init.d/logstash`.

34
docs/static/settings-file.asciidoc vendored
View file

@ -1,13 +1,11 @@
[[logstash-settings-file]]
=== Settings File
added[5.0.0-alpha3]
You can set options in the Logstash settings file, `logstash.yml`, to control Logstash execution. For example,
you can specify pipeline settings, the location of configuration files, logging options, and other settings.
Most of the settings in the `logstash.yml` file are also available as <<command-line-flags,command-line flags>>
when you run Logstash. Any flags that you set at the command line override the corresponding settings in the
`logstash.yml` file.
`logstash.yml` file.
The `logstash.yml` file, which is written in http://http://yaml.org/[YAML], is located in `LOGSTASH_HOME/config`. You can
specify settings in hierarchical form or use flat keys. For example, to use hierarchical form to set the pipeline batch
@ -44,7 +42,7 @@ The `logstash.yml` file includes the following settings:
|`LOGSTASH_HOME/data`
| `pipeline.workers`
| The number of workers that will, in parallel, execute the filter and output stages of the pipeline.
| The number of workers that will, in parallel, execute the filter and output stages of the pipeline.
If you find that events are backing up, or that the
CPU is not saturated, consider increasing this number to better utilize machine processing power.
| Number of the host's CPU cores
@ -55,23 +53,23 @@ The `logstash.yml` file includes the following settings:
| `pipeline.batch.size`
| The maximum number of events an individual worker thread will collect from inputs
before attempting to execute its filters and outputs.
before attempting to execute its filters and outputs.
Larger batch sizes are generally more efficient, but come at the cost of increased memory
overhead. You may have to increase the JVM heap size by setting the `LS_HEAP_SIZE`
variable to effectively use the option.
| `125`
| `pipeline.batch.delay`
| When creating pipeline event batches, how long in milliseconds to wait before dispatching an undersized
batch to filters and workers.
| `5`
| `pipeline.unsafe_shutdown`
| When set to `true`, forces Logstash to exit during shutdown even if there are still inflight events
in memory. By default, Logstash will refuse to quit until all received events
have been pushed to the outputs. Enabling this option can lead to data loss during shutdown.
| `false`
| `path.config`
| The path to the Logstash config for the main pipeline. If you specify a directory or wildcard,
config files are read from the directory in alphabetical order.
@ -81,19 +79,19 @@ The `logstash.yml` file includes the following settings:
| A string that contains the pipeline configuration to use for the main pipeline. Use the same syntax as
the config file.
| None
| `config.test_and_exit`
| When set to `true`, checks that the configuration is valid and then exits. Note that grok patterns are not checked for
correctness with this setting. Logstash can read multiple config files from a directory. If you combine this
setting with `log.level: debug`, Logstash will log the combined config file, annotating
each config block with the source file it came from.
| `false`
| `config.reload.automatic`
| When set to `true`, periodically checks if the configuration has changed and reloads the configuration whenever it is changed.
| When set to `true`, periodically checks if the configuration has changed and reloads the configuration whenever it is changed.
This can also be triggered manually through the SIGHUP signal.
| `false`
| `config.reload.interval`
| How often in seconds Logstash checks the config files for changes.
| `3`
@ -103,15 +101,15 @@ The `logstash.yml` file includes the following settings:
WARNING: The log message will include any 'password' options passed to plugin configs as plaintext, and may result
in plaintext passwords appearing in your logs!
| `false`
| `http.host`
| The bind address for the metrics REST endpoint.
| `"127.0.0.1"`
| `http.port`
| The bind port for the metrics REST endpoint.
| The bind port for the metrics REST endpoint.
| `9600`
| `log.level`
a|
The log level. Valid options are:
@ -122,7 +120,7 @@ The log level. Valid options are:
* `debug`: log debugging info (for developers)
| `warn`
| `log.format`
| The log format. Set to `json` to log in JSON format, or `plain` to use `Object#.inspect`.
| `plain`
@ -130,13 +128,13 @@ The log level. Valid options are:
| `path.log`
| The file to log to.
| Logs to stdout
| `path.plugins`
| Where to find custom plugins. You can specify this setting multiple times to include
multiple paths. Plugins are expected to be in a specific directory hierarchy:
`PATH/logstash/TYPE/NAME.rb` where `TYPE` is `inputs`, `filters`, `outputs`, or `codecs`,
and `NAME` is the name of the plugin.
| Platform-specific. See <<dir-layout>>.
|=======================================================================