mirror of
https://github.com/elastic/logstash.git
synced 2025-04-24 14:47:19 -04:00
[DOCS] Adding translated GS docs for jp & kr.
This commit is contained in:
Deb Adair
parent
e3b01d1191
commit
99f3d7b821
75 changed files with 8642 additions and 0 deletions
34
docs/jp/gs-index.asciidoc
Normal file
34
docs/jp/gs-index.asciidoc
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
[[logstash-reference]]
|
||||
= Logstashリファレンス
|
||||
|
||||
:branch: 5.4
|
||||
:major-version: 5.4
|
||||
:logstash_version: 5.4.0
|
||||
:elasticsearch_version: 5.4.0
|
||||
:docker-image: docker.elastic.co/logstash/logstash:{logstash_version}
|
||||
|
||||
//////////
|
||||
release-state can be: released | prerelease | unreleased
|
||||
//////////
|
||||
:release-state: released
|
||||
|
||||
:jdk: 1.8.0
|
||||
:guide: https://www.elastic.co/guide/en/elasticsearch/guide/current/
|
||||
:ref: https://www.elastic.co/guide/en/elasticsearch/reference/{major-version}/
|
||||
:xpack: https://www.elastic.co/guide/en/x-pack/current/
|
||||
:logstash: https://www.elastic.co/guide/en/logstash/{major-version}/
|
||||
:filebeat: https://www.elastic.co/guide/en/beats/filebeat/{major-version}/
|
||||
:lsissue: https://github.com/elastic/logstash/issues/
|
||||
:security: X-Pack Security
|
||||
:stack: https://www.elastic.co/guide/en/elastic-stack/current/
|
||||
|
||||
[[introduction]]
|
||||
== Logstashの紹介
|
||||
|
||||
Logstashは、リアルタイムのパイプライン機能を備えたオープンソースのデータ収集エンジンです。Logstashは、異なるデータソースのデータを動的に統合し、そのデータを選択した出力先に合わせます。多様で高度なダウンストリーム分析と可視化の事例向けにすべてのデータをクレンジングし、誰でも使えるようにします。
|
||||
|
||||
本来、Logstashはログ収集における革新を牽引していますが、その機能はその事例に留まりません。収集プロセスをさらにシンプルにする多数のネイティブコーディックにより、数多くのinput、filter、およびoutputのプラグインであらゆる種類のイベントを整形し変換することができます。Logstashは、大量かつ多様なデータを利用して、分析を促進します。
|
||||
|
||||
include::static/introduction.asciidoc[]
|
||||
|
||||
include::static/getting-started-with-logstash.asciidoc[]
|
||||
156
docs/jp/index.asciidoc
Normal file
156
docs/jp/index.asciidoc
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
[[logstash-reference]]
|
||||
= Logstash Reference
|
||||
|
||||
:branch: 5.x
|
||||
:major-version: 5.x
|
||||
:logstash_version: 5.x.0
|
||||
:elasticsearch_version: 5.x.0
|
||||
:docker-image: docker.elastic.co/logstash/logstash:{logstash_version}
|
||||
|
||||
//////////
|
||||
release-state can be: released | prerelease | unreleased
|
||||
//////////
|
||||
:release-state: unreleased
|
||||
|
||||
:jdk: 1.8.0
|
||||
:guide: https://www.elastic.co/guide/en/elasticsearch/guide/current/
|
||||
:ref: https://www.elastic.co/guide/en/elasticsearch/reference/5.x/
|
||||
:xpack: https://www.elastic.co/guide/en/x-pack/current/
|
||||
:logstash: https://www.elastic.co/guide/en/logstash/5.x/
|
||||
:filebeat: https://www.elastic.co/guide/en/beats/filebeat/5.x/
|
||||
:lsissue: https://github.com/elastic/logstash/issues/
|
||||
:security: X-Pack Security
|
||||
:stack: https://www.elastic.co/guide/en/elastic-stack/current/
|
||||
|
||||
[[introduction]]
|
||||
== Logstash Introduction
|
||||
|
||||
Logstash is an open source data collection engine with real-time pipelining capabilities. Logstash can dynamically
|
||||
unify data from disparate sources and normalize the data into destinations of your choice. Cleanse and democratize all
|
||||
your data for diverse advanced downstream analytics and visualization use cases.
|
||||
|
||||
While Logstash originally drove innovation in log collection, its capabilities extend well beyond that use case. Any
|
||||
type of event can be enriched and transformed with a broad array of input, filter, and output plugins, with many
|
||||
native codecs further simplifying the ingestion process. Logstash accelerates your insights by harnessing a greater
|
||||
volume and variety of data.
|
||||
|
||||
// The pass blocks here point to the correct repository for the edit links in the guide.
|
||||
|
||||
// Introduction
|
||||
|
||||
include::static/introduction.asciidoc[]
|
||||
|
||||
// Glossary and core concepts go here
|
||||
|
||||
// Getting Started with Logstash
|
||||
|
||||
include::static/getting-started-with-logstash.asciidoc[]
|
||||
|
||||
// Advanced LS Pipelines
|
||||
|
||||
include::static/advanced-pipeline.asciidoc[]
|
||||
|
||||
// Processing Pipeline
|
||||
|
||||
include::static/life-of-an-event.asciidoc[]
|
||||
|
||||
// Lostash setup
|
||||
|
||||
include::static/setting-up-logstash.asciidoc[]
|
||||
|
||||
|
||||
include::static/settings-file.asciidoc[]
|
||||
|
||||
|
||||
include::static/running-logstash-command-line.asciidoc[]
|
||||
|
||||
|
||||
include::static/running-logstash.asciidoc[]
|
||||
|
||||
|
||||
include::static/docker.asciidoc[]
|
||||
|
||||
|
||||
include::static/logging.asciidoc[]
|
||||
|
||||
|
||||
include::static/persistent-queues.asciidoc[]
|
||||
|
||||
|
||||
include::static/shutdown.asciidoc[]
|
||||
|
||||
// Breaking Changes
|
||||
|
||||
include::static/breaking-changes.asciidoc[]
|
||||
|
||||
// Upgrading Logstash
|
||||
|
||||
include::static/upgrading.asciidoc[]
|
||||
|
||||
// Configuring Logstash
|
||||
|
||||
include::static/configuration.asciidoc[]
|
||||
|
||||
|
||||
include::static/reloading-config.asciidoc[]
|
||||
|
||||
|
||||
include::static/managing-multiline-events.asciidoc[]
|
||||
|
||||
|
||||
include::static/glob-support.asciidoc[]
|
||||
|
||||
// Deploying & Scaling
|
||||
|
||||
include::static/deploying.asciidoc[]
|
||||
|
||||
// Troubleshooting performance
|
||||
|
||||
include::static/performance-checklist.asciidoc[]
|
||||
|
||||
// Monitoring APIs
|
||||
|
||||
include::static/monitoring-apis.asciidoc[]
|
||||
|
||||
// Working with Plugins
|
||||
|
||||
include::static/plugin-manager.asciidoc[]
|
||||
|
||||
// These files do their own pass blocks
|
||||
|
||||
include::plugins/inputs.asciidoc[]
|
||||
include::plugins/outputs.asciidoc[]
|
||||
include::plugins/filters.asciidoc[]
|
||||
include::plugins/codecs.asciidoc[]
|
||||
|
||||
:edit_url:
|
||||
|
||||
// Contributing to Logstash
|
||||
|
||||
include::static/contributing-to-logstash.asciidoc[]
|
||||
|
||||
include::static/input.asciidoc[]
|
||||
include::static/codec.asciidoc[]
|
||||
include::static/filter.asciidoc[]
|
||||
include::static/output.asciidoc[]
|
||||
|
||||
// Contributing a Patch to a Logstash Plugin
|
||||
|
||||
include::static/contributing-patch.asciidoc[]
|
||||
|
||||
// Logstash Community Maintainer Guide
|
||||
|
||||
include::static/maintainer-guide.asciidoc[]
|
||||
|
||||
// A space is necessary here ^^^
|
||||
|
||||
|
||||
// Submitting a Plugin
|
||||
|
||||
include::static/submitting-a-plugin.asciidoc[]
|
||||
|
||||
|
||||
// Glossary of Terms
|
||||
|
||||
include::static/glossary.asciidoc[]
|
||||
|
||||
835
docs/jp/static/advanced-pipeline.asciidoc
Normal file
835
docs/jp/static/advanced-pipeline.asciidoc
Normal file
|
|
@ -0,0 +1,835 @@
|
|||
[[advanced-pipeline]]
|
||||
=== Parsing Logs with Logstash
|
||||
|
||||
In <<first-event>>, you created a basic Logstash pipeline to test your Logstash setup. In the real world, a Logstash
|
||||
pipeline is a bit more complex: it typically has one or more input, filter, and output plugins.
|
||||
|
||||
In this section, you create a Logstash pipeline that uses Filebeat to take Apache web logs as input, parses those
|
||||
logs to create specific, named fields from the logs, and writes the parsed data to an Elasticsearch cluster. Rather than
|
||||
defining the pipeline configuration at the command line, you'll define the pipeline in a config file.
|
||||
|
||||
To get started, go https://download.elastic.co/demos/logstash/gettingstarted/logstash-tutorial.log.gz[here] to
|
||||
download the sample data set used in this example. Unpack the file.
|
||||
|
||||
|
||||
[[configuring-filebeat]]
|
||||
==== Configuring Filebeat to Send Log Lines to Logstash
|
||||
|
||||
Before you create the Logstash pipeline, you'll configure Filebeat to send log lines to Logstash.
|
||||
The https://github.com/elastic/beats/tree/master/filebeat[Filebeat] client is a lightweight, resource-friendly tool
|
||||
that collects logs from files on the server and forwards these logs to your Logstash instance for processing.
|
||||
Filebeat is designed for reliability and low latency. Filebeat has a light resource footprint on the host machine,
|
||||
and the {logstash}plugins-inputs-beats.html[`Beats input`] plugin minimizes the resource demands on the Logstash
|
||||
instance.
|
||||
|
||||
NOTE: In a typical use case, Filebeat runs on a separate machine from the machine running your
|
||||
Logstash instance. For the purposes of this tutorial, Logstash and Filebeat are running on the
|
||||
same machine.
|
||||
|
||||
The default Logstash installation includes the {logstash}plugins-inputs-beats.html[`Beats input`] plugin. The Beats
|
||||
input plugin enables Logstash to receive events from the Elastic Beats framework, which means that any Beat written
|
||||
to work with the Beats framework, such as Packetbeat and Metricbeat, can also send event data to Logstash.
|
||||
|
||||
To install Filebeat on your data source machine, download the appropriate package from the Filebeat https://www.elastic.co/downloads/beats/filebeat[product page]. You can also refer to
|
||||
{filebeat}filebeat-getting-started.html[Getting Started with Filebeat] in the Beats documentation for additional
|
||||
installation instructions.
|
||||
|
||||
After installing Filebeat, you need to configure it. Open the `filebeat.yml` file located in your Filebeat installation
|
||||
directory, and replace the contents with the following lines. Make sure `paths` points to the example Apache log file,
|
||||
`logstash-tutorial.log`, that you downloaded earlier:
|
||||
|
||||
[source,yaml]
|
||||
--------------------------------------------------------------------------------
|
||||
filebeat.prospectors:
|
||||
- input_type: log
|
||||
paths:
|
||||
- /path/to/file/logstash-tutorial.log <1>
|
||||
output.logstash:
|
||||
hosts: ["localhost:5043"]
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
<1> Absolute path to the file or files that Filebeat processes.
|
||||
|
||||
Save your changes.
|
||||
|
||||
To keep the configuration simple, you won't specify TLS/SSL settings as you would in a real world
|
||||
scenario.
|
||||
|
||||
At the data source machine, run Filebeat with the following command:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
sudo ./filebeat -e -c filebeat.yml -d "publish"
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Filebeat will attempt to connect on port 5043. Until Logstash starts with an active Beats plugin, there
|
||||
won’t be any answer on that port, so any messages you see regarding failure to connect on that port are normal for now.
|
||||
|
||||
==== Configuring Logstash for Filebeat Input
|
||||
|
||||
Next, you create a Logstash configuration pipeline that uses the Beats input plugin to receive
|
||||
events from Beats.
|
||||
|
||||
The following text represents the skeleton of a configuration pipeline:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
# The # character at the beginning of a line indicates a comment. Use
|
||||
# comments to describe your configuration.
|
||||
input {
|
||||
}
|
||||
# The filter part of this file is commented out to indicate that it is
|
||||
# optional.
|
||||
# filter {
|
||||
#
|
||||
# }
|
||||
output {
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
This skeleton is non-functional, because the input and output sections don’t have any valid options defined.
|
||||
|
||||
To get started, copy and paste the skeleton configuration pipeline into a file named `first-pipeline.conf` in your home
|
||||
Logstash directory.
|
||||
|
||||
Next, configure your Logstash instance to use the Beats input plugin by adding the following lines to the `input` section
|
||||
of the `first-pipeline.conf` file:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
beats {
|
||||
port => "5043"
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
You'll configure Logstash to write to Elasticsearch later. For now, you can add the following line
|
||||
to the `output` section so that the output is printed to stdout when you run Logstash:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
stdout { codec => rubydebug }
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
When you're done, the contents of `first-pipeline.conf` should look like this:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
input {
|
||||
beats {
|
||||
port => "5043"
|
||||
}
|
||||
}
|
||||
# The filter part of this file is commented out to indicate that it is
|
||||
# optional.
|
||||
# filter {
|
||||
#
|
||||
# }
|
||||
output {
|
||||
stdout { codec => rubydebug }
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
To verify your configuration, run the following command:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
bin/logstash -f first-pipeline.conf --config.test_and_exit
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
The `--config.test_and_exit` option parses your configuration file and reports any errors.
|
||||
|
||||
If the configuration file passes the configuration test, start Logstash with the following command:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
bin/logstash -f first-pipeline.conf --config.reload.automatic
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
The `--config.reload.automatic` option enables automatic config reloading so that you don't have to stop and restart Logstash
|
||||
every time you modify the configuration file.
|
||||
|
||||
If your pipeline is working correctly, you should see a series of events like the following written to the console:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
{
|
||||
"@timestamp" => 2016-10-11T20:54:06.733Z,
|
||||
"offset" => 325,
|
||||
"@version" => "1",
|
||||
"beat" => {
|
||||
"hostname" => "My-MacBook-Pro.local",
|
||||
"name" => "My-MacBook-Pro.local"
|
||||
},
|
||||
"input_type" => "log",
|
||||
"host" => "My-MacBook-Pro.local",
|
||||
"source" => "/path/to/file/logstash-tutorial.log",
|
||||
"message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
|
||||
"type" => "log",
|
||||
"tags" => [
|
||||
[0] "beats_input_codec_plain_applied"
|
||||
]
|
||||
}
|
||||
...
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
|
||||
[float]
|
||||
[[configuring-grok-filter]]
|
||||
==== Parsing Web Logs with the Grok Filter Plugin
|
||||
|
||||
Now you have a working pipeline that reads log lines from Filebeat. However you'll notice that the format of the log messages
|
||||
is not ideal. You want to parse the log messages to create specific, named fields from the logs.
|
||||
To do this, you'll use the `grok` filter plugin.
|
||||
|
||||
The {logstash}plugins-filters-grok.html[`grok`] filter plugin is one of several plugins that are available by default in
|
||||
Logstash. For details on how to manage Logstash plugins, see the <<working-with-plugins,reference documentation>> for
|
||||
the plugin manager.
|
||||
|
||||
The `grok` filter plugin enables you to parse the unstructured log data into something structured and queryable.
|
||||
|
||||
Because the `grok` filter plugin looks for patterns in the incoming log data, configuring the plugin requires you to
|
||||
make decisions about how to identify the patterns that are of interest to your use case. A representative line from the
|
||||
web server log sample looks like this:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png
|
||||
HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel
|
||||
Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
The IP address at the beginning of the line is easy to identify, as is the timestamp in brackets. To parse the data, you can use the `%{COMBINEDAPACHELOG}` grok pattern, which structures lines from the Apache log using the following schema:
|
||||
|
||||
[horizontal]
|
||||
*Information*:: *Field Name*
|
||||
IP Address:: `clientip`
|
||||
User ID:: `ident`
|
||||
User Authentication:: `auth`
|
||||
timestamp:: `timestamp`
|
||||
HTTP Verb:: `verb`
|
||||
Request body:: `request`
|
||||
HTTP Version:: `httpversion`
|
||||
HTTP Status Code:: `response`
|
||||
Bytes served:: `bytes`
|
||||
Referrer URL:: `referrer`
|
||||
User agent:: `agent`
|
||||
|
||||
Edit the `first-pipeline.conf` file and replace the entire `filter` section with the following text:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
filter {
|
||||
grok {
|
||||
match => { "message" => "%{COMBINEDAPACHELOG}"}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
When you're done, the contents of `first-pipeline.conf` should look like this:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
input {
|
||||
beats {
|
||||
port => "5043"
|
||||
}
|
||||
}
|
||||
filter {
|
||||
grok {
|
||||
match => { "message" => "%{COMBINEDAPACHELOG}"}
|
||||
}
|
||||
}
|
||||
output {
|
||||
stdout { codec => rubydebug }
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Save your changes. Because you've enabled automatic config reloading, you don't have to restart Logstash to
|
||||
pick up your changes. However, you do need to force Filebeat to read the log file from scratch. To do this,
|
||||
go to the terminal window where Filebeat is running and press Ctrl+C to shut down Filebeat. Then delete the
|
||||
Filebeat registry file. For example, run:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
sudo rm data/registry
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Since Filebeat stores the state of each file it harvests in the registry, deleting the registry file forces
|
||||
Filebeat to read all the files it's harvesting from scratch.
|
||||
|
||||
Next, restart Filebeat with the following command:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
sudo ./filebeat -e -c filebeat.yml -d "publish"
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
After processing the log file with the grok pattern, the events will have the following JSON representation:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
{
|
||||
"request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png",
|
||||
"agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
|
||||
"offset" => 325,
|
||||
"auth" => "-",
|
||||
"ident" => "-",
|
||||
"input_type" => "log",
|
||||
"verb" => "GET",
|
||||
"source" => "/path/to/file/logstash-tutorial.log",
|
||||
"message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
|
||||
"type" => "log",
|
||||
"tags" => [
|
||||
[0] "beats_input_codec_plain_applied"
|
||||
],
|
||||
"referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
|
||||
"@timestamp" => 2016-10-11T21:04:36.167Z,
|
||||
"response" => "200",
|
||||
"bytes" => "203023",
|
||||
"clientip" => "83.149.9.216",
|
||||
"@version" => "1",
|
||||
"beat" => {
|
||||
"hostname" => "My-MacBook-Pro.local",
|
||||
"name" => "My-MacBook-Pro.local"
|
||||
},
|
||||
"host" => "My-MacBook-Pro.local",
|
||||
"httpversion" => "1.1",
|
||||
"timestamp" => "04/Jan/2015:05:13:42 +0000"
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Notice that the event includes the original message, but the log message is also broken down into specific fields.
|
||||
|
||||
|
||||
[float]
|
||||
[[configuring-geoip-plugin]]
|
||||
==== Enhancing Your Data with the Geoip Filter Plugin
|
||||
|
||||
In addition to parsing log data for better searches, filter plugins can derive supplementary information from existing
|
||||
data. As an example, the {logstash}plugins-filters-geoip.html[`geoip`] plugin looks up IP addresses, derives geographic
|
||||
location information from the addresses, and adds that location information to the logs.
|
||||
|
||||
Configure your Logstash instance to use the `geoip` filter plugin by adding the following lines to the `filter` section
|
||||
of the `first-pipeline.conf` file:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
geoip {
|
||||
source => "clientip"
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
The `geoip` plugin configuration requires you to specify the name of the source field that contains the IP address to look up. In this example, the `clientip` field contains the IP address.
|
||||
|
||||
Since filters are evaluated in sequence, make sure that the `geoip` section is after the `grok` section of
|
||||
the configuration file and that both the `grok` and `geoip` sections are nested within the `filter` section.
|
||||
|
||||
When you're done, the contents of `first-pipeline.conf` should look like this:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
input {
|
||||
beats {
|
||||
port => "5043"
|
||||
}
|
||||
}
|
||||
filter {
|
||||
grok {
|
||||
match => { "message" => "%{COMBINEDAPACHELOG}"}
|
||||
}
|
||||
geoip {
|
||||
source => "clientip"
|
||||
}
|
||||
}
|
||||
output {
|
||||
stdout { codec => rubydebug }
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Save your changes. To force Filebeat to read the log file from scratch, as you did earlier, shut down Filebeat (press Ctrl+C),
|
||||
delete the registry file, and then restart Filebeat with the following command:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
sudo ./filebeat -e -c filebeat.yml -d "publish"
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Notice that the event now contains geographic location information:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
{
|
||||
"request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png",
|
||||
"agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
|
||||
"geoip" => {
|
||||
"timezone" => "Europe/Moscow",
|
||||
"ip" => "83.149.9.216",
|
||||
"latitude" => 55.7522,
|
||||
"continent_code" => "EU",
|
||||
"city_name" => "Moscow",
|
||||
"country_code2" => "RU",
|
||||
"country_name" => "Russia",
|
||||
"dma_code" => nil,
|
||||
"country_code3" => "RU",
|
||||
"region_name" => "Moscow",
|
||||
"location" => [
|
||||
[0] 37.6156,
|
||||
[1] 55.7522
|
||||
],
|
||||
"postal_code" => "101194",
|
||||
"longitude" => 37.6156,
|
||||
"region_code" => "MOW"
|
||||
},
|
||||
...
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
|
||||
[float]
|
||||
[[indexing-parsed-data-into-elasticsearch]]
|
||||
==== Indexing Your Data into Elasticsearch
|
||||
|
||||
Now that the web logs are broken down into specific fields, the Logstash pipeline can index the data into an
|
||||
Elasticsearch cluster. Edit the `first-pipeline.conf` file and replace the entire `output` section with the following
|
||||
text:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
output {
|
||||
elasticsearch {
|
||||
hosts => [ "localhost:9200" ]
|
||||
}
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
With this configuration, Logstash uses http protocol to connect to Elasticsearch. The above example assumes that
|
||||
Logstash and Elasticsearch are running on the same instance. You can specify a remote Elasticsearch instance by using
|
||||
the `hosts` configuration to specify something like `hosts => [ "es-machine:9092" ]`.
|
||||
|
||||
At this point, your `first-pipeline.conf` file has input, filter, and output sections properly configured, and looks
|
||||
something like this:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
input {
|
||||
beats {
|
||||
port => "5043"
|
||||
}
|
||||
}
|
||||
filter {
|
||||
grok {
|
||||
match => { "message" => "%{COMBINEDAPACHELOG}"}
|
||||
}
|
||||
geoip {
|
||||
source => "clientip"
|
||||
}
|
||||
}
|
||||
output {
|
||||
elasticsearch {
|
||||
hosts => [ "localhost:9200" ]
|
||||
}
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Save your changes. To force Filebeat to read the log file from scratch, as you did earlier, shut down Filebeat (press Ctrl+C),
|
||||
delete the registry file, and then restart Filebeat with the following command:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
sudo ./filebeat -e -c filebeat.yml -d "publish"
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
[float]
|
||||
[[testing-initial-pipeline]]
|
||||
===== Testing Your Pipeline
|
||||
|
||||
Now that the Logstash pipeline is configured to index the data into an
|
||||
Elasticsearch cluster, you can query Elasticsearch.
|
||||
|
||||
Try a test query to Elasticsearch based on the fields created by the `grok` filter plugin.
|
||||
Replace $DATE with the current date, in YYYY.MM.DD format:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
curl -XGET 'localhost:9200/logstash-$DATE/_search?pretty&q=response=200'
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
NOTE: The date used in the index name is based on UTC, not the timezone where Logstash is running.
|
||||
If the query returns `index_not_found_exception`, make sure that `logstash-$DATE` reflects the actual
|
||||
name of the index. To see a list of available indexes, use this query: `curl 'localhost:9200/_cat/indices?v'`.
|
||||
|
||||
You should get multiple hits back. For example:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
{
|
||||
"took" : 21,
|
||||
"timed_out" : false,
|
||||
"_shards" : {
|
||||
"total" : 5,
|
||||
"successful" : 5,
|
||||
"failed" : 0
|
||||
},
|
||||
"hits" : {
|
||||
"total" : 98,
|
||||
"max_score" : 3.745223,
|
||||
"hits" : [
|
||||
{
|
||||
"_index" : "logstash-2016.10.11",
|
||||
"_type" : "log",
|
||||
"_id" : "AVe14gMiYMkU36o_eVsA",
|
||||
"_score" : 3.745223,
|
||||
"_source" : {
|
||||
"request" : "/presentations/logstash-monitorama-2013/images/frontend-response-codes.png",
|
||||
"agent" : "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
|
||||
"geoip" : {
|
||||
"timezone" : "Europe/Moscow",
|
||||
"ip" : "83.149.9.216",
|
||||
"latitude" : 55.7522,
|
||||
"continent_code" : "EU",
|
||||
"city_name" : "Moscow",
|
||||
"country_code2" : "RU",
|
||||
"country_name" : "Russia",
|
||||
"dma_code" : null,
|
||||
"country_code3" : "RU",
|
||||
"region_name" : "Moscow",
|
||||
"location" : [
|
||||
37.6156,
|
||||
55.7522
|
||||
],
|
||||
"postal_code" : "101194",
|
||||
"longitude" : 37.6156,
|
||||
"region_code" : "MOW"
|
||||
},
|
||||
"offset" : 2932,
|
||||
"auth" : "-",
|
||||
"ident" : "-",
|
||||
"input_type" : "log",
|
||||
"verb" : "GET",
|
||||
"source" : "/path/to/file/logstash-tutorial.log",
|
||||
"message" : "83.149.9.216 - - [04/Jan/2015:05:13:45 +0000] \"GET /presentations/logstash-monitorama-2013/images/frontend-response-codes.png HTTP/1.1\" 200 52878 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
|
||||
"type" : "log",
|
||||
"tags" : [
|
||||
"beats_input_codec_plain_applied"
|
||||
],
|
||||
"referrer" : "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
|
||||
"@timestamp" : "2016-10-11T22:34:25.317Z",
|
||||
"response" : "200",
|
||||
"bytes" : "52878",
|
||||
"clientip" : "83.149.9.216",
|
||||
"@version" : "1",
|
||||
"beat" : {
|
||||
"hostname" : "My-MacBook-Pro.local",
|
||||
"name" : "My-MacBook-Pro.local"
|
||||
},
|
||||
"host" : "My-MacBook-Pro.local",
|
||||
"httpversion" : "1.1",
|
||||
"timestamp" : "04/Jan/2015:05:13:45 +0000"
|
||||
}
|
||||
}
|
||||
},
|
||||
...
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Try another search for the geographic information derived from the IP address.
|
||||
Replace $DATE with the current date, in YYYY.MM.DD format:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
curl -XGET 'localhost:9200/logstash-$DATE/_search?pretty&q=geoip.city_name=Buffalo'
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
A few log entries come from Buffalo, so the query produces the following response:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
{
|
||||
"took" : 3,
|
||||
"timed_out" : false,
|
||||
"_shards" : {
|
||||
"total" : 5,
|
||||
"successful" : 5,
|
||||
"failed" : 0
|
||||
},
|
||||
"hits" : {
|
||||
"total" : 3,
|
||||
"max_score" : 2.6390574,
|
||||
"hits" : [
|
||||
{
|
||||
"_index" : "logstash-2016.10.11",
|
||||
"_type" : "log",
|
||||
"_id" : "AVe14gMjYMkU36o_eVtO",
|
||||
"_score" : 2.6390574,
|
||||
"_source" : {
|
||||
"request" : "/?flav=rss20",
|
||||
"agent" : "\"-\"",
|
||||
"geoip" : {
|
||||
"timezone" : "America/New_York",
|
||||
"ip" : "108.174.55.234",
|
||||
"latitude" : 42.9864,
|
||||
"continent_code" : "NA",
|
||||
"city_name" : "Buffalo",
|
||||
"country_code2" : "US",
|
||||
"country_name" : "United States",
|
||||
"dma_code" : 514,
|
||||
"country_code3" : "US",
|
||||
"region_name" : "New York",
|
||||
"location" : [
|
||||
-78.7279,
|
||||
42.9864
|
||||
],
|
||||
"postal_code" : "14221",
|
||||
"longitude" : -78.7279,
|
||||
"region_code" : "NY"
|
||||
},
|
||||
"offset" : 21471,
|
||||
"auth" : "-",
|
||||
"ident" : "-",
|
||||
"input_type" : "log",
|
||||
"verb" : "GET",
|
||||
"source" : "/path/to/file/logstash-tutorial.log",
|
||||
"message" : "108.174.55.234 - - [04/Jan/2015:05:27:45 +0000] \"GET /?flav=rss20 HTTP/1.1\" 200 29941 \"-\" \"-\"",
|
||||
"type" : "log",
|
||||
"tags" : [
|
||||
"beats_input_codec_plain_applied"
|
||||
],
|
||||
"referrer" : "\"-\"",
|
||||
"@timestamp" : "2016-10-11T22:34:25.318Z",
|
||||
"response" : "200",
|
||||
"bytes" : "29941",
|
||||
"clientip" : "108.174.55.234",
|
||||
"@version" : "1",
|
||||
"beat" : {
|
||||
"hostname" : "My-MacBook-Pro.local",
|
||||
"name" : "My-MacBook-Pro.local"
|
||||
},
|
||||
"host" : "My-MacBook-Pro.local",
|
||||
"httpversion" : "1.1",
|
||||
"timestamp" : "04/Jan/2015:05:27:45 +0000"
|
||||
}
|
||||
},
|
||||
...
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
If you are using Kibana to visualize your data, you can also explore the Filebeat data in Kibana:
|
||||
|
||||
image::static/images/kibana-filebeat-data.png[Discovering Filebeat data in Kibana]
|
||||
|
||||
See the {filebeat}filebeat-getting-started.html[Filebeat getting started docs] for info about loading the Kibana
|
||||
index pattern for Filebeat.
|
||||
|
||||
You've successfully created a pipeline that uses Filebeat to take Apache web logs as input, parses those logs to
|
||||
create specific, named fields from the logs, and writes the parsed data to an Elasticsearch cluster. Next, you
|
||||
learn how to create a pipeline that uses multiple input and output plugins.
|
||||
|
||||
[[multiple-input-output-plugins]]
|
||||
=== Stitching Together Multiple Input and Output Plugins
|
||||
|
||||
The information you need to manage often comes from several disparate sources, and use cases can require multiple
|
||||
destinations for your data. Your Logstash pipeline can use multiple input and output plugins to handle these
|
||||
requirements.
|
||||
|
||||
In this section, you create a Logstash pipeline that takes input from a Twitter feed and the Filebeat client, then
|
||||
sends the information to an Elasticsearch cluster as well as writing the information directly to a file.
|
||||
|
||||
[float]
|
||||
[[twitter-configuration]]
|
||||
==== Reading from a Twitter Feed
|
||||
|
||||
To add a Twitter feed, you use the {logstash}plugins-inputs-twitter.html[`twitter`] input plugin. To
|
||||
configure the plugin, you need several pieces of information:
|
||||
|
||||
* A _consumer key_, which uniquely identifies your Twitter app.
|
||||
* A _consumer secret_, which serves as the password for your Twitter app.
|
||||
* One or more _keywords_ to search in the incoming feed. The example shows using "cloud" as a keyword, but you can use whatever you want.
|
||||
* An _oauth token_, which identifies the Twitter account using this app.
|
||||
* An _oauth token secret_, which serves as the password of the Twitter account.
|
||||
|
||||
Visit https://dev.twitter.com/apps[https://dev.twitter.com/apps] to set up a Twitter account and generate your consumer
|
||||
key and secret, as well as your access token and secret. See the docs for the {logstash}plugins-inputs-twitter.html[`twitter`] input plugin if you're not sure how to generate these keys.
|
||||
|
||||
Like you did earlier when you worked on <<advanced-pipeline>>, create a config file (called `second-pipeline.conf`) that
|
||||
contains the skeleton of a configuration pipeline. If you want, you can reuse the file you created earlier, but make
|
||||
sure you pass in the correct config file name when you run Logstash.
|
||||
|
||||
Add the following lines to the `input` section of the `second-pipeline.conf` file, substituting your values for the
|
||||
placeholder values shown here:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
twitter {
|
||||
consumer_key => "enter_your_consumer_key_here"
|
||||
consumer_secret => "enter_your_secret_here"
|
||||
keywords => ["cloud"]
|
||||
oauth_token => "enter_your_access_token_here"
|
||||
oauth_token_secret => "enter_your_access_token_secret_here"
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
[float]
|
||||
[[configuring-lsf]]
|
||||
==== Configuring Filebeat to Send Log Lines to Logstash
|
||||
|
||||
As you learned earlier in <<configuring-filebeat>>, the https://github.com/elastic/beats/tree/master/filebeat[Filebeat]
|
||||
client is a lightweight, resource-friendly tool that collects logs from files on the server and forwards these logs to your
|
||||
Logstash instance for processing.
|
||||
|
||||
After installing Filebeat, you need to configure it. Open the `filebeat.yml` file located in your Filebeat installation
|
||||
directory, and replace the contents with the following lines. Make sure `paths` points to your syslog:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
filebeat.prospectors:
|
||||
- input_type: log
|
||||
paths:
|
||||
- /var/log/*.log <1>
|
||||
fields:
|
||||
type: syslog <2>
|
||||
output.logstash:
|
||||
hosts: ["localhost:5043"]
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
<1> Absolute path to the file or files that Filebeat processes.
|
||||
<2> Adds a field called `type` with the value `syslog` to the event.
|
||||
|
||||
Save your changes.
|
||||
|
||||
To keep the configuration simple, you won't specify TLS/SSL settings as you would in a real world
|
||||
scenario.
|
||||
|
||||
Configure your Logstash instance to use the Filebeat input plugin by adding the following lines to the `input` section
|
||||
of the `second-pipeline.conf` file:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
beats {
|
||||
port => "5043"
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
[float]
|
||||
[[logstash-file-output]]
|
||||
==== Writing Logstash Data to a File
|
||||
|
||||
You can configure your Logstash pipeline to write data directly to a file with the
|
||||
{logstash}plugins-outputs-file.html[`file`] output plugin.
|
||||
|
||||
Configure your Logstash instance to use the `file` output plugin by adding the following lines to the `output` section
|
||||
of the `second-pipeline.conf` file:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
file {
|
||||
path => "/path/to/target/file"
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
[float]
|
||||
[[multiple-es-nodes]]
|
||||
==== Writing to Multiple Elasticsearch Nodes
|
||||
|
||||
Writing to multiple Elasticsearch nodes lightens the resource demands on a given Elasticsearch node, as well as
|
||||
providing redundant points of entry into the cluster when a particular node is unavailable.
|
||||
|
||||
To configure your Logstash instance to write to multiple Elasticsearch nodes, edit the `output` section of the `second-pipeline.conf` file to read:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
output {
|
||||
elasticsearch {
|
||||
hosts => ["IP Address 1:port1", "IP Address 2:port2", "IP Address 3"]
|
||||
}
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Use the IP addresses of three non-master nodes in your Elasticsearch cluster in the host line. When the `hosts`
|
||||
parameter lists multiple IP addresses, Logstash load-balances requests across the list of addresses. Also note that
|
||||
the default port for Elasticsearch is `9200` and can be omitted in the configuration above.
|
||||
|
||||
[float]
|
||||
[[testing-second-pipeline]]
|
||||
===== Testing the Pipeline
|
||||
|
||||
At this point, your `second-pipeline.conf` file looks like this:
|
||||
|
||||
[source,json]
|
||||
--------------------------------------------------------------------------------
|
||||
input {
|
||||
twitter {
|
||||
consumer_key => "enter_your_consumer_key_here"
|
||||
consumer_secret => "enter_your_secret_here"
|
||||
keywords => ["cloud"]
|
||||
oauth_token => "enter_your_access_token_here"
|
||||
oauth_token_secret => "enter_your_access_token_secret_here"
|
||||
}
|
||||
beats {
|
||||
port => "5043"
|
||||
}
|
||||
}
|
||||
output {
|
||||
elasticsearch {
|
||||
hosts => ["IP Address 1:port1", "IP Address 2:port2", "IP Address 3"]
|
||||
}
|
||||
file {
|
||||
path => "/path/to/target/file"
|
||||
}
|
||||
}
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Logstash is consuming data from the Twitter feed you configured, receiving data from Filebeat, and
|
||||
indexing this information to three nodes in an Elasticsearch cluster as well as writing to a file.
|
||||
|
||||
At the data source machine, run Filebeat with the following command:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
sudo ./filebeat -e -c filebeat.yml -d "publish"
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Filebeat will attempt to connect on port 5043. Until Logstash starts with an active Beats plugin, there
|
||||
won’t be any answer on that port, so any messages you see regarding failure to connect on that port are normal for now.
|
||||
|
||||
To verify your configuration, run the following command:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
bin/logstash -f second-pipeline.conf --config.test_and_exit
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
The `--config.test_and_exit` option parses your configuration file and reports any errors. When the configuration file
|
||||
passes the configuration test, start Logstash with the following command:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
bin/logstash -f second-pipeline.conf
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Use the `grep` utility to search in the target file to verify that information is present:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
grep syslog /path/to/target/file
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Run an Elasticsearch query to find the same information in the Elasticsearch cluster:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
curl -XGET 'localhost:9200/logstash-$DATE/_search?pretty&q=fields.type:syslog'
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Replace $DATE with the current date, in YYYY.MM.DD format.
|
||||
|
||||
To see data from the Twitter feed, try this query:
|
||||
|
||||
[source,shell]
|
||||
--------------------------------------------------------------------------------
|
||||
curl -XGET 'http://localhost:9200/logstash-$DATE/_search?pretty&q=client:iphone'
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Again, remember to replace $DATE with the current date, in YYYY.MM.DD format.
|
||||
|
||||
|
||||
|
||||
310
docs/jp/static/breaking-changes.asciidoc
Normal file
310
docs/jp/static/breaking-changes.asciidoc
Normal file
|
|
@ -0,0 +1,310 @@
|
|||
[[breaking-changes]]
|
||||
== Breaking changes
|
||||
|
||||
This section discusses the changes that you need to be aware of when migrating your application to Logstash 5.0 from the previous major release of Logstash (2.x).
|
||||
|
||||
[float]
|
||||
=== Changes in Logstash Core
|
||||
|
||||
These changes can impact any instance of Logstash and are plugin agnostic, but only if you are using the features that are impacted.
|
||||
|
||||
[float]
|
||||
==== Application Settings
|
||||
|
||||
[IMPORTANT]
|
||||
Logstash 5.0 introduces a new way to <<logstash-settings-file, configure application settings>> for Logstash through a
|
||||
`logstash.yml` file.
|
||||
|
||||
This file is typically located in `${LS_HOME}/config`, or `/etc/logstash` when installed via packages. Logstash will not be
|
||||
able to start without this file, so please make sure to pass in `--path.settings` if you are starting Logstash manually
|
||||
after installing it via a package (RPM, DEB).
|
||||
|
||||
[source,bash]
|
||||
----------------------------------
|
||||
bin/logstash --path.settings /path/to/logstash.yml
|
||||
----------------------------------
|
||||
|
||||
[float]
|
||||
==== URL Changes for DEB/RPM Packages
|
||||
|
||||
The previous `packages.elastic.co` URL has been altered to `artifacts.elastic.co`.
|
||||
Ensure you update your repository files before running the upgrade process, or
|
||||
your operating system may not see the new packages.
|
||||
|
||||
[float]
|
||||
==== Release Packages
|
||||
|
||||
When Logstash 5.0 is installed via DEB or RPM packages, it now uses `/usr/share/logstash` to
|
||||
install binaries. Previously it used to install in `/opt/logstash` directory. This change was done to make the user experience consistent with other products in the Elastic Stack.
|
||||
|
||||
[cols="3", options="header"]
|
||||
|===
|
||||
| |DEB |RPM
|
||||
|Logstash 2.x
|
||||
|`/opt/logstash`
|
||||
|`/opt/logstash`
|
||||
|Logstash 5.0
|
||||
|`/usr/share/logstash`
|
||||
|`/usr/share/logstash`
|
||||
|===
|
||||
|
||||
A complete directory layout is described in <<dir-layout>>. This will likely impact any scripts that you may have written
|
||||
to support installing or manipulating Logstash, such as via Puppet.
|
||||
|
||||
[float]
|
||||
==== Default Logging Level
|
||||
|
||||
The default log severity level changed to `INFO` instead of `WARN` to match Elasticsearch. Existing logs
|
||||
(in core and plugins) were too noisy at the `INFO` level, so we audited our log messages and switched some of them to
|
||||
`DEBUG` level.
|
||||
|
||||
You can use the new `logstash.yml` file to configure the `log.level` setting or continue to pass the new
|
||||
`--log.level` command line flag.
|
||||
|
||||
[source,bash]
|
||||
----------------------------------
|
||||
bin/logstash --log.level warn
|
||||
----------------------------------
|
||||
|
||||
[float]
|
||||
==== Plugin Manager Renamed
|
||||
|
||||
`bin/plugin` has been renamed to `bin/logstash-plugin`. This occurred in Logstash 2.3 and it was mainly prevent `PATH` being
|
||||
polluted when other components of the Elastic Stack are installed on the same machine. Also, this provides a foundation
|
||||
for future change which will allow Elastic Stack packs to be installed via this script.
|
||||
|
||||
Logstash 5.0 also adds a `remove` option, which is an alias for the now-deprecated `uninstall` option.
|
||||
|
||||
As with earlier releases, the updated script allows both online and offline plugin installation. For example, to install a
|
||||
plugin named “my-plugin”, it’s as simple as running:
|
||||
|
||||
[source,bash]
|
||||
----------------------------------
|
||||
bin/logstash-plugin install my-plugin
|
||||
----------------------------------
|
||||
|
||||
Similar to the package changes, this is likely to impact and scripts that have been written to follow Logstash
|
||||
installations.
|
||||
|
||||
Like earlier releases of Logstash, most plugins are bundled directly with Logstash, so no additional action is required
|
||||
while upgrading from earlier Logstash releases. However, if you are attempting to install a non-bundled plugin, then make
|
||||
sure that it supports Logstash 5.0 before upgrading!
|
||||
|
||||
[float]
|
||||
==== Logstash with All Plugins Download
|
||||
|
||||
The Logstash All Plugins download option has been removed. For users previously using this option as a convenience for
|
||||
offline plugin management purposes (air-gapped environments), please see the <<offline-plugins>> documentation page.
|
||||
|
||||
There were 17 plugins removed from 5.0 default bundle. These plugins can still be installed manually for use:
|
||||
|
||||
* logstash-codec-oldlogstashjson
|
||||
* logstash-input-eventlog
|
||||
* logstash-input-log4j
|
||||
* logstash-input-zeromq
|
||||
* logstash-filter-anonymize
|
||||
* logstash-filter-checksum
|
||||
* logstash-filter-multiline
|
||||
* logstash-output-email
|
||||
* logstash-output-exec
|
||||
* logstash-output-ganglia
|
||||
* logstash-output-gelf
|
||||
* logstash-output-hipchat
|
||||
* logstash-output-juggernaut
|
||||
* logstash-output-lumberjack
|
||||
* logstash-output-nagios_nsca
|
||||
* logstash-output-opentsdb
|
||||
* logstash-output-zeromq
|
||||
|
||||
[float]
|
||||
==== Command Line Interface
|
||||
|
||||
Some CLI Options changed in Logstash 5.0. If you were using the “long form” of the <<command-line-flags,options>>,
|
||||
then this will impact the way that you launch Logstash. They were changed to match the `logstash.yml` format used to
|
||||
simplify future setup, as well as behave in the same way as other products in the Elastic Stack. For example, here’s two
|
||||
before-and-after examples. In Logstash 2.x, you may have run something:
|
||||
|
||||
[source,bash]
|
||||
----------------------------------
|
||||
bin/logstash --config my.conf --pipeline-workers 8 <1>
|
||||
bin/logstash -f my.conf -w 8 <2>
|
||||
----------------------------------
|
||||
<1> Long form options `config` and `pipeline-workers` are used here.
|
||||
<2> Short form options `f` and `w` (aliases for the former` are used here.
|
||||
|
||||
But, in Logstash 5.0, this becomes:
|
||||
|
||||
[source,bash]
|
||||
----------------------------------
|
||||
bin/logstash --path.config my.conf --pipeline.workers 8 <1>
|
||||
bin/logstash -f my.conf -w 8 <2>
|
||||
----------------------------------
|
||||
<1> Long form options are changed to reflect the new options.
|
||||
<2> Short form options are unchanged.
|
||||
|
||||
NOTE: None of the short form options have changed!
|
||||
|
||||
[float]
|
||||
==== RSpec testing script
|
||||
|
||||
The `rspec` script is no longer bundled with Logstash release artifacts. This script has been used previously to
|
||||
run unit tests for validating Logstash configurations. While this was useful to some users, this mechanism assumed that Logstash users
|
||||
were familiar with the RSpec framework, which is a Ruby testing framework.
|
||||
|
||||
|
||||
[float]
|
||||
=== Breaking Changes in Plugins
|
||||
|
||||
[float]
|
||||
==== Elasticsearch Output `workers` Setting Removed
|
||||
|
||||
Starting with Logstash 5.0, the `workers` setting in the Elasticsearch output
|
||||
plugin is no longer supported. Pipelines that specify this setting will no
|
||||
longer start up. You need to specify the `pipeline.workers` setting at the
|
||||
pipeline level instead. For more information about setting
|
||||
`pipeline.workers`, see <<logstash-settings-file>>.
|
||||
|
||||
[float]
|
||||
==== Elasticsearch Output Index Template
|
||||
|
||||
The index template for Elasticsearch 5.0 has been changed to reflect
|
||||
https://www.elastic.co/guide/en/elasticsearch/reference/5.0/breaking_50_mapping_changes.html[Elasticsearch's mapping changes]. Most
|
||||
importantly, the subfield for string multi-fields has changed from `.raw` to `.keyword` to match Elasticsearch's default
|
||||
behavior. The impact of this change to various user groups is detailed below:
|
||||
|
||||
** New Logstash 5.0 and Elasticsearch 5.0 users: Multi-fields (often called sub-fields) use `.keyword` from the
|
||||
outset. In Kibana, you can use `my_field.keyword` to perform aggregations against text-based fields, in the same way that it
|
||||
used to be `my_field.raw`.
|
||||
** Existing users with custom templates: Using a custom template means that you control the template completely, and our
|
||||
template changes do not impact you.
|
||||
** Existing users with default template: Logstash does not force you to upgrade templates if one already exists. If you
|
||||
intend to move to the new template and want to use `.keyword`, you will most likely want to reindex existing data so that it
|
||||
also uses the `.keyword` field, unless you are able to transition from `.raw` to `.keyword`. Elasticsearch's
|
||||
{ref}docs-reindex.html[reindexing API] can help move your data from using `.raw` subfields to `.keyword`, thereby avoiding any
|
||||
transition time. You _can_ use a custom template to get both `.raw` and `.keyword` so that you can wait until all `.raw` data
|
||||
has stopped existing before transitioning to only using `.keyword`; this will waste some storage space and memory, but it does
|
||||
help users to avoid having to relearn operations.
|
||||
|
||||
[float]
|
||||
[[plugin-versions]]
|
||||
==== Plugin Versions
|
||||
|
||||
Logstash is unique amongst the Elastic Stack with respect to its plugins. Unlike Elasticsearch and Kibana, which both
|
||||
require plugins to be targeted to a specific release, Logstash’s plugin ecosystem provides more flexibility so that it can
|
||||
support outside ecosystems _within the same release_. Unfortunately,
|
||||
that flexibility can cause issues when handling upgrades.
|
||||
|
||||
Non-standard plugins must always be checked for compatibility, but some bundled plugins are upgraded in order to remain
|
||||
compatible with the tools or frameworks that they use for communication. For example, the
|
||||
<<plugins-inputs-kafka, Kafka Input>> and <<plugins-outputs-kafka, Kafka Output>> plugins serve as a primary example of
|
||||
such compatibility changes. The latest version of the Kafka plugins is only compatible with Kafka 0.10, but as the
|
||||
compatibility matrices show: earlier plugin versions are required for earlier versions of Kafka (e.g., Kafka 0.9).
|
||||
|
||||
Automatic upgrades generally lead to improved features and support, but network layer changes like those above may make part
|
||||
of your architecture incompatible. You should always test your Logstash configurations in a test environment before
|
||||
deploying to production, which would catch these kinds of issues. If you do face such an issue, then you should also check
|
||||
the specific plugin’s page to see how to get a compatible, older plugin version if necessary.
|
||||
|
||||
For example, if you upgrade to Logstash 5.0, but you want to run against Kafka 0.9, then you need to remove the
|
||||
bundled plugin(s) that only work with Kafka 0.10 and replace them:
|
||||
|
||||
[source,bash]
|
||||
----------------------------------
|
||||
bin/logstash-plugin remove logstash-input-kafka
|
||||
bin/logstash-plugin remove logstash-output-kafka
|
||||
bin/logstash-plugin install --version 4.0.0 logstash-input-kafka
|
||||
bin/logstash-plugin install --version 4.0.1 logstash-output-kafka
|
||||
----------------------------------
|
||||
|
||||
The version numbers were found by checking the compatibility matrix for the individual plugins.
|
||||
|
||||
[float]
|
||||
==== Kafka Input Configuration Changes
|
||||
|
||||
As described in the section <<plugin-versions, above>>, the Kafka plugin has been updated to bring in new consumer features.
|
||||
In addition, to the plugin being incompatible with 0.8.x version of the Kafka broker, _most_ of the config options have
|
||||
been changed to match the new consumer configurations from the Kafka Java consumer. Here's a list of important config options that have changed:
|
||||
|
||||
* `topic_id` is renamed to `topics` and accepts an array of topics to consume from.
|
||||
* `zk_connect` has been dropped; you should use `bootstrap_servers`. There is no need for the consumer to go through ZooKeeper.
|
||||
* `consumer_id` is renamed to `client_id`.
|
||||
|
||||
We recommend users of the Kafka plugin to check the documentation for the latest <<plugins-inputs-kafka, config options>>.
|
||||
|
||||
[float]
|
||||
==== File Input
|
||||
|
||||
The <<plugins-inputs-file, File Input>> `SinceDB` file is now saved at `<path.data>/plugins/inputs/file` location,
|
||||
where `path.data` is the path defined in the new `logstash.yml` file.
|
||||
|
||||
[cols="2", options="header"]
|
||||
|===
|
||||
| |Default `sincedb_path`
|
||||
|Logstash 2.x
|
||||
|`$HOME/.sincedb*`
|
||||
|Logstash 5.0
|
||||
|`<path.data>/plugins/inputs/file`
|
||||
|===
|
||||
|
||||
If you have manually specified `sincedb_path` as part of the configuration, this change will not affect you.
|
||||
If you are moving from Logstash 2.x to Logstash 5.0, and you would like to use the existing SinceDB file,
|
||||
then it must be copied over to `path.data` manually to use the save state (or the path needs to be changed to point to it).
|
||||
|
||||
[float]
|
||||
==== GeoIP Filter
|
||||
|
||||
The GeoIP filter has been updated to use MaxMind's GeoIP2 database. Previous GeoIP version is now considered legacy
|
||||
by MaxMind. As a result of this, `.dat` version files are no longer supported, and only `.mmdb` format is supported.
|
||||
The new database will not include ASN data in the basic free database file.
|
||||
|
||||
Previously, when the filter encountered an IP address for which there were no results in the database, the event
|
||||
would just pass through the filter without modification. It will now add a `_geoip_lookup_failure` tag to the
|
||||
event which will allow for some subsequent stage of the pipeline to identify those events and perform some other
|
||||
operation. To simply get the same behavior as the earlier versions, just add a filter conditional on that tag
|
||||
which then drops the tag from the event.
|
||||
|
||||
[float]
|
||||
=== Ruby Filter and Custom Plugin Developers
|
||||
|
||||
With the migration to the new <<event-api>>, we have changed how you can access internal data compared to previous release.
|
||||
The `event` object no longer returns a reference to the data. Instead, it returns a copy. This might change how you perform
|
||||
manipulation of your data, especially when working with nested hashes. When working with nested hashes, it’s recommended that
|
||||
you use the <<logstash-config-field-references, `field reference` syntax>> instead of using multiple square brackets.
|
||||
|
||||
As part of this change, Logstash has introduced new Getter/Setter APIs for accessing information in the `event` object.
|
||||
|
||||
**Examples:**
|
||||
|
||||
Prior to Logstash 5.0, you may have used Ruby filters like so:
|
||||
|
||||
[source, js]
|
||||
----------------------------------
|
||||
filter {
|
||||
ruby {
|
||||
code => "event['name'] = 'Logstash'"
|
||||
}
|
||||
ruby {
|
||||
code => "event['product']['version'] = event['major'] + '.' + event['minor']"
|
||||
}
|
||||
}
|
||||
----------------------------------
|
||||
|
||||
The above syntax, which uses the `event` object as a reference, is no longer supported in
|
||||
Logstash 5.0. Fortunately, the change to make it work is very simple:
|
||||
|
||||
[source, js]
|
||||
----------------------------------
|
||||
filter {
|
||||
ruby {
|
||||
code => "event.set('name', 'Logstash')"
|
||||
}
|
||||
ruby {
|
||||
code => "event.set('[product][version]', event.get('major') + '.' + event.get('minor'))"
|
||||
}
|
||||
}
|
||||
----------------------------------
|
||||
|
||||
NOTE: Moving from the old syntax to the new syntax, it can be easy to miss that `['product']['version']` became
|
||||
`'[product][version]'`. The quotes moved from inside of the square brackets to outside of the square brackets!
|
||||
|
||||
The <<event-api>> documentation describes the available syntax in great detail.
|
||||
15
docs/jp/static/codec.asciidoc
Normal file
15
docs/jp/static/codec.asciidoc
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
:register_method: true
|
||||
:encode_method: true
|
||||
:decode_method: true
|
||||
:plugintype: codec
|
||||
:pluginclass: Codecs
|
||||
:pluginname: example
|
||||
:pluginnamecap: Example
|
||||
|
||||
:blockinput: true
|
||||
|
||||
:getstarted: Let's step through creating a {plugintype} plugin using the https://github.com/logstash-plugins/logstash-codec-example/[example {plugintype} plugin].
|
||||
|
||||
:methodheader: Logstash codecs must implement the `register` method, and the `decode` method or the `encode` method (or both).
|
||||
|
||||
include::include/pluginbody.asciidoc[]
|
||||
1062
docs/jp/static/configuration.asciidoc
Normal file
1062
docs/jp/static/configuration.asciidoc
Normal file
File diff suppressed because it is too large
Load diff
397
docs/jp/static/contributing-patch.asciidoc
Normal file
397
docs/jp/static/contributing-patch.asciidoc
Normal file
|
|
@ -0,0 +1,397 @@
|
|||
[[contributing-patch-plugin]]
|
||||
=== Contributing a Patch to a Logstash Plugin
|
||||
|
||||
This section discusses the information you need to know to successfully contribute a patch to a Logstash plugin.
|
||||
|
||||
Each plugin defines its own configuration options. These control the behaviour of the plugin to some degree. Configuration
|
||||
option definitions commonly include:
|
||||
|
||||
* Data validation
|
||||
* The default value
|
||||
* Any required flags
|
||||
|
||||
Plugins are subclasses of a Logstash base class. A plugin's base class defines common configuration and methods.
|
||||
|
||||
==== Input Plugins
|
||||
|
||||
Input plugins ingest data from an external source. Input plugins are always associated with a codec. An input plugin
|
||||
always has an associated codec plugin. Input and codec plugins operate in conjunction to create a Logstash event and add
|
||||
that event to the processing queue. An input codec is a subclass of the `LogStash::Inputs::Base` class.
|
||||
|
||||
.Input API
|
||||
[horizontal]
|
||||
`#register() -> nil`:: Required. This API sets up resources for the plugin, typically the connection to the
|
||||
external source.
|
||||
`#run(queue) -> nil`:: Required. This API fetches or listens for source data, typically looping until stopped. Must handle
|
||||
errors inside the loop. Pushes any created events to the queue object specified in the method argument. Some inputs may
|
||||
receive batched data to minimize the external call overhead.
|
||||
`#stop() -> nil`:: Optional. Stops external connections and cleans up.
|
||||
|
||||
==== Codec Plugins
|
||||
|
||||
Codec plugins decode input data that has a specific structure, such as JSON input data. A codec plugin is a subclass of
|
||||
`LogStash::Codecs::Base`.
|
||||
|
||||
.Codec API
|
||||
[horizontal]
|
||||
`#register() -> nil`:: Identical to the API of the same name for input plugins.
|
||||
`#decode(data){|event| block} -> nil`:: Must be implemented. Used to create an Event from the raw data given in the method
|
||||
argument. Must handle errors. The caller must provide a Ruby block. The block is called with the created Event.
|
||||
`#encode(event) -> nil`:: Required. Used to create a structured data object from the given Event. May handle
|
||||
errors. This method calls a block that was previously stored as @on_event with two arguments: the original event and the
|
||||
data object.
|
||||
|
||||
==== Filter Plugins
|
||||
|
||||
A mechanism to change, mutate or merge one or more Events. A filter plugin is a subclass of the `LogStash::Filters::Base`
|
||||
class.
|
||||
|
||||
.Filter API
|
||||
[horizontal]
|
||||
`#register() -> nil`:: Identical to the API of the same name for input plugins.
|
||||
`#filter(event) -> nil`:: Required. May handle errors. Used to apply a mutation function to the given event.
|
||||
|
||||
==== Output Plugins
|
||||
|
||||
A mechanism to send an event to an external destination. This process may require serialization. An output plugin is a
|
||||
subclass of the `LogStash::Outputs::Base` class.
|
||||
|
||||
.Output API
|
||||
[horizontal]
|
||||
`#register() -> nil`:: Identical to the API of the same name for input plugins.
|
||||
`#receive(event) -> nil`:: Required. Must handle errors. Used to prepare the given event for transmission to
|
||||
the external destination. Some outputs may buffer the prepared events to batch transmit to the destination.
|
||||
|
||||
[[patch-process]]
|
||||
==== Process
|
||||
|
||||
A bug or feature is identified. An issue is created in the plugin repository. A patch is created and a pull request (PR)
|
||||
is submitted. After review and possible rework the PR is merged and the plugin is published.
|
||||
|
||||
The <<community-maintainer,Community Maintainer Guide>> explains, in more detail, the process of getting a patch accepted,
|
||||
merged and published. The Community Maintainer Guide also details the roles that contributors and maintainers are
|
||||
expected to perform.
|
||||
|
||||
==== Testing Methodologies
|
||||
|
||||
===== Test Driven Development
|
||||
|
||||
Test Driven Development, colloquially known as TDD, describes a methodology for using tests to guide evolution of source
|
||||
code. For our purposes, we are only going to use a part of it, that is, before writing the fix - we create tests that
|
||||
illustrate the bug by failing. We stop when we have written enough code to make the tests pass and submit the fix and
|
||||
tests as a patch. It is not necessary to write the tests before the fix, but it is very easy to write a passing test
|
||||
afterwards that may not actually verify that the fault is really fixed especially if the fault can be triggered via
|
||||
multiple execution paths or varying input data.
|
||||
|
||||
===== The RSpec Framework
|
||||
|
||||
Logstash uses Rspec, a Ruby testing framework, to define and run the test suite. What follows is a summary of various
|
||||
sources.
|
||||
|
||||
. Rspec Example
|
||||
[source,ruby]
|
||||
1 # encoding: utf-8
|
||||
2 require "logstash/devutils/rspec/spec_helper"
|
||||
3 require "logstash/plugin"
|
||||
4
|
||||
5 describe "outputs/riemann" do
|
||||
6 describe "#register" do
|
||||
7 let(:output) do
|
||||
8 LogStash::Plugin.lookup("output", "riemann").new(configuration)
|
||||
9 end
|
||||
10
|
||||
11 context "when no protocol is specified" do
|
||||
12 let(:configuration) { Hash.new }
|
||||
13
|
||||
14 it "the method completes without error" do
|
||||
15 expect {output.register}.not_to raise_error
|
||||
16 end
|
||||
17 end
|
||||
18
|
||||
19 context "when a bad protocol is specified" do
|
||||
20 let(:configuration) { {"protocol" => "fake"} }
|
||||
21
|
||||
22 it "the method fails with error" do
|
||||
23 expect {output.register}.to raise_error
|
||||
24 end
|
||||
25 end
|
||||
26
|
||||
27 context "when the tcp protocol is specified" do
|
||||
28 let(:configuration) { {"protocol" => "tcp"} }
|
||||
29
|
||||
30 it "the method completes without error" do
|
||||
31 expect {output.register}.not_to raise_error
|
||||
32 end
|
||||
33 end
|
||||
34 end
|
||||
35
|
||||
36 describe "#receive" do
|
||||
37 let(:output) do
|
||||
38 LogStash::Plugin.lookup("output", "riemann").new(configuration)
|
||||
39 end
|
||||
40
|
||||
41 context "when operating normally" do
|
||||
42 let(:configuration) { Hash.new }
|
||||
43 let(:event) do
|
||||
44 data = {"message"=>"hello", "@version"=>"1",
|
||||
45 "@timestamp"=>"2015-06-03T23:34:54.076Z",
|
||||
46 "host"=>"vagrant-ubuntu-trusty-64"}
|
||||
47 LogStash::Event.new(data)
|
||||
48 end
|
||||
49
|
||||
50 before(:example) do
|
||||
51 output.register
|
||||
52 end
|
||||
53
|
||||
54 it "should accept the event" do
|
||||
55 expect { output.receive event }.not_to raise_error
|
||||
56 end
|
||||
57 end
|
||||
58 end
|
||||
59 end
|
||||
|
||||
.Describe blocks (lines 5, 6 and 36 in Example 1)
|
||||
[source,ruby]
|
||||
describe(string){block} -> nil
|
||||
describe(Class){block} -> nil
|
||||
|
||||
With RSpec, we are always describing the plugin method behavior. The describe block is added in logical sections and can
|
||||
accept either an existing class name or a string. The string used in line 5 is the plugin name. Line 6 is the register
|
||||
method, line 36 is the receive method. It is a RSpec convention to prefix instance methods with one hash and class
|
||||
methods with one dot.
|
||||
|
||||
.Context blocks (lines 11, 19, 27 and 41)
|
||||
[source,ruby]
|
||||
context(string){block} -> nil
|
||||
|
||||
In RSpec, context blocks define sections that group tests by a variation. The string should start with the word `when`
|
||||
and then detail the variation. See line 11. The tests in the content block should should only be for that variation.
|
||||
|
||||
.Let blocks (lines 7, 12, 20, 28, 37, 42 and 43)
|
||||
[source,ruby]
|
||||
let(symbol){block} -> nil
|
||||
|
||||
In RSpec, `let` blocks define resources for use in the test blocks. These resources are reinitialized for every test
|
||||
block. They are available as method calls inside the test block. Define `let` blocks in `describe` and `context` blocks,
|
||||
which scope the `let` block and any other nested blocks.
|
||||
You can use other `let` methods defined later within the `let` block body. See lines 7-9, which define the output resource
|
||||
and use the configuration method, defined with different variations in lines 12, 20 and 28.
|
||||
|
||||
.Before blocks (line 50)
|
||||
[source,ruby]
|
||||
before(symbol){block} -> nil - symbol is one of :suite, :context, :example, but :all and :each are synonyms for :suite and :example respectively.
|
||||
|
||||
In RSpec, `before` blocks are used to further set up any resources that would have been initialized in a `let` block.
|
||||
You cannot define `let` blocks inside `before` blocks.
|
||||
|
||||
You can also define `after` blocks, which are typically used to clean up any setup activity performed by a `before` block.
|
||||
|
||||
.It blocks (lines 14, 22, 30 and 54)
|
||||
[source,ruby]
|
||||
it(string){block} -> nil
|
||||
|
||||
In RSpec, `it` blocks set the expectations that verify the behavior of the tested code. The string should not start with
|
||||
'it' or 'should', but needs to express the outcome of the expectation. When put together the texts from the enclosing
|
||||
describe, `context` and `it` blocks should form a fairly readable sentence, as in lines 5, 6, 11 and 14:
|
||||
|
||||
[source,ruby]
|
||||
outputs/riemann
|
||||
#register when no protocol is specified the method completes without error
|
||||
|
||||
Readable code like this make the goals of tests easy to understand.
|
||||
|
||||
.Expect method (lines 15, 23, 31, 55)
|
||||
[source,ruby]
|
||||
expect(object){block} -> nil
|
||||
|
||||
In RSpec, the expect method verifies a statement that compares an actual result to an expected result. The `expect` method
|
||||
is usually paired with a call to the `to` or `not_to` methods. Use the block form when expecting errors or observing for
|
||||
changes. The `to` or `not_to` methods require a `matcher` object that encapsulates the expected value. The argument form
|
||||
of the `expect` method encapsulates the actual value. When put together the whole line tests the actual against the
|
||||
expected value.
|
||||
|
||||
.Matcher methods (lines 15, 23, 31, 55)
|
||||
[source,ruby]
|
||||
raise_error(error class|nil) -> matcher instance
|
||||
be(object) -> matcher instance
|
||||
eq(object) -> matcher instance
|
||||
eql(object) -> matcher instance
|
||||
for more see http://www.relishapp.com/rspec/rspec-expectations/docs/built-in-matchers
|
||||
|
||||
In RSpec, a matcher is an object generated by the equivalent method call (be, eq) that will be used to evaluate the
|
||||
expected against the actual values.
|
||||
|
||||
==== Putting it all together
|
||||
|
||||
This example fixes an https://github.com/logstash-plugins/logstash-output-zeromq/issues/9[issue] in the ZeroMQ output
|
||||
plugin. The issue does not require knowledge of ZeroMQ.
|
||||
|
||||
The activities in this example have the following prerequisites:
|
||||
|
||||
--
|
||||
* A minimal knowledge of Git and Github. See the https://help.github.com/categories/bootcamp/[Github boot camp].
|
||||
* A text editor.
|
||||
* A JRuby https://www.ruby-lang.org/en/documentation/installation/#managers[runtime]
|
||||
https://howistart.org/posts/ruby/1[environment]. The `chruby` tool manages Ruby versions.
|
||||
* JRuby 1.7.22 or later.
|
||||
* The `bundler` and `rake` gems installed.
|
||||
* ZeroMQ http://zeromq.org/intro:get-the-software[installed].
|
||||
--
|
||||
|
||||
. In Github, fork the ZeroMQ https://github.com/logstash-plugins/logstash-output-zeromq[output plugin repository].
|
||||
|
||||
. On your local machine, https://help.github.com/articles/fork-a-repo/[clone] the fork to a known folder such as
|
||||
`logstash/`.
|
||||
|
||||
. Open the following files in a text editor:
|
||||
* `logstash-output-zeromq/lib/logstash/outputs/zeromq.rb`
|
||||
* `logstash-output-zeromq/lib/logstash/util/zeromq.rb`
|
||||
* `logstash-output-zeromq/spec/outputs/zeromq_spec.rb`
|
||||
|
||||
. According to the issue, log output in server mode must indicate `bound`. Furthermore, the test file contains no tests.
|
||||
+
|
||||
NOTE: Line 21 of `util/zeromq.rb` reads `@logger.info("0mq: #{server? ? 'connected' : 'bound'}", :address => address)`
|
||||
|
||||
. In the text editor, set the file encoding and require `zeromq.rb` for the file `zeromq_spec.rb` by adding the following
|
||||
lines:
|
||||
+
|
||||
[source,ruby]
|
||||
# encoding: utf-8
|
||||
require "logstash/outputs/zeromq"
|
||||
require "logstash/devutils/rspec/spec_helper"
|
||||
|
||||
. The desired error message should read:
|
||||
+
|
||||
[source,ruby]
|
||||
LogStash::Outputs::ZeroMQ when in server mode a 'bound' info line is logged
|
||||
+
|
||||
To properly generate this message, add a `describe` block with the fully qualified class name as the argument, a context
|
||||
block, and an `it` block.
|
||||
+
|
||||
[source,ruby]
|
||||
describe LogStash::Outputs::ZeroMQ do
|
||||
context "when in server mode" do
|
||||
it "a 'bound' info line is logged" do
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
. To add the missing test, use an instance of the ZeroMQ output and a substitute logger. This example uses an RSpec feature
|
||||
called _test doubles_ as the substitute logger.
|
||||
+
|
||||
Add the following lines to `zeromq_spec.rb`, after `describe LogStash::Outputs::ZeroMQ do` and before `context "when in
|
||||
server mode" do`:
|
||||
[source,ruby]
|
||||
let(:output) { described_class.new("mode" => "server", "topology" => "pushpull" }
|
||||
let(:tracer) { double("logger") }
|
||||
|
||||
. Add the body to the `it` block. Add the following five lines after the line `context "when in server mode" do`:
|
||||
[source,ruby]
|
||||
allow(tracer).to receive(:debug)<1>
|
||||
output.logger = logger<2>
|
||||
expect(tracer).to receive(:info).with("0mq: bound", {:address=>"tcp://127.0.0.1:2120"})<3>
|
||||
output.register<4>
|
||||
output.do_close<5>
|
||||
|
||||
<1> Allow the double to receive `debug` method calls.
|
||||
<2> Make the output use the test double.
|
||||
<3> Set an expectation on the test to receive an `info` method call.
|
||||
<4> Call `register` on the output.
|
||||
<5> Call `do_close` on the output so the test does not hang.
|
||||
|
||||
At the end of the modifications, the relevant code section reads:
|
||||
|
||||
[source,ruby]
|
||||
--------
|
||||
# encoding: utf-8
|
||||
require "logstash/outputs/zeromq"
|
||||
require "logstash/devutils/rspec/spec_helper"
|
||||
|
||||
describe LogStash::Outputs::ZeroMQ do
|
||||
let(:output) { described_class.new("mode" => "server", "topology" => "pushpull") }
|
||||
let(:tracer) { double("logger") }
|
||||
|
||||
context "when in server mode" do
|
||||
it "a ‘bound’ info line is logged" do
|
||||
allow(tracer).to receive(:debug)
|
||||
output.logger = tracer
|
||||
expect(tracer).to receive(:info).with("0mq: bound", {:address=>"tcp://127.0.0.1:2120"})
|
||||
output.register
|
||||
output.do_close
|
||||
end
|
||||
end
|
||||
end
|
||||
--------
|
||||
|
||||
To run this test:
|
||||
|
||||
. Open a terminal window
|
||||
. Navigate to the cloned plugin folder
|
||||
. The first time you run the test, run the command `bundle install`
|
||||
. Run the command `bundle exec rspec`
|
||||
|
||||
Assuming all prerequisites were installed correctly, the test fails with output similar to:
|
||||
|
||||
[source,shell]
|
||||
--------
|
||||
Using Accessor#strict_set for specs
|
||||
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :couchdb=>true, :elasticsearch=>true,
|
||||
:elasticsearch_secure=>true, :export_cypher=>true, :integration=>true, :windows=>true}
|
||||
|
||||
LogStash::Outputs::ZeroMQ
|
||||
when in server mode
|
||||
a ‘bound’ info line is logged (FAILED - 1)
|
||||
|
||||
Failures:
|
||||
|
||||
1) LogStash::Outputs::ZeroMQ when in server mode a ‘bound’ info line is logged
|
||||
Failure/Error: output.register
|
||||
Double "logger" received :info with unexpected arguments
|
||||
expected: ("0mq: bound", {:address=>"tcp://127.0.0.1:2120"})
|
||||
got: ("0mq: connected", {:address=>"tcp://127.0.0.1:2120"})
|
||||
# ./lib/logstash/util/zeromq.rb:21:in `setup'
|
||||
# ./lib/logstash/outputs/zeromq.rb:92:in `register'
|
||||
# ./lib/logstash/outputs/zeromq.rb:91:in `register'
|
||||
# ./spec/outputs/zeromq_spec.rb:13:in `(root)'
|
||||
# /Users/guy/.gem/jruby/1.9.3/gems/rspec-wait-0.0.7/lib/rspec/wait.rb:46:in `(root)'
|
||||
|
||||
Finished in 0.133 seconds (files took 1.28 seconds to load)
|
||||
1 example, 1 failure
|
||||
|
||||
Failed examples:
|
||||
|
||||
rspec ./spec/outputs/zeromq_spec.rb:10 # LogStash::Outputs::ZeroMQ when in server mode a ‘bound’ info line is logged
|
||||
|
||||
Randomized with seed 2568
|
||||
--------
|
||||
|
||||
To correct the error, open the `util/zeromq.rb` file in your text editor and swap the positions of the words `connected`
|
||||
and `bound` on line 21. Line 21 now reads:
|
||||
|
||||
[source,ruby]
|
||||
@logger.info("0mq: #{server? ? 'bound' : 'connected'}", :address => address)
|
||||
|
||||
Run the test again with the `bundle exec rspec` command.
|
||||
|
||||
The test passes with output similar to:
|
||||
|
||||
[source,shell]
|
||||
--------
|
||||
Using Accessor#strict_set for specs
|
||||
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :couchdb=>true, :elasticsearch=>true, :elasticsearch_secure=>true, :export_cypher=>true, :integration=>true, :windows=>true}
|
||||
|
||||
LogStash::Outputs::ZeroMQ
|
||||
when in server mode
|
||||
a ‘bound’ info line is logged
|
||||
|
||||
Finished in 0.114 seconds (files took 1.22 seconds to load)
|
||||
1 example, 0 failures
|
||||
|
||||
Randomized with seed 45887
|
||||
--------
|
||||
|
||||
https://help.github.com/articles/fork-a-repo/#next-steps[Commit] the changes to git and Github.
|
||||
|
||||
Your pull request is visible from the https://github.com/logstash-plugins/logstash-output-zeromq/pulls[Pull Requests]
|
||||
section of the original Github repository. The plugin maintainers review your work, suggest changes if necessary, and
|
||||
merge and publish a new version of the plugin.
|
||||
51
docs/jp/static/contributing-to-logstash.asciidoc
Normal file
51
docs/jp/static/contributing-to-logstash.asciidoc
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
[[contributing-to-logstash]]
|
||||
== Contributing to Logstash
|
||||
|
||||
Before version 1.5, Logstash included all plugins in each release. This made it
|
||||
easy to make use of any plugin, but it complicated plugin development--a new
|
||||
release of Logstash became necessary if a plugin needed patching. Since version
|
||||
1.5, all plugins are independent of the Logstash core. Now you can add your own
|
||||
input, codec, filter, or output plugins to Logstash much more easily!
|
||||
|
||||
[float]
|
||||
=== Adding plugins
|
||||
|
||||
Since plugins can now be developed and deployed independently of the Logstash
|
||||
core, there are documents which guide you through the process of coding and
|
||||
deploying your own plugins:
|
||||
|
||||
* <<plugin-generator,Generating a New Plugin>>
|
||||
* http://www.elasticsearch.org/guide/en/logstash/current/_how_to_write_a_logstash_input_plugin.html[How to write a Logstash input plugin]
|
||||
* http://www.elasticsearch.org/guide/en/logstash/current/_how_to_write_a_logstash_codec_plugin.html[How to write a Logstash codec plugin]
|
||||
* http://www.elasticsearch.org/guide/en/logstash/current/_how_to_write_a_logstash_filter_plugin.html[How to write a Logstash filter plugin]
|
||||
* http://www.elasticsearch.org/guide/en/logstash/current/_how_to_write_a_logstash_output_plugin.html[How to write a Logstash output plugin]
|
||||
* <<contributing-patch-plugin,Contributing a Patch to a Logstash Plugin>>
|
||||
* <<community-maintainer,Community Maintainer's Guide>>
|
||||
* <<submitting-plugin,Submitting a Plugin>>
|
||||
|
||||
[float]
|
||||
==== Plugin Shutdown APIs
|
||||
|
||||
Starting in Logstash 2.0, we changed how input plugins shut down to increase shutdown reliability. There are three methods
|
||||
for plugin shutdown: `stop`, `stop?`, and `close`.
|
||||
|
||||
* Call the `stop` method from outside the plugin thread. This method signals the plugin to stop.
|
||||
* The `stop?` method returns `true` when the `stop` method has already been called for that plugin.
|
||||
* The `close` method performs final bookkeeping and cleanup after the plugin's `run` method and the plugin's thread both
|
||||
exit. The `close` method is a a new name for the method known as `teardown` in previous versions of Logstash.
|
||||
|
||||
The `shutdown`, `finished`, `finished?`, `running?`, and `terminating?` methods are redundant and no longer present in the
|
||||
Plugin Base class.
|
||||
|
||||
Sample code for the plugin shutdown APIs is https://github.com/logstash-plugins/logstash-input-example/blob/master/lib/logstash/inputs/example.rb[available].
|
||||
|
||||
[float]
|
||||
=== Extending Logstash core
|
||||
|
||||
We also welcome contributions and bug fixes to the Logstash core feature set.
|
||||
|
||||
Please read through our
|
||||
https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md[contribution]
|
||||
guide, and the Logstash
|
||||
https://github.com/elastic/logstash/blob/master/README.md[readme]
|
||||
document.
|
||||
146
docs/jp/static/deploying.asciidoc
Normal file
146
docs/jp/static/deploying.asciidoc
Normal file
|
|
@ -0,0 +1,146 @@
|
|||
[[deploying-and-scaling]]
|
||||
== Deploying and Scaling Logstash
|
||||
|
||||
As your use case for Logstash evolves, the preferred architecture at a given scale will change. This section discusses
|
||||
a range of Logstash architectures in increasing order of complexity, starting from a minimal installation and adding
|
||||
elements to the system. The example deployments in this section write to an Elasticsearch cluster, but Logstash can
|
||||
write to a large variety of {logstash}output-plugins.html[endpoints].
|
||||
|
||||
[float]
|
||||
[[deploying-minimal-install]]
|
||||
=== The Minimal Installation
|
||||
|
||||
The minimal Logstash installation has one Logstash instance and one Elasticsearch instance. These instances are
|
||||
directly connected. Logstash uses an {logstash}input-plugins.html[_input plugin_] to ingest data and an
|
||||
Elasticsearch {logstash}output-plugins.html[_output plugin_] to index the data in Elasticsearch, following the Logstash
|
||||
{logstash}pipeline.html[_processing pipeline_]. A Logstash instance has a fixed pipeline constructed at startup,
|
||||
based on the instance’s configuration file. You must specify an input plugin. Output defaults to `stdout`, and the
|
||||
filtering section of the pipeline, which is discussed in the next section, is optional.
|
||||
|
||||
image::static/images/deploy_1.png[]
|
||||
|
||||
[float]
|
||||
[[deploying-filter-threads]]
|
||||
=== Using Filters
|
||||
|
||||
Log data is typically unstructured, often contains extraneous information that isn’t relevant to your use case, and
|
||||
sometimes is missing relevant information that can be derived from the log contents. You can use a
|
||||
{logstash}filter-plugins.html[filter plugin] to parse the log into fields, remove unnecessary information, and derive
|
||||
additional information from the existing fields. For example, filters can derive geolocation information from an IP
|
||||
address and add that information to the logs, or parse and structure arbitrary text with the
|
||||
{logstash}plugins-filters-grok.html[grok] filter.
|
||||
|
||||
Adding a filter plugin can significantly affect performance, depending on the amount of computation the filter plugin
|
||||
performs, as well as on the volume of the logs being processed. The `grok` filter’s regular expression computation is
|
||||
particularly resource-intensive. One way to address this increased demand for computing resources is to use
|
||||
parallel processing on multicore machines. Use the `-w` switch to set the number of execution threads for Logstash
|
||||
filtering tasks. For example the `bin/logstash -w 8` command uses eight different threads for filter processing.
|
||||
|
||||
image::static/images/deploy_2.png[]
|
||||
|
||||
[float]
|
||||
[[deploying-filebeat]]
|
||||
=== Using Filebeat
|
||||
|
||||
https://www.elastic.co/guide/en/beats/filebeat/current/index.html[Filebeat] is a lightweight, resource-friendly tool
|
||||
written in Go that collects logs from files on the server and forwards these logs to other machines for processing.
|
||||
Filebeat uses the https://www.elastic.co/guide/en/beats/libbeat/current/index.html[Beats] protocol to communicate with a
|
||||
centralized Logstash instance. Configure the Logstash instances that receive Beats data to use the
|
||||
{logstash}plugins-inputs-beats.html[Beats input plugin].
|
||||
|
||||
Filebeat uses the computing resources of the machine hosting the source data, and the Beats input plugin minimizes the
|
||||
resource demands on the Logstash instance, making this architecture attractive for use cases with resource constraints.
|
||||
|
||||
image::static/images/deploy_3.png[]
|
||||
|
||||
[float]
|
||||
[[deploying-larger-cluster]]
|
||||
=== Scaling to a Larger Elasticsearch Cluster
|
||||
|
||||
Typically, Logstash does not communicate with a single Elasticsearch node, but with a cluster that comprises several
|
||||
nodes. By default, Logstash uses the HTTP protocol to move data into the cluster.
|
||||
|
||||
You can use the Elasticsearch HTTP REST APIs to index data into the Elasticsearch cluster. These APIs represent the
|
||||
indexed data in JSON. Using the REST APIs does not require the Java client classes or any additional JAR
|
||||
files and has no performance disadvantages compared to the transport or node protocols. You can secure communications
|
||||
that use the HTTP REST APIs by using {xpack}xpack-security.html[{security}], which supports SSL and HTTP basic authentication.
|
||||
|
||||
When you use the HTTP protocol, you can configure the Logstash Elasticsearch output plugin to automatically
|
||||
load-balance indexing requests across a
|
||||
specified set of hosts in the Elasticsearch cluster. Specifying multiple Elasticsearch nodes also provides high availability for the Elasticsearch cluster by routing traffic to active Elasticsearch nodes.
|
||||
|
||||
You can also use the Elasticsearch Java APIs to serialize the data into a binary representation, using
|
||||
the transport protocol. The transport protocol can sniff the endpoint of the request and select an
|
||||
arbitrary client or data node in the Elasticsearch cluster.
|
||||
|
||||
Using the HTTP or transport protocols keep your Logstash instances separate from the Elasticsearch cluster. The node
|
||||
protocol, by contrast, has the machine running the Logstash instance join the Elasticsearch cluster, running an
|
||||
Elasticsearch instance. The data that needs indexing propagates from this node to the rest of the cluster. Since the
|
||||
machine is part of the cluster, the cluster topology is available, making the node protocol a good fit for use cases
|
||||
that use a relatively small number of persistent connections.
|
||||
|
||||
You can also use a third-party hardware or software load balancer to handle connections between Logstash and
|
||||
external applications.
|
||||
|
||||
NOTE: Make sure that your Logstash configuration does not connect directly to Elasticsearch dedicated
|
||||
{ref}modules-node.html[master nodes], which perform dedicated cluster management. Connect Logstash to client or data
|
||||
nodes to protect the stability of your Elasticsearch cluster.
|
||||
|
||||
image::static/images/deploy_4.png[]
|
||||
|
||||
[float]
|
||||
[[deploying-message-queueing]]
|
||||
=== Managing Throughput Spikes with Message Queueing
|
||||
|
||||
When the data coming into a Logstash pipeline exceeds the Elasticsearch cluster's ability to ingest the data, you can
|
||||
use a message broker as a buffer. By default, Logstash throttles incoming events when
|
||||
indexer consumption rates fall below incoming data rates. Since this throttling can lead to events being buffered at
|
||||
the data source, preventing back pressure with message brokers becomes an important part of managing your deployment.
|
||||
|
||||
Adding a message broker to your Logstash deployment also provides a level of protection from data loss. When a Logstash
|
||||
instance that has consumed data from the message broker fails, the data can be replayed from the message broker to an
|
||||
active Logstash instance.
|
||||
|
||||
Several third-party message brokers exist, such as Redis, Kafka, or RabbitMQ. Logstash provides input and output plugins
|
||||
to integrate with several of these third-party message brokers. When your Logstash deployment has a message broker
|
||||
configured, Logstash functionally exists in two phases: shipping instances, which handles data ingestion and storage in
|
||||
the message broker, and indexing instances, which retrieve the data from the message broker, apply any configured
|
||||
filtering, and write the filtered data to an Elasticsearch index.
|
||||
|
||||
image::static/images/deploy_5.png[]
|
||||
|
||||
[float]
|
||||
[[deploying-logstash-ha]]
|
||||
=== Multiple Connections for Logstash High Availability
|
||||
|
||||
To make your Logstash deployment more resilient to individual instance failures, you can set up a load balancer between
|
||||
your data source machines and the Logstash cluster. The load balancer handles the individual connections to the
|
||||
Logstash instances to ensure continuity of data ingestion and processing even when an individual instance is unavailable.
|
||||
|
||||
image::static/images/deploy_6.png[]
|
||||
|
||||
The architecture in the previous diagram is unable to process input from a specific type, such as an RSS feed or a
|
||||
file, if the Logstash instance dedicated to that input type becomes unavailable. For more robust input processing,
|
||||
configure each Logstash instance for multiple inputs, as in the following diagram:
|
||||
|
||||
image::static/images/deploy_7.png[]
|
||||
|
||||
This architecture parallelizes the Logstash workload based on the inputs you configure. With more inputs, you can add
|
||||
more Logstash instances to scale horizontally. Separate parallel pipelines also increases the reliability of your stack
|
||||
by eliminating single points of failure.
|
||||
|
||||
[float]
|
||||
[[deploying-scaling]]
|
||||
=== Scaling Logstash
|
||||
|
||||
A mature Logstash deployment typically has the following pipeline:
|
||||
|
||||
* The _input_ tier consumes data from the source, and consists of Logstash instances with the proper input plugins.
|
||||
* The _message broker_ serves as a buffer to hold ingested data and serve as failover protection.
|
||||
* The _filter_ tier applies parsing and other processing to the data consumed from the message broker.
|
||||
* The _indexing_ tier moves the processed data into Elasticsearch.
|
||||
|
||||
Any of these layers can be scaled by adding computing resources. Examine the performance of these components regularly
|
||||
as your use case evolves and add resources as needed. When Logstash routinely throttles incoming events, consider
|
||||
adding storage for your message broker. Alternately, increase the Elasticsearch cluster's rate of data consumption by
|
||||
adding more Logstash indexing instances.
|
||||
119
docs/jp/static/docker.asciidoc
Normal file
119
docs/jp/static/docker.asciidoc
Normal file
|
|
@ -0,0 +1,119 @@
|
|||
[[docker]]
|
||||
=== Running Logstash on Docker
|
||||
Docker images for Logstash are available from the Elastic Docker
|
||||
registry.
|
||||
|
||||
Obtaining Logstash for Docker is as simple as issuing a +docker pull+
|
||||
command against the Elastic Docker registry.
|
||||
|
||||
ifeval::["{release-state}"=="unreleased"]
|
||||
|
||||
However, version {logstash_version} of Logstash has not yet been
|
||||
released, so no Docker image is currently available for this version.
|
||||
|
||||
endif::[]
|
||||
|
||||
ifeval::["{release-state}"!="unreleased"]
|
||||
|
||||
The Docker image for Logstash {logstash_version} can be retrieved with
|
||||
the following command:
|
||||
|
||||
["source","sh",subs="attributes"]
|
||||
--------------------------------------------
|
||||
docker pull {docker-image}
|
||||
--------------------------------------------
|
||||
|
||||
endif::[]
|
||||
|
||||
==== Configuring Logstash for Docker
|
||||
|
||||
Logstash differentiates between two types of configuration:
|
||||
<<config-setting-files,Settings and Pipeline Configuration>>.
|
||||
|
||||
===== Pipeline Configuration
|
||||
|
||||
It is essential to place your pipeline configuration where it can be
|
||||
found by Logstash. By default, the container will look in
|
||||
+/usr/share/logstash/pipeline/+ for pipeline configuration files.
|
||||
|
||||
In this example we use a bind-mounted volume to provide the
|
||||
configuration via the +docker run+ command:
|
||||
|
||||
["source","sh",subs="attributes"]
|
||||
--------------------------------------------
|
||||
docker run --rm -it -v ~/pipeline/:/usr/share/logstash/pipeline/ {docker-image}
|
||||
--------------------------------------------
|
||||
|
||||
Every file in the host directory +~/pipeline/+ will then be parsed
|
||||
by Logstash as pipeline configuration.
|
||||
|
||||
If you don't provide configuration to Logstash, it will run with a
|
||||
minimal config that listens for messages from the
|
||||
<<plugins-inputs-beats,Beats input plugin>> and echoes any that are
|
||||
received to `stdout`. In this case, the startup logs will be similar
|
||||
to the following:
|
||||
|
||||
["source","text"]
|
||||
--------------------------------------------
|
||||
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties.
|
||||
[2016-10-26T05:11:34,992][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
|
||||
[2016-10-26T05:11:35,068][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
|
||||
[2016-10-26T05:11:35,078][INFO ][org.logstash.beats.Server] Starting server on port: 5044
|
||||
[2016-10-26T05:11:35,078][INFO ][logstash.pipeline ] Pipeline main started
|
||||
[2016-10-26T05:11:35,105][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
|
||||
--------------------------------------------
|
||||
|
||||
This is the default configuration for the image, defined in
|
||||
+/usr/share/logstash/pipeline/logstash.conf+. If this is the
|
||||
behaviour that you are observing, ensure that your pipeline
|
||||
configuration is being picked up correctly, and that you are replacing
|
||||
either +logstash.conf+ or the entire +pipeline+ directory.
|
||||
|
||||
===== Settings Files
|
||||
|
||||
Settings files can also be provided through bind-mounts. Logstash
|
||||
expects to find them at +/usr/share/logstash/config/+.
|
||||
|
||||
It's possible to provide an entire directory containing all needed
|
||||
files:
|
||||
|
||||
["source","sh",subs="attributes"]
|
||||
--------------------------------------------
|
||||
docker run --rm -it -v ~/settings/:/usr/share/logstash/config/ {docker-image}
|
||||
--------------------------------------------
|
||||
|
||||
Alternatively, a single file can be mounted:
|
||||
|
||||
["source","sh",subs="attributes"]
|
||||
--------------------------------------------
|
||||
docker run --rm -it -v ~/settings/logstash.yml:/usr/share/logstash/config/logstash.yml {docker-image}
|
||||
--------------------------------------------
|
||||
|
||||
NOTE: Bind-mounted configuration files will retain the same permissions and
|
||||
ownership within the container that they have on the host system. Be sure
|
||||
to set permissions such that the files will be readable and, ideally, not
|
||||
writeable by the container's +logstash+ user (UID 1000).
|
||||
|
||||
===== Custom Images
|
||||
|
||||
Bind-mounted configuration is not the only option, naturally. If you
|
||||
prefer the _Immutable Infrastructure_ approach, you can prepare a
|
||||
custom image containing your configuration by using a +Dockerfile+
|
||||
like this one:
|
||||
|
||||
["source","dockerfile",subs="attributes"]
|
||||
--------------------------------------------
|
||||
FROM {docker-image}
|
||||
RUN rm -f /usr/share/logstash/pipeline/logstash.conf
|
||||
ADD pipeline/ /usr/share/logstash/pipeline/
|
||||
ADD config/ /usr/share/logstash/config/
|
||||
--------------------------------------------
|
||||
|
||||
Be sure to replace or delete `logstash.conf` in your custom image, so
|
||||
that you don't retain the example config from the base image.
|
||||
|
||||
==== Logging Configuration
|
||||
|
||||
Under Docker, Logstash logs go to standard output by default. To
|
||||
change this behaviour, use any of the techniques above to replace the
|
||||
file at +/usr/share/logstash/config/log4j2.properties+.
|
||||
120
docs/jp/static/event-api.asciidoc
Normal file
120
docs/jp/static/event-api.asciidoc
Normal file
|
|
@ -0,0 +1,120 @@
|
|||
[[event-api]]
|
||||
=== Event API
|
||||
|
||||
This section is targeted for plugin developers and users of Logstash's Ruby filter. Below we document recent
|
||||
changes (starting with version 5.0) in the way users have been accessing Logstash's event based data in
|
||||
custom plugins and in the Ruby filter. Note that <<event-dependent-configuration>>
|
||||
data flow in Logstash's config files -- using <<logstash-config-field-references>> -- is
|
||||
not affected by this change, and will continue to use existing syntax.
|
||||
|
||||
[float]
|
||||
==== Event Object
|
||||
|
||||
Event is the main object that encapsulates data flow internally in Logstash and provides an API for the plugin
|
||||
developers to interact with the event's content. Typically, this API is used in plugins and in a Ruby filter to
|
||||
retrieve data and use it for transformations. Event object contains the original data sent to Logstash and any additional
|
||||
fields created during Logstash's filter stages.
|
||||
|
||||
In 5.0, we've re-implemented the Event class and its supporting classes in pure Java. Since Event is a critical component
|
||||
in data processing, a rewrite in Java improves performance and provides efficient serialization when storing data on disk. For the most part, this change aims at keeping backward compatibility and is transparent to the users. To this extent we've updated and published most of the plugins in Logstash's ecosystem to adhere to the new API changes. However, if you are maintaining a custom plugin, or have a Ruby filter, this change will affect you. The aim of this guide is to describe the new API and provide examples to migrate to the new changes.
|
||||
|
||||
[float]
|
||||
==== Event API
|
||||
|
||||
Prior to version 5.0, developers could access and manipulate event data by directly using Ruby hash syntax. For
|
||||
example, `event[field] = foo`. While this is powerful, our goal is to abstract the internal implementation details
|
||||
and provide well-defined getter and setter APIs.
|
||||
|
||||
**Get API**
|
||||
|
||||
The getter is a read-only access of field-based data in an Event.
|
||||
|
||||
**Syntax:** `event.get(field)`
|
||||
|
||||
**Returns:** Value for this field or nil if the field does not exist. Returned values could be a string,
|
||||
numeric or timestamp scalar value.
|
||||
|
||||
`field` is a structured field sent to Logstash or created after the transformation process. `field` can also
|
||||
be a nested field reference such as `[field][bar]`.
|
||||
|
||||
Examples:
|
||||
|
||||
[source,ruby]
|
||||
--------------------------------------------------
|
||||
event.get("foo" ) # => "baz"
|
||||
event.get("[foo]") # => "zab"
|
||||
event.get("[foo][bar]") # => 1
|
||||
event.get("[foo][bar]") # => 1.0
|
||||
event.get("[foo][bar]") # => [1, 2, 3]
|
||||
event.get("[foo][bar]") # => {"a" => 1, "b" => 2}
|
||||
event.get("[foo][bar]") # => {"a" => 1, "b" => 2, "c" => [1, 2]}
|
||||
--------------------------------------------------
|
||||
|
||||
Accessing @metadata
|
||||
|
||||
[source,ruby]
|
||||
--------------------------------------------------
|
||||
event.get("[@metadata][foo]") # => "baz"
|
||||
--------------------------------------------------
|
||||
|
||||
**Set API**
|
||||
|
||||
This API can be used to mutate data in an Event.
|
||||
|
||||
**Syntax:** `event.set(field, value)`
|
||||
|
||||
**Returns:** The current Event after the mutation, which can be used for chainable calls.
|
||||
|
||||
Examples:
|
||||
|
||||
[source,ruby]
|
||||
--------------------------------------------------
|
||||
event.set("foo", "baz")
|
||||
event.set("[foo]", "zab")
|
||||
event.set("[foo][bar]", 1)
|
||||
event.set("[foo][bar]", 1.0)
|
||||
event.set("[foo][bar]", [1, 2, 3])
|
||||
event.set("[foo][bar]", {"a" => 1, "b" => 2})
|
||||
event.set("[foo][bar]", {"a" => 1, "b" => 2, "c" => [1, 2]})
|
||||
event.set("[@metadata][foo]", "baz")
|
||||
--------------------------------------------------
|
||||
|
||||
Mutating a collection after setting it in the Event has an undefined behaviour and is not allowed.
|
||||
|
||||
[source,ruby]
|
||||
--------------------------------------------------
|
||||
h = {"a" => 1, "b" => 2, "c" => [1, 2]}
|
||||
event.set("[foo][bar]", h)
|
||||
|
||||
h["c"] = [3, 4]
|
||||
event.get("[foo][bar][c]") # => undefined
|
||||
|
||||
Suggested way of mutating collections:
|
||||
|
||||
h = {"a" => 1, "b" => 2, "c" => [1, 2]}
|
||||
event.set("[foo][bar]", h)
|
||||
|
||||
h["c"] = [3, 4]
|
||||
event.set("[foo][bar]", h)
|
||||
|
||||
# Alternatively,
|
||||
event.set("[foo][bar][c]", [3, 4])
|
||||
--------------------------------------------------
|
||||
|
||||
[float]
|
||||
==== Ruby Filter
|
||||
|
||||
The <<plugins-filters-ruby,Ruby Filter>> can be used to execute any ruby code and manipulate event data using the
|
||||
API described above. For example, using the new API:
|
||||
|
||||
[source,ruby]
|
||||
--------------------------------------------------
|
||||
filter {
|
||||
ruby {
|
||||
code => 'event.set("lowercase_field", event.get("message").downcase)'
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
This filter will lowercase the `message` field, and set it to a new field called `lowercase_field`
|
||||
|
||||
39
docs/jp/static/filebeat_modules/apache2/access/pipeline.conf
Normal file
39
docs/jp/static/filebeat_modules/apache2/access/pipeline.conf
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
input {
|
||||
beats {
|
||||
# The port to listen on for filebeat connections.
|
||||
port => 5044
|
||||
# The IP address to listen for filebeat connections.
|
||||
host => "0.0.0.0"
|
||||
}
|
||||
}
|
||||
filter {
|
||||
grok {
|
||||
match => { "message" => ["%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \[%{HTTPDATE:[apache2][access][time]}\] \"%{WORD:[apache2][access][method]} %{DATA:[apache2][access][url]} HTTP/%{NUMBER:[apache2][access][http_version]}\" %{NUMBER:[apache2][access][response_code]} %{NUMBER:[apache2][access][body_sent][bytes]}( \"%{DATA:[apache2][access][referrer]}\")?( \"%{DATA:[apache2][access][agent]}\")?",
|
||||
"%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \\[%{HTTPDATE:[apache2][access][time]}\\] \"-\" %{NUMBER:[apache2][access][response_code]} -" ] }
|
||||
remove_field => "message"
|
||||
}
|
||||
mutate {
|
||||
add_field => { "read_timestamp" => "%{@timestamp}" }
|
||||
}
|
||||
date {
|
||||
match => [ "[apache2][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
|
||||
remove_field => "[apache2][access][time]"
|
||||
}
|
||||
useragent {
|
||||
source => "[apache2][access][agent]"
|
||||
target => "[apache2][access][user_agent]"
|
||||
remove_field => "[apache2][access][agent]"
|
||||
}
|
||||
geoip {
|
||||
source => "[apache2][access][remote_ip]"
|
||||
target => "[apache2][access][geoip]"
|
||||
}
|
||||
}
|
||||
output {
|
||||
elasticsearch {
|
||||
hosts => localhost
|
||||
manage_template => false
|
||||
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
|
||||
document_type => "%{[@metadata][type]}"
|
||||
}
|
||||
}
|
||||
33
docs/jp/static/filebeat_modules/apache2/error/pipeline.conf
Normal file
33
docs/jp/static/filebeat_modules/apache2/error/pipeline.conf
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
input {
|
||||
beats {
|
||||
# The port to listen on for filebeat connections.
|
||||
port => 5044
|
||||
# The IP address to listen for filebeat connections.
|
||||
host => "0.0.0.0"
|
||||
}
|
||||
}
|
||||
filter {
|
||||
grok {
|
||||
match => { "message" => ["\[%{APACHE_TIME:[apache2][error][timestamp]}\] \[%{LOGLEVEL:[apache2][error][level]}\]( \[client %{IPORHOST:[apache2][error][client]}\])? %{GREEDYDATA:[apache2][error][message]}",
|
||||
"\[%{APACHE_TIME:[apache2][error][timestamp]}\] \[%{DATA:[apache2][error][module]}:%{LOGLEVEL:[apache2][error][level]}\] \[pid %{NUMBER:[apache2][error][pid]}(:tid %{NUMBER:[apache2][error][tid]})?\]( \[client %{IPORHOST:[apache2][error][client]}\])? %{GREEDYDATA:[apache2][error][message1]}" ] }
|
||||
pattern_definitions => {
|
||||
"APACHE_TIME" => "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}"
|
||||
}
|
||||
remove_field => "message"
|
||||
}
|
||||
mutate {
|
||||
rename => { "[apache2][error][message1]" => "[apache2][error][message]" }
|
||||
}
|
||||
date {
|
||||
match => [ "[apache2][error][timestamp]", "EEE MMM dd H:m:s YYYY", "EEE MMM dd H:m:s.SSSSSS YYYY" ]
|
||||
remove_field => "[apache2][error][timestamp]"
|
||||
}
|
||||
}
|
||||
output {
|
||||
elasticsearch {
|
||||
hosts => localhost
|
||||
manage_template => false
|
||||
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
|
||||
document_type => "%{[@metadata][type]}"
|
||||
}
|
||||
}
|
||||
37
docs/jp/static/filebeat_modules/mysql/error/pipeline.conf
Normal file
37
docs/jp/static/filebeat_modules/mysql/error/pipeline.conf
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
input {
|
||||
beats {
|
||||
# The port to listen on for filebeat connections.
|
||||
port => 5044
|
||||
# The IP address to listen for filebeat connections.
|
||||
host => "0.0.0.0"
|
||||
}
|
||||
}
|
||||
filter {
|
||||
grok {
|
||||
match => { "message" => ["%{LOCALDATETIME:[mysql][error][timestamp]} (\[%{DATA:[mysql][error][level]}\] )?%{GREEDYDATA:[mysql][error][message]}",
|
||||
"%{TIMESTAMP_ISO8601:[mysql][error][timestamp]} %{NUMBER:[mysql][error][thread_id]} \[%{DATA:[mysql][error][level]}\] %{GREEDYDATA:[mysql][error][message1]}",
|
||||
"%{GREEDYDATA:[mysql][error][message2]}"] }
|
||||
pattern_definitions => {
|
||||
"LOCALDATETIME" => "[0-9]+ %{TIME}"
|
||||
}
|
||||
remove_field => "message"
|
||||
}
|
||||
mutate {
|
||||
rename => { "[mysql][error][message1]" => "[mysql][error][message]" }
|
||||
}
|
||||
mutate {
|
||||
rename => { "[mysql][error][message2]" => "[mysql][error][message]" }
|
||||
}
|
||||
date {
|
||||
match => [ "[mysql][error][timestamp]", "ISO8601", "YYMMdd H:m:s" ]
|
||||
remove_field => "[apache2][access][time]"
|
||||
}
|
||||
}
|
||||
output {
|
||||
elasticsearch {
|
||||
hosts => localhost
|
||||
manage_template => false
|
||||
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
|
||||
document_type => "%{[@metadata][type]}"
|
||||
}
|
||||
}
|
||||
31
docs/jp/static/filebeat_modules/mysql/slowlog/pipeline.conf
Normal file
31
docs/jp/static/filebeat_modules/mysql/slowlog/pipeline.conf
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
input {
|
||||
beats {
|
||||
# The port to listen on for filebeat connections.
|
||||
port => 5044
|
||||
# The IP address to listen for filebeat connections.
|
||||
host => "0.0.0.0"
|
||||
}
|
||||
}
|
||||
filter {
|
||||
grok {
|
||||
match => { "message" => ["^# User@Host: %{USER:[mysql][slowlog][user]}(\[[^\]]+\])? @ %{HOSTNAME:[mysql][slowlog][host]} \[(IP:[mysql][slowlog][ip])?\](\s*Id:\s* %{NUMBER:[mysql][slowlog][id]})?\n# Query_time: %{NUMBER:[mysql][slowlog][query_time][sec]}\s* Lock_time: %{NUMBER:[mysql][slowlog][lock_time][sec]}\s* Rows_sent: %{NUMBER:[mysql][slowlog][rows_sent]}\s* Rows_examined: %{NUMBER:[mysql][slowlog][rows_examined]}\n(SET timestamp=%{NUMBER:[mysql][slowlog][timestamp]};\n)?%{GREEDYMULTILINE:[mysql][slowlog][query]}"] }
|
||||
pattern_definitions => {
|
||||
"GREEDYMULTILINE" => "(.|\n)*"
|
||||
}
|
||||
remove_field => "message"
|
||||
}
|
||||
date {
|
||||
match => [ "[mysql][slowlog][timestamp]", "UNIX" ]
|
||||
}
|
||||
mutate {
|
||||
gsub => ["[mysql][slowlog][query]", "\n# Time: [0-9]+ [0-9][0-9]:[0-9][0-9]:[0-9][0-9](\\.[0-9]+)?$", ""]
|
||||
}
|
||||
}
|
||||
output {
|
||||
elasticsearch {
|
||||
hosts => localhost
|
||||
manage_template => false
|
||||
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
|
||||
document_type => "%{[@metadata][type]}"
|
||||
}
|
||||
}
|
||||
38
docs/jp/static/filebeat_modules/nginx/access/pipeline.conf
Normal file
38
docs/jp/static/filebeat_modules/nginx/access/pipeline.conf
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
input {
|
||||
beats {
|
||||
# The port to listen on for filebeat connections.
|
||||
port => 5044
|
||||
# The IP address to listen for filebeat connections.
|
||||
host => "0.0.0.0"
|
||||
}
|
||||
}
|
||||
filter {
|
||||
grok {
|
||||
match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{HTTPDATE:[nginx][access][time]}\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}\" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} \"%{DATA:[nginx][access][referrer]}\" \"%{DATA:[nginx][access][agent]}\""] }
|
||||
remove_field => "message"
|
||||
}
|
||||
mutate {
|
||||
rename => { "@timestamp" => "read_timestamp" }
|
||||
}
|
||||
date {
|
||||
match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
|
||||
remove_field => "[nginx][access][time]"
|
||||
}
|
||||
useragent {
|
||||
source => "[nginx][access][agent]"
|
||||
target => "[nginx][access][user_agent]"
|
||||
remove_field => "[nginx][access][agent]"
|
||||
}
|
||||
geoip {
|
||||
source => "[nginx][access][remote_ip]"
|
||||
target => "[nginx][access][geoip]"
|
||||
}
|
||||
}
|
||||
output {
|
||||
elasticsearch {
|
||||
hosts => localhost
|
||||
manage_template => false
|
||||
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
|
||||
document_type => "%{[@metadata][type]}"
|
||||
}
|
||||
}
|
||||
29
docs/jp/static/filebeat_modules/nginx/error/pipeline.conf
Normal file
29
docs/jp/static/filebeat_modules/nginx/error/pipeline.conf
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
input {
|
||||
beats {
|
||||
# The port to listen on for filebeat connections.
|
||||
port => 5044
|
||||
# The IP address to listen for filebeat connections.
|
||||
host => "0.0.0.0"
|
||||
}
|
||||
}
|
||||
filter {
|
||||
grok {
|
||||
match => { "message" => ["%{DATA:[nginx][error][time]} \[%{DATA:[nginx][error][level]}\] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: (\*%{NUMBER:[nginx][error][connection_id]} )?%{GREEDYDATA:[nginx][error][message]}"] }
|
||||
remove_field => "message"
|
||||
}
|
||||
mutate {
|
||||
rename => { "@timestamp" => "read_timestamp" }
|
||||
}
|
||||
date {
|
||||
match => [ "[nginx][error][time]", "YYYY/MM/dd H:m:s" ]
|
||||
remove_field => "[nginx][error][time]"
|
||||
}
|
||||
}
|
||||
output {
|
||||
elasticsearch {
|
||||
hosts => localhost
|
||||
manage_template => false
|
||||
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
|
||||
document_type => "%{[@metadata][type]}"
|
||||
}
|
||||
}
|
||||
38
docs/jp/static/filebeat_modules/system/auth/pipeline.conf
Normal file
38
docs/jp/static/filebeat_modules/system/auth/pipeline.conf
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
input {
|
||||
beats {
|
||||
# The port to listen on for filebeat connections.
|
||||
port => 5044
|
||||
# The IP address to listen for filebeat connections.
|
||||
host => "0.0.0.0"
|
||||
}
|
||||
}
|
||||
filter {
|
||||
grok {
|
||||
match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
|
||||
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
|
||||
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
|
||||
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
|
||||
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
|
||||
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",
|
||||
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
|
||||
pattern_definitions => {
|
||||
"GREEDYMULTILINE"=> "(.|\n)*"
|
||||
}
|
||||
remove_field => "message"
|
||||
}
|
||||
date {
|
||||
match => [ "[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
|
||||
}
|
||||
geoip {
|
||||
source => "[system][auth][ssh][ip]"
|
||||
target => "[system][auth][ssh][geoip]"
|
||||
}
|
||||
}
|
||||
output {
|
||||
elasticsearch {
|
||||
hosts => localhost
|
||||
manage_template => false
|
||||
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
|
||||
document_type => "%{[@metadata][type]}"
|
||||
}
|
||||
}
|
||||
26
docs/jp/static/filebeat_modules/system/syslog/pipeline.conf
Normal file
26
docs/jp/static/filebeat_modules/system/syslog/pipeline.conf
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
input {
|
||||
beats {
|
||||
# The port to listen on for filebeat connections.
|
||||
port => 5044
|
||||
# The IP address to listen for filebeat connections.
|
||||
host => "0.0.0.0"
|
||||
}
|
||||
}
|
||||
filter {
|
||||
grok {
|
||||
match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
|
||||
pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
|
||||
remove_field => "message"
|
||||
}
|
||||
date {
|
||||
match => [ "[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
|
||||
}
|
||||
}
|
||||
output {
|
||||
elasticsearch {
|
||||
hosts => localhost
|
||||
manage_template => false
|
||||
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
|
||||
document_type => "%{[@metadata][type]}"
|
||||
}
|
||||
}
|
||||
14
docs/jp/static/filter.asciidoc
Normal file
14
docs/jp/static/filter.asciidoc
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
:register_method: true
|
||||
:filter_method: true
|
||||
:plugintype: filter
|
||||
:pluginclass: Filters
|
||||
:pluginname: example
|
||||
:pluginnamecap: Example
|
||||
|
||||
:blockcodec: true
|
||||
|
||||
:getstarted: Let's step through creating a {plugintype} plugin using the https://github.com/logstash-plugins/logstash-filter-example/[example {plugintype} plugin].
|
||||
|
||||
:methodheader: Logstash filters must implement the `register` and `filter` methods.
|
||||
|
||||
include::include/pluginbody.asciidoc[]
|
||||
189
docs/jp/static/getting-started-with-logstash.asciidoc
Normal file
189
docs/jp/static/getting-started-with-logstash.asciidoc
Normal file
|
|
@ -0,0 +1,189 @@
|
|||
[[getting-started-with-logstash]]
|
||||
== Logstashを始めてみよう
|
||||
|
||||
このセクションでは、Logstashのインストール手順と正常動作の確認について説明します。
|
||||
最初のイベントの格納方法を学習した後、Apache Webログをinputとして使用し、ログを解析して、解析したデータをElasticsearchクラスタに書き込む高度なパイプラインを作成します。次に、複数のinputプラグインとoutputプラグインをまとめて、多様な異なるソースからのデータを統合する方法について学習します。
|
||||
|
||||
このセクションには、次のトピックが含まれます。
|
||||
|
||||
* <<installing-logstash>>
|
||||
* <<first-event>>
|
||||
* {logstash}advanced-pipeline.html[Advanced pipeline]
|
||||
* {logstash}multiple-input-output-plugins[Multiple input output plugins]
|
||||
|
||||
[[installing-logstash]]
|
||||
=== Logstashのインストール
|
||||
|
||||
NOTE: LogstashにはJava 8が必要です。Java 9はサポートされていません。 http://www.oracle.com/technetwork/java/javase/downloads/index.html[official Oracle distribution]または http://openjdk.java.net/[OpenJDK]などのオープンソースのデストリビューションを使用します。
|
||||
|
||||
お使いのJavaのバージョンを確認するには、次のコマンドを実行します。
|
||||
|
||||
[source,shell]
|
||||
java -version
|
||||
|
||||
Javaがインストールされたシステムでは、このコマンドにより次のようなoutputが表示されます。
|
||||
|
||||
[source,shell]
|
||||
java version "1.8.0_65"
|
||||
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
|
||||
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
|
||||
|
||||
一部のLinuxシステムでは、特にtarファイルからJavaをインストールした場合、インストールを試行する前に、`JAVA_HOME`環境のエクスポートが必要になることがあります。これは、インストール中にLogstashがJavaを使用して環境を自動的に検知し、正しいスタートアップ方法（SysV initスクリプト、Upstart、またはsystemd）をインストールするためです。パッケージのインストール中に、LogstashがJAVA_HOME環境変数を検出できない場合、エラーメッセージが表示され、Logstashが適切に起動できなくなります。
|
||||
|
||||
[float]
|
||||
[[installing-binary]]
|
||||
=== ダウンロードしたバイナリのインストール
|
||||
|
||||
ホスト環境に合った https://www.elastic.co/downloads/logstash[Logstash installation file]をダウンロードします。
|
||||
ファイルを解凍します。Logstashはコロン（:）を含むディレクトリパスにインストールしないでください。
|
||||
|
||||
サポートされるLinuxオペレーティングシステムで、パッケージマネージャを使用してLogstashをインストールできます。
|
||||
|
||||
[float]
|
||||
[[package-repositories]]
|
||||
=== パッケージリポジトリからのインストール
|
||||
|
||||
APTおよびYUMベースのデストリビューション用にリポジトリも用意しています。パッケージはLogstashビルドの一部として作成されているため、バイナリパッケージだけが提供され、ソースパッケージは提供されないことに注意してください。
|
||||
|
||||
メジャーバージョンでの不注意によるアップグレードを避けるため、Logstashパッケージリポジトリは個別のURLにバージョンごとに分けられています。すべての{major-version}.yリリースに対して、{major-version}をバージョン番号として使用します。
|
||||
|
||||
次のフィンガープリントが付いたPGPキー https://pgp.mit.edu/pks/lookup?op=vindex&search=0xD27D666CD88E42B4[D88E42B4]（Elasticの署名キー）
|
||||
|
||||
4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4
|
||||
|
||||
を使用して、弊社のすべてのパッケージに署名しています。これは、 https://pgp.mit.edu から入手できます。
|
||||
|
||||
[float]
|
||||
==== APT
|
||||
|
||||
ifeval::["{release-state}"=="unreleased"]
|
||||
|
||||
バージョン{logstash_version}のLogstashはまだリリースされていません。
|
||||
|
||||
endif::[]
|
||||
|
||||
ifeval::["{release-state}"!="unreleased"]
|
||||
|
||||
次のように、公開署名キーをダウンロードしてインストールします。
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------------
|
||||
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
|
||||
--------------------------------------------------
|
||||
|
||||
先に進む前に、Debianに`apt-transport-https`パッケージをインストールする必要がある場合があります。
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------------
|
||||
sudo apt-get install apt-transport-https
|
||||
--------------------------------------------------
|
||||
|
||||
リポジトリ定義を+/etc/apt/sources.list.d/elastic-{major-version}.list+に保存します。
|
||||
|
||||
["source","sh",subs="attributes,callouts"]
|
||||
--------------------------------------------------
|
||||
echo "deb https://artifacts.elastic.co/packages/{major-version}/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-{major-version}.list
|
||||
--------------------------------------------------
|
||||
|
||||
[WARNING]
|
||||
==================================================
|
||||
上記の`echo`メソッドを使用して、Logstashリポジトリを追加します。`deb-src`エントリも追加されますが、ソースパッケージを提供していないため、`add-apt-repository`を使用しないでください。`deb-src`エントリを追加していない場合、次のようなエラーが表示されます。
|
||||
|
||||
Unable to find expected entry 'main/source/Sources' in Release file (Wrong sources.list entry or malformed file)
|
||||
|
||||
`/etc/apt/sources.list`ファイルから`deb-src`エントリを削除すると、インストールは正しく動作します。
|
||||
==================================================
|
||||
|
||||
`sudo apt-get update`を実行します。リポジトリが使用できるようになります。次のコマンドで、インストールできます。
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------------
|
||||
sudo apt-get update && sudo apt-get install logstash
|
||||
--------------------------------------------------
|
||||
|
||||
Logstashをシステムサービスとして管理する方法の詳細については、{logstash}running-logstash.html[Logstashの実行]を参照してください。
|
||||
|
||||
endif::[]
|
||||
|
||||
[float]
|
||||
==== YUM
|
||||
|
||||
ifeval::["{release-state}"=="unreleased"]
|
||||
|
||||
バージョン{logstash_version}のLogstashはまだリリースされていません。
|
||||
|
||||
endif::[]
|
||||
|
||||
ifeval::["{release-state}"!="unreleased"]
|
||||
|
||||
次のように、公開署名キーをダウンロードしてインストールします。
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------------
|
||||
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
|
||||
--------------------------------------------------
|
||||
|
||||
`/etc/yum.repos.d/`ディレクトリ内で`.repo`サフィックスが付いたファイル（例: `logstash.repo`）に次を追加します。
|
||||
|
||||
["source","sh",subs="attributes,callouts"]
|
||||
--------------------------------------------------
|
||||
[logstash-{major-version}]
|
||||
name=Elastic repository for {major-version} packages
|
||||
baseurl=https://artifacts.elastic.co/packages/{major-version}/yum
|
||||
gpgcheck=1
|
||||
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
|
||||
enabled=1
|
||||
autorefresh=1
|
||||
type=rpm-md
|
||||
--------------------------------------------------
|
||||
|
||||
リポジトリが使用できるようになります。次のコマンドで、インストールできます。
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------------
|
||||
sudo yum install logstash
|
||||
--------------------------------------------------
|
||||
|
||||
WARNING: CentOS 5のようにまだRPM v3を使用している古いRPMベースのデストリビューションでは、リポジトリは機能しません。
|
||||
|
||||
Logstashをシステムサービスとして管理するには、{logstash}running-logstash.html[Logstashの実行]を参照してください。
|
||||
|
||||
endif::[]
|
||||
|
||||
==== Docker
|
||||
|
||||
DockerコンテナとしてLogstashを実行するためのイメージが用意されています。Elastic Dockerレジストリから利用できます。Logstash Dockerコンテナの設定方法と実行方法の詳細については、{logstash}docker.html[Docker]を参照してください。
|
||||
|
||||
[[first-event]]
|
||||
=== 最初のイベントの格納
|
||||
|
||||
最初に、最も基本的な_Logstash pipeline_を実行してLogstashインストールをテストします。
|
||||
|
||||
Logstash pipelineには、`input`と`output`という2つの必須要素と、`filter`という1つのオプション要素があります。inputプラグインはソースからデータを取得し、filterプラグインは指定どおりにデータを変更し、outputプラグインはデータを出力先に書き込みます。
|
||||
|
||||
//TODO: REPLACE WITH NEW IMAGE
|
||||
|
||||
image::static/images/basic_logstash_pipeline.png[]
|
||||
|
||||
Logstashインストールをテストするには、最も基本的なLogstash pipelineを実行します。例:
|
||||
|
||||
["source","sh",subs="attributes"]
|
||||
--------------------------------------------------
|
||||
cd logstash-{logstash_version}
|
||||
bin/logstash -e 'input { stdin { } } output { stdout {} }'
|
||||
--------------------------------------------------
|
||||
|
||||
NOTE: `bin`ディレクトリのロケーションはプラットフォームにより異なります。お使いのシステムで`bin\logstash`の場所を見つけるには、{logstash}dir-layout.html[dir layout]を参照してください。
|
||||
|
||||
`-e`フラグを使用すると、コマンドラインから設定ディレクトリを指定できます。コマンドラインで設定を指定すると、繰り返しの間にファイルを編集しなくても、設定をすばやくテストできます。
|
||||
例のパイプラインは、標準input`stdin`のinputを使用し、そのinputを標準output`stdout`に構造化された形式で移します。
|
||||
|
||||
Logstashの起動後、「Pipeline main started」と表示されるまで待って、コマンドプロンプトに`hello world`と入力します。
|
||||
|
||||
[source,shell]
|
||||
hello world
|
||||
2013-11-21T01:22:14.405+0000 0.0.0.0 hello world
|
||||
|
||||
Logstashは、メッセージにタイムスタンプとIPアドレス情報を追加します。Logstashの稼働中にシェルで*CTRL-D*コマンドを発行して、Logstashを終了します。
|
||||
|
||||
おめでとうございます! 基本的なLogstash pipelineを作成し、実行しました。次は、より現実的なパイプラインの作成方法について学習します。
|
||||
50
docs/jp/static/glob-support.asciidoc
Normal file
50
docs/jp/static/glob-support.asciidoc
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
[[glob-support]]
|
||||
=== Glob Pattern Support
|
||||
|
||||
Logstash supports the following patterns wherever glob patterns are allowed:
|
||||
|
||||
*`*`*::
|
||||
Match any file. You can also use an `*` to restrict other values in the glob.
|
||||
For example, `*conf` matches all files that end in `conf`. `*apache*` matches
|
||||
any files with `apache` in the name. This pattern does not match hidden files
|
||||
(dot files) on Unix-like operating systems. To match dot files, use a pattern
|
||||
like `{*,.*}`.
|
||||
|
||||
*`**`*::
|
||||
Match directories recursively.
|
||||
|
||||
*`?`*::
|
||||
Match any one character.
|
||||
|
||||
*`[set]`*::
|
||||
Match any one character in a set. For example, `[a-z]`. Also supports set negation
|
||||
(`[^a-z]`).
|
||||
|
||||
*`{p,q}`*::
|
||||
Match either literal `p` or literal `q`. The matching literal can be more than one
|
||||
character, and you can specify more than two literals. This pattern is the equivalent
|
||||
to using alternation with the vertical bar in regular expressions (`foo|bar`).
|
||||
|
||||
*`\`*::
|
||||
Escape the next metacharacter. This means that you cannot use a backslash in Windows
|
||||
as part of a glob. The pattern `c:\foo*` will not work, so use `foo*` instead.
|
||||
|
||||
[float]
|
||||
[[example-glob-patterns]]
|
||||
==== Example Patterns
|
||||
|
||||
Here are some common examples of glob patterns:
|
||||
|
||||
`"/path/to/*.conf"`::
|
||||
Matches config files ending in `.conf` in the specified path.
|
||||
|
||||
`"/var/log/*.log"`::
|
||||
Matches log files ending in `.log` in the specified path.
|
||||
|
||||
`"/var/log/**/*.log`::
|
||||
Matches log files ending in `.log` in subdirectories under the specified path.
|
||||
|
||||
`"/path/to/logs/{app1,app2,app3}/data.log"`::
|
||||
Matches app log files in the `app1`, `app2`, and `app3` subdirectories under the
|
||||
specified path.
|
||||
|
||||
79
docs/jp/static/glossary.asciidoc
Normal file
79
docs/jp/static/glossary.asciidoc
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
[[glossary]]
|
||||
== Glossary of Terms
|
||||
|
||||
[[glossary-metadata]]@metadata ::
|
||||
A special field for storing content that you don't want to include in output <<glossary-event,events>>. For example, the `@metadata`
|
||||
field is useful for creating transient fields for use in <<glossary-conditional,conditional>> statements.
|
||||
|
||||
[[glossary-codec-plugin]]codec plugin::
|
||||
A Logstash <<glossary-plugin,plugin>> that changes the data representation of an <<glossary-event,event>>. Codecs are essentially stream filters that can operate as part of an input or output. Codecs enable you to separate the transport of messages from the serialization process. Popular codecs include json, msgpack, and plain (text).
|
||||
|
||||
[[glossary-conditional]]conditional::
|
||||
A control flow that executes certain actions based on whether a statement (also called a condition) is true or false. Logstash supports `if`, `else if`, and `else` statements. You can use conditional statements to apply filters and send events to a specific output based on conditions that you specify.
|
||||
|
||||
[[glossary-event]]event::
|
||||
A single unit of information, containing a timestamp plus additional data. An event arrives via an input, and is subsequently parsed, timestamped, and passed through the Logstash <<glossary-pipeline,pipeline>>.
|
||||
|
||||
[[glossary-field]]field::
|
||||
An <<glossary-event,event>> property. For example, each event in an apache access log has properties, such as a status
|
||||
code (200, 404), request path ("/", "index.html"), HTTP verb (GET, POST), client IP address, and so on. Logstash uses
|
||||
the term "fields" to refer to these properties.
|
||||
|
||||
[[glossary-field-reference]]field reference::
|
||||
A reference to an event <<glossary-field,field>>. This reference may appear in an output block or filter block in the
|
||||
Logstash config file. Field references are typically wrapped in square (`[]`) brackets, for example `[fieldname]`. If
|
||||
you are referring to a top-level field, you can omit the `[]` and simply use the field name. To refer to a nested
|
||||
field, you specify the full path to that field: `[top-level field][nested field]`.
|
||||
|
||||
[[glossary-filter-plugin]]filter plugin::
|
||||
A Logstash <<glossary-plugin,plugin>> that performs intermediary processing on an <<glossary-event,event>>. Typically, filters act upon
|
||||
event data after it has been ingested via inputs, by mutating, enriching, and/or modifying the data according to
|
||||
configuration rules. Filters are often applied conditionally depending on the characteristics of the event. Popular
|
||||
filter plugins include grok, mutate, drop, clone, and geoip. Filter stages are optional.
|
||||
|
||||
[[glossary-gem]]gem::
|
||||
A self-contained package of code that's hosted on https://rubygems.org[RubyGems.org]. Logstash <<glossary-plugin,plugins>> are packaged as
|
||||
Ruby Gems. You can use the Logstash <<glossary-plugin-manager,plugin manager>> to manage Logstash gems.
|
||||
|
||||
[[glossary-hot-thread]]hot thread::
|
||||
A Java thread that has high CPU usage and executes for a longer than normal period of time.
|
||||
|
||||
[[glossary-input-plugin]]input plugin::
|
||||
A Logstash <<glossary-plugin,plugin>> that reads <<glossary-event,event>> data from a specific source. Input plugins are the first stage in the Logstash event processing <<glossary-pipeline,pipeline>>. Popular input plugins include file, syslog, redis, and beats.
|
||||
|
||||
[[glossary-indexer]]indexer::
|
||||
A Logstash instance that is tasked with interfacing with an Elasticsearch cluster in order to index <<glossary-event,event>> data.
|
||||
|
||||
[[glossary-message-broker]]message broker::
|
||||
Also referred to as a _message buffer_ or _message queue_, a message broker is external software (such as Redis, Kafka, or RabbitMQ) that stores messages from the Logstash shipper instance as an intermediate store, waiting to be processed by the Logstash indexer instance.
|
||||
|
||||
[[glossary-output-plugin]]output plugin::
|
||||
A Logstash <<glossary-plugin,plugin>> that writes <<glossary-event,event>> data to a specific destination. Outputs are the final stage in
|
||||
the event <<glossary-pipeline,pipeline>>. Popular output plugins include elasticsearch, file, graphite, and
|
||||
statsd.
|
||||
|
||||
[[glossary-pipeline]]pipeline::
|
||||
A term used to describe the flow of <<glossary-event,events>> through the Logstash workflow. A pipeline typically consists of a series of
|
||||
input, filter, and output stages. <<glossary-input-plugin,Input>> stages get data from a source and generate events,
|
||||
<<glossary-filter-plugin,filter>> stages, which are optional, modify the event data, and
|
||||
<<glossary-output-plugin,output>> stages write the data to a destination. Inputs and outputs support <<glossary-codec-plugin,codecs>> that enable you to encode or decode the data as it enters or exits the pipeline without having to use
|
||||
a separate filter.
|
||||
|
||||
[[glossary-plugin]]plugin::
|
||||
A self-contained software package that implements one of the stages in the Logstash event processing
|
||||
<<glossary-pipeline,pipeline>>. The list of available plugins includes <<glossary-input-plugin,input plugins>>,
|
||||
<<glossary-output-plugin,output plugins>>, <<glossary-codec-plugin,codec plugins>>, and
|
||||
<<glossary-filter-plugin,filter plugins>>. The plugins are implemented as Ruby <<glossary-gem,gems>> and hosted on
|
||||
https://rubygems.org[RubyGems.org]. You define the stages of an event processing <<glossary-pipeline,pipeline>> by configuring plugins.
|
||||
|
||||
[[glossary-plugin-manager]]plugin manager::
|
||||
Accessed via the `bin/logstash-plugin` script, the plugin manager enables you to manage the lifecycle of
|
||||
<<glossary-plugin,plugins>> in your Logstash deployment. You can install, remove, and upgrade plugins by using the
|
||||
plugin manager Command Line Interface (CLI).
|
||||
|
||||
[[shipper]]shipper::
|
||||
An instance of Logstash that send events to another instance of Logstash, or some other application.
|
||||
|
||||
[[worker]]worker::
|
||||
The filter thread model used by Logstash, where each worker receives an <<glossary-event,event>> and applies all filters, in order, before emitting the event to the output queue. This allows scalability across CPUs because many filters are CPU intensive.
|
||||
|
||||
BIN
docs/jp/static/images/basic_logstash_pipeline.png
Normal file
BIN
docs/jp/static/images/basic_logstash_pipeline.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 39 KiB |
BIN
docs/jp/static/images/deploy_1.png
Normal file
BIN
docs/jp/static/images/deploy_1.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
BIN
docs/jp/static/images/deploy_2.png
Normal file
BIN
docs/jp/static/images/deploy_2.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 45 KiB |
BIN
docs/jp/static/images/deploy_3.png
Normal file
BIN
docs/jp/static/images/deploy_3.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 41 KiB |
BIN
docs/jp/static/images/deploy_4.png
Normal file
BIN
docs/jp/static/images/deploy_4.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 56 KiB |
BIN
docs/jp/static/images/deploy_5.png
Normal file
BIN
docs/jp/static/images/deploy_5.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 105 KiB |
BIN
docs/jp/static/images/deploy_6.png
Normal file
BIN
docs/jp/static/images/deploy_6.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 164 KiB |
BIN
docs/jp/static/images/deploy_7.png
Normal file
BIN
docs/jp/static/images/deploy_7.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 172 KiB |
BIN
docs/jp/static/images/kibana-filebeat-data.png
Normal file
BIN
docs/jp/static/images/kibana-filebeat-data.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 179 KiB |
BIN
docs/jp/static/images/logstash.png
Normal file
BIN
docs/jp/static/images/logstash.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 51 KiB |
BIN
docs/jp/static/images/pipeline_correct_load.png
Normal file
BIN
docs/jp/static/images/pipeline_correct_load.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 118 KiB |
BIN
docs/jp/static/images/pipeline_overload.png
Normal file
BIN
docs/jp/static/images/pipeline_overload.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 113 KiB |
1302
docs/jp/static/include/pluginbody.asciidoc
Normal file
1302
docs/jp/static/include/pluginbody.asciidoc
Normal file
File diff suppressed because it is too large
Load diff
12
docs/jp/static/input.asciidoc
Normal file
12
docs/jp/static/input.asciidoc
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
:register_method: true
|
||||
:run_method: true
|
||||
:plugintype: input
|
||||
:pluginclass: Inputs
|
||||
:pluginname: example
|
||||
:pluginnamecap: Example
|
||||
|
||||
:getstarted: Let's step through creating an {plugintype} plugin using the https://github.com/logstash-plugins/logstash-input-example/[example {plugintype} plugin].
|
||||
|
||||
:methodheader: Logstash inputs must implement two main methods: `register` and `run`.
|
||||
|
||||
include::include/pluginbody.asciidoc[]
|
||||
116
docs/jp/static/introduction.asciidoc
Normal file
116
docs/jp/static/introduction.asciidoc
Normal file
|
|
@ -0,0 +1,116 @@
|
|||
[float]
|
||||
[[power-of-logstash]]
|
||||
== Logstashの威力
|
||||
|
||||
*Elasticsearchの収集エンジン以上*
|
||||
|
||||
強力なElasticsearchとKibanaのシナジーを備えた水平にスケーラブルなデータ処理パイプライン
|
||||
|
||||
*プラグ可能なパイプラインアーキテクチャ*
|
||||
|
||||
さまざまなinput、filter、およびoutputを組み合わせ、調整して、パイプラインを調和
|
||||
|
||||
*コミュニティによる拡張が可能で開発者が使いやすいプラグインエコシステム*
|
||||
|
||||
200を超えるプラグインが利用可能であるだけでなく、独自のプラグインを柔軟に作成、共有することが可能
|
||||
|
||||
image:static/images/logstash.png[]
|
||||
|
||||
[float]
|
||||
== Logstashはデータ重視
|
||||
|
||||
データを収集するほど、より多くのことがわかります。Logstashは、あらゆる形式やサイズのデータを扱えます。
|
||||
|
||||
[float]
|
||||
=== ログとメトリクス
|
||||
|
||||
すべてここから始まります。
|
||||
|
||||
* あらゆる種類のログデータを処理
|
||||
** 多数の{logstash}advanced-pipeline.html[Apache]などのWebログやJavaの{logstash}plugins-inputs-log4j.html[log4j]などのアプリケーションログを容易に収集
|
||||
** {logstash}plugins-inputs-syslog.html[syslog]、{logstash}plugins-inputs-eventlog.html[Windows event logs]、ネットワークやファイアウォールのログなど、多様な形式のログをキャプチャ
|
||||
* https://www.elastic.co/products/beats/filebeat[Filebeat]との安全で補足的なログ転送を利用
|
||||
* {logstash}plugins-inputs-ganglia.html[Ganglia]、{logstash}plugins-codecs-collectd.html[collectd]、{logstash}plugins-codecs-netflow.html[NetFlow]、{logstash}plugins-inputs-jmx.html[JMX]を始めとする数多くのインフラストラクチャとアプリケーションプラットフォームのメトリクスを{logstash}plugins-inputs-tcp.html[TCP]と{logstash}plugins-inputs-udp.html[UDP]を通して収集
|
||||
|
||||
[float]
|
||||
=== Web
|
||||
|
||||
World Wide Webを解放します。
|
||||
|
||||
* {logstash}plugins-inputs-http.html[HTTPリクエスト]をイベントに変換
|
||||
** ソーシャルセンチメント分析に{logstash}plugins-inputs-twitter.html[Twitter]などのWebサービスFirehoseから習得
|
||||
** GitHub、HipChat、JIRAを始めとする無数のアプリケーションのWebhookサポート
|
||||
** 数多くの https://www.elastic.co/products/x-pack/alerting[Watcher]アラート事例
|
||||
* オンデマンド方式の{logstash}plugins-inputs-http_poller.html[HTTPエンドポイント]のポーリングによりイベントを作成
|
||||
** Webアプリケーションインターフェイスからヘルス、パフォーマンス、メトリクスといったさまざまな種類のデータを広く一般にキャプチャ
|
||||
** 受信よりもポーリングの制御が優先される状況に最適
|
||||
[float]
|
||||
|
||||
=== データの格納とストリーム
|
||||
すでに所有しているデータからさらなる価値を見つけます。
|
||||
|
||||
* {logstash}plugins-inputs-jdbc.html[JDBC]インターフェイスを使用して、リレーショナルデータベースまたはNoSQLストアからのデータの理解を深める
|
||||
* Apache {logstash}plugins-outputs-kafka.html[Kafka]、{logstash}plugins-outputs-rabbitmq.html[RabbitMQ]、{logstash}plugins-outputs-sqs.html[Amazon SQS]、{logstash}plugins-outputs-zeromq.html[ZeroMQ]のようなメッセージングキューからのさまざまなデータストリームを統合
|
||||
|
||||
[float]
|
||||
=== センサとIoT
|
||||
|
||||
その他の広大なデータを探索します。
|
||||
|
||||
* この技術の進歩した時代に、大規模なIoT世界は、接続したセンサーからのデータの利用を通して、無数の使用事例を解放しています。
|
||||
* Logstashは、モバイル機器からインテリジェントホーム、接続された車、ヘルスケアセンサなど数多くの業界固有アプリケーションに送信されたデータを取得する、イベント収集の共通バックボーンです。
|
||||
|
||||
[float]
|
||||
== 簡単にすべてを整形
|
||||
|
||||
データが良ければ良いほど、知識は深まります。取得時にデータをきれいにして変換し、インデックス作成時または出力時にほぼリアルタイムの情報をすぐに取得します。Logstashは、パターン照合、ジオマッピング、動的なルックアップ機能とともに、数多くの集約や変形ですぐに使用できます。
|
||||
|
||||
* {logstash}plugins-filters-grok.html[Grok]は、主要なLogstashフィルタで、非構造データから構造を展開するために至る所で使用されます。Web、システム、ネットワークを始めとする各種のイベント形式をすばやく分析することを目的とした数多くの統合パターンを利用してください。
|
||||
* IPアドレスからの{logstash}plugins-filters-geoip.html[位置座標]の解読、複雑な{logstash}plugins-filters-date.html[日付]の標準化、{logstash}plugins-filters-kv.html[キーと値のペア]および{logstash}plugins-filters-csv.html[CSV]データの単純化、機密情報の{logstash}plugins-filters-anonymize.html[匿名化]、および{logstash}plugins-filters-translate.html[ローカルルックアップ]やElasticsearchの{logstash}plugins-filters-elasticsearch.html[クエリ]でのデータの整形により視野が広がります。
|
||||
* {logstash}plugins-codecs-json.html[JSON]イベントや{logstash}plugins-codecs-multiline.html[multiline]イベントのような共通イベント構造の処理を容易にするために、コーディックがよく使用されます。
|
||||
|
||||
[float]
|
||||
== 格納庫の選択
|
||||
|
||||
最も重要なデータを転送します。データの格納、分析、および措置により、さまざまなダウンストリームの分析事例と運用事例を明らかにします。
|
||||
|
||||
[cols="a,a"]
|
||||
|=======================================================================
|
||||
|
|
||||
|
||||
*分析*
|
||||
|
||||
* {logstash}plugins-outputs-elasticsearch.html[Elasticsearch]
|
||||
* {logstash}plugins-outputs-mongodb.html[MongoDB]や{logstash}plugins-outputs-riak.html[Riak]などのデータストア
|
||||
|
||||
|
|
||||
|
||||
*アーカイブ*
|
||||
|
||||
* {logstash}plugins-outputs-webhdfs.html[HDFS]
|
||||
* {logstash}plugins-outputs-s3.html[S3]
|
||||
* {logstash}plugins-outputs-google_cloud_storage.html[Google Cloud Storage]
|
||||
|
||||
|
|
||||
|
||||
*監視*
|
||||
|
||||
* {logstash}plugins-outputs-nagios,html[Nagios]
|
||||
* {logstash}plugins-outputs-ganglia.html[Ganglia]
|
||||
* {logstash}plugins-outputs-zabbix.html[Zabbix]
|
||||
* {logstash}plugins-outputs-graphite.html[Graphite]
|
||||
* {logstash}plugins-outputs-datadog.html[Datadog]
|
||||
* {logstash}plugins-outputs-cloudwatch.html[CloudWatch]
|
||||
|
||||
|
|
||||
|
||||
*アラート*
|
||||
|
||||
* Elasticsearchでの https://www.elastic.co/products/watcher[Watcher]
|
||||
* {logstash}plugins-outputs-email.html[Email]
|
||||
* {logstash}plugins-outputs-pagerduty.html[Pagerduty]
|
||||
* {logstash}plugins-outputs-hipchat.html[HipChat]
|
||||
* {logstash}plugins-outputs-irc.html[IRC]
|
||||
* {logstash}plugins-outputs-sns.html[SNS]
|
||||
|
||||
|=======================================================================
|
||||
94
docs/jp/static/life-of-an-event.asciidoc
Normal file
94
docs/jp/static/life-of-an-event.asciidoc
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
[[pipeline]]
|
||||
== How Logstash Works
|
||||
|
||||
The Logstash event processing pipeline has three stages: inputs -> filters ->
|
||||
outputs. Inputs generate events, filters modify them, and outputs ship them
|
||||
elsewhere. Inputs and outputs support codecs that enable you to encode or decode
|
||||
the data as it enters or exits the pipeline without having to use a separate
|
||||
filter.
|
||||
|
||||
[float]
|
||||
=== Inputs
|
||||
You use inputs to get data into Logstash. Some of the more commonly-used inputs
|
||||
are:
|
||||
|
||||
* *file*: reads from a file on the filesystem, much like the UNIX command
|
||||
`tail -0F`
|
||||
* *syslog*: listens on the well-known port 514 for syslog messages and parses
|
||||
according to the RFC3164 format
|
||||
* *redis*: reads from a redis server, using both redis channels and redis lists.
|
||||
Redis is often used as a "broker" in a centralized Logstash installation, which
|
||||
queues Logstash events from remote Logstash "shippers".
|
||||
* *beats*: processes events sent by https://www.elastic.co/downloads/beats/filebeat[Filebeat].
|
||||
|
||||
For more information about the available inputs, see
|
||||
<<input-plugins,Input Plugins>>.
|
||||
|
||||
[float]
|
||||
=== Filters
|
||||
Filters are intermediary processing devices in the Logstash pipeline. You can
|
||||
combine filters with conditionals to perform an action on an event if it meets
|
||||
certain criteria. Some useful filters include:
|
||||
|
||||
* *grok*: parse and structure arbitrary text. Grok is currently the best way in
|
||||
Logstash to parse unstructured log data into something structured and queryable.
|
||||
With 120 patterns built-in to Logstash, it's more than likely you'll find one
|
||||
that meets your needs!
|
||||
* *mutate*: perform general transformations on event fields. You can rename,
|
||||
remove, replace, and modify fields in your events.
|
||||
* *drop*: drop an event completely, for example, 'debug' events.
|
||||
* *clone*: make a copy of an event, possibly adding or removing fields.
|
||||
* *geoip*: add information about geographical location of IP addresses (also
|
||||
displays amazing charts in Kibana!)
|
||||
|
||||
For more information about the available filters, see
|
||||
<<filter-plugins,Filter Plugins>>.
|
||||
|
||||
[float]
|
||||
=== Outputs
|
||||
Outputs are the final phase of the Logstash pipeline. An event can pass through
|
||||
multiple outputs, but once all output processing is complete, the event has
|
||||
finished its execution. Some commonly used outputs include:
|
||||
|
||||
* *elasticsearch*: send event data to Elasticsearch. If you're planning to save
|
||||
your data in an efficient, convenient, and easily queryable format...
|
||||
Elasticsearch is the way to go. Period. Yes, we're biased :)
|
||||
* *file*: write event data to a file on disk.
|
||||
* *graphite*: send event data to graphite, a popular open source tool for
|
||||
storing and graphing metrics. http://graphite.readthedocs.io/en/latest/
|
||||
* *statsd*: send event data to statsd, a service that "listens for statistics,
|
||||
like counters and timers, sent over UDP and sends aggregates to one or more
|
||||
pluggable backend services". If you're already using statsd, this could be
|
||||
useful for you!
|
||||
|
||||
For more information about the available outputs, see
|
||||
<<output-plugins,Output Plugins>>.
|
||||
|
||||
[float]
|
||||
=== Codecs
|
||||
Codecs are basically stream filters that can operate as part of an input or
|
||||
output. Codecs enable you to easily separate the transport of your messages from
|
||||
the serialization process. Popular codecs include `json`, `msgpack`, and `plain`
|
||||
(text).
|
||||
|
||||
* *json*: encode or decode data in the JSON format.
|
||||
* *multiline*: merge multiple-line text events such as java exception and
|
||||
stacktrace messages into a single event.
|
||||
|
||||
For more information about the available codecs, see
|
||||
<<codec-plugins,Codec Plugins>>.
|
||||
|
||||
[[execution-model]]
|
||||
=== Execution Model
|
||||
|
||||
The Logstash event processing pipeline coordinates the execution of inputs,
|
||||
filters, and outputs.
|
||||
|
||||
Each input stage in the Logstash pipeline runs in its own thread. Inputs write events to a common Java https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/SynchronousQueue.html[SynchronousQueue]. This queue holds no events, instead transferring each pushed event to a free worker, blocking if all workers are busy. Each pipeline worker thread takes a batch of events off this queue, creating a buffer per worker, runs the batch of events through the configured filters, then runs the filtered events through any outputs. The size of the batch and number of pipeline worker threads are configurable (see <<tuning-logstash>>).
|
||||
|
||||
By default, Logstash uses in-memory bounded queues between pipeline stages
|
||||
(input → filter and filter → output) to buffer events. If Logstash terminates
|
||||
unsafely, any events that are stored in memory will be lost. To prevent data
|
||||
loss, you can enable Logstash to persist in-flight events to disk. See
|
||||
<<persistent-queues>> for more information.
|
||||
|
||||
101
docs/jp/static/logging.asciidoc
Normal file
101
docs/jp/static/logging.asciidoc
Normal file
|
|
@ -0,0 +1,101 @@
|
|||
[[logging]]
|
||||
=== Logging
|
||||
|
||||
Logstash emits internal logs during its operation, which are placed in `LS_HOME/logs` (or `/var/log/logstash` for
|
||||
DEB/RPM). The default logging level is `INFO`. Logstash's logging framework is based on
|
||||
http://logging.apache.org/log4j/2.x/[Log4j 2 framework], and much of its functionality is exposed directly to users.
|
||||
|
||||
When debugging problems, particularly problems with plugins, it can be helpful to increase the logging level to `DEBUG`
|
||||
to emit more verbose messages. Previously, you could only set a log level that applied to the entire Logstash product.
|
||||
Starting with 5.0, you can configure logging for a particular subsystem in Logstash. For example, if you are
|
||||
debugging issues with Elasticsearch Output, you can increase log levels just for that component. This way
|
||||
you can reduce noise due to excessive logging and focus on the problem area effectively.
|
||||
|
||||
==== Log file location
|
||||
|
||||
You can specify the log file location using `--path.logs` setting.
|
||||
|
||||
==== Log4j 2 Configuration
|
||||
|
||||
Logstash ships with a `log4j2.properties` file with out-of-the-box settings. You can modify this file directly to change the
|
||||
rotation policy, type, and other https://logging.apache.org/log4j/2.x/manual/configuration.html#Loggers[log4j2 configuration].
|
||||
You must restart Lostash to apply any changes that you make to this file.
|
||||
|
||||
==== Slowlog
|
||||
|
||||
Slow-log for Logstash adds the ability to log when a specific event takes an abnormal amount of time to make its way
|
||||
through the pipeline. Just like the normal application log, you can find slow-logs in your `--path.logs` directory.
|
||||
Slowlog is configured in the `logstash.yml` settings file with the following options:
|
||||
|
||||
[source,yaml]
|
||||
------------------------------
|
||||
slowlog.threshold.warn (default: -1)
|
||||
slowlog.threshold.info (default: -1)
|
||||
slowlog.threshold.debug (default: -1)
|
||||
slowlog.threshold.trace (default: -1)
|
||||
------------------------------
|
||||
|
||||
By default, these values are set to `-1nanos` to represent an infinite threshold where no slowlog will be invoked. These `slowlog.threshold`
|
||||
fields are configured using a time-value format which enables a wide range of trigger intervals. The positive numeric ranges
|
||||
can be specified using the following time units: `nanos` (nanoseconds), `micros` (microseconds), `ms` (milliseconds), `s` (second), `m` (minute),
|
||||
`h` (hour), `d` (day).
|
||||
|
||||
Here is an example:
|
||||
|
||||
[source,yaml]
|
||||
------------------------------
|
||||
slowlog.threshold.warn: 2s
|
||||
slowlog.threshold.info: 1s
|
||||
slowlog.threshold.debug: 500ms
|
||||
slowlog.threshold.trace: 100ms
|
||||
------------------------------
|
||||
|
||||
In the above configuration, events that take longer than two seconds to be processed within a filter will be logged.
|
||||
The logs will include the full event and filter configuration that are responsible for the slowness.
|
||||
|
||||
==== Logging APIs
|
||||
|
||||
You could modify the `log4j2.properties` file and restart your Logstash, but that is both tedious and leads to unnecessary
|
||||
downtime. Instead, you can dynamically update logging levels through the logging API. These settings are effective
|
||||
immediately and do not need a restart. To update logging levels, take the subsystem/module you are interested in and prepend
|
||||
`logger.` to it. For example:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
PUT /_node/logging
|
||||
{
|
||||
"logger.logstash.outputs.elasticsearch" : "DEBUG"
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
While this setting is in effect, Logstash will begin to emit DEBUG-level logs for __all__ the Elasticsearch outputs
|
||||
specified in your configuration. Please note this new setting is transient and will not survive a restart.
|
||||
|
||||
To retrieve a list of logging subsystems available at runtime, you can do a `GET` request to `_node/logging`
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET /_node/logging?pretty
|
||||
--------------------------------------------------
|
||||
|
||||
Example response:
|
||||
|
||||
["source","js"]
|
||||
--------------------------------------------------
|
||||
{
|
||||
...
|
||||
"loggers" : {
|
||||
"logstash.registry" : "WARN",
|
||||
"logstash.instrument.periodicpoller.os" : "WARN",
|
||||
"logstash.instrument.collector" : "WARN",
|
||||
"logstash.runner" : "WARN",
|
||||
"logstash.inputs.stdin" : "WARN",
|
||||
"logstash.outputs.stdout" : "WARN",
|
||||
"logstash.agent" : "WARN",
|
||||
"logstash.api.service" : "WARN",
|
||||
"logstash.instrument.periodicpoller.jvm" : "WARN",
|
||||
"logstash.pipeline" : "WARN",
|
||||
"logstash.codecs.line" : "WARN"
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
132
docs/jp/static/logstash-glossary.asciidoc
Normal file
132
docs/jp/static/logstash-glossary.asciidoc
Normal file
|
|
@ -0,0 +1,132 @@
|
|||
== Glossary
|
||||
Logstash Glossary
|
||||
|
||||
apache ::
|
||||
A very common open source web server application, which produces logs easily consumed by Logstash (Apache Common/Combined Log Format).
|
||||
|
||||
agent ::
|
||||
An invocation of Logstash with a particular configuration, allowing it to operate as a "shipper", a "collector", or a combination of functionalities.
|
||||
|
||||
|
||||
broker ::
|
||||
An intermediary used in a multi-tiered Logstash deployment which allows a queueing mechanism to be used. Examples of brokers are Redis, RabbitMQ, and Apache Kafka. This pattern is a common method of building fault-tolerance into a Logstash architecture.
|
||||
|
||||
buffer::
|
||||
Within Logstash, a temporary storage area where events can queue up, waiting to be processed. The default queue size is 20 events, but it is not recommended to increase this, as Logstash is not designed to operate as a queueing mechanism.
|
||||
|
||||
centralized::
|
||||
A configuration of Logstash in which the Logstash agent, input and output sources live on multiple machines, and the pipeline passes through these tiers.
|
||||
|
||||
codec::
|
||||
A Logstash plugin which works within an input or output plugin, and usually aims to serialize or deserialize data flowing through the Logstash pipeline. A common example is the JSON codec, which allows Logstash inputs to receive data which arrives in JSON format, or output event data in JSON format.
|
||||
|
||||
collector::
|
||||
An instance of Logstash which receives external events from another instance of Logstash, or perhaps some other client, either remote or local.
|
||||
|
||||
conditional::
|
||||
In a computer programming context, a control flow which executes certain actions based on true/false values of a statement (called the condition). Often expressed in the form of "if ... then ... (elseif ...) else". Logstash has built-in conditionals to allow users control of the plugin pipeline.
|
||||
|
||||
elasticsearch::
|
||||
An open-source, Lucene-based, RESTful search and analytics engine written in Java, with supported clients in various languages such as Perl, Python, Ruby, Java, etc.
|
||||
|
||||
event::
|
||||
In Logstash parlance, a single unit of information, containing a timestamp plus additional data. An event arrives via an input, and is subsequently parsed, timestamped, and passed through the Logstash pipeline.
|
||||
|
||||
field::
|
||||
A data point (often a key-value pair) within a full Logstash event, e.g. "timestamp", "message", "hostname", "ipaddress". Also used to describe a key-value pair in Elasticsearch.
|
||||
|
||||
file::
|
||||
A resource storing binary data (which might be text, image, application, etc.) on a physical storage media. In the Logstash context, a common input source which monitors a growing collection of text-based log lines.
|
||||
|
||||
filter:
|
||||
An intermediary processing mechanism in the Lostash pipeline. Typically, filters act upon event data after it has been ingested via inputs, by mutating, enriching, and/or modifying the data according to configuration rules. The second phase of the typical Logstash pipeline (inputs->filters->outputs).
|
||||
|
||||
fluentd::
|
||||
Like Logstash, another open-source tool for collecting logs and events, with plugins to extend functionality.
|
||||
|
||||
ganglia::
|
||||
A scalable, distributed monitoring system suitable for large clusters. Logstash features both an input and an output to enable reading from, and writing to Ganglia.
|
||||
|
||||
graphite::
|
||||
A highly-scalable realtime graphing application, which presents graphs through web interfaces. Logstash provides an output which ships event data to Graphite for visualization.
|
||||
|
||||
heka::
|
||||
An open-source event processing system developed by Mozilla and often compared to Logstash.
|
||||
|
||||
index::
|
||||
An index can be seen as a named collection of documents in Elasticsearch which are available for searching and querying. It is a logical namespace which maps to one or more primary shards and can have zero or more replica shards.
|
||||
|
||||
indexer::
|
||||
Refers to a Logstash instance which is tasked with interfacing with an Elasticsearch cluster in order to index event data.
|
||||
|
||||
input::
|
||||
The means for ingesting data into Logstash. Inputs allow users to pull data from files, network sockets, other applications, etc. The initial phase of the typical Logstash pipeline (inputs->filters->outputs).
|
||||
|
||||
jar / jarfile::
|
||||
A packaging method for Java libraries. Since Logstash runs on the JRuby runtime environment, it is possible to use these Java libraries to provide extra functionality to Logstash.
|
||||
|
||||
java::
|
||||
An object-oriented programming language popular for its flexibility, extendability and portability.
|
||||
|
||||
jRuby:
|
||||
JRuby is a 100% Java implementation of the Ruby programming language, which allows Ruby to run in the JVM. Logstash typically runs in JRuby, which provides it with a fast, extensible runtime environment.
|
||||
|
||||
kibana::
|
||||
A visual tool for viewing time-based data which has been stored in Elasticsearch. Kibana features a powerful set of functionality based on panels which query Elasticsearch in different ways.
|
||||
|
||||
log::
|
||||
A snippet of textual information emitted by a device, ostensibly with some pertinent information about the status of said device.
|
||||
|
||||
log4j::
|
||||
A very common Java-based logging utility.
|
||||
|
||||
Logstash::
|
||||
An application which offers a powerful data processing pipeline, allowing users to consume information from various sources, enrich the data, and output it to any number of other sources.
|
||||
|
||||
lumberjack::
|
||||
A protocol for shipping logs from one location to another, in a secure and optimized manner. Also the (deprecated) name of a software application, now known as Logstash Forwarder (LSF).
|
||||
|
||||
output::
|
||||
The means for passing event data out of Logstash into other applications, network endpoints, files, etc. The last phase of the typical Logstash pipeline (inputs->filters->outputs).
|
||||
|
||||
pipeline::
|
||||
A term used to describe the flow of events through the Logstash workflow. The pipeline typically consists of a series of inputs, filters, and outputs.
|
||||
|
||||
plugin::
|
||||
A generic term referring to an input, codec, filter, or output which extends basic Logstash functionality.
|
||||
|
||||
redis::
|
||||
An open-source key-value store and cache which is often used in conjunction with Logstash as a message broker.
|
||||
|
||||
ruby::
|
||||
A popular, open-source, object-oriented programming language in which Logstash is implemented.
|
||||
|
||||
shell::
|
||||
A command-line interface to an operating system.
|
||||
|
||||
shipper::
|
||||
An instance of Logstash which send events to another instance of Logstash, or some other application.
|
||||
|
||||
statsd::
|
||||
A network daemon for aggregating statistics, such as counters and timers, and shipping over UDP to backend services, such as Graphite or Datadog. Logstash provides an output to statsd.
|
||||
|
||||
stdin::
|
||||
An I/O stream providing input to a software application. In Logstash, an input which receives data from this stream.
|
||||
|
||||
stdout::
|
||||
An I/O stream producing output from a software application. In Logstash, an output which produces data from this stream.
|
||||
|
||||
syslog::
|
||||
A popular method for logging messages from a computer. The standard is somewhat loose, but Logstash has tools (input, grok patterns) to make this simpler.
|
||||
|
||||
standalone::
|
||||
A configuration of Logstash in which the Logstash agent, input and output sources typically live on the same host machine.
|
||||
|
||||
thread::
|
||||
Parallel sequences of execution within a process which allow a computer to perform several tasks simultaneously, in a multi-processor environment. Logstash takes advantage of this functionality, by specifying the "-w" flag
|
||||
|
||||
type::
|
||||
In Elasticsearch type, a type can be compared to a table in a relational database. Each type has a list of fields that can be specified for documents of that type. The mapping defines how each field in the document is analyzed. To index documents, it is required to specify both an index and a type.
|
||||
|
||||
worker::
|
||||
The filter thread model used by Logstash, where each worker receives an event and applies all filters, in order, before emitting the event to the output queue. This allows scalability across CPUs because many filters are CPU intensive (permitting that we have thread safety).
|
||||
222
docs/jp/static/maintainer-guide.asciidoc
Normal file
222
docs/jp/static/maintainer-guide.asciidoc
Normal file
|
|
@ -0,0 +1,222 @@
|
|||
[[community-maintainer]]
|
||||
=== Logstash Plugins Community Maintainer Guide
|
||||
|
||||
This document, to be read by new Maintainers, should explain their responsibilities. It was inspired by the
|
||||
http://rfc.zeromq.org/spec:22[C4] document from the ZeroMQ project. This document is subject to change and suggestions
|
||||
through Pull Requests and issues are strongly encouraged.
|
||||
|
||||
[float]
|
||||
=== Contribution Guidelines
|
||||
|
||||
For general guidance around contributing to Logstash Plugins, see the
|
||||
https://www.elastic.co/guide/en/logstash/current/contributing-to-logstash.html[_Contributing to Logstash_] section.
|
||||
|
||||
[float]
|
||||
=== Document Goals
|
||||
|
||||
To help make the Logstash plugins community participation easy with positive feedback.
|
||||
|
||||
To increase diversity.
|
||||
|
||||
To reduce code review, merge and release dependencies on the core team by providing support and tools to the Community and
|
||||
Maintainers.
|
||||
|
||||
To support the natural life cycle of a plugin.
|
||||
|
||||
To codify the roles and responsibilities of: Maintainers and Contributors with specific focus on patch testing, code
|
||||
review, merging and release.
|
||||
|
||||
[float]
|
||||
=== Development Workflow
|
||||
|
||||
All Issues and Pull Requests must be tracked using the Github issue tracker.
|
||||
|
||||
The plugin uses the http://www.apache.org/licenses/LICENSE-2.0[Apache 2.0 license]. Maintainers should check whether a
|
||||
patch introduces code which has an incompatible license. Patch ownership and copyright is defined in the Elastic
|
||||
https://www.elastic.co/contributor-agreement[Contributor License Agreement] (CLA).
|
||||
|
||||
[float]
|
||||
==== Terminology
|
||||
|
||||
A "Contributor" is a role a person assumes when providing a patch. Contributors will not have commit access to the
|
||||
repository. They need to sign the Elastic https://www.elastic.co/contributor-agreement[Contributor License Agreement]
|
||||
before a patch can be reviewed. Contributors can add themselves to the plugin Contributor list.
|
||||
|
||||
A "Maintainer" is a role a person assumes when maintaining a plugin and keeping it healthy, including triaging issues, and
|
||||
reviewing and merging patches.
|
||||
|
||||
[float]
|
||||
==== Patch Requirements
|
||||
|
||||
A patch is a minimal and accurate answer to exactly one identified and agreed upon problem. It must conform to the
|
||||
https://github.com/elastic/logstash/blob/master/STYLE.md[code style guidelines] and must include RSpec tests that verify
|
||||
the fitness of the solution.
|
||||
|
||||
A patch will be automatically tested by a CI system that will report on the Pull Request status.
|
||||
|
||||
A patch CLA will be automatically verified and reported on the Pull Request status.
|
||||
|
||||
A patch commit message has a single short (less than 50 character) first line summarizing the change, a blank second line,
|
||||
and any additional lines as necessary for change explanation and rationale.
|
||||
|
||||
A patch is mergeable when it satisfies the above requirements and has been reviewed positively by at least one other
|
||||
person.
|
||||
|
||||
[float]
|
||||
==== Development Process
|
||||
|
||||
A user will log an issue on the issue tracker describing the problem they face or observe with as much detail as possible.
|
||||
|
||||
To work on an issue, a Contributor forks the plugin repository and then works on their forked repository and submits a
|
||||
patch by creating a pull request back to the plugin.
|
||||
|
||||
Maintainers must not merge patches where the author has not signed the CLA.
|
||||
|
||||
Before a patch can be accepted it should be reviewed. Maintainers should merge accepted patches without delay.
|
||||
|
||||
Maintainers should not merge their own patches except in exceptional cases, such as non-responsiveness from other
|
||||
Maintainers or core team for an extended period (more than 2 weeks).
|
||||
|
||||
Reviewer’s comments should not be based on personal preferences.
|
||||
|
||||
The Maintainers should label Issues and Pull Requests.
|
||||
|
||||
Maintainers should involve the core team if help is needed to reach consensus.
|
||||
|
||||
Review non-source changes such as documentation in the same way as source code changes.
|
||||
|
||||
[float]
|
||||
==== Branch Management
|
||||
|
||||
The plugin has a master branch that always holds the latest in-progress version and should always build. Topic branches
|
||||
should kept to the minimum.
|
||||
|
||||
[float]
|
||||
==== Changelog Management
|
||||
|
||||
Every plugin should have a changelog (CHANGELOG.md). If not, please create one. When changes are made to a plugin, make sure to include a changelog entry under the respective version to document the change. The changelog should be easily understood from a user point of view. As we iterate and release plugins rapidly, users use the changelog as a mechanism for deciding whether to update.
|
||||
|
||||
Changes that are not user facing should be tagged as `internal:`. For example:
|
||||
|
||||
[source,markdown]
|
||||
- internal: Refactored specs for better testing
|
||||
- config: Default timeout configuration changed from 10s to 5s
|
||||
|
||||
[float]
|
||||
===== Detailed format of CHANGELOG.md
|
||||
|
||||
Sharing a similar format of CHANGELOG.md in plugins ease readability for users.
|
||||
Please see following annotated example and see a concrete example in https://raw.githubusercontent.com/logstash-plugins/logstash-filter-date/master/CHANGELOG.md[logstash-filter-date].
|
||||
|
||||
[source,markdown]
|
||||
----
|
||||
## 1.0.x // <1> <2>
|
||||
- change description // <3>
|
||||
- tag: change description // <3> <4>
|
||||
- tag1,tag2: change description // <3> <5>
|
||||
- tag: Multi-line description // <3> <6>
|
||||
must be indented and can use
|
||||
additional markdown syntax
|
||||
// <7>
|
||||
## 1.0.0 // <8>
|
||||
[...]
|
||||
|
||||
----
|
||||
<1> Latest version is the first line of CHANGELOG.md
|
||||
<2> Each version identifier should be a level-2 header using `##`
|
||||
<3> One change description is described as a list item using a dash `-` aligned under the version identifier
|
||||
<4> One change can be tagged by a word and suffixed by `:`. +
|
||||
Common tags are `bugfix`, `feature`, `doc`, `test` or `internal`.
|
||||
<5> One change can have multiple tags separated by a comma and suffixed by `:`
|
||||
<6> A multi-line change description must be properly indented
|
||||
<7> Please take care to *separate versions with an empty line*
|
||||
<8> Previous version identifier
|
||||
|
||||
[float]
|
||||
==== Continuous Integration
|
||||
|
||||
Plugins are setup with automated continuous integration (CI) environments and there should be a corresponding badge on each Github page. If it’s missing, please contact the Logstash core team.
|
||||
|
||||
Every Pull Request opened automatically triggers a CI run. To conduct a manual run, comment “Jenkins, please test this.” on the Pull Request.
|
||||
|
||||
[float]
|
||||
=== Versioning Plugins
|
||||
|
||||
Logstash core and its plugins have separate product development lifecycles. Hence the versioning and release strategy for
|
||||
the core and plugins do not have to be aligned. In fact, this was one of our goals during the great separation of plugins
|
||||
work in Logstash 1.5.
|
||||
|
||||
At times, there will be changes in core API in Logstash, which will require mass update of plugins to reflect the changes
|
||||
in core. However, this does not happen frequently.
|
||||
|
||||
For plugins, we would like to adhere to a versioning and release strategy that can better inform our users, about any
|
||||
breaking changes to the Logstash configuration formats and functionality.
|
||||
|
||||
Plugin releases follows a three-placed numbering scheme X.Y.Z. where X denotes a major release version which may break
|
||||
compatibility with existing configuration or functionality. Y denotes releases which includes features which are backward
|
||||
compatible. Z denotes releases which includes bug fixes and patches.
|
||||
|
||||
[float]
|
||||
==== Changing the version
|
||||
|
||||
Version can be changed in the Gemspec, which needs to be associated with a changelog entry. Following this, we can publish
|
||||
the gem to RubyGem.org manually. At this point only the core developers can publish a gem.
|
||||
|
||||
[float]
|
||||
==== Labeling
|
||||
|
||||
Labeling is a critical aspect of maintaining plugins. All issues in GitHub should be labeled correctly so it can:
|
||||
|
||||
* Provide good feedback to users/developers
|
||||
* Help prioritize changes
|
||||
* Be used in release notes
|
||||
|
||||
Most labels are self explanatory, but here’s a quick recap of few important labels:
|
||||
|
||||
* `bug`: Labels an issue as an unintentional defect
|
||||
* `needs details`: If a the issue reporter has incomplete details, please ask them for more info and label as needs
|
||||
details.
|
||||
* `missing cla`: Contributor License Agreement is missing and patch cannot be accepted without it
|
||||
* `adopt me`: Ask for help from the community to take over this issue
|
||||
* `enhancement`: New feature, not a bug fix
|
||||
* `needs tests`: Patch has no tests, and cannot be accepted without unit/integration tests
|
||||
* `docs`: Documentation related issue/PR
|
||||
|
||||
[float]
|
||||
=== Logging
|
||||
|
||||
Although it’s important not to bog down performance with excessive logging, debug level logs can be immensely helpful when
|
||||
diagnosing and troubleshooting issues with Logstash. Please remember to liberally add debug logs wherever it makes sense
|
||||
as users will be forever gracious.
|
||||
|
||||
[source,shell]
|
||||
@logger.debug("Logstash loves debug logs!", :actions => actions)
|
||||
|
||||
[float]
|
||||
=== Contributor License Agreement (CLA) Guidance
|
||||
|
||||
[qanda]
|
||||
Why is a https://www.elastic.co/contributor-agreement[CLA] required?::
|
||||
We ask this of all Contributors in order to assure our users of the origin and continuing existence of the code. We
|
||||
are not asking Contributors to assign copyright to us, but to give us the right to distribute a Contributor’s code
|
||||
without restriction.
|
||||
|
||||
Please make sure the CLA is signed by every Contributor prior to reviewing PRs and commits.::
|
||||
Contributors only need to sign the CLA once and should sign with the same email as used in Github. If a Contributor
|
||||
signs the CLA after a PR is submitted, they can refresh the automated CLA checker by pushing another
|
||||
comment on the PR after 5 minutes of signing.
|
||||
|
||||
[float]
|
||||
=== Need Help?
|
||||
|
||||
Ping @logstash-core on Github to get the attention of the Logstash core team.
|
||||
|
||||
[float]
|
||||
=== Community Administration
|
||||
|
||||
The core team is there to support the plugin Maintainers and overall ecosystem.
|
||||
|
||||
Maintainers should propose Contributors to become a Maintainer.
|
||||
|
||||
Contributors and Maintainers should follow the Elastic Community https://www.elastic.co/community/codeofconduct[Code of
|
||||
Conduct]. The core team should block or ban "bad actors".
|
||||
106
docs/jp/static/managing-multiline-events.asciidoc
Normal file
106
docs/jp/static/managing-multiline-events.asciidoc
Normal file
|
|
@ -0,0 +1,106 @@
|
|||
[[multiline]]
|
||||
=== Managing Multiline Events
|
||||
|
||||
Several use cases generate events that span multiple lines of text. In order to correctly handle these multiline events,
|
||||
Logstash needs to know how to tell which lines are part of a single event.
|
||||
|
||||
Multiline event processing is complex and relies on proper event ordering. The best way to guarantee ordered log
|
||||
processing is to implement the processing as early in the pipeline as possible. The preferred tool in the Logstash
|
||||
pipeline is the {logstash}plugins-codecs-multiline.html[multiline codec], which merges lines from a single input using
|
||||
a simple set of rules.
|
||||
|
||||
|
||||
The most important aspects of configuring the multiline codec are the following:
|
||||
|
||||
* The `pattern` option specifies a regular expression. Lines that match the specified regular expression are considered
|
||||
either continuations of a previous line or the start of a new multiline event. You can use
|
||||
{logstash}plugins-filters-grok.html[grok] regular expression templates with this configuration option.
|
||||
* The `what` option takes two values: `previous` or `next`. The `previous` value specifies that lines that match the
|
||||
value in the `pattern` option are part of the previous line. The `next` value specifies that lines that match the value
|
||||
in the `pattern` option are part of the following line.* The `negate` option applies the multiline codec to lines that
|
||||
_do not_ match the regular expression specified in the `pattern` option.
|
||||
|
||||
See the full documentation for the {logstash}plugins-codecs-multiline.html[multiline codec] plugin for more information
|
||||
on configuration options.
|
||||
|
||||
==== Examples of Multiline Codec Configuration
|
||||
|
||||
The examples in this section cover the following use cases:
|
||||
|
||||
* Combining a Java stack trace into a single event
|
||||
* Combining C-style line continuations into a single event
|
||||
* Combining multiple lines from time-stamped events
|
||||
|
||||
===== Java Stack Traces
|
||||
|
||||
Java stack traces consist of multiple lines, with each line after the initial line beginning with whitespace, as in
|
||||
this example:
|
||||
|
||||
[source,java]
|
||||
Exception in thread "main" java.lang.NullPointerException
|
||||
at com.example.myproject.Book.getTitle(Book.java:16)
|
||||
at com.example.myproject.Author.getBookTitles(Author.java:25)
|
||||
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
|
||||
|
||||
To consolidate these lines into a single event in Logstash, use the following configuration for the multiline codec:
|
||||
|
||||
[source,json]
|
||||
input {
|
||||
stdin {
|
||||
codec => multiline {
|
||||
pattern => "^\s"
|
||||
what => "previous"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
This configuration merges any line that begins with whitespace up to the previous line.
|
||||
|
||||
===== Line Continuations
|
||||
|
||||
Several programming languages use the `\` character at the end of a line to denote that the line continues, as in this
|
||||
example:
|
||||
|
||||
[source,c]
|
||||
printf ("%10.10ld \t %10.10ld \t %s\
|
||||
%f", w, x, y, z );
|
||||
|
||||
To consolidate these lines into a single event in Logstash, use the following configuration for the multiline codec:
|
||||
|
||||
[source,json]
|
||||
input {
|
||||
stdin {
|
||||
codec => multiline {
|
||||
pattern => "\\$"
|
||||
what => "next"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
This configuration merges any line that ends with the `\` character with the following line.
|
||||
|
||||
===== Timestamps
|
||||
|
||||
Activity logs from services such as Elasticsearch typically begin with a timestamp, followed by information on the
|
||||
specific activity, as in this example:
|
||||
|
||||
[source,shell]
|
||||
[2015-08-24 11:49:14,389][INFO ][env ] [Letha] using [1] data paths, mounts [[/
|
||||
(/dev/disk1)]], net usable_space [34.5gb], net total_space [118.9gb], types [hfs]
|
||||
|
||||
To consolidate these lines into a single event in Logstash, use the following configuration for the multiline codec:
|
||||
|
||||
[source,json]
|
||||
input {
|
||||
file {
|
||||
path => "/var/log/someapp.log"
|
||||
codec => multiline {
|
||||
pattern => "^%{TIMESTAMP_ISO8601} "
|
||||
negate => true
|
||||
what => previous
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
This configuration uses the `negate` option to specify that any line that does not begin with a timestamp belongs to
|
||||
the previous line.
|
||||
661
docs/jp/static/monitoring-apis.asciidoc
Normal file
661
docs/jp/static/monitoring-apis.asciidoc
Normal file
|
|
@ -0,0 +1,661 @@
|
|||
[[monitoring]]
|
||||
== Monitoring APIs
|
||||
|
||||
experimental[]
|
||||
|
||||
Logstash provides the following monitoring APIs to retrieve runtime metrics
|
||||
about Logstash:
|
||||
|
||||
* <<node-info-api>>
|
||||
* <<plugins-api>>
|
||||
* <<node-stats-api>>
|
||||
* <<hot-threads-api>>
|
||||
|
||||
|
||||
You can use the root resource to retrieve general information about the Logstash instance, including
|
||||
the host and version.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET /
|
||||
--------------------------------------------------
|
||||
|
||||
Example response:
|
||||
|
||||
["source","js",subs="attributes"]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"host": "skywalker",
|
||||
"version": "{logstash_version}",
|
||||
"http_address": "127.0.0.1:9600"
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
NOTE: By default, the monitoring API attempts to bind to `tcp:9600`. If this port is already in use by another Logstash
|
||||
instance, you need to launch Logstash with the `--http.port` flag specified to bind to a different port. See
|
||||
<<command-line-flags>> for more information.
|
||||
|
||||
[float]
|
||||
[[monitoring-common-options]]
|
||||
=== Common Options
|
||||
|
||||
The following options can be applied to all of the Logstash monitoring APIs.
|
||||
|
||||
[float]
|
||||
==== Pretty Results
|
||||
|
||||
When appending `?pretty=true` to any request made, the JSON returned
|
||||
will be pretty formatted (use it for debugging only!).
|
||||
|
||||
[float]
|
||||
==== Human-Readable Output
|
||||
|
||||
NOTE: For Logstash {logstash_version}, the `human` option is supported for the <<hot-threads-api>>
|
||||
only. When you specify `human=true`, the results are returned in plain text instead of
|
||||
JSON format. The default is false.
|
||||
|
||||
Statistics are returned in a format suitable for humans
|
||||
(eg `"exists_time": "1h"` or `"size": "1kb"`) and for computers
|
||||
(eg `"exists_time_in_millis": 3600000` or `"size_in_bytes": 1024`).
|
||||
The human-readable values can be turned off by adding `?human=false`
|
||||
to the query string. This makes sense when the stats results are
|
||||
being consumed by a monitoring tool, rather than intended for human
|
||||
consumption. The default for the `human` flag is
|
||||
`false`.
|
||||
|
||||
[[node-info-api]]
|
||||
=== Node Info API
|
||||
|
||||
experimental[]
|
||||
|
||||
The node info API retrieves information about the node.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET /_node/<types>
|
||||
--------------------------------------------------
|
||||
|
||||
Where `<types>` is optional and specifies the types of node info you want to return.
|
||||
|
||||
You can limit the info that's returned by combining any of the following types in a comma-separated list:
|
||||
|
||||
[horizontal]
|
||||
<<node-pipeline-info,`pipeline`>>::
|
||||
Gets pipeline-specific information and settings.
|
||||
<<node-os-info,`os`>>::
|
||||
Gets node-level info about the OS.
|
||||
<<node-jvm-info,`jvm`>>::
|
||||
Gets node-level JVM info, including info about threads.
|
||||
|
||||
See <<monitoring-common-options, Common Options>> for a list of options that can be applied to all
|
||||
Logstash monitoring APIs.
|
||||
|
||||
[[node-pipeline-info]]
|
||||
==== Pipeline Info
|
||||
|
||||
The following request returns a JSON document that shows pipeline info, such as the number of workers,
|
||||
batch size, and batch delay:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET /_node/pipeline
|
||||
--------------------------------------------------
|
||||
|
||||
If you want to view additional information about the pipeline, such as stats for each configured input, filter,
|
||||
or output stage, see the <<pipeline-stats>> section under the <<node-stats-api>>.
|
||||
|
||||
Example response:
|
||||
|
||||
["source","js",subs="attributes"]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"pipeline": {
|
||||
"workers": 8,
|
||||
"batch_size": 125,
|
||||
"batch_delay": 5,
|
||||
"config_reload_automatic": true,
|
||||
"config_reload_interval": 3
|
||||
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
[[node-os-info]]
|
||||
==== OS Info
|
||||
|
||||
The following request returns a JSON document that shows the OS name, architecture, version, and
|
||||
available processors:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET /_node/os
|
||||
--------------------------------------------------
|
||||
|
||||
Example response:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"os": {
|
||||
"name": "Mac OS X",
|
||||
"arch": "x86_64",
|
||||
"version": "10.12.1",
|
||||
"available_processors": 8
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
[[node-jvm-info]]
|
||||
==== JVM Info
|
||||
|
||||
The following request returns a JSON document that shows node-level JVM stats, such as the JVM process id, version,
|
||||
VM info, memory usage, and info about garbage collectors:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET /_node/jvm
|
||||
--------------------------------------------------
|
||||
|
||||
Example response:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"jvm": {
|
||||
"pid": 59616,
|
||||
"version": "1.8.0_65",
|
||||
"vm_name": "Java HotSpot(TM) 64-Bit Server VM",
|
||||
"vm_version": "1.8.0_65",
|
||||
"vm_vendor": "Oracle Corporation",
|
||||
"start_time_in_millis": 1484251185878,
|
||||
"mem": {
|
||||
"heap_init_in_bytes": 268435456,
|
||||
"heap_max_in_bytes": 1037959168,
|
||||
"non_heap_init_in_bytes": 2555904,
|
||||
"non_heap_max_in_bytes": 0
|
||||
},
|
||||
"gc_collectors": [
|
||||
"ParNew",
|
||||
"ConcurrentMarkSweep"
|
||||
]
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
[[plugins-api]]
|
||||
=== Plugins Info API
|
||||
|
||||
experimental[]
|
||||
|
||||
The plugins info API gets information about all Logstash plugins that are currently installed.
|
||||
This API basically returns the output of running the `bin/logstash-plugin list --verbose` command.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET /_node/plugins
|
||||
--------------------------------------------------
|
||||
|
||||
See <<monitoring-common-options, Common Options>> for a list of options that can be applied to all
|
||||
Logstash monitoring APIs.
|
||||
|
||||
The output is a JSON document.
|
||||
|
||||
Example response:
|
||||
|
||||
["source","js",subs="attributes"]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"total": 92,
|
||||
"plugins": [
|
||||
{
|
||||
"name": "logstash-codec-cef",
|
||||
"version": "4.1.2"
|
||||
},
|
||||
{
|
||||
"name": "logstash-codec-collectd",
|
||||
"version": "3.0.3"
|
||||
},
|
||||
{
|
||||
"name": "logstash-codec-dots",
|
||||
"version": "3.0.2"
|
||||
},
|
||||
{
|
||||
"name": "logstash-codec-edn",
|
||||
"version": "3.0.2"
|
||||
},
|
||||
.
|
||||
.
|
||||
.
|
||||
]
|
||||
--------------------------------------------------
|
||||
|
||||
[[node-stats-api]]
|
||||
=== Node Stats API
|
||||
|
||||
experimental[]
|
||||
|
||||
The node stats API retrieves runtime stats about Logstash.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET /_node/stats/<types>
|
||||
--------------------------------------------------
|
||||
|
||||
Where `<types>` is optional and specifies the types of stats you want to return.
|
||||
|
||||
By default, all stats are returned. You can limit the info that's returned by combining any of the following types in a comma-separated list:
|
||||
|
||||
[horizontal]
|
||||
<<jvm-stats,`jvm`>>::
|
||||
Gets JVM stats, including stats about threads, memory usage, garbage collectors,
|
||||
and uptime.
|
||||
<<process-stats,`process`>>::
|
||||
Gets process stats, including stats about file descriptors, memory consumption, and CPU usage.
|
||||
<<pipeline-stats,`pipeline`>>::
|
||||
Gets runtime stats about the Logstash pipeline.
|
||||
<<reload-stats,`reloads`>>::
|
||||
Gets runtime stats about config reload successes and failures.
|
||||
<<os-stats,`os`>>::
|
||||
Gets runtime stats about cgroups when Logstash is running in a container.
|
||||
|
||||
See <<monitoring-common-options, Common Options>> for a list of options that can be applied to all
|
||||
Logstash monitoring APIs.
|
||||
|
||||
[[jvm-stats]]
|
||||
==== JVM Stats
|
||||
|
||||
The following request returns a JSON document containing JVM stats:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET /_node/stats/jvm
|
||||
--------------------------------------------------
|
||||
|
||||
Example response:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"jvm": {
|
||||
"threads": {
|
||||
"count": 35,
|
||||
"peak_count": 36
|
||||
},
|
||||
"mem": {
|
||||
"heap_used_in_bytes": 318691184,
|
||||
"heap_used_percent": 15,
|
||||
"heap_committed_in_bytes": 519045120,
|
||||
"heap_max_in_bytes": 2075918336,
|
||||
"non_heap_used_in_bytes": 189382304,
|
||||
"non_heap_committed_in_bytes": 200728576,
|
||||
"pools": {
|
||||
"survivor": {
|
||||
"peak_used_in_bytes": 8912896,
|
||||
"used_in_bytes": 9538656,
|
||||
"peak_max_in_bytes": 35782656,
|
||||
"max_in_bytes": 71565312,
|
||||
"committed_in_bytes": 17825792
|
||||
},
|
||||
"old": {
|
||||
"peak_used_in_bytes": 106946320,
|
||||
"used_in_bytes": 181913072,
|
||||
"peak_max_in_bytes": 715849728,
|
||||
"max_in_bytes": 1431699456,
|
||||
"committed_in_bytes": 357957632
|
||||
},
|
||||
"young": {
|
||||
"peak_used_in_bytes": 71630848,
|
||||
"used_in_bytes": 127239456,
|
||||
"peak_max_in_bytes": 286326784,
|
||||
"max_in_bytes": 572653568,
|
||||
"committed_in_bytes": 143261696
|
||||
}
|
||||
}
|
||||
},
|
||||
"gc": {
|
||||
"collectors": {
|
||||
"old": {
|
||||
"collection_time_in_millis": 58,
|
||||
"collection_count": 2
|
||||
},
|
||||
"young": {
|
||||
"collection_time_in_millis": 338,
|
||||
"collection_count": 26
|
||||
}
|
||||
}
|
||||
},
|
||||
"uptime_in_millis": 382701
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
[[process-stats]]
|
||||
==== Process Stats
|
||||
|
||||
The following request returns a JSON document containing process stats:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET /_node/stats/process
|
||||
--------------------------------------------------
|
||||
|
||||
Example response:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"process": {
|
||||
"open_file_descriptors": 164,
|
||||
"peak_open_file_descriptors": 166,
|
||||
"max_file_descriptors": 10240,
|
||||
"mem": {
|
||||
"total_virtual_in_bytes": 5399474176
|
||||
},
|
||||
"cpu": {
|
||||
"total_in_millis": 72810537000,
|
||||
"percent": 0,
|
||||
"load_average": {
|
||||
"1m": 2.41943359375
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
[[pipeline-stats]]
|
||||
==== Pipeline Stats
|
||||
|
||||
The following request returns a JSON document containing pipeline stats,
|
||||
including:
|
||||
|
||||
* the number of events that were input, filtered, or output by the pipeline
|
||||
* stats for each configured filter or output stage
|
||||
* info about config reload successes and failures
|
||||
(when <<reloading-config,config reload>> is enabled)
|
||||
* info about the persistent queue (when
|
||||
<<persistent-queues,persistent queues>> are enabled)
|
||||
|
||||
NOTE: Detailed pipeline stats for input plugins are not currently available, but
|
||||
will be available in a future release. For now, the node stats API returns an
|
||||
empty set array for inputs (`"inputs": []`).
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET /_node/stats/pipeline
|
||||
--------------------------------------------------
|
||||
|
||||
Example response:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"pipeline": {
|
||||
"events": {
|
||||
"duration_in_millis": 6304989,
|
||||
"in": 200,
|
||||
"filtered": 200,
|
||||
"out": 200
|
||||
},
|
||||
"plugins": {
|
||||
"inputs": [],
|
||||
"filters": [
|
||||
{
|
||||
"id": "4e3d4bed6ba821ebb47f4752bb757b04a754d736-2",
|
||||
"events": {
|
||||
"duration_in_millis": 113,
|
||||
"in": 200,
|
||||
"out": 200
|
||||
},
|
||||
"matches": 200,
|
||||
"patterns_per_field": {
|
||||
"message": 1
|
||||
},
|
||||
"name": "grok"
|
||||
},
|
||||
{
|
||||
"id": "4e3d4bed6ba821ebb47f4752bb757b04a754d736-3",
|
||||
"events": {
|
||||
"duration_in_millis": 526,
|
||||
"in": 200,
|
||||
"out": 200
|
||||
},
|
||||
"name": "geoip"
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"id": "4e3d4bed6ba821ebb47f4752bb757b04a754d736-4",
|
||||
"events": {
|
||||
"duration_in_millis": 2312,
|
||||
"in": 200,
|
||||
"out": 200
|
||||
},
|
||||
"name": "stdout"
|
||||
}
|
||||
]
|
||||
},
|
||||
"reloads": {
|
||||
"last_error": null,
|
||||
"successes": 0,
|
||||
"last_success_timestamp": null,
|
||||
"last_failure_timestamp": null,
|
||||
"failures": 0
|
||||
},
|
||||
"queue": {
|
||||
"events": 26,
|
||||
"type": "persisted",
|
||||
"capacity": {
|
||||
"page_capacity_in_bytes": 262144000,
|
||||
"max_queue_size_in_bytes": 4294967296,
|
||||
"max_unread_events": 0
|
||||
},
|
||||
"data": {
|
||||
"path": "/path/to/data/queue",
|
||||
"free_space_in_bytes": 123027787776,
|
||||
"storage_type": "hfs"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
[[reload-stats]]
|
||||
==== Reload Stats
|
||||
|
||||
The following request returns a JSON document that shows info about config reload successes and failures.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET /_node/stats/reloads
|
||||
--------------------------------------------------
|
||||
|
||||
Example response:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"reloads": {
|
||||
"successes": 0,
|
||||
"failures": 0
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
[[os-stats]]
|
||||
==== OS Stats
|
||||
|
||||
When Logstash is running in a container, the following request returns a JSON document that
|
||||
contains cgroup information to give you a more accurate view of CPU load, including whether
|
||||
the container is being throttled.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET /_node/stats/os
|
||||
--------------------------------------------------
|
||||
|
||||
Example response:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"os" : {
|
||||
"cgroup" : {
|
||||
"cpuacct" : {
|
||||
"control_group" : "/elastic1",
|
||||
"usage_nanos" : 378477588075
|
||||
},
|
||||
"cpu" : {
|
||||
"control_group" : "/elastic1",
|
||||
"cfs_period_micros" : 1000000,
|
||||
"cfs_quota_micros" : 800000,
|
||||
"stat" : {
|
||||
"number_of_elapsed_periods" : 4157,
|
||||
"number_of_times_throttled" : 460,
|
||||
"time_throttled_nanos" : 581617440755
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
|
||||
[[hot-threads-api]]
|
||||
=== Hot Threads API
|
||||
|
||||
experimental[]
|
||||
|
||||
The hot threads API gets the current hot threads for Logstash. A hot thread is a
|
||||
Java thread that has high CPU usage and executes for a longer than normal period
|
||||
of time.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET /_node/hot_threads
|
||||
--------------------------------------------------
|
||||
|
||||
The output is a JSON document that contains a breakdown of the top hot threads for
|
||||
Logstash.
|
||||
|
||||
Example response:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"time": "2017-01-12T12:09:45-08:00",
|
||||
"busiest_threads": 3,
|
||||
"threads": [
|
||||
{
|
||||
"name": "LogStash::Runner",
|
||||
"percent_of_cpu_time": 1.07,
|
||||
"state": "timed_waiting",
|
||||
"traces": [
|
||||
"java.lang.Object.wait(Native Method)",
|
||||
"java.lang.Thread.join(Thread.java:1253)",
|
||||
"org.jruby.internal.runtime.NativeThread.join(NativeThread.java:75)",
|
||||
"org.jruby.RubyThread.join(RubyThread.java:697)",
|
||||
"org.jruby.RubyThread$INVOKER$i$0$1$join.call(RubyThread$INVOKER$i$0$1$join.gen)",
|
||||
"org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:663)",
|
||||
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:198)",
|
||||
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)",
|
||||
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)",
|
||||
"org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "[main]>worker7",
|
||||
"percent_of_cpu_time": 0.71,
|
||||
"state": "waiting",
|
||||
"traces": [
|
||||
"sun.misc.Unsafe.park(Native Method)",
|
||||
"java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)",
|
||||
"java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)",
|
||||
"java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:897)",
|
||||
"java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1222)",
|
||||
"java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)",
|
||||
"org.jruby.RubyThread.lockInterruptibly(RubyThread.java:1470)",
|
||||
"org.jruby.ext.thread.Mutex.lock(Mutex.java:91)",
|
||||
"org.jruby.ext.thread.Mutex.synchronize(Mutex.java:147)",
|
||||
"org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "[main]>worker3",
|
||||
"percent_of_cpu_time": 0.71,
|
||||
"state": "waiting",
|
||||
"traces": [
|
||||
"sun.misc.Unsafe.park(Native Method)",
|
||||
"java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)",
|
||||
"java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)",
|
||||
"java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:897)",
|
||||
"java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1222)",
|
||||
"java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)",
|
||||
"org.jruby.RubyThread.lockInterruptibly(RubyThread.java:1470)",
|
||||
"org.jruby.ext.thread.Mutex.lock(Mutex.java:91)",
|
||||
"org.jruby.ext.thread.Mutex.synchronize(Mutex.java:147)",
|
||||
"org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
The parameters allowed are:
|
||||
|
||||
[horizontal]
|
||||
`threads`:: The number of hot threads to return. The default is 3.
|
||||
`human`:: If true, returns plain text instead of JSON format. The default is false.
|
||||
`ignore_idle_threads`:: If true, does not return idle threads. The default is true.
|
||||
|
||||
See <<monitoring-common-options, Common Options>> for a list of options that can be applied to all
|
||||
Logstash monitoring APIs.
|
||||
|
||||
You can use the `?human` parameter to return the document in a human-readable format.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET /_node/hot_threads?human=true
|
||||
--------------------------------------------------
|
||||
|
||||
Example of a human-readable response:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
::: {}
|
||||
Hot threads at 2017-01-12T12:10:15-08:00, busiestThreads=3:
|
||||
================================================================================
|
||||
1.02 % of cpu usage, state: timed_waiting, thread name: 'LogStash::Runner'
|
||||
java.lang.Object.wait(Native Method)
|
||||
java.lang.Thread.join(Thread.java:1253)
|
||||
org.jruby.internal.runtime.NativeThread.join(NativeThread.java:75)
|
||||
org.jruby.RubyThread.join(RubyThread.java:697)
|
||||
org.jruby.RubyThread$INVOKER$i$0$1$join.call(RubyThread$INVOKER$i$0$1$join.gen)
|
||||
org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:663)
|
||||
org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:198)
|
||||
org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
|
||||
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
|
||||
org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
|
||||
--------------------------------------------------------------------------------
|
||||
0.71 % of cpu usage, state: waiting, thread name: '[main]>worker7'
|
||||
sun.misc.Unsafe.park(Native Method)
|
||||
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
|
||||
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
|
||||
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:897)
|
||||
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1222)
|
||||
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
|
||||
org.jruby.RubyThread.lockInterruptibly(RubyThread.java:1470)
|
||||
org.jruby.ext.thread.Mutex.lock(Mutex.java:91)
|
||||
org.jruby.ext.thread.Mutex.synchronize(Mutex.java:147)
|
||||
org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)
|
||||
--------------------------------------------------------------------------------
|
||||
0.71 % of cpu usage, state: timed_waiting, thread name: '[main]>worker3'
|
||||
sun.misc.Unsafe.park(Native Method)
|
||||
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
|
||||
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
|
||||
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
|
||||
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
|
||||
sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
|
||||
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
|
||||
java.lang.reflect.Method.invoke(Method.java:497)
|
||||
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:466)
|
||||
org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:324)
|
||||
|
||||
--------------------------------------------------
|
||||
|
||||
142
docs/jp/static/offline-plugins.asciidoc
Normal file
142
docs/jp/static/offline-plugins.asciidoc
Normal file
|
|
@ -0,0 +1,142 @@
|
|||
[[offline-plugins]]
|
||||
=== Offline Plugin Management
|
||||
|
||||
The Logstash <<working-with-plugins,plugin manager>> provides support for preparing offline plugin packs that you can
|
||||
use to install Logstash plugins on systems that don't have Internet access.
|
||||
|
||||
This procedure requires a staging machine running Logstash that has access to a public or
|
||||
<<private-rubygem,private Rubygems>> server. The staging machine downloads and packages all the files and dependencies
|
||||
required for offline installation.
|
||||
|
||||
NOTE: If you used offline plugin management prior to Logstash 5.2, you used the `pack` and `unpack` subcommands. Those
|
||||
subcommands are now deprecated, but the procedure for using them is still available in the documentation
|
||||
<<building-offline-packages-deprecated,here>>.
|
||||
|
||||
[[building-offline-packs]]
|
||||
[float]
|
||||
=== Building Offline Plugin Packs
|
||||
|
||||
An _offline plugin pack_ is a compressed file that contains all the plugins your offline Logstash installation requires,
|
||||
along with the dependencies for those plugins.
|
||||
|
||||
To build an offline plugin pack:
|
||||
|
||||
. Make sure all the plugins that you want to package are installed on the staging server and that the staging server can
|
||||
access the Internet.
|
||||
|
||||
. Run the `bin/logstash-plugin prepare-offline-pack` subcommand to package the plugins and dependencies:
|
||||
+
|
||||
[source, shell]
|
||||
-------------------------------------------------------------------------------
|
||||
bin/logstash-plugin prepare-offline-pack --output OUTPUT [PLUGINS] --overwrite
|
||||
-------------------------------------------------------------------------------
|
||||
+
|
||||
where:
|
||||
+
|
||||
* `OUTPUT` specifies the zip file where the compressed plugin pack will be written. The default file is
|
||||
+/LOGSTASH_HOME/logstash-offline-plugins-{logstash_version}.zip+. If you are using 5.2.x and 5.3.0, this location should be a zip file whose contents will be overwritten.
|
||||
* `[PLUGINS]` specifies one or more plugins that you want to include in the pack.
|
||||
* `--overwrite` specifies if you want to override an existing file at the location
|
||||
|
||||
Examples:
|
||||
|
||||
["source","sh",subs="attributes"]
|
||||
-------------------------------------------------------------------------------
|
||||
bin/logstash-plugin prepare-offline-pack logstash-input-beats <1>
|
||||
bin/logstash-plugin prepare-offline-pack logstash-filter-* <2>
|
||||
bin/logstash-plugin prepare-offline-pack logstash-filter-* logstash-input-beats <3>
|
||||
-------------------------------------------------------------------------------
|
||||
<1> Packages the Beats input plugin and any dependencies.
|
||||
<2> Uses a wildcard to package all filter plugins and any dependencies.
|
||||
<3> Packages all filter plugins, the Beats input plugin, and any dependencies.
|
||||
|
||||
NOTE: Downloading all dependencies for the specified plugins may take some time, depending on the plugins listed.
|
||||
|
||||
[[installing-offline-packs]]
|
||||
[float]
|
||||
=== Installing Offline Plugin Packs
|
||||
|
||||
To install an offline plugin pack:
|
||||
|
||||
. Move the compressed bundle to the machine where you want to install the plugins.
|
||||
|
||||
. Run the `bin/logstash-plugin install` subcommand to install the packaged plugins:
|
||||
+
|
||||
["source","sh",subs="attributes"]
|
||||
-------------------------------------------------------------------------------
|
||||
bin/logstash-plugin install file:///path/to/logstash-offline-plugins-{logstash_version}.zip
|
||||
-------------------------------------------------------------------------------
|
||||
+
|
||||
Where +path/to/logstash-offline-plugins-{logstash_version}.zip+ is the path to the offline plugin pack.
|
||||
|
||||
[float]
|
||||
=== Updating Offline Plugins
|
||||
|
||||
To update offline plugins, you update the plugins on the staging server and then use the same process that you followed to
|
||||
build and install the plugin pack:
|
||||
|
||||
. On the staging server, run the `bin/logstash-plugin update` subcommand to update the plugins. See <<updating-plugins>>.
|
||||
|
||||
. Create a new version of the plugin pack. See <<building-offline-packs>>.
|
||||
|
||||
. Install the new version of the plugin pack. See <<installing-offline-packs>>.
|
||||
|
||||
|
||||
[[building-offline-packages-deprecated]]
|
||||
[float]
|
||||
=== Building the Offline Package (Deprecated Procedure)
|
||||
|
||||
deprecated[5.2, Starting with Logstash 5.2, the `pack` and `unpack` commands are deprecated and replaced by the `prepare-offline-pack` and `install` commands]
|
||||
|
||||
Working with offline plugins requires you to create an _offline package_, which is a compressed file that contains all of
|
||||
the plugins your offline Logstash installation requires, along with the dependencies for those plugins.
|
||||
|
||||
. Create the offline package with the `bin/logstash-plugin pack` subcommand.
|
||||
+
|
||||
When you run the `bin/logstash-plugin pack` subcommand, Logstash creates a compressed bundle that contains all of the currently
|
||||
installed plugins and the dependencies for those plugins. By default, the compressed bundle is a GZipped TAR file when you
|
||||
run the `bin/logstash-plugin pack` subcommand on a UNIX machine. By default, when you run the `bin/logstash-plugin pack` subcommand on a
|
||||
Windows machine, the compressed bundle is a ZIP file. See <<managing-packs,Managing Plugin Packs>> for details on changing
|
||||
these default behaviors.
|
||||
+
|
||||
NOTE: Downloading all dependencies for the specified plugins may take some time, depending on the plugins listed.
|
||||
|
||||
. Move the compressed bundle to the offline machines that are the source for offline plugin installation, then use the
|
||||
`bin/logstash-plugin unpack` subcommand to make the packaged plugins available.
|
||||
|
||||
[float]
|
||||
=== Install or Update a local plugin (Deprecated Procedure)
|
||||
|
||||
deprecated[5.2]
|
||||
|
||||
To install or update a local plugin, use the `--local` option with the install and update commands, as in the following
|
||||
examples:
|
||||
|
||||
.Installing a local plugin
|
||||
============
|
||||
`bin/logstash-plugin install --local logstash-input-jdbc`
|
||||
============
|
||||
|
||||
.Updating a local plugin
|
||||
============
|
||||
`bin/logstash-plugin update --local logstash-input-jdbc`
|
||||
============
|
||||
|
||||
.Updating all local plugins in one command
|
||||
============
|
||||
`bin/logstash-plugin update --local`
|
||||
============
|
||||
|
||||
[float]
|
||||
[[managing-packs]]
|
||||
=== Managing Plugin Packs
|
||||
|
||||
deprecated[5.2]
|
||||
|
||||
The `pack` and `unpack` subcommands for `bin/logstash-plugin` take the following options:
|
||||
|
||||
[horizontal]
|
||||
`--tgz`:: Generate the offline package as a GZipped TAR file. The default behavior on UNIX systems.
|
||||
`--zip`:: Generate the offline package as a ZIP file. The default behavior on Windows systems.
|
||||
`[packname] --override`:: Generates a new offline package that overwrites an existing offline with the specified name.
|
||||
`[packname] --[no-]clean`: Deletes offline packages matching the specified name.
|
||||
14
docs/jp/static/output.asciidoc
Normal file
14
docs/jp/static/output.asciidoc
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
:register_method: true
|
||||
:multi_receive_method: true
|
||||
:plugintype: output
|
||||
:pluginclass: Outputs
|
||||
:pluginname: example
|
||||
:pluginnamecap: Example
|
||||
|
||||
:blockfilter: true
|
||||
|
||||
:getstarted: Let's step through creating an {plugintype} plugin using the https://github.com/logstash-plugins/logstash-output-example/[example {plugintype} plugin].
|
||||
|
||||
:methodheader: Logstash outputs must implement the `register` and `multi_receive` methods.
|
||||
|
||||
include::include/pluginbody.asciidoc[]
|
||||
103
docs/jp/static/performance-checklist.asciidoc
Normal file
103
docs/jp/static/performance-checklist.asciidoc
Normal file
|
|
@ -0,0 +1,103 @@
|
|||
[[performance-tuning]]
|
||||
== Performance Tuning
|
||||
|
||||
This section includes the following information about tuning Logstash
|
||||
performance:
|
||||
|
||||
* <<performance-troubleshooting>>
|
||||
* <<tuning-logstash>>
|
||||
|
||||
[[performance-troubleshooting]]
|
||||
=== Performance Troubleshooting Guide
|
||||
|
||||
You can use this troubleshooting guide to quickly diagnose and resolve Logstash performance problems. Advanced knowledge of pipeline internals is not required to understand this guide. However, the <<pipeline,pipeline documentation>> is recommended reading if you want to go beyond this guide.
|
||||
|
||||
You may be tempted to jump ahead and change settings like `pipeline.workers` (`-w`) as a first attempt to improve performance. In our experience, changing this setting makes it more difficult to troubleshoot performance problems because you increase the number of variables in play. Instead, make one change at a time and measure the results. Starting at the end of this list is a sure-fire way to create a confusing situation.
|
||||
|
||||
[float]
|
||||
==== Performance Checklist
|
||||
|
||||
. *Check the performance of input sources and output destinations:*
|
||||
+
|
||||
* Logstash is only as fast as the services it connects to. Logstash can only consume and produce data as fast as its input and output destinations can!
|
||||
|
||||
. *Check system statistics:*
|
||||
+
|
||||
* CPU
|
||||
** Note whether the CPU is being heavily used. On Linux/Unix, you can run `top -H` to see process statistics broken out by thread, as well as total CPU statistics.
|
||||
** If CPU usage is high, skip forward to the section about checking the JVM heap and then read the section about tuning Logstash worker settings.
|
||||
* Memory
|
||||
** Be aware of the fact that Logstash runs on the Java VM. This means that Logstash will always use the maximum amount of memory you allocate to it.
|
||||
** Look for other applications that use large amounts of memory and may be causing Logstash to swap to disk. This can happen if the total memory used by applications exceeds physical memory.
|
||||
* I/O Utilization
|
||||
** Monitor disk I/O to check for disk saturation.
|
||||
*** Disk saturation can happen if you’re using Logstash plugins (such as the file output) that may saturate your storage.
|
||||
*** Disk saturation can also happen if you're encountering a lot of errors that force Logstash to generate large error logs.
|
||||
*** On Linux, you can use iostat, dstat, or something similar to monitor disk I/O.
|
||||
** Monitor network I/O for network saturation.
|
||||
*** Network saturation can happen if you’re using inputs/outputs that perform a lot of network operations.
|
||||
*** On Linux, you can use a tool like dstat or iftop to monitor your network.
|
||||
|
||||
. *Check the JVM heap:*
|
||||
+
|
||||
* Often times CPU utilization can go through the roof if the heap size is too low, resulting in the JVM constantly garbage collecting.
|
||||
* A quick way to check for this issue is to double the heap size and see if performance improves. Do not increase the heap size past the amount of physical memory. Leave at least 1GB free for the OS and other processes.
|
||||
* You can make more accurate measurements of the JVM heap by using either the `jmap` command line utility distributed with Java or by using VisualVM. For more info, see <<profiling-the-heap>>.
|
||||
|
||||
. *Tune Logstash worker settings:*
|
||||
+
|
||||
* Begin by scaling up the number of pipeline workers by using the `-w` flag. This will increase the number of threads available for filters and outputs. It is safe to scale this up to a multiple of CPU cores, if need be, as the threads can become idle on I/O.
|
||||
* Each output can only be active in a single pipeline worker thread by default. You can increase this by changing the `workers` setting in the configuration block for each output. Never make this value larger than the number of pipeline workers.
|
||||
* You may also tune the output batch size. For many outputs, such as the Elasticsearch output, this setting will correspond to the size of I/O operations. In the case of the Elasticsearch output, this setting corresponds to the batch size.
|
||||
|
||||
[[tuning-logstash]]
|
||||
=== Tuning and Profiling Logstash Performance
|
||||
|
||||
The Logstash defaults are chosen to provide fast, safe performance for most
|
||||
users. However if you notice performance issues, you may need to modify
|
||||
some of the defaults. Logstash provides the following configurable options
|
||||
for tuning pipeline performance: `pipeline.workers`, `pipeline.batch.size`, and `pipeline.batch.delay`. For more information about setting these options, see <<logstash-settings-file>>.
|
||||
|
||||
Make sure you've read the <<performance-troubleshooting>> before modifying these options.
|
||||
|
||||
* The `pipeline.workers` setting determines how many threads to run for filter and output processing. If you find that events are backing up, or that the CPU is not saturated, consider increasing the value of this parameter to make better use of available processing power. Good results can even be found increasing this number past the number of available processors as these threads may spend significant time in an I/O wait state when writing to external systems. Legal values for this parameter are positive integers.
|
||||
|
||||
* The `pipeline.batch.size` setting defines the maximum number of events an individual worker thread collects before attempting to execute filters and outputs. Larger batch sizes are generally more efficient, but increase memory overhead. Some hardware configurations require you to increase JVM heap size by setting the `LS_HEAP_SIZE` variable to avoid performance degradation with this option. Values of this parameter in excess of the optimum range cause performance degradation due to frequent garbage collection or JVM crashes related to out-of-memory exceptions. Output plugins can process each batch as a logical unit. The Elasticsearch output, for example, issues https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html[bulk requests] for each batch received. Tuning the `pipeline.batch.size` setting adjusts the size of bulk requests sent to Elasticsearch.
|
||||
|
||||
* The `pipeline.batch.delay` setting rarely needs to be tuned. This setting adjusts the latency of the Logstash pipeline. Pipeline batch delay is the maximum amount of time in milliseconds that Logstash waits for new messages after receiving an event in the current pipeline worker thread. After this time elapses, Logstash begins to execute filters and outputs.The maximum time that Logstash waits between receiving an event and processing that event in a filter is the product of the `pipeline.batch.delay` and `pipeline.batch.size` settings.
|
||||
|
||||
[float]
|
||||
==== Notes on Pipeline Configuration and Performance
|
||||
|
||||
If you plan to modify the default pipeline settings, take into account the
|
||||
following suggestions:
|
||||
|
||||
* The total number of inflight events is determined by the product of the `pipeline.workers` and `pipeline.batch.size` settings. This product is referred to as the _inflight count_. Keep the value of the inflight count in mind as you adjust the `pipeline.workers` and `pipeline.batch.size` settings. Pipelines that intermittently receive large events at irregular intervals require sufficient memory to handle these spikes. Configure the `LS_HEAP_SIZE` variable accordingly.
|
||||
|
||||
* Measure each change to make sure it increases, rather than decreases, performance.
|
||||
|
||||
* Ensure that you leave enough memory available to cope with a sudden increase in event size. For example, an application that generates exceptions that are represented as large blobs of text.
|
||||
|
||||
* The number of workers may be set higher than the number of CPU cores since outputs often spend idle time in I/O wait conditions.
|
||||
|
||||
* Threads in Java have names and you can use the `jstack`, `top`, and the VisualVM graphical tools to figure out which resources a given thread uses.
|
||||
|
||||
* On Linux platforms, Logstash labels all the threads it can with something descriptive. For example, inputs show up as `[base]<inputname`, filter/output workers show up as `[base]>workerN`, where N is an integer. Where possible, other threads are also labeled to help you identify their purpose.
|
||||
|
||||
[float]
|
||||
[[profiling-the-heap]]
|
||||
==== Profiling the Heap
|
||||
|
||||
When tuning Logstash you may have to adjust the heap size. You can use the https://visualvm.java.net/[VisualVM] tool to profile the heap. The *Monitor* pane in particular is useful for checking whether your heap allocation is sufficient for the current workload. The screenshots below show sample *Monitor* panes. The first pane examines a Logstash instance configured with too many inflight events. The second pane examines a Logstash instance configured with an appropriate amount of inflight events. Note that the specific batch sizes used here are most likely not applicable to your specific workload, as the memory demands of Logstash vary in large part based on the type of messages you are sending.
|
||||
|
||||
image::static/images/pipeline_overload.png[]
|
||||
|
||||
image::static/images/pipeline_correct_load.png[]
|
||||
|
||||
In the first example we see that the CPU isn’t being used very efficiently. In fact, the JVM is often times having to stop the VM for “full GCs”. Full garbage collections are a common symptom of excessive memory pressure. This is visible in the spiky pattern on the CPU chart. In the more efficiently configured example, the GC graph pattern is more smooth, and the CPU is used in a more uniform manner. You can also see that there is ample headroom between the allocated heap size, and the maximum allowed, giving the JVM GC a lot of room to work with.
|
||||
|
||||
Examining the in-depth GC statistics with a tool similar to the excellent https://visualvm.java.net/plugins.html[VisualGC] plugin shows that the over-allocated VM spends very little time in the efficient Eden GC, compared to the time spent in the more resource-intensive Old Gen “Full” GCs.
|
||||
|
||||
NOTE: As long as the GC pattern is acceptable, heap sizes that occasionally increase to the maximum are acceptable. Such heap size spikes happen in response to a burst of large events passing through the pipeline. In general practice, maintain a gap between the used amount of heap memory and the maximum.
|
||||
This document is not a comprehensive guide to JVM GC tuning. Read the official http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html[Oracle guide] for more information on the topic. We also recommend reading http://www.semicomplete.com/blog/geekery/debugging-java-performance.html[Debugging Java Performance].
|
||||
|
||||
166
docs/jp/static/persistent-queues.asciidoc
Normal file
166
docs/jp/static/persistent-queues.asciidoc
Normal file
|
|
@ -0,0 +1,166 @@
|
|||
[[persistent-queues]]
|
||||
=== Persistent Queues
|
||||
|
||||
WARNING: This functionality is in beta and is subject to change. Deployment in production is at your own risk.
|
||||
|
||||
By default, Logstash uses in-memory bounded queues between pipeline stages
|
||||
(inputs → pipeline workers) to buffer events. The size of these in-memory
|
||||
queues is fixed and not configurable. If Logstash experiences a temporary
|
||||
machine failure, the contents of the in-memory queue will be lost. Temporary machine
|
||||
failures are scenarios where Logstash or its host machine are terminated
|
||||
abnormally but are capable of being restarted.
|
||||
|
||||
In order to protect against data loss during abnormal termination, Logstash has
|
||||
a persistent queue feature which will store the message queue on disk.
|
||||
Persistent queues provide durability of data within Logstash.
|
||||
|
||||
Persistent queues are also useful for Logstash deployments that need large buffers.
|
||||
Instead of deploying and managing a message broker, such as Redis, RabbitMQ, or
|
||||
Apache Kafka, to facilitate a buffered publish-subscriber model, you can enable
|
||||
persistent queues to buffer events on disk and remove the message broker.
|
||||
|
||||
In summary, the benefits of enabling persistent queues are as follows:
|
||||
|
||||
* Absorbs bursts of events without needing an external buffering mechanism like
|
||||
Redis or Apache Kafka.
|
||||
* Provides an at-least-once delivery guarantee against message loss during
|
||||
a normal shutdown as well as when Logstash is terminated abnormally. If Logstash
|
||||
is restarted while events are in-flight, Logstash will attempt to deliver
|
||||
messages stored in the persistent queue until delivery succeeds at least once.
|
||||
|
||||
[[persistent-queues-limitations]]
|
||||
==== Limitations of Persistent Queues
|
||||
|
||||
The following are problems not solved by the persistent queue feature:
|
||||
|
||||
* Input plugins that do not use a request-response protocol cannot be protected from data loss. For example: tcp, udp, zeromq push+pull, and many other inputs do not have a mechanism to acknowledge receipt to the sender. Plugins such as beats and http, which *do* have an acknowledgement capability, are well protected by this queue.
|
||||
* It does not handle permanent machine failures such as disk corruption, disk failure, and machine loss. The data persisted to disk is not replicated.
|
||||
|
||||
[[persistent-queues-architecture]]
|
||||
==== How Persistent Queues Work
|
||||
|
||||
The queue sits between the input and filter stages in the same
|
||||
process:
|
||||
|
||||
input → queue → filter + output
|
||||
|
||||
When an input has events ready to process, it writes them to the queue. When
|
||||
the write to the queue is successful, the input can send an acknowledgement to
|
||||
its data source.
|
||||
|
||||
When processing events from the queue, Logstash acknowledges events as
|
||||
completed, within the queue, only after filters and outputs have completed.
|
||||
The queue keeps a record of events that have been processed by the pipeline.
|
||||
An event is recorded as processed (in this document, called "acknowledged" or
|
||||
"ACKed") if, and only if, the event has been processed completely by the
|
||||
Logstash pipeline.
|
||||
|
||||
What does acknowledged mean? This means the event has been handled by all
|
||||
configured filters and outputs. For example, if you have only one output,
|
||||
Elasticsearch, an event is ACKed when the Elasticsearch output has successfully
|
||||
sent this event to Elasticsearch.
|
||||
|
||||
During a normal shutdown (*CTRL+C* or SIGTERM), Logstash will stop reading
|
||||
from the queue and will finish processing the in-flight events being processed
|
||||
by the filters and outputs. Upon restart, Logstash will resume processing the
|
||||
events in the persistent queue as well as accepting new events from inputs.
|
||||
|
||||
If Logstash is abnormally terminated, any in-flight events will not have been
|
||||
ACKed and will be reprocessed by filters and outputs when Logstash is
|
||||
restarted. Logstash processes events in batches, so it is possible
|
||||
that for any given batch, some of that batch may have been successfully
|
||||
completed, but not recorded as ACKed, when an abnormal termination occurs.
|
||||
|
||||
For more details specific behaviors of queue writes and acknowledgement, see
|
||||
<<durability-persistent-queues>>.
|
||||
|
||||
[[configuring-persistent-queues]]
|
||||
==== Configuring Persistent Queues
|
||||
|
||||
To configure persistent queues, you can specify the following options in the
|
||||
Logstash <<logstash-settings-file,settings file>>:
|
||||
|
||||
* `queue.type`: Specify `persisted` to enable persistent queues. By default, persistent queues are disabled (default: `queue.type: memory`).
|
||||
* `path.queue`: The directory path where the data files will be stored. By default, the files are stored in `path.data/queue`.
|
||||
* `queue.page_capacity`: The maximum size of a queue page in bytes. The queue data consists of append-only files called "pages". The default size is 250mb. Changing this value is unlikely to have performance benefits.
|
||||
// Technically, I know, this isn't "maximum number of events" it's really maximum number of events not yet read by the pipeline worker. We only use this for testing and users generally shouldn't be setting this.
|
||||
* `queue.max_events`: The maximum number of events that are allowed in the queue. The default is 0 (unlimited). This value is used internally for the Logstash test suite.
|
||||
* `queue.max_bytes`: The total capacity of the queue in number of bytes. The
|
||||
default is 1024mb (1gb). Make sure the capacity of your disk drive is greater
|
||||
than the value you specify here.
|
||||
|
||||
If both `queue.max_events` and
|
||||
`queue.max_bytes` are specified, Logstash uses whichever criteria is reached
|
||||
first. See <<backpressure-persistent-queue>> for behavior when these queue limits are reached.
|
||||
|
||||
You can also specify options that control when the checkpoint file gets updated (`queue.checkpoint.acks`, `queue.checkpoint.writes`). See <<durability-persistent-queues>>.
|
||||
|
||||
Example configuration:
|
||||
|
||||
[source, yaml]
|
||||
queue.type: persisted
|
||||
queue.max_bytes: 4gb
|
||||
|
||||
[[backpressure-persistent-queue]]
|
||||
==== Handling Back Pressure
|
||||
|
||||
When the queue is full, Logstash puts back pressure on the inputs to stall data
|
||||
flowing into Logstash. This mechanism helps Logstash control the rate of data
|
||||
flow at the input stage without overwhelming outputs like Elasticsearch.
|
||||
|
||||
Use `queue.max_bytes` setting to configure the total capacity of the queue on
|
||||
disk. The following example sets the total capacity of the queue to 8gb:
|
||||
|
||||
[source, yaml]
|
||||
queue.type: persisted
|
||||
queue.max_bytes: 8gb
|
||||
|
||||
With these settings specified, Logstash will buffer events on disk until the
|
||||
size of the queue reaches 8gb. When the queue is full of unACKed events, and
|
||||
the size limit has been reached, Logstash will no longer accept new events.
|
||||
|
||||
Each input handles back pressure independently. For example, when the
|
||||
<<plugins-inputs-beats,beats>> input encounters back pressure, it no longer
|
||||
accepts new connections and waits until the persistent queue has space to accept
|
||||
more events. After the filter and output stages finish processing existing
|
||||
events in the queue and ACKs them, Logstash automatically starts accepting new
|
||||
events.
|
||||
|
||||
[[durability-persistent-queues]]
|
||||
==== Controlling Durability
|
||||
|
||||
Durability is a property of storage writes that ensures data will be available after it's written.
|
||||
|
||||
When the persistent queue feature is enabled, Logstash will store events on
|
||||
disk. Logstash commits to disk in a mechanism called checkpointing.
|
||||
|
||||
To discuss durability, we need to introduce a few details about how the persistent queue is implemented.
|
||||
|
||||
First, the queue itself is a set of pages. There are two kinds of pages: head pages and tail pages. The head page is where new events are written. There is only one head page. When the head page is of a certain size (see `queue.page_capacity`), it becomes a tail page, and a new head page is created. Tail pages are immutable, and the head page is append-only.
|
||||
Second, the queue records details about itself (pages, acknowledgements, etc) in a separate file called a checkpoint file.
|
||||
|
||||
When recording a checkpoint, Logstash will:
|
||||
|
||||
* Call fsync on the head page.
|
||||
* Atomically write to disk the current state of the queue.
|
||||
|
||||
The following settings are available to let you tune durability:
|
||||
|
||||
* `queue.checkpoint.writes`: Logstash will checkpoint after this many writes into the queue. Currently, one event counts as one write, but this may change in future releases.
|
||||
* `queue.checkpoint.acks`: Logstash will checkpoint after this many events are acknowledged. This configuration controls the durability at the processing (filter + output)
|
||||
part of Logstash.
|
||||
|
||||
Disk writes have a resource cost. Tuning the above values higher or lower will trade durability for performance. For instance, if you want to the strongest durability for all input events, you can set `queue.checkpoint.writes: 1`.
|
||||
|
||||
The process of checkpointing is atomic, which means any update to the file is saved if successful.
|
||||
|
||||
If Logstash is terminated, or if there is a hardware level failure, any data
|
||||
that is buffered in the persistent queue, but not yet checkpointed, is lost.
|
||||
To avoid this possibility, you can set `queue.checkpoint.writes: 1`, but keep in
|
||||
mind that this setting can severely impact performance.
|
||||
|
||||
[[garbage-collection]]
|
||||
==== Disk Garbage Collection
|
||||
|
||||
On disk, the queue is stored as a set of pages where each page is one file. Each page can be at most `queue.page_capacity` in size. Pages are deleted (garbage collected) after all events in that page have been ACKed. If an older page has at least one event that is not yet ACKed, that entire page will remain on disk until all events in that page are successfully processed. Each page containing unprocessed events will count against the `queue.max_bytes` byte size.
|
||||
|
||||
19
docs/jp/static/plugin-generator.asciidoc
Normal file
19
docs/jp/static/plugin-generator.asciidoc
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
[[plugin-generator]]
|
||||
=== Generating Plugins
|
||||
|
||||
You can now create your own Logstash plugin in seconds! The generate subcommand of `bin/logstash-plugin` creates the foundation
|
||||
for a new Logstash plugin with templatized files. It creates the correct directory structure, gemspec files, and dependencies so you
|
||||
can start adding custom code to process data with Logstash.
|
||||
|
||||
**Example Usage**
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
bin/logstash-plugin generate --type input --name xkcd --path ~/ws/elastic/plugins
|
||||
-------------------------------------------
|
||||
|
||||
* `--type`: Type of plugin - input, filter, output, or codec
|
||||
* `--name`: Name for the new plugin
|
||||
* `--path`: Directory path where the new plugin structure will be created. If not specified, it will be
|
||||
created in the current directory.
|
||||
|
||||
135
docs/jp/static/plugin-manager.asciidoc
Normal file
135
docs/jp/static/plugin-manager.asciidoc
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
[[working-with-plugins]]
|
||||
== Working with plugins
|
||||
|
||||
Logstash has a rich collection of input, filter, codec and output plugins. Plugins are available as self-contained
|
||||
packages called gems and hosted on RubyGems.org. The plugin manager accessed via `bin/logstash-plugin` script is used to manage the
|
||||
lifecycle of plugins in your Logstash deployment. You can install, remove and upgrade plugins using the Command Line
|
||||
Interface (CLI) invocations described below.
|
||||
|
||||
[[http-proxy]]
|
||||
=== Proxy configuration
|
||||
|
||||
The majority of the plugin manager commands require access to the internet to reach https://rubygems.org[RubyGems.org].
|
||||
If your organization is behind a firewall you can set these environments variables to configure Logstash to use your proxy.
|
||||
|
||||
[source, shell]
|
||||
----------------------------------
|
||||
export http_proxy=http://localhost:3128
|
||||
export https_proxy=http://localhost:3128
|
||||
----------------------------------
|
||||
|
||||
[float]
|
||||
[[listing-plugins]]
|
||||
=== Listing plugins
|
||||
|
||||
Logstash release packages bundle common plugins so you can use them out of the box. To list the plugins currently
|
||||
available in your deployment:
|
||||
|
||||
[source,shell]
|
||||
----------------------------------
|
||||
bin/logstash-plugin list <1>
|
||||
bin/logstash-plugin list --verbose <2>
|
||||
bin/logstash-plugin list '*namefragment*' <3>
|
||||
bin/logstash-plugin list --group output <4>
|
||||
----------------------------------
|
||||
<1> Will list all installed plugins
|
||||
|
||||
<2> Will list installed plugins with version information
|
||||
|
||||
<3> Will list all installed plugins containing a namefragment
|
||||
|
||||
<4> Will list all installed plugins for a particular group (input, filter, codec, output)
|
||||
|
||||
[float]
|
||||
[[installing-plugins]]
|
||||
=== Adding plugins to your deployment
|
||||
|
||||
The most common situation when dealing with plugin installation is when you have access to internet. Using this method,
|
||||
you will be able to retrieve plugins hosted on the public repository (RubyGems.org) and install on top of your Logstash
|
||||
installation.
|
||||
|
||||
[source,shell]
|
||||
----------------------------------
|
||||
bin/logstash-plugin install logstash-output-kafka
|
||||
----------------------------------
|
||||
|
||||
Once the plugin is successfully installed, you can start using it in your configuration file.
|
||||
|
||||
[[installing-local-plugins]]
|
||||
[float]
|
||||
==== Advanced: Adding a locally built plugin
|
||||
|
||||
In some cases, you want to install plugins which have not yet been released and not hosted on RubyGems.org. Logstash
|
||||
provides you the option to install a locally built plugin which is packaged as a ruby gem. Using a file location:
|
||||
|
||||
[source,shell]
|
||||
----------------------------------
|
||||
bin/logstash-plugin install /path/to/logstash-output-kafka-1.0.0.gem
|
||||
----------------------------------
|
||||
|
||||
[[installing-local-plugins-path]]
|
||||
[float]
|
||||
==== Advanced: Using `--path.plugins`
|
||||
|
||||
Using the Logstash `--path.plugins` flag, you can load a plugin source code located on your file system. Typically this is used by
|
||||
developers who are iterating on a custom plugin and want to test it before creating a ruby gem.
|
||||
|
||||
The path needs to be in a specific directory hierarchy: `PATH/logstash/TYPE/NAME.rb`, where TYPE is 'inputs' 'filters', 'outputs' or 'codecs' and NAME is the name of the plugin.
|
||||
|
||||
[source,shell]
|
||||
----------------------------------
|
||||
# supposing the code is in /opt/shared/lib/logstash/inputs/my-custom-plugin-code.rb
|
||||
bin/logstash --path.plugins /opt/shared/lib
|
||||
----------------------------------
|
||||
|
||||
[[updating-plugins]]
|
||||
[float]
|
||||
=== Updating plugins
|
||||
|
||||
Plugins have their own release cycle and are often released independent of Logstash’s core release cycle. Using the update
|
||||
subcommand you can get the latest or update to a particular version of the plugin.
|
||||
|
||||
[source,shell]
|
||||
----------------------------------
|
||||
bin/logstash-plugin update <1>
|
||||
bin/logstash-plugin update logstash-output-kafka <2>
|
||||
----------------------------------
|
||||
<1> will update all installed plugins
|
||||
|
||||
<2> will update only this plugin
|
||||
|
||||
[[removing-plugins]]
|
||||
[float]
|
||||
=== Removing plugins
|
||||
|
||||
If you need to remove plugins from your Logstash installation:
|
||||
|
||||
[source,shell]
|
||||
----------------------------------
|
||||
bin/logstash-plugin remove logstash-output-kafka
|
||||
----------------------------------
|
||||
|
||||
[[proxy-plugins]]
|
||||
[float]
|
||||
=== Proxy Support
|
||||
|
||||
The previous sections relied on Logstash being able to communicate with RubyGems.org. In certain environments, Forwarding
|
||||
Proxy is used to handle HTTP requests. Logstash Plugins can be installed and updated through a Proxy by setting the
|
||||
`HTTP_PROXY` environment variable:
|
||||
|
||||
[source,shell]
|
||||
----------------------------------
|
||||
export HTTP_PROXY=http://127.0.0.1:3128
|
||||
|
||||
bin/logstash-plugin install logstash-output-kafka
|
||||
----------------------------------
|
||||
|
||||
Once set, plugin commands install, update can be used through this proxy.
|
||||
|
||||
include::plugin-generator.asciidoc[]
|
||||
|
||||
include::offline-plugins.asciidoc[]
|
||||
|
||||
include::private-gem-repo.asciidoc[]
|
||||
|
||||
include::event-api.asciidoc[]
|
||||
53
docs/jp/static/private-gem-repo.asciidoc
Normal file
53
docs/jp/static/private-gem-repo.asciidoc
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
[[private-rubygem]]
|
||||
=== Private Gem Repositories
|
||||
|
||||
The Logstash plugin manager connects to a Ruby gems repository to install and update Logstash plugins. By default, this
|
||||
repository is http://rubygems.org.
|
||||
|
||||
Some use cases are unable to use the default repository, as in the following examples:
|
||||
|
||||
* A firewall blocks access to the default repository.
|
||||
* You are developing your own plugins locally.
|
||||
* Airgap requirements on the local system.
|
||||
|
||||
When you use a custom gem repository, be sure to make plugin dependencies available.
|
||||
|
||||
Several open source projects enable you to run your own plugin server, among them:
|
||||
|
||||
* https://github.com/geminabox/geminabox[Geminabox]
|
||||
* https://github.com/PierreRambaud/gemirro[Gemirro]
|
||||
* https://gemfury.com/[Gemfury]
|
||||
* http://www.jfrog.com/open-source/[Artifactory]
|
||||
|
||||
==== Editing the Gemfile
|
||||
|
||||
The gemfile is a configuration file that specifies information required for plugin management. Each gem file has a
|
||||
`source` line that specifies a location for plugin content.
|
||||
|
||||
By default, the gemfile's `source` line reads:
|
||||
|
||||
[source,shell]
|
||||
----------
|
||||
# This is a Logstash generated Gemfile.
|
||||
# If you modify this file manually all comments and formatting will be lost.
|
||||
|
||||
source "https://rubygems.org"
|
||||
----------
|
||||
|
||||
To change the source, edit the `source` line to contain your preferred source, as in the following example:
|
||||
|
||||
[source,shell]
|
||||
----------
|
||||
# This is a Logstash generated Gemfile.
|
||||
# If you modify this file manually all comments and formatting will be lost.
|
||||
|
||||
source "https://my.private.repository"
|
||||
----------
|
||||
|
||||
After saving the new version of the gemfile, use <<working-with-plugins,plugin management commands>> normally.
|
||||
|
||||
The following links contain further material on setting up some commonly used repositories:
|
||||
|
||||
* https://github.com/geminabox/geminabox/blob/master/README.markdown[Geminabox]
|
||||
* https://www.jfrog.com/confluence/display/RTF/RubyGems+Repositories[Artifactory]
|
||||
* Running a http://guides.rubygems.org/run-your-own-gem-server/[rubygems mirror]
|
||||
46
docs/jp/static/reloading-config.asciidoc
Normal file
46
docs/jp/static/reloading-config.asciidoc
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
[[reloading-config]]
|
||||
=== Reloading the Config File
|
||||
|
||||
Starting with Logstash 2.3, you can set Logstash to detect and reload configuration
|
||||
changes automatically.
|
||||
|
||||
To enable automatic config reloading, start Logstash with the `--config.reload.automatic` (or `-r`)
|
||||
command-line option specified. For example:
|
||||
|
||||
[source,shell]
|
||||
----------------------------------
|
||||
bin/logstash –f apache.config --config.reload.automatic
|
||||
----------------------------------
|
||||
|
||||
NOTE: The `--config.reload.automatic` option is not available when you specify the `-e` flag to pass
|
||||
in configuration settings from the command-line.
|
||||
|
||||
By default, Logstash checks for configuration changes every 3 seconds. To change this interval,
|
||||
use the `--config.reload.interval <seconds>` option, where `seconds` specifies how often Logstash
|
||||
checks the config files for changes.
|
||||
|
||||
If Logstash is already running without auto-reload enabled, you can force Logstash to
|
||||
reload the config file and restart the pipeline by sending a SIGHUP (signal hangup) to the
|
||||
process running Logstash. For example:
|
||||
|
||||
[source,shell]
|
||||
----------------------------------
|
||||
kill -1 14175
|
||||
----------------------------------
|
||||
|
||||
Where 14175 is the ID of the process running Logstash.
|
||||
|
||||
==== How Automatic Config Reloading Works
|
||||
|
||||
When Logstash detects a change in a config file, it stops the current pipeline by stopping
|
||||
all inputs, and it attempts to create a new pipeline that uses the updated configuration.
|
||||
After validating the syntax of the new configuration, Logstash verifies that all inputs
|
||||
and outputs can be initialized (for example, that all required ports are open). If the checks
|
||||
are successful, Logstash swaps the existing pipeline with the new pipeline. If the checks
|
||||
fail, the old pipeline continues to function, and the errors are propagated to the console.
|
||||
|
||||
During automatic config reloading, the JVM is not restarted. The creating and swapping of
|
||||
pipelines all happens within the same process.
|
||||
|
||||
Changes to <<plugins-filters-grok,grok>> pattern files are also reloaded, but only when
|
||||
a change in the config file triggers a reload (or the pipeline is restarted).
|
||||
183
docs/jp/static/roadmap/index.asciidoc
Normal file
183
docs/jp/static/roadmap/index.asciidoc
Normal file
|
|
@ -0,0 +1,183 @@
|
|||
= Logstash Roadmap
|
||||
|
||||
:ISSUES: https://github.com/elastic/logstash/issues/
|
||||
:LABELS: https://github.com/elastic/logstash/labels/
|
||||
|
||||
== Overview
|
||||
|
||||
Welcome to the Logstash roadmap page!
|
||||
|
||||
While GitHub is great for sharing our work, it can be difficult to get an
|
||||
overview of the current state of affairs from an issues list. This page outlines
|
||||
major themes for our future plans, with pointers to additional resources if you
|
||||
want to contribute to the Logstash project.
|
||||
|
||||
We will not track concrete milestones on this page, because we often make
|
||||
adjustments to our timelines based on community feedback. For the latest release
|
||||
status information, please search for the {LABELS}roadmap[roadmap] tag in
|
||||
GitHub.
|
||||
|
||||
== Resiliency
|
||||
[float]
|
||||
=== status: ongoing; v2.x
|
||||
|
||||
The Logstash team is committed to continuously improving the resiliency of
|
||||
Logstash. As with any modular system, Logstash has many moving parts and a
|
||||
multitude of deployment architectures, all of which need to be considered in the
|
||||
context of resiliency. Our resiliency project is an ongoing effort to identify
|
||||
and enhance areas where Logstash can provide additional resiliency guarantees.
|
||||
You can follow this effort on GitHub by searching for issues that have the
|
||||
{LABELS}resiliency[resiliency] tag.
|
||||
|
||||
*Event persistence ({ISSUES}2605[#2605]).* Logstash relies on bounded in-memory
|
||||
queues between pipeline stages to buffer events (see the
|
||||
http://www.elastic.co/guide/en/logstash/current/pipeline.html#_fault_tolerance[documentation]
|
||||
for more information). Currently, these queues are not persisted to disk.
|
||||
To prevent loss in the event of a plugin crash or a restart, we plan to persist
|
||||
these queues to disk.
|
||||
|
||||
*Variable internal queues ({ISSUES}2606[#2606]).* Logstash currently uses
|
||||
fixed-sized queues between pipeline stages. When the processing rates differ
|
||||
widely between stages (such as parsing and indexing), users typically deploy a
|
||||
message broker, such as Redis or RabbitMQ, to provide an external queueing
|
||||
mechanism. We plan to offer a built-in alternative to using an external message
|
||||
broker by adding a variable queueing option to Logstash.
|
||||
|
||||
*Dead letter queue (https://github.com/elastic/logstash/issues/2607[#2607]).*
|
||||
Today, when Logstash cannot process an event due to an error, it has two
|
||||
choices: drop or retry. If the condition is temporary (for example, the next
|
||||
stage in the pipeline is temporarily overloaded), retry is a good approach.
|
||||
However, if the failure is permanent (such as bad encoding or a mapping error)
|
||||
retrying could cause an indefinite stall in processing. In this case, dropping
|
||||
the event is preferred. As a third option, we plan to introduce a dead letter
|
||||
queue (DLQ), which will store events that could stall the pipeline. Users can
|
||||
then examine these events and resolve problems as needed. The DLQ could also
|
||||
receive events that abuse the grok filter (e.g. runaway regular expressions
|
||||
which cause expensive backtracking), failures in grok patterns, date filters,
|
||||
and so on.
|
||||
|
||||
*End-to-end acknowledgement of message delivery ({ISSUES}2609[#2609]).* Logstash
|
||||
currently does not provide end-to-end delivery guarantees. When a plugin fails
|
||||
to process an event, it does not signal to an earlier stage in the pipeline that
|
||||
an error has occurred. In the longer term, we plan to introduce an optional
|
||||
notification mechanism to give operators an easier way to track and replay
|
||||
failed events.
|
||||
|
||||
*Known issues affecting resiliency.* There are certain categories of defects
|
||||
that affect resiliency, such as plugin crashes, failures in retry logic, and
|
||||
exhausting system resources. We respond to critical bug requests in real-time
|
||||
and perform weekly triaging of less urgent requests. All known issues are
|
||||
flagged with the
|
||||
https://github.com/elastic/logstash/labels/resiliency[resiliency] tag.
|
||||
|
||||
*Known unknowns.* If we don’t know it’s happening, it’s hard for us to fix it!
|
||||
Please report your issues in GitHub, under the
|
||||
https://github.com/elastic/logstash/issues[Logstash] or
|
||||
individual https://github.com/logstash-plugins/[Logstash plugin] repositories.
|
||||
|
||||
== Manageability
|
||||
[float]
|
||||
=== status: ongoing; v2.x
|
||||
|
||||
As Logstash deployments scale up, managing and monitoring multiple Logstash
|
||||
instances using configuration and log files can become challenging. Our
|
||||
manageability project aims to improve this experience by adding functionality
|
||||
that makes administration of Logstash more efficient and less error-prone. You
|
||||
can follow this effort on GitHub by searching for issues that have the
|
||||
{LABELS}manageability[manageability] tag.
|
||||
|
||||
*Better Defaults.* Today, some Logstash defaults are geared toward the development experience, rather than production environments. We plan to audit and re-evaluate a number of defaults to alleviate the burden of tuning Logstash performance in production ({ISSUES}1512[#1512]). In addition, we are undertaking additional benchmarking to evaluate the performance of node, transport, and HTTP protocols in the Elasticsearch output to provide additional confirmation for our proposal to switch the default from node to HTTP (https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/150[#150]).
|
||||
|
||||
*Logstash Monitoring API ({ISSUES}2611[#2611]).* Today, most Logstash monitoring
|
||||
functions are accomplished by tailing logs or outputting debug messages. As a
|
||||
result, it is hard to monitor the Logstash health and track success or failure
|
||||
of events passing through the pipeline. We plan to introduce a Logstash
|
||||
monitoring API to improve visibility into pipeline activity and provide
|
||||
performance metrics such as number of events processed, success/failure rates,
|
||||
and time spent in each plugin.
|
||||
|
||||
*Logstash Management API ({ISSUES}2612[#2612]).* Currently, updating the
|
||||
Logstash configuration requires editing a configuration file and restarting
|
||||
the Logstash process. This means you either have to temporarily halt the
|
||||
pipeline or accept an interruption in processing. While file-based configuration
|
||||
management will continue to be supported, we plan to add a robust Logstash
|
||||
management API that enables you to update the configuration dynamically without
|
||||
restarting the Logstash process. As the API matures, it will provide us with a
|
||||
strong foundation for building a user interface for monitoring and managing
|
||||
Logstash.
|
||||
|
||||
*Clustering ({ISSUES}2632[#2632]).* In large-scale Logstash deployments, users
|
||||
run multiple instances of Logstash to horizontally scale event processing.
|
||||
Currently, this requires manual management of individual configuration files, or
|
||||
custom/3rd party configuration automation tools, some of which are maintained
|
||||
and supported by us (e.g. puppet-logstash). We plan to introduce an option to
|
||||
centrally store and manage Logstash configuration options to provide an
|
||||
alternative for scaling out your deployment that doesn’t rely on manual
|
||||
configuration file management or or 3rd party configuration management tools.
|
||||
|
||||
*High availability and load balancing ({ISSUES}2633[#2633]).* Currently, if a
|
||||
specific instance of Logstash becomes overloaded or unavailable, it can result
|
||||
in a performance degradation or outage until the problem is resolved, unless you
|
||||
use a dedicated load balancer to distribute traffic over the available
|
||||
instances. In a clustered deployment, we have the option of automatically
|
||||
distributing the load between instances based on the latest cluster state. This
|
||||
is a complex use case that will require input from the community on current
|
||||
approaches to implementing HA and load balancing of Logstash instances.
|
||||
|
||||
== Performance
|
||||
[float]
|
||||
=== status: ongoing; v1.5, v2.x
|
||||
|
||||
In the 1.5 release, we significantly improved the performance of the Grok
|
||||
filter, which is used to parse text via regular expressions. Based on our
|
||||
internal benchmarks, parsing common log formats, such as Apache logs, was 2x
|
||||
faster in Logstash 1.5 compared to previous versions. We also sped up JSON
|
||||
serialization and deserialization. In future releases of Logstash, we plan to
|
||||
incorporate additional JRuby optimizations to make the code even more efficient.
|
||||
We also plan to seek community feedback in terms of prioritizing other aspects
|
||||
of performance, such as startup time, resource utilization, and pipeline
|
||||
latency. You can follow our benchmarking and performance improvements in this issue ({ISSUES}3499[#3499]).
|
||||
|
||||
== Windows Support
|
||||
[float]
|
||||
=== status: ongoing; v1.5, v2.x
|
||||
|
||||
Leading up to the 1.5 release, we greatly improved automated Windows testing of
|
||||
Logstash. As a result of this testing, we identified and
|
||||
https://github.com/elastic/logstash/issues?q=is%3Aissue+label%3Awindows+is%3Aclosed[resolved]
|
||||
a number of critical issues affecting the Windows platform, pertaining to
|
||||
initial setup, upgrade, and file input plugin. You can follow the outstanding
|
||||
issues we are still working on using the GitHub
|
||||
https://github.com/elastic/logstash/issues?q=is%3Aissue+label%3Awindows+is%3Aopen[windows]
|
||||
label.
|
||||
|
||||
== Plugin Framework
|
||||
[float]
|
||||
=== status: completed; v1.5
|
||||
|
||||
Logstash has a rich collection of 165+ plugins, which are developed by
|
||||
Elasticsearch and contributed by the community. Previously, most commonly-used
|
||||
plugins were bundled with Logstash to make the getting started experience
|
||||
easier. However, there was no way to update plugins outside of the Logstash
|
||||
release cycle. In Logstash 1.5, we created a powerful plugin framework based on
|
||||
https://rubygems.org/[RubyGems.org] to facilitate per-plugin installation and
|
||||
updates. We will continue to distribute commonly-used plugins with Logstash, but
|
||||
now users will be able to install new plugins and receive plugin updates at any
|
||||
time. Read more about these changes in the
|
||||
http://www.elastic.co/blog/plugin-ecosystem-changes/[Logstash Plugin Ecosystem Changes]
|
||||
announcement.
|
||||
|
||||
== New Plugins
|
||||
[float]
|
||||
=== status: ongoing
|
||||
|
||||
Logstash plugins are continuously added to the Logstash plugin ecosystem, both
|
||||
by us and by our wonderful community of plugin contributors. Recent additions
|
||||
include https://github.com/logstash-plugins?query=kafka[Kafka],
|
||||
https://github.com/logstash-plugins?query=couchdb[CouchDB], and
|
||||
https://github.com/logstash-plugins/logstash-input-rss[RSS], just to name a few.
|
||||
In Logstash 1.5, we made it easier than ever to add and maintain plugins by
|
||||
putting each plugin into its own repository (see "Plugin Framework" section).
|
||||
We also greatly improved the S3, Twitter, RabbitMQ plugins. To follow requests
|
||||
for new Logstash plugins or contribute to the discussion, look for issues that
|
||||
have the {LABELS}new-plugin[new-plugin] tag in Github.
|
||||
159
docs/jp/static/running-logstash-command-line.asciidoc
Normal file
159
docs/jp/static/running-logstash-command-line.asciidoc
Normal file
|
|
@ -0,0 +1,159 @@
|
|||
[[running-logstash-command-line]]
|
||||
=== Running Logstash from the Command Line
|
||||
|
||||
To run Logstash from the command line, use the following command:
|
||||
|
||||
[source,shell]
|
||||
----
|
||||
bin/logstash [options]
|
||||
----
|
||||
|
||||
Where `options` are <<command-line-flags,command-line>> flags that you can
|
||||
specify to control Logstash execution. The location of the `bin` directory
|
||||
varies by platform. See <<dir-layout>> to find the location of `bin\logstash` on
|
||||
your system.
|
||||
|
||||
The following example runs Logstash and loads the Logstash config defined in
|
||||
the `mypipeline.conf` file:
|
||||
|
||||
[source,shell]
|
||||
----
|
||||
bin/logstash -f mypipeline.conf
|
||||
----
|
||||
|
||||
Any flags that you set at the command line override the corresponding settings
|
||||
in the Logstash <<logstash-settings-file,settings file>>, but the settings file
|
||||
itself is not changed. It remains as-is for subsequent Logstash runs.
|
||||
|
||||
Specifying command line options is useful when you are testing Logstash.
|
||||
However, in a production environment, we recommend that you use the Logstash
|
||||
<<logstash-settings-file,settings file>> to control Logstash execution. Using
|
||||
the settings file makes it easier for you to specify multiple options, and it
|
||||
provides you with a single, versionable file that you can use to start up
|
||||
Logstash consistently for each run.
|
||||
|
||||
[[command-line-flags]]
|
||||
==== Command-Line Flags
|
||||
|
||||
Logstash has the following flags. You can use the `--help` flag to display this information.
|
||||
|
||||
*`--node.name NAME`*::
|
||||
Specify the name of this Logstash instance. If no value is given it will default to the current
|
||||
hostname.
|
||||
|
||||
*`-f, --path.config CONFIG_PATH`*::
|
||||
Load the Logstash config from a specific file or directory. If a directory is given, all
|
||||
files in that directory will be concatenated in lexicographical order and then parsed as a
|
||||
single config file. Specifying this flag multiple times is not supported. If you specify
|
||||
this flag multiple times, Logstash uses the last occurrence (for example, `-f foo -f bar`
|
||||
is the same as `-f bar`).
|
||||
+
|
||||
You can specify wildcards (<<glob-support,globs>>) and any matched files will
|
||||
be loaded in the order described above. For example, you can use the wildcard feature to
|
||||
load specific files by name:
|
||||
+
|
||||
[source,shell]
|
||||
---------------------------------------------
|
||||
bin/logstash --debug -f '/tmp/{one,two,three}'
|
||||
---------------------------------------------
|
||||
+
|
||||
With this command, Logstash concatenates three config files, `/tmp/one`, `/tmp/two`, and
|
||||
`/tmp/three`, and parses them into a single config.
|
||||
|
||||
*`-e, --config.string CONFIG_STRING`*::
|
||||
Use the given string as the configuration data. Same syntax as the config file. If no
|
||||
input is specified, then the following is used as the default input:
|
||||
`input { stdin { type => stdin } }` and if no output is specified, then the
|
||||
following is used as the default output: `output { stdout { codec => rubydebug } }`.
|
||||
If you wish to use both defaults, please use the empty string for the `-e` flag.
|
||||
The default is nil.
|
||||
|
||||
*`-w, --pipeline.workers COUNT`*::
|
||||
Sets the number of pipeline workers to run. This option sets the number of workers that will,
|
||||
in parallel, execute the filter and output stages of the pipeline. If you find that events are
|
||||
backing up, or that the CPU is not saturated, consider increasing this number to better utilize
|
||||
machine processing power. The default is the number of the host's CPU cores.
|
||||
|
||||
*`-b, --pipeline.batch.size SIZE`*::
|
||||
Size of batches the pipeline is to work in. This option defines the maximum number of events an
|
||||
individual worker thread will collect from inputs before attempting to execute its filters and outputs.
|
||||
The default is 125 events. Larger batch sizes are generally more efficient, but come at the cost of
|
||||
increased memory overhead. You may have to increase the JVM heap size by setting the `LS_HEAP_SIZE`
|
||||
variable to effectively use the option.
|
||||
|
||||
*`-u, --pipeline.batch.delay DELAY_IN_MS`*::
|
||||
When creating pipeline batches, how long to wait while polling for the next event. This option defines
|
||||
how long in milliseconds to wait while polling for the next event before dispatching an undersized batch
|
||||
to filters and workers. The default is 250ms.
|
||||
|
||||
*`--pipeline.unsafe_shutdown`*::
|
||||
Force Logstash to exit during shutdown even if there are still inflight events
|
||||
in memory. By default, Logstash will refuse to quit until all received events
|
||||
have been pushed to the outputs. Enabling this option can lead to data loss during shutdown.
|
||||
|
||||
*`--path.data PATH`*::
|
||||
This should point to a writable directory. Logstash will use this directory whenever it needs to store
|
||||
data. Plugins will also have access to this path. The default is the `data` directory under
|
||||
Logstash home.
|
||||
|
||||
*`-p, --path.plugins PATH`*::
|
||||
A path of where to find custom plugins. This flag can be given multiple times to include
|
||||
multiple paths. Plugins are expected to be in a specific directory hierarchy:
|
||||
`PATH/logstash/TYPE/NAME.rb` where `TYPE` is `inputs`, `filters`, `outputs`, or `codecs`,
|
||||
and `NAME` is the name of the plugin.
|
||||
|
||||
*`-l, --path.logs PATH`*::
|
||||
Directory to write Logstash internal logs to.
|
||||
|
||||
*`--log.level LEVEL`*::
|
||||
Set the log level for Logstash. Possible values are:
|
||||
* `fatal`: log very severe error messages that will usually be followed by the application aborting
|
||||
* `error`: log errors
|
||||
* `warn`: log warnings
|
||||
* `info`: log verbose info (this is the default)
|
||||
* `debug`: log debugging info (for developers)
|
||||
* `trace`: log finer-grained messages beyond debugging info
|
||||
|
||||
*`--config.debug`*::
|
||||
Show the fully compiled configuration as a debug log message (you must also have `--log.level=debug` enabled).
|
||||
WARNING: The log message will include any 'password' options passed to plugin configs as plaintext, and may result
|
||||
in plaintext passwords appearing in your logs!
|
||||
|
||||
*`-i, --interactive SHELL`*::
|
||||
Drop to shell instead of running as normal. Valid shells are "irb" and "pry".
|
||||
|
||||
*`-V, --version`*::
|
||||
Emit the version of Logstash and its friends, then exit.
|
||||
|
||||
*`-t, --config.test_and_exit`*::
|
||||
Check configuration for valid syntax and then exit. Note that grok patterns are not checked for
|
||||
correctness with this flag. Logstash can read multiple config files from a directory. If you combine this
|
||||
flag with `--log.level=debug`, Logstash will log the combined config file, annotating
|
||||
each config block with the source file it came from.
|
||||
|
||||
*`-r, --config.reload.automatic`*::
|
||||
Monitor configuration changes and reload whenever the configuration is changed.
|
||||
NOTE: Use SIGHUP to manually reload the config. The default is false.
|
||||
|
||||
*`--config.reload.interval RELOAD_INTERVAL`*::
|
||||
How frequently to poll the configuration location for changes, in seconds. The default is every 3 seconds.
|
||||
|
||||
*`--http.host HTTP_HOST`*::
|
||||
Web API binding host. This option specifies the bind address for the metrics REST endpoint. The default is "127.0.0.1".
|
||||
|
||||
*`--http.port HTTP_PORT`*::
|
||||
Web API http port. This option specifies the bind port for the metrics REST endpoint. The default is 9600-9700.
|
||||
This setting accepts a range of the format 9600-9700. Logstash will pick up the first available port.
|
||||
|
||||
*`--log.format FORMAT`*::
|
||||
Specify if Logstash should write its own logs in JSON form (one event per line) or in plain text
|
||||
(using Ruby's Object#inspect). The default is "plain".
|
||||
|
||||
*`--path.settings SETTINGS_DIR`*::
|
||||
Set the directory containing the `logstash.yml` <<logstash-settings-file,settings file>> as well
|
||||
as the log4j logging configuration. This can also be set through the LS_SETTINGS_DIR environment variable.
|
||||
The default is the `config` directory under Logstash home.
|
||||
|
||||
*`-h, --help`*::
|
||||
Print help
|
||||
|
||||
53
docs/jp/static/running-logstash.asciidoc
Normal file
53
docs/jp/static/running-logstash.asciidoc
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
[[running-logstash]]
|
||||
=== Running Logstash as a Service on Debian or RPM
|
||||
|
||||
Logstash is not started automatically after installation. How to start and stop Logstash depends on whether your system
|
||||
uses systemd, upstart, or SysV.
|
||||
|
||||
Here are some common operating systems and versions, and the corresponding
|
||||
startup styles they use. This list is intended to be informative, not exhaustive.
|
||||
|
||||
|=======================================================================
|
||||
| Distribution | Service System |
|
||||
| Ubuntu 16.04 and newer | <<running-logstash-systemd,systemd>> |
|
||||
| Ubuntu 12.04 through 15.10 | <<running-logstash-upstart,upstart>> |
|
||||
| Debian 8 "jessie" and newer | <<running-logstash-systemd,systemd>> |
|
||||
| Debian 7 "wheezy" and older | <<running-logstash-sysv,sysv>> |
|
||||
| CentOS (and RHEL) 7 and newer | <<running-logstash-systemd,systemd>> |
|
||||
| CentOS (and RHEL) 6 | <<running-logstash-upstart,upstart>> |
|
||||
|=======================================================================
|
||||
|
||||
[[running-logstash-systemd]]
|
||||
==== Running Logstash by Using Systemd
|
||||
|
||||
Distributions like Debian Jessie, Ubuntu 15.10+, and many of the SUSE derivatives use systemd and the
|
||||
`systemctl` command to start and stop services. Logstash places the systemd unit files in `/etc/systemd/system` for both deb and rpm. After installing the package, you can start up Logstash with:
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
sudo systemctl start logstash.service
|
||||
-------------------------------------------
|
||||
|
||||
[[running-logstash-upstart]]
|
||||
==== Running Logstash by Using Upstart
|
||||
|
||||
For systems that use upstart, you can start Logstash with:
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
sudo initctl start logstash
|
||||
-------------------------------------------
|
||||
|
||||
The auto-generated configuration file for upstart systems is `/etc/init/logstash.conf`.
|
||||
|
||||
[[running-logstash-sysv]]
|
||||
==== Running Logstash by Using SysV
|
||||
|
||||
For systems that use SysV, you can start Logstash with:
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------
|
||||
sudo /etc/init.d/logstash start
|
||||
-------------------------------------------
|
||||
|
||||
The auto-generated configuration file for SysV systems is `/etc/init.d/logstash`.
|
||||
179
docs/jp/static/setting-up-logstash.asciidoc
Normal file
179
docs/jp/static/setting-up-logstash.asciidoc
Normal file
|
|
@ -0,0 +1,179 @@
|
|||
[[setup-logstash]]
|
||||
== Setting Up and Running Logstash
|
||||
|
||||
Before reading this section, see <<installing-logstash>> for basic installation instructions to get you started.
|
||||
|
||||
This section includes additional information on how to set up and run Logstash, including:
|
||||
|
||||
* <<dir-layout>>
|
||||
* <<config-setting-files>>
|
||||
* <<logstash-settings-file>>
|
||||
* <<running-logstash-command-line>>
|
||||
* <<running-logstash>>
|
||||
* <<docker>>
|
||||
* <<logging>>
|
||||
* <<persistent-queues>>
|
||||
* <<shutdown>>
|
||||
|
||||
|
||||
[[dir-layout]]
|
||||
=== Logstash Directory Layout
|
||||
|
||||
This section describes the default directory structure that is created when you unpack the Logstash installation packages.
|
||||
|
||||
[[zip-targz-layout]]
|
||||
==== Directory Layout of `.zip` and `.tar.gz` Archives
|
||||
|
||||
The `.zip` and `.tar.gz` packages are entirely self-contained. All files and
|
||||
directories are, by default, contained within the home directory -- the directory
|
||||
created when unpacking the archive.
|
||||
|
||||
This is very convenient because you don't have to create any directories to start using Logstash, and uninstalling
|
||||
Logstash is as easy as removing the home directory. However, it is advisable to change the default locations of the
|
||||
config and the logs directories so that you do not delete important data later on.
|
||||
|
||||
[cols="<h,<,<m,<m",options="header",]
|
||||
|=======================================================================
|
||||
| Type | Description | Default Location | Setting
|
||||
| home
|
||||
| Home directory of the Logstash installation.
|
||||
| `{extract.path}`- Directory created by unpacking the archive
|
||||
d|
|
||||
|
||||
| bin
|
||||
| Binary scripts, including `logstash` to start Logstash
|
||||
and `logstash-plugin` to install plugins
|
||||
| `{extract.path}/bin`
|
||||
d|
|
||||
|
||||
| settings
|
||||
| Configuration files, including `logstash.yml` and `jvm.options`
|
||||
| `{extract.path}/config`
|
||||
| `path.settings`
|
||||
|
||||
| logs
|
||||
| Log files
|
||||
| `{extract.path}/logs`
|
||||
| `path.logs`
|
||||
|
||||
| plugins
|
||||
| Local, non Ruby-Gem plugin files. Each plugin is contained in a subdirectory. Recommended for development only.
|
||||
| `{extract.path}/plugins`
|
||||
| `path.plugins`
|
||||
|
||||
|=======================================================================
|
||||
|
||||
[[deb-layout]]
|
||||
==== Directory Layout of Debian and RPM Packages
|
||||
|
||||
The Debian package and the RPM package each place config files, logs, and the settings files in the appropriate
|
||||
locations for the system:
|
||||
|
||||
[cols="<h,<,<m,<m",options="header",]
|
||||
|=======================================================================
|
||||
| Type | Description | Default Location | Setting
|
||||
| home
|
||||
| Home directory of the Logstash installation.
|
||||
| `/usr/share/logstash`
|
||||
d|
|
||||
|
||||
| bin
|
||||
| Binary scripts including `logstash` to start Logstash
|
||||
and `logstash-plugin` to install plugins
|
||||
| `/usr/share/logstash/bin`
|
||||
d|
|
||||
|
||||
| settings
|
||||
| Configuration files, including `logstash.yml`, `jvm.options`, and `startup.options`
|
||||
| `/etc/logstash`
|
||||
| `path.settings`
|
||||
|
||||
| conf
|
||||
| Logstash pipeline configuration files
|
||||
| `/etc/logstash/conf.d`
|
||||
| `path.config`
|
||||
|
||||
| logs
|
||||
| Log files
|
||||
| `/var/log/logstash`
|
||||
| `path.logs`
|
||||
|
||||
| plugins
|
||||
| Local, non Ruby-Gem plugin files. Each plugin is contained in a subdirectory. Recommended for development only.
|
||||
| `/usr/share/logstash/plugins`
|
||||
| `path.plugins`
|
||||
|
||||
|=======================================================================
|
||||
|
||||
[[docker-layout]]
|
||||
==== Directory Layout of Docker Images
|
||||
|
||||
The Docker images are created from the `.tar.gz` packages, and follow a
|
||||
similar directory layout.
|
||||
|
||||
[cols="<h,<,<m,<m",options="header",]
|
||||
|=======================================================================
|
||||
| Type | Description | Default Location | Setting
|
||||
| home
|
||||
| Home directory of the Logstash installation.
|
||||
| `/usr/share/logstash`
|
||||
d|
|
||||
|
||||
| bin
|
||||
| Binary scripts, including `logstash` to start Logstash
|
||||
and `logstash-plugin` to install plugins
|
||||
| `/usr/share/logstash/bin`
|
||||
d|
|
||||
|
||||
| settings
|
||||
| Configuration files, including `logstash.yml` and `jvm.options`
|
||||
| `/usr/share/logstash/config`
|
||||
| `path.settings`
|
||||
|
||||
| conf
|
||||
| Logstash pipeline configuration files
|
||||
| `/usr/share/logstash/pipeline`
|
||||
| `path.config`
|
||||
|
||||
| plugins
|
||||
| Local, non Ruby-Gem plugin files. Each plugin is contained in a subdirectory. Recommended for development only.
|
||||
| `/usr/share/logstash/plugins`
|
||||
| `path.plugins`
|
||||
|
||||
|=======================================================================
|
||||
|
||||
NOTE: Logstash Docker containers do not create log files by default. They log
|
||||
to standard output.
|
||||
|
||||
[[config-setting-files]]
|
||||
=== Logstash Configuration Files
|
||||
|
||||
Logstash has two types of configuration files: _pipeline configuration files_, which define the Logstash processing
|
||||
pipeline, and _settings files_, which specify options that control Logstash startup and execution.
|
||||
|
||||
==== Pipeline Configuration Files
|
||||
|
||||
You create pipeline configuration files when you define the stages of your Logstash processing pipeline. On deb and
|
||||
rpm, you place the pipeline configuration files in the `/etc/logstash/conf.d` directory. Logstash tries to load all
|
||||
files in the `/etc/logstash/conf.d directory`, so don't store any non-config files or backup files in this directory.
|
||||
|
||||
See <<configuration>> for more info.
|
||||
|
||||
==== Settings Files
|
||||
|
||||
The settings files are already defined in the Logstash installation. Logstash includes the following settings files:
|
||||
|
||||
*`logstash.yml`*::
|
||||
Contains Logstash configuration flags. You can set flags in this file instead of passing the flags at the command
|
||||
line. Any flags that you set at the command line override the corresponding settings in the `logstash.yml` file. See <<logstash-settings-file>> for more info.
|
||||
*`jvm.options`*::
|
||||
Contains JVM configuration flags. Specify each flag on a separate line. You can also use this file to set the locale
|
||||
for Logstash.
|
||||
*`startup.options` (Linux)*::
|
||||
Contains options used by the `system-install` script in `/usr/share/logstash/bin` to build the appropriate startup
|
||||
script for your system. When you install the Logstash package, the `system-install` script executes at the end of the
|
||||
installation process and uses the settings specified in `startup.options` to set options such as the user, group,
|
||||
service name, and service description. By default, Logstash services are installed under the user `logstash`. The `startup.options` file makes it easier for you to install multiple instances of the Logstash service. You can copy
|
||||
the file and change the values for specific settings. Note that the `startup.options` file is not read at startup. If
|
||||
you want to change the Logstash startup script (for example, to change the Logstash user or read from a different
|
||||
configuration path), you must re-run the `system-install` script (as root) to pass in the new settings.
|
||||
174
docs/jp/static/settings-file.asciidoc
Normal file
174
docs/jp/static/settings-file.asciidoc
Normal file
|
|
@ -0,0 +1,174 @@
|
|||
[[logstash-settings-file]]
|
||||
=== Settings File
|
||||
|
||||
You can set options in the Logstash settings file, `logstash.yml`, to control Logstash execution. For example,
|
||||
you can specify pipeline settings, the location of configuration files, logging options, and other settings.
|
||||
Most of the settings in the `logstash.yml` file are also available as <<command-line-flags,command-line flags>>
|
||||
when you run Logstash. Any flags that you set at the command line override the corresponding settings in the
|
||||
`logstash.yml` file.
|
||||
|
||||
The `logstash.yml` file is written in http://yaml.org/[YAML]. Its location varies by platform (see
|
||||
<<dir-layout>>). You can specify settings in hierarchical form or use flat keys. For example, to use
|
||||
hierarchical form to set the pipeline batch size and batch delay, you specify:
|
||||
|
||||
[source,yaml]
|
||||
-------------------------------------------------------------------------------------
|
||||
pipeline:
|
||||
batch:
|
||||
size: 125
|
||||
delay: 5
|
||||
-------------------------------------------------------------------------------------
|
||||
|
||||
To express the same values as flat keys, you specify:
|
||||
|
||||
[source,yaml]
|
||||
-------------------------------------------------------------------------------------
|
||||
pipeline.batch.size: 125
|
||||
pipeline.batch.delay: 5
|
||||
-------------------------------------------------------------------------------------
|
||||
|
||||
The `logstash.yml` file includes the following settings:
|
||||
|
||||
[options="header"]
|
||||
|=======================================================================
|
||||
| Setting | Description | Default value
|
||||
|
||||
| `node.name`
|
||||
| A descriptive name for the node.
|
||||
| Machine's hostname
|
||||
|
||||
| `path.data`
|
||||
| The directory that Logstash and its plugins use for any persistent needs.
|
||||
|`LOGSTASH_HOME/data`
|
||||
|
||||
| `pipeline.workers`
|
||||
| The number of workers that will, in parallel, execute the filter and output stages of the pipeline.
|
||||
If you find that events are backing up, or that the
|
||||
CPU is not saturated, consider increasing this number to better utilize machine processing power.
|
||||
| Number of the host's CPU cores
|
||||
|
||||
| `pipeline.output.workers`
|
||||
| The number of workers to use per output plugin instance.
|
||||
| `1`
|
||||
|
||||
| `pipeline.batch.size`
|
||||
| The maximum number of events an individual worker thread will collect from inputs
|
||||
before attempting to execute its filters and outputs.
|
||||
Larger batch sizes are generally more efficient, but come at the cost of increased memory
|
||||
overhead. You may have to increase the JVM heap size by setting the `LS_HEAP_SIZE`
|
||||
variable to effectively use the option.
|
||||
| `125`
|
||||
|
||||
| `pipeline.batch.delay`
|
||||
| When creating pipeline event batches, how long in milliseconds to wait before dispatching an undersized
|
||||
batch to filters and workers.
|
||||
| `5`
|
||||
|
||||
| `pipeline.unsafe_shutdown`
|
||||
| When set to `true`, forces Logstash to exit during shutdown even if there are still inflight events
|
||||
in memory. By default, Logstash will refuse to quit until all received events
|
||||
have been pushed to the outputs. Enabling this option can lead to data loss during shutdown.
|
||||
| `false`
|
||||
|
||||
| `path.config`
|
||||
| The path to the Logstash config for the main pipeline. If you specify a directory or wildcard,
|
||||
config files are read from the directory in alphabetical order.
|
||||
| Platform-specific. See <<dir-layout>>.
|
||||
|
||||
| `config.string`
|
||||
| A string that contains the pipeline configuration to use for the main pipeline. Use the same syntax as
|
||||
the config file.
|
||||
| None
|
||||
|
||||
| `config.test_and_exit`
|
||||
| When set to `true`, checks that the configuration is valid and then exits. Note that grok patterns are not checked for
|
||||
correctness with this setting. Logstash can read multiple config files from a directory. If you combine this
|
||||
setting with `log.level: debug`, Logstash will log the combined config file, annotating
|
||||
each config block with the source file it came from.
|
||||
| `false`
|
||||
|
||||
| `config.reload.automatic`
|
||||
| When set to `true`, periodically checks if the configuration has changed and reloads the configuration whenever it is changed.
|
||||
This can also be triggered manually through the SIGHUP signal.
|
||||
| `false`
|
||||
|
||||
| `config.reload.interval`
|
||||
| How often in seconds Logstash checks the config files for changes.
|
||||
| `3`
|
||||
|
||||
| `config.debug`
|
||||
| When set to `true`, shows the fully compiled configuration as a debug log message. You must also set `log.level: debug`.
|
||||
WARNING: The log message will include any 'password' options passed to plugin configs as plaintext, and may result
|
||||
in plaintext passwords appearing in your logs!
|
||||
| `false`
|
||||
|
||||
| `queue.type`
|
||||
| The internal queuing model to use for event buffering. Specify `memory` for legacy in-memory based queuing, or `persisted` for disk-based ACKed queueing (<<persistent-queues,persistent queues>>).
|
||||
| `memory`
|
||||
|
||||
| `path.queue`
|
||||
| The directory path where the data files will be stored when persistent queues are enabled (`queue.type: persisted`).
|
||||
| `path.data/queue`
|
||||
|
||||
| `queue.page_capacity`
|
||||
| The size of the page data files used when persistent queues are enabled (`queue.type: persisted`). The queue data consists of append-only data files separated into pages.
|
||||
| 250mb
|
||||
|
||||
| `queue.max_events`
|
||||
| The maximum number of unread events in the queue when persistent queues are enabled (`queue.type: persisted`).
|
||||
| 0 (unlimited)
|
||||
|
||||
| `queue.max_bytes`
|
||||
| The total capacity of the queue in number of bytes. Make sure the capacity of your disk drive is greater than the value you specify here. If both `queue.max_events` and `queue.max_bytes` are specified, Logstash uses whichever criteria is reached first.
|
||||
| 1024mb (1g)
|
||||
|
||||
| `queue.checkpoint.acks`
|
||||
| The maximum number of ACKed events before forcing a checkpoint when persistent queues are enabled (`queue.type: persisted`). Specify `queue.checkpoint.acks: 0` to set this value to unlimited.
|
||||
|1024
|
||||
|
||||
| `queue.checkpoint.writes`
|
||||
| The maximum number of written events before forcing a checkpoint when persistent queues are enabled (`queue.type: persisted`). Specify `queue.checkpoint.writes: 0` to set this value to unlimited.
|
||||
| 1024
|
||||
|
||||
| `queue.checkpoint.interval`
|
||||
| The interval in milliseconds when a checkpoint is forced on the head page when persistent queues are enabled (`queue.type: persisted`). Specify `queue.checkpoint.interval: 0` for no periodic checkpoint.
|
||||
| 1000
|
||||
|
||||
| `http.host`
|
||||
| The bind address for the metrics REST endpoint.
|
||||
| `"127.0.0.1"`
|
||||
|
||||
| `http.port`
|
||||
| The bind port for the metrics REST endpoint.
|
||||
| `9600`
|
||||
|
||||
| `log.level`
|
||||
a|
|
||||
The log level. Valid options are:
|
||||
|
||||
* `fatal`
|
||||
* `error`
|
||||
* `warn`
|
||||
* `info`
|
||||
* `debug`
|
||||
* `trace`
|
||||
|
||||
| `info`
|
||||
|
||||
| `log.format`
|
||||
| The log format. Set to `json` to log in JSON format, or `plain` to use `Object#.inspect`.
|
||||
| `plain`
|
||||
|
||||
| `path.logs`
|
||||
| The directory where Logstash will write its log to.
|
||||
| `LOGSTASH_HOME/logs
|
||||
|
||||
| `path.plugins`
|
||||
| Where to find custom plugins. You can specify this setting multiple times to include
|
||||
multiple paths. Plugins are expected to be in a specific directory hierarchy:
|
||||
`PATH/logstash/TYPE/NAME.rb` where `TYPE` is `inputs`, `filters`, `outputs`, or `codecs`,
|
||||
and `NAME` is the name of the plugin.
|
||||
| Platform-specific. See <<dir-layout>>.
|
||||
|
||||
|=======================================================================
|
||||
|
||||
92
docs/jp/static/shutdown.asciidoc
Normal file
92
docs/jp/static/shutdown.asciidoc
Normal file
|
|
@ -0,0 +1,92 @@
|
|||
[[shutdown]]
|
||||
=== Shutting Down Logstash
|
||||
|
||||
To shut down Logstash, use one of the following commands:
|
||||
|
||||
* On systemd, use:
|
||||
+
|
||||
[source,shell]
|
||||
----
|
||||
systemctl stop logstash
|
||||
----
|
||||
|
||||
* On upstart, use:
|
||||
+
|
||||
[source,shell]
|
||||
----
|
||||
initctl stop logstash
|
||||
----
|
||||
|
||||
* On sysv, use:
|
||||
+
|
||||
[source,shell]
|
||||
----
|
||||
/etc/init.d/logstash stop
|
||||
----
|
||||
|
||||
* If you have the PID, use:
|
||||
+
|
||||
[source,shell]
|
||||
----
|
||||
kill -TERM {logstash_pid}
|
||||
----
|
||||
|
||||
==== What Happens During a Controlled Shutdown?
|
||||
|
||||
When you attempt to shut down a running Logstash instance, Logstash performs several steps before it can safely shut down. It must:
|
||||
|
||||
* Stop all input, filter and output plugins
|
||||
* Process all in-flight events
|
||||
* Terminate the Logstash process
|
||||
|
||||
The following conditions affect the shutdown process:
|
||||
|
||||
* An input plugin receiving data at a slow pace.
|
||||
* A slow filter, like a Ruby filter executing `sleep(10000)` or an Elasticsearch filter that is executing a very heavy
|
||||
query.
|
||||
* A disconnected output plugin that is waiting to reconnect to flush in-flight events.
|
||||
|
||||
These situations make the duration and success of the shutdown process unpredictable.
|
||||
|
||||
Logstash has a stall detection mechanism that analyzes the behavior of the pipeline and plugins during shutdown.
|
||||
This mechanism produces periodic information about the count of inflight events in internal queues and a list of busy
|
||||
worker threads.
|
||||
|
||||
To enable Logstash to forcibly terminate in the case of a stalled shutdown, use the `--pipeline.unsafe_shutdown` flag when
|
||||
you start Logstash.
|
||||
|
||||
WARNING: Unsafe shutdowns, force-kills of the Logstash process, or crashes of the Logstash process for any other reason may result in data loss (unless you've
|
||||
enabled Logstash to use <<persistent-queues,persistent queues>>). Shut down
|
||||
Logstash safely whenever possible.
|
||||
|
||||
[[shutdown-stall-example]]
|
||||
==== Stall Detection Example
|
||||
|
||||
In this example, slow filter execution prevents the pipeline from performing a clean shutdown. Because Logstash is
|
||||
started with the `--pipeline.unsafe_shutdown` flag, the shutdown results in the loss of 20 events.
|
||||
|
||||
========
|
||||
[source,shell]
|
||||
bin/logstash -e 'input { generator { } } filter { ruby { code => "sleep 10000" } }
|
||||
output { stdout { codec => dots } }' -w 1 --pipeline.unsafe_shutdown
|
||||
Pipeline main started
|
||||
^CSIGINT received. Shutting down the agent. {:level=>:warn}
|
||||
stopping pipeline {:id=>"main", :level=>:warn}
|
||||
Received shutdown signal, but pipeline is still waiting for in-flight events
|
||||
to be processed. Sending another ^C will force quit Logstash, but this may cause
|
||||
data loss. {:level=>:warn}
|
||||
{"inflight_count"=>125, "stalling_thread_info"=>{["LogStash::Filters::Ruby",
|
||||
{"code"=>"sleep 10000"}]=>[{"thread_id"=>19, "name"=>"[main]>worker0",
|
||||
"current_call"=>"(ruby filter code):1:in `sleep'"}]}} {:level=>:warn}
|
||||
The shutdown process appears to be stalled due to busy or blocked plugins.
|
||||
Check the logs for more information. {:level=>:error}
|
||||
{"inflight_count"=>125, "stalling_thread_info"=>{["LogStash::Filters::Ruby",
|
||||
{"code"=>"sleep 10000"}]=>[{"thread_id"=>19, "name"=>"[main]>worker0",
|
||||
"current_call"=>"(ruby filter code):1:in `sleep'"}]}} {:level=>:warn}
|
||||
{"inflight_count"=>125, "stalling_thread_info"=>{["LogStash::Filters::Ruby",
|
||||
{"code"=>"sleep 10000"}]=>[{"thread_id"=>19, "name"=>"[main]>worker0",
|
||||
"current_call"=>"(ruby filter code):1:in `sleep'"}]}} {:level=>:warn}
|
||||
Forcefully quitting logstash.. {:level=>:fatal}
|
||||
========
|
||||
|
||||
When `--pipeline.unsafe_shutdown` isn't enabled, Logstash continues to run and produce these reports periodically.
|
||||
107
docs/jp/static/submitting-a-plugin.asciidoc
Normal file
107
docs/jp/static/submitting-a-plugin.asciidoc
Normal file
|
|
@ -0,0 +1,107 @@
|
|||
[[submitting-plugin]]
|
||||
=== Submitting your plugin to RubyGems.org and the logstash-plugins repository
|
||||
|
||||
Logstash uses http://rubygems.org[RubyGems.org] as its repository for all plugin
|
||||
artifacts. Once you have developed your new plugin, you can make it available to
|
||||
Logstash users by simply publishing it to RubyGems.org.
|
||||
|
||||
==== Licensing
|
||||
Logstash and all its plugins are licensed under
|
||||
https://github.com/elasticsearch/logstash/blob/master/LICENSE[Apache License, version 2 ("ALv2")].
|
||||
If you make your plugin publicly available via http://rubygems.org[RubyGems.org],
|
||||
please make sure to have this line in your gemspec:
|
||||
|
||||
* `s.licenses = ['Apache License (2.0)']`
|
||||
|
||||
==== Publishing to http://rubygems.org[RubyGems.org]
|
||||
|
||||
To begin, you’ll need an account on RubyGems.org
|
||||
|
||||
* https://rubygems.org/sign_up[Sign-up for a RubyGems account].
|
||||
|
||||
After creating an account,
|
||||
http://guides.rubygems.org/rubygems-org-api/#api-authorization[obtain] an API
|
||||
key from RubyGems.org. By default, RubyGems uses the file `~/.gem/credentials`
|
||||
to store your API key. These credentials will be used to publish the gem.
|
||||
Replace `username` and `password` with the credentials you created at
|
||||
RubyGems.org:
|
||||
|
||||
[source,sh]
|
||||
----------------------------------
|
||||
curl -u username:password https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials
|
||||
chmod 0600 ~/.gem/credentials
|
||||
----------------------------------
|
||||
|
||||
Before proceeding, make sure you have the right version in your gemspec file
|
||||
and commit your changes.
|
||||
|
||||
* `s.version = '0.1.0'`
|
||||
|
||||
To publish version 0.1.0 of your new logstash gem:
|
||||
|
||||
[source,sh]
|
||||
----------------------------------
|
||||
bundle install
|
||||
bundle exec rake vendor
|
||||
bundle exec rspec
|
||||
bundle exec rake publish_gem
|
||||
----------------------------------
|
||||
|
||||
[NOTE]
|
||||
========
|
||||
Executing `rake publish_gem`:
|
||||
|
||||
. Reads the version from the gemspec file (`s.version = '0.1.0'`)
|
||||
. Checks in your local repository if a tag exists for that version. If the tag
|
||||
already exists, it aborts the process. Otherwise, it creates a new version tag
|
||||
in your local repository.
|
||||
. Builds the gem
|
||||
. Publishes the gem to RubyGems.org
|
||||
========
|
||||
|
||||
That's it! Your plugin is published! Logstash users can now install your plugin
|
||||
by running:
|
||||
|
||||
[source,sh]
|
||||
[subs="attributes"]
|
||||
----------------------------------
|
||||
bin/plugin install logstash-{plugintype}-mypluginname
|
||||
----------------------------------
|
||||
|
||||
==== Contributing your source code to https://github.com/logstash-plugins[logstash-plugins]
|
||||
|
||||
It is not required to contribute your source code to
|
||||
https://github.com/logstash-plugins[logstash-plugins] github organization, but
|
||||
we always welcome new plugins!
|
||||
|
||||
==== Benefits
|
||||
|
||||
Some of the many benefits of having your plugin in the logstash-plugins
|
||||
repository are:
|
||||
|
||||
* **Discovery** Your plugin will appear in the http://www.elasticsearch.org/guide/en/logstash/current/index.html[Logstash Reference],
|
||||
where Logstash users look first for plugins and documentation.
|
||||
* **Documentation** Your plugin documentation will automatically be added to the
|
||||
http://www.elasticsearch.org/guide/en/logstash/current/index.html[Logstash Reference].
|
||||
* **Testing** With our testing infrastructure, your plugin will be continuously
|
||||
tested against current and future releases of Logstash. As a result, users will
|
||||
have the assurance that if incompatibilities arise, they will be quickly
|
||||
discovered and corrected.
|
||||
|
||||
==== Acceptance Guidelines
|
||||
|
||||
* **Code Review** Your plugin must be reviewed by members of the community for
|
||||
coherence, quality, readability, stability and security.
|
||||
* **Tests** Your plugin must contain tests to be accepted. These tests are also
|
||||
subject to code review for scope and completeness. It's ok if you don't know
|
||||
how to write tests -- we will guide you. We are working on publishing a guide to
|
||||
creating tests for Logstash which will make it easier. In the meantime, you can
|
||||
refer to http://betterspecs.org/ for examples.
|
||||
|
||||
To begin migrating your plugin to logstash-plugins, simply create a new
|
||||
https://github.com/elasticsearch/logstash/issues[issue] in
|
||||
the Logstash repository. When the acceptance guidelines are completed, we will
|
||||
facilitate the move to the logstash-plugins organization using the recommended
|
||||
https://help.github.com/articles/transferring-a-repository/#transferring-from-a-user-to-an-organization[github process].
|
||||
|
||||
|
||||
82
docs/jp/static/upgrading.asciidoc
Normal file
82
docs/jp/static/upgrading.asciidoc
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
[[upgrading-logstash]]
|
||||
== Upgrading Logstash
|
||||
|
||||
[IMPORTANT]
|
||||
===========================================
|
||||
Before upgrading Logstash:
|
||||
|
||||
* Consult the <<breaking-changes,breaking changes>> docs.
|
||||
* Test upgrades in a development environment before upgrading your production cluster.
|
||||
===========================================
|
||||
|
||||
If you are installing Logstash with other components in the Elastic Stack, also see the
|
||||
{stack}index.html[Elastic Stack installation and upgrade documentation].
|
||||
|
||||
See the following topics for information about upgrading Logstash:
|
||||
|
||||
* <<upgrading-using-package-managers>>
|
||||
* <<upgrading-using-direct-download>>
|
||||
* <<upgrading-logstash-5.0>>
|
||||
|
||||
[[upgrading-using-package-managers]]
|
||||
=== Upgrading Using Package Managers
|
||||
|
||||
This procedure uses <<package-repositories,package managers>> to upgrade Logstash.
|
||||
|
||||
1. Shut down your Logstash pipeline, including any inputs that send events to Logstash.
|
||||
2. Using the directions in the _Package Repositories_ section, update your repository links to point to the 5.x repositories
|
||||
instead of the previous version.
|
||||
3. Run the `apt-get upgrade logstash` or `yum update logstash` command as appropriate for your operating system.
|
||||
4. Test your configuration file with the `logstash --config.test_and_exit -f <configuration-file>` command. Configuration options for
|
||||
some Logstash plugins have changed in the 5.x release.
|
||||
5. Restart your Logstash pipeline after updating your configuration file.
|
||||
|
||||
[[upgrading-using-direct-download]]
|
||||
=== Upgrading Using a Direct Download
|
||||
|
||||
This procedure downloads the relevant Logstash binaries directly from Elastic.
|
||||
|
||||
1. Shut down your Logstash pipeline, including any inputs that send events to Logstash.
|
||||
2. Download the https://www.elastic.co/downloads/logstash[Logstash installation file] that matches your host environment.
|
||||
3. Unpack the installation file into your Logstash directory.
|
||||
4. Test your configuration file with the `logstash --config.test_and_exit -f <configuration-file>` command. Configuration options for
|
||||
some Logstash plugins have changed in the 5.x release.
|
||||
5. Restart your Logstash pipeline after updating your configuration file.
|
||||
|
||||
[[upgrading-logstash-5.0]]
|
||||
=== Upgrading Logstash to 5.0
|
||||
|
||||
Before upgrading Logstash, remember to read the <<breaking-changes,breaking changes>>.
|
||||
|
||||
If you are installing Logstash with other components in the Elastic Stack, also see the
|
||||
{stack}index.html[Elastic Stack installation and upgrade documentation].
|
||||
|
||||
==== When to Upgrade
|
||||
|
||||
Fresh installations can and should start with the same version across the Elastic Stack.
|
||||
|
||||
Elasticsearch 5.0 does not require Logstash 5.0. An Elasticsearch 5.0 cluster will happily receive data from a
|
||||
Logstash 2.x instance via the default HTTP communication layer. This provides some flexibility to decide when to upgrade
|
||||
Logstash relative to an Elasticsearch upgrade. It may or may not be convenient for you to upgrade them together, and it
|
||||
is
|
||||
not required to be done at the same time as long as Elasticsearch is upgraded first.
|
||||
|
||||
You should upgrade in a timely manner to get the performance improvements that come with Logstash 5.0, but do so in
|
||||
the way that makes the most sense for your environment.
|
||||
|
||||
==== When Not to Upgrade
|
||||
|
||||
If any Logstash plugin that you require is not compatible with Logstash 5.0, then you should wait until it is ready
|
||||
before upgrading.
|
||||
|
||||
Although we make great efforts to ensure compatibility, Logstash 5.0 is not completely backwards compatible. As noted
|
||||
in the Elastic Stack upgrade guide, Logstash 5.0 should not be upgraded before Elasticsearch 5.0. This is both
|
||||
practical and because some Logstash 5.0 plugins may attempt to use features of Elasticsearch 5.0 that did not exist
|
||||
in earlier versions. For example, if you attempt to send the 5.x template to a cluster before Elasticsearch 5.0, then it
|
||||
will not be able to use it and all indexing will fail likely fail. If you use your own, custom template with Logstash,
|
||||
then this issue can be ignored.
|
||||
|
||||
Note the Elasticsearch Output Index Template change in the <<breaking-changes>> documentation for further insight into
|
||||
this change and how it impacts operations.
|
||||
|
||||
|
||||
34
docs/kr/gs-index.asciidoc
Normal file
34
docs/kr/gs-index.asciidoc
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
[[logstash-reference]]
|
||||
= Logstash 참조
|
||||
|
||||
:branch: 5.4
|
||||
:major-version: 5.4
|
||||
:logstash_version: 5.4.0
|
||||
:elasticsearch_version: 5.4.0
|
||||
:docker-image: docker.elastic.co/logstash/logstash:{logstash_version}
|
||||
|
||||
//////////
|
||||
release-state can be: released | prerelease | unreleased
|
||||
//////////
|
||||
:release-state: released
|
||||
|
||||
:jdk: 1.8.0
|
||||
:guide: https://www.elastic.co/guide/en/elasticsearch/guide/current/
|
||||
:ref: https://www.elastic.co/guide/en/elasticsearch/reference/5.x/
|
||||
:xpack: https://www.elastic.co/guide/en/x-pack/current/
|
||||
:logstash: https://www.elastic.co/guide/en/logstash/5.x/
|
||||
:filebeat: https://www.elastic.co/guide/en/beats/filebeat/5.x/
|
||||
:lsissue: https://github.com/elastic/logstash/issues/
|
||||
:security: X-Pack 보안
|
||||
:stack: https://www.elastic.co/guide/en/elastic-stack/current/
|
||||
|
||||
[[introduction]]
|
||||
== Logstash 소개
|
||||
|
||||
Logstash는 실시간 파이프라인 기능을 가진 오픈소스 데이터 수집 엔진입니다. Logstash는 서로 다른 소스의 데이터를 탄력적으로 통합하고 사용자가 선택한 목적지로 데이터를 정규화할 수 있습니다. 다양한 고급 다운스트림 분석 및 시각화 활용 사례를 위해 모든 데이터를 정리하고 대중화(democratization)합니다.
|
||||
|
||||
Logstash는 근본적으로 로그 수집의 혁신을 주도해 왔으며 그 기능은 이러한 활용 사례 이외의 영역으로 확장되고 있습니다. 어떤 유형의 이벤트도 다양한 입력, 필터, 출력 플러그인을 통해 강화하고 전환할 수 있으며 기본 제공되는 여러 코덱으로 수집(ingestion) 프로세스를 한층 더 간소화할 수 있습니다. Logstash로 더 방대하고 다양한 데이터를 활용함으로써 더 신속하게 인사이트를 개발할 수 있습니다.
|
||||
|
||||
include::static/introduction.asciidoc[]
|
||||
|
||||
include::static/getting-started-with-logstash.asciidoc[]
|
||||
190
docs/kr/static/getting-started-with-logstash.asciidoc
Normal file
190
docs/kr/static/getting-started-with-logstash.asciidoc
Normal file
|
|
@ -0,0 +1,190 @@
|
|||
[[getting-started-with-logstash]]
|
||||
== Logstash 시작하기
|
||||
|
||||
이 섹션에서는 Logstash를 설치하고 모든 것이 제대로 실행되고 있는지 확인하는 프로세스를 안내합니다.
|
||||
첫 번째 이벤트를 보관(stash)하는 방법을 학습한 다음 Apache 웹 로그를 입력으로 사용하여 로그를 파싱하고 파싱된 데이터를 Elasticsearch 클러스터에 기록하는 더 발전된 파이프라인을 생성해보겠습니다. 그런 다음 여러 입력 및 출력 플러그인을 연결하여 다양한 소스의 데이터를 통합하는 방법을 알아봅니다.
|
||||
|
||||
이 섹션은 다음 주제로 구성되어 있습니다.
|
||||
|
||||
* <<installing-logstash>>
|
||||
* <<first-event>>
|
||||
* {logstash}advanced-pipeline.html[Parsing Logs with Logstash]
|
||||
* {logstash}/multiple-input-output-plugins.html[Stitching Together Multiple Input Output Plugins]
|
||||
|
||||
[[installing-logstash]]
|
||||
=== Logstash 설치
|
||||
|
||||
NOTE: Logstash에는 Java 8이 필요합니다. Java 9는 지원되지 않습니다. http://www.oracle.com/technetwork/java/javase/downloads/index.html[official Oracle distribution] 또는 오픈소스 배포판(예: http://openjdk.java.net/[OpenJDK])을 사용합니다.
|
||||
|
||||
Java 버전을 확인하려면 다음 명령을 실행합니다.
|
||||
|
||||
[source,shell]
|
||||
java -version
|
||||
|
||||
이 명령은 Java가 설치된 시스템에서 다음과 비슷한 출력을 생성합니다.
|
||||
|
||||
[source,shell]
|
||||
java version "1.8.0_65"
|
||||
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
|
||||
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
|
||||
|
||||
일부 Linux 시스템에서는, 특히 tarball에서 Java를 설치했다면 설치를 시도하기에 앞서 `JAVA_HOME` 환경을 내보내야 할 수도 있습니다. Logstash는 설치 과정에서 Java를 사용하여 자동으로 사용자의 환경을 탐지하고 알맞은 시동 방식(SysV init 스크립트, Upstart, systemd)을 설치합니다. Logstash가 패키지 설치 시 JAVA_HOME 환경 변수를 찾지 못할 경우 오류 메시지가 나타날 수 있으며 Logstash는 제대로 시작하지 못합니다.
|
||||
|
||||
[float]
|
||||
[[installing-binary]]
|
||||
=== 다운로드된 바이너리에서 설치
|
||||
|
||||
호스트 환경에 맞는 https://www.elastic.co/downloads/logstash[Logstash installation file]을 다운로드합니다.
|
||||
파일의 압축을 풉니다. 콜론(:) 문자를 포함하는 디렉토리 경로에 Logstash를 설치하지 마십시오.
|
||||
|
||||
지원되는 Linux 운영 체제에서는 패키지 관리자를 사용하여 Logstash를 설치할 수 있습니다.
|
||||
|
||||
[float]
|
||||
[[package-repositories]]
|
||||
=== 패키지 리포지토리에서 설치
|
||||
|
||||
APT 및 YUM 기반 배포판을 위한 리포지토리도 제공됩니다. 패키지가 Logstash 빌드의 일부로 생성되므로 소스 패키지 없이 바이너리 패키지만 제공됩니다.
|
||||
|
||||
실수로 주요 버전 사이에 업그레이드를 하지 않도록 버전별 Logstash 패키지 리포지토리를 개별 URL로 분리했습니다. 모든 {major-version}.y 릴리스에서는 {major-version}을(를) 버전 번호로 사용합니다.
|
||||
|
||||
PGP 키 https://pgp.mit.edu/pks/lookup?op=vindex&search=0xD27D666CD88E42B4[D88E42B4]인 Elastic Signing Key와 핑거프린트
|
||||
|
||||
4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4
|
||||
|
||||
를 사용하여 모든 패키지를 서명합니다. 이는 https://pgp.mit.edu 에서 구할 수 있습니다.
|
||||
|
||||
[float]
|
||||
==== APT
|
||||
|
||||
ifeval::["{release-state}"=="unreleased"]
|
||||
|
||||
Logstash의 {logstash_version} 버전은 아직 릴리스되지 않았습니다.
|
||||
|
||||
endif::[]
|
||||
|
||||
ifeval::["{release-state}"!="unreleased"]
|
||||
|
||||
공개 서명 키를 다운로드하고 설치합니다.
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------------
|
||||
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
|
||||
--------------------------------------------------
|
||||
|
||||
계속하기 전에 Debian에 `apt-transport-https` 패키지를 설치해야 할 수도 있습니다.
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------------
|
||||
sudo apt-get install apt-transport-https
|
||||
--------------------------------------------------
|
||||
|
||||
리포지토리 정의를 +/etc/apt/sources.list.d/elastic-{major-version}.list+에 저장합니다.
|
||||
|
||||
["source","sh",subs="attributes,callouts"]
|
||||
--------------------------------------------------
|
||||
echo "deb https://artifacts.elastic.co/packages/{major-version}/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-{major-version}.list
|
||||
--------------------------------------------------
|
||||
|
||||
[WARNING]
|
||||
==================================================
|
||||
앞서 설명한 `echo` 메서드를 사용하여 Logstash 리포지토리를 추가합니다. `add-apt-repository`는 사용하지 마십시오. 이는 `deb-src` 항목도 추가하는데 소스 패키지가 제공되지 않습니다. `deb-src` 항목을 추가하면 다음과 같은 오류가 표시됩니다.
|
||||
|
||||
릴리스 파일에서 예상된 항목 'main/source/Sources'를 찾을 수 없음(잘못된 sources.list 항목 또는 잘못된 형식의 파일)
|
||||
|
||||
`/etc/apt/sources.list` 파일에서 `deb-src` 항목을 삭제하면 설치가 정상적으로 진행됩니다.
|
||||
==================================================
|
||||
|
||||
`sudo apt-get update`를 실행하면 리포지토리는 사용 가능한 상태가 됩니다. 다음 명령으로 설치할 수 있습니다.
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------------
|
||||
sudo apt-get update && sudo apt-get install logstash
|
||||
--------------------------------------------------
|
||||
|
||||
Logstash를 시스템 서비스로 관리하는 자세한 내용은 {logstash}running-logstash.html[Logstash 실행]을 참조하십시오.
|
||||
|
||||
endif::[]
|
||||
|
||||
[float]
|
||||
==== YUM
|
||||
|
||||
ifeval::["{release-state}"=="unreleased"]
|
||||
|
||||
Logstash의 {logstash_version} 버전은 아직 릴리스되지 않았습니다.
|
||||
|
||||
endif::[]
|
||||
|
||||
ifeval::["{release-state}"!="unreleased"]
|
||||
|
||||
공개 서명 키를 다운로드하고 설치합니다.
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------------
|
||||
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
|
||||
--------------------------------------------------
|
||||
|
||||
`/etc/yum.repos.d/` 디렉토리에서 `.repo` 접미사를 사용하는 파일(예: `logstash.repo`)에 다음 내용을 추가합니다.
|
||||
|
||||
["source","sh",subs="attributes,callouts"]
|
||||
--------------------------------------------------
|
||||
[logstash-{major-version}]
|
||||
name=Elastic repository for {major-version} packages
|
||||
baseurl=https://artifacts.elastic.co/packages/{major-version}/yum
|
||||
gpgcheck=1
|
||||
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
|
||||
enabled=1
|
||||
autorefresh=1
|
||||
type=rpm-md
|
||||
--------------------------------------------------
|
||||
|
||||
그러면 리포지토리는 사용 가능한 상태가 됩니다. 다음 명령으로 설치할 수 있습니다.
|
||||
|
||||
[source,sh]
|
||||
--------------------------------------------------
|
||||
sudo yum install logstash
|
||||
--------------------------------------------------
|
||||
|
||||
WARNING: 이 리포지토리는 CentOS5와 같이 아직 RPM v3를 사용하는 오래된 RPM 기반 배포판을 지원하지 않습니다.
|
||||
|
||||
Logstash를 시스템 서비스로 관리하는 것에 대한 자세한 내용은 {logstash}running-logstash.html[Logstash 실행] 문서를 참조하십시오.
|
||||
|
||||
endif::[]
|
||||
|
||||
==== Docker
|
||||
|
||||
Logstash를 Docker 컨테이너로 실행할 수 있도록 이미지가 제공됩니다. Elastic Docker 레지스트리에서 사용할 수 있습니다. Logstash Docker 컨테이너를 구성하고 실행하는 방법에 대한 자세한 내용은 {logstash}docker.html[Running Logstash on Docker]를 참조하십시오.
|
||||
|
||||
[[first-event]]
|
||||
=== 첫 번째 이벤트 보관
|
||||
|
||||
먼저 가장 기본적인 _Logstash 파이프라인_을 실행하여 Logstash 설치를 테스트해보겠습니다.
|
||||
|
||||
Logstash 파이프라인에는 2개의 필수 요소 `input`과 `output`, 1개의 선택 요소 `filter`가 있습니다. 입력 플러그인은 소스의 데이터를 사용하고, 필터 플러그인은 사용자가 지정한 대로 데이터를 수정하며, 출력 플러그인은 목적지에 데이터를 기록합니다.
|
||||
|
||||
//TODO: 새 이미지로 대체
|
||||
|
||||
image::static/images/basic_logstash_pipeline.png[]
|
||||
|
||||
Logstash 설치를 테스트하기 위해 가장 기본적인 Logstash 파이프라인을 실행합니다. 예:
|
||||
|
||||
["source","sh",subs="attributes"]
|
||||
--------------------------------------------------
|
||||
cd logstash-{logstash_version}
|
||||
bin/logstash -e 'input { stdin { } } output { stdout {} }'
|
||||
--------------------------------------------------
|
||||
|
||||
NOTE: `bin` 디렉토리의 위치는 플랫폼에 따라 다릅니다. 시스템에서 `bin\logstash`의 위치를 찾으려면
|
||||
{logstash}dir-layout.html[Logstash Directory Layout]을 참조하십시오.
|
||||
|
||||
`-e` 플래그를 사용하면 명령행에서 직접 구성을 지정할 수 있습니다. 명령행에서 구성을 지정함으로써 반복 사이에 파일을 편집할 필요 없이 신속하게 구성을 테스트할 수 있습니다.
|
||||
예시된 파이프라인은 표준 입력, `stdin`으로부터 입력을 받아 정형화된 형태로 표준 출력, `stdout`로 이동합니다.
|
||||
|
||||
Logstash를 시작하고 "파이프라인 메인 시작했음"이 표시될 때까지 기다린 다음 명령 프롬프트에 `hello world`를 입력합니다.
|
||||
|
||||
[source,shell]
|
||||
hello world
|
||||
2013-11-21T01:22:14.405+0000 0.0.0.0 hello world
|
||||
|
||||
Logstash는 메시지에 타임스탬프 및 IP 주소 정보를 추가합니다. Logstash가 실행 중인 셸에서 *CTRL-D* 명령을 실행하여 Logstash를 종료합니다.
|
||||
|
||||
축하합니다! 기본 Logstash 파이프라인을 생성하고 실행했습니다. 이제 더 현실적인 파이프라인을 생성하는 방법을 알아보겠습니다.
|
||||
BIN
docs/kr/static/images/basic_logstash_pipeline.png
Normal file
BIN
docs/kr/static/images/basic_logstash_pipeline.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 39 KiB |
BIN
docs/kr/static/images/deploy_1.png
Normal file
BIN
docs/kr/static/images/deploy_1.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
BIN
docs/kr/static/images/deploy_2.png
Normal file
BIN
docs/kr/static/images/deploy_2.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 45 KiB |
BIN
docs/kr/static/images/deploy_3.png
Normal file
BIN
docs/kr/static/images/deploy_3.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 41 KiB |
BIN
docs/kr/static/images/deploy_4.png
Normal file
BIN
docs/kr/static/images/deploy_4.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 56 KiB |
BIN
docs/kr/static/images/deploy_5.png
Normal file
BIN
docs/kr/static/images/deploy_5.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 105 KiB |
BIN
docs/kr/static/images/deploy_6.png
Normal file
BIN
docs/kr/static/images/deploy_6.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 164 KiB |
BIN
docs/kr/static/images/deploy_7.png
Normal file
BIN
docs/kr/static/images/deploy_7.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 172 KiB |
BIN
docs/kr/static/images/kibana-filebeat-data.png
Normal file
BIN
docs/kr/static/images/kibana-filebeat-data.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 179 KiB |
BIN
docs/kr/static/images/logstash.png
Normal file
BIN
docs/kr/static/images/logstash.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 51 KiB |
BIN
docs/kr/static/images/pipeline_correct_load.png
Normal file
BIN
docs/kr/static/images/pipeline_correct_load.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 118 KiB |
BIN
docs/kr/static/images/pipeline_overload.png
Normal file
BIN
docs/kr/static/images/pipeline_overload.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 113 KiB |
117
docs/kr/static/introduction.asciidoc
Normal file
117
docs/kr/static/introduction.asciidoc
Normal file
|
|
@ -0,0 +1,117 @@
|
|||
[float]
|
||||
[[power-of-logstash]]
|
||||
== Logstash의 강점
|
||||
|
||||
*Elasticsearch 등을 위한 강력한 수집 기능*
|
||||
|
||||
뛰어난 Elasticsearch 및 Kibana 시너지 효과를 발휘하는, 수평 확장이 가능한 데이터 처리 파이프라인
|
||||
|
||||
*플러그형 파이프라인 아키텍처*
|
||||
|
||||
다양한 입력, 필터, 출력을 믹스매치하고 조정하면서 파이프라인에서 조화롭게 운용
|
||||
|
||||
*커뮤니티에서 확장 가능하고 개발자에게 편리한 플러그인 에코시스템*
|
||||
|
||||
200여 개 플러그인 사용 가능, 또한 직접 플러그인을 만들어 제공할 수 있는 유연성
|
||||
|
||||
image:static/images/logstash.png[]
|
||||
|
||||
[float]
|
||||
== 데이터의 가치를 높이는 Logstash
|
||||
|
||||
더 많은 데이터를 수집하여 지식을 확장할 수 있습니다. Logstash는 모든 형태 및 규모의 데이터를 환영합니다.
|
||||
|
||||
[float]
|
||||
=== 로그와 메트릭
|
||||
|
||||
모든 것의 출발점입니다.
|
||||
|
||||
* 모든 유형의 로깅 데이터 처리
|
||||
** 다양한 웹 로그(예: {logstash}advanced-pipeline.html[Apache]) 및 애플리케이션 로그(예: {logstash}plugins-inputs-log4j.html[log4j] for Java)를 손쉽게 수집
|
||||
** {logstash}plugins-inputs-syslog.html[syslog], {logstash}plugins-inputs-eventlog.html[Windows 이벤트 로그], 네트워킹 및 방화벽 로그 등 다른 여러 로그 형식도 수집
|
||||
* https://www.elastic.co/products/beats/filebeat[Filebeat]
|
||||
와 연계하여 보충적인 보안 로그 전달 기능 활용 * {logstash}plugins-inputs-ganglia.html[Ganglia], {logstash}plugins-codecs-collectd.html[collectd], {logstash}plugins-codecs-netflow.html[NetFlow], {logstash}plugins-inputs-jmx.html[JMX], 기타 여러 인프라 및 애플리케이션 플랫폼의 메트릭을 {logstash}plugins-inputs-tcp.html[TCP] 및 {logstash}plugins-inputs-udp.html[UDP]를 통해 수집
|
||||
|
||||
[float]
|
||||
=== 웹
|
||||
|
||||
월드 와이드 웹의 진정한 효용성을 실현합니다.
|
||||
|
||||
* {logstash}plugins-inputs-http.html[HTTP 요청]을 이벤트로 변환
|
||||
** 소셜 정서 분석을 위해 {logstash}plugins-inputs-twitter.html[Twitter]와 같은 웹 서비스 파이어호스 활용
|
||||
** GitHub, HipChat, JIRA, 기타 많은 애플리케이션을 위한 웹후크 지원
|
||||
** 다양한 https://www.elastic.co/products/x-pack/alerting[Watcher]알림 활용 사례 지원
|
||||
* 필요에 따라 {logstash}plugins-inputs-http_poller.html[HTTP 엔드포인트] 폴링으로 이벤트 생성
|
||||
** 보편적으로 웹 애플리케이션 인터페이스로부터 상태, 성능, 메트릭, 기타 데이터 유형 수집
|
||||
** 수신보다 폴링 제어를 선호하는 시나리오에 안성맞춤
|
||||
|
||||
[float]
|
||||
=== 데이터 저장소와 스트림
|
||||
|
||||
이미 보유한 데이터에서 더 큰 가치를 발굴할 수 있습니다.
|
||||
|
||||
* {logstash}plugins-inputs-jdbc.html[JDBC] 인터페이스를 통해 관련 데이터베이스 또는 NoSQL 저장소의 데이터를 더 정확히 이해
|
||||
* Apache {logstash}plugins-outputs-kafka.html[Kafka], {logstash}plugins-outputs-rabbitmq.html[RabbitMQ], {logstash}plugins-outputs-sqs.html[Amazon SQS], {logstash}plugins-outputs-zeromq.html[ZeroMQ]
|
||||
와 같은 메시징 큐가 제공하는 각종 데이터 스트림 통합
|
||||
[float]
|
||||
=== 센서와 IoT
|
||||
|
||||
다른 데이터를 폭넓게 탐색합니다.
|
||||
|
||||
* 이 기술 발전의 시대에 거대한 IoT 세상은 연결된 센서로부터 데이터를 수집하고 이용하는 무궁무진한 활용 사례를 내놓고 있습니다.
|
||||
* Logstash는 모바일 디바이스, 지능형 홈, 커넥티드 자동차, 의료 센서, 기타 다양한 산업용 애플리케이션에서 제공하는 데이터를 수집하는 공통 이벤트 수집 백본입니다.
|
||||
|
||||
[float]
|
||||
== 손쉽게 모든 요소 강화
|
||||
|
||||
양질의 데이터로부터 양질의 지식을 얻을 수 있습니다. 색인 또는 출력 시점에 거의 실시간으로 인사이트를 확보할 수 있도록 수집 과정에 데이터를 정리하고 변환합니다. Logstash에서는 패턴 일치, 지리적 맵핑, 동적 조회 기능과 함께 여러 집계 및 변이 기능을 즉시 사용할 수 있습니다.
|
||||
|
||||
* {logstash}plugins-filters-grok.html[Grok]은 Logstash 필터의 가장 기본적인 요소로서 비정형 데이터로부터 구조를 도출하는 데 광범위하게 사용됩니다. 웹, 시스템, 네트워킹, 기타 유형의 이벤트 형식을 신속하게 확인할 수 있도록 제공된 여러 통합형 패턴을 활용하십시오.
|
||||
* IP 주소에서 {logstash}plugins-filters-geoip.html[지리 좌표]를 판독하고 {logstash}plugins-filters-date.html[날짜] 복잡성을 정규화하고 {logstash}plugins-filters-kv.html[키-값 쌍] 및 {logstash}plugins-filters-csv.html[CSV] 데이터를 간소화하고 민감한 정보를 {logstash}plugins-filters-anonymize.html[익명 처리]하고 {logstash}plugins-filters-translate.html[로컬 조회] 또는 Elasticsearch {logstash}plugins-filters-elasticsearch.html[쿼리]로 데이터를 한층 더 강화함으로써 영역을 확장할 수 있습니다.
|
||||
* 코덱은 대개 {logstash}plugins-codecs-json.html[JSON] 및 {logstash}plugins-codecs-multiline.html[multiline] 이벤트와 같은 일반적인 이벤트 구조를 손쉽게 처리하는 데 사용됩니다.
|
||||
|
||||
[float]
|
||||
== 보관(Stash) 선택
|
||||
|
||||
데이터가 가장 중요해질 때 데이터를 라우팅합니다. 데이터를 저장, 분석하고 그에 대한 작업을 수행하여 각종 다운스트림 분석 및 운영 활용 사례에서 최고의 효과를 거둡니다.
|
||||
|
||||
[cols="a,a"]
|
||||
|=======================================================================
|
||||
|
|
||||
|
||||
*분석*
|
||||
|
||||
* {logstash}plugins-outputs-elasticsearch.html[Elasticsearch]
|
||||
* {logstash}plugins-outputs-mongodb.html[MongoDB] 및 {logstash}plugins-outputs-riak.html[Riak]와 같은 데이터 저장소
|
||||
|
||||
|
|
||||
|
||||
*아카이빙*
|
||||
|
||||
* {logstash}plugins-outputs-webhdfs.html[HDFS]
|
||||
* {logstash}plugins-outputs-s3.html[S3]
|
||||
* {logstash}plugins-outputs-google_cloud_storage.html[Google Cloud Storage]
|
||||
|
||||
|
|
||||
|
||||
*모니터링*
|
||||
|
||||
* {logstash}plugins-outputs-nagios,Nagios]
|
||||
* {logstash}plugins-outputs-ganglia.html[Ganglia]
|
||||
* {logstash}plugins-outputs-zabbix.html[Zabbix]
|
||||
* {logstash}plugins-outputs-graphite.html[Graphite]
|
||||
* {logstash}plugins-outputs-datadog.html[Datadog]
|
||||
* {logstash}plugins-outputs-cloudwatch.html[CloudWatch]
|
||||
|
||||
|
|
||||
|
||||
*알림*
|
||||
|
||||
* https://www.elastic.co/products/watcher[Watcher](Elasticsearch와 함께)
|
||||
* {logstash}plugins-outputs-email.html[Email]
|
||||
* {logstash}plugins-outputs-pagerduty.html[Pagerduty]
|
||||
* {logstash}plugins-outputs-hipchat.html[HipChat]
|
||||
* {logstash}plugins-outputs-irc.html[IRC]
|
||||
* {logstash}plugins-outputs-sns.html[SNS]
|
||||
|
||||
|=======================================================================
|
||||
Loading…
Add table
Add a link
Reference in a new issue