github-mirrors/logstash
1
0
Fork 0
mirror of https://github.com/elastic/logstash.git synced 2025-04-23 22:27:21 -04:00
Code Issues Projects Releases Packages Wiki Activity

incorporate @wibaa feedback

Browse source
This commit is contained in:
Kurt Hurtado 2014-10-14 15:17:10 -07:00
parent a08f60c270
commit eef2687c62
9 changed files with 48 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files

8
docs/asciidoc/static/command-line-flags.asciidoc
View file

@ -2,15 +2,15 @@
== Agent
The logstash agent has the following flags (also try using the '--help' flag)
The Logstash agent has the following flags (also try using the '--help' flag)
[source,js]
----------------------------------
-f, --config CONFIGFILE
Load the logstash config from a specific file, directory, or a wildcard. If given a directory or wildcard, config files will be read from the directory in alphabetical order.
Load the Logstash config from a specific file, directory, or a wildcard. If given a directory or wildcard, config files will be read from the directory in alphabetical order.
-e CONFIGSTRING
Use the given string as the configuration data. Same syntax as the config file. If not input is specified, 'stdin { type => stdin }' is default. If no output is specified, 'stdout { debug => true }}' is default.
Use the given string as the configuration data. Same syntax as the config file. If not input is specified, 'stdin { type => stdin }' is default. If no output is specified, 'stdout { codec => rubydebug }}' is default.
-w, --filterworkers COUNT
Run COUNT filter workers (default: 1)
@ -32,7 +32,7 @@ The logstash agent has the following flags (also try using the '--help' flag)
'-vv' currently being the highest
--pluginpath PLUGIN_PATH
A colon-delimted path to find other logstash plugins in
A colon-delimited path to find other Logstash plugins in
----------------------------------

9
docs/asciidoc/static/configuration.asciidoc
View file

@ -27,7 +27,7 @@ output {
== Filters and Ordering
For a given event, are applied in the order of appearance in the configuration file.
For a given event, filters are applied in the order of appearance in the configuration file.
== Comments
@ -57,7 +57,8 @@ input {
}
file {
path => "/var/log/apache/access.log" => "apache"
path => "/var/log/apache/access.log"
type => "apache"
}
}
----------------------------------
@ -80,7 +81,7 @@ Examples:
[source,js]
----------------------------------
debug => true
ssl_enable => true
----------------------------------
=== String
@ -94,7 +95,7 @@ Example:
name => "Hello world"
----------------------------------
Single, unquoted words are valid as strings, too, but you should use quotes.
You should use quotes around string values.
=== Number

37
docs/asciidoc/static/contributing-to-logstash.asciidoc
View file

@ -1,24 +1,24 @@
[[contributing-to-logstash]]
== Extending logstash
== Extending Logstash
You can add your own input, output, or filter plugins to logstash.
You can add your own input, output, or filter plugins to Logstash.
If you're looking to extend logstash today, please look at the existing plugins.
If you're looking to extend Logstash today, the best way is to look at how some existing plugins are written.
[float]
=== Good examples of plugins
* [inputs/tcp](https://github.com/logstash/logstash/blob/master/lib/logstash/inputs/tcp.rb)
* [filters/multiline](https://github.com/logstash/logstash/blob/master/lib/logstash/filters/multiline.rb)
* [outputs/mongodb](https://github.com/logstash/logstash/blob/master/lib/logstash/outputs/mongodb.rb)
* https://github.com/logstash/logstash/blob/master/lib/logstash/inputs/tcp.rb[inputs/tcp]
* https://github.com/logstash/logstash/blob/master/lib/logstash/filters/multiline.rb[filters/multiline]
* https://github.com/elasticsearch/logstash-contrib/blob/master/lib/logstash/outputs/mongodb.rb[outputs/mongodb]
[float]
=== Common concepts
* The `config_name` sets the name used in the config file.
* The `milestone` sets the milestone number of the plugin. See <../plugin-milestones> for more info.
* The `config` lines define config options.
* The `milestone` sets the milestone number of the plugin. See link:plugin-milestones[Plugin Milestones] for more info.
* The `config` lines define this plugin's configuration options.
* The `register` method is called per plugin instantiation. Do any of your initialization here.
[float]
@ -62,11 +62,15 @@ Boolean).
[float]
=== Inputs
All inputs require the LogStash::Inputs::Base class:
All inputs require and extend the LogStash::Inputs::Base class, like so:
[source,js]
----------------------------------
require 'logstash/inputs/base'
...
class LogStash::Inputs::YourPlugin < LogStash::Inputs::Base
...
----------------------------------
Inputs have two methods: `register` and `run`.
@ -77,11 +81,15 @@ Inputs have two methods: `register` and `run`.
[float]
=== Filters
All filters require the LogStash::Filters::Base class:
All filters require and extend the LogStash::Filters::Base class, like so:
[source,js]
----------------------------------
require 'logstash/filters/base'
...
class LogStash::Filters::YourPlugin < LogStash::Filters::Base
...
----------------------------------
Filters have two methods: `register` and `filter`.
@ -94,21 +102,24 @@ Filters have two methods: `register` and `filter`.
[float]
=== Outputs
All outputs require the LogStash::Outputs::Base class:
All outputs require and extend the LogStash::Outputs::Base class, like so:
[source,js]
----------------------------------
require 'logstash/outputs/base'
...
class LogStash::Outputs::YourPlugin < LogStash::Outputs::Base
...
----------------------------------
Outputs have two methods: `register` and `receive`.
* The `register` method is called per plugin instantiation. Do any of your initialization here.
* The `receive` method is called when an event gets pushed to your output
[float]
=== Example: a new filter
Learn by example how to [add a new filter to logstash](example-add-a-new-filter)
Learn by example how to [add a new filter to Logstash](example-add-a-new-filter)

10
docs/asciidoc/static/example-add-a-new-filter.asciidoc
View file

@ -47,12 +47,12 @@ class LogStash::Filters::Foo < LogStash::Filters::Base
public
def register
# nothing to do end # def register
# nothing to do
end # def register
public
def filter(event)
# return nothing unless there's an actual filter event
return unless filter?(event)
if @message
# Replace the event message with our message as configured in the
# config file.
@ -109,9 +109,9 @@ command.
[source,js]
----------------------------------
% bin/logstash -f example.conf
% bin/logstash --pluginpath your/plugin/root -f example.conf
the quick brown fox
2011-05-12T01:05:09.495000Z stdin://snack.home/: Hello world!
2011-05-12T01:05:09.495000Z mylocalhost: Hello world!
----------------------------------
The output is the standard Logstash stdout output, but in this case our "the quick brown fox" message was replaced with "Hello world!"

8
docs/asciidoc/static/getting-started-with-logstash.asciidoc
View file

@ -143,7 +143,7 @@ Another very useful tool for querying your Logstash data (and Elasticsearch in g
----------------------------------
bin/plugin -install lmenezes/elasticsearch-kopf
----------------------------------
Now you can browse to http://localhost:9200/_plugin/kopf[http://localhost:9200/_plugin/kopf] to browse your Elasticsearch data, settings and mappings!
Now you can browse to http://localhost:9200/_plugin/kopf/[http://localhost:9200/_plugin/kopf/] to browse your Elasticsearch data, settings and mappings!
=== Multiple Outputs
As a quick exercise in configuring multiple Logstash outputs, let's invoke Logstash again, using both the 'stdout' as well as the 'elasticsearch' output:
@ -155,7 +155,7 @@ bin/logstash -e 'input { stdin { } } output { elasticsearch { host => localhost
Typing a phrase will now echo back to your terminal, as well as save in Elasticsearch! (Feel free to verify this using curl or elasticsearch-kopf).
=== Default - Daily Indices
You might notice that Logstash was smart enough to create a new index in Elasticsearch... The default index name is in the form of 'logstash-YYYY.MM.DD', which essentially creates one index per day. At midnight (GMT?), Logstash will automagically rotate the index to a fresh new one, with the new current day's timestamp. This allows you to keep windows of data, based on how far retroactively you'd like to query your log data. Of course, you can always archive (or re-index) your data to an alternate location, where you are able to query further into the past. If you'd like to simply delete old indices after a certain time period, you can use the https://github.com/elasticsearch/curator[Elasticsearch Curator tool].
You might notice that Logstash was smart enough to create a new index in Elasticsearch... The default index name is in the form of 'logstash-YYYY.MM.DD', which essentially creates one index per day. At midnight (UTC), Logstash will automagically rotate the index to a fresh new one, with the new current day's timestamp. This allows you to keep windows of data, based on how far retroactively you'd like to query your log data. Of course, you can always archive (or re-index) your data to an alternate location, where you are able to query further into the past. If you'd like to simply delete old indices after a certain time period, you can use the https://github.com/elasticsearch/curator[Elasticsearch Curator tool].
== Moving On
Now you're ready for more advanced configurations. At this point, it makes sense for a quick discussion of some of the core features of Logstash, and how they interact with the Logstash engine.
@ -278,7 +278,7 @@ You should see something returned to STDOUT which looks like this:
}
----------------------------------
As you can see, Logstash (with help from the *grok* filter) was able to parse the log line (which happens to be in Apache "combined log" format) and break it up into many different discrete bits of information. This will be extremely useful later when we start querying and analyzing our log data... for example, we'll be able to run reports on HTTP response codes, IP addresses, referrers, etc. very easily. There are quite a few grok patterns included with Logstash out-of-the-box, so it's quite likely if you're attempting to parse a fairly common log format, someone has already done the work for you. For more details, see the list of https://github.com/logstash/logstash/blob/master/patterns/grok-patterns[logstash grok patterns] on github.
As you can see, Logstash (with help from the *grok* filter) was able to parse the log line (which happens to be in Apache "combined log" format) and break it up into many different discrete bits of information. This will be extremely useful later when we start querying and analyzing our log data... for example, we'll be able to run reports on HTTP response codes, IP addresses, referrers, etc. very easily. There are quite a few grok patterns included with Logstash out-of-the-box, so it's quite likely if you're attempting to parse a fairly common log format, someone has already done the work for you. For more details, see the list of https://github.com/logstash/logstash/blob/master/patterns/[logstash grok patterns] on github.
The other filter used in this example is the *date* filter. This filter parses out a timestamp and uses it as the timestamp for the event (regardless of when you're ingesting the log data). You'll notice that the @timestamp field in this example is set to December 11, 2013, even though Logstash is ingesting the event at some point afterwards. This is handy when backfilling logs, for example... the ability to tell Logstash "use this value as the timestamp for this event".
@ -292,7 +292,7 @@ Now, let's configure something actually *useful*... apache2 access log files! We
input {
file {
path => "/tmp/access_log"
start_position => beginning
start_position => "beginning"
}
}

2
docs/asciidoc/static/howtos-and-tutorials.asciidoc
View file

@ -6,7 +6,7 @@ Pretty self-explanatory, really
* http://elasticsearch.org/#[Getting Started with Logstash]
* http://elasticsearch.org/#[Configuration file overview]
* http://elasticsearch.org/#[Command-line flags]
* http://elasticsearch.org/#[The life of an event in logstash]
* http://elasticsearch.org/#[The life of an event in Logstash]
* http://elasticsearch.org/#[Using conditional logic]
* http://elasticsearch.org/#[Glossary]
* http://elasticsearch.org/#[referring to fields `[like][this]`]

4
docs/asciidoc/static/logstash-docs-home.asciidoc
View file

@ -3,7 +3,7 @@
Pretty self-explanatory, really
=== Downloads and Releases
* http://www.elasticsearch.org/overview/logstash/download/[Download logstash 1.4.2]
* http://www.elasticsearch.org/overview/logstash/download/[Download Logstash 1.4.2]
* http://www.elasticsearch.org/blog/apt-and-yum-repositories/[package repositories]
* http://www.elasticsearch.org/blog/logstash-1-4-2/[release notes]
* https://github.com/elasticsearch/logstash/blob/master/CHANGELOG[view changelog]
@ -20,7 +20,7 @@ Pretty self-explanatory, really
* http://elasticsearch.org/#[Getting Started with Logstash]
* http://elasticsearch.org/#[Configuration file overview]
* http://elasticsearch.org/#[Command-line flags]
* http://elasticsearch.org/#[The life of an event in logstash]
* http://elasticsearch.org/#[The life of an event in Logstash]
* http://elasticsearch.org/#[Using conditional logic]
* http://elasticsearch.org/#[Glossary]
* http://elasticsearch.org/#[(more)]

4
docs/asciidoc/static/logstash-glossary.asciidoc
View file

@ -1,5 +1,5 @@
== bar
bar bar
== Glossary
Logstash Glossary
apache ::
A very common open source web server application, which produces logs easily consumed by Logstash (Apache Common/Combined Log Format).

2
docs/asciidoc/static/plugin-milestones.asciidoc
View file

@ -1,7 +1,7 @@
= Plugin Milestones
== Why Milestones?
Plugins (inputs/outputs/filters/codecs) have a milestone label in logstash. This is to provide an indicator to the end-user as to the kinds of changes a given plugin could have between logstash releases.
Plugins (inputs/outputs/filters/codecs) have a milestone label in Logstash. This is to provide an indicator to the end-user as to the kinds of changes a given plugin could have between Logstash releases.
The desire here is to allow plugin developers to quickly iterate on possible new plugins while conveying to the end-user a set of expectations about that plugin.