logstash/docs/static/include/pluginbody.asciidoc

=== How to write a Logstash {plugintype} plugin

To develop a new {plugintype} for Logstash, you build a self-contained Ruby gem
whose source code lives in its own GitHub repository. The Ruby gem can then be
hosted and shared on RubyGems.org. You can use the example {plugintype}
implementation as a starting point. (If you're unfamiliar with
Ruby, you can find an excellent quickstart guide at
https://www.ruby-lang.org/en/documentation/quickstart/[].)

==== Get started

{getstarted}

==== Create a GitHub repo for your new plugin
Each Logstash plugin lives in its own GitHub repository. To create a new repository for your plugin:

. Log in to GitHub.
. Click the **Repositories** tab. You'll see a list of other repositories you've forked or contributed to.
. Click the green **New** button in the upper right.
. Specify the following settings for your new repo:
** **Repository name** -- a unique name of the form +logstash-pass:attributes[{plugintype}]-pluginname+.
** **Public or Private** -- your choice, but the repository must be Public if you want to submit it as an official plugin.
** **Initialize this repository with a README** -- enables you to immediately clone the repository to your computer.
. Click **Create Repository**.

==== Use the plugin generator tool

You can now create your own Logstash plugin in seconds! The `generate` subcommand of `bin/logstash-plugin` creates the foundation 
for a new Logstash plugin with templatized files. It creates the correct directory structure, gemspec files, and dependencies so you 
can start adding custom code to process data with Logstash.

For more information, see <<plugin-generator>>

==== Copy the {plugintype} code

Alternatively, you can use the examples repo we host on github.com

. **Clone your plugin.** Replace `GITUSERNAME` with your github username, and
`MYPLUGINNAME` with your plugin name.
** `git clone https://github.com/GITUSERNAME/logstash-`+pass:attributes[{plugintype}]-MYPLUGINNAME.git+
*** alternately, via ssh: `git clone git@github.com:GITUSERNAME/logstash`+-pass:attributes[{plugintype}]-MYPLUGINNAME.git+
** +cd logstash-pass:attributes[{plugintype}]-MYPLUGINNAME+

. **Clone the {inputtype} plugin example and copy it to your plugin branch.**
+
You don't want to include the example .git directory or its contents, so delete
it before you copy the example.
+
** `cd /tmp`
** `git clone https://github.com/logstash-plugins/logstash`+-{plugintype}-{pluginname}.git+
** +cd logstash-pass:attributes[{plugintype}]-pass:attributes[{pluginname}]+
** +rm -rf .git+
** +cp -R * /path/to/logstash-pass:attributes[{plugintype}]-mypluginname/+

. **Rename the following files to match the name of your plugin.**
** +logstash-pass:attributes[{plugintype}]-pass:attributes[{pluginname}].gemspec+
** +pass:attributes[{pluginname}].rb+
** +pass:attributes[{pluginname}]_spec.rb+
+
[source,txt]
[subs="attributes"]
----
cd /path/to/logstash-{plugintype}-mypluginname
mv logstash-{plugintype}-{pluginname}.gemspec logstash-{plugintype}-mypluginname.gemspec
mv lib/logstash/{plugintype}s/example.rb lib/logstash/{plugintype}s/mypluginname.rb
mv spec/{plugintype}s/{pluginname}_spec.rb spec/{plugintype}s/mypluginname_spec.rb
----

Your file structure should look like this:

[source,txt]
[subs="attributes"]
----------------------------------
$ tree logstash-{plugintype}-mypluginname
├── Gemfile
├── LICENSE
├── README.md
├── Rakefile
├── lib
│   └── logstash
│       └── {plugintype}s
│           └── mypluginname.rb
├── logstash-{plugintype}-mypluginname.gemspec
└── spec
    └── {plugintype}s
        └── mypluginname_spec.rb
----------------------------------

For more information about the Ruby gem file structure and an excellent
walkthrough of the Ruby gem creation process, see
http://timelessrepo.com/making-ruby-gems

==== See what your plugin looks like

Before we dive into the details, open up the plugin file in your favorite text editor
and take a look.

// /////////////////////////////////////////////////////////////////////////////
// Input (conditionally recognized by the presence of the run_method attribute)
// /////////////////////////////////////////////////////////////////////////////
ifdef::run_method[]
ifndef::blockinput[]
[source,ruby]
[subs="attributes"]
----------------------------------
# encoding: utf-8
require "logstash/{plugintype}s/base"
require "logstash/namespace"
require "stud/interval"
require "socket" # for Socket.gethostname

# Add any asciidoc formatted documentation here
# Generate a repeating message.
#
# This plugin is intended only as an example.

class LogStash::{pluginclass}::{pluginnamecap} < LogStash::{pluginclass}::Base
  config_name "example"

  # If undefined, Logstash will complain, even if codec is unused.
  default :codec, "plain"

  # The message string to use in the event.
  config :message, :validate => :string, :default => "Hello World!"

  # Set how frequently messages should be sent.
  #
  # The default, `1`, means send a message every second.
  config :interval, :validate => :number, :default => 1

  public
  def register
    @host = Socket.gethostname
  end # def register

  def run(queue)
    Stud.interval(@interval) do
      event = LogStash::Event.new("message" => @message, "host" => @host)
      decorate(event)
      queue << event
    end # loop
  end # def run

end # class LogStash::{pluginclass}::{pluginnamecap}
----------------------------------
endif::blockinput[]
endif::run_method[]

// /////////////////////////////////////////////////////////////////////////////
// Codec (conditionally recognized by the presence of the encode_method
// attribute)
// /////////////////////////////////////////////////////////////////////////////
ifdef::encode_method[]
ifndef::blockcodec[]
[source,ruby]
[subs="attributes"]
----------------------------------
# encoding: utf-8
require "logstash/{plugintype}s/base"
require "logstash/codecs/line"

# Add any asciidoc formatted documentation here
class LogStash::{pluginclass}::{pluginnamecap} < LogStash::{pluginclass}::Base

  # This example codec will append a string to the message field
  # of an event, either in the decoding or encoding methods
  #
  # This is only intended to be used as an example.
  #
  # input {
  #   stdin { codec => example }
  # }
  #
  # or
  #
  # output {
  #   stdout { codec => example }
  # }
  config_name "example"

  # Append a string to the message
  config :append, :validate => :string, :default => ', Hello World!'

  public
  def register
    @lines = LogStash::Codecs::Line.new
    @lines.charset = "UTF-8"
  end

  public
  def decode(data)
    @lines.decode(data) do |line|
      replace = { "message" => line["message"].to_s + @append }
      yield LogStash::Event.new(replace)
    end
  end # def decode

  public
  def encode(event)
    @on_event.call(event, event.get("message").to_s + @append + NL)
  end # def encode

end # class LogStash::{pluginclass}::{pluginnamecap}
----------------------------------
endif::blockcodec[]
endif::encode_method[]

// /////////////////////////////////////////////////////////////////////////////
// Filter (conditionally recognized by the presence of the filter_method
// attribute)
// /////////////////////////////////////////////////////////////////////////////
ifdef::filter_method[]
ifndef::blockfilter[]
[source,ruby]
[subs="attributes"]
----------------------------------
# encoding: utf-8
require "logstash/{plugintype}s/base"
require "logstash/namespace"

# Add any asciidoc formatted documentation here
# This example filter will replace the contents of the default
# message field with whatever you specify in the configuration.
#
# It is only intended to be used as an example.
class LogStash::{pluginclass}::{pluginnamecap} < LogStash::{pluginclass}::Base

  # Setting the config_name here is required. This is how you
  # configure this {plugintype} from your Logstash config.
  #
  # {plugintype} {
  #   {pluginname} { message => "My message..." }
  # }
  config_name "example"

  # Replace the message with this value.
  config :message, :validate => :string, :default => "Hello World!"


  public
  def register
    # Add instance variables
  end # def register

  public
  def filter(event)

    if @message
      # Replace the event message with our message as configured in the
      # config file.
      event.set("message", @message)
    end

    # filter_matched should go in the last line of our successful code
    filter_matched(event)
  end # def {plugintype}

end # class LogStash::{pluginclass}::{pluginnamecap}
----------------------------------
endif::blockfilter[]
endif::filter_method[]

// /////////////////////////////////////////////////////////////////////////////
// Output (conditionally recognized by the presence of the receive_method
// attribute)
// /////////////////////////////////////////////////////////////////////////////
ifdef::multi_receive_method[]
[source,ruby]
[subs="attributes"]
----------------------------------
# encoding: utf-8
require "logstash/{plugintype}s/base"
require "logstash/namespace"

# Add any asciidoc formatted documentation here
# An example output that does nothing.
class LogStash::{pluginclass}::{pluginnamecap} < LogStash::{pluginclass}::Base
  config_name "example"

  # This sets the concurrency behavior of this plugin. By default it is :legacy, which was the standard
  # way concurrency worked before Logstash 2.4
  # 
  # You should explicitly set it to either :single or :shared as :legacy will be removed in Logstash 6.0
  # 
  # When configured as :single a single instance of the Output will be shared among the
  # pipeline worker threads. Access to the `#multi_receive/#multi_receive_encoded/#receive` method will be synchronized
  # i.e. only one thread will be active at a time making threadsafety much simpler.
  # 
  # You can set this to :shared if your output is threadsafe. This will maximize
  # concurrency but you will need to make appropriate uses of mutexes in `#multi_receive/#receive`.
  #
  # Only the `#multi_receive/#multi_receive_encoded` methods need to actually be threadsafe, the other methods
  # will only be executed in a single thread
  concurrency :single
  
  public
  def register    
  end # def register

  public
  # Takes an array of events
  # Must be threadsafe if `concurrency :shared` is set
  def multi_receive(events)
  end # def multi_receive
end # class LogStash::{pluginclass}::{pluginnamecap}
----------------------------------
endif::multi_receive_method[]

==== Coding {plugintype} plugins

Now let's take a line-by-line look at the example plugin.

==== `encoding`

It seems like a small thing, but remember to specify the encoding at the
beginning of your plugin code:

[source,sh]
----------------------------------
# encoding: utf-8
----------------------------------

Logstash depends on things being in UTF-8, so we put this here to tell the Ruby
interpreter that we’re going to be using the UTF-8 encoding.

==== `require` Statements

Logstash {plugintype} plugins require parent classes defined in
+logstash/pass:attributes[{plugintype}]s/base+ and logstash/namespace:

[source,ruby]
[subs="attributes"]
----------------------------------
require "logstash/{plugintype}s/base"
require "logstash/namespace"
----------------------------------

Of course, the plugin you build may depend on other code, or even gems. Just put
them here along with these Logstash dependencies.

==== Plugin Body

Let's go through the various elements of the plugin itself.

==== Inline Documentation
Logstash provides infrastructure to automatically generate documentation for
plugins. We use the asciidoc format to write documentation so _any_ comments in
the source code will be first converted into asciidoc and then into html.

All plugin documentation is then rendered and placed in
http://www.elasticsearch.org/guide/en/logstash/current/index.html[the Logstash section of the Elasticsearch Guide].

The inline documentation can include code blocks and config examples! To include
Ruby code, use the asciidoc `[source,ruby]` directive:

[source,txt]
[subs="attributes"]
----------------------------------
# Using hashes:
# [source,ruby]
# ----------------------------------
# match => {
#  "field1" => "value1"
#  "field2" => "value2"
#  ...
# }
# ----------------------------------
----------------------------------

In the rendered HTML document, this block would look like:

[]
=========================
Using hashes:
[source,ruby]
----------------------------------
match => {
  "field1" => "value1"
  "field2" => "value2"
  ...
 }
----------------------------------
=========================


TIP: For more asciidoc formatting tips, see the excellent reference at
https://github.com/elastic/docs#asciidoc-guide

==== `class` Declaration
The {plugintype} plugin class should be a subclass of
+LogStash::pass:attributes[{pluginclass}]::Base+:

[source,ruby]
[subs="attributes"]
----------------------------------
class LogStash::{pluginclass}::{pluginnamecap} < LogStash::{pluginclass}::Base
----------------------------------

The class name should closely mirror the plugin name, for example:
[source,ruby]
[subs="attributes"]
----
LogStash::{pluginclass}::{pluginnamecap}
----

==== `config_name`
[source,ruby]
[subs="attributes"]
----------------------------------
  config_name "{pluginname}"
----------------------------------
This is the name your plugin will call inside the {plugintype} configuration
block.

If you set +config_name "pass:attributes[{pluginname}]"+ in your plugin code,
the corresponding Logstash configuration block would need to look like this:

// /////////////////////////////////////////////////////////////////////////////
// If encode_method is NOT defined (not a codec)
// /////////////////////////////////////////////////////////////////////////////
ifndef::encode_method[]
ifndef::blockcodec[]
[source,js]
[subs="attributes"]
----------------------------------
{plugintype} {
  {pluginname} {...}
}
----------------------------------
endif::blockcodec[]
endif::encode_method[]

// /////////////////////////////////////////////////////////////////////////////
// If encode_method IS defined (for codecs only)
// /////////////////////////////////////////////////////////////////////////////
ifdef::encode_method[]
ifndef::blockinput[]
[source,js]
[subs="attributes"]
----------------------------------
input {
  codec => {pluginname} {...}
}
----------------------------------

Or if using the codec in an output block:

[source,js]
[subs="attributes"]
----------------------------------
output {
  codec => {pluginname} {...}
}
----------------------------------
endif::blockinput[]
endif::encode_method[]

==== Configuration Parameters
[source,ruby]
----------------------------------
  config :variable_name, :validate => :variable_type, :default => "Default value", :required => boolean, :deprecated => boolean, :obsolete => string
----------------------------------
The configuration, or `config` section allows you to define as many (or as few)
parameters as are needed to enable Logstash to process events.

There are several configuration attributes:

* `:validate` - allows you to enforce passing a particular data type to Logstash
for this configuration option, such as `:string`, `:password`, `:boolean`,
`:number`, `:array`, `:hash`, `:path` (a file-system path), `uri` (starting in 5.0.0), `:codec` (since
1.2.0), `:bytes` (starting in 1.5.0).  Note that this also works as a coercion 
in that if I specify "true" for boolean (even though technically a string), it
will become a valid boolean in the config.  This coercion works for the
`:number` type as well where "1.2" becomes a float and "22" is an integer.
* `:default` - lets you specify a default value for a parameter
* `:required` - whether or not this parameter is mandatory (a Boolean `true` or `false`)
* `:list` - whether or not this value should be a list of values. Will typecheck the list members, and convert scalars to one element lists. Note that this mostly obviates the array type, though if you need lists of complex objects that will be more suitable. 
* `:deprecated` - informational (also a Boolean `true` or `false`)
* `:obsolete` - used to declare that a given setting has been removed and is no longer functioning. The idea is to provide an informed upgrade path to users who are still using a now-removed setting.

==== Plugin Methods

{methodheader}

// /////////////////////////////////////////////////////////////////////////////
// If register_method is defined (should be all types)
// /////////////////////////////////////////////////////////////////////////////
ifdef::register_method[]

==== `register` Method
[source,ruby]
[subs="attributes"]
----------------------------------
  public
  def register
  end # def register
----------------------------------

The Logstash `register` method is like an `initialize` method. It was originally
created to enforce having `super` called, preventing headaches for newbies.
(Note: It may go away in favor of `initialize`, in conjunction with some
enforced testing to ensure `super` is called.)

`public` means the method can be called anywhere, not just within the class.
This is the default behavior for methods in Ruby, but it is specified explicitly
here anyway.

You can also assign instance variables here (variables prepended by `@`).
Configuration variables are now in scope as instance variables, like `@message`

endif::register_method[]

// /////////////////////////////////////////////////////////////////////////////
// If filter_method is defined (should only be for filter plugin page)
// /////////////////////////////////////////////////////////////////////////////
ifdef::filter_method[]
ifndef::blockfilter[]

==== `filter` Method

[source,ruby]
[subs="attributes"]
----------------------------------
  public
  def filter(event)

    if @message
      # Replace the event message with our message as configured in the
      # config file.
      event.set("message", @message)
    end

  # filter_matched should go in the last line of our successful code
  filter_matched(event)
end # def filter
----------------------------------
The plugin's `filter` method is where the actual filtering work takes place!
Inside the `filter` method you can refer to the event data using the `Event`
object. Event is the main object that encapsulates data flow internally in Logstash 
and provides an <<event-api, API>> for the plugin developers to interact with the 
event's content.

The `filter` method should also handle any <<event-dependent-configuration, event dependent configuration>> by 
explicitly calling the `sprintf` method available in Event class. For example:

[source,ruby]
----------------------------------
field_foo = event.sprintf(field)
----------------------------------

Note that configuration variables are now in scope as instance variables, like
`@message`

[source,ruby]
----------------------------------
  filter_matched(event)
----------------------------------
Calling the `filter_matched` method upon successful execution of the plugin will
ensure that any fields or tags added through the Logstash configuration for this
filter will be handled correctly. For example, any `add_field`, `remove_field`,
`add_tag` and/or `remove_tag` actions will be performed at this time.

Event methods such as `event.cancel` are now available to control the workflow
of the event being processed.
endif::blockfilter[]
endif::filter_method[]

// /////////////////////////////////////////////////////////////////////////////
// If decode_method is defined (should only be for codec plugin page)
// /////////////////////////////////////////////////////////////////////////////
ifdef::decode_method[]
ifndef::blockcodec[]

==== `decode` Method

[source,ruby]
[subs="attributes"]
----------------------------------
  public
  def decode(data)
    @lines.decode(data) do |line|
      replace = { "message" => line["message"].to_s + @append }
      yield LogStash::Event.new(replace)
    end
  end # def decode
----------------------------------
The codec's `decode` method is where data coming in from an input is transformed
into an event.  There are complex examples like the
https://github.com/logstash-plugins/logstash-codec-collectd/blob/master/lib/logstash/codecs/collectd.rb#L386-L484[collectd]
codec, and simpler examples like the https://github.com/logstash-plugins/logstash-codec-spool/blob/master/lib/logstash/codecs/spool.rb#L11-L16[spool]
codec.

There must be a `yield` statement as part of the `decode` method which will
return decoded events to the pipeline.
endif::blockcodec[]
endif::decode_method[]

// /////////////////////////////////////////////////////////////////////////////
// If encode_method is defined (should only be for codec plugin page)
// /////////////////////////////////////////////////////////////////////////////
ifdef::encode_method[]
ifndef::blockcodec[]

==== `encode` Method

[source,ruby]
[subs="attributes"]
----------------------------------
  public
  def encode(event)
    @on_event.call(event, event.get("message").to_s + @append + NL)
  end # def encode
----------------------------------
The `encode` method takes an event and serializes it (_encodes_) into another
format.  Good examples of `encode` methods include the simple https://github.com/logstash-plugins/logstash-codec-plain/blob/master/lib/logstash/codecs/plain.rb#L39-L46[plain]
codec, the slightly more involved https://github.com/logstash-plugins/logstash-codec-msgpack/blob/master/lib/logstash/codecs/msgpack.rb#L38-L46[msgpack]
codec, and even an https://github.com/logstash-plugins/logstash-codec-avro/blob/master/lib/logstash/codecs/avro.rb#L38-L45[avro]
codec.

In most cases, your `encode` method should have an `@on_event.call()` statement.
This call will output data per event in the described way.
endif::blockcodec[]
endif::encode_method[]

// /////////////////////////////////////////////////////////////////////////////
// If run_method is defined (should only be for input plugin page)
// /////////////////////////////////////////////////////////////////////////////
ifdef::run_method[]
ifndef::blockinput[]

==== `run` Method

The {pluginname} input plugin has the following `run` Method:

[source,ruby]
[subs="attributes"]
----------------------------------
  def run(queue)
    Stud.interval(@interval) do
      event = LogStash::Event.new("message" => @message, "host" => @host)
      decorate(event)
      queue << event
    end # loop
  end # def run
----------------------------------
The `run` method is where a stream of data from an input becomes an event.

The stream can be plain or generated as with the
https://github.com/logstash-plugins/logstash-input-heartbeat/blob/master/lib/logstash/inputs/heartbeat.rb#L43-L61[heartbeat]
input plugin.  In these cases, though no codec is used,
https://github.com/logstash-plugins/logstash-input-heartbeat/blob/master/lib/logstash/inputs/heartbeat.rb#L17[a default codec]
must be set in the code to avoid errors.

Here's another example `run` method:
[source,ruby]
[subs="attributes"]
----------------------------------
  def run(queue)
    while true
      begin
        # Based on some testing, there is no way to interrupt an IO.sysread nor
        # IO.select call in JRuby.
        data = $stdin.sysread(16384)
        @codec.decode(data) do |event|
          decorate(event)
          event.set("host", @host) if !event.include?("host")
          queue << event
        end
      rescue IOError, EOFError, LogStash::ShutdownSignal
        # stdin closed or a requested shutdown
        break
      end
    end # while true
    finished
  end # def run
----------------------------------
In this example, the `data` is being sent to the codec defined in the
configuration block to `decode` the data stream and return an event.

In both examples, the resulting `event` is passed to the `decorate` method:
[source,ruby]
[subs="attributes"]
----------------------------------
      decorate(event)
----------------------------------
This applies any tags you might have set in the input configuration block. For
example, `tags => ["tag1", "tag2"]`.

Also in both examples, the `event`, after being "decorated," is appended to the
queue:
[source,ruby]
[subs="attributes"]
----------------------------------
      queue << event
----------------------------------

This inserts the event into the pipeline.

[TIP]
======
Because input plugins can range from simple to complex, it is helpful to see
more examples of how they have been created:

 - https://github.com/logstash-plugins/logstash-input-syslog/blob/master/lib/logstash/inputs/syslog.rb[syslog]
 - https://github.com/logstash-plugins/logstash-input-zeromq/blob/master/lib/logstash/inputs/zeromq.rb[zeromq]
 - https://github.com/logstash-plugins/logstash-input-stdin/blob/master/lib/logstash/inputs/stdin.rb[stdin]
 - https://github.com/logstash-plugins/logstash-input-tcp/blob/master/lib/logstash/inputs/tcp.rb[tcp]

There are many more more examples in the https://github.com/logstash-plugins?query=logstash-input[logstash-plugin github repository].
======
endif::blockinput[]
endif::run_method[]

// /////////////////////////////////////////////////////////////////////////////
// If multi_receive_method is defined (should only be for output plugin page)
// /////////////////////////////////////////////////////////////////////////////
ifdef::receive_method[]

==== `receive` Method

[source,ruby]
[subs="attributes"]
----------------------------------
  public
  def receive(event)
  end # def event
----------------------------------
This empty `receive` method does absolutely nothing, which is valid code from
the https://github.com/logstash-plugins/logstash-output-null/blob/master/lib/logstash/outputs/null.rb[null]
output plugin.

The `receive` method _receives_ events from the pipeline and further processes
them before sending them to their final destination.  These destinations can
be as varied as https://github.com/logstash-plugins/logstash-output-file/blob/master/lib/logstash/outputs/file.rb[files],
https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch.rb[elasticsearch],
https://github.com/logstash-plugins/logstash-output-email/blob/master/lib/logstash/outputs/email.rb[email],
https://github.com/logstash-plugins/logstash-output-rabbitmq/blob/master/lib/logstash/outputs/rabbitmq.rb[RabbitMQ],
https://github.com/logstash-plugins/logstash-output-tcp/blob/master/lib/logstash/outputs/tcp.rb[tcp],
https://github.com/logstash-plugins/logstash-output-stdout/blob/master/lib/logstash/outputs/stdout.rb[stdout],
and dozens more!

Output plugins do not require you to make use of codecs, but you can use a
codec by including code similar to this:
[source,ruby]
[subs="attributes"]
----------------------------------
  public
  def receive(event)
    return unless output?(event)
    @codec.encode(event)
  end # def receive
----------------------------------

For more examples of output plugins, see the https://github.com/logstash-plugins?query=logstash-output[logstash-plugin github repository].

endif::receive_method[]

// Teardown is now in the base class... can be pruned?
// /////////////////////////////////////////////////////////////////////////////
// If close_method is defined (should only be for input or output plugin page)
// /////////////////////////////////////////////////////////////////////////////
// ifdef::close_method[]
// [float]
// ==== `close` Method
// [source,ruby]
// [subs="attributes"]
// ----------------------------------
// public
// def close
//   @udp.close if @udp && !@udp.closed?
// end
// ----------------------------------
// The `close` method is not present in all input or output plugins.  It is
// called when a shutdown happens to ensure that sockets, files, connections and
// threads are all closed down properly.  If your plugin uses these connections,
// you should include a close method.
// endif::close_method[]

==== Building the Plugin

At this point in the process you have coded your plugin and are ready to build
a Ruby Gem from it.  The following steps will help you complete the process.

==== External dependencies

A `require` statement in Ruby is used to include necessary code. In some cases
your plugin may require additional files.  For example, the collectd plugin
https://github.com/logstash-plugins/logstash-codec-collectd/blob/master/lib/logstash/codecs/collectd.rb#L148[uses]
the `types.db` file provided by collectd.  In the main directory of your plugin,
a file called `vendor.json` is where these files are described.

The `vendor.json` file contains an array of JSON objects, each describing a file
dependency. This example comes from the
https://github.com/logstash-plugins/logstash-codec-collectd/blob/master/vendor.json[collectd]
codec plugin:

[source,txt]
----------------------------------
[{
        "sha1": "a90fe6cc53b76b7bdd56dc57950d90787cb9c96e",
        "url": "http://collectd.org/files/collectd-5.4.0.tar.gz",
        "files": [ "/src/types.db" ]
}]
----------------------------------

** `sha1` is the sha1 signature used to verify the integrity of the file
referenced by `url`.
** `url` is the address from where Logstash will download the file.
** `files` is an optional array of files to extract from the downloaded file.
Note that while tar archives can use absolute or relative paths, treat them as
absolute in this array.  If `files` is not present, all files will be
uncompressed and extracted into the vendor directory.

Another example of the `vendor.json` file is the
https://github.com/logstash-plugins/logstash-filter-geoip/blob/master/vendor.json[`geoip` filter]

The process used to download these dependencies is to call `rake vendor`.  This
will be discussed further in the testing section of this document.

Another kind of external dependency is on jar files.  This will be described
in the "Add a `gemspec` file" section.

==== Add a Gemfile

Gemfiles allow Ruby's Bundler to maintain the dependencies for your plugin.
Currently, all we'll need is the Logstash gem, for testing, but if you require
other gems, you should add them in here.

TIP: See http://bundler.io/gemfile.html[Bundler's Gemfile page] for more details.

[source,ruby]
[subs="attributes"]
----------------------------------
source 'https://rubygems.org'
gemspec
gem "logstash", :github => "elastic/logstash", :branch => "{branch}"
----------------------------------

==== Add a `gemspec` file
Gemspecs define the Ruby gem which will be built and contain your plugin.

TIP: More information can be found on the
http://guides.rubygems.org/specification-reference/[Rubygems Specification page].

[source,ruby]
[subs="attributes"]
----------------------------------
Gem::Specification.new do |s|
  s.name = 'logstash-{plugintype}-{pluginname}'
  s.version = '0.1.0'
  s.licenses = ['Apache License (2.0)']
  s.summary = "This {plugintype} does x, y, z in Logstash"
  s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
  s.authors = ["Elastic"]
  s.email = 'info@elastic.co'
  s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
  s.require_paths = ["lib"]

  # Files
  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
   # Tests
  s.test_files = s.files.grep(%r{^(test|spec|features)/})

  # Special flag to let us know this is actually a logstash plugin
  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "{plugintype}" }

  # Gem dependencies
  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
  s.add_development_dependency 'logstash-devutils'
end
----------------------------------

It is appropriate to change these values to fit your plugin. In particular,
`s.name` and `s.summary` shoud reflect your plugin's name and behavior.

`s.licenses` and `s.version` are also important and will come into play when
you are ready to publish your plugin.

Logstash and all its plugins are licensed under
https://github.com/elastic/logstash/blob/master/LICENSE[Apache License, version 2 ("ALv2")].
If you make your plugin publicly available via http://rubygems.org[RubyGems.org],
please make sure to have this line in your gemspec:

* `s.licenses = ['Apache License (2.0)']`

The gem version, designated by `s.version`, helps track changes to plugins over
time. You should use http://semver.org/[semver versioning] strategy for version numbers. 

==== Runtime & Development Dependencies

At the bottom of the `gemspec` file is a section with a comment:
`Gem dependencies`.  This is where any other needed gems must be mentioned. If
a gem is necessary for your plugin to function, it is a runtime dependency. If
a gem are only used for testing, then it would be a development dependency.

[NOTE]
=========================
You can also have versioning requirements for your dependencies--including other
Logstash plugins:

[source,ruby]
[subs="attributes"]
----------------------------------
  # Gem dependencies
  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
  s.add_development_dependency 'logstash-devutils'
----------------------------------
This gemspec has a runtime dependency on the logstash-core-plugin-api and requires that
it have a version number greater than or equal to version 1.60 and less than or equal to version 2.99.
=========================


IMPORTANT: All plugins have a runtime dependency on the `logstash-core-plugin-api` gem, and
a development dependency on `logstash-devutils`.

==== Jar dependencies

In some cases, such as the
https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/logstash-output-elasticsearch.gemspec#L22-L23[Elasticsearch output plugin],
your code may depend on a jar file.  In cases such as this, the dependency is
added in the gemspec file in this manner:

[source,ruby]
[subs="attributes"]
----------------------------------
  # Jar dependencies
  s.requirements << "jar 'org.elasticsearch:elasticsearch', '5.0.0'"
  s.add_runtime_dependency 'jar-dependencies'
----------------------------------

With these both defined, the install process will search for the required jar
file at http://mvnrepository.com and download the specified version.

==== Add Tests

Logstash loves tests. Lots of tests. If you're using your new {plugintype}
plugin in a production environment, you'll want to have some tests to ensure you
are not breaking any existing functionality.

NOTE: A full exposition on RSpec is outside the scope of this document. Learn
more about RSpec at http://rspec.info

For help learning about tests and testing, look in the
+spec/pass:attributes[{plugintype}]s/+ directory of several other similar
plugins.

==== Clone and test!

Now let's start with a fresh clone of the plugin, build it and run the tests.

* **Clone your plugin into a temporary location** Replace `GITUSERNAME` with
your github username, and `MYPLUGINNAME` with your plugin name.
** `git clone https://github.com/GITUSERNAME/logstash-`+pass:attributes[{plugintype}]-MYPLUGINNAME.git+
*** alternately, via ssh: `git clone git@github.com:GITUSERNAME/logstash-`+pass:attributes[{plugintype}]-MYPLUGINNAME.git+
** +cd logstash-pass:attributes[{plugintype}]-MYPLUGINNAME+

Then, you'll need to install your plugins dependencies with bundler:

----------------------------------
bundle install
----------------------------------

[IMPORTANT]
======

If your plugin has an external file dependency described in `vendor.json`, you
must download that dependency before running or testing.  You can do this by
running:

----------------------------------
rake vendor
----------------------------------
======

And finally, run the tests:

----------------------------------
bundle exec rspec
----------------------------------

You should see a success message, which looks something like this:

----------------------------------
Finished in 0.034 seconds
1 example, 0 failures
----------------------------------

Hooray! You're almost there! (Unless you saw failures... you should fix those
  first).

==== Building and Testing

Now you're ready to build your (well-tested) plugin into a Ruby gem.

==== Build
You already have all the necessary ingredients, so let's go ahead and run the
build command:

[source,sh]
[subs="attributes"]
----------------------------------
gem build logstash-{plugintype}-{pluginname}.gemspec
----------------------------------

That's it!  Your gem should be built and be in the same path with the name
[source,sh]
[subs="attributes"]
----------------------------------
logstash-{plugintype}-mypluginname-0.1.0.gem
----------------------------------
The `s.version` number from your gemspec file will provide the gem version, in
this case, `0.1.0`.

==== Test installation

You should test install your plugin into a clean installation of Logstash.
Download the latest version from the
https://www.elastic.co/downloads/logstash/[Logstash downloads page].

. Untar and cd in to the directory:
+
[source,sh]
[subs="attributes"]
----------------------------------
curl -O https://download.elastic.co/logstash/logstash/logstash-{logstash_version}.tar.gz
tar xzvf logstash-{logstash_version}.tar.gz
cd logstash-{logstash_version}
----------------------------------
. Using the plugin tool, we can install the gem we just built.
+
* Replace `/my/logstash/plugins` with  the correct path to the gem for your
environment, and `0.1.0` with the correct version number from the gemspec file.
+
[source,sh]
[subs="attributes"]
----------------------------------
bin/logstash-plugin install /my/logstash/plugins/logstash-{plugintype}-{pluginname}/logstash-{plugintype}-{pluginname}-0.1.0.gem
----------------------------------
+
* After running this, you should see feedback from Logstash that it was
successfully installed:
+
[source,sh]
[subs="attributes"]
----------------------------------
validating /my/logstash/plugins/logstash-{plugintype}-{pluginname}/logstash-{plugintype}-{pluginname}-0.1.0.gem >= 0
Valid logstash plugin. Continuing...
Successfully installed 'logstash-{plugintype}-{pluginname}' with version '0.1.0'
----------------------------------
+
[TIP]
=======
You can also use the Logstash plugin tool to determine which plugins are
currently available:

[source,sh]
----------------------------------
bin/logstash-plugin list
----------------------------------
Depending on what you have installed, you might see a short or long list of
plugins: inputs, codecs, filters and outputs.
=======
+
. Now try running Logstash with a simple configuration passed in via the
command-line, using the `-e` flag.
[NOTE]
Your results will depend on what your {plugintype} plugin is designed to do.

ifdef::run_method[]
ifndef::blockinput[]
[source,sh]
[subs="attributes"]
----------------------------------
bin/logstash -e 'input { {pluginname}{} } output {stdout { codec => rubydebug }}'
----------------------------------

The {pluginname} {plugintype} plugin will send the contents of `message` (with a
default message of "Hello World!") every second.

[source,sh]
[subs="attributes"]
----------------------------------
{
       "message" => "Hello World!",
      "@version" => "1",
    "@timestamp" => "2015-01-27T19:17:18.932Z",
          "host" => "cadenza"
}
----------------------------------

Feel free to experiment and test this by changing the `message` and `interval` parameters:
[source,sh]
[subs="attributes"]
----------------------------------
bin/logstash -e 'input { {pluginname}{ message => "A different message" interval => 5 } } output {stdout { codec => rubydebug }}'
----------------------------------
endif::blockinput[]
endif::run_method[]

ifdef::encode_method[]
ifndef::blockcodec[]
[source,sh]
[subs="attributes"]
----------------------------------
bin/logstash -e 'input { stdin{ codec => {pluginname}{}} } output {stdout { codec => rubydebug }}'
----------------------------------

The {pluginname} {plugintype} plugin will append the contents of `append` (which
by default appends ", Hello World!")

After starting Logstash, type something, for example "Random output string".
The resulting output message field contents should be,
"Random output string, Hello World!":

[source,sh]
[subs="attributes"]
----------------------------------
Random output string
{
       "message" => "Random output string, Hello World!",
      "@version" => "1",
    "@timestamp" => "2015-01-27T19:17:18.932Z",
          "host" => "cadenza"
}
----------------------------------

Feel free to experiment and test this by changing the `append` parameter:
[source,sh]
[subs="attributes"]
----------------------------------
bin/logstash -e 'input { stdin{ codec => {pluginname}{ append => ", I am appending this! }} } output {stdout { codec => rubydebug }}'
----------------------------------

endif::blockcodec[]
endif::encode_method[]

ifdef::filter_method[]
ifndef::blockfilter[]
[source,sh]
[subs="attributes"]
----------------------------------
bin/logstash -e 'input { stdin{} } {plugintype} { {pluginname} {} } output {stdout { codec => rubydebug }}'
----------------------------------

Test your filter by sending input through `stdin` and output (after filtering)
through `stdout` with the `rubydebug` codec, which enhances readability.

In the case of the {pluginname} {plugintype} plugin, any text you send will be
replaced by the contents of the `message` configuration parameter, the default
value being "Hello World!":

[source,sh]
[subs="attributes"]
----------------------------------
Testing 1, 2, 3
{
       "message" => "Hello World!",
      "@version" => "1",
    "@timestamp" => "2015-01-27T19:17:18.932Z",
          "host" => "cadenza"
}
----------------------------------

Feel free to experiment and test this by changing the `message` parameter:
[source,sh]
[subs="attributes"]
----------------------------------
bin/logstash -e 'input { stdin{} } {plugintype} { {pluginname} { message => "This is a new message!"} } output {stdout { codec => rubydebug }}'
----------------------------------

endif::blockfilter[]
endif::filter_method[]

ifdef::receive_method[]

It is harder to display console proof of a working output--with the notable
exception of the `stdout` plugin.

[source,sh]
[subs="attributes"]
----------------------------------
bin/logstash -e 'input { stdin {} } output { {pluginname}{} }'
----------------------------------

Depending on your plugin, you may or may not be immediately able to tell it is
working.  If your output plugin sends to a network socket, you may want to set
up a listener to test the results.  Through your testing, you should be able to
verify that your target is receiving the expected results.

endif::receive_method[]

Congratulations! You've built, deployed and successfully run a Logstash
{plugintype}.

==== Submitting your plugin to http://rubygems.org[RubyGems.org] and https://github.com/logstash-plugins[logstash-plugins]

Logstash uses http://rubygems.org[RubyGems.org] as its repository for all plugin
artifacts. Once you have developed your new plugin, you can make it available to
Logstash users by simply publishing it to RubyGems.org.

==== Licensing
Logstash and all its plugins are licensed under
https://github.com/elasticsearch/logstash/blob/master/LICENSE[Apache License, version 2 ("ALv2")].
If you make your plugin publicly available via http://rubygems.org[RubyGems.org],
please make sure to have this line in your gemspec:

* `s.licenses = ['Apache License (2.0)']`

==== Publishing to http://rubygems.org[RubyGems.org]

To begin, you’ll need an account on RubyGems.org

* https://rubygems.org/sign_up[Sign-up for a RubyGems account].

After creating an account,
http://guides.rubygems.org/rubygems-org-api/#api-authorization[obtain] an API
key from RubyGems.org. By default, RubyGems uses the file `~/.gem/credentials`
to store your API key. These credentials will be used to publish the gem.
Replace `username` and `password` with the credentials you created at
RubyGems.org:

[source,sh]
----------------------------------
curl -u username:password https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials
chmod 0600 ~/.gem/credentials
----------------------------------

Before proceeding, make sure you have the right version in your gemspec file
and commit your changes.

* `s.version = '0.1.0'`

To publish version 0.1.0 of your new logstash gem:

[source,sh]
----------------------------------
bundle install
bundle exec rake vendor
bundle exec rspec
bundle exec rake publish_gem
----------------------------------

[NOTE]
========
Executing `rake publish_gem`:

. Reads the version from the gemspec file (`s.version = '0.1.0'`)
. Checks in your local repository if a tag exists for that version. If the tag
already exists, it aborts the process. Otherwise, it creates a new version tag
in your local repository.
. Builds the gem
. Publishes the gem to RubyGems.org
========

That's it! Your plugin is published! Logstash users can now install your plugin
by running:

[source,sh]
[subs="attributes"]
----------------------------------
bin/logstash-plugin install logstash-{plugintype}-mypluginname
----------------------------------

==== Contributing your source code to https://github.com/logstash-plugins[logstash-plugins]

It is not required to contribute your source code to
https://github.com/logstash-plugins[logstash-plugins] github organization, but
we always welcome new plugins!

==== Benefits

Some of the many benefits of having your plugin in the logstash-plugins
repository are:

* **Discovery** Your plugin will appear in the http://www.elasticsearch.org/guide/en/logstash/current/index.html[Logstash Reference],
where Logstash users look first for plugins and documentation.
* **Documentation** Your plugin documentation will automatically be added to the
 http://www.elasticsearch.org/guide/en/logstash/current/index.html[Logstash Reference].
* **Testing** With our testing infrastructure, your plugin will be continuously
tested against current and future releases of Logstash.  As a result, users will
have the assurance that if incompatibilities arise, they will be quickly
discovered and corrected.

==== Acceptance Guidelines

* **Code Review** Your plugin must be reviewed by members of the community for
coherence, quality, readability, stability and security.
* **Tests** Your plugin must contain tests to be accepted.  These tests are also
subject to code review for scope and completeness.  It's ok if you don't know
how to write tests -- we will guide you. We are working on publishing a guide to
creating tests for Logstash which will make it easier.  In the meantime, you can
refer to http://betterspecs.org/ for examples.

To begin migrating your plugin to logstash-plugins, simply create a new
https://github.com/elasticsearch/logstash/issues[issue] in
the Logstash repository. When the acceptance guidelines are completed, we will
facilitate the move to the logstash-plugins organization using the recommended
https://help.github.com/articles/transferring-a-repository/#transferring-from-a-user-to-an-organization[github process].