github-mirrors/logstash
1
0
Fork 0
mirror of https://github.com/elastic/logstash.git synced 2025-04-24 06:37:19 -04:00
Code Issues Projects Releases Packages Wiki Activity

Adds Contributing a Patch doc

Browse source 
Fixes #4442
This commit is contained in:
Paul Echeverri 2016-01-09 19:55:57 -08:00
parent 62f56ec274
commit 9e57d05de1
1 changed files with 397 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

397
docs/static/contributing-patch.asciidoc vendored Normal file
View file

@ -0,0 +1,397 @@
[[contributing-patch-plugin]]
=== Contributing a Patch to a Logstash Plugin
This section discusses the information you need to know to successfully contribute a patch to a Logstash plugin.
Each plugin defines its own configuration options. These control the behaviour of the plugin to some degree. Configuration
option definitions commonly include:
* Data validation
* The default value
* Any required flags
Plugins are subclasses of a Logstash base class. A plugin's base class defines common configuration and methods.
==== Input Plugins
Input plugins ingest data from an external source. Input plugins are always associated with a codec. An input plugin
always has an associated codec plugin. Input and codec plugins operate in conjuction to create a Logstash event and add
that event to the processing queue. An input codec is a subclass of the `LogStash::Inputs::Base` class.
.Input API
[horizontal]
`#register() -> nil`:: Required. This API sets up resources for the plugin, typically the connection to the
external source.
`#run(queue) -> nil`:: Required. This API fetches or listens for source data, typically looping until stopped. Must handle
errors inside the loop. Pushes any created events to the queue object specified in the method argument. Some inputs may
receive batched data to minimize the external call overhead.
`#stop() -> nil`:: Optional. Stops external connections and cleans up.
==== Codec Plugins
Codec plugins decode input data that has a specific structure, such as JSON input data. A codec plugin is a subclass of
`LogStash::Codecs::Base`.
.Codec API
[horizontal]
`#register() -> nil`:: Identical to the API of the same name for input plugins.
`#decode(data){|event| block} -> nil`:: Must be implemented. Used to create an Event from the raw data given in the method
argument. Must handle errors. The caller must provide a Ruby block. The block is called with the created Event.
`#encode(event) -> nil`:: Required. Used to create a structured data object from the given Event. May handle
errors. This method calls a block that was previously stored as @on_event with two arguments: the original event and the
data object.
==== Filter Plugins
A mechanism to change, mutate or merge one or more Events. A filter plugin is a subclass of the `LogStash::Filters::Base`
class.
.Filter API
[horizontal]
`#register() -> nil`:: Identical to the API of the same name for input plugins.
`#filter(event) -> nil`:: Required. May handle errors. Used to apply a mutation function to the given event.
==== Output Plugins
A mechanism to send an event to an external destination. This process may require serialization. An output plugin is a
subclass of the `LogStash::Outputs::Base` class.
.Output API
[horizontal]
`#register() -> nil`:: Identical to the API of the same name for input plugins.
`#receive(event) -> nil`:: Required. Must handle errors. Used to prepare the given event for transmission to
the external destination. Some outputs may buffer the prepared events to batch transmit to the destination.
[[patch-process]]
==== Process
A bug or feature is identified. An issue is created in the plugin repository. A patch is created and a pull request (PR)
is submitted. After review and possible rework the PR is merged and the plugin is published.
The <<community-maintainer,Community Maintainer Guide>> explains, in more detail, the process of getting a patch accepted,
merged and published. The Community Maintainer Guide also details the roles that contributors and maintainers are
expected to perform.
==== Testing Methodologies
===== Test Driven Development
Test Driven Development, colloquially known as TDD, describes a methodology for using tests to guide evolution of source
code. For our purposes, we are only going to use a part of it, that is, before writing the fix - we create tests that
illustrate the bug by failing. We stop when we have written enough code to make the tests pass and submit the fix and
tests as a patch. It is not necessary to write the tests before the fix, but it is very easy to write a passing test
afterwards that may not actually verify that the fault is really fixed especially if the fault can be triggered via
multiple execution paths or varying input data.
===== The RSpec Framework
Logstash uses Rspec, a Ruby testing framework, to define and run the test suite. What follows is a summary of various
sources.
. Rspec Example
[source,ruby]
1 # encoding: utf-8
2 require "logstash/devutils/rspec/spec_helper"
3 require "logstash/plugin"
4
5 describe "outputs/riemann" do
6 describe "#register" do
7 let(:output) do
8 LogStash::Plugin.lookup("output", "riemann").new(configuration)
9 end
10
11 context "when no protocol is specified" do
12 let(:configuration) { Hash.new }
13
14 it "the method completes without error" do
15 expect {output.register}.not_to raise_error
16 end
17 end
18
19 context "when a bad protocol is specified" do
20 let(:configuration) { {"protocol" => "fake"} }
21
22 it "the method fails with error" do
23 expect {output.register}.to raise_error
24 end
25 end
26
27 context "when the tcp protocol is specified" do
28 let(:configuration) { {"protocol" => "tcp"} }
29
30 it "the method completes without error" do
31 expect {output.register}.not_to raise_error
32 end
33 end
34 end
35
36 describe "#receive" do
37 let(:output) do
38 LogStash::Plugin.lookup("output", "riemann").new(configuration)
39 end
40
41 context "when operating normally" do
42 let(:configuration) { Hash.new }
43 let(:event) do
44 data = {"message"=>"hello", "@version"=>"1",
45 "@timestamp"=>"2015-06-03T23:34:54.076Z",
46 "host"=>"vagrant-ubuntu-trusty-64"}
47 LogStash::Event.new(data)
48 end
49
50 before(:example) do
51 output.register
52 end
53
54 it "should accept the event" do
55 expect { output.receive event }.not_to raise_error
56 end
57 end
58 end
59 end
.Describe blocks (lines 5, 6 and 36 in Example 1)
[source,ruby]
describe(string){block} -> nil
describe(Class){block} -> nil
With RSpec, we are always describing the plugin method behavior. The describe block is added in logical sections and can
accept either an existing class name or a string. The string used in line 5 is the plugin name. Line 6 is the register
method, line 36 is the receive method. It is a RSpec convention to prefix instance methods with one hash and class
methods with one dot.
.Context blocks (lines 11, 19, 27 and 41)
[source,ruby]
context(string){block} -> nil
In RSpec, context blocks define sections that group tests by a variation. The string should start with the word `when`
and then detail the variation. See line 11. The tests in the content block should should only be for that variation.
.Let blocks (lines 7, 12, 20, 28, 37, 42 and 43)
[source,ruby]
let(symbol){block} -> nil
In RSpec, `let` blocks define resources for use in the test blocks. These resources are reinitialized for every test
block. They are available as method calls inside the test block. Define `let` blocks in `describe` and `context` blocks,
which scope the `let` block and any other nested blocks.
You can use other `let` methods defined later within the `let` block body. See lines 7-9, which define the output resource
and use the configuration method, defined with different variations in lines 12, 20 and 28.
.Before blocks (line 50)
[source,ruby]
before(symbol){block} -> nil - symbol is one of :suite, :context, :example, but :all and :each are synonyms for :suite and :example respectively.
In RSpec, `before` blocks are used to further set up any resources that would have been initialized in a `let` block.
You cannot define `let` blocks inside `before` blocks.
You can also define `after` blocks, which are typically used to clean up any setup activity performed by a `before` block.
.It blocks (lines 14, 22, 30 and 54)
[source,ruby]
it(string){block} -> nil
In RSpec, `it` blocks set the expectations that verify the behavior of the tested code. The string should not start with
'it' or 'should', but needs to express the outcome of the expectation. When put together the texts from the enclosing
describe, `context` and `it` blocks should form a fairly readable sentence, as in lines 5, 6, 11 and 14:
[source,ruby]
outputs/riemann
#register when no protocol is specified the method completes without error
Readable code like this make the goals of tests easy to understand.
.Expect method (lines 15, 23, 31, 55)
[source,ruby]
expect(object){block} -> nil
In RSpec, the expect method verifies a statement that compares an actual result to an expected result. The `expect` method
is usually paired with a call to the `to` or `not_to` methods. Use the block form when expecting errors or observing for
changes. The `to` or `not_to` methods require a `matcher` object that encapsulates the expected value. The argument form
of the `expect` method encapsulates the actual value. When put together the whole line tests the actual against the
expected value.
.Matcher methods (lines 15, 23, 31, 55)
[source,ruby]
raise_error(error class|nil) -> matcher instance
be(object) -> matcher instance
eq(object) -> matcher instance
eql(object) -> matcher instance
for more see http://www.relishapp.com/rspec/rspec-expectations/docs/built-in-matchers
In RSpec, a matcher is an object generated by the equivalent method call (be, eq) that will be used to evaluate the
expected against the actual values.
==== Putting it all together
This example fixes an https://github.com/logstash-plugins/logstash-output-zeromq/issues/9[issue] in the ZeroMQ output
plugin. The issue does not require knowledge of ZeroMQ.
The activities in this example have the following prerequisites:
--
* A minimal knowledge of Git and Github. See the https://help.github.com/categories/bootcamp/[Github boot camp].
* A text editor.
* A JRuby https://www.ruby-lang.org/en/documentation/installation/#managers[runtime]
https://howistart.org/posts/ruby/1[environment]. The `chruby` tool manages Ruby versions.
* JRuby 1.7.22 or later.
* The `bundler` and `rake` gems installed.
* ZeroMQ http://zeromq.org/intro:get-the-software[installed].
--
. In Github, fork the ZeroMQ https://github.com/logstash-plugins/logstash-output-zeromq[output plugin repository].
. On your local machine, https://help.github.com/articles/fork-a-repo/[clone] the fork to a known folder such as
`logstash/`.
. Open the following files in a text editor:
* `logstash-output-zeromq/lib/logstash/outputs/zeromq.rb`
* `logstash-output-zeromq/lib/logstash/util/zeromq.rb`
* `logstash-output-zeromq/spec/outputs/zeromq_spec.rb`
. According to the issue, log output in server mode must indicate `bound`. Furthermore, the test file contains no tests.
+
NOTE: Line 21 of `util/zeromq.rb` reads `@logger.info("0mq: #{server? ? 'connected' : 'bound'}", :address => address)`
. In the text editor, set the file encoding and require `zeromq.rb` for the file `zeromq_spec.rb` by adding the following
lines:
+
[source,ruby]
# encoding: utf-8
require "logstash/outputs/zeromq"
require "logstash/devutils/rspec/spec_helper"
. The desired error message should read:
+
[source,ruby]
LogStash::Outputs::ZeroMQ when in server mode a 'bound' info line is logged
+
To properly generate this message, add a `describe` block with the fully qualified class name as the argument, a context
block, and an `it` block.
+
[source,ruby]
describe LogStash::Outputs::ZeroMQ do
context "when in server mode" do
it "a 'bound' info line is logged" do
end
end
end
. To add the missing test, use an instance of the ZeroMQ output and a substitute logger. This examle uses an RSpec feature
called _test doubles_ as the substitute logger.
+
Add the following lines to `zeromq_spec.rb`, after `describe LogStash::Outputs::ZeroMQ do` and before `context "when in
server mode" do`:
[source,ruby]
let(:output) { described_class.new("mode" => "server", "topology" => "pushpull" }
let(:tracer) { double("logger") }
. Add the body to the `it` block. Add the following five lines after the line `context "when in server mode" do`:
[source,ruby]
allow(tracer).to receive(:debug)<1>
output.logger = logger<2>
expect(tracer).to receive(:info).with("0mq: bound", {:address=>"tcp://127.0.0.1:2120"})<3>
output.register<4>
output.do_close<5>
<1> Allow the double to receive `debug` method calls.
<2> Make the output use the test double.
<3> Set an expectation on the test to receive an `info` method call.
<4> Call `register` on the output.
<5> Call `do_close` on the output so the test does not hang.
At the end of the modifications, the relevant code section reads:
[source,ruby]
--------
# encoding: utf-8
require "logstash/outputs/zeromq"
require "logstash/devutils/rspec/spec_helper"
describe LogStash::Outputs::ZeroMQ do
let(:output) { described_class.new("mode" => "server", "topology" => "pushpull" }
let(:tracer) { double("logger") }
context "when in server mode" do
it "a ‘bound’ info line is logged" do
allow(tracer).to receive(:debug)
output.logger = logger
expect(tracer).to receive(:info).with("0mq: bound", {:address=>"tcp://127.0.0.1:2120"})
output.register
output.do_close
end
end
end
--------
To run this test:
. Open a terminal window
. Mavigate to the cloned plugin folder
. The first time you run the test, run the command `bundle install`
. Run the command `bundle exec rspec`
Assuming all prerequisites were installed correctly, the test fails with output similar to:
[source,shell]
--------
Using Accessor#strict_set for specs
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :couchdb=>true, :elasticsearch=>true,
:elasticsearch_secure=>true, :export_cypher=>true, :integration=>true, :windows=>true}
LogStash::Outputs::ZeroMQ
when in server mode
a ‘bound’ info line is logged (FAILED - 1)
Failures:
1) LogStash::Outputs::ZeroMQ when in server mode a ‘bound’ info line is logged
Failure/Error: output.register
Double "logger" received :info with unexpected arguments
expected: ("0mq: bound", {:address=>"tcp://127.0.0.1:2120"})
got: ("0mq: connected", {:address=>"tcp://127.0.0.1:2120"})
# ./lib/logstash/util/zeromq.rb:21:in `setup'
# ./lib/logstash/outputs/zeromq.rb:92:in `register'
# ./lib/logstash/outputs/zeromq.rb:91:in `register'
# ./spec/outputs/zeromq_spec.rb:13:in `(root)'
# /Users/guy/.gem/jruby/1.9.3/gems/rspec-wait-0.0.7/lib/rspec/wait.rb:46:in `(root)'
Finished in 0.133 seconds (files took 1.28 seconds to load)
1 example, 1 failure
Failed examples:
rspec ./spec/outputs/zeromq_spec.rb:10 # LogStash::Outputs::ZeroMQ when in server mode a ‘bound’ info line is logged
Randomized with seed 2568
--------
To correct the error, open the `util/zeromq.rb` file in your text editor and swap the positions of the words `connected`
and `bound` on line 21. Line 21 now reads:
[source,ruby]
@logger.info("0mq: #{server? ? 'bound' : 'connected'}", :address => address)
Run the test again with the `bundle exec rspec` command.
The test passes with output similar to:
[source,shell]
--------
Using Accessor#strict_set for specs
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :couchdb=>true, :elasticsearch=>true, :elasticsearch_secure=>true, :export_cypher=>true, :integration=>true, :windows=>true}
LogStash::Outputs::ZeroMQ
when in server mode
a ‘bound’ info line is logged
Finished in 0.114 seconds (files took 1.22 seconds to load)
1 example, 0 failures
Randomized with seed 45887
--------
https://help.github.com/articles/fork-a-repo/#next-steps[Commit] the changes to git and Github.
Your pull request is visible from the https://github.com/logstash-plugins/logstash-output-zeromq/pulls[Pull Requests]
section of the original Github repository. The plugin maintainers review your work, suggest changes if necessary, and
merge and publish a new version of the plugin.