Please make corrections and changes as necessary.
See Part 1 of this tutorial for Overview of why I am doing this, and logstash config blocks.
Below is a breakdown of a logstash custom filter. It is meant to be read as one Ruby class, with comments separating the sections.
At the end of this page is the Grok filter used to parse a Atlassian Stash Auth log.
Obviously comment your custom plugin.
1 # Atlassian Crowd Filter
2 #
3 # This filter will lookup user email from Crowd REST API using username.
4 # Before using this you must create an application account in Crowd and
5 # allow the IP of your logstash indexing server access.
6 #
All custom plugins require the following two require lines.
7 require "logstash/filters/base"
8 require "logstash/namespace"
Provide an example of how to call / use this plugin from the logstash.conf file. If some parameters are required or have special needs provide documentation.
9 # The Atlassian Crowd filter performs a lookup of a user email address
10 # if given a username.
11 #
12 # The config should look like this:
13 #
14 # filter {
15 # crowd {
16 # crowdURL => "https:/crowd/rest/usermanagement/1/user"
17 # crowdUsername => "username"
18 # crowdPassword => "password"
19 # timeout => 4
20 # username_field => "user1"
21 # }
22 # }
23 #
24
25 # username_field = Is a required field that points to the field in the event
26 # before it's passed to this plugin that contains the
27 # Key / Value pair needed to perform the Atlassian Crowd
28 # REST lookup.
29 # crowdURL = Is a required field that provides the URL to your Atlassian
30 # Crowd REST service.
31 # crowdUsername = Is a required field that you must have previously setup
32 # in your Atlassian Crowd UI to allow REST calls. This is
33 # not a standard user account, or admin account. Crowd
34 # refers to these special accounts as Application Accounts.
35 # crowdPassword = Is a required field that specifies the password used by
36 # this Atlassian Crowd Application Account.
37 # timeout = Defines the amount of time the plugin should wait for the
38 # Atlassian Crowd REST service to respond. This is not required
39 # since a default of 2 seconds is set below. If you want to change
40 # the value you can do so in the logstash configuration file as
41 # shown above.
42 #
43 #
44 #
See Official Extending Logstash Doc for up to date changes.
The actual plugin class is defined below and is a subclass of LogStash::Filters::Base as denoted by the < symbol on line 45.
You must name your plugin using the config_name parameter as shown on line 47.
A milestone is also required, and it is suggested by logstash to use milestone 1 as shown on line 48. Logstash Milestone Doc
45 class LogStash::Filters::Crowd < LogStash::Filters::Base
46
47 config_name "crowd"
48 milestone 1
Next you define your expected logstash configuration settings that should be passed to this plugin. There are multiple possibilities here, but each line starts with the config keyword.
See the logstash-1.4.2/lib/logstash/config/mixin.rb source file for more details, or the Extending Logstash Doc for more details.
You can also specify config items that can be defined later in the custom plugin, for example the dns.rb plugin has an 'action' configuration item that provides a way to 'append' or 'update' a value.
It is also advisable to include comments above each config item as shown below.
49 # Username field that contains look up value, in my grok filter I parse the
50 # stash logs and assign the user having a captcha problem to user1
51
52 config :username_field, :validate => :string, :required => true
53
54 # Atlassian Crowd REST API URL
55
56 config :crowdURL, :validate => :string, :required => true
57
58 # Atlassian Crowd REST API Username
59
60 config :crowdUsername, :validate => :string, :required => true
61
62 # Atlassian Crowd REST API Password
63
64 config :crowdPassword, :validate => :string, :required => true
65
66 # RestClient timeout
67
68 config :timeout, :validate => :number, :default => 2
The first method is called register, it sets up additional required modules, and initiates objects that can be used for the lifetime of the logstash process. The resource variable below is defined as an instance variable. When you start logstash your plugin is loaded one time and stays in memory for the entire time that logstash is running. This could be minutes, days, or months.
In the example below I created the RestClient object and assigned it to the @resource class variable on line 73. I did this in the register method instead of the filter method which is called every time a new event is received from logstash. If it was in the event method a new RestClient object would need to be created every time.
69 public
70 def register
71 require "json"
72 require "rest_client"
73 @resource = RestClient::Resource.new(@crowdURL,
74 :user => @crowdUsername,
75 :password => @crowdPassword,
76 :timeout => timeout)
77 end
The next method is called filter which has one input argument called event. This is where logstash hands the raw event to the plugin. Previously the plugin was setup a.k.a initialized, but not used. This is the place where it is actually used. When inspecting the logstash event you get logstashevent, not very meaningful. According to logstash documentation a logstash event is just a hash with some added features. Click here to view the code for event.rb.
First we check if an actual event was received, if not than exit on line 80.
Then we check to see if the event hash passed from logstash includes a key that matches our value for the variable username_field on line 81. Remember earlier we set our config to include an item that said username_field = "user1". If the event came through but did not include a user1 field in the hash, then this event would fail the Atlassian Crowd REST call. So we skip the REST call and exit the plugin if it's missing.
On line 82 we lookup the value stored in the event username_field, and assign it to the username local variable.
We now have the opportunity to set some unique parameters of the @resource object that will be different each time the object is used. For example on line 83 we assign the RestResource a parameter of username with the value obtained on line 82.
Line 84 sends the response which is a string in JSON format to the JSON parse method, which then returns a ruby hash and assigns it to responseHash.
Line 85 extracts the value for the hash key "email" from responseHash and assigns it to the local variable 'email'.
Line 86 updates the logstash event originally passed into the filter with a new Key called 'email' and assigns the Value from the local email variable.
Finally the result is returned to logstash in the form of a filter_matched method that accepts the new event on line 87. Click here for the filter_matched method code.
78 public
79 def filter(event)
80 return unless filter?(event)
81 if event[username_field]
82 username = event[username_field]
83 response = @resource.get(:accept => 'json', :params => {:username => username})
84 responseHash = JSON.parse(response)
85 email = responseHash["email"]
86 event['email'] = email
87 filter_matched(event)
88 end
89 end
90 end
Custom Grok filter for Atlassian Stash Auth Log.
STASH_CAPTCHA %{IP:proxy},%{IP:client} \| %{WORD:error} \| %{WORD:user1} \| %{INT:epoch_time} \| %{WORD:user2} \| (?<error>{%{QS}:%{QS},%{QS}:"For security reasons you must answer a CAPTCHA question."}) \| %{INT:minuteinday}x%{INT:reqnumsincerestart}x%{INT:concurrentreqs} \| %{DATA:something}