github-mirrors/logstash
1
0
Fork 0
mirror of https://github.com/elastic/logstash.git synced 2025-04-24 22:57:16 -04:00
Code Issues Projects Releases Packages Wiki Activity

Merge branch 'master' of git://github.com/logstash/logstash

Browse source
This commit is contained in:
Alex Wheeler 2012-11-14 14:33:23 -05:00
commit 5abc9b3e95
44 changed files with 985 additions and 274 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.gitignore vendored
View file

@ -1,6 +1,7 @@
.*.swp .*.swp
*.gem *.gem
*.class *.class
Gemfile.lock
*.tar.gz *.tar.gz
*.jar *.jar
.bundle .bundle

48
CHANGELOG
View file

@ -1,5 +1,49 @@
1.1.5 (????????????????) 1.1.6 (???)
- irc input now stores nick ## Overview of this release:
- bug fixes
## general
- fixed internal dependency versioning on 'addressable' gem (LOGSTASH-694)
1.1.5 (November 10, 2012)
## Overview of this release:
* New inputs: zenoss, gemfire
* New outputs: lumberjack, gemfire
* Many UTF-8 crashing bugs were resolved
## general
- new runner command 'rspec' - lets you run rspec tests from the jar
This means you should now be able to write external tests that execute your
logstash configs and verify functionality.
- "file not found" errors related to paths that had "jar:" prefixes should
now work. (Fixes LOGSTASH-649, LOGSTASH-642, LOGSTASH-655)
- several plugins received UTF-8-related fixes (file, lumberjack, etc)
File bugs if you see any UTF-8 related crashes.
- 'json_event' format inputs will now respect 'tags' (#239, patch by
Tim Laszlo)
- logstash no longer uses nor recommends bundler (see 'gembag.rb'). The
Gemfile will be purged in the near future.
- amqp plugins are now marked 'unsupported' as there is no active maintainer
nor is there source of active support in the community. If you're interested
in maintainership, please email the mailling list or contact Jordan!
## inputs
- irc: now stores irc nick
- new: zenoss (#232, patch by Chet Luther)
- new: gemfire (#235, patch by Andrea Campi)
- bugfix: udp: skip close() call if we're already closed (#238, patch by kcrayon)
## filters
- bugfix: fix for zeromq filter initializer (#237, patch by Tom Howe)
## outputs
- new: lumberjack
- new: gemfire output (#234, patch by Andrea Campi)
- improved: nagios_ncsa (patch by Tomas Doran)
- improved: elasticsearch: permit setting 'host' even if embedded. Also set the
host default to 'localhost' when using embedded. These fixes should help resolve
issues new users have when their distros surprisingly block multicast by default.
- improved: elasticsearch: failed index attempts will be retried
1.1.4 (October 28, 2012) 1.1.4 (October 28, 2012)
## Overview of this release: ## Overview of this release:

2
CONTRIBUTORS
View file

@ -58,7 +58,7 @@ Contributors:
* Mike Worth (MikeWorth) * Mike Worth (MikeWorth)
* Nic Williams (drnic) * Nic Williams (drnic)
* Tomas Doran (bobtfish) * Tomas Doran (bobtfish)
* zuazo * Xabier de Zuazo (zuazo)
Note: If you've sent me patches, bug reports, or otherwise contributed to Note: If you've sent me patches, bug reports, or otherwise contributed to
logstash, and you aren't on the list above and want to be, please let me know logstash, and you aren't on the list above and want to be, please let me know

191
Gemfile.lock
View file

@ -1,191 +0,0 @@
PATH
remote: .
specs:
logstash (1.1.4-java)
addressable (= 2.2.6)
aws-sdk
bunny
cabin (= 0.4.4)
cinch
ffi
ffi-rzmq (= 0.9.3)
filewatch (= 0.5.0)
ftw (~> 0.0.22)
gelf (= 1.3.2)
gelfd (= 0.2.0)
geoip (>= 1.1.0)
gmetric (= 0.1.3)
haml
heroku
jls-grok (= 0.10.7)
jls-lumberjack
jruby-elasticsearch (= 0.0.14)
jruby-httpclient
jruby-openssl
jruby-win32ole
json
mail
minitest
mongo
onstomp
pry
rack
redis
riak-client (= 1.0.3)
riemann-client (= 0.0.6)
sass
sinatra
statsd-ruby (= 0.3.0)
stud
uuidtools
xml-simple
xmpp4r (= 0.5)
GEM
remote: http://rubygems.org/
specs:
activesupport (3.2.8)
i18n (~> 0.6)
multi_json (~> 1.0)
addressable (2.2.6)
aws-sdk (1.6.9)
httparty (~> 0.7)
json (~> 1.4)
nokogiri (>= 1.4.4)
uuidtools (~> 2.1)
backports (2.3.0)
beefcake (0.3.7)
bouncy-castle-java (1.5.0146.1)
bson (1.7.0-java)
builder (3.1.4)
bunny (0.8.0)
cabin (0.4.4)
json
cinch (2.0.3)
coderay (1.0.8)
diff-lcs (1.1.3)
excon (0.16.7)
ffi (1.1.5-java)
ffi-rzmq (0.9.3)
ffi
filewatch (0.5.0)
ftw (0.0.22)
addressable (= 2.2.6)
backports (= 2.3.0)
cabin (> 0)
http_parser.rb (= 0.5.3)
json (= 1.6.5)
minitest (> 0)
gelf (1.3.2)
json
gelfd (0.2.0)
geoip (1.2.0)
gmetric (0.1.3)
haml (3.1.7)
heroku (2.33.0)
heroku-api (~> 0.3.5)
launchy (>= 0.3.2)
netrc (~> 0.7.7)
rest-client (~> 1.6.1)
rubyzip
heroku-api (0.3.5)
excon (~> 0.16.1)
http_parser.rb (0.5.3-java)
httparty (0.9.0)
multi_json (~> 1.0)
multi_xml
i18n (0.6.1)
insist (0.0.7)
jls-grok (0.10.7)
cabin (~> 0.4.0)
jls-lumberjack (0.0.2)
jruby-elasticsearch (0.0.14)
jruby-httpclient (1.1.0-java)
jruby-openssl (0.7.7)
bouncy-castle-java (>= 1.5.0146.1)
jruby-win32ole (0.8.5)
json (1.6.5-java)
launchy (2.0.3-java)
spoon (~> 0.0.1)
mail (2.4.4)
i18n (>= 0.4.0)
mime-types (~> 1.16)
treetop (~> 1.4.8)
metaclass (0.0.1)
method_source (0.8.1)
mime-types (1.19)
minitest (4.1.0)
mocha (0.12.7)
metaclass (~> 0.0.1)
mongo (1.7.0)
bson (~> 1.7.0)
mtrc (0.0.4)
multi_json (1.3.6)
multi_xml (0.5.1)
netrc (0.7.7)
nokogiri (1.5.5-java)
onstomp (1.0.7)
polyglot (0.3.3)
pry (0.9.10-java)
coderay (~> 1.0.5)
method_source (~> 0.8)
slop (~> 3.3.1)
spoon (~> 0.0)
rack (1.4.1)
rack-protection (1.2.0)
rack
redis (3.0.2)
rest-client (1.6.7)
mime-types (>= 1.16)
riak-client (1.0.3)
beefcake (~> 0.3.7)
builder (>= 2.1.2)
i18n (>= 0.4.0)
multi_json (~> 1.0)
riemann-client (0.0.6)
beefcake (>= 0.3.5)
mtrc (>= 0.0.4)
trollop (>= 1.16.2)
rspec (2.11.0)
rspec-core (~> 2.11.0)
rspec-expectations (~> 2.11.0)
rspec-mocks (~> 2.11.0)
rspec-core (2.11.1)
rspec-expectations (2.11.3)
diff-lcs (~> 1.1.3)
rspec-mocks (2.11.3)
rubyzip (0.9.9)
sass (3.2.1)
shoulda (3.3.2)
shoulda-context (~> 1.0.1)
shoulda-matchers (~> 1.4.1)
shoulda-context (1.0.1)
shoulda-matchers (1.4.1)
activesupport (>= 3.0.0)
sinatra (1.3.3)
rack (~> 1.3, >= 1.3.6)
rack-protection (~> 1.2)
tilt (~> 1.3, >= 1.3.3)
slop (3.3.3)
spoon (0.0.1)
statsd-ruby (0.3.0)
stud (0.0.6)
tilt (1.3.3)
treetop (1.4.11)
polyglot
polyglot (>= 0.3.1)
trollop (2.0)
uuidtools (2.1.3)
xml-simple (1.1.2)
xmpp4r (0.5)
PLATFORMS
java
DEPENDENCIES
insist (= 0.0.7)
logstash!
mocha
rspec
shoulda
spoon

47
Makefile
View file

@ -17,10 +17,9 @@ JRUBYC=$(WITH_JRUBY) jrubyc
ELASTICSEARCH_URL=http://github.com/downloads/elasticsearch/elasticsearch ELASTICSEARCH_URL=http://github.com/downloads/elasticsearch/elasticsearch
ELASTICSEARCH=vendor/jar/elasticsearch-$(ELASTICSEARCH_VERSION) ELASTICSEARCH=vendor/jar/elasticsearch-$(ELASTICSEARCH_VERSION)
JODA=vendor/jar/joda-time-$(JODA_VERSION)/joda-time-$(JODA_VERSION).jar JODA=vendor/jar/joda-time-$(JODA_VERSION)/joda-time-$(JODA_VERSION).jar
GEOIP=vendor/geoip/GeoCityLite.dat GEOIP=vendor/geoip/GeoLiteCity.dat
GEOIP_URL=http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz GEOIP_URL=http://logstash.objects.dreamhost.com/maxmind/GeoLiteCity-2012-11-09.dat.gz
PLUGIN_FILES=$(shell git ls-files | egrep '^lib/logstash/(inputs|outputs|filters)/' | egrep -v '/(base|threadable).rb$$|/inputs/ganglia/') PLUGIN_FILES=$(shell git ls-files | egrep '^lib/logstash/(inputs|outputs|filters)/' | egrep -v '/(base|threadable).rb$$|/inputs/ganglia/')
GEM_HOME=build/gems
QUIET=@ QUIET=@
WGET=$(shell which wget 2>/dev/null) WGET=$(shell which wget 2>/dev/null)
@ -39,15 +38,15 @@ default: jar
# Figure out if we're using wget or curl # Figure out if we're using wget or curl
.PHONY: wget-or-curl .PHONY: wget-or-curl
wget-or-curl: wget-or-curl:
ifeq ($(WGET),)
ifeq ($(CURL),) ifeq ($(CURL),)
ifeq ($(WGET),)
@echo "wget or curl are required." @echo "wget or curl are required."
exit 1 exit 1
else else
DOWNLOAD_COMMAND=curl -L -k -o DOWNLOAD_COMMAND=wget -q --no-check-certificate -O
endif endif
else else
DOWNLOAD_COMMAND=wget --no-check-certificate -O DOWNLOAD_COMMAND=curl -s -L -k -o
endif endif
# Compile config grammar (ragel -> ruby) # Compile config grammar (ragel -> ruby)
@ -79,8 +78,7 @@ copy-ruby-files: | build/ruby
| (cd lib; cpio -p --make-directories ../build/ruby) | (cd lib; cpio -p --make-directories ../build/ruby)
$(QUIET)find ./test -name '*.rb' | sed -e 's,^\./test/,,' \ $(QUIET)find ./test -name '*.rb' | sed -e 's,^\./test/,,' \
| (cd test; cpio -p --make-directories ../build/ruby) | (cd test; cpio -p --make-directories ../build/ruby)
$(QUIET)find ./spec -name '*.rb' | sed -e 's,^\./spec/,,' \ $(QUIET)rsync -av ./spec build/ruby
| (cd spec; cpio -p --make-directories ../build/ruby)
vendor: vendor:
$(QUIET)mkdir $@ $(QUIET)mkdir $@
@ -119,8 +117,7 @@ vendor/geoip: | vendor
$(QUIET)mkdir $@ $(QUIET)mkdir $@
$(GEOIP): | vendor/geoip $(GEOIP): | vendor/geoip
$(QUIET)wget -q -O - http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz \ $(QUIET)wget -q -O - $(GEOIP_URL) | gzip -dc - > $@
| gzip -dc - > $@
# Always run vendor/bundle # Always run vendor/bundle
.PHONY: fix-bundler .PHONY: fix-bundler
@ -130,14 +127,11 @@ fix-bundler:
.PHONY: vendor-gems .PHONY: vendor-gems
vendor-gems: | vendor/bundle vendor-gems: | vendor/bundle
$(GEM_HOME)/bin/bundle: | $(JRUBY)
@echo "=> Installing bundler ($@)"
$(QUIET)GEM_HOME=$(GEM_HOME) $(WITH_JRUBY) gem install bundler
.PHONY: vendor/bundle .PHONY: vendor/bundle
vendor/bundle: | $(GEM_HOME)/bin/bundle fix-bundler vendor/bundle: | vendor
@echo "=> Installing gems to $@..." @echo "=> Installing gems to $@..."
$(QUIET)GEM_HOME=$(GEM_HOME) $(JRUBY_CMD) --1.9 $(GEM_HOME)/bin/bundle install --deployment #$(QUIET)GEM_HOME=$(GEM_HOME) $(JRUBY_CMD) --1.9 $(GEM_HOME)/bin/bundle install --deployment
$(QUIET)GEM_HOME=./vendor/bundle/jruby/1.9/ GEM_PATH= $(JRUBY_CMD) --1.9 ./gembag.rb logstash.gemspec
@# Purge any junk that fattens our jar without need! @# Purge any junk that fattens our jar without need!
@# The riak gem includes previous gems in the 'pkg' dir. :( @# The riak gem includes previous gems in the 'pkg' dir. :(
-rm -rf $@/jruby/1.9/gems/riak-client-1.0.3/pkg -rm -rf $@/jruby/1.9/gems/riak-client-1.0.3/pkg
@ -197,10 +191,31 @@ build/logstash-$(VERSION)-monolithic.jar: JAR_ARGS+=-C lib logstash/certs
build/logstash-$(VERSION)-monolithic.jar: JAR_ARGS+=-C lib logstash/web/views build/logstash-$(VERSION)-monolithic.jar: JAR_ARGS+=-C lib logstash/web/views
build/logstash-$(VERSION)-monolithic.jar: JAR_ARGS+=patterns build/logstash-$(VERSION)-monolithic.jar: JAR_ARGS+=patterns
build/logstash-$(VERSION)-monolithic.jar: build/logstash-$(VERSION)-monolithic.jar:
$(QUIET)rm -f $@
$(QUIET)jar cfe $@ logstash.runner $(JAR_ARGS) $(QUIET)jar cfe $@ logstash.runner $(JAR_ARGS)
$(QUIET)jar i $@ $(QUIET)jar i $@
@echo "Created $@" @echo "Created $@"
build/flatgems: | build vendor/bundle
mkdir $@
for i in $(VENDOR_DIR)/gems/*/lib; do \
rsync -av $$i/ $@/lib ; \
done
flatjar: build/logstash-$(VERSION)-flatjar.jar
build/jar: | build build/flatgems build/monolith
$(QUIET)mkdir build/jar
$(QUIET)rsync -av --delete build/flatgems/lib/ build/monolith/ build/ruby/ patterns build/jar/
$(QUIET)(cd lib; rsync -av --delete logstash/web/public ../build/jar/logstash/web/public)
$(QUIET)(cd lib; rsync -av --delete logstash/web/views ../build/jar/logstash/web/views)
$(QUIET)(cd lib; rsync -av --delete logstash/certs ../build/jar/logstash/certs)
build/logstash-$(VERSION)-flatjar.jar: | build/jar
$(QUIET)rm -f $@
$(QUIET)jar cfe $@ logstash.runner -C build/jar .
$(QUIET)jar i $@
@echo "Created $@"
update-jar: copy-ruby-files update-jar: copy-ruby-files
$(QUIET)jar uf build/logstash-$(VERSION)-monolithic.jar -C build/ruby . $(QUIET)jar uf build/logstash-$(VERSION)-monolithic.jar -C build/ruby .

18
README.md
View file

@ -9,16 +9,22 @@ are pretty much free to use it however you want in whatever way.
For more info, see <http://logstash.net/> For more info, see <http://logstash.net/>
Need help? Try #logstash on freenode irc or the logstash-users@googlegroups.com mailing list.
## Building ## Building
To work on the code without building a jar, install rvm and run the following: To work on the code without building a jar, install rvm and run the following:
rvm install 1.6.8 # Install JRuby with rvm
rvm use 1.6.8 rvm install jruby-1.7.0
export JRUBY_OPTS=--1.9 rvm use jruby-1.7.0
bundle install
bundle exec ruby bin/logstash agent [options]
jar releases are available here: <http://semicomplete.com/files/logstash/> # Install logstash dependencies
ruby gembag.rb logstash.gemspec
# Run logstash
bin/logstash agent [options]
jar releases are available here: <http://logstash.objects.dreamhost.com/>
If you want to build the jar yourself, run: If you want to build the jar yourself, run:
make jar make jar

7
STYLE.md
View file

@ -18,13 +18,15 @@ Do this:
## Code Style ## Code Style
* comment everything you can think of.
* indentation: 2 spaces * indentation: 2 spaces
* between methods: 1 line * between methods: 1 line
* sort your requires * sort your requires
* long lines should wrap at 80 characters. If you wrap at an operator (or, +, * long lines should wrap at 80 characters. If you wrap at an operator ('or',
etc) start the next line with that operator. '+', etc) start the next line with that operator.
* parentheses on function definitions/calls * parentheses on function definitions/calls
* explicit is better than implicit * explicit is better than implicit
* implicit returns are forbidden except in the case of a single expression
The point is consistency and documentation. If you see inconsistencies, let me The point is consistency and documentation. If you see inconsistencies, let me
know, and I'll fix them :) know, and I'll fix them :)
@ -55,6 +57,7 @@ Short example:
some_really_long_function_call_blah_blah_blah(arg1, some_really_long_function_call_blah_blah_blah(arg1,
arg2, arg3, arg4) arg2, arg3, arg4)
# indent the 'when' inside a 'case'.
case foo case foo
when "bar" when "bar"
puts "Hello world" puts "Hello world"

18
bin/logstash
View file

@ -1,13 +1,9 @@
#!/usr/bin/env ruby #!/bin/sh
$: << File.dirname($0) + "/../lib" if [ -d .git ] ; then
export GEM_HOME=./vendor/bundle/jruby/1.9
export GEM_PATH=
fi
require "rubygems" export RUBYLIB=./lib
require "logstash/runner" ruby lib/logstash/runner.rb "$@"
# If the first argument is a flag, assume agent.
if ARGV[0] =~ /^-/
ARGV.unshift("agent")
end
LogStash::Runner.new.main(ARGV)

2
docs/index.html.erb
View file

@ -6,7 +6,7 @@ layout: content_right
<h3> general </h3> <h3> general </h3>
<ul> <ul>
<li> <a href="http://semicomplete.com/files/logstash/logstash-%VERSION%-monolithic.jar"> download logstash %VERSION% </a> </li> <li> <a href="https://logstash.objects.dreamhost.com/release/logstash-%VERSION%-monolithic.jar"> download logstash %VERSION% </a> </li>
<li> <a href="flags"> command-line flags </a> </li> <li> <a href="flags"> command-line flags </a> </li>
<li> <a href="configuration"> configuration file overview </a> </li> <li> <a href="configuration"> configuration file overview </a> </li>
<li> <a href="extending"> writing your own plugins </a> </li> <li> <a href="extending"> writing your own plugins </a> </li>

7
docs/learn.md
View file

@ -16,10 +16,15 @@ presentation I gave at CarolinaCon 2011:
logstash, how you can use it, some alternatives, logging best practices, logstash, how you can use it, some alternatives, logging best practices,
parsing tools, etc. Video also below: parsing tools, etc. Video also below:
<!--
<embed src="http://blip.tv/play/gvE9grjcdQI" type="application/x-shockwave-flash" width="480" height="296" allowscriptaccess="always" allowfullscreen="true"></embed> <embed src="http://blip.tv/play/gvE9grjcdQI" type="application/x-shockwave-flash" width="480" height="296" allowscriptaccess="always" allowfullscreen="true"></embed>
The slides are available online here: [slides](http://goo.gl/68c62). The slides The slides are available online here: [slides](http://goo.gl/68c62). The slides
include speaker notes (click 'actions' then 'speaker notes'). include speaker notes (click 'actions' then 'speaker notes').
-->
<iframe width="480" height="296" src="http://www.youtube.com/embed/RuUFnog29M4" frameborder="0" allowfullscreen></iframe>
The slides are available online here: [slides](http://semicomplete.com/presentations/logstash-puppetconf-2012/).
## Getting Help ## Getting Help
@ -33,7 +38,7 @@ for such things, that works for me, too.)
## Download It ## Download It
[Download logstash-%VERSION%](http://semicomplete.com/files/logstash/logstash-%VERSION%-monolithic.jar) [Download logstash-%VERSION%](https://logstash.objects.dreamhost.com/release/logstash-%VERSION%-monolithic.jar)
## What's next? ## What's next?

7
docs/plugin-status.md
View file

@ -15,6 +15,13 @@ while conveying to the end-user a set of expectations about that plugin. This
allows you to make more informed decisions about when and where to use the allows you to make more informed decisions about when and where to use the
functionality provided by the new plugin. functionality provided by the new plugin.
## Unsupported
This plugin is not supported. It should work, but if you have any problems with
it you are unlikely to find any help due to lack of experience in the community.
Additionally, this label may mean that a plugin has no active maintainer.
## Experimental ## Experimental
When a plugin is in the `experimental` state, it is essentially untested. This When a plugin is in the `experimental` state, it is essentially untested. This

2
docs/tutorials/getting-started-simple.md
View file

@ -83,7 +83,7 @@ _This assumes you have the `curl` command installed._
You should get back some output like so: You should get back some output like so:
"logstash-2012.07.02" : { "logstash-2012.07.02" : {
           "index" : "logstash-2012.07.02" "index" : "logstash-2012.07.02"
This means Logstash created a new index based on today's date. Likely your data is in there as well: This means Logstash created a new index based on today's date. Likely your data is in there as well:

33
gembag.rb Normal file
View file

@ -0,0 +1,33 @@
#!/usr/bin/env ruby
require "rubygems/specification"
require "rubygems/commands/install_command"
gemspec = ARGV.shift
spec = Gem::Specification.load(gemspec)
deps = [spec.development_dependencies, spec.runtime_dependencies].flatten
# target for now
target = "vendor/bundle/jruby/1.9/"
deps.each do |dep|
#cmd = "gem install --install-dir #{target} #{dep.name} -v '#{dep.requirement}'"
installer = Gem::Commands::InstallCommand.new
installer.options[:generate_rdoc] = false
installer.options[:generate_ri] = false
installer.options[:version] = dep.requirement
installer.options[:args] = [dep.name]
installer.options[:install_dir] = target
begin
installer.execute
rescue Gem::SystemExitException => e
if e.exit_code != 0
puts "Installation of #{dep.to_s} failed"
raise
end
end
end

2
lib/logstash/config/mixin.rb
View file

@ -172,6 +172,8 @@ module LogStash::Config::Mixin
def validate_plugin_status def validate_plugin_status
docmsg = "For more information about plugin statuses, see http://logstash.net/docs/#{LOGSTASH_VERSION}/plugin-status " docmsg = "For more information about plugin statuses, see http://logstash.net/docs/#{LOGSTASH_VERSION}/plugin-status "
case @plugin_status case @plugin_status
when "unsupported"
@logger.warn("Using unsupported plugin '#{@config_name}'. This plugin isn't well supported by the community and likely has no maintainer. #{docmsg}")
when "experimental" when "experimental"
@logger.warn("Using experimental plugin '#{@config_name}'. This plugin is untested and may change in the future. #{docmsg}") @logger.warn("Using experimental plugin '#{@config_name}'. This plugin is untested and may change in the future. #{docmsg}")
when "beta" when "beta"

5
lib/logstash/event.rb
View file

@ -40,6 +40,11 @@ class LogStash::Event
@cancelled = true @cancelled = true
end # def cancel end # def cancel
public
def uncancel
@cancelled = false
end # def uncancel
public public
def cancelled? def cancelled?
return @cancelled return @cancelled

1
lib/logstash/filters/multiline.rb
View file

@ -227,6 +227,7 @@ class LogStash::Filters::Multiline < LogStash::Filters::Base
def flush def flush
events = [] events = []
@pending.each do |key, value| @pending.each do |key, value|
value.uncancel
events << value events << value
end end
@pending.clear @pending.clear

4
lib/logstash/filters/zeromq.rb
View file

@ -51,8 +51,8 @@ class LogStash::Filters::ZeroMQ < LogStash::Filters::Base
config :sockopt, :validate => :hash config :sockopt, :validate => :hash
public public
def initialize def initialize(params)
super super(params)
@threadsafe = false @threadsafe = false
end end

7
lib/logstash/inputs/amqp.rb
View file

@ -3,6 +3,11 @@ require "logstash/namespace"
# Pull events from an AMQP exchange. # Pull events from an AMQP exchange.
# #
# <b> NOTE: THIS IS ONLY KNOWN TO WORK WITH RECENT RELEASES OF RABBITMQ. Any
# other amqp broker will not work with this plugin. I do not know why. If you
# need support for brokers other than rabbitmq, please file bugs here:
# <https://github.com/ruby-amqp/bunny> </b>
#
# AMQP is a messaging system. It requires you to run an AMQP server or 'broker' # AMQP is a messaging system. It requires you to run an AMQP server or 'broker'
# Examples of AMQP servers are [RabbitMQ](http://www.rabbitmq.com/) and # Examples of AMQP servers are [RabbitMQ](http://www.rabbitmq.com/) and
# [QPid](http://qpid.apache.org/) # [QPid](http://qpid.apache.org/)
@ -12,7 +17,7 @@ require "logstash/namespace"
class LogStash::Inputs::Amqp < LogStash::Inputs::Threadable class LogStash::Inputs::Amqp < LogStash::Inputs::Threadable
config_name "amqp" config_name "amqp"
plugin_status "beta" plugin_status "unsupported"
# Your amqp broker's custom arguments. For mirrored queues in RabbitMQ: [ "x-ha-policy", "all" ] # Your amqp broker's custom arguments. For mirrored queues in RabbitMQ: [ "x-ha-policy", "all" ]
config :arguments, :validate => :array, :default => [] config :arguments, :validate => :array, :default => []

1
lib/logstash/inputs/base.rb
View file

@ -95,6 +95,7 @@ class LogStash::Inputs::Base < LogStash::Plugin
when "json_event" when "json_event"
begin begin
event = LogStash::Event.from_json(raw) event = LogStash::Event.from_json(raw)
event.tags += @tags
if @message_format if @message_format
event.message ||= event.sprintf(@message_format) event.message ||= event.sprintf(@message_format)
end end

2
lib/logstash/inputs/file.rb
View file

@ -126,6 +126,8 @@ class LogStash::Inputs::File < LogStash::Inputs::Base
hostname = Socket.gethostname hostname = Socket.gethostname
@tail.subscribe do |path, line| @tail.subscribe do |path, line|
path = path
line = line.force_encoding("UTF-8")
source = Addressable::URI.new(:scheme => "file", :host => hostname, :path => path).to_s source = Addressable::URI.new(:scheme => "file", :host => hostname, :path => path).to_s
@logger.debug("Received line", :path => path, :line => line) @logger.debug("Received line", :path => path, :line => line)
e = to_event(line, source) e = to_event(line, source)

227
lib/logstash/inputs/gemfire.rb Normal file
View file

@ -0,0 +1,227 @@
require "logstash/inputs/threadable"
require "logstash/namespace"
# Push events to a GemFire region.
#
# GemFire is an object database.
#
# To use this plugin you need to add gemfire.jar to your CLASSPATH.
# Using format=json requires jackson.jar too; use of continuous
# queries requires antlr.jar.
#
# Note: this plugin has only been tested with GemFire 7.0.
#
class LogStash::Inputs::Gemfire < LogStash::Inputs::Threadable
config_name "gemfire"
plugin_status "experimental"
# Your client cache name
config :name, :validate => :string, :default => "logstash"
# The path to a GemFire client cache XML file.
#
# Example:
#
# <client-cache>
# <pool name="client-pool" subscription-enabled="true" subscription-redundancy="1">
# <locator host="localhost" port="31331"/>
# </pool>
# <region name="Logstash">
# <region-attributes refid="CACHING_PROXY" pool-name="client-pool" >
# </region-attributes>
# </region>
# </client-cache>
#
config :cache_xml_file, :validate => :string, :default => nil
# The region name
config :region_name, :validate => :string, :default => "Logstash"
# A regexp to use when registering interest for cache events.
# Ignored if a :query is specified.
config :interest_regexp, :validate => :string, :default => ".*"
# A query to run as a GemFire "continuous query"; if specified it takes
# precedence over :interest_regexp which will be ignore.
#
# Important: use of continuous queries requires subscriptions to be enabled on the client pool.
config :query, :validate => :string, :default => nil
# How the message is serialized in the cache. Can be one of "json" or "plain"; default is plain
config :serialization, :validate => :string, :default => nil
public
def initialize(params)
super
@format ||= "plain"
end # def initialize
public
def register
import com.gemstone.gemfire.cache.AttributesMutator
import com.gemstone.gemfire.cache.InterestResultPolicy
import com.gemstone.gemfire.cache.client.ClientCacheFactory
import com.gemstone.gemfire.cache.client.ClientRegionShortcut
import com.gemstone.gemfire.cache.query.CqQuery
import com.gemstone.gemfire.cache.query.CqAttributes
import com.gemstone.gemfire.cache.query.CqAttributesFactory
import com.gemstone.gemfire.cache.query.QueryService
import com.gemstone.gemfire.cache.query.SelectResults
import com.gemstone.gemfire.pdx.JSONFormatter
@logger.info("Registering input", :plugin => self)
end # def register
def run(queue)
return if terminating?
connect
@logstash_queue = queue
if @query
continuous_query(@query)
else
register_interest(@interest_regexp)
end
end # def run
def teardown
@cache.close if @cache
@cache = nil
finished
end # def teardown
protected
def connect
begin
@logger.debug("Connecting to GemFire #{@name}")
@cache = ClientCacheFactory.new.
set("name", @name).
set("cache-xml-file", @cache_xml_file).create
@logger.debug("Created cache #{@cache.inspect}")
rescue => e
if terminating?
return
else
@logger.error("Gemfire connection error (during connect), will reconnect",
:exception => e, :backtrace => e.backtrace)
sleep(1)
retry
end
end
@region = @cache.getRegion(@region_name);
@logger.debug("Created region #{@region.inspect}")
end # def connect
protected
def continuous_query(query)
qs = @cache.getQueryService
cqAf = CqAttributesFactory.new
cqAf.addCqListener(self)
cqa = cqAf.create
@logger.debug("Running continuous query #{query}")
cq = qs.newCq("logstashCQ" + self.object_id.to_s, query, cqa)
cq.executeWithInitialResults
end
def register_interest(interest)
@region.getAttributesMutator.addCacheListener(self)
@region.registerInterestRegex(interest, InterestResultPolicy::NONE, false, true)
end
def deserialize_message(message)
if @serialization == "json"
message ? JSONFormatter.toJSON(message) : "{}"
else
message
end
end
def process_event(event, event_name, source)
message = deserialize_message(event)
e = to_event(message, source)
if e
@logstash_queue << e
end
end
# multiple interfaces
def close
end
#
# CqListener interface
#
def onEvent(event)
key = event.getKey
newValue = event.getNewValue
@logger.debug("onEvent #{event.getQueryOperation} #{key} #{newValue}")
process_event(event.getNewValue, "onEvent", "gemfire://query/#{key}/#{event.getQueryOperation}")
end
def onError(event)
@logger.debug("onError #{event}")
end
#
# CacheListener interface
#
protected
def afterCreate(event)
regionName = event.getRegion.getName
key = event.getKey
newValue = event.getNewValue
@logger.debug("afterCreate #{regionName} #{key} #{newValue}")
process_event(event.getNewValue, "afterCreate", "gemfire://#{regionName}/#{key}/afterCreate")
end
def afterDestroy(event)
regionName = event.getRegion.getName
key = event.getKey
newValue = event.getNewValue
@logger.debug("afterDestroy #{regionName} #{key} #{newValue}")
process_event(nil, "afterDestroy", "gemfire://#{regionName}/#{key}/afterDestroy")
end
def afterUpdate(event)
regionName = event.getRegion.getName
key = event.getKey
oldValue = event.getOldValue
newValue = event.getNewValue
@logger.debug("afterUpdate #{regionName} #{key} #{oldValue} -> #{newValue}")
process_event(event.getNewValue, "afterUpdate", "gemfire://#{regionName}/#{key}/afterUpdate")
end
def afterRegionLive(event)
@logger.debug("afterRegionLive #{event}")
end
def afterRegionCreate(event)
@logger.debug("afterRegionCreate #{event}")
end
def afterRegionClear(event)
@logger.debug("afterRegionClear #{event}")
end
def afterRegionDestroy(event)
@logger.debug("afterRegionDestroy #{event}")
end
def afterRegionInvalidate(event)
@logger.debug("afterRegionInvalidate #{event}")
end
end # class LogStash::Inputs::Amqp

4
lib/logstash/inputs/lumberjack.rb
View file

@ -44,7 +44,9 @@ class LogStash::Inputs::Lumberjack < LogStash::Inputs::Base
event = to_event(l.delete("line"), source) event = to_event(l.delete("line"), source)
# take any remaining fields in the lumberjack event and merge it as a # take any remaining fields in the lumberjack event and merge it as a
# field in the logstash event. # field in the logstash event.
event.fields.merge!(l) l.each do |key, value|
event[key] = value
end
output_queue << event output_queue << event
end end
end # def run end # def run

2
lib/logstash/inputs/udp.rb
View file

@ -48,7 +48,7 @@ class LogStash::Inputs::Udp < LogStash::Inputs::Base
def udp_listener(output_queue) def udp_listener(output_queue)
@logger.info("Starting UDP listener", :address => "#{@host}:#{@port}") @logger.info("Starting UDP listener", :address => "#{@host}:#{@port}")
if @udp if @udp && ! @udp.closed?
@udp.close @udp.close
end end

1
lib/logstash/inputs/xmpp.rb
View file

@ -65,6 +65,7 @@ class LogStash::Inputs::Xmpp < LogStash::Inputs::Base
queue << e queue << e
end end
end # @client.add_message_callback end # @client.add_message_callback
sleep
end # def run end # def run
end # def class LogStash:Inputs::Xmpp end # def class LogStash:Inputs::Xmpp

2
lib/logstash/inputs/zeromq.rb
View file

@ -32,7 +32,7 @@ class LogStash::Inputs::ZeroMQ < LogStash::Inputs::Base
# you can change the 'mode' setting # you can change the 'mode' setting
# TODO (lusis) add req/rep MAYBE # TODO (lusis) add req/rep MAYBE
# TODO (lusis) add router/dealer # TODO (lusis) add router/dealer
config :topology, :validate => ["pushpull", "pubsub", "pair"] config :topology, :validate => ["pushpull", "pubsub", "pair"], :required => true
# 0mq topic # 0mq topic
# This is used for the 'pubsub' topology only # This is used for the 'pubsub' topology only

7
lib/logstash/outputs/amqp.rb
View file

@ -3,6 +3,11 @@ require "logstash/namespace"
# Push events to an AMQP exchange. # Push events to an AMQP exchange.
# #
# <b> NOTE: THIS IS ONLY KNOWN TO WORK WITH RECENT RELEASES OF RABBITMQ. Any
# other amqp broker will not work with this plugin. I do not know why. If you
# need support for brokers other than rabbitmq, please file bugs here:
# <https://github.com/ruby-amqp/bunny> </b>
#
# AMQP is a messaging system. It requires you to run an AMQP server or 'broker' # AMQP is a messaging system. It requires you to run an AMQP server or 'broker'
# Examples of AMQP servers are [RabbitMQ](http://www.rabbitmq.com/) and # Examples of AMQP servers are [RabbitMQ](http://www.rabbitmq.com/) and
# [QPid](http://qpid.apache.org/) # [QPid](http://qpid.apache.org/)
@ -10,7 +15,7 @@ class LogStash::Outputs::Amqp < LogStash::Outputs::Base
MQTYPES = [ "fanout", "direct", "topic" ] MQTYPES = [ "fanout", "direct", "topic" ]
config_name "amqp" config_name "amqp"
plugin_status "beta" plugin_status "unsupported"
# Your amqp server address # Your amqp server address
config :host, :validate => :string, :required => true config :host, :validate => :string, :required => true

20
lib/logstash/outputs/elasticsearch.rb
View file

@ -82,19 +82,14 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
@logger.setup_log4j @logger.setup_log4j
if @embedded if @embedded
# Check for settings that are incompatible with @embedded # Default @host with embedded to localhost. This should help avoid
%w(host).each do |name| # newbies tripping on ubuntu and other distros that have a default
if instance_variable_get("@#{name}") # firewall that blocks multicast.
@logger.error("outputs/elasticsearch: You cannot specify " \ @host ||= "localhost"
"'embedded => true' and also set '#{name}'")
raise "Invalid configuration detected. Please fix."
end
end
# Start elasticsearch local. # Start elasticsearch local.
start_local_elasticsearch start_local_elasticsearch
end end
require "jruby-elasticsearch" require "jruby-elasticsearch"
@logger.info("New ElasticSearch output", :cluster => @cluster, @logger.info("New ElasticSearch output", :cluster => @cluster,
@ -167,10 +162,15 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
#timer.stop #timer.stop
decrement_inflight_request_count decrement_inflight_request_count
end.on(:failure) do |exception| end.on(:failure) do |exception|
@logger.warn("Failed to index an event", :exception => exception, @logger.warn("Failed to index an event, will retry", :exception => exception,
:event => event.to_hash) :event => event.to_hash)
#timer.stop #timer.stop
decrement_inflight_request_count decrement_inflight_request_count
# Failed to index, try again after a short sleep (incase our hammering is
# the problem).
sleep(0.200)
receive(event)
end end
# Execute this request asynchronously. # Execute this request asynchronously.

102
lib/logstash/outputs/gemfire.rb Normal file
View file

@ -0,0 +1,102 @@
require "logstash/outputs/base"
require "logstash/namespace"
# Push events to a GemFire region.
#
# GemFire is an object database.
#
# To use this plugin you need to add gemfire.jar to your CLASSPATH;
# using format=json requires jackson.jar too.
#
# Note: this plugin has only been tested with GemFire 7.0.
#
class LogStash::Outputs::Gemfire < LogStash::Outputs::Base
config_name "gemfire"
plugin_status "experimental"
# Your client cache name
config :name, :validate => :string, :default => "logstash"
# The path to a GemFire client cache XML file.
#
# Example:
#
# <client-cache>
# <pool name="client-pool">
# <locator host="localhost" port="31331"/>
# </pool>
# <region name="Logstash">
# <region-attributes refid="CACHING_PROXY" pool-name="client-pool" >
# </region-attributes>
# </region>
# </client-cache>
#
config :cache_xml_file, :validate => :string, :default => nil
# The region name
config :region_name, :validate => :string, :default => "Logstash"
# A sprintf format to use when building keys
config :key_format, :validate => :string, :default => "%{@source}-%{@timestamp}"
public
def register
import com.gemstone.gemfire.cache.client.ClientCacheFactory
import com.gemstone.gemfire.pdx.JSONFormatter
@logger.info("Registering output", :plugin => self)
connect
end # def register
public
def connect
begin
@logger.debug("Connecting to GemFire #{@name}")
@cache = ClientCacheFactory.new.
set("name", @name).
set("cache-xml-file", @cache_xml_file).create
@logger.debug("Created cache #{@cache.inspect}")
rescue => e
if terminating?
return
else
@logger.error("Gemfire connection error (during connect), will reconnect",
:exception => e, :backtrace => e.backtrace)
sleep(1)
retry
end
end
@region = @cache.getRegion(@region_name);
@logger.debug("Created region #{@region.inspect}")
end # def connect
public
def receive(event)
return unless output?(event)
@logger.debug("Sending event", :destination => to_s, :event => event)
key = event.sprintf @key_format
message = JSONFormatter.fromJSON(event.to_json)
@logger.debug("Publishing message", { :destination => to_s, :message => message, :key => key })
@region.put(key, message)
end # def receive
public
def to_s
return "gemfire://#{name}"
end
public
def teardown
@cache.close if @cache
@cache = nil
finished
end # def teardown
end # class LogStash::Outputs::Gemfire

50
lib/logstash/outputs/lumberjack.rb Normal file
View file

@ -0,0 +1,50 @@
class LogStash::Outputs::Lumberjack < LogStash::Outputs::Base
config_name "lumberjack"
plugin_status "experimental"
# list of addresses lumberjack can send to
config :hosts, :validate => :array, :required => true
# the port to connect to
config :port, :validate => :number, :required => true
# ssl certificate to use
config :ssl_certificate, :validate => :string, :required => true
# window size
config :window_size, :validate => :number, :default => 5000
public
def register
require 'lumberjack/client'
connect
end # def register
public
def receive(event)
return unless output?(event)
begin
@client.write("line" => event.message, "host" => event.source_host, "file" => event.source_path)
rescue Exception => e
@logger.log("Client write error", :e => e, :backtrace => e.backtrace)
connect
retry
end
end # def receive
private
def connect
@logger.info("Connecting to lumberjack server.", :addresses => @hosts, :port => @port,
:ssl_certificate => @ssl_certificate, :window_size => @window_size)
begin
@client = Lumberjack::Client.new(:addresses => @hosts, :port => @port,
:ssl_certificate => @ssl_certificate, :window_size => @window_size)
rescue Exception => e
@logger.error("All hosts unavailable, sleeping", :hosts => @hosts, :e => e,
:backtrace => e.backtrace, :host => @client.host)
sleep(10)
retry
end
end
end

3
lib/logstash/outputs/nagios_nsca.rb
View file

@ -85,10 +85,11 @@ class LogStash::Outputs::NagiosNsca < LogStash::Outputs::Base
msg.gsub!("'", "&#146;") msg.gsub!("'", "&#146;")
status = event.sprintf(@nagios_status) status = event.sprintf(@nagios_status)
if status.to_i != status if status.to_i.to_s != status # Check it round-trips to int correctly
msg = "status '#{status}' is not numeric" msg = "status '#{status}' is not numeric"
status = 2 status = 2
else else
status = status.to_i
if status > 3 || status < 0 if status > 3 || status < 0
msg "status must be > 0 and <= 3, not #{status}" msg "status must be > 0 and <= 3, not #{status}"
status = 2 status = 2

2
lib/logstash/outputs/zeromq.rb
View file

@ -31,7 +31,7 @@ class LogStash::Outputs::ZeroMQ < LogStash::Outputs::Base
# you can change the 'mode' setting # you can change the 'mode' setting
# TODO (lusis) add req/rep MAYBE # TODO (lusis) add req/rep MAYBE
# TODO (lusis) add router/dealer # TODO (lusis) add router/dealer
config :topology, :validate => ["pushpull", "pubsub", "pair"] config :topology, :validate => ["pushpull", "pubsub", "pair"], :required => true
# 0mq topic # 0mq topic
# This is used for the 'pubsub' topology only # This is used for the 'pubsub' topology only

43
lib/logstash/runner.rb
View file

@ -1,3 +1,16 @@
# Monkeypatch for JRUBY-6970
module Kernel
alias_method :require_JRUBY_6970_hack, :require
def require(path)
if path =~ /^jar:file:.+!.+/
path = path.gsub(/^jar:/, "")
puts "JRUBY-6970: require(#{path})" if ENV["REQUIRE_DEBUG"] == "1"
end
return require_JRUBY_6970_hack(path)
end
end
require "rubygems" require "rubygems"
require "logstash/namespace" require "logstash/namespace"
require "logstash/program" require "logstash/program"
@ -74,6 +87,36 @@ class LogStash::Runner
@runners << test @runners << test
return test.run(args) return test.run(args)
end, end,
"rspec" => lambda do
require "rspec/core/runner"
require "rspec"
fixedargs = args.collect do |arg|
# if the arg ends in .rb or has a "/" in it, assume it's a path.
if arg =~ /\.rb$/ || arg =~ /\//
# check if it's a file, if not, try inside the jar if we are in it.
if !File.exists?(arg) && __FILE__ =~ /file:.*\.jar!\//
# Try inside the jar.
jar_root = __FILE__.gsub(/!.*/,"!")
newpath = File.join(jar_root, args.first)
if File.exists?(newpath)
# Add the 'spec' dir to the load path so specs can run
specpath = File.join(jar_root, "spec")
$: << specpath unless $:.include?(specpath)
newpath
else
arg
end
else
arg
end
else
arg
end
end # args.collect
RSpec::Core::Runner.run(fixedargs)
return []
end,
"irb" => lambda do "irb" => lambda do
require "irb" require "irb"
return IRB.start(__FILE__) return IRB.start(__FILE__)

2
lib/logstash/version.rb
View file

@ -1,5 +1,5 @@
# The version of logstash. # The version of logstash.
LOGSTASH_VERSION = "1.1.4" LOGSTASH_VERSION = "1.1.6.dev"
# Note to authors: this should not include dashes because 'gem' barfs if # Note to authors: this should not include dashes because 'gem' barfs if
# you include a dash in the version string. # you include a dash in the version string.

1
logstash-event.gemspec
View file

@ -7,6 +7,7 @@ Gem::Specification.new do |gem|
gem.description = %q{Library that contains the classes required to create LogStash events} gem.description = %q{Library that contains the classes required to create LogStash events}
gem.summary = %q{Library that contains the classes required to create LogStash events} gem.summary = %q{Library that contains the classes required to create LogStash events}
gem.homepage = "https://github.com/logstash/logstash" gem.homepage = "https://github.com/logstash/logstash"
gem.license = "Apache License (2.0)"
gem.files = %w{ gem.files = %w{
lib/logstash-event.rb lib/logstash-event.rb

21
logstash.gemspec
View file

@ -17,14 +17,14 @@ Gem::Specification.new do |gem|
gem.version = LOGSTASH_VERSION gem.version = LOGSTASH_VERSION
# Core dependencies # Core dependencies
gem.add_runtime_dependency "cabin", ["0.4.4"] gem.add_runtime_dependency "cabin", ["0.5.0"]
gem.add_runtime_dependency "json" gem.add_runtime_dependency "json"
gem.add_runtime_dependency "minitest" # for running the tests from the jar gem.add_runtime_dependency "minitest" # for running the tests from the jar
gem.add_runtime_dependency "pry" gem.add_runtime_dependency "pry"
gem.add_runtime_dependency "stud" gem.add_runtime_dependency "stud"
# Web dependencies # Web dependencies
gem.add_runtime_dependency "ftw", ["~> 0.0.22"] gem.add_runtime_dependency "ftw", ["~> 0.0.23"]
gem.add_runtime_dependency "haml" gem.add_runtime_dependency "haml"
gem.add_runtime_dependency "rack" gem.add_runtime_dependency "rack"
gem.add_runtime_dependency "sass" gem.add_runtime_dependency "sass"
@ -34,7 +34,7 @@ Gem::Specification.new do |gem|
#TODO Can these be optional? #TODO Can these be optional?
gem.add_runtime_dependency "aws-sdk" gem.add_runtime_dependency "aws-sdk"
gem.add_runtime_dependency "heroku" gem.add_runtime_dependency "heroku"
gem.add_runtime_dependency "addressable", ["2.2.6"] gem.add_runtime_dependency "addressable", ["~> 2.2.6"]
gem.add_runtime_dependency "bunny" gem.add_runtime_dependency "bunny"
gem.add_runtime_dependency "ffi" gem.add_runtime_dependency "ffi"
gem.add_runtime_dependency "ffi-rzmq", ["0.9.3"] gem.add_runtime_dependency "ffi-rzmq", ["0.9.3"]
@ -42,7 +42,7 @@ Gem::Specification.new do |gem|
gem.add_runtime_dependency "gelfd", ["0.2.0"] gem.add_runtime_dependency "gelfd", ["0.2.0"]
gem.add_runtime_dependency "gelf", ["1.3.2"] gem.add_runtime_dependency "gelf", ["1.3.2"]
gem.add_runtime_dependency "gmetric", ["0.1.3"] gem.add_runtime_dependency "gmetric", ["0.1.3"]
gem.add_runtime_dependency "jls-grok", ["0.10.7"] gem.add_runtime_dependency "jls-grok", ["0.10.8"]
gem.add_runtime_dependency "mail" gem.add_runtime_dependency "mail"
gem.add_runtime_dependency "mongo" gem.add_runtime_dependency "mongo"
gem.add_runtime_dependency "onstomp" gem.add_runtime_dependency "onstomp"
@ -53,7 +53,7 @@ Gem::Specification.new do |gem|
gem.add_runtime_dependency "uuidtools" # For generating amqp queue names gem.add_runtime_dependency "uuidtools" # For generating amqp queue names
gem.add_runtime_dependency "xml-simple" gem.add_runtime_dependency "xml-simple"
gem.add_runtime_dependency "xmpp4r", ["0.5"] gem.add_runtime_dependency "xmpp4r", ["0.5"]
gem.add_runtime_dependency "jls-lumberjack" gem.add_runtime_dependency "jls-lumberjack", ["0.0.4"]
gem.add_runtime_dependency "geoip", [">= 1.1.0"] gem.add_runtime_dependency "geoip", [">= 1.1.0"]
gem.add_runtime_dependency "beefcake", "0.3.7" gem.add_runtime_dependency "beefcake", "0.3.7"
@ -71,9 +71,10 @@ Gem::Specification.new do |gem|
gem.add_runtime_dependency "cinch" # cinch requires 1.9.1+ gem.add_runtime_dependency "cinch" # cinch requires 1.9.1+
end end
gem.add_development_dependency "spoon" # These are runtime-deps so you can do 'java -jar logstash.jar rspec <test>'
gem.add_development_dependency "mocha" gem.add_runtime_dependency "spoon"
gem.add_development_dependency "shoulda" gem.add_runtime_dependency "mocha"
gem.add_development_dependency "rspec" gem.add_runtime_dependency "shoulda"
gem.add_development_dependency "insist", "0.0.7" gem.add_runtime_dependency "rspec"
gem.add_runtime_dependency "insist", "0.0.8"
end end

2
patterns/firewalls
View file

@ -1,2 +1,2 @@
# NetScreen firewall logs # NetScreen firewall logs
NETSCREENSESSIONLOG %{SYSLOGDATE:date} %{IPORHOST:device} %{IPORHOST}: NetScreen device_id=%{WORD:device_id}%{DATA}: start_time=%{QUOTEDSTRING:start_time} duration=%{INT:duration} policy_id=%{INT:policy_id} service=%{DATA:service} proto=%{INT:proto} src zone=%{WORD:src_zone} dst zone=%{WORD:dst_zone} action=%{WORD:action} sent=%{INT:sent} rcvd=%{INT:rcvd} src=%{IPORHOST:src_ip} dst=%{IPORHOST:dst_ip} src_port=%{INT:src_port} dst_port=%{INT:dst_port} src-xlated ip=%{IPORHOST:src_xlated_ip} port=%{INT:src_xlated_port} dst-xlated ip=%{IPORHOST:dst_xlated_ip} port=%{INT:dst_xlated_port} session_id=%{INT:session_id} reason=%{GREEDYDATA:reason} NETSCREENSESSIONLOG %{SYSLOGTIMESTAMP:date} %{IPORHOST:device} %{IPORHOST}: NetScreen device_id=%{WORD:device_id}%{DATA}: start_time=%{QUOTEDSTRING:start_time} duration=%{INT:duration} policy_id=%{INT:policy_id} service=%{DATA:service} proto=%{INT:proto} src zone=%{WORD:src_zone} dst zone=%{WORD:dst_zone} action=%{WORD:action} sent=%{INT:sent} rcvd=%{INT:rcvd} src=%{IPORHOST:src_ip} dst=%{IPORHOST:dst_ip} src_port=%{INT:src_port} dst_port=%{INT:dst_port} src-xlated ip=%{IPORHOST:src_xlated_ip} port=%{INT:src_xlated_port} dst-xlated ip=%{IPORHOST:dst_xlated_ip} port=%{INT:dst_xlated_port} session_id=%{INT:session_id} reason=%{GREEDYDATA:reason}

8
patterns/grok-patterns
View file

@ -30,12 +30,12 @@ HOSTPORT (?:%{IPORHOST=~/\./}:%{POSINT})
# paths # paths
PATH (?:%{UNIXPATH}|%{WINPATH}) PATH (?:%{UNIXPATH}|%{WINPATH})
UNIXPATH (?:/(?:[\w_%!$@:.,-]+|\\.)*)+ UNIXPATH (?>/(?>[\w_%!$@:.,-]+|\\.)*)+
#UNIXPATH (?<![\w\/])(?:/[^\/\s?*]*)+ #UNIXPATH (?<![\w\/])(?:/[^\/\s?*]*)+
LINUXTTY (?:/dev/pts/%{NONNEGINT}) LINUXTTY (?>/dev/pts/%{NONNEGINT})
BSDTTY (?:/dev/tty[pq][a-z0-9]) BSDTTY (?>/dev/tty[pq][a-z0-9])
TTY (?:%{BSDTTY}|%{LINUXTTY}) TTY (?:%{BSDTTY}|%{LINUXTTY})
WINPATH (?:[A-Za-z]+:|\\)(?:\\[^\\?*]*)+ WINPATH (?>[A-Za-z]+:|\\)(?:\\[^\\?*]*)+
URIPROTO [A-Za-z]+(\+[A-Za-z+]+)? URIPROTO [A-Za-z]+(\+[A-Za-z+]+)?
URIHOST %{IPORHOST}(?::%{POSINT:port})? URIHOST %{IPORHOST}(?::%{POSINT:port})?
# uripath comes loosely from RFC1738, but mostly from what Firefox # uripath comes loosely from RFC1738, but mostly from what Firefox

91
pl.rb Normal file
View file

@ -0,0 +1,91 @@
# pipeline tests
$: << "lib"
require "logstash/config/file"
config = LogStash::Config::File.new(nil, ARGV[0])
agent = LogStash::Agent.new
inputs, filters, outputs = agent.instance_eval { parse_config(config) }
inputs.collect(&:register)
filters.collect(&:register)
outputs.collect(&:register)
i2f = SizedQueue.new(16)
f2o = SizedQueue.new(16)
i2f = f2o if filters.empty?
input_threads = inputs.collect do |i|
t = Thread.new do
begin
i.run(i2f)
rescue => e
puts :input => i.class, :exception => e
end
end
t[:name] = i.class
t
end
#input_supervisor_thread = Thread.new do
#while true
#input_threads.collect(&:join)
#i2f << :shutdown
#end
#end
filter_thread = Thread.new(filters) do |filters|
if filters.any?
event = i2f.pop
filters.each do |filter|
filter.filter(event)
end
f2o << event
end
end
filter_thread[:name] = "filterworker"
output_thread = Thread.new do
begin
while true
event = f2o.pop
outputs.each do |output|
output.receive(event)
end
end
rescue => e
puts :output_thread => e
end
end
output_thread[:name] = "outputworker"
def twait(thread)
begin
puts :waiting => thread[:name]
thread.join
puts :donewaiting => thread[:name]
rescue => e
puts thread => e
end
end
def shutdown(input, filter, output)
input.each do |i|
i.raise("SHUTDOWN")
twait(i)
end
#filter.raise("SHUTDOWN")
#twait(filter)
output.raise("SHUTDOWN")
twait(output)
end
trap("INT") do
puts "SIGINT"; shutdown(input_threads, filter_thread, output_thread)
exit 1
end
#[*input_threads, filter_thread, output_thread].collect(&:join)
sleep 30

118
pl2.rb Normal file
View file

@ -0,0 +1,118 @@
$: << "lib"
require "logstash/config/file"
class Pipeline
class ShutdownSignal; end
def initialize(configstr)
# hacks for now to parse a config string
config = LogStash::Config::File.new(nil, configstr)
agent = LogStash::Agent.new
@inputs, @filters, @outputs = agent.instance_eval { parse_config(config) }
@inputs.collect(&:register)
@filters.collect(&:register)
@outputs.collect(&:register)
@input_to_filter = SizedQueue(16)
@filter_to_output = SizedQueue(16)
# If no filters, pipe inputs to outputs
if @filters.empty?
input_to_filter = filter_to_output
end
end
def run
# one thread per input
@input_threads = @inputs.collect do |input|
Thread.new(input) do |input|
inputworker(input)
end
end
# one filterworker thread
#@filter_threads = @filters.collect do |input
# TODO(sissel): THIS IS WHERE I STOPPED WORKING
# one outputworker thread
# Now monitor input threads state
# if all inputs are terminated, send shutdown signal to @input_to_filter
end
def inputworker(plugin)
begin
plugin.run(@input_to_filter)
rescue ShutdownSignal
plugin.teardown
rescue => e
@logger.error("Exception in plugin #{plugin.class}, restarting plugin.",
"plugin" => plugin.inspect, "exception" => e)
plugin.teardown
retry
end
end # def
def filterworker
begin
while true
event << @input_to_filter
break if event == :shutdown
@filters.each do |filter|
filter.filter(event)
end
next if event.cancelled?
@filter_to_output << event
end
rescue => e
@logger.error("Exception in plugin #{plugin.class}",
"plugin" => plugin.inspect, "exception" => e)
end
@filters.each(&:teardown)
end # def filterworker
def outputworker
begin
while true
event << @filter_to_output
break if event == :shutdown
@outputs.each do |output|
output.receive(event)
end
end
rescue => e
@logger.error("Exception in plugin #{plugin.class}",
"plugin" => plugin.inspect, "exception" => e)
end
@outputs.each(&:teardown)
end # def filterworker
end # class Pipeline
def twait(thread)
begin
puts :waiting => thread[:name]
thread.join
puts :donewaiting => thread[:name]
rescue => e
puts thread => e
end
end
def shutdown(input, filter, output)
input.each do |i|
i.raise("SHUTDOWN")
end
#filter.raise("SHUTDOWN")
#twait(filter)
output.raise("SHUTDOWN")
twait(output)
end
trap("INT") do
puts "SIGINT"; shutdown(input_threads, filter_thread, output_thread)
exit 1
end

2
spec/filters/date_performance.rb
View file

@ -7,7 +7,7 @@ describe LogStash::Filters::Date do
describe "performance test of java syntax parsing" do describe "performance test of java syntax parsing" do
event_count = 100000 event_count = 100000
min_rate = 10000 min_rate = 5000
max_duration = event_count / min_rate max_duration = event_count / min_rate
input = "Nov 24 01:29:01 -0800" input = "Nov 24 01:29:01 -0800"

27
spec/filters/grok/timeout2.rb Normal file
View file

@ -0,0 +1,27 @@
require "test_utils"
require "grok-pure"
require "timeout"
describe "grok known timeout failures" do
extend LogStash::RSpec
describe "user reported timeout" do
config <<-'CONFIG'
filter {
grok {
match => [ "@message", "%{SYSLOGBASE:ts1} \[\#\|%{TIMESTAMP_ISO8601:ts2}\|%{DATA} for %{PATH:url} = %{POSINT:delay} ms.%{GREEDYDATA}" ]
}
}
CONFIG
start = Time.now
line = 'Nov 13 19:23:34 qa-api1 glassfish: [#|2012-11-13T19:23:25.604+0000|INFO|glassfish3.1.2|com.locusenergy.platform.messages.LocusMessage|_ThreadID=59;_ThreadName=Thread-2;|API TIMER - Cache HIT user: null for /kiosks/194/energyreadings/data?tz=America/New_York&fields=kwh&type=gen&end=2012-11-13T23:59:59&start=2010-12-16T00:00:00-05:00&gran=yearly = 5 ms.|#]'
sample line do
duration = Time.now - start
insist { duration } < 0.03
end
end
end
__END__

16
spec/jar.rb Normal file
View file

@ -0,0 +1,16 @@
require "insist"
describe "logstash jar features" do
before :each do
@jar_root = __FILE__.split("!").first + "!"
end
it "must be only run from a jar" do
insist { __FILE__ } =~ /file:.*!/
end
it "must contain GeoLiteCity.dat" do
path = File.join(@jar_root, "GeoLiteCity.dat")
insist { File }.exists?(path)
end
end

66
spec/support/Ec8ISgamIfo.rb Normal file
View file

@ -0,0 +1,66 @@
# This spec covers the question here:
# https://groups.google.com/forum/?fromgroups=#!topic/logstash-users/Ec8ISgamIfo
require "test_utils"
describe "https://groups.google.com/forum/?fromgroups=#!topic/logstash-users/Ec8ISgamIfo" do
extend LogStash::RSpec
config <<-'CONFIG'
filter {
multiline {
type => "java-log"
pattern => "^20"
negate => "true"
what => "previous"
}
grok {
type => "java-log"
tags => [ "dev", "console", "multiline" ]
singles => true
add_tag => "mytag"
match => [ "@message", "^%{DATESTAMP:log_time}%{SPACE}\[%{PROG:thread}\]%{SPACE}%{LOGLEVEL:log_level}%{SPACE}%{WORD:class_name}%{GREEDYDATA}"]
}
}
CONFIG
type "java-log"
tags "dev", "console"
line1 = '2012-11-13 13:55:37,706 [appname.connector.http.mule.default.receiver.14] INFO LoggerMessageProcessor - SRC message is <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:icc="http://researchnow.com/schema/icc"'
line2 = "hello world"
sample line1 do
insist { subject.tags }.include?("dev")
insist { subject.tags }.include?("console")
# This is not a multiline event, so it won't get tagged as multiline
reject { subject.tags }.include?("multiline")
# Since this event doesn't have the 'multiline' tag, grok will not act on
# it, so it should not have the 'mytag' tag given in the grok filter's
# add_tag setting.
reject { subject.tags }.include?("mytag")
end
# Try with a proper multiline event
sample [ line1, line2 ] do
insist { subject.count } == 1
event = subject.first # get the first event.
insist { event.tags }.include?("dev")
insist { event.tags }.include?("console")
insist { event.tags }.include?("multiline")
# grok shouldn't fail.
reject { event.tags }.include?("_grokparsefailure")
# Verify grok is working and pulling out certain fields
insist { event.tags }.include?("mytag")
insist { event["log_time"] } == "2012-11-13 13:55:37,706"
insist { event["thread"] } == "appname.connector.http.mule.default.receiver.14"
insist { event["log_level"] } == "INFO"
insist { event["class_name"] } == "LoggerMessageProcessor"
end
end

37
spec/test_utils.rb
View file

@ -18,12 +18,24 @@ module LogStash
@config_str = configstr @config_str = configstr
end # def config end # def config
def type(default_type)
@default_type = default_type
end
def tags(*tags)
@default_tags = tags
puts "Setting default tags: #{@default_tags}"
end
def sample(event, &block) def sample(event, &block)
default_type = @default_type || "default"
default_tags = @default_tags || []
require "logstash/config/file" require "logstash/config/file"
config = LogStash::Config::File.new(nil, @config_str) config = LogStash::Config::File.new(nil, @config_str)
agent = LogStash::Agent.new agent = LogStash::Agent.new
@inputs, @filters, @outputs = agent.instance_eval { parse_config(config) } @inputs, @filters, @outputs = agent.instance_eval { parse_config(config) }
[@inputs, @filters, @outputs].flatten.each do |plugin| [@inputs, @filters, @outputs].flatten.each do |plugin|
plugin.logger = Cabin::Channel.get
plugin.register plugin.register
end end
@ -38,7 +50,8 @@ module LogStash
event = [event] unless event.is_a?(Array) event = [event] unless event.is_a?(Array)
event = event.collect do |e| event = event.collect do |e|
if e.is_a?(String) if e.is_a?(String)
LogStash::Event.new("@message" => e) LogStash::Event.new("@message" => e, "@type" => default_type,
"@tags" => default_tags)
else else
LogStash::Event.new(e) LogStash::Event.new(e)
end end
@ -53,13 +66,25 @@ module LogStash
results << e unless e.cancelled? results << e unless e.cancelled?
end end
filters.select { |f| f.respond_to?(:flush) }.each do |filter| # do any flushing.
event = filter.flush filters.each_with_index do |filter, i|
results += event if event if filter.respond_to?(:flush)
end # get any event from flushing
list = filter.flush
if list
list.each do |e|
filters[i+1 .. -1].each do |f|
f.filter(e)
end
results << e unless e.cancelled?
end
end # if list
end # filter.respond_to?(:flush)
end # filters.each_with_index
@results = results @results = results
end end # before :all
if multiple_events if multiple_events
subject { @results } subject { @results }
else else