Commit graph

19 commits

Author SHA1 Message Date
Pere Urbon-Bayes
f5a89636b3 Hide sensitive AST/config data unless --debug-config argument passed 2016-04-05 16:29:52 -05:00
Joao Duarte
6c800198b6 ensure Plugin#original_params is obfuscated
Fixes #4952
2016-03-31 19:15:32 +00:00
Andrew Cholakian
60a6d9c2c7 Fix threadsafe workers in output delegators to not instantiate extras Cherry pick of db9e67802fb1f6cd097bcac0a7aa59a3a453ceb1 from master Originally from https://github.com/elastic/logstash/pull/4763
Fixes #4894
2016-03-24 21:11:53 +00:00
Andrew Cholakian
0b02d1a4eb Add specs to code around flusher initialization order
Fixes #4623
2016-02-04 16:41:33 +00:00
Pere Urbon-Bayes
2248fc35bb add test to verify that a cloned codec is only registerd once
Fixes #4533
2016-01-21 16:33:58 +00:00
Guy Boertje
88e399238c Better method of ensuring that args are not mutated by plugin initialize
add test for subclasses of codec, filter, input and output

Register codec clone, use String clone not dup so frozen strings stay so

Fixes #4444

Fixes #4473
2016-01-14 14:15:55 +00:00
Joao Duarte
0b358480ca distinguish between user set and default pipeline settings 2016-01-13 10:13:12 +00:00
Joao Duarte
902e127d71 rename pipeline-worker settings to symbols
Fixes #4430
2016-01-07 18:58:40 +00:00
Andrew Cholakian
d93b4d60f2 Add debug log for determining the number of output workers for output delegator
Fixes #4391
2016-01-07 17:19:10 +00:00
Andrew Cholakian
00c74d84e1 Improve OutputDelegator implementation and specs.
Backwards compatibility is now implemented for existing workers_not_supported uses.

This clears up a few bugs in the initial pass as well.

Fixes #4391
2016-01-07 17:19:10 +00:00
Andrew Cholakian
67d0a8e4ef Improved semantics for OutputDelegator This commit greatly improves the semantics for the OutputDelegator. It now: * Has an API for declaring if an output is threadsafe to avoid instance creation * By default sets the number of workers for an output to be == the number of pipeline workers * Moves the worker safe API to the class, not instance level * Strongly enforces the worker safety API * Should provide better performance for threadsafe plugins
Fixes #4391
2016-01-07 17:19:10 +00:00
Pier-Hugues Pellerin
27e675ce7a Replace def #{type}_func with define_singleton_method
When you run multiples pipeline and the config code get evaluated, on
every evaluation the class cached is clear, everytime you were calling a
`func` method you had the latest evaluated code. The `filter_func` and
the `output_func` need to be unique for every instance of the pipeline,
this PR replace the `def` with a `define_single_method` call ensuring
the uniqueness of the code on every instance.

This PR is based on #4254

Fixes #4298
2016-01-06 20:11:21 +00:00
Joao Duarte
b887705acd rename ShutdownController to ShutdownWatcher
slightly adjust the metaphor for the class that is responsible for
monitoring a pipeline's shutdown process, report on its status and
intervene if it's stalled
2016-01-05 17:41:05 +00:00
Pier-Hugues Pellerin
1d8b88a706 Add a WrappedSynchronousQueue#offer method
Expose the Java's `#offer` method of the `SynchronousQueue` class,
This method allow the input developper to correctly apply back pressure
to the network clients.

If you use `#push` it will block forever until the pipeline free some
space, this behavior is sufficient if the pipeline outputs are healthy.
But if the output stale the backpressure will be applied up to the input
(producers) when reading a file this scenario is fine since we will just
stop reading the file until the thread unblock again.

In the context of network clients, the story is a bit different the
clients will timeout and try to reconnect to restransmit their payload
creating multiple new connection thread block on the queue. In some case
this will lead into a OOM issues.

This PR is the first step to communicate that the queue is under
pressure.

Fixes #4408
2016-01-05 14:33:28 +00:00
Andrew Cholakian
4dcdc0eef7 Warn on potential memory issues when a large number of inflight messages are present
Fixes #4410
2016-01-04 19:14:14 +00:00
Andrew Cholakian
8de54e46ce NG Pipeline
Fixes #4363
2015-12-16 20:03:05 +00:00
Joao Duarte
21dfae1652 fix shutdown controller when using no filters 2015-11-27 10:48:19 +00:00
guyboertje
447b47d9ce add provision for worker threads based on threadsafe workers
updates requested by code review

changes requested by colin: make workers override from -w arg

do not set workers unless user actually specified it via cmdline

fix defaults printing

add describe block to improve test output readability

Closes #4130
2015-11-17 14:37:56 +00:00
Colin Surprenant
12a5982da2 logstash-core & logstash-core-event extraction to support logstash-core-event-java impl, relates to #4191
fixed timezone issue

extracted logstash-core and reorganized specs

extracted logstash-core-event

extract java Event into logstash-core-event-java in a proper gem

remove obsolete jruby_event bootstrapping

fix require path

add java code bootstrap

use logstash-core-event/logstash-core-event.rb

remove obsolete files

basic instructions

LogStash::Json need to be initialized from event

update jruby and gradle versions

update compile:logstash-core-event-java rake task

WIP tasks refactor

fix gem.files

skip test if class is not defined

fix gem related tasks for new structure

add gem spec dirs in core tests

bootstrap java implementation when requiring timestamp

new Cloner class and Event clone impl

fix array fields assignments, see #4140

don't rely on json implementation ordering

fix skipped last iterpolation char

remove implementation specific unnecessary check

also require ruby classes

define error class in ruby

raise exception on invalid format

remove implementation specific tests and extract and put logger related test in pending

missing bits for having all core timestamp specs pass

run all core specs

remove leftover

comment regex

missing encoding header

revert to logstash-core-event by default

finished proper gemification

useless require

dynamically pick specs depending on logstash-core-event-* implementation

logstash root package version

missing file for proper gemification

do not build java event by default

always check for root logstash lib dir

fix concurrent-ruby version confict

fix rebase conflict

re-enable specs

user vars instead of constants

move non core code in bootstrap

document version files

move version file

remove useless code

use version in logstash-core

fix gem files list

put back concurrent-ruby version constrain as in master

add dependency on logstash-core-event

remove dependency on logstash-core to avoid circular dependency

fix rebase conflict

remove circular dependency

fix specs

update README
2015-11-16 17:16:10 -05:00