add test for subclasses of codec, filter, input and output
Register codec clone, use String clone not dup so frozen strings stay so
Fixes#4444Fixes#4473
We were getting some bizzare
errors (https://github.com/elastic/logstash/pull/4391#issuecomment-169477728)
on the file output with the new OutputDelegator changes. By switching
away from method(:mymethod) to blocks with define_singleton_method they
went away.
This bug is only present with Jruby 1.7.x (tested with 1.7.22 and
1.7.23). Jruby 9.x is fine.
Fixes#4391
When you run multiples pipeline and the config code get evaluated, on
every evaluation the class cached is clear, everytime you were calling a
`func` method you had the latest evaluated code. The `filter_func` and
the `output_func` need to be unique for every instance of the pipeline,
this PR replace the `def` with a `define_single_method` call ensuring
the uniqueness of the code on every instance.
This PR is based on #4254Fixes#4298
slightly adjust the metaphor for the class that is responsible for
monitoring a pipeline's shutdown process, report on its status and
intervene if it's stalled
This is predicated on the fact that with the ng_pipeline it is expected that workers
will spend a significant amount of time in iowait due to outputs like the Elasticsearch
output. In benchmarks based on real-world Apache log files the best performance came out
of scenarios where pipeline_workers > num_cpu_cores. Setting this to default to the # of
cores is a defensive decision that should handle cases where users have particularly IO
heavy inputs.
For most users we should recommend setting the number of workers to be as high as possible
until performance decreases.
Previous benchmark information:
https://github.com/elastic/logstash/pull/4340Fixes#4414
Expose the Java's `#offer` method of the `SynchronousQueue` class,
This method allow the input developper to correctly apply back pressure
to the network clients.
If you use `#push` it will block forever until the pipeline free some
space, this behavior is sufficient if the pipeline outputs are healthy.
But if the output stale the backpressure will be applied up to the input
(producers) when reading a file this scenario is fine since we will just
stop reading the file until the thread unblock again.
In the context of network clients, the story is a bit different the
clients will timeout and try to reconnect to restransmit their payload
creating multiple new connection thread block on the queue. In some case
this will lead into a OOM issues.
This PR is the first step to communicate that the queue is under
pressure.
Fixes#4408
* start logstash with --allow-unsafe-shutdown to force_exit on stalled shutdown
* by default --allow-unsafe-shutdown is disabled
* stall detection kicks in when SIGTERM/SIGINT is received
* check if inflight event count isn't going down and if there are blocked/blocking plugin threads
including dependencies, and reuse them in an offline installation by
providing an package for it. It adds two important commands to the
plugin manager, the pack and upack, to handle package creation and
installation and adds the --local flag to install and update to pull
plugins from the installed local packages.
Former commits:
add a task to create a bundle, under vendor/cache of the installed gems + their dependencies, this can be used later on to be installed offline
add an option to pass the --local flag to the bin/plugin update task, so it fetch plugins from a local cache, by default under vendor/cache
rename package:build to package:build-cache as is more meaningfull
add a --local flag to bin/plugin install to users can install plugins from the local cache, under the default directory of vendor/cache
add a plugin manager command to build the local cache of installed plugins using bundler package alike command
It adds code for compressing and extracting in zip and tar formats to
the utils module of logstash. The zip module is only jruby dependant as
it uses functions provided by java.
There is also code add in the plugin manager package command to handle
compression of plugins dumping.
Cleanup the custom java code to compress and extract zip files as it has
been known that using rubyzip is good and it has a more ruby like
features so the code is more clean and portable.
set of smallish improvement requested during review
added several options to handle situation when the package of plugins we want to generate is already there
clean up old code
applyed feedback from review, mostly changed in documentating behaviour plus better wording for messages
relocate the Environment.windows? check to the bootstrap side as it's used also in the plugin manager
add an unpack bin/plugin command so users can install packages of plugins throw the plugin manager
document override behaviour in the compress helpers (for zip and tar) plus add a fix for the tar extract when reading entries
made the unpack filename parameter a required one
add a force option to the bin/plugin unpack command
add a notice to that if using a local cache the --local flag should be passed with
Code cleanup and refactor introduced during review
add two wording suggestions comming from review
ammend more wording
skip the major version validation as in situation where there is no internet connection like when using the local vendor/cache to do the update
move compress to the bootstrap environment as being used in the plugin manager means not being loaded with logstash-core
Bring pack cached gems in the vendor directory so they can be used for bundler when packaging dependencies
Revert "Bring pack cached gems in the vendor directory so they can be used for bundler when packaging dependencies"
This reverts commit a9d7f46649932b06efaafebdd0eed2b4c63c2235.
patch the Bundler::Source::Rubygems to fetch only gems from a remote source
small changes in several parts of the plugin manager and the creation of a common pack command with shared code
change compress to read in chuncks
fix wrong var name in the bootstrap compress utils module
fix namespacing conflicts
add basic test for the compress utility module
ammend more namespace issues
add a comment to the rubygems mockey patch to rebuild the gem cache
apply cosmetic changes
make the compress module raise CompressError
remove vendor_path and pattern_path form environment as they where mix up during rebase
change the bin/pack force option to be force-delete-cache
rename force_delete_cache to overwrite
change force for overwrite in tha pack command
remove the override option for bin/plugin unpack
revert Gemfile and Genfile.lock wrong committed
