We were getting some bizzare
errors (https://github.com/elastic/logstash/pull/4391#issuecomment-169477728)
on the file output with the new OutputDelegator changes. By switching
away from method(:mymethod) to blocks with define_singleton_method they
went away.
This bug is only present with Jruby 1.7.x (tested with 1.7.22 and
1.7.23). Jruby 9.x is fine.
Fixes#4391
When you run multiples pipeline and the config code get evaluated, on
every evaluation the class cached is clear, everytime you were calling a
`func` method you had the latest evaluated code. The `filter_func` and
the `output_func` need to be unique for every instance of the pipeline,
this PR replace the `def` with a `define_single_method` call ensuring
the uniqueness of the code on every instance.
This PR is based on #4254Fixes#4298
slightly adjust the metaphor for the class that is responsible for
monitoring a pipeline's shutdown process, report on its status and
intervene if it's stalled
This is predicated on the fact that with the ng_pipeline it is expected that workers
will spend a significant amount of time in iowait due to outputs like the Elasticsearch
output. In benchmarks based on real-world Apache log files the best performance came out
of scenarios where pipeline_workers > num_cpu_cores. Setting this to default to the # of
cores is a defensive decision that should handle cases where users have particularly IO
heavy inputs.
For most users we should recommend setting the number of workers to be as high as possible
until performance decreases.
Previous benchmark information:
https://github.com/elastic/logstash/pull/4340Fixes#4414
Expose the Java's `#offer` method of the `SynchronousQueue` class,
This method allow the input developper to correctly apply back pressure
to the network clients.
If you use `#push` it will block forever until the pipeline free some
space, this behavior is sufficient if the pipeline outputs are healthy.
But if the output stale the backpressure will be applied up to the input
(producers) when reading a file this scenario is fine since we will just
stop reading the file until the thread unblock again.
In the context of network clients, the story is a bit different the
clients will timeout and try to reconnect to restransmit their payload
creating multiple new connection thread block on the queue. In some case
this will lead into a OOM issues.
This PR is the first step to communicate that the queue is under
pressure.
Fixes#4408
These changes allow Logstash to be launched from a symlink _only_ when the `readlink`, or suitable version of `stat` binaries are present. An error message will result if Logstash is launched from a symlink and the path cannot be discovered by one of these two methods. The message will advise the user to use the full path instead of a symlink to launch Logstash.
fixes#4291
