This PR fix an issue where the time was calculated but no work was done
on the event. This code make sure we have at least one event to start
recording the time spend.
This was causing the `events/duratin_in_millis` to not be in sync with
the time spend on the filtera, since `take_batch` was called in a tight
loop and could return an empty array this made the duration was way off.
Fixes: #5952Fixes#5953
This PR changes how we modify the STDOUT/STDERR in puma, instande of
using `const_set`, we override the constants using a module.
This operation should be thread safe and will make sure the STDERR are
correctly visible.
After when we receive the logger we can swap the null logger with the
real thing in a thread safe way.
Fixes: #5912Fixes#5918
sets default to INFO and updates some verbose logging to
more appropriate, less verbose log levels where it makes sense.
Closes#5735.
Closes#5893.
Fixes#5898
This PR introduce a changes to make sure the API endpoint correctly
return 404 in every case. To make it work it uses an exception that get
pickup up in the base class by the `error` handler.
Using the exception template allow the code to be dry and make sure all
other errors can be returned with the same format.
Example or a 404.
```json
{
"error": { "message": "Not Found"},
"path": "/this-should-not-exist"
"status": 404
}
```
The code will also set the content-type and the right error code in the
header.
Fixes: #5874, #5622Fixes#5897
When Logstash encounter an exception Puma was getting really noisy about
the file descriptor it used. Its because Puma has an infinite loop into
his reactor and when logstash is dying from the exception the FD get
removed under the puma threads.
Puma is using the constants `STDERR` and `STDOUT` to give information
about the errors, there is no easy way to provide Puma with a custom IO.
This PR hacks the Puma name to redefine both constants and send the
information to the logger using the `debug` level.
Also when we start the server we can pass a custom Puma::Events instance
to record some of the errors happening in the internal threads, this PR
also make sure we send all theses message to the configured logger.
Fixes: #5822Fixes#5869
The plugin args are not really necessary for reporting. Now that we have unique IDs
that's a better way to figure which output is which without potentially sharing
sensitive data.
This fixes hanging tests in the previous commit
Fixes#5827
These methods allow outputs to receive their events pre-encoded for them
by the pipeline. This is mostly useful in the context of `#shared` outputs, for whom
encoding a discrete batch in a threadsafe way is not necessarily straightforward.
It would be advised for codecs to prefer `#multi_encode` as the main way of operating
as the standard `#encode` method is not threadsafe.
Fixes#5770
This PR introduce GC stats in the form of `collection_time` and `collection_count`, it takes the configured gc and categorize them
into old and young.
Example output:
```
gc: {
collectors: {
young: {
collection_time_in_millis: 22101,
collection_count: 5452
},
old: {
collection_time_in_millis: 57,
collection_count: 1
}
}
}
```
Fixes: #5730Fixes#5788
This PR will expose the time spend of a batch in the filter + output
Ouput:
http://localhost:9600/_node/stats/pipeline
```
events: {
duration_in_millis: 18364,
in: 309325,
filtered: 309325,
out: 309325
},
```
Fixes: #5731Fixes#5792
This fixes two issues:
1. This fully removes the xopts parameter which for Shared and Single concurrency would prevent outputs from receiving their parameters
2. This cleans up the injection of the OutputDelegatorStrategyRegistry.
This also bumps the plugin api version to 2.1.12
Fixes#5794
This PR does a few modifications for our webserver:
- Add a PortRange setting type.
- allow the --http.port option to accept a port range or a single port, when a range is provided Logstash will incrementally try this list to find a free port for the webserver.
- Logstash will report in the log which port it is using. (INFO LEVEL)
- It refactors a few methods of the webserver.
- It adds test for the binding of the ports.
Notes:
Port range can be defined in the logstash.yml or pass via the command line like this.
`bin/logstash -e 'input { generator {}} output { null {}}' --log.level verbose --http.port 2000-2005`
Fixes#5774
The new way Output Delegators work is that events flow from:
OutputDelegator -> OutputDelegatorStrategy -> Output
The output delegator handles all the common denominator tasks (like metrics) and
a few other things. The OutputDelegatorStrategy handles concurrency and Output instantiation.
This is a significant improvement and simplification over the past where we used mixins and dynamic method
redifinition.
This removes the concept of plugin 'unique_names' and replaces it with the 'id'.
Also consistently autogenerates plugin IDs based on a given config file using SHA1 hashing.
Fixes#5752
