This PR changes where the `events.in` are calculated, previously the
values were calculated in the `ReadClient` which was fine before the
addition of the PQ, but this make the stats not accurate when the PQ was
enabled and the producer are a lot faster than the consumer.
These commits change the collection of the metric inside an
instrumented `WriteClient` so both implementation of the client queues will use
the same code.
This also make possible to record `events.out` for every inputs and the
time waiting to push to the queue.
The API is now exposing theses values for each plugins, the events level
and and the pipeline.
Using a pipeline with a sleep filter and PQ we will see this kind of
response from the API.
```json
{
"duration_in_millis": 438624,
"in": 3011436,
"filtered": 2189,
"out": 2189,
"queue_push_duration_in_millis": 49845
}
```
Fixes: #6512Fixes#6532
This PR add new information in the /_node/stats api and will return the
load average of the machine in the following formats depending of the
platforms that logstash is running on:
**Linux**
```json
{
"cpu" : {
"percent" : 26,
"load_average" : {
"1m" : 2.826171875,
"5m": 1.8261718,
"15m": 1.56566
}
}
}
```
**MacOS and other platform that the OperatingMXBean understand**
```json
{
"cpu" : {
"percent" : 26,
"load_average" : {
"1m" : 2.826171875,
}
}
}
```
Load average is not available on Windows
Fixes: #6214Fixes#6240
This adds two new fields 'id', and 'name' to the base metadata for API requests.
These fields are now returned at all API endpoints by default.
The `id` field is the persisted UUID, the name field is the custom name
the user has passed in (defaulted to the hostname).
I renamed `node_uuid` and `node_name` to just `id` and `name` to be
inline with Elasticsearch.
This also fixed a broken test double in `webserver_spec.rb` that was
using doubles across threads which created hidden errors.
Fixes#6224
sets default to INFO and updates some verbose logging to
more appropriate, less verbose log levels where it makes sense.
Closes#5735.
Closes#5893.
Fixes#5898
This PR introduce a changes to make sure the API endpoint correctly
return 404 in every case. To make it work it uses an exception that get
pickup up in the base class by the `error` handler.
Using the exception template allow the code to be dry and make sure all
other errors can be returned with the same format.
Example or a 404.
```json
{
"error": { "message": "Not Found"},
"path": "/this-should-not-exist"
"status": 404
}
```
The code will also set the content-type and the right error code in the
header.
Fixes: #5874, #5622Fixes#5897
This PR introduce GC stats in the form of `collection_time` and `collection_count`, it takes the configured gc and categorize them
into old and young.
Example output:
```
gc: {
collectors: {
young: {
collection_time_in_millis: 22101,
collection_count: 5452
},
old: {
collection_time_in_millis: 57,
collection_count: 1
}
}
}
```
Fixes: #5730Fixes#5788
This PR will expose the time spend of a batch in the filter + output
Ouput:
http://localhost:9600/_node/stats/pipeline
```
events: {
duration_in_millis: 18364,
in: 309325,
filtered: 309325,
out: 309325
},
```
Fixes: #5731Fixes#5792
This PR does a few modifications for our webserver:
- Add a PortRange setting type.
- allow the --http.port option to accept a port range or a single port, when a range is provided Logstash will incrementally try this list to find a free port for the webserver.
- Logstash will report in the log which port it is using. (INFO LEVEL)
- It refactors a few methods of the webserver.
- It adds test for the binding of the ports.
Notes:
Port range can be defined in the logstash.yml or pass via the command line like this.
`bin/logstash -e 'input { generator {}} output { null {}}' --log.level verbose --http.port 2000-2005`
Fixes#5774
Setting error handling conditions in sinatra dynamically is not possible.
The values such as :show_exceptions get set, but have no effect.
This is esp. important in testing where we want failures to raise exceptions.
Otherwise, debugging is a matter of parsing instructable HTML responses containing
the debug info.
We do not want this behavior by default however, as doing so would mean a single API
bug would take down a production app.
This change adds a new agent setting (not CLI setting) 'http.environment' that is defaulted to 'production'.
If set to 'test' exceptions get raised.
This change also removes sinatra error handlers that previously did nothing, and injects
custom rack middleware to properly log and handle errors.
Fixes#5411
format test properly for the node_spec api
Introduce a more proper organization of the dummy/mock runner to avoid
having wrong state propagaged across the different tests. This was
discovered by the unnecessary need to have the rubydebug codec as it was
due to wrong pipeline startup.
make the node stats test sync with last changes in data formats
improve web api service logic from reviews
remove the requirement of using output-null as only with output {} you get the same effect
Fixes#4782
Test improvements:
* Cleanup non used fixture.
* Improve node_stats tests by adding omre context and format checks.
* Add a new node_specs endpoint test that checks the expected data is
returned for the hot_threads resource.
Fixes#4782
As discussed in this adds:
/
/_node/hot_threads
/_node/stats/
/_node/stats/events
/_node/stats/jvm
/_stats/jvm
introduces also small refactors and cleanup necessary to improve the
webapi code quality.
adds also a way to handle references from the webapi to the internals of
logstash by passing the agent refrences to the collector, this is not
perfect, but for now it solve the communication situation until furder
refactor.
Fixes#4652Fixes#4653
This commit introduce the collection of the main metrics inside the
logstash pipeline and records it to the metric store. This code is also
the initial stone to define an internal metric api.
Collected metrics from the pipeline and system will be exposed by the web
api.
This work was done in collaboration by @ph and @purbon, commits were
squashed to simplify the merging process.
Fixes#4653
