I actually am not sure if this matters, but it seems like it won't hurt,
and is a more sane ordering of things. I have a suspicion that in some cases
the agents get stuck and don't shut down because of this ordering, but I can't
prove it
Fixes#9628
* Fixes#8344 - Ruby Unit Test can hang with no information as to why
* Fixes#7227 - MetricStore::MetricNotFound
* Fixes#8204 - `config.reload.automatic` is set to`FALSE
I am only certain it fixes#8344 and #7227, since I can not reproduce #8204.
Fixes#8376
Some of the test were still depending on the multiline filter, this
filter is now shipped anymore with Logstash. I've keep the same logic
but created a simple dummy filter.
Fixes#8284
This commit further improves hashing performance, using SourceWithMetadata
objects where present to determine Vertex IDs and the hash of the source
file. If those objects are not present LIR will revert to graph hashing,
which is slower, but always works. This is still useful for things like
testing, so it makes sense to leave that code in.
In the future it might be nice to extract the hashing code using the
strategy pattern to clean things up.
This commit also improves the Hashable interface, moving hashSource into
its own interface, letting classes that do their own hashing simply
implement uniqueHash instead of requiring them to also implement
hashSource.
A requirement of these changes was also that SourceWithMetadata no
longer accept null or empty arguments. This patch now enforces that
requirement. This also is nicer from an API standpoint, as you now know
that if a SourceWithMetadata object exists, it will actually have all
its fields in use.
Fixes#7408
* add multi_local source for multi pipelines
* introduce pipelines.yml
* introduce PipelineSettings class
* support reloading of pipeline parameters
* fix pipeline api call for _node/pipelines
* inform user pipelines.yml is ignored if -e or -f is enabled
This is a big chang, it:
1. Moves API specs out of their special hierarchy
2. Removes the API spec spec_helper
3. Reactivates that stats command spec (that was accidentally not being
run before due to it not having _spec as a suffix
This was required to fix the preceeding commit, where we added a
before(:each) hook to the spec_helper that wasn't being picked up in
some cases due to the existence of two spec helpers and a $LOAD_PATH
that could change.
Fixes#7132
* Introduce a DeadLetterQueueFactory
DeadLetterQueueFactory is a static class that keeps
a static collection of DeadLetterQueueWriteManagers per
pipeline that has plugins requesting to use it.
* DeadLetterQueue was added as a first-class field in the execution context that input/filter/output plugins can leverage
This unifies the two different config classes that represented mainly
the same data. While this does expose a plain java class into ruby
this works fine because ruby only needs to access and set values, not
work with ruby return types.
Fixes#7003Fixes#7004
This PR introduces majors changes inside Logstash, to help build future
features like supporting multiple pipeline or java execution.
The previous implementation of the agent made the class hard to refactor or add new feature to it, because it needed to
know about too much internal of the pipeline and how the configuration existed.
The changes includes:
- Externalize the loading of the configuration file using a `SourceLoader`
- The source loader can support multiple sources and will aggregate them.
- We keep some metadata about the original file so the LIR can give better feedback.
- The Agent now ask the `SourceLoader` to know which configuration need to be run
- The Agent now uses a converge state strategy to handle: start, reload, stop
- Each actions executed on the pipeline are now extracted into their own classes to help with migration to a new pipeline execution.
- The pipeline now has a start method that handle the tread
- Better out of the box support for multiple pipeline (monitoring)
- Refactor of the spec of the agent
Fixes#6632
This PR changes where the `events.in` are calculated, previously the
values were calculated in the `ReadClient` which was fine before the
addition of the PQ, but this make the stats not accurate when the PQ was
enabled and the producer are a lot faster than the consumer.
These commits change the collection of the metric inside an
instrumented `WriteClient` so both implementation of the client queues will use
the same code.
This also make possible to record `events.out` for every inputs and the
time waiting to push to the queue.
The API is now exposing theses values for each plugins, the events level
and and the pipeline.
Using a pipeline with a sleep filter and PQ we will see this kind of
response from the API.
```json
{
"duration_in_millis": 438624,
"in": 3011436,
"filtered": 2189,
"out": 2189,
"queue_push_duration_in_millis": 49845
}
```
Fixes: #6512Fixes#6532
The assertions was using dummy outputs and kept received events into an
array in memory, but the test actually only needed to match the number
of events it received, this PR add a DroppingDummyOutput that wont
retain the events in memory.
The previous implementation was causing a OOM issue when running the
test on a very fast machine.
Fixes: #6335Fixes#6346
This adds two new fields 'id', and 'name' to the base metadata for API requests.
These fields are now returned at all API endpoints by default.
The `id` field is the persisted UUID, the name field is the custom name
the user has passed in (defaulted to the hostname).
I renamed `node_uuid` and `node_name` to just `id` and `name` to be
inline with Elasticsearch.
This also fixed a broken test double in `webserver_spec.rb` that was
using doubles across threads which created hidden errors.
Fixes#6224
This PR introduce a changes to make sure the API endpoint correctly
return 404 in every case. To make it work it uses an exception that get
pickup up in the base class by the `error` handler.
Using the exception template allow the code to be dry and make sure all
other errors can be returned with the same format.
Example or a 404.
```json
{
"error": { "message": "Not Found"},
"path": "/this-should-not-exist"
"status": 404
}
```
The code will also set the content-type and the right error code in the
header.
Fixes: #5874, #5622Fixes#5897
When Logstash encounter an exception Puma was getting really noisy about
the file descriptor it used. Its because Puma has an infinite loop into
his reactor and when logstash is dying from the exception the FD get
removed under the puma threads.
Puma is using the constants `STDERR` and `STDOUT` to give information
about the errors, there is no easy way to provide Puma with a custom IO.
This PR hacks the Puma name to redefine both constants and send the
information to the logger using the `debug` level.
Also when we start the server we can pass a custom Puma::Events instance
to record some of the errors happening in the internal threads, this PR
also make sure we send all theses message to the configured logger.
Fixes: #5822Fixes#5869
The current test where only checking if the `NullMetric` Interface was
matching the original metric interface, this PR add more test around the
actual methods and fix an issue with `decrement` using too much
arguments in the context of a namespaced metric.
The test were also move into a shared example to be used in both the `NullMetric`
and the `NamedspacedMetric`.
Also the NullMetric class will no use the same validation for keys as the namespaced class and will
reject empty or nil keys.
Fixes#5545
Make sure we pass the metric when we create a pipeline this make sure
the metric is correctly visible in the pipeline and remove the need to
lock the variables.
Make the expectation more robust in the metric reload scenario.
Fixes#5400 for the greater good.
Fixes#5446
When logstash reload a configuration the collector should remove all the
collected metrics from the store since it wont make any sense with a new
configuration. You should have the same behavior as when you restart
logstash.
Fixes#4801
This commit introduce the collection of the main metrics inside the
logstash pipeline and records it to the metric store. This code is also
the initial stone to define an internal metric api.
Collected metrics from the pipeline and system will be exposed by the web
api.
This work was done in collaboration by @ph and @purbon, commits were
squashed to simplify the merging process.
Fixes#4653
