* specs: detangle out-of-band pipeline initialization
Our API tests were initializing their pipelines-to-test in an out-of-band
manner that prevented the agent from having complete knowledge of the
pipelines that were running. By providing a ConfigSource to our Agent's
SourceLoader, we can rely on the normal pipeline reload behaviour to ensure
that the agent fully-manages the pipelines in question.
* api: do not emit pipeline that is not fully-initialized
This commit updates the version of jruby used in Logstash to `9.3.4.0`.
* Updates the references of `jruby` from `9.2.20.1` to `9.3.4.0`
* Updates references/locations of ruby from `2.5.0` to `2.6.0`
* Updates java imports including `org.logstash.util` to be quoted
* Without quoting the name of the import, the following error is observed in tests:
* `java.lang.NoClassDefFoundError: org/logstash/Util (wrong name: org/logstash/util)`
* Maybe an instance of https://github.com/jruby/jruby/issues/4861
* Adds a monkey patch to `require` to resolve compatibility issue between latest `jruby` and `polyglot` gem
* The addition of https://github.com/jruby/jruby/pull/7145 to disallow circular
causes, will throw when `polyglot` is thrown into the mix, and stop logstash from
starting and building - any gems that use an exception to determine whether or not
to load the native gem, will trigger the code added in that commit.
* This commit adds a monkey patch of `require` to rollback the circular cause exception
back to the original cause.
* Removes the use of the deprecated `JavaClass`
* Adds additional `require time` in `generate_build_metadata`
* Rewrites a test helper to avoid potentially calling `~>` on `FalseClass`
Co-authored-by: Joao Duarte <jsvduarte@gmail.com>
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Reimplement the Ruby class PipelinceConfig in Java trying to keep the method signatures to limit the changes in client code, this is a step of other that intend to move all the configuration code in Java language.
Having all that code in Java unlock some reasoning about how to better implement it and probably an improvement in performance during process startup.
Moved also the spec into a JUnit and fixed here and there the failing tests
Closes: #11824
* Fixes#8344 - Ruby Unit Test can hang with no information as to why
* Fixes#7227 - MetricStore::MetricNotFound
* Fixes#8204 - `config.reload.automatic` is set to`FALSE
I am only certain it fixes#8344 and #7227, since I can not reproduce #8204.
Fixes#8376
This commit further improves hashing performance, using SourceWithMetadata
objects where present to determine Vertex IDs and the hash of the source
file. If those objects are not present LIR will revert to graph hashing,
which is slower, but always works. This is still useful for things like
testing, so it makes sense to leave that code in.
In the future it might be nice to extract the hashing code using the
strategy pattern to clean things up.
This commit also improves the Hashable interface, moving hashSource into
its own interface, letting classes that do their own hashing simply
implement uniqueHash instead of requiring them to also implement
hashSource.
A requirement of these changes was also that SourceWithMetadata no
longer accept null or empty arguments. This patch now enforces that
requirement. This also is nicer from an API standpoint, as you now know
that if a SourceWithMetadata object exists, it will actually have all
its fields in use.
Fixes#7408
* add multi_local source for multi pipelines
* introduce pipelines.yml
* introduce PipelineSettings class
* support reloading of pipeline parameters
* fix pipeline api call for _node/pipelines
* inform user pipelines.yml is ignored if -e or -f is enabled
This is a big chang, it:
1. Moves API specs out of their special hierarchy
2. Removes the API spec spec_helper
3. Reactivates that stats command spec (that was accidentally not being
run before due to it not having _spec as a suffix
This was required to fix the preceeding commit, where we added a
before(:each) hook to the spec_helper that wasn't being picked up in
some cases due to the existence of two spec helpers and a $LOAD_PATH
that could change.
Fixes#7132
This unifies the two different config classes that represented mainly
the same data. While this does expose a plain java class into ruby
this works fine because ruby only needs to access and set values, not
work with ruby return types.
Fixes#7003Fixes#7004
This PR introduces majors changes inside Logstash, to help build future
features like supporting multiple pipeline or java execution.
The previous implementation of the agent made the class hard to refactor or add new feature to it, because it needed to
know about too much internal of the pipeline and how the configuration existed.
The changes includes:
- Externalize the loading of the configuration file using a `SourceLoader`
- The source loader can support multiple sources and will aggregate them.
- We keep some metadata about the original file so the LIR can give better feedback.
- The Agent now ask the `SourceLoader` to know which configuration need to be run
- The Agent now uses a converge state strategy to handle: start, reload, stop
- Each actions executed on the pipeline are now extracted into their own classes to help with migration to a new pipeline execution.
- The pipeline now has a start method that handle the tread
- Better out of the box support for multiple pipeline (monitoring)
- Refactor of the spec of the agent
Fixes#6632
This adds two new fields 'id', and 'name' to the base metadata for API requests.
These fields are now returned at all API endpoints by default.
The `id` field is the persisted UUID, the name field is the custom name
the user has passed in (defaulted to the hostname).
I renamed `node_uuid` and `node_name` to just `id` and `name` to be
inline with Elasticsearch.
This also fixed a broken test double in `webserver_spec.rb` that was
using doubles across threads which created hidden errors.
Fixes#6224
When Logstash encounter an exception Puma was getting really noisy about
the file descriptor it used. Its because Puma has an infinite loop into
his reactor and when logstash is dying from the exception the FD get
removed under the puma threads.
Puma is using the constants `STDERR` and `STDOUT` to give information
about the errors, there is no easy way to provide Puma with a custom IO.
This PR hacks the Puma name to redefine both constants and send the
information to the logger using the `debug` level.
Also when we start the server we can pass a custom Puma::Events instance
to record some of the errors happening in the internal threads, this PR
also make sure we send all theses message to the configured logger.
Fixes: #5822Fixes#5869
