* My changes (#7218)
* First upstream PR commit (#7172)
No tests yet. Just for code review for now
* move all inner classes to their own folder + client and importer
* Fixes and tests (#7228)
Add tests for the `LogStash::Modules:CLIParser` class in `cli_parser.rb`
Fix a typo in `cli_parser.rb` (`uparsed` vs `unparsed`)
Fix a bad variable name found by testing in `cli_parser.rb` and update the error message accordingly in `en.yml`
* Remove fb_modules (#7280)
* fixes to import index-pattern & var updates & savedsearch capability (#7283)
* fixes to import index-pattern & var updates & savedsearch capability
fixes to import index-pattern & var updates
add savedsearch capability
* minimise merge conflicts with PR End-to-End test with filebeat apache2
* End-to-End test with filebeat apache2 (#7279)
This is a first run, but data flows from filebeat through Elasticsearch.
Template uploads from `$LS_HOME/modules/MODULENAME/configuration/elasticsearch/MODULENAME.json`
Specifying `--modules filebeat` from the command-line, with `-M "filebeat.var.elasticsearch.output.host=localhost:9200"`
Some of the saved searches don't get uploaded. @guyboertje is on this already.
The logstash configuration needs tweaking to allow receiving both access logs _and_ error logs. The dashboards and visualizations all seem to expect the presence of both.
Set default to `localhost` in `elasticsearch_client.rb`
Changed command-line variable parsing to allow for a variable with only `modulename.key.subkey=value`, and updated the error message accordingly.
First draft of the filebeat module, as extracted from filebeat 5.4.0
* Add documentation for Modules
This is specific to the Master branch. Multiple modules will not be supported in 5.5.
* Add READMEs and prune post-code comments
* Add comment regarding the variable name `modul`
Also, fix the default username for the Elasticsearch output in Logstash. The default x-pack credentials are `elastic:changeme` rather than `elasticsearch:changeme`
* add cef module files (#7292)
* fixes from reviews of PR #7284
Ruby 2.0's stdlib no longer allows relative IPv6 URLs like: //[::1]
This is a key feature of the Elasticsearch output. See more in
https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/604
The only way to fix this is to use a better URI class which means
introducing a slight incompatibility here. I don't believe that anyone
is using the URI class other than the ES output, and I don't that there
is anyone using any portions of the API here that are no longer
compatible.
Normally we would only make such a change in a major version, but our
hands are tied here. If we are to go with JRuby 9k, which is critical
for support we must use a different, better, library.
java.net.URI is stable and works well, so its probably superior in the
long run anyway.
Fixes#7236
* add multi_local source for multi pipelines
* introduce pipelines.yml
* introduce PipelineSettings class
* support reloading of pipeline parameters
* fix pipeline api call for _node/pipelines
* inform user pipelines.yml is ignored if -e or -f is enabled
Fixes#7230
Further improves on the threadsafety improvements in #7206
This commit fixes various tests to access the Agent#pipelines property
in a fully threadsafe way.
Fixes#7233
Universal plugins may access the #pipelines property of the agent.
This PR makes accessing it in a threadsafe way possible with a
`with_pipelines` helper function in Agent.
This uses a reentrant lock since its anticipated that users may want to
nest calls here. A ruby Mutex is not reentrant, so that won't work.
Fixes#7206
Java code is mostly the same as that from the JRMonitor gem with some general tidying and DRYing up
Report classes are now straight Java
Added junit tests for the Java monitor classes
Ruby code basically stays the same, other than calling out to pure Java report classes
Fixes#7167
When running this test with a newer version of rspec we got the warning
that some values where leaking between example:
```
1) LogStash::ShutdownWatcher when pipeline is not stalled .unsafe_shutdown = false shouldn't force the shutdown
Failure/Error: m.call(*args)
The use of doubles or partial doubles from rspec-mocks outside of the per-test lifecycle is not supported.
# ./logstash-core/lib/logstash/shutdown_watcher.rb:67:in `pipeline_report_snapshot'
# ./logstash-core/spec/logstash/shutdown_watcher_spec.rb:22:in `block in /Users/ph/es/second/logstash/logstash-core/spec/logstash/shutdown_watcher_spec.rb'
# ./logstash-core/lib/logstash/shutdown_watcher.rb:44:in `block in start'
# ./vendor/bundle/jruby/2.3.0/gems/stud-0.0.22/lib/stud/interval.rb:20:in `interval'
# ./logstash-core/lib/logstash/shutdown_watcher.rb:42:in `start'
# ./logstash-core/spec/logstash/shutdown_watcher_spec.rb:93:in `block in (root)'
```
We have added a few helpers method in the class that allowed us to
simplify the expectation and removed any usage of `Thread#kill`
Fixes#7152
The central problem with recording a mean stat is that on a non-realtime
system you can't guarantee that samples were evenly spaced out. A series
of timestamped gauges does a better job. This is what we can do in our
x-pack monitoring extension, but not something we should do in core. The
central difference is that x-pack has Elasticsearch, which can act as a
time series database. I'd prefer not to build a tsdb into logstash. That
would obviously involve a lot of complexity.
Fixes#7094.
Fixes#7105
This PR helps enable https://github.com/elastic/logstash/issues/7076
This also fixes a bug where when concatenating pipelines for PipelineIR
the to_s versions of the SourceWithMetadata objects were conjoined
instead of just the `text`.
Fixes#7079
This is a temporary change until we have LIR execution. This allows the
LIR Pipeline object to be the only object passed into the ruby Pipeline.
The ruby Pipeline needs the source to parse the file into the legacy AST
format.
Fixes#7055
* Introduce a DeadLetterQueueFactory
DeadLetterQueueFactory is a static class that keeps
a static collection of DeadLetterQueueWriteManagers per
pipeline that has plugins requesting to use it.
* DeadLetterQueue was added as a first-class field in the execution context that input/filter/output plugins can leverage
this means the configuration read from path.config (-f) is no longer auto completed with stdin/stdout if the input/output sections are missing. This behaviour will only occur with config.string (-e).
This cleans up the code from a design patterns standpoint and makes testing plugins easier since you can just create/destroy agents at will.
Without this change the SOURCE_LOADER singleton's state will become dirty as agents are created/destroyed and be problematic.
Fixes#7048
When the fetch is call we are aggregating all the pipeline_config from
the different sources, if we encounter duplicates ids we will return a
failure, making the pipeline skip that fetch.
Fixes#6866
This expose some of the internal of the registry to the outside world to
allow other part of the system to retrieves plugins.
This change was motivated by #6851 to retrieve the installed list of
modules.
Fixes#7011
fix Issue #6352
On Windows, Logstash can't find log4j2.properties file with above message at startup.
```
Could not find log4j2 configuration at path /LS_HOME/config/log4j2.properties.
```
Fixes#6903
This unifies the two different config classes that represented mainly
the same data. While this does expose a plain java class into ruby
this works fine because ruby only needs to access and set values, not
work with ruby return types.
Fixes#7003Fixes#7004
This commit is to provide more information when an action fails and will
allow us hopefully to debug a random failure in the test suite for the
environment variable.
ref: #6978Fixes#6982
PipelineAction now have an `execution_priority`, we use this method to change the priority of a create action when we are creating a system pipeline
Fixes#6885
With the creation of the x-pack we have added our first internal
pipeline, but if you were running the monitoring pipeline with a
*finite* pipeline (LIKE generator count => X) when the finite has
completed processing all the events logstash would refuse to stop.
This PR fixes the problem by adding a new pipeline settings called
`system` in the shutdown loop we will check if all the user defined
pipeline are completed if its the case we will shutdown any internal
pipeline and logtash will stop gracefully.
Fixes#6885
This PR introduces majors changes inside Logstash, to help build future
features like supporting multiple pipeline or java execution.
The previous implementation of the agent made the class hard to refactor or add new feature to it, because it needed to
know about too much internal of the pipeline and how the configuration existed.
The changes includes:
- Externalize the loading of the configuration file using a `SourceLoader`
- The source loader can support multiple sources and will aggregate them.
- We keep some metadata about the original file so the LIR can give better feedback.
- The Agent now ask the `SourceLoader` to know which configuration need to be run
- The Agent now uses a converge state strategy to handle: start, reload, stop
- Each actions executed on the pipeline are now extracted into their own classes to help with migration to a new pipeline execution.
- The pipeline now has a start method that handle the tread
- Better out of the box support for multiple pipeline (monitoring)
- Refactor of the spec of the agent
Fixes#6632
Because we sync listeners with emitters when adding or creating hook
this could lead to duplicates of listeners, this PR fixes the problem by using a set
instead of a list. Making sure we can only have one instance of a specific
listener at any time.
Fixes#6916
This PR add the initial building block to pass some `ExecutionContext`
from the pipeline to the plugin, currently we only pass the `pipeline_id`.
We use the accessor `execution_context=` to set the context, in a future
refactor we will pass the object to the constructor.
Fixes#6890
Some codecs are context-specific and not threadsafe. If, for instance,
you want to use `generator { threads => 3 }` you will run into buggy
behavior with the line and multiline codecs which are not threadsafe.
This patch is a quick workaround for this behavior. This does not fix
this issue for inputs that do their own multithreading. Those inputs
should handle codec cloning / lifecycle internally according to their
specific requirements.
Fixes#6865
