This change moves the definition of the counters and gauges from Ruby to Java, providing a Ruby bridge that allows support for Ruby duck typing to Java strict typing.
This change also moves the serialzation from JRuby (Jackson) to Java (Jackson).
This change also removes the decrement from counter api and requires counters start at zero. This helps to allowing for reliable aggregrations over time.
This change removes the base Ruby class for metrics, and replaces with base Java class.
This change removes the to_hash method (no known usages).
This change also has minor changes to the Ruby type() and inspect() methods.
Fixes#7509Fixes#7595
* Add support for using kibana dashboard import api
- Add support for versioned folders below 'module'/configuration/kibana
- Rename cef to module_test_files/tester and simplify it.
- Remove skips on tests using above modules test files.
- In pipeline.rb, moved method default_logging_keys to the base class,
it was being called from BasePipeline during a config-test.
- In modules_common, added a backtrace to LogStash::ConfigLoadingError
for better debugging
- Improve layout when user_lines et al were empty in logstash_config
* Change specificity of maj/min/patch folder search
* Use logger to surface manticore error
* Add suppor for Kibana auth
* improve error message
* support es and kibana authentication in the logstash modules
* Add more options for the modules templates
This commits add the following new features:
Allow users to use the internal settings classes to have better handling
over the primitives values that a user can use in the modules templates,
by default we are using a `NullableString` which is the more flexible
type.
This change open the way to have more strict validation on the module
templates and will allow us to preprocess configuration to retrieve the
list of the settings and their settings types.
We also add a new type of settings called `SplittableStringArray`, this
class is a subclass or `ArrayCoercible` but target module CLI
configuration. It will take the following form `-M
"arcsight.var.input.hosts=127.0.0.1:3333,192.168.1.23:9999"` and
transform it into a ruby array of string.
We can also add alias of settings values in the configuration by using the
`alias_settings_keys!()`, this is usefull when the language domain is
different from the logstash domain.
* adjust test
New boolean setting `config.support_escapes` which defaults to false
(the historical behavior). When set to true, the following escapes are
handled:
* backslash doublequote -> doublequote
* backslash quote -> quote
* backslash n -> newline (ascii 10)
* backslash r -> carriage return (ascii 13)
* backslash backslash -> backslash
* backslash t -> tab (ascii 9)
This will solve #1645.
* Update runner, source/local, source/modules, source/multi_local
improve startup config problem detection across all sources.
* fix multi_local spec to use not `empty` values.
This corrects a mistake where the digest of the string source should be
used when present, and the hash of the graph should be used when not.
Fixes#7467
Theses id wont be persisted between restart of logstash or between
creation of pipeline. When we reload pipelines the ephemeral_id will be
generated again.
Fixes#7423
This commit further improves hashing performance, using SourceWithMetadata
objects where present to determine Vertex IDs and the hash of the source
file. If those objects are not present LIR will revert to graph hashing,
which is slower, but always works. This is still useful for things like
testing, so it makes sense to leave that code in.
In the future it might be nice to extract the hashing code using the
strategy pattern to clean things up.
This commit also improves the Hashable interface, moving hashSource into
its own interface, letting classes that do their own hashing simply
implement uniqueHash instead of requiring them to also implement
hashSource.
A requirement of these changes was also that SourceWithMetadata no
longer accept null or empty arguments. This patch now enforces that
requirement. This also is nicer from an API standpoint, as you now know
that if a SourceWithMetadata object exists, it will actually have all
its fields in use.
Fixes#7408
Work done by @guyboertje and @ph
Since JRuby 1.7.25 is now EOL we are migrating Logstash to use JRuby 9k and JDK8 only,
Not much needed updating to make this work, its was mostly a drop in replacement from the previous version.
The major point was the change in the implementation of Time in JRuby, JRuby now use `java.time`
instead of joda time, this allow JRuby to have nanoseconds precision on time object.
Add an initial set of metrics for the dead letter queue.
Metrics are supplied under the pipeline in the following format:
"dead_letter_queue": {
"queue_size_in_bytes": ...,
}
Metrics are populated via a PeriodicPoller
Also fixed up calculation of currentQueueSize to take account
of version headers, which was previously being skipped.
Additionally, whether the dlq is enabled, and if so, the path
of the dlq is supplied under the pipelines API endpoint
Resolves#7287Fixes#7338
* My changes (#7218)
* First upstream PR commit (#7172)
No tests yet. Just for code review for now
* move all inner classes to their own folder + client and importer
* Fixes and tests (#7228)
Add tests for the `LogStash::Modules:CLIParser` class in `cli_parser.rb`
Fix a typo in `cli_parser.rb` (`uparsed` vs `unparsed`)
Fix a bad variable name found by testing in `cli_parser.rb` and update the error message accordingly in `en.yml`
* Remove fb_modules (#7280)
* fixes to import index-pattern & var updates & savedsearch capability (#7283)
* fixes to import index-pattern & var updates & savedsearch capability
fixes to import index-pattern & var updates
add savedsearch capability
* minimise merge conflicts with PR End-to-End test with filebeat apache2
* End-to-End test with filebeat apache2 (#7279)
This is a first run, but data flows from filebeat through Elasticsearch.
Template uploads from `$LS_HOME/modules/MODULENAME/configuration/elasticsearch/MODULENAME.json`
Specifying `--modules filebeat` from the command-line, with `-M "filebeat.var.elasticsearch.output.host=localhost:9200"`
Some of the saved searches don't get uploaded. @guyboertje is on this already.
The logstash configuration needs tweaking to allow receiving both access logs _and_ error logs. The dashboards and visualizations all seem to expect the presence of both.
Set default to `localhost` in `elasticsearch_client.rb`
Changed command-line variable parsing to allow for a variable with only `modulename.key.subkey=value`, and updated the error message accordingly.
First draft of the filebeat module, as extracted from filebeat 5.4.0
* Add documentation for Modules
This is specific to the Master branch. Multiple modules will not be supported in 5.5.
* Add READMEs and prune post-code comments
* Add comment regarding the variable name `modul`
Also, fix the default username for the Elasticsearch output in Logstash. The default x-pack credentials are `elastic:changeme` rather than `elasticsearch:changeme`
* add cef module files (#7292)
* fixes from reviews of PR #7284
Ruby 2.0's stdlib no longer allows relative IPv6 URLs like: //[::1]
This is a key feature of the Elasticsearch output. See more in
https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/604
The only way to fix this is to use a better URI class which means
introducing a slight incompatibility here. I don't believe that anyone
is using the URI class other than the ES output, and I don't that there
is anyone using any portions of the API here that are no longer
compatible.
Normally we would only make such a change in a major version, but our
hands are tied here. If we are to go with JRuby 9k, which is critical
for support we must use a different, better, library.
java.net.URI is stable and works well, so its probably superior in the
long run anyway.
Fixes#7236
* add multi_local source for multi pipelines
* introduce pipelines.yml
* introduce PipelineSettings class
* support reloading of pipeline parameters
* fix pipeline api call for _node/pipelines
* inform user pipelines.yml is ignored if -e or -f is enabled
Fixes#7230
Further improves on the threadsafety improvements in #7206
This commit fixes various tests to access the Agent#pipelines property
in a fully threadsafe way.
Fixes#7233
Universal plugins may access the #pipelines property of the agent.
This PR makes accessing it in a threadsafe way possible with a
`with_pipelines` helper function in Agent.
This uses a reentrant lock since its anticipated that users may want to
nest calls here. A ruby Mutex is not reentrant, so that won't work.
Fixes#7206
We remove a flaky test "LogStash::Agent#execute when auto_reload is true if state is clean should periodically reload_state"
in favor of the refactored test in `converge_spec`, since it doesn't use
rspec mock but instead use a threadsafe spies to make sure we do
multiple call on the source loader.
Fixes: #7196Fixes#7200
When running this test with a newer version of rspec we got the warning
that some values where leaking between example:
```
1) LogStash::ShutdownWatcher when pipeline is not stalled .unsafe_shutdown = false shouldn't force the shutdown
Failure/Error: m.call(*args)
The use of doubles or partial doubles from rspec-mocks outside of the per-test lifecycle is not supported.
# ./logstash-core/lib/logstash/shutdown_watcher.rb:67:in `pipeline_report_snapshot'
# ./logstash-core/spec/logstash/shutdown_watcher_spec.rb:22:in `block in /Users/ph/es/second/logstash/logstash-core/spec/logstash/shutdown_watcher_spec.rb'
# ./logstash-core/lib/logstash/shutdown_watcher.rb:44:in `block in start'
# ./vendor/bundle/jruby/2.3.0/gems/stud-0.0.22/lib/stud/interval.rb:20:in `interval'
# ./logstash-core/lib/logstash/shutdown_watcher.rb:42:in `start'
# ./logstash-core/spec/logstash/shutdown_watcher_spec.rb:93:in `block in (root)'
```
We have added a few helpers method in the class that allowed us to
simplify the expectation and removed any usage of `Thread#kill`
Fixes#7152
This is a big chang, it:
1. Moves API specs out of their special hierarchy
2. Removes the API spec spec_helper
3. Reactivates that stats command spec (that was accidentally not being
run before due to it not having _spec as a suffix
This was required to fix the preceeding commit, where we added a
before(:each) hook to the spec_helper that wasn't being picked up in
some cases due to the existence of two spec helpers and a $LOAD_PATH
that could change.
Fixes#7132
Instead of depending on the now deprecated multiline filter we use a
dummy filter that just emits events. This simplifies the test and
dramatically reduces timing issues.
I also increased the max-wait for the timer just in case
Fixes#7024Fixes#7131
This PR helps enable https://github.com/elastic/logstash/issues/7076
This also fixes a bug where when concatenating pipelines for PipelineIR
the to_s versions of the SourceWithMetadata objects were conjoined
instead of just the `text`.
Fixes#7079
The failure:
Failures:
1) LogStash::Pipeline defaulting the pipeline workers based on thread safety when there are threadsafe filters only starts multiple filter threads
Failure/Error: expect(pipeline.worker_threads.size).to eq(worker_thread_count)
expected: 5
got: 8
Related issues: #6855, #6245, #6355Fixes#7071
