Commit graph

1569 commits

Author SHA1 Message Date
Joao Duarte
d163917044 avoid class variable during pipeline compilation 2016-05-17 08:47:54 +01:00
Tal Levy
4e6c0da4ec undo allow_env flag (#5265)
allow-env flag was introduced because of a backwards incompatible
change, this flag is no longer needed for 5.0 and onwards.

Closes #5263.
2016-05-12 09:17:32 -07:00
Joao Duarte
b2f04ba9b9 better error message for reload incompatible pipelines 2016-05-10 16:41:21 +01:00
Jordan Sissel
f241dd808e Enable JSON logging for all test runs.
Use an internal subscriber to verify that JSON output is valid JSON. The
purpose is to catch any json serialization errors that would occur while
logging.

Also had to update a few logger calls to log values that could be
serialized (Class instances and similar, at this time, fail to serialize
to JSON).

Fixes #4820
2016-05-09 19:53:17 +00:00
Jordan Sissel
af41218650 Allow Logstash to write its logs in JSON format
This is made available by a new `--log-in-json` flag. Default is false.
When false, the old behavior [1] is used. When true, JSON logs are
emitted.

[1] The old behavior is realy two things. First, using Object#inspect to
serialize. Second, to color the output if the IO is a tty.

Fixes #1569

Fixes #4820
2016-05-09 19:53:17 +00:00
Colin Surprenant
ee2c8af8c8 refactor Ruby Event getter and setter
refactor wip gemfiles

refactor Java Event getter and setter

bump plugin-api to 2.0

use plugin-api 2.0

switch to core-event-java

include logstash-core-event-java.jar jar file so that gem dependency using the source tree work

updated core plugins to core-api 2.0

added grok for refactor branch

fix rebased specs

remove temp plugins github paths

remove commented out tmp alias_method
2016-05-06 13:33:31 -04:00
Pier-Hugues Pellerin
5c667620e2 Remove constant redefinition
The Rack config was redefining a constant in the configuration, since
this value isn't used anywhere else there was no need to use a constant.

The conflicting ROOT contants was defined in `lib/logstash/patches/profile_require_calls.rb`

Fixes: #5163

Fixes #5208
2016-05-06 16:54:06 +00:00
Suyog Rao
4a3cbb4868 Remove unused java version check
Fixes #5243
2016-05-04 20:31:43 +00:00
Suyog Rao
bcf3faa6cd Make Java 8 minimum version for LS 5.0
Fixes #5242

Fixes #5243
2016-05-04 20:31:43 +00:00
Joao Duarte
dd02422912 converted static I18n test to rspec
Fixes #5158
2016-04-27 14:49:41 +00:00
Joao Duarte
4942f38b2d make the plugin incompatibility tests less deterministic
Fixes #5175
2016-04-26 09:46:31 +00:00
Pier-Hugues Pellerin
708a269a62 Fix non derterministic test when accessing the metric store
This commit introduce a mutex around the structured hash for the metric,
this hash is not updated frequently and its used mostly for the api to
be able to do search on the collected metric. Adding a mutex make sure
the changes are visible accross thread.

Fixes #5152

Fixes #5178
2016-04-22 17:22:05 +00:00
Colin Surprenant
5de3ce40d1 upgrade to JRuby 1.7.25 and required jruby-openssl 0.9.16 to match JRuby 2016-04-20 14:04:53 -04:00
Pier-Hugues Pellerin
fb62bc991c Clear the collector when logstash reload configuration
When logstash reload a configuration the collector should remove all the
collected metrics from the store since it wont make any sense with a new
configuration. You should have the same behavior as when you restart
logstash.

Fixes #4801
2016-04-18 20:03:28 +00:00
Pere Urbon-Bayes
263232a5e4 not use elasticsearch input in test as this is present in branches that already had one release
Fixes #5107
2016-04-14 08:12:53 +00:00
Tony Laidig
78e25f9e02 change of I18n varname, possible fix for #5108
Fixes #5111
2016-04-14 00:51:32 +00:00
Pere Urbon-Bayes
8af93d0f3a Add generic code used to load any kind of plugins across logstash
Introduce the idea of a registry plugin placeholder where all necessary
interface to plugins is mantained, also simplified the internal registry
calls to be more generic.

Add a way to handle registrations for plugins explicitly

introduced the idea of self.plugin_type method to fetch plugin type from base clase, also removed the former plugins:mixin used to annotate defined plugins

make the config_name method also handle the registration to the plugin registry, that way old plugins get registration out of the box and we can simply incoming plugin registry without automatic loading

simplify the plugin registry by removing former need to load classes, now they all get registered automatically when using the config_name method

cleanup unnecessary former changes

updated typo in comments for the plugins registry and also removed internal attr_reader for the same class

renamed plugin annotate to declare_plugin to have a more meaningful name

change Registry::Plugin.gem_name -> cannonic_gem_name to reflect the idea of having probably also other non cannonic gem names

Fixes #4535
2016-04-12 14:44:52 +00:00
Tal Levy
a7c602b8cc fix path to --allow-env I18n description
Fixes #5007
2016-04-04 21:43:57 +00:00
Andrew Cholakian
f4a07f90b1 Prevent config leak on output delegator close
On shutdown logstash may leak config information for outputs including
plaintext passwords

Fixes #5001
2016-04-04 21:19:15 +00:00
Tal Levy
ae53d5119f add experimental flag for environment variables in config
Fixes #4963
2016-04-04 18:45:13 +00:00
Andrew Cholakian
edd03d98bd Hide sensitive AST/config data unless --debug-config argument passed
We now hide this because displaying it is dangerous. Generated code
and other AST data may contain plaintext copies of 'password' type fields.
Users can force this to appear with the --debug-config flag.

Fixes https://github.com/elastic/logstash/issues/4964

Fixes #4965
2016-04-04 15:47:02 +00:00
Joao Duarte
120ca6d7fe test identity of generated conditional functions
Fixes #4970
2016-04-01 17:22:03 +00:00
Joao Duarte
46af6ce109 ensure cond_func methods are singletons
Fixes #4970
2016-04-01 17:22:02 +00:00
Joao Duarte
bb30cc773b move Pipeline#validate_config to Runner
Fixes #4946
2016-03-31 21:01:33 +00:00
Joao Duarte
f52820a58f fix --config-test and add a couple of tests
fixes #4933

Fixes #4946
2016-03-31 21:01:33 +00:00
Joao Duarte
6369a43c9d avoid reusing generic validation method during params securization
Fixes #4952
2016-03-31 19:15:31 +00:00
Joao Duarte
b0b7a8c5e9 ensure Plugin#original_params is obfuscated
Fixes #4952
2016-03-31 19:15:31 +00:00
Pere Urbon-Bayes
4dbbbde1fb bump jruby-monitoring version to >= 0.3.1 to include last fixes
Fixes #4911
2016-03-31 07:20:02 +00:00
Suyog Rao
e971dacfce Bump current master to 5.0.0.dev
Fixes #4906
2016-03-29 16:26:31 +00:00
Joao Duarte
563236649e fix race condition in agent spec
Fixes #4912
2016-03-29 14:41:42 +00:00
Andrew Cholakian
76f0dcc320 Revert to use only a single worker by default on outputs
The move to auto-scale output workers was great in theory, but a lot of outputs
just weren't built to support it well, they often used too many resources or had
logical errors.

Fixes #4904
2016-03-28 18:04:39 +00:00
Andrew Cholakian
b0f4d4e318 Fix threadsafe workers in output delegators to not instantiate extras
Fixes #4763
2016-03-24 18:59:18 +00:00
Joao Duarte
0dac96aa11 prevent reloading of configurations containing non reloadable plugins
Fixes #4884
2016-03-24 18:41:42 +00:00
Pier-Hugues Pellerin
15aed859de Update the gem.authors and gem.email with the Elastic information
Fix #4880
2016-03-23 17:36:10 -07:00
Pere Urbon-Bayes
bfd35a6e5c remove hot_threads ignore_idle_threads test
Fixes #4853
2016-03-21 15:37:58 +00:00
Pier-Hugues Pellerin
3b8d5a1f0b Make sure we clear the store before running the test
Fixes: #4824

Fixes #4839
2016-03-20 02:27:36 +00:00
Pere Urbon-Bayes
c233059d74 change JIT threads for pool threads in the idle threads check, due to the JIT threads not showing up randominly in some tests
Fixes #4833
2016-03-18 20:54:05 +00:00
Joao Duarte
23f0c5bcce expose reload_interval in Runner
Fixes #4831
2016-03-18 18:15:31 +00:00
Pere Urbon-Bayes
a860e5bc97 initial version for the _plugins api root resource
Fixes #4794
2016-03-17 11:00:04 +00:00
Suyog Rao
b9369abc0e Revert commit 4f7299a
Need to revert bump to 5.0.0.dev because individual plugins
will not install on top of this version.

In plugins' gemspec, the constraint is
`s.add_runtime_dependency "logstash-core", ">= 2.0.0", "< 3.0.0"`

This needs to be mass updated, and then bumped to 5.0.0

Fixes #4828
2016-03-17 02:12:56 +00:00
Joao Duarte
4f7299a555 bump to 5.0.0.dev
Fixes #4814
2016-03-17 00:02:03 +00:00
Pere Urbon-Bayes
42710a4ffd add expectations for json and plain text responses for the hot threads api tests
format test properly for the node_spec api

Introduce a more proper organization of the dummy/mock runner to avoid
having wrong state propagaged across the different tests. This was
discovered by the unnecessary need to have the rubydebug codec as it was
due to wrong pipeline startup.

make the node stats test sync with last changes in data formats

improve web api service logic from reviews

remove the requirement of using output-null as only with output {} you get the same effect

Fixes #4782
2016-03-14 14:07:25 +00:00
Pere Urbon-Bayes
7f112bccff fix issue with unhandled requests prior to shutdown by puma
Fixes #4782
2016-03-14 14:07:25 +00:00
Pere Urbon-Bayes
79e2d0040f Make sure all web api test are actually going through the hole pipe and getting data out of a real logstash execution.
Test improvements:

* Cleanup non used fixture.
* Improve node_stats tests by adding omre context and format checks.
* Add a new node_specs endpoint test that checks the expected data is
  returned for the hot_threads resource.

Fixes #4782
2016-03-14 14:07:25 +00:00
Pier-Hugues Pellerin
5ce51933b6 Missing a method for the delegator test
Fixes #4802
2016-03-11 21:14:25 +00:00
Pier-Hugues Pellerin
e3231a0c31 Add test for the filter delegator
Fixes #4802
2016-03-11 21:14:25 +00:00
Pier-Hugues Pellerin
04b2b72cbd Adding more test to test the flow of the metrics inside the pipeline_spec
This an integration test where the pipeline should emit specific
metrics. Also fix an issue of the output delegator not correctly
recording the metrics.

Normalize the naming of the metrics collected at the plugin level
following #4492

Fixes #4802
2016-03-11 21:14:25 +00:00
Pere Urbon-Bayes
ee690ac3ee make sure _stats and _node/stats data format is the same
Fixes #4780
2016-03-11 19:31:14 +00:00
fbaligand
587db82c15 add complementary tests to 'environment variable evaluation'
Fixes #3944
2016-03-07 10:58:38 -08:00
Jordan Sissel
f644df6732 Use structured logging. 2016-03-07 10:58:17 -08:00