Commit graph

440 commits

Author SHA1 Message Date
Pier-Hugues Pellerin
5c667620e2 Remove constant redefinition
The Rack config was redefining a constant in the configuration, since
this value isn't used anywhere else there was no need to use a constant.

The conflicting ROOT contants was defined in `lib/logstash/patches/profile_require_calls.rb`

Fixes: #5163

Fixes #5208
2016-05-06 16:54:06 +00:00
Suyog Rao
4a3cbb4868 Remove unused java version check
Fixes #5243
2016-05-04 20:31:43 +00:00
Suyog Rao
bcf3faa6cd Make Java 8 minimum version for LS 5.0
Fixes #5242

Fixes #5243
2016-05-04 20:31:43 +00:00
Pier-Hugues Pellerin
708a269a62 Fix non derterministic test when accessing the metric store
This commit introduce a mutex around the structured hash for the metric,
this hash is not updated frequently and its used mostly for the api to
be able to do search on the collected metric. Adding a mutex make sure
the changes are visible accross thread.

Fixes #5152

Fixes #5178
2016-04-22 17:22:05 +00:00
Pier-Hugues Pellerin
fb62bc991c Clear the collector when logstash reload configuration
When logstash reload a configuration the collector should remove all the
collected metrics from the store since it wont make any sense with a new
configuration. You should have the same behavior as when you restart
logstash.

Fixes #4801
2016-04-18 20:03:28 +00:00
Tony Laidig
78e25f9e02 change of I18n varname, possible fix for #5108
Fixes #5111
2016-04-14 00:51:32 +00:00
Pere Urbon-Bayes
8af93d0f3a Add generic code used to load any kind of plugins across logstash
Introduce the idea of a registry plugin placeholder where all necessary
interface to plugins is mantained, also simplified the internal registry
calls to be more generic.

Add a way to handle registrations for plugins explicitly

introduced the idea of self.plugin_type method to fetch plugin type from base clase, also removed the former plugins:mixin used to annotate defined plugins

make the config_name method also handle the registration to the plugin registry, that way old plugins get registration out of the box and we can simply incoming plugin registry without automatic loading

simplify the plugin registry by removing former need to load classes, now they all get registered automatically when using the config_name method

cleanup unnecessary former changes

updated typo in comments for the plugins registry and also removed internal attr_reader for the same class

renamed plugin annotate to declare_plugin to have a more meaningful name

change Registry::Plugin.gem_name -> cannonic_gem_name to reflect the idea of having probably also other non cannonic gem names

Fixes #4535
2016-04-12 14:44:52 +00:00
Tal Levy
a7c602b8cc fix path to --allow-env I18n description
Fixes #5007
2016-04-04 21:43:57 +00:00
Andrew Cholakian
f4a07f90b1 Prevent config leak on output delegator close
On shutdown logstash may leak config information for outputs including
plaintext passwords

Fixes #5001
2016-04-04 21:19:15 +00:00
Tal Levy
ae53d5119f add experimental flag for environment variables in config
Fixes #4963
2016-04-04 18:45:13 +00:00
Andrew Cholakian
edd03d98bd Hide sensitive AST/config data unless --debug-config argument passed
We now hide this because displaying it is dangerous. Generated code
and other AST data may contain plaintext copies of 'password' type fields.
Users can force this to appear with the --debug-config flag.

Fixes https://github.com/elastic/logstash/issues/4964

Fixes #4965
2016-04-04 15:47:02 +00:00
Joao Duarte
46af6ce109 ensure cond_func methods are singletons
Fixes #4970
2016-04-01 17:22:02 +00:00
Joao Duarte
bb30cc773b move Pipeline#validate_config to Runner
Fixes #4946
2016-03-31 21:01:33 +00:00
Joao Duarte
f52820a58f fix --config-test and add a couple of tests
fixes #4933

Fixes #4946
2016-03-31 21:01:33 +00:00
Joao Duarte
6369a43c9d avoid reusing generic validation method during params securization
Fixes #4952
2016-03-31 19:15:31 +00:00
Joao Duarte
b0b7a8c5e9 ensure Plugin#original_params is obfuscated
Fixes #4952
2016-03-31 19:15:31 +00:00
Suyog Rao
e971dacfce Bump current master to 5.0.0.dev
Fixes #4906
2016-03-29 16:26:31 +00:00
Andrew Cholakian
76f0dcc320 Revert to use only a single worker by default on outputs
The move to auto-scale output workers was great in theory, but a lot of outputs
just weren't built to support it well, they often used too many resources or had
logical errors.

Fixes #4904
2016-03-28 18:04:39 +00:00
Andrew Cholakian
b0f4d4e318 Fix threadsafe workers in output delegators to not instantiate extras
Fixes #4763
2016-03-24 18:59:18 +00:00
Joao Duarte
0dac96aa11 prevent reloading of configurations containing non reloadable plugins
Fixes #4884
2016-03-24 18:41:42 +00:00
Joao Duarte
23f0c5bcce expose reload_interval in Runner
Fixes #4831
2016-03-18 18:15:31 +00:00
Pere Urbon-Bayes
a860e5bc97 initial version for the _plugins api root resource
Fixes #4794
2016-03-17 11:00:04 +00:00
Suyog Rao
b9369abc0e Revert commit 4f7299a
Need to revert bump to 5.0.0.dev because individual plugins
will not install on top of this version.

In plugins' gemspec, the constraint is
`s.add_runtime_dependency "logstash-core", ">= 2.0.0", "< 3.0.0"`

This needs to be mass updated, and then bumped to 5.0.0

Fixes #4828
2016-03-17 02:12:56 +00:00
Joao Duarte
4f7299a555 bump to 5.0.0.dev
Fixes #4814
2016-03-17 00:02:03 +00:00
Pere Urbon-Bayes
42710a4ffd add expectations for json and plain text responses for the hot threads api tests
format test properly for the node_spec api

Introduce a more proper organization of the dummy/mock runner to avoid
having wrong state propagaged across the different tests. This was
discovered by the unnecessary need to have the rubydebug codec as it was
due to wrong pipeline startup.

make the node stats test sync with last changes in data formats

improve web api service logic from reviews

remove the requirement of using output-null as only with output {} you get the same effect

Fixes #4782
2016-03-14 14:07:25 +00:00
Pere Urbon-Bayes
7f112bccff fix issue with unhandled requests prior to shutdown by puma
Fixes #4782
2016-03-14 14:07:25 +00:00
Pere Urbon-Bayes
79e2d0040f Make sure all web api test are actually going through the hole pipe and getting data out of a real logstash execution.
Test improvements:

* Cleanup non used fixture.
* Improve node_stats tests by adding omre context and format checks.
* Add a new node_specs endpoint test that checks the expected data is
  returned for the hot_threads resource.

Fixes #4782
2016-03-14 14:07:25 +00:00
Pier-Hugues Pellerin
5ce51933b6 Missing a method for the delegator test
Fixes #4802
2016-03-11 21:14:25 +00:00
Pier-Hugues Pellerin
e3231a0c31 Add test for the filter delegator
Fixes #4802
2016-03-11 21:14:25 +00:00
Pier-Hugues Pellerin
04b2b72cbd Adding more test to test the flow of the metrics inside the pipeline_spec
This an integration test where the pipeline should emit specific
metrics. Also fix an issue of the output delegator not correctly
recording the metrics.

Normalize the naming of the metrics collected at the plugin level
following #4492

Fixes #4802
2016-03-11 21:14:25 +00:00
Pere Urbon-Bayes
ee690ac3ee make sure _stats and _node/stats data format is the same
Fixes #4780
2016-03-11 19:31:14 +00:00
Jordan Sissel
f644df6732 Use structured logging. 2016-03-07 10:58:17 -08:00
Jordan Sissel
d45e303dca Make ${FOO} evaluation fail if FOO is not an environment variable
This updates the specs to cover this expectation. It also makes
assertions about default values if a default `${FOO:default value}` is
given.

Specs pass for me (18 specs)

(Improves on #4710)
2016-03-07 10:58:17 -08:00
Jordan Sissel
7f6bb9a782 Small refactor to use String#gsub and named captures for text replacements.
The config/mixin_spec.rb specs are passing for me (20 of 20 passing)

(Improves upon #4710)
2016-03-07 10:58:17 -08:00
fbaligand
9b89b884a0 Fix issue #3944 : Add support for environment variable injection in logstash plugin configuration 2016-03-07 10:58:17 -08:00
Joao Duarte
592835cea2 improve pipeline start and stop logging
Fixes #4764
2016-03-07 11:54:18 +00:00
Andrew Cholakian
4835024598 Fix oops backtrace logging, it should be structured
Fixes #4717
2016-02-26 18:53:56 +00:00
Pier-Hugues Pellerin
82dde31afe Fix an issue with the expectation on the agent#uptime
Fixes #4687

Fixes #4653
2016-02-12 19:00:11 +00:00
Pere Urbon-Bayes
7cdd77b822 make default values constants inside the webserver
Fixes #4685

Fixes #4653
2016-02-12 19:00:11 +00:00
Pere Urbon-Bayes
fecf02401c Cleanup debug variable used inside the pipeline and also fixed small typo in the memory command.
Fixes #4681

Fixes #4653
2016-02-12 19:00:11 +00:00
Pere Urbon-Bayes
e24fcdd8c1 make default bind to 127.0.0.1
Fixes #4676

Fixes #4653
2016-02-12 19:00:11 +00:00
Pere Urbon-Bayes
5b0c136e37 add an http-host option to bind to something else than 0.0.0.0 (default), also cleanup a bit the webserver logging code from unnecessary options
Fixes #4676

Fixes #4653
2016-02-12 19:00:11 +00:00
Pere Urbon-Bayes
93a3baa007 add human version to not_found error and rescue errors on webserver
Fixes #4662

Fixes #4653
2016-02-12 19:00:10 +00:00
Pere Urbon-Bayes
effff76588 ammend the way we show idle threads to skip as idle internal JVM threads, Ruby JIT threads and internal thread pool ones, also the concurent tasks ones used by jruby.
added comments to describe what we skip as idle and why inside the hot threads api

Fixes #4662

Fixes #4653
2016-02-12 19:00:10 +00:00
Pere Urbon-Bayes
a3280d345a add default content for not found errors
ignore waiting threads in the hotthreads and also add back the default threads number to 3

Fixes #4662

Fixes #4653
2016-02-12 19:00:10 +00:00
Pier-Hugues Pellerin
e06c5106c5 Threadsafety problem when creating the initial metric instance
This PR fix an issue where we were incorrectly creating the first
instance of the metric. This was causing missing increments

Fixes #4665

Fixes #4653
2016-02-12 19:00:10 +00:00
Pere Urbon-Bayes
2c4b8e8fd5 add a way to ask for pretty print json for the webapi
Fixes #4658

Fixes #4653
2016-02-12 19:00:10 +00:00
Pere Urbon-Bayes
65ee298280 make the API listen to 9600 port
add support to select the port from the runner cli option

move defaults pickup to the webserver for port and also cleanup debug code

Fixes #4655

Fixes #4653
2016-02-12 19:00:10 +00:00
Pere Urbon-Bayes
23dcf467e4 This make all designed resources for the metrics api available, this follows the initial sketches created in #4446.
As discussed in this adds:

/
/_node/hot_threads
/_node/stats/
/_node/stats/events
/_node/stats/jvm
/_stats/jvm

introduces also small refactors and cleanup necessary to improve the
webapi code quality.

adds also a way to handle references from the webapi to the internals of
logstash by passing the agent refrences to the collector, this is not
perfect, but for now it solve the communication situation until furder
refactor.

Fixes #4652

Fixes #4653
2016-02-12 19:00:10 +00:00
Pere Urbon-Bayes
81f83a231c introduced basic error logging
Fixes #4635

Fixes #4653
2016-02-12 19:00:10 +00:00