Commit graph

330 commits

Author SHA1 Message Date
Andrew Cholakian
2d64792d91 Add more node stats + refactor stats internals
Add node stats mem endpoint

Fixes #5381
2016-06-15 14:34:15 -04:00
Andrew Cholakian
9caae09baf Add new :list property to configuration parameters.
If set to try this will allow the user to specify one or more values.
This generally replaces the :array type, which had fewer type checks.

The array type is still needed for lists of complex objects, e.g. hashes.

Fixes #5453
2016-06-13 14:13:56 -04:00
Pier-Hugues Pellerin
a9e71f5234 Code review
Fixes #5446
2016-06-10 09:51:37 -04:00
Pier-Hugues Pellerin
420abaef0b Fix the ApiTest and the Metric inputs tests
Fixes #5446
2016-06-10 09:51:37 -04:00
Pier-Hugues Pellerin
20cede5b0a typos
Fixes #5446
2016-06-10 09:51:37 -04:00
Pier-Hugues Pellerin
1502fc9348 Fix a thread safety issue with the agent.
Make sure we pass the metric when we create a pipeline this make sure
the metric is correctly visible in the pipeline and remove the need to
lock the variables.

Make the expectation more robust in the metric reload scenario.

Fixes #5400 for the greater good.

Fixes #5446
2016-06-10 09:51:37 -04:00
Pier-Hugues Pellerin
189b201019 Refactor the metric library interaction
The metric library now require an instance of the collector and will not
use the singleton

Fixes #5446
2016-06-10 09:51:36 -04:00
Pier-Hugues Pellerin
189adea066 Do not declare the collector as a singleton
Fixes #5446
2016-06-10 09:51:36 -04:00
Andrew Cholakian
53ed1defd0 Add URI config validator/type
Often times plugins (like the Elasticsearch output) can naturally use URIs for their configuration.
Unfortunately using the :string type here means that the password portion of the URI can easily be leaked.

This wraps the URI class in a new LogStash::Util::SafeURI class that proxies all regular URI methods but masks
the password when `#to_s` and `#inspect` are invoked.

Fixes #5439
2016-06-09 08:33:01 -04:00
Joao Duarte
ed46e475ea ensure plugins are correctly compiled outside of same plugin section
Fixes #5459
2016-06-07 16:17:05 -04:00
Suyog Rao
45ee712ac3 Release 5.0.0-alpha4.snapshot1 (#5441) 2016-06-03 10:47:10 +02:00
Andrew Cholakian
ef18693115 Improve error handling in API / Rack App.
Setting error handling conditions in sinatra dynamically is not possible.
The values such as :show_exceptions get set, but have no effect.
This is esp. important in testing where we want failures to raise exceptions.
Otherwise, debugging is a matter of parsing instructable HTML responses containing
the debug info.

We do not want this behavior by default however, as doing so would mean a single API
bug would take down a production app.

This change adds a new agent setting (not CLI setting) 'http.environment' that is defaulted to 'production'.
If set to 'test' exceptions get raised.

This change also removes sinatra error handlers that previously did nothing, and injects
custom rack middleware to properly log and handle errors.

Fixes #5411
2016-06-01 20:20:13 +00:00
Suyog Rao
68bf39e659 Release 5.0.0-alpha3 gems and lock file (#5394) 2016-05-31 12:05:53 +02:00
Joao Duarte
1ed838fd48 abort if path.settings does not contain logstash.yml
Fixes #5393
2016-05-31 09:09:08 +00:00
Joao Duarte
1520a36cd4 tolerate incorrect path.settings value
Fixes #5393
2016-05-31 09:09:08 +00:00
Suyog Rao
9764584698 Release 5.0.0.alpha3 RC8 (#5386) 2016-05-29 21:15:41 +02:00
Andrew Cholakian
8b8a0b0f52 Fix resolution of ::Gem constant in API
Fixes #5384
2016-05-27 22:47:47 +00:00
Andrew Cholakian
67347441dd Always run sinatra in production, it makes debugging tests easier
Fixes #5384
2016-05-27 22:47:47 +00:00
Andrew Cholakian
b2ec0cffbc Refactor API + Add node_stats
Fixes #5307
2016-05-27 20:01:24 +00:00
Suyog Rao
4bd26f2c53 Release 5.0.0-alpha3.snpashot5 (#5368) 2016-05-26 15:02:51 -07:00
Pier-Hugues Pellerin
60c0aa072a Fixing the faillings metrics tests
This PR make sure that all the tests using the agent or the pipeline are
correctly shutting down their pipelines to make sure the metrics state
are coherent with the config defined in the test.

Fixes: #5291

Fixes #5365
2016-05-26 21:21:45 +00:00
Pier-Hugues Pellerin
12debcf529 Initialize the path.settings inside the runner class
The `path.settings` requires the LOGSTASH_HOME constant to be defined,
the problem is that constant is only defined when you are actually
inside the logstash application, This was causing a bug when you were
testing plugin individually because that constant wasn't defined.

Fixes: #5361

Fixes #5363
2016-05-26 17:19:07 +00:00
Suyog Rao
03ece707c7 Release 5.0.0-alpha3 snapshot4 (#5360) 2016-05-26 07:56:11 -07:00
Joao Duarte
3a6b1dafaa introduce yaml settings file and cli long switches
* by default lives in LOGSTASH_HOME/config/logstash.yml
* location can be changed by $LS_SETTINGS_DIR or --path.settings
* overrides defaults of LogStash::Environment, BUT
* CLI flags override the yaml file
* several breaking changes to cli flag names (see config/logstash.yml)
* setting values are now type checked (see Setting and Settings classes)

Fixes #5313
2016-05-19 19:20:37 +01:00
Joao Duarte
ce473ab502 avoid dynamic use of singleton methods in pipeline compilation 2016-05-17 08:46:37 +01:00
Colin Surprenant
98abb22258 use closure table instead of ivars 2016-05-17 08:46:37 +01:00
Joao Duarte
723046fb1e reduce overhead of create_pipeline if config is known 2016-05-17 08:46:37 +01:00
Joao Duarte
c79ca77bd8 avoid creating pipeline if fetched config is the same as previous 2016-05-17 08:46:37 +01:00
Joao Duarte
fb40deee17 avoid class variable during pipeline compilation 2016-05-17 08:46:37 +01:00
Tal Levy
fdd98591d4 undo allow_env flag
allow-env flag was introduced because of a backwards incompatible
change, this flag is no longer needed for 5.0 and onwards.

Closes #5263.

Fixes #5265
2016-05-16 01:50:02 +00:00
Pier-Hugues Pellerin
2d5f32cbd9 mixing the logstash-core
Fixes #5275
2016-05-11 16:02:49 +00:00
Pier-Hugues Pellerin
f883d90622 Release a new beta1.snapshot2 gem
Releated to #5273

Fixes #5275
2016-05-11 16:02:49 +00:00
Jordan Sissel
3fa057e3bc Enable JSON logging for all test runs.
Use an internal subscriber to verify that JSON output is valid JSON. The
purpose is to catch any json serialization errors that would occur while
logging.

Also had to update a few logger calls to log values that could be
serialized (Class instances and similar, at this time, fail to serialize
to JSON).

Fixes #4820
2016-05-09 19:53:17 +00:00
Jordan Sissel
755e4b5710 Allow Logstash to write its logs in JSON format
This is made available by a new `--log-in-json` flag. Default is false.
When false, the old behavior [1] is used. When true, JSON logs are
emitted.

[1] The old behavior is realy two things. First, using Object#inspect to
serialize. Second, to color the output if the IO is a tty.

Fixes #1569

Fixes #4820
2016-05-09 19:53:17 +00:00
Colin Surprenant
537d6a2780 refactor Ruby Event getter and setter
refactor wip gemfiles

refactor Java Event getter and setter

bump plugin-api to 2.0

use plugin-api 2.0

switch to core-event-java

include logstash-core-event-java.jar jar file so that gem dependency using the source tree work

updated core plugins to core-api 2.0

added grok for refactor branch

fix rebased specs

remove temp plugins github paths

remove commented out tmp alias_method
2016-05-06 13:47:03 -04:00
Pier-Hugues Pellerin
f73cff65f0 Remove constant redefinition
The Rack config was redefining a constant in the configuration, since
this value isn't used anywhere else there was no need to use a constant.

The conflicting ROOT contants was defined in `lib/logstash/patches/profile_require_calls.rb`

Fixes: #5163

Fixes #5208
2016-05-06 16:54:06 +00:00
Suyog Rao
3de59fec01 Remove unused java version check
Fixes #5243
2016-05-04 20:31:43 +00:00
Suyog Rao
0f27338495 Make Java 8 minimum version for LS 5.0
Fixes #5242

Fixes #5243
2016-05-04 20:31:43 +00:00
Suyog Rao
59e113c852 Release 5.0 alpha2 (#5233)
* Release 5.0.0-alpha2
2016-05-02 11:15:16 -07:00
Suyog Rao
a8380099e4 Release 5.0.0-alpha2 snapshot2 (#5225)
* Release 5.0.0-alpha2 snapshot2 release
2016-04-29 10:21:09 -07:00
Suyog Rao
0fbded8e43 Release alpha2 snapshot1 2016-04-26 13:04:41 -07:00
Pier-Hugues Pellerin
2a7b5668ca Fix non derterministic test when accessing the metric store
This commit introduce a mutex around the structured hash for the metric,
this hash is not updated frequently and its used mostly for the api to
be able to do search on the collected metric. Adding a mutex make sure
the changes are visible accross thread.

Fixes #5152

Fixes #5178
2016-04-22 17:22:05 +00:00
Pier-Hugues Pellerin
62c609f135 Clear the collector when logstash reload configuration
When logstash reload a configuration the collector should remove all the
collected metrics from the store since it wont make any sense with a new
configuration. You should have the same behavior as when you restart
logstash.

Fixes #4801
2016-04-18 20:03:28 +00:00
Tony Laidig
9f7fe0d4e5 change of I18n varname, possible fix for #5108
Fixes #5111
2016-04-14 00:51:32 +00:00
Pere Urbon-Bayes
907f85177c Add generic code used to load any kind of plugins across logstash
Introduce the idea of a registry plugin placeholder where all necessary
interface to plugins is mantained, also simplified the internal registry
calls to be more generic.

Add a way to handle registrations for plugins explicitly

introduced the idea of self.plugin_type method to fetch plugin type from base clase, also removed the former plugins:mixin used to annotate defined plugins

make the config_name method also handle the registration to the plugin registry, that way old plugins get registration out of the box and we can simply incoming plugin registry without automatic loading

simplify the plugin registry by removing former need to load classes, now they all get registered automatically when using the config_name method

cleanup unnecessary former changes

updated typo in comments for the plugins registry and also removed internal attr_reader for the same class

renamed plugin annotate to declare_plugin to have a more meaningful name

change Registry::Plugin.gem_name -> cannonic_gem_name to reflect the idea of having probably also other non cannonic gem names

Fixes #4535
2016-04-12 14:44:52 +00:00
Pier-Hugues Pellerin
0f5089d1f0 Logstash 5.0.0 alpha1
Fixes #5011
2016-04-05 14:08:06 +00:00
Pier-Hugues Pellerin
793b867ce1 5.0.0.snapshot2
Fixes #5006
2016-04-04 21:45:38 +00:00
Tal Levy
4f45adf940 fix path to --allow-env I18n description
Fixes #5007
2016-04-04 21:43:57 +00:00
Andrew Cholakian
2670668f9b Prevent config leak on output delegator close
On shutdown logstash may leak config information for outputs including
plaintext passwords

Fixes #5001
2016-04-04 21:19:15 +00:00
Tal Levy
82ed28bf86 add experimental flag for environment variables in config
Fixes #4963
2016-04-04 18:45:13 +00:00