Commit graph

330 commits

Author SHA1 Message Date
Andrew Cholakian
4926974ef5 Cleans up output delegators and simplifies a few other aspects of plugins
The new way Output Delegators work is that events flow from:
OutputDelegator -> OutputDelegatorStrategy -> Output

The output delegator handles all the common denominator tasks (like metrics) and
a few other things. The OutputDelegatorStrategy handles concurrency and Output instantiation.
This is a significant improvement and simplification over the past where we used mixins and dynamic method
redifinition.

This removes the concept of plugin 'unique_names' and replaces it with the 'id'.
Also consistently autogenerates plugin IDs based on a given config file using SHA1 hashing.

Fixes #5752
2016-08-18 16:50:41 -04:00
Joao Duarte
086582ddcd add more numeric setting classes and control cli numeric coersion
Fixes #5773
2016-08-18 09:20:27 -04:00
Pier-Hugues Pellerin
16532ff345 make the human? method part of the base api class
Fixes #5766
2016-08-18 09:17:00 -04:00
Pier-Hugues Pellerin
aa5c5518cd adding test for the human hot_threads call
Fixes #5766
2016-08-18 09:17:00 -04:00
Pier-Hugues Pellerin
4ec6e8ae11 Small improvements for the hot_threads api
- Fix the typo in the human format
- Make sure we use the I18n string for the output

Fixes #5766
2016-08-18 09:17:00 -04:00
Pier-Hugues Pellerin
8fe0b05ba6 Add GC Strategy under /note/jvm
Fixes: #5755

Fixes #5775
2016-08-18 08:55:01 -04:00
Joao Duarte
c868c07154 show meaningful error message with an unknown command
Fixes #5772
2016-08-17 13:42:29 -04:00
Andrew Cholakian
c31bb3e8e7 Fix broken plugin tests that didn't have a defined data.home
This is done by providing a default data.home in tmp for environments other than a packaged logstash distro.

This is a hack and should be replaced by something better in the future.

Fixes #5706
2016-07-28 16:36:06 -04:00
Suyog Rao
244e88ae25 Bump version to 5.0.0-alpha5 (#5698) 2016-07-27 11:32:01 -07:00
Guy Boertje
a0688ac964 Add queue client support (and batch object) to pipeline (#5560)
* add batch object and queue client support to pipeline

* post review updates

* add note about inflight_batches to pipeline and wrapped_sync_queue

* remove more whitespace changes

* remove more whitespace changes

* rename add to merge + refactor event fill on ReadBatch.new

* better handle the initial filling of the batch, handle merging better
2016-07-27 16:27:15 +01:00
Pere Urbon-Bayes
108a56d0e6 add namespaced metrics for inputs
Fixes #5668
2016-07-21 11:25:52 -04:00
Pere Urbon-Bayes
3ec5d30038 move the payload filter to the respond_with method helper
Fixes #5630
2016-07-20 06:01:35 -04:00
Pere Urbon-Bayes
a951193d01 add generic filter for _node/stats api
ammend wrong rebase in terms of the hot_threads api fix

Fixes #5630
2016-07-20 06:01:34 -04:00
Pere Urbon-Bayes
bee0199eaf add the magic header # encoding: utf-8 to a few files, plus refactoring the hot threads code to be in single location
Fixes #5630
2016-07-20 06:01:34 -04:00
Pere Urbon-Bayes
a53ed94315 Introduce the first level, comma separated selector for fields. Also corrected an issue where memory refence was also using mem.
Fixes #5630
2016-07-20 06:01:34 -04:00
Pere Urbon-Bayes
4a8472c211 add option to filter payload (first level) in the node api
Fixes #5630
2016-07-20 06:01:34 -04:00
Pere Urbon-Bayes
1886c158c6 rename _node/stats/pipeline pipeline into plugins as is more clean that the resources means
Fixes #5623
2016-07-19 08:50:42 -04:00
Pere Urbon-Bayes
154b8ebafa /_node/stats/events object+resource is removed as these metrics now instead reside under the pipeline stats /_node/stats/pipeline
Fixes #5623
2016-07-19 08:50:42 -04:00
Pere Urbon-Bayes
f9dc80d6ec Removed /_node/stats/mem and insted nested them under /_node/stats/jvm along with the ‘thread’ stats
Fixes #5623
2016-07-19 08:50:42 -04:00
Pere Urbon-Bayes
4df49e836d add /_node/stats/pipeline inside /_node/stats main resource
Fixes #5623
2016-07-19 08:50:42 -04:00
Pere Urbon-Bayes
330845e2fa fix human parameter processing
Fixes #5621
2016-07-15 03:12:15 -04:00
Joao Duarte
1c16e14c66 fix string setting validation of possible values
Fixes #5628
2016-07-14 04:01:24 -04:00
Pier-Hugues Pellerin
8fc605d164 Review changes
Fixes #5542
2016-06-27 18:37:03 -04:00
Pier-Hugues Pellerin
8bf7c97990 Making sure the input plugin have access to the metric
Fixes #5542
2016-06-27 18:37:02 -04:00
Pier-Hugues Pellerin
c1feec058c Always provide a default metric to the plugin instance
Fix an issue when running tests of the plugin in isolation, no default
metric was created and the metric was nil.

This commit change the behavior to always return a `NullMetric` when
plugin instances are created outside of the pipeline.

Added tests to cover all the base class of plugin that can record
metric.

Fixes #5542
2016-06-27 18:37:02 -04:00
Pier-Hugues Pellerin
8443705d11 When running logstash in debug mode, display the stacktrace
Fixes #5542
2016-06-27 18:37:02 -04:00
Pier-Hugues Pellerin
0105547f57 Add more testing around the NullMetric implementation
The current test where only checking if the `NullMetric` Interface was
matching the original metric interface, this PR add more test around the
actual methods and fix an issue with `decrement` using too much
arguments in the context of a namespaced metric.

The test were also move into a shared example to be used in both the `NullMetric`
and the `NamedspacedMetric`.

Also the NullMetric class will no use the same validation for keys as the namespaced class and will
reject empty or nil keys.

Fixes #5545
2016-06-27 11:27:20 -04:00
Joao Duarte
c03c5523a3 add data.path setting
Fixes #5528
2016-06-24 07:32:25 -04:00
Pier-Hugues Pellerin
e2871db786 Make sure the namespace metric is send to the plugin correctly and MetricType#To_hash wont return all the values
Fixes: #5529

Fixes #5532
2016-06-22 09:24:34 -04:00
Joao Duarte
a6db0c9d54 update version to 5.0.0-alpha4 2016-06-21 12:29:23 +01:00
Joao Duarte
fa0eeb579f add single command release task and other release support tooling
Purpose:

* manage releases through a minimum number of rake tasks
* simplify building of snapshot builds
* create staged artifacts, candidates for releases, that required no changes to become releases
* this means the snapshot release process will not involve publishing gems, therefore:
* the gem artifacts should only be published to rubygems as a final artifact, at the time of GA

Changes:

* release artifacts no longer depend on gems of core components
* all core components are used locally AS-IS to minimize code changes between snapshot, RC and GA
* `versions.yml` describes the versions of all logstash parts and package
  * `rake version:set[version]` manage the yaml file and push the changes to the gemspecs/version.rb files
  * `rake version:set_plugin_api[version]` manage the yaml file and push the changes to the gemspecs/version.rb files
  * `rake artifact:all` generates SNAPSHOT artifacts: tar.gz, zip, rpm, deb
  * `RELEASE=1 rake artifact:all` creates release candidate artifacts + 4 gems: logstash-core, logstash-core-event, logstash-core-event-java and logstash-core-plugin-api

implements #5416 and #5414

Fixes #5460
2016-06-21 12:29:18 +01:00
Andrew Cholakian
2ce01fc92f Add stats default fields (hostname, http_address, version) 2016-06-20 16:20:33 -05:00
Suyog Rao
6f64b7031d Release 5.0.0-alpha3.snapshot3 (#5519) 2016-06-17 12:57:52 -07:00
Jordan Sissel
c09ca89316 Add plugin name and wallclock duration to output metrics.
Fixes #5472
2016-06-17 15:17:44 -04:00
Jordan Sissel
1ed9ef8181 Store duration_in_millis under the 'events' section in metrics
Fixes #5472
2016-06-17 15:17:44 -04:00
Jordan Sissel
aa04f830dc Add a gauge 'name' that shows the plugin's name in metrics
Fixes #5472
2016-06-17 15:17:44 -04:00
Jordan Sissel
c7523ba523 Use metric#time instead of computing duration locally.
Fix a bug in the metric that called 'milliseconds' 1_000_000 for a
multiplier when it should be 1000.

Fixes #5472
2016-06-17 15:17:44 -04:00
Jordan Sissel
8492482eb6 Rename :time_ms to :duration_in_millis
This was chosen based on discussion w/ Suyog, Alvin, and others.

Fixes #5472
2016-06-17 15:17:44 -04:00
Jordan Sissel
f4ee6f326c Structure the "plugins" stats as an array of data points.
Each data point contains the stats for a plugin instance (one particular
grok, for example).

Fixes #5472
2016-06-17 15:17:44 -04:00
Jordan Sissel
85f5bfe3bb Expose pipeline metrics to /_node/stats/pipelines
Fixes #5472
2016-06-17 15:17:44 -04:00
Jordan Sissel
5f8ee2b3fb Track wallclock time spent in each filter.
Fixes #5472
2016-06-17 15:17:44 -04:00
Pier-Hugues Pellerin
83c0f2f276 Namespace should use a copy of the arrays when creating a sub space
Fix: #5510

Fixes #5513
2016-06-16 17:26:05 -04:00
Andrew Cholakian
18fe37d993 Fix direct require of environment.rb to properly load Socket class
This was causing devutils to fail

Fixes #5511
2016-06-16 15:37:57 -04:00
Andrew Cholakian
422fa85962 Fix a regression where untyped fields would return null values
This patch fixes a bug where config fields declared without a type
would just return null.

Fixes #5512
2016-06-16 15:37:23 -04:00
Pere Urbon-Bayes
5f6b580fa5 add cleaner error for when the webserver port is already in use
Fixes #5489
2016-06-15 16:51:52 -04:00
Suyog Rao
f5d6942daa Release 5.0.0-alpha4.snapshot2 (#5505) 2016-06-15 13:47:32 -07:00
Andrew Cholakian
9e02bb8d26 Bring back accidentally deleted regexp param required validation
Fixes #5500
2016-06-15 14:40:52 -04:00
Andrew Cholakian
2f4a1e6c39 Remove hot_threads from default _node API endpoint results
Fixes #5381
2016-06-15 14:34:15 -04:00
Andrew Cholakian
d68ac5d2d6 Move _plugin api to _node/plugin + add tests
Fixes #5381
2016-06-15 14:34:15 -04:00
Andrew Cholakian
1567c5a966 Refactor MetricStore#extract_metrics to handle every possible situation that makes sense
Fixes #5381
2016-06-15 14:34:15 -04:00