Commit graph

1123 commits

Author SHA1 Message Date
Pier-Hugues Pellerin
988793dbb1 Add more testing around the NullMetric implementation
The current test where only checking if the `NullMetric` Interface was
matching the original metric interface, this PR add more test around the
actual methods and fix an issue with `decrement` using too much
arguments in the context of a namespaced metric.

The test were also move into a shared example to be used in both the `NullMetric`
and the `NamedspacedMetric`.

Also the NullMetric class will no use the same validation for keys as the namespaced class and will
reject empty or nil keys.

Fixes #5545
2016-06-27 11:27:20 -04:00
Joao Duarte
c0c9a3babc add data.path setting
Fixes #5528
2016-06-24 07:32:25 -04:00
Pier-Hugues Pellerin
1dd44383c7 Make sure the namespace metric is send to the plugin correctly and MetricType#To_hash wont return all the values
Fixes: #5529

Fixes #5532
2016-06-22 09:24:33 -04:00
Andrew Cholakian
d01b5f6b9c Add stats default fields (hostname, http_address, version)
Fixes #5520
2016-06-20 14:13:40 -04:00
Joao Duarte
a859bd4a3f add single command release task and other release support tooling
Purpose:

* manage releases through a minimum number of rake tasks
* simplify building of snapshot builds
* create staged artifacts, candidates for releases, that required no changes to become releases
* this means the snapshot release process will not involve publishing gems, therefore:
* the gem artifacts should only be published to rubygems as a final artifact, at the time of GA

Changes:

* release artifacts no longer depend on gems of core components
* all core components are used locally AS-IS to minimize code changes between snapshot, RC and GA
* `versions.yml` describes the versions of all logstash parts and package
  * `rake version:set[version]` manage the yaml file and push the changes to the gemspecs/version.rb files
  * `rake version:set_plugin_api[version]` manage the yaml file and push the changes to the gemspecs/version.rb files
  * `rake artifact:all` generates SNAPSHOT artifacts: tar.gz, zip, rpm, deb
  * `RELEASE=1 rake artifact:all` creates release candidate artifacts + 4 gems: logstash-core, logstash-core-event, logstash-core-event-java and logstash-core-plugin-api

implements #5416 and #5414

Fixes #5460
2016-06-20 04:23:48 -04:00
Jordan Sissel
246a2e7ad3 Fix tests that broke as a result of recent changes
Fixes #5472
2016-06-17 15:17:44 -04:00
Jordan Sissel
d2400ac93a Add plugin name and wallclock duration to output metrics.
Fixes #5472
2016-06-17 15:17:44 -04:00
Jordan Sissel
fdf19eab2e Store duration_in_millis under the 'events' section in metrics
Fixes #5472
2016-06-17 15:17:44 -04:00
Jordan Sissel
97a2f16b48 Add a gauge 'name' that shows the plugin's name in metrics
Fixes #5472
2016-06-17 15:17:44 -04:00
Jordan Sissel
07be0dcb3b Use metric#time instead of computing duration locally.
Fix a bug in the metric that called 'milliseconds' 1_000_000 for a
multiplier when it should be 1000.

Fixes #5472
2016-06-17 15:17:44 -04:00
Jordan Sissel
c8f7aee9b1 Rename :time_ms to :duration_in_millis
This was chosen based on discussion w/ Suyog, Alvin, and others.

Fixes #5472
2016-06-17 15:17:44 -04:00
Jordan Sissel
f5335c990a Structure the "plugins" stats as an array of data points.
Each data point contains the stats for a plugin instance (one particular
grok, for example).

Fixes #5472
2016-06-17 15:17:44 -04:00
Jordan Sissel
a8282184f3 Slight tidying of specs
Any metric calls are now allowed. This should help us grow our metrics
usage without accidentally failing tests that were too rigid.

I also split some testing up to have more alignment between the 'it'
description and the execution.

Fixes #5472
2016-06-17 15:17:44 -04:00
Jordan Sissel
6cb0b538e3 Fix failing tests. Allow metric time_ms capture now.
Fixes #5472
2016-06-17 15:17:43 -04:00
Jordan Sissel
1eef8c9be7 Expose pipeline metrics to /_node/stats/pipelines
Fixes #5472
2016-06-17 15:17:43 -04:00
Jordan Sissel
5a4d4bc3ba Track wallclock time spent in each filter.
Fixes #5472
2016-06-17 15:17:43 -04:00
Pier-Hugues Pellerin
73c1e68050 Namespace should use a copy of the arrays when creating a sub space
Fix: #5510

Fixes #5513
2016-06-16 17:26:05 -04:00
Andrew Cholakian
74672c994b Fix direct require of environment.rb to properly load Socket class
This was causing devutils to fail

Fixes #5511
2016-06-16 15:37:57 -04:00
Andrew Cholakian
48b78e8c58 Fix a regression where untyped fields would return null values
This patch fixes a bug where config fields declared without a type
would just return null.

Fixes #5512
2016-06-16 15:37:23 -04:00
Pere Urbon-Bayes
95cdab3583 add cleaner error for when the webserver port is already in use
Fixes #5489
2016-06-15 16:51:52 -04:00
Andrew Cholakian
11c3b83d10 Bring back accidentally deleted regexp param required validation
Fixes #5500
2016-06-15 14:40:52 -04:00
Andrew Cholakian
0f478b4faa Remove hot_threads from default _node API endpoint results
Fixes #5381
2016-06-15 14:34:15 -04:00
Andrew Cholakian
6268e063e3 Move _plugin api to _node/plugin + add tests
Fixes #5381
2016-06-15 14:34:15 -04:00
Andrew Cholakian
4c38288d31 Refactor MetricStore#extract_metrics to handle every possible situation that makes sense
Fixes #5381
2016-06-15 14:34:14 -04:00
Andrew Cholakian
86f776fafa Add more node stats + refactor stats internals
Add node stats mem endpoint

Fixes #5381
2016-06-15 14:34:14 -04:00
Andrew Cholakian
76c4bc9971 Add new :list property to configuration parameters.
If set to try this will allow the user to specify one or more values.
This generally replaces the :array type, which had fewer type checks.

The array type is still needed for lists of complex objects, e.g. hashes.

Fixes #5453
2016-06-13 14:13:56 -04:00
Pier-Hugues Pellerin
a741534f7c Code review
Fixes #5446
2016-06-10 09:51:36 -04:00
Pier-Hugues Pellerin
6acbdc3015 Make the test more reliable by used a fixed amount of generated events
Fixes #5446
2016-06-10 09:51:36 -04:00
Pier-Hugues Pellerin
47239b30c7 Fix the ApiTest and the Metric inputs tests
Fixes #5446
2016-06-10 09:51:36 -04:00
Pier-Hugues Pellerin
587debd3e6 typos
Fixes #5446
2016-06-10 09:51:36 -04:00
Pier-Hugues Pellerin
d81ad6172f Fix a thread safety issue with the agent.
Make sure we pass the metric when we create a pipeline this make sure
the metric is correctly visible in the pipeline and remove the need to
lock the variables.

Make the expectation more robust in the metric reload scenario.

Fixes #5400 for the greater good.

Fixes #5446
2016-06-10 09:51:36 -04:00
Pier-Hugues Pellerin
21a47b8db9 Fix an issue when data was not present in the collector
When you ran this test in isolation it failed to find the corresponding
metric, Changed the behavior to force a collect before running the
tests.

Fixes #5446
2016-06-10 09:51:36 -04:00
Pier-Hugues Pellerin
d02d4e4429 Refactor the metric library interaction
The metric library now require an instance of the collector and will not
use the singleton

Fixes #5446
2016-06-10 09:51:36 -04:00
Pier-Hugues Pellerin
0b50e2e624 Do not declare the collector as a singleton
Fixes #5446
2016-06-10 09:51:35 -04:00
Andrew Cholakian
35cffb5b39 Add URI config validator/type
Often times plugins (like the Elasticsearch output) can naturally use URIs for their configuration.
Unfortunately using the :string type here means that the password portion of the URI can easily be leaked.

This wraps the URI class in a new LogStash::Util::SafeURI class that proxies all regular URI methods but masks
the password when `#to_s` and `#inspect` are invoked.

Fixes #5439
2016-06-09 08:33:00 -04:00
Joao Duarte
f29ff3186f ensure plugins are correctly compiled outside of same plugin section
Fixes #5459
2016-06-07 16:17:05 -04:00
Andrew Cholakian
f0dfea9d80 Improve error handling in API / Rack App.
Setting error handling conditions in sinatra dynamically is not possible.
The values such as :show_exceptions get set, but have no effect.
This is esp. important in testing where we want failures to raise exceptions.
Otherwise, debugging is a matter of parsing instructable HTML responses containing
the debug info.

We do not want this behavior by default however, as doing so would mean a single API
bug would take down a production app.

This change adds a new agent setting (not CLI setting) 'http.environment' that is defaulted to 'production'.
If set to 'test' exceptions get raised.

This change also removes sinatra error handlers that previously did nothing, and injects
custom rack middleware to properly log and handle errors.

Fixes #5411
2016-06-01 20:19:58 +00:00
Joao Duarte
595658f512 abort if path.settings does not contain logstash.yml
Fixes #5393
2016-05-31 09:09:07 +00:00
Joao Duarte
6749c87572 tolerate incorrect path.settings value
Fixes #5393
2016-05-31 09:09:07 +00:00
Pier-Hugues Pellerin
83c14614c0 Remove XIT for metric test in the agent_spec.
The XIT was added to make sure we were able to build LS packages.
This PR revert the xit since the bug was fixed in #5307

Fixes: #5371

Fixes #5392
2016-05-30 17:05:38 +00:00
Andrew Cholakian
d7c4895d85 Fix resolution of ::Gem constant in API
Fixes #5384
2016-05-27 22:34:09 +00:00
Andrew Cholakian
53b164fd44 Always run sinatra in production, it makes debugging tests easier
Fixes #5384
2016-05-27 22:34:09 +00:00
Andrew Cholakian
e402062d77 Fix rake test:core path loading in API specs
Fixes #5384
2016-05-27 22:34:09 +00:00
Andrew Cholakian
6bf60f71ca Downgrade puma to 2.x to keep it compatible with the HTTP input for now
Fixes #5384
2016-05-27 22:34:08 +00:00
Andrew Cholakian
90d12f4e0a Refactor API + Add node_stats 2016-05-27 13:28:47 -05:00
Suyog Rao
ec95659dbd Fix failing tests on master and 5.0 (#5369)
License tests was failing because of GPL2 in mime-types.Added an
exception for this gem because v2.6.2 is triple-licensed and we pick
MIT.

Metrics test is failing because of address in use which is fixed in
PR #5307. Skipping this test for now.
2016-05-26 16:31:17 -07:00
Pier-Hugues Pellerin
1033df7be4 Fixing the faillings metrics tests (#5365)
This PR make sure that all the tests using the agent or the pipeline are
correctly shutting down their pipelines to make sure the metrics state
are coherent with the config defined in the test.

Fixes: #5291
2016-05-26 14:06:04 -07:00
Pier-Hugues Pellerin
f1f1a542e9 Initialize the path.settings inside the runner class
The `path.settings` requires the LOGSTASH_HOME constant to be defined,
the problem is that constant is only defined when you are actually
inside the logstash application, This was causing a bug when you were
testing plugin individually because that constant wasn't defined.

Fixes: #5361

Fixes #5363
2016-05-26 17:19:07 +00:00
Joao Duarte
7783f76a4c introduce yaml settings file and cli long switches
* by default lives in LOGSTASH_HOME/config/logstash.yml
* location can be changed by $LS_SETTINGS_DIR or --path.settings
* overrides defaults of LogStash::Environment, BUT
* CLI flags override the yaml file
* several breaking changes to cli flag names (see config/logstash.yml)
* setting values are now type checked (see Setting and Settings classes)

Fixes #5313
2016-05-19 18:15:12 +00:00
Joao Duarte
9a4d3631d5 adapt config_ast test to support lack of ivars 2016-05-17 08:47:54 +01:00