Commit graph

1654 commits

Author SHA1 Message Date
Dan Hermann
eaeffd47b5 don't put nulls in the map
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
e0135c4e63 add missing lock statement
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
b9da367d39 don't perform long-running pipeline actions inside calls to ConcurrentHashMap.compute to avoid deadlocks
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
cfa31467f4 serialize access to PipelineBus methods on a per-plugin basis, code cleanup in AddressState
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
1b16eca32a Fix pipeline shutdown ordering
Fixes #10872
2019-06-29 17:27:22 +00:00
Mike Place
2dc6150fd8
Restore UUID lookup to node (#10884)
* Restore UUID lookup to node

* Fix incorrect merge

* Move private method to bottom

* Change method name per review suggestion

* Update method description
2019-06-26 14:32:19 +02:00
Dan Hermann
6ba1b1ff67 Remove debug code for p2p plus formatting
Fixes #10840
2019-06-25 18:31:37 +00:00
Dan Hermann
ea0485d5bf Do not shut down API webserver until after pipelines have been shut down
Fixes #10880
2019-06-25 18:30:21 +00:00
Mike Place
03001fa1f5
Add workers and batch_size to root request (#10853)
* Add workers and batch_size to root request

* Add batch delay per review recommendation
2019-06-25 13:44:25 +02:00
Mike Place
aa0588000c
Inject hash and ephemeral_id into stats (#10885) 2019-06-25 13:19:23 +02:00
Mike Place
252d5e7686
Enhance GET _node/stats/pipelines API for Metricbeat monitoring (#10576)
* parent 8c5697c748
author Guy Boertje <guy@elastic.co> 1556806171 +0100
committer Mike Place <mike.place@elastic.co> 1557234770 +0200

Bump JrJackson to 0.4.8

Fixes #10748

LIR serializer refactor

Remove commented code

Remove more commented code

Remove license and add encoding

Style change to make code more vertical.

eid and hash

Use pipelines_info to construct the stats

Add tests for new fields

Add queue stats

* bad merge resolution

* bad merge resolution

* Don't merge if nil

* Better merge strategy

* add vertex gate

* Guard against nil

* Use extended queue stats in pipeline report

* Add cluster uuids to Elasticsearch outputters in pipeline output

* move uuid

* remove old uuid lookup

* Only populate cluster_uuids when present

* remove print

* cluster_uuids -> cluster_uuid

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Make var singular

* Match singular var name

* Remove unnecessary nil check

* Pass in the matching pipeline for the report

* Remove old way of inserting cluster_uuids

* Update logstash-core/lib/logstash/api/commands/stats.rb

I like this much better and in testing it seems to work correctly.

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Remove unreferenced code that was part of debugging

* Remove events var which was unused

* Don't try to remove before insert

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Make pipeline extended stats generation more efficient

* Implement suggestion to improve readability

* Cleaner merging per review recommendation

* Only generate extended_stats once

* remove unneeded comments

* Add cluster_uuid to node vertex

* remove top-level cluster_uuids

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Implement change to make logic more simple suggested in review

* Rely on options gate to insert graph

Resolves concern here:
https://github.com/elastic/logstash/pull/10576#issuecomment-501774635

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Move UUID lookup to API layer

* Move private method to bottom per review recommandation
2019-06-22 08:31:13 +02:00
Joao Duarte
7468ed058f avoid clashes between Environment class methods
Fixes #10860
2019-06-14 17:44:38 +00:00
Dan Hermann
b477869eb3 fix parsing of boolean options provided to Java plugins
Fixes #10848
2019-06-07 19:54:08 +00:00
Ry Biesemeyer
4fab0f531e field_refefence: handle illegal field references in converted maps
When using the Jruby event API, re-cast java exceptions produced by illegal
field references to ruby `RuntimeError`s, which can be caught by the ruby-based
plugins.

This is similar to what we already do in the Jruby event API when directly
handling field references, but catches a case where the `Valuifier` encounters
an illegal reference when creating a `ConvertedMap`.

Fixes #10839
2019-06-06 21:27:40 +00:00
Dan Hermann
d9c60bfe35 plain codec for Java
Fixes #10791
2019-05-30 14:45:18 +00:00
Dan Hermann
2449e94164 Fixes unit test failures on some runs of ConfigCompilerTest::testComplexConfigToPipelineIR
Fixes #10837
2019-05-30 14:40:51 +00:00
Dan Hermann
2f5aff2362 Merge config values in LIR
Fixes #10832
2019-05-30 12:59:09 +00:00
Dan Hermann
fd74ce0156 LIR support for octal literals in pipeline definitions
Fixes #10828
2019-05-28 16:51:03 +00:00
Dan Hermann
9a78a6ce80 Document copy semantics of QueueWriter::push method
Fixes #10808
2019-05-22 10:51:52 +00:00
Dan Hermann
08d2443a37 Provide DLQ writer interface to Java plugins
Fixes #10790
2019-05-22 03:38:44 +00:00
Dan Hermann
7cd6bb89ae Default stack trace size to 50 and make it configurable
Fixes #10793
2019-05-21 15:42:06 +00:00
Jordan Doyle
15ac5db739 Expose Metrics API to Java plugins (#10761)
Expose Metric API to Java plugins and migrate JavaCodecDelegator to new Metric API
2019-05-20 11:24:07 -04:00
Jordan Doyle
bf1881dff2 Include G1 in JVM heap metrics (#10728) 2019-05-10 15:34:07 -05:00
João Duarte
43f28479e9
fix javadoc warning for Codec.java (#10756) 2019-05-06 09:34:50 +01:00
Dan Hermann
0de53939e9 * Adds a java_generator input with jdots codec to facilitate testing, adds float config type
* Breaking change to codec.encode method

* Sink output for discarding events

* URI and password config types

* Utility methods for packaging Java plugins

* Plugin API validation, fix gemspec generation

* Plugin Jar validation

* Update developer documentation

* Update codec metrics for new encode method

* Beta: Isolated classloaders for Java plugins

* Address code review comments

Fixes #10620
2019-05-03 11:54:37 +00:00
Colin Surprenant
c28ded78c9
fix JRuby resolv.rb leak (#10734) 2019-05-01 13:58:49 -04:00
Mike Place
3f56d93e01
Enhance GET / API for Metricbeat (#10589)
* ephemeral id

* hard-code status

* Add snapshot field
2019-05-01 13:14:47 +02:00
Mike Place
cc3c5ec00f
Refactoring of LIR serializer and exposing pipeline metrics (#10561)
* [WIP] Add ephemeral id and hash to pipeline stats

Refs #10119

* Fix incorrect hash id

* Add graph metrics

* Include node update

* LIR serializer refactor

* Remove commented code

* Remove more commented code

* Update spec

* Remove license and add encoding

* Style change to make code more vertical.

* Implement review suggestions

* Remove commented code

* Fix spec path

* Add cluster UUID

* Add graph?=true parameter

* Add options to pipelines/

* Use pluginmetadata

* Replicate change from java_pipeline to ruby pipeline
2019-05-01 12:45:11 +02:00
Rob Bavey
b088147cf1 Move plugin cleanup before retry
Previously the `do_close` method would never be called on a failing
plugin, because the retry call stops the code under `ensure` from
ever being called.

Fixes #10691
2019-04-26 13:15:46 +00:00
Rob Bavey
4f270295a1 Adds cleanup after shutdown of plugin
Fixes #10691
2019-04-26 13:15:46 +00:00
Rob Bavey
9c09f1ee50 Add methods to clear PluginMetadata repositories
Fixes #10691
2019-04-26 13:15:46 +00:00
Ry Biesemeyer
1fa9454c7c adds LogStash::PluginMetadata for simple key/value plugin metadata
We need a way for a plugin to register simple metadata about external
resources it connects to in order to implement a Monitoring feature in which
an Elasticsearch Output Plugin can store the connected cluster's uuid (#10602)

Here, we add a generic `LogStash::PluginMetadata` along with a registry, and
expose an accessor on `LogStash::Plugin#plugin_metadata` so that instances
can access their own metadata object.

Fixes #10691
2019-04-26 13:15:45 +00:00
Jordan Johnson-Doyle
30879b293a Add tests for LogStash::Codec::Delegator
Fixes #10614
2019-04-11 13:10:18 +00:00
Jordan Johnson-Doyle
58dcfae199 Add tests for JavaCodecDelegator
Fixes #10614
2019-04-11 13:10:18 +00:00
Jordan Johnson-Doyle
68e9c6ff04 Collect and expose metrics from Ruby codecs
Fixes #10614
2019-04-11 13:10:18 +00:00
Jordan Johnson-Doyle
310541f41b Collect and expose metrics from Java codecs
Fixes #10614
2019-04-11 13:10:18 +00:00
Dan Hermann
03df1316ce Correctly sets the default codec to java_line. Fixes buffer handling for events whose encodings do not fit into the buffer.
Fixes #10673
2019-04-10 18:32:05 +00:00
Dan Hermann
a16e67d332 Handle duplicate config entries
Fixes #10619
2019-04-04 17:53:44 +00:00
Dan Hermann
a808c8fcb3 Log successful shutdowns
Fixes #10628
2019-04-04 08:51:00 +00:00
Rob Bavey
fb5eba8c8a Fix issue setting 'enable_metric => false' on a plugin
This commit fixes a ClassCastException which happens when
 a plugin has the `enable_metric` setting set to false - a
 NullMetricExt is assumed, but that is only created when
 'metric.collect' is set to 'false' in the Logstash configuration,
 not when an individual plugin disables its metrics.

Fixes #10538
2019-03-13 10:43:17 +00:00
Ry Biesemeyer
fcd52c1573 prevent concurrent convergence (e.g., SIGHUP during in-flight convergence)
There are several scenarios in which we can trigger concurrent convergence in
the agent, resulting in two or more threads working to perform interleaved and
potentially conflicting or overlapping pipeline actions. Notably, our trap on
`SIGHUP` will be resolved in its own thread, so if we are sent `SIGHUP` while
in the process of converging, the second in-flight convergence may get its
starting state before, during, or after the effects of the first convergence.

By mutually excluding execution of the convergence cycle, we eliminate the
class of bugs in which one convergence acquires actions that cannot succeed due
to the prior success of actions given to the other convergence.

Fixes #10537
2019-03-12 17:37:40 +00:00
Dan Hermann
80c966b03e fix bug with explicitly-specified Java codecs
Fixes #10520
2019-03-08 02:05:57 +00:00
Colin Surprenant
0cdefb9141
remove exclusive lock for Ruby pipeline initialization (#10431) 2019-02-15 17:27:43 -05:00
Rob Bavey
95a0362fb9 Mute Failing Test on Windows
Mute mixin_spec, tracked by #10454

Fixes #10455
2019-02-15 19:44:21 +00:00
Dan Hermann
0efc0d58d5 One more character encoding fix
Fixes #10452
2019-02-14 21:36:51 +00:00
Dan Hermann
5505f7c9d5 fix Stdin tests to work on platforms that do not have UTF-8 as the default character encoding
Fixes #10449
2019-02-14 16:56:07 +00:00
Dan Hermann
9eb2aac36a Fix line codec tests so they work on platforms such as Windows that do not have UTF-8 as the default character encoding
Fixes #10446
2019-02-14 14:47:56 +00:00
Ry Biesemeyer
b605161662 ast/lir: simplify concurrent use of AST, which is globally stateful
Fixes #10415
2019-02-09 00:24:46 +00:00
Rob Bavey
51d7723fb6 Remove unnecessary option to not require host in SafeURI
Fixes #10414
2019-02-08 14:14:33 +00:00
Rob Bavey
795fd98e52 Better handle malformed URIs
Raise an error if a URI has inadvertently been supplied without a host

Fixes #10414
2019-02-08 14:14:33 +00:00