Commit graph

1202 commits

Author SHA1 Message Date
João Duarte
a5823baafe Remove license header from pipelines_info.rb
This code was moved to the apache 2.0 license part of the distribution and should not have the license header any more

Fixes #11345
2019-11-26 14:02:29 +00:00
Colin Surprenant
aa39a7e0a9 support remove_field on metadata and tests
Fixes #11334
2019-11-25 21:29:00 +00:00
amitav.mohanty
d65ef8edb6 Mention the path of DLQ to indicate DLQ if full for which pipeline
Fixes #11280
2019-11-14 08:09:38 +00:00
Joao Duarte
e853377d5e dont mutate SETTINGS object in keystore specs
Fixes #11270
2019-10-25 14:18:33 +00:00
Joao Duarte
a1930f25ac remove logging setting in cgroup_spec.rb
Fixes #11271
2019-10-25 10:24:28 +00:00
Joao Duarte
68dea86c54 improve reliability of webserver port binding
use "127.0.0.1" instead of "localhost" to avoid binding to ipv4 and ipv6
don't assume port 10006 will be open in the machine
rely on the ranges and the actual bound port for the assertions

Fixes #11263
2019-10-24 08:00:39 +00:00
Colin Surprenant
281baf4218 use 2048 bits key
Fixes #11115
2019-09-03 21:03:32 +00:00
Joao Duarte
6f8c677bf6 support substitutions in pipelines.yml file
Fixes #11081
2019-08-30 14:02:59 +00:00
Joao Duarte
7488ca768e give multiple pipelines all the settings
Previously we'd only give a pipeline the settings related to pipelines
The PipelineSettings class was used for this.
However a pipeline may need other settings like the keystore location.

For this we instead clone the settings object and merge all the pipeline
specific settings. This is accomplished with a new method that ensures
that only pipeline level settings are overwritten in the clone.

Fixes #11076
2019-08-30 11:20:35 +00:00
andsel
a3ac21d6e5 Add pipeline.id to log lines fixes #8290, #10521
Fixes #11075
2019-08-29 10:36:05 +00:00
Mike Place
6d140f0a7b
Enhanced API testing (#10972)
* Starting to audit tests

* Additional field checking in stats

* Add epehemeral id

* More tests

* Test new structure of pipeline report

* Add default_metadata testing

* Add node command tests

* add jvm

* test no mutate

* Add check for graph flag

* Break apart test per review suggestion

* Remove test that doesn't test much
2019-08-29 11:32:50 +01:00
Dan Hermann
3e99e1f203 Update Reflections library
Fixes #10951
2019-08-29 10:31:07 +00:00
Dan Hermann
5125e0729e Write generated Java files to disk only if debug flag is set
Fixes #11082
2019-08-28 14:24:13 +00:00
Dan Hermann
c75ef874ae Use correct execution engine for test-and-exit mode
Fixes #11067
2019-08-21 17:40:55 +00:00
Rob Bavey
0f25107cde Fix misleading log message
Fix DLQ error log message incorrectly referring to PQ.

Fixes #11063
2019-08-20 15:36:29 +00:00
João Duarte
accc636126 avoid variable collision in pipeline stats api (#11059)
Fixes #11062
2019-08-20 15:22:46 +00:00
amitav.mohanty
c2391b1849 Improved logging of version mismatch in DLQ file reader (RecordIOReader)
Fixes #11039
2019-08-16 12:50:23 +00:00
Dan Hermann
8f78612e93 fix javadoc warning
Fixes #11046
2019-08-13 21:18:25 +00:00
Dan Hermann
2375ae1b17 Expand config variables for Java plugins
Fixes #11043
2019-08-13 19:36:41 +00:00
Joao Duarte
085d93650e update jruby to 9.2.8.0
Fixes #11041
2019-08-13 06:45:58 +00:00
Rob Bavey
b2e8b0b6c4 Update jrjackson, jackson versions
Also splits out jackson and jackson databind versions to allow separate revisions
 for databind.

Fixes #11040
2019-08-12 18:08:22 +00:00
Dan Hermann
d64e5a7853 support truthy eval of constants
Fixes #11032
2019-08-09 17:23:55 +00:00
Dan Hermann
4b5dc343ae fix compilation of [field] in [field] event conditions
Fixes #11026
2019-08-08 20:02:24 +00:00
Dan Hermann
120b6a7d04 add regex support for constant conditionals
Fixes #11017
2019-08-06 10:30:10 +00:00
Rodrigo López Dato
b4f01e88fc Expose DLQ writers to all types of plugins, not just Ruby outputs
Fixes #10749
2019-07-12 17:21:54 +00:00
Rodrigo López Dato
fe4b6ee308 Allow LogStash::Event to be instantiated with a Java Event
Allows inputs such as dead_letter_queue to create Ruby events that
include both data and metadata from deserialised Java events

Fixes #10749
2019-07-12 17:21:54 +00:00
Joao Duarte
67bddcee12 disable testInvalidInputPlugin
Test has been failing on windows for many weeks.

See https://github.com/elastic/logstash/issues/10926

Fixes #10927
2019-07-02 18:54:54 +00:00
Dan Hermann
5e8a694d59 simplify unit test
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
46c1611dff use method-level ensure blocks
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
1802869d80 remote != remove
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
e925b4c48d don't put nulls in the map
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
f48da1e461 add missing lock statement
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
0ff6d11344 don't perform long-running pipeline actions inside calls to ConcurrentHashMap.compute to avoid deadlocks
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
d1e92862c3 serialize access to PipelineBus methods on a per-plugin basis, code cleanup in AddressState
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
7d206b78d1 Fix pipeline shutdown ordering
Fixes #10872
2019-06-29 17:27:22 +00:00
Mike Place
ed57c5dc53 Inject hash and ephemeral_id into stats
Fixes #10885
2019-06-27 08:45:52 +00:00
Mike Place
20fc465510
Add workers and batch_size to root request (#10853) (#10910)
* Add workers and batch_size to root request

* Add batch delay per review recommendation
2019-06-26 23:14:14 +02:00
Mike Place
c918082617
Restore UUID lookup to node (#10884) (#10911)
* Restore UUID lookup to node

* Fix incorrect merge

* Move private method to bottom

* Change method name per review suggestion

* Update method description
2019-06-26 23:05:33 +02:00
Mike Place
9a5faf24bf
Enhance GET _node/stats/pipelines API for Metricbeat monitoring (#10576) (#10909)
* parent 8c5697c748
author Guy Boertje <guy@elastic.co> 1556806171 +0100
committer Mike Place <mike.place@elastic.co> 1557234770 +0200

Bump JrJackson to 0.4.8

Fixes #10748

LIR serializer refactor

Remove commented code

Remove more commented code

Remove license and add encoding

Style change to make code more vertical.

eid and hash

Use pipelines_info to construct the stats

Add tests for new fields

Add queue stats

* bad merge resolution

* bad merge resolution

* Don't merge if nil

* Better merge strategy

* add vertex gate

* Guard against nil

* Use extended queue stats in pipeline report

* Add cluster uuids to Elasticsearch outputters in pipeline output

* move uuid

* remove old uuid lookup

* Only populate cluster_uuids when present

* remove print

* cluster_uuids -> cluster_uuid

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Make var singular

* Match singular var name

* Remove unnecessary nil check

* Pass in the matching pipeline for the report

* Remove old way of inserting cluster_uuids

* Update logstash-core/lib/logstash/api/commands/stats.rb

I like this much better and in testing it seems to work correctly.

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Remove unreferenced code that was part of debugging

* Remove events var which was unused

* Don't try to remove before insert

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Make pipeline extended stats generation more efficient

* Implement suggestion to improve readability

* Cleaner merging per review recommendation

* Only generate extended_stats once

* remove unneeded comments

* Add cluster_uuid to node vertex

* remove top-level cluster_uuids

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Implement change to make logic more simple suggested in review

* Rely on options gate to insert graph

Resolves concern here:
https://github.com/elastic/logstash/pull/10576#issuecomment-501774635

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Move UUID lookup to API layer

* Move private method to bottom per review recommandation
2019-06-26 22:58:20 +02:00
Dan Hermann
d03c0f14a4 Remove debug code for p2p plus formatting
Fixes #10840
2019-06-25 18:31:37 +00:00
Dan Hermann
8a048ccc27 Do not shut down API webserver until after pipelines have been shut down
Fixes #10880
2019-06-25 18:30:20 +00:00
Joao Duarte
f18992c647 avoid clashes between Environment class methods
Fixes #10860
2019-06-14 17:44:38 +00:00
Dan Hermann
b519dc1e1e fix parsing of boolean options provided to Java plugins
Fixes #10848
2019-06-07 19:54:08 +00:00
Ry Biesemeyer
b76ea1989e field_refefence: handle illegal field references in converted maps
When using the Jruby event API, re-cast java exceptions produced by illegal
field references to ruby `RuntimeError`s, which can be caught by the ruby-based
plugins.

This is similar to what we already do in the Jruby event API when directly
handling field references, but catches a case where the `Valuifier` encounters
an illegal reference when creating a `ConvertedMap`.

Fixes #10839
2019-06-06 21:27:40 +00:00
Dan Hermann
8a2b7be327 plain codec for Java
Fixes #10791
2019-05-30 14:45:18 +00:00
Dan Hermann
6dd85ca2a8 Fixes unit test failures on some runs of ConfigCompilerTest::testComplexConfigToPipelineIR
Fixes #10837
2019-05-30 14:40:51 +00:00
Dan Hermann
8685d1d483 Merge config values in LIR
Fixes #10832
2019-05-30 12:59:09 +00:00
Dan Hermann
78607d2bbb LIR support for octal literals in pipeline definitions
Fixes #10828
2019-05-28 16:51:03 +00:00
Dan Hermann
2d35566ab2 Document copy semantics of QueueWriter::push method
Fixes #10808
2019-05-22 10:51:53 +00:00
Dan Hermann
9a06b204bf Provide DLQ writer interface to Java plugins
Fixes #10790
2019-05-22 03:38:44 +00:00