Commit graph

1654 commits

Author SHA1 Message Date
Ry Biesemeyer
0b22b00893
support Environment and Keystore substitutions in password-type plugin options (#11774)
Resolves: https://github.com/elastic/logstash/pull/10583
Co-authored-by: Tyler Gregory <td.g@icloud.com>
2020-04-13 17:28:18 -07:00
Rob Bavey
2899c9f379 Handle Boolean Edges
Fixes #11779
2020-04-13 20:09:27 +00:00
Rob Bavey
830b6ce64e Remove separator vertices and associated edges from serialized output
The separator vertices are an implementation detail of the serialized
output of the LIR, and are not meaningful to the pipeline viewer.

This commit removes the separator vertices, and reworks the edges to
account for this.

Fixes #11779
2020-04-13 20:09:27 +00:00
João Duarte
3a84beda3d
remove :cluster_uuids gauge (#11628) 2020-04-13 09:36:11 +01:00
Ry Biesemeyer
70f4c3294f performance: share a single secret store
Loading a Java Keystore can take anywhere from ~0.3s to upwards of 3s, so the
pattern of loading one per variable we need to replace adds a significant
amount of overhead on pipelines that use these variables, whether or not they
are provided by the keystore.

By providing a private, constant, lazy singleton, we ensure that we don't
incur the cost of repeatedly building the keystore.

Fixes #10794
2020-04-10 15:41:17 +00:00
Andrew Pan
8793fb3c49
Update local.rb for pipe file (#11109)
Load pipe file for process communication

Co-authored-by: Andrew Pan <px2557@foxmail.com>
2020-04-10 09:52:12 +02:00
Rob Bavey
58314a7f0e
Update gradle version to 6.3 (#11742)
* Update gradle version to 6.3

Gradle versions prior to 6.3 cannot run under JDK14.
This commit upgrades the version of Gradle to 6.3, and removes all deprecation warnings that can currently be removed.

Changes include:
* Increase gradle memory to 2g
* Increase gradle memory in the license check job to 2g
* Replace use of `testCompile`
* Replace `runtime` with `runtimeOnly`
* Remove`compile` depedencies from gradle files
* Replace deprecated archive methods
* Fix dependencies report build
* Make jruby dependencies 'api', fix archiveVersion
* Set `duplicatesStrategy` for all tasks of type Copy
* Use `configureEach` for global 'withType' calls
** Use the recommended Tasks API calls
(https://blog.gradle.org/preview-avoiding-task-configuration-time)
* Run `./gradlew wrapper` earlier to improve caching
* Use copy with chown for resources that need to be run during `./gradlew wrapper`
2020-04-07 12:49:36 -04:00
Colin Surprenant
5a25c6f8e9 simplify batch classes, do not compute JE empty batches, refactor RE worker loop (#11737)
cleanup RubyArray "rawtypes"
remove all LinkedHashSet from batch and queue classes
avoid processing empty batches in Java worker loop
cleanup AckedReadBatch and MemoryReadBatch
refactor Ruby worker loop similar to Java Execution to not use batch merge
remove QueueBatch merge and replace LinkedHashSet with ArrayList
2020-04-02 16:26:54 -04:00
Karol Bucek
24d3348776 Refactor: avoid array in case of single event
while also making the array case cleaner & effective

(JRuby uses specialized array holder for 1 / 2 values)

+ Refactor: minor - use true/false constants directly
+ Refactor: do not allocate empty array

Fixes #11732
2020-04-01 07:37:02 +00:00
Colin Surprenant
dbca0b36a1
separate filter & output execution, rebatch after filter when ordered (#11710) 2020-03-27 15:36:23 -04:00
João Duarte
d522a7cb30
bring back unicode trimmer for LogStash::Util (#11712)
this is used by the SNS output:
5b796a0095/lib/logstash/outputs/sns.rb (L98)
2020-03-23 15:11:54 +00:00
andsel
70da647f54 Simplified if..else if in PluginFactory for Java plugins part, moved to template method pattern
Fixes #11457
2020-03-20 09:43:43 +00:00
Colin Surprenant
9126fdb7e0
support quoted plugin option key (#11688) 2020-03-16 16:58:01 -04:00
Mark Ramotowski
6486624ee7 Fixed typo in guage metric of unknown type log.
Fixes #11689
2020-03-16 09:19:28 +00:00
João Duarte
e9c9865f40
Add apache and elastic license headers to source code files (#11673)
* add license header to ruby and java files
* add license header to erb and rake files
* add license headers to gradle files
2020-03-11 11:53:38 +00:00
Karol Bucek
22d07520e4 Updated: JRuby to 9.2.11.0
- fix: deprecation warnings due JRuby 9.2.11
- fix: compat for upgraded RubyGems 3.0

Fixes #11647
2020-03-05 11:40:47 +00:00
Karol Bucek
81a339e746 Fix: use l/w match-ing (which does not depend on frames)
this was wrong on LS' end - String#match impl expects a frame (due $~)
... started failing due JRuby reducing frame usage (for blocks)

Fixes #11653
2020-03-04 16:39:10 +00:00
Karol Bucek
7b4808bd10 Fix: broken --help due sinatra 2.x upgrade
Fixes #11634
2020-03-03 21:00:10 +00:00
andsel
97d211012b Fixes JDK13's javadoc build failure
Fixes #11642
2020-03-03 09:58:37 +00:00
andsel
3695580b92 Adaptations to internal collector to send data directly to monitoring cluster Close 11573
Fixes #11541
2020-02-28 14:26:19 +00:00
andsel
fc1ae6f119 Fixes #11598 enabling the users to use completely commented config files
Fixes #11615
2020-02-27 09:43:05 +00:00
andsel
05fe308c64 Changed PluginFactory to resolve id field with environment variables docs: plugin ids variable expansion cannot use secret store
closes 10546

Fixes #11592
2020-02-20 08:57:46 +00:00
Karol Bucek
b25be7432b Review: more places for RUBY.getCurrentContext()
Fixes #11587
2020-02-10 17:41:09 +00:00
Karol Bucek
d7caa7d3cc Review: more consistent getCurrentContext()
Fixes #11587
2020-02-10 17:41:09 +00:00
Karol Bucek
f4f1a61cbb Refactor: do not keep around JRuby context reference
there's no need for this and makes code base inconsistent
... also the original intent seems no longer relevant :

was introduced at 57e7a8a56b
> allows for a massive simplification for the invocation of filters and
outputs from the Java execution

Fixes #11587
2020-02-10 17:41:09 +00:00
Colin Surprenant
8481bd0838
revert #11482 and fix redundant code generation (#11564) 2020-02-04 12:23:37 -05:00
João Duarte
5fcab21194
update licenses for master (#11549) 2020-01-31 12:43:01 +00:00
Joao Duarte
a36fdb9050 fail license report job on missing licenses
Fixes #11554
2020-01-31 12:36:41 +00:00
Colin Surprenant
0bc9fa5665
add support for pipeline.ordered setting for java execution (#11524)
reuse rubyArray for single element batches

rename preserveBatchOrder to preserveEventOrder

allow boolean and string values for the pipeline.ordered setting, reorg validation

update docs

yml typo

Update docs/static/running-logstash-command-line.asciidoc

Co-Authored-By: Karen Metts <35154725+karenzone@users.noreply.github.com>

Update docs/static/running-logstash-command-line.asciidoc

Co-Authored-By: Karen Metts <35154725+karenzone@users.noreply.github.com>

java execution specs and spec support

docs corrections per review

typo

close not shutdown

Ruby pipeline spec
2020-01-29 14:01:38 -05:00
Mike Place
020e87efc4 Add cluster_uuid setting to default config file, displaying it in Node stats HTTP API
Fixes #11106
2020-01-27 08:12:39 +00:00
andsel
6eb25173e0 Added plugin.id to fish tag log lines related to plugins
Fixes #11078
2020-01-23 15:33:31 +00:00
Joao Duarte
d579bf4de5 pin google-java-format back to 1.1 for licensing reasons
Fixes #11515
2020-01-17 13:16:54 +00:00
Colin Surprenant
b5f203ce24
start inputs only when all WorkerLoop are fully initialized (#11492) 2020-01-14 15:46:48 -05:00
Colin Surprenant
4b29112efd
cache compiled datasets (#11482) 2020-01-13 12:52:58 -05:00
Karol Bucek
1bc0aeab34 Test: try to deal with (potentially) flaky spec
Fixes #11483
2020-01-13 10:22:01 +00:00
andsel
8ccb24bf99 Changed plugin factory creation to use SourceWithMedata and not destructured line and column
Fixes #11456
2020-01-10 16:23:43 +00:00
Ioana Tagirta
c9b09bb369
Remove NodeStats#mem_payload (#9051) 2020-01-02 11:53:52 +01:00
Karol Bucek
abd4253708 Fix: SafeURI normalize and eql? to work as expected
normalize wasn't doing what was expected (URI.normalize)
also make sure SafeURI can be stored in Hash (proper eql?)

Fixes #11443
2019-12-31 10:06:40 +00:00
Karol Bucek
18af2d2c67 Test: get_thread_id (native thread being gc-d)
Fixes #11458
2019-12-30 13:04:16 +00:00
Karol Bucek
7b4e3585bc Fix: (move and) make get_thread_id "nil safe"
necessary since native (Java) thread is kept as a weak ref
so un-wrapping should deal with a potentially GCd instance

resolves GH-11450

Fixes #11458
2019-12-30 13:04:16 +00:00
Karol Bucek
b9fb9a7063 Refactor: minor one sub-stitution should do
Fixes #11458
2019-12-30 13:04:16 +00:00
Karol Bucek
850d270703 Refactor: we're always on JRuby these days
(other part of same method assume JRuby already)

Fixes #11458
2019-12-30 13:04:16 +00:00
andsel
71eed80d3c Fixes to JMH launch after upgrade to Gradle 5
Fixes #11459
2019-12-30 10:43:44 +00:00
Karol Bucek
a7cf23d9e8 Test: Java proxied QueueReadClient unwrapping
Fixes #11391
2019-12-22 22:06:08 +00:00
Karol Bucek
af3513dfba Changed: avoid JavaObject wrapping in Ruby methods
aligns the Ruby/Java returns as they happen in scripted Java
e.g. as `java.lang.Thread.new` returns a JavaProxy instance

there's really no reason to use JavaObject which always needs `to_java`
conversion to be useful (and is considered legacy in JRuby).

considered breaking change e.g. `LogStash::MemoryReadClient#read_batch`
will now return a proper JavaProxy instead of the JavaObject

Fixes #11391
2019-12-22 22:06:08 +00:00
Karol Bucek
5fd5fb2db2 Test: a lengthy real-world cloud-id test
Fixes #11435
2019-12-12 21:12:30 +00:00
Karol Bucek
3cbe7e922a Fix: handle cloud-id with an empty kibana part
fixes GH-10747

Fixes #11435
2019-12-12 21:12:30 +00:00
Colin Surprenant
702efea88f test codec against class name string to prevent class equivalence bug with a Delegator
Fixes #11401
2019-12-12 17:14:39 +00:00
andsel
71e702c16c Covered the read of logfile content with try-resource to avoid to keep a file descriptor open that later prohibited access to the process itself. Also added clean shutdown of LogManager before deleting log files used by log.
Fixes #11399
2019-12-12 16:00:18 +00:00
Joao Duarte
7f5aa186c1 remove use of thread.exclusive in plugin_metadata.rb
Fixes #11388
2019-12-10 19:43:29 +00:00