Commit graph

538 commits

Author SHA1 Message Date
Armin
6352001177 BUG: Fix incorrect type handling between Java pipeline and Ruby pipeline
Fixes #9671
2018-05-30 11:24:14 +00:00
Armin
417afd1c5a JAVAFICATION: Ported ShutdownWatcher to Java
Fixes #9670
2018-05-29 15:57:13 +00:00
Tamara Braun
7d6acf1799 Port HooksRegistry to Java
Fixes #9665
2018-05-28 06:45:51 +00:00
Dan Hermann
75684020e5 port SecretStore to Java
Fixes #9662
2018-05-25 20:47:34 +00:00
Armin
8103bff757 JAVAFICATION: Port RubyTimestamp test to JUnit
Fixes #9661
2018-05-25 12:07:50 +00:00
Andrew Cholakian
2c4cfeb990 Fix converge spec that didn't wait for agent to converge
This spec had a race, the start_agent invocation this depends on doesn't wait
until the state has converged at least once.

An example failure is here: https://logstash-ci.elastic.co/job/elastic+logstash+6.x+multijob-unix-compatibility/os=debian/121/console

The message was:

```
22:34:08     Failures:
22:34:08
22:34:08       1) LogStash::Agent Agent execute options when `config.reload.automatic` is set to`FALSE` and successfully load the config converge only once
22:34:08          Failure/Error: expect(source_loader.fetch_count).to eq(1)
22:34:08
22:34:08            expected: 1
22:34:08                 got: 0
22:34:08
22:34:08            (compared using ==)
22:34:08          # /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/logstash-core/spec/logstash/agent/converge_spec.rb:120:in `block in (root)'
22:34:08          # /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/spec/spec_helper.rb:50:in `block in /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/spec/spec_helper.rb'
22:34:08          # /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/spec/spec_helper.rb:43:in `block in /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/spec/spec_helper.rb'
22:34:08          # /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/vendor/bundle/jruby/2.3.0/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `block in /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/vendor/bundle/jruby/2.3.0/gems/rspec-wait-0.0.9/lib/rspec/wait.rb'
22:34:08          # /var/lib/jenkins/workspace/elastic+logstash+6.x+multijob-unix-compatibility/os/debian/lib/bootstrap/rspec.rb:13:in `<main>'
22:34:08
22:34:08     Finished in 5 minutes 27 seconds (files took 8.75 seconds to load)
22:34:08     2877 examples, 1 failure, 5 pending
```

Fixes #9643
2018-05-23 00:35:30 +00:00
Andrew Cholakian
4c767c059e Wait longer and more reliably for agent specs.
These agent specs occasionally time out. This removes our use of the unreliable Timeout gem
and also lengthens the waiting period before failure

Fixes #9639
2018-05-22 20:58:19 +00:00
Andrew Cholakian
fb16b7b984 Shutdown agent after pipelines
I actually am not sure if this matters, but it seems like it won't hurt,
and is a more sane ordering of things. I have a suspicion that in some cases
the agents get stuck and don't shut down because of this ordering, but I can't
prove it

Fixes #9628
2018-05-22 14:51:17 +00:00
Armin
a9217255a7 JAVAFICATION: Port pipeline reporter to Java
Fixes #9631
2018-05-22 12:47:13 +00:00
Armin
0dc5c4af64 JAVAFICATION: Port EventDispatcher to Java
Fixes #9633
2018-05-22 12:19:45 +00:00
Joao Duarte
1fd34822b9 add custom clamp appender to fix path.plugin
the path.plugin setting is a multi valued option, clamp uses
an implicit appender method for adding values to an array.
because we patch Clamp to use LogStash::SETTINGS underneath,
we must tune both reading/writing and appending values.
This last wasn't being customized, so this path redefines the
append_method method to add an element to a setting's value

Fixes #8887
2018-05-18 08:58:31 +00:00
Armin
c0abde28a6 JAVAFICATION: Port Loggable Module to Java
Fixes #9587
2018-05-17 12:17:31 +00:00
Shaunak Kashyap
6a895a417e
Revert "[X-Pack Monitoring] Report pipeline protocol (#9516)" (#9568)
This reverts commit dc168a10b5.
2018-05-14 07:51:54 -07:00
Shaunak Kashyap
dc168a10b5
[X-Pack Monitoring] Report pipeline protocol (#9516)
* Add getter for pipeline protocol

* Refactoring: extract bare string into constant

* Report pipeline protocol to Monitoring as part of logstash_state docs

* Report all protocols, not just first one

* Refactoring: rename constant to be more descriptive

* De-dupe protocols list before sending to monitoring

* Checking for single protocol

* Adding missing require

This was being masked and was uncovered when this test suite was run by itself

* Raising error if multiple protocols are specified

* Adding back comma
2018-05-11 13:30:42 -07:00
Armin
d38e117246 JAVAFICATION: Port Plugin Factory to Java
Fixes #9491
2018-04-26 17:21:31 +00:00
Armin
1cc89532d7 JAVAFICATION: Port Ruby Error Definitions to Java
Fixes #9489
2018-04-26 12:44:00 +00:00
Armin
6179c9662a JAVAFICATION: Port namespace definitions to Java
Fixes #9490
2018-04-26 12:00:15 +00:00
Armin
ccb4daf518 JAVAFICATION: Port java integration hacks to Java
Fixes #9486
2018-04-26 11:57:41 +00:00
Armin
fb01e68185 remove unused file
Fixes #9413
2018-04-25 12:48:14 +00:00
Armin
4877f5370e JAVAFICATION: ExecutionContext ported to Java
Fixes #9482
2018-04-25 12:27:25 +00:00
Armin
a9bf332103 DLQ Ruby wrapper ported to Java
Fixes #9422
2018-04-25 09:17:04 +00:00
Armin
9c581a11fc JAVAFICATION: Port metric.rb to Java
Fixes #9418
2018-04-25 08:35:15 +00:00
Tamara Braun
63b43f9451 ported buftok.rb to java
Fixes #9414
2018-04-25 07:44:22 +00:00
Armin
155801520c JAVAFICATION+PERFORMANCE: Port OutputDelegatorStrategies to Java
Fixes #9410
2018-04-20 16:37:25 +00:00
Armin
7a861f2b48 #8888: Fix Logstash::Util.deep_clone for Timestamp
Fixes #9405
2018-04-19 16:26:18 +00:00
Armin
57e7a8a56b CLEANUP+JAVAFICATION+PERFORMANCE: Keep hard reference to Ruby Thread Context, simplify code accordingly
Fixes #9401
2018-04-18 13:30:57 +00:00
Armin
4afc94fb6a PERFORMANCE+JAVAFICATION: Java Filter Delegator
Fixes #9380
2018-04-16 12:10:12 +00:00
Armin
d691c5cbeb #9361 skip failing test for the time being
Fixes #9374
2018-04-13 14:39:06 +00:00
Andrew Cholakian
a1c0e417e5 Support for inter-pipeline comms with a new pipeline input/output
This also makes the load / reload cycle of pipelines threadsafe
and concurrent in the Agent class.

Fixes #9225
2018-04-10 23:48:58 +00:00
Armin
d483a01600 PERFORMANCE+JAVAFICATION: Ported outputdelegator to Java
Fixes #9340
2018-04-09 13:38:15 +00:00
Armin
7cea457eef CLEANUP: Dry up filter wrapper setup
Fixes #9337
2018-04-08 16:49:56 +00:00
Armin
5553f5886a MINOR: Cleanup dead code in PQ, fix unsafe resource handling in PQ pages
Fixes #9332
2018-04-06 12:26:17 +00:00
Armin
8f40c594f3 MINOR: Remove ununsed queue method
Fixes #9329
2018-04-05 12:08:35 +00:00
Andrew Cholakian
3622d9aee1 Remove unnecessary require for reporter specs from pipeline_specs
Using require this way breaks the ability to run a specific spec with rspec since the require triggers
them to run. This patch fixes that.

Fixes #9291
2018-03-26 22:18:00 +00:00
Andrew Cholakian
cc72641447 Fix race condition in shutdown of pipelines
Prior to this a single worker could slurp down multiple shutdown messages, this prevents that from happening by using a flag that can't be overconsumed.

Fixes #9285
2018-03-26 17:01:34 -05:00
Armin
8b89a139a6 BUG: Fix Java execution not handling nil return from flush invocations
Fixes #9289
2018-03-26 14:47:17 +00:00
Guy Boertje
daeb4f4071
Fix: Tags collision with user-defined tags and built-in tags (#9166)
* return early if there's no remove_tag work todo.

* add spec to prevent regression

* fix typo as per review
2018-03-20 16:50:57 +13:00
Joao Duarte
b0d1caeb6e tolerate different types for ssl.enabled module options
Fixes #8600
2018-03-19 15:34:15 +00:00
Armin
5ab31cc5a2 BUG: Fix input shutting down prematurely
Fixes #9190
2018-02-22 17:52:20 +00:00
Dan Hermann
f024216db8
Common read clients and batches (#9111) 2018-02-15 10:05:32 -06:00
Dan Hermann
43ff4ba19f JAVAFICATION: Java wrapped acked queue
Fixes #9080
2018-02-07 16:06:37 +00:00
Colin Surprenant
da15c5b3b7 verify available disk space for PQ #8978. This code was originally written by @original-brownbear for #6998
localize error message

refactor & simplify hasFreeSpace() per @yaauie suggestion

reformat for clarity
2018-01-24 10:51:41 -05:00
Armin
28e7467f9f Remove in memory serialized queue Ruby wrappers
Fixes #8999
2018-01-22 23:22:15 +00:00
Armin
3b2ff7b2e3 Remove mock checkpoint io
Fixes #8999
2018-01-22 23:22:15 +00:00
Dan Hermann
dee283a270 JAVAFICATION: Port remainder of synchronous queue to Java (#8991)
* JAVAFICATION: Port WrappedSynchronousQueue.WriteClient and WrappedSynchronousQueue to Java
2018-01-22 10:24:32 +01:00
Dan Hermann
d7cd3b29cc JAVAFICATION: Port memory queue read client to Java. Also uses Java counters more efficiently per #8985. Rough throughput testing with generator input shows 15-20% increase due mainly to the metrics improvements.
Fixes #8987
2018-01-19 14:11:18 +00:00
Armin
edd8844670 JAVAFICATION: Port wrapped write client to Java
Fixes #8977
2018-01-18 13:10:32 +00:00
Armin
9bead1c93d MINOR: Remove unnecessary logger fields from output delegator strategy and plugin factory
Fixes #8964
2018-01-17 11:38:06 +00:00
Armin
879454af3b JAVAFICATION: Remove Ruby method_missing deprecation on Event
Fixes #8970
2018-01-17 11:28:49 +00:00
Armin
4644ece0b6 JAVAFICATION: Move remaining Logstash::Timestamp Ruby Code to Java
Fixes #8969
2018-01-17 11:27:15 +00:00