Commit graph

870 commits

Author SHA1 Message Date
Rob Bavey
1e4e0dbb8b Removed unnecessary Password type check
Fixes #8226
2017-09-13 17:57:11 +00:00
Rob Bavey
70844f02d3 Fix password support for Modules
Module settings should use value field for setting password in
Elasticsearch output config, not the implicit, obfuscated to_s value.

Fixes #8226
2017-09-13 17:57:11 +00:00
Jordan Sissel
acc6d50ab8 Log an error if the require fails while loading a plugin.
Fixes #6834

Fixes #8147
2017-09-13 17:50:41 +00:00
Armin
d1b053cf9b MINOR: Clean up some dead code around PQ + PQ Benchmarks
Fixes #8232
2017-09-13 13:19:19 +00:00
Armin
0ce82557d7 MINOR: Some DLQ Stability Fixes
Fixes #8171
2017-09-12 21:57:31 +00:00
Armin
fbeb49cf39 MINOR: Remove redundant throws in PipelineIR path
Fixes #8212
2017-09-12 13:42:05 +00:00
Armin
b66562342d MINOR: Cleanup JAVAC warnings around our Jruby Extensions
Fixes #8201
2017-09-12 08:05:11 +00:00
Armin
cfddcf61bc #8128 Fix timestamp not getting properly deserialized under custom key
Fixes #8130
2017-09-11 17:14:03 +00:00
Armin
89af438309 PERFORMANCE: Flatten logic for write batch instantiation
Fixes #8163
2017-09-11 09:48:43 +00:00
Armin
baf1c8f03d PERFORMANCE: Remove SymbolBiValue
Fixes #8189
2017-09-11 08:15:47 +00:00
Armin
a252ec6178 #8172 Move timestamp handling for metrics to java.lang.System.nano_time
Fixes #8180
2017-09-08 14:02:16 +00:00
Armin
029be86c6d PERFORMANCE: Bring Javafier design in line with Rubyfier and Valuefier
Fixes #8158
2017-09-08 13:54:38 +00:00
Armin
0467f0dbda MINOR: Cleanup all dead imports in Java
Fixes #8183
2017-09-08 13:33:03 +00:00
Colin Surprenant
5d8c79ad4f add jruby-runtime-override in versions.yml, update jruby version to 9.1.13.0, relax jruby-ssl version to >= 0.9.20
add option for different jruby runtime version

relax jruby-ssl version to >= 0.9.20

rename jruby-runtime to jruby-runtime-override and add sha1 IGNORE option

use JRuby released version of 9.1.13.0
2017-09-07 12:17:32 -04:00
Andrew Cholakian
ffff418839 Always log backtrace for converge / pipeline exceptions
Previously users had to enable --log.level=debug to see backtraces. These errors catch everything which can make debugging based on user reports difficult if not impossible.

Enabling --log.level=debug to solve these issues is not useful because users cannot enable it in production for rare errors.

Fixes #8156
2017-09-07 13:30:03 +00:00
Armin
340e879379 PERFORMANCE: Remove WriteBatch, it's 100% redundant
Fixes #8155
2017-09-07 07:55:57 +00:00
Armin
d82ba45d90 #8148 properly join threads in webserver_spec
Fixes #8150
2017-09-07 07:03:40 +00:00
Armin
c7c98b6744 PERFORMANCE: Cleanup Rubyfier
Fixes #8146
2017-09-06 16:02:48 +00:00
Armin
3f57ea10dc MINOR: Remove dead if branch from Vertex class
Fixes #8136
2017-09-06 15:52:40 +00:00
Armin
d66e0c507c PERFORMANCE: Move Memory Queue push loop into Java
Fixes #8140
2017-09-06 15:51:38 +00:00
Armin
d5530a42c4 #7712 integer and long bivalues removed
Fixes #8107
2017-09-05 18:14:56 +00:00
Armin
78689716b7 MINOR: Fix a few more compiler warnings
Fixes #8137
2017-09-05 17:13:06 +00:00
Guy Boertje
efa7f71584 until dev_utils spec_helper is changed to new method name, alias it.
Fixes #8117
2017-09-01 05:54:17 +00:00
Guy Boertje
dde2730d10 Really fix verbose logging in tests
Fixes #8117
2017-09-01 05:54:16 +00:00
Colin Surprenant
f5480128f7 post backport fix of windows fixes for missing timeout 2017-08-31 20:39:55 -04:00
Colin Surprenant
d382248caf fixes for Windows platform tests/specs
use rm_rf to delete dir and shutdown pipeline after run

avoid the use of rescue nil, bad practice

do not mock close as it prevents closing the file an prevents removing it on Windows

cleanup temporary files and relax file name check for Windows

fix paths handling for Windows

flag the read file string as UTF-8 which is what we expect

fix Windows issues

ignore load_average on windows

windows safe URI

cleanup and fix file handling for windows

wait for pipeline shutdown to complete

revert to original puts

cleanups

use environment for windows platform check

fix hash path

wait for pipeline thread to complete after shutdown
2017-08-31 20:35:52 -04:00
Colin Surprenant
d7eec97a80 wait for pipeline thread to end before returning from the stop action 2017-08-31 20:33:20 -04:00
Armin
e23780d042 PERFORMANCE: Much faster ConvertedMap
* Use IdentityHashMap since we intern all possible keys anyways and can look them up more efficiently than `String.intern()` from our PathCache
* Use the PathCache to never have to instantiate new `String` when converting a RubyHash

Fixes #8104
2017-08-30 16:55:49 +00:00
Armin
7f3df27fa8 MINOR: Cleanup Valuefier to have the same efficient approach that is already used by BiValues
Fixes #8103
2017-08-30 12:29:28 +00:00
Armin
a658c46b43 MINOR: Use more optimal loop in draining QUeue
Fixes #8101
2017-08-30 07:39:43 +00:00
Guy Boertje
848e1c1aef remove the unneeded require "logstash/util" call, has great expectations that cannot be met if required too early.
fixes #8096

Fixes #8097
2017-08-30 07:09:42 +00:00
Armin
6ba9720052 #7712 & #7945 BiValue Improvements
* Turn Enum into HashMap for direct lookup
* Use stateless Lambdas instead of named classes as converter functions for better performance
* Remove redundant instanceof checks by splitting existing converter functions between Java and Ruby version
* Use most restrictive instanceof check possible for Ruby converters for performance improvements

Fixes #8087
2017-08-29 19:21:43 +00:00
Armin
c7119fa608 PERFORMANCE: Move Memory Queue Drain Loop to Java fully
Fixes #8083
2017-08-29 13:08:15 +00:00
Jake Landis
76aa2bbc76 Metrics / Bug Fix: Protect against race condition in metrics reporting with multiple inputs.
Protect initialization of metrics obejcts with  mutex to prevent race condition. See https://github.com/elastic/logstash/issues/8011 for additional information.

Fixes #8011

Fixes #8027

Fixes #8047
2017-08-28 21:51:14 +00:00
Guy Boertje
e2b2e7d979 Modules cloud id auth (#8059)
* add newlines to generated json

* Implement cloud.id and cloud.auth settings merge to module settings

* Fixes from review plus convert to using Password for any Module Setting

* Review changes

* update modules.asciidoc to include a section on Cloud

* Capitalize Id

* remove unnecessesary require lines
2017-08-28 08:51:21 -07:00
Armin
9f389c494a MINOR: Cleanup magic string for LS JRuby Module name
Fixes #8084
2017-08-28 14:48:26 +00:00
Armin
6f786144a8 MINOR: Fix a lot of new compiler warnings in Metrics code
Fixes #8086
2017-08-28 14:20:41 +00:00
Rob Bavey
ccb3b7978f Fix DLQ support for multi-pipelines
Add missing settings to the multi-pipeline settings white list to
allow DLQ to be used with multi-pipelines

Fixes #8069

Fixes #8076
2017-08-25 20:34:19 +00:00
Colin Surprenant
b50bcd17e0 relax time gap constrain in expectation 2017-08-25 13:13:27 -04:00
Jake Landis
dbda826060 Metrics: move Witness class to better package names
Part of #7788

Fixes #8065
2017-08-24 20:25:23 +00:00
Jake Landis
b7fef5d837 Fix path for local development (attempt 2)
With IntelliJ 2017.02 the gradle output changed from 'build/classes' to 'out/production/classes'. See https://youtrack.jetbrains.com/issue/IDEA-175172 for additional information.

This PR changes the local developement behavior to prefer the classes build with IntelliJ (if they exist) over the classes built directly with gradle.

Fixes #8056
2017-08-23 22:35:22 +00:00
Andrew Cholakian
9449b1909a Use rspec wait to fix flakey converge specs
Some converge specs have races, assuming that a running Agent means that pipelines have started running.
This patch lets us wait a while in the event of a race. One such failure is here: https://logstash-ci.elastic.co/job/elastic+logstash+master+multijob-unix-compatibility/os=centos/72/console

Additionally, this removes a harmless double invocation in these specs

Fixes #7912
2017-08-23 21:56:48 +00:00
Jake Landis
9b961c46d7 Metrics: Add custom witness for plugins, and code clean up
Remove LongGauge and DoubleGauge and Replace with NumberGauge
 - This is an unecessary distiction for Gauges and can complicate the serialization. Double/Float should serialize with decimal precision, and Jackson handles this by default.

Add custom witness to match existing Ruby API, such that existing Ruby plugins should not need changes

Remove namespace from LazyDelegatingGauge
 - It was only there for logging, but newer versions (not here) will completely remove the concept of namespace.

Part of #7788

Fixes #8053
2017-08-23 17:09:01 +00:00
Jake Landis
17c9d668cd Metrics: More consistent serialization, Witness support for DLQ, PQ, and Java clean up
* Additional consistent serialization (don't check for null value, continuation of https://github.com/elastic/logstash/pull/8009 / 5d83a71aa4)
* Add support for Dead Letter Queue
* Add support for persitent Queue
* Scope clean up

Part of #7788

Fixes #8023
2017-08-22 15:52:25 +00:00
Rob Bavey
f08a628b5c Moved test from RecordIOWriterTest to RecordIOReaderTest
Moved testReadWhileWriteAcrossBoundary to RecordIOReader as it verifies a fix in the
 RecordIOReader class

Fixes #8024
2017-08-18 19:15:05 +00:00
Rob Bavey
9f64fccef9 DLQ: Fix reading while writing boundary issues
Fixes an issue where if the DLQ is actively being read and written to, the consumeBlock method would
write data to the position where the data was last read from, rather than written to, leading to
plugin crashes.

Fixes #8024
2017-08-18 19:15:04 +00:00
Jake Landis
5b71afcdb9 Metrics: Consistent serialization
This change allows for the serialized output of the metrics to be consistent w/r/t the un-initialized values. Note - the code presented here is not the current code in use, rather it is the code staged, but unused to replace the metrics store / namespace implementation.

Fixes #7885 (for a future release)

Fixes #8009
2017-08-17 23:35:50 +00:00
Armin
62cbb0613e Performance: Remove BooleanBiValue
Fixes #8015
2017-08-17 17:04:41 +00:00
Armin
1066d61dae #8016 remove Max bucket setting from specs to bring them in line with #8010
Fixes #8017
2017-08-17 15:07:15 +00:00
Suyog Rao
4dc530ddbb Remove metrics max bucket setting
PR #7451 introduced a way to set metrics max bucket settign in Kibana. This ain't good because
it can destabilize Kibana and ES as well

Fixes #8010
2017-08-16 23:17:53 +00:00