Commit graph

886 commits

Author SHA1 Message Date
Jordan Sissel
cfca18a4d7 Log an error if the require fails while loading a plugin.
Fixes #6834

Fixes #8147
2017-09-13 17:50:41 +00:00
Armin
2514c6b801 PERFORMANCE: Flatten logic for write batch instantiation
Fixes #8163
2017-09-11 09:48:43 +00:00
Armin
6ad193410b #8172 Move timestamp handling for metrics to java.lang.System.nano_time
Fixes #8180
2017-09-08 14:02:16 +00:00
Andrew Cholakian
95b9daaed9 Always log backtrace for converge / pipeline exceptions
Previously users had to enable --log.level=debug to see backtraces. These errors catch everything which can make debugging based on user reports difficult if not impossible.

Enabling --log.level=debug to solve these issues is not useful because users cannot enable it in production for rare errors.

Fixes #8156
2017-09-07 13:30:03 +00:00
Armin
0c04964cef PERFORMANCE: Remove WriteBatch, it's 100% redundant
Fixes #8155
2017-09-07 07:55:57 +00:00
Armin
5d71a2b2b6 PERFORMANCE: Move Memory Queue push loop into Java
Fixes #8140
2017-09-06 15:51:38 +00:00
Guy Boertje
456f3c30f0 until dev_utils spec_helper is changed to new method name, alias it.
Fixes #8117
2017-09-01 05:54:16 +00:00
Guy Boertje
3ce46f5473 Really fix verbose logging in tests
Fixes #8117
2017-09-01 05:54:16 +00:00
Colin Surprenant
21dbde2567 fixes for Windows platform tests/specs
use rm_rf to delete dir and shutdown pipeline after run

avoid the use of rescue nil, bad practice

do not mock close as it prevents closing the file an prevents removing it on Windows

cleanup temporary files and relax file name check for Windows

fix paths handling for Windows

flag the read file string as UTF-8 which is what we expect

fix Windows issues

ignore load_average on windows

windows safe URI

cleanup and fix file handling for windows

wait for pipeline shutdown to complete

revert to original puts

cleanups

use environment for windows platform check

fix hash path

wait for pipeline thread to complete after shutdown
2017-08-31 19:42:49 -04:00
Colin Surprenant
825da10cbb wait for pipeline thread to end before returning from the stop action 2017-08-31 19:28:01 -04:00
Guy Boertje
2fc04c9edd remove the unneeded require "logstash/util" call, has great expectations that cannot be met if required too early.
fixes #8096

Fixes #8097
2017-08-30 07:09:41 +00:00
Armin
69fc14f765 PERFORMANCE: Move Memory Queue Drain Loop to Java fully
Fixes #8083
2017-08-29 13:08:15 +00:00
Jake Landis
71fa64bf9c Metrics / Bug Fix: Protect against race condition in metrics reporting with multiple inputs.
Protect initialization of metrics obejcts with  mutex to prevent race condition. See https://github.com/elastic/logstash/issues/8011 for additional information.

Fixes #8011

Fixes #8027

Fixes #8047
2017-08-28 21:51:15 +00:00
Guy Boertje
c5f56e84d8 Modules cloud id auth (#8059)
* add newlines to generated json

* Implement cloud.id and cloud.auth settings merge to module settings

* Fixes from review plus convert to using Password for any Module Setting

* Review changes

* update modules.asciidoc to include a section on Cloud

* Capitalize Id

* remove unnecessesary require lines
2017-08-28 08:46:02 -07:00
Rob Bavey
75028f3592 Fix DLQ support for multi-pipelines
Add missing settings to the multi-pipeline settings white list to
allow DLQ to be used with multi-pipelines

Fixes #8069

Fixes #8076
2017-08-25 20:34:19 +00:00
Jake Landis
976e918d21 Fix path for local development (attempt 2)
With IntelliJ 2017.02 the gradle output changed from 'build/classes' to 'out/production/classes'. See https://youtrack.jetbrains.com/issue/IDEA-175172 for additional information.

This PR changes the local developement behavior to prefer the classes build with IntelliJ (if they exist) over the classes built directly with gradle.

Fixes #8056
2017-08-23 22:35:22 +00:00
Jake Landis
166ac8b2bd Metrics: Add custom witness for plugins, and code clean up
Remove LongGauge and DoubleGauge and Replace with NumberGauge
 - This is an unecessary distiction for Gauges and can complicate the serialization. Double/Float should serialize with decimal precision, and Jackson handles this by default.

Add custom witness to match existing Ruby API, such that existing Ruby plugins should not need changes

Remove namespace from LazyDelegatingGauge
 - It was only there for logging, but newer versions (not here) will completely remove the concept of namespace.

Part of #7788

Fixes #8053
2017-08-23 17:09:01 +00:00
Suyog Rao
eaee1e9289 Remove metrics max bucket setting (#8010)
PR #7451 introduced a way to set metrics max bucket settign in Kibana. This ain't good because
it can destabilize Kibana and ES as well
2017-08-16 16:17:25 -07:00
Andrew Cholakian
0b03e2f873 Revert "create source_loader after logging setup"
This commit broke the plugin contract with plugins like xpack and caused many issues there. Reverted until it can be better handled.

This reverts commit 92cdbe2dbd.

Fixes #7995
2017-08-14 23:11:32 +00:00
Armin
240048b88d #7990 bring back batch acking for PQ
Fixes #7991
2017-08-14 17:35:11 +00:00
Joao Duarte
92cdbe2dbd create source_loader after logging setup
Fixes #7970
2017-08-11 13:18:32 +00:00
Aaron Mildenstein
fb7f455f1c Add --setup phase for modules (#7965)
* Add --setup phase for modules

Without the `--setup` flag, the index template for elasticsearch, and the index-pattern, saved searches, visualizations, and dashboards for Kibana will not be published.

fixes #7959
2017-08-09 16:50:56 -06:00
Suyog Rao
203d419762 Add log statement about modules that are run
Fixes #7963
2017-08-09 21:48:11 +00:00
Andrew Cholakian
9f1830d119 Generate codec IDs randomly
Fixes #7930
2017-08-09 15:42:12 +00:00
Andrew Cholakian
0e7cba783e Use LIR to generate vertex plugin IDs
In this patch we unify the IDs reported by LIR with those generated using config_ast.rb

This is a temporary fix until LIR execution is built. It relies on the fact that currently both LIR and config_ast.rb only operate on a single concatenated string of configurations. In the future LIR will compile different files separately, then merge their IRs. For now, however, this solution will hold.

This change also exposes the :id attribute of FilterDelegator to bring it to parity with OutputDelegator and InputDelegator which is required for testing purposes.

Fixes #7930
2017-08-09 15:42:12 +00:00
Joao Duarte
7c514689bb fix debug logging messages with thread values
Fixes #7954
2017-08-09 14:27:32 +00:00
Joao Duarte
01110a9b48 cleanup force shutdown code
Fixes #7955
2017-08-09 14:25:47 +00:00
Joao Duarte
79ca3b9996 force exit on double SIGINT
Fixes #7918
2017-08-09 08:29:36 +00:00
Joao Duarte
06b63e28f9 prevent force shutdown to be blocked by with_pipelines
if force shutdown uses `with_pipelines` then it will never acquire the
`@pipelines_lock` from `shutdown_pipelines` until it terminates.

This PR makes force shutdown use `@pipelines` directly, fixing it.

Fixes #7918
2017-08-09 08:29:36 +00:00
Armin
5e2becbe40 Revert "PERFORMANCE: Generate more optimal filter_func"
This reverts commit 99a21bc5c1.

Fixes #7943
2017-08-08 19:17:10 +00:00
Armin
99a21bc5c1 PERFORMANCE: Generate more optimal filter_func
Fixes #7924
2017-08-08 15:41:21 +00:00
Jake Landis
fe7cd1fe07 Metrics: Clean up, remove and deprecate in support of pure Java based metrics.
This changes the following:
* DRY Gauge metrics
* Remove Namespace from metrics API (never used, not needed)
* Introduce a 'dirty' flag. This will be used to assist with serialization (in a later commit).
* Change getKey to getName in the API
* Deprecate (only from Java) the future non-supported Guage types.
* Remove NumericGauge and replace with LongGuage and DoubleGuage.
* Bring code coverage back to 100%

Note this change is internally non-passive, however the Java metrics API (recently introduced in 6.0) has not been advertised as a re-usable asset.

Changes in support of #7788

Fixes #7872
2017-08-08 02:04:42 +00:00
Armin
324da31837 PERFORMANCE: Cache some metrics
Fixes #7919
2017-08-07 21:11:52 +00:00
Armin
e6424f3e97 PERFORMANCE: Don't use slow metric clocks where avoidable
Fixes #7919
2017-08-07 21:11:52 +00:00
Armin
e10af362be PERFORMANCE: Cache some metrics
Fixes #7919
2017-08-07 21:11:51 +00:00
Jake Landis
21c178cbe5 Revert "Performance: Use RubyArray.hash for metric's fast lookup key"
This reverts commit 9b4d7c51ae (master)

Fixes: #7925

Fixes #7927
2017-08-07 18:59:18 +00:00
Armin
ce4b854b3d PERFORMANCE: remove redundant return call
Fixes #7902
2017-08-04 10:40:53 +00:00
Armin
a0fea76370 MINOR: Improve batch iteration
Fixes #7837
2017-08-03 19:19:43 +00:00
Rob Bavey
64388f34a5 Remove closed DeadLetterQueueWriters from 'Factory'
Fix bug where reloading a pipeline would close the DLQWriter, but
leave the closed version cached in the factory object, stopping
the reloaded pipeline from being able to write to the dead letter
queue.

Fixes #7840
2017-08-02 20:04:17 +00:00
Joao Duarte
eb622e507b fix a few issues with pipeline source loading
- allow empty config.string
- move all config autocompletion logic to the ConfigStringLoader
- gracefully handle absense of files in path.config
- ensure original_settings are restored in multi_local source after PipelineConfig creation

Fixes #7866
2017-08-02 12:33:56 +00:00
Armin
46dfcad215 #7754 ensure we actually test periodic flushing in specs + make flush work without data input data
Fixes #7863
2017-08-02 12:24:54 +00:00
Jake Landis
5d2c803847 Metrics: Cache the hostname in class variable Fixes: #7857
Fixes #7858
2017-08-01 18:40:56 +00:00
Armin
7a73c5fd7b #7851 allow UTF-8 in config file
Fixes #7852
2017-07-31 11:54:27 +00:00
Jake Landis
9b4d7c51ae Performance: Use RubyArray.hash for metric's fast lookup key
The existing implementation uses the RubyArray as the key for the fast lookup Map. Under the covers the Map implementation is comparing equal operators (many times per event), and JRuby Array equals operator is VERY expensive since it literally walks each value of array for each equality. Based on profiling via YourKit, the JRuby Array equals operator is a very hot method consuming upto 60% of sampled CPU cycles while under high load (and no other CPU dominators).

This change is to use the .hash value of the JRuby Array as the key of the fast lookup Map. This implementation still calls .hash for each and every call, which is also expensive, since it also walks the arrays to compute the hash. However, the equality check of the hash value is very fast, and net gain is significant. Upto a 15% increase of throughput.

Fixes #7772

Fixes #7798
2017-07-29 20:57:47 +00:00
Armin
8d216a31c0 MINOR: Flatten Pipeline Worker Threads
Fixes #7836
2017-07-28 12:38:57 +00:00
Armin
6fb0f7db21 #7690 Fix needless waiting by using an ArrayBlockingQueue
Fixes #7691
2017-07-28 06:08:29 +00:00
Jake Landis
0e342f3d9e Revert "Fix path for local development"
This reverts commit ae21bf374c.

This change was only applicable to IntelliJ, see https://youtrack.jetbrains.com/issue/IDEA-175172

Fixes #7806
2017-07-25 19:22:37 +00:00
Armin
dfb423a60f PERFORMANCE: Remve needless locking on batch reads
Fixes #7623
2017-07-25 18:33:09 +00:00
Suyog Rao
b6bcbda8e4 bump master version to 7.0.0-alpha1 (#7791) 2017-07-24 11:51:40 -07:00
Jake Landis
ae21bf374c Fix path for local development
Fixes #7786
2017-07-24 16:55:26 +00:00