Commit graph

1649 commits

Author SHA1 Message Date
Joao Duarte
06b63e28f9 prevent force shutdown to be blocked by with_pipelines
if force shutdown uses `with_pipelines` then it will never acquire the
`@pipelines_lock` from `shutdown_pipelines` until it terminates.

This PR makes force shutdown use `@pipelines` directly, fixing it.

Fixes #7918
2017-08-09 08:29:36 +00:00
Armin
14e6d78cef PERFORMANCE: Remove TimestampBiValue
Fixes #7937
2017-08-09 06:44:19 +00:00
Armin
5e2becbe40 Revert "PERFORMANCE: Generate more optimal filter_func"
This reverts commit 99a21bc5c1.

Fixes #7943
2017-08-08 19:17:10 +00:00
Jake Landis
793275c8b5 Performance: remove volatile from hot code
Fixes #7935

Fixes #7942
2017-08-08 16:33:21 +00:00
Armin
c7028199fd MINOR: Save a few null checks in Valuefier
Fixes #7938
2017-08-08 16:27:53 +00:00
Armin
ec8015225b MINOR: Remove useless test object
Fixes #7939
2017-08-08 16:04:09 +00:00
Armin
99a21bc5c1 PERFORMANCE: Generate more optimal filter_func
Fixes #7924
2017-08-08 15:41:21 +00:00
Andrew Cholakian
9720dfba73 Shorten Vertex generated IDs
Currently generated Vertex IDs are unwieldy concatenations of two sha256 hashes. This is just a pain to deal with.

This patch hashes that concatenation instead, yielding something far easier to deal with.

Fixes #7931
2017-08-08 15:34:41 +00:00
Jake Landis
fe7cd1fe07 Metrics: Clean up, remove and deprecate in support of pure Java based metrics.
This changes the following:
* DRY Gauge metrics
* Remove Namespace from metrics API (never used, not needed)
* Introduce a 'dirty' flag. This will be used to assist with serialization (in a later commit).
* Change getKey to getName in the API
* Deprecate (only from Java) the future non-supported Guage types.
* Remove NumericGauge and replace with LongGuage and DoubleGuage.
* Bring code coverage back to 100%

Note this change is internally non-passive, however the Java metrics API (recently introduced in 6.0) has not been advertised as a re-usable asset.

Changes in support of #7788

Fixes #7872
2017-08-08 02:04:42 +00:00
Armin
324da31837 PERFORMANCE: Cache some metrics
Fixes #7919
2017-08-07 21:11:52 +00:00
Armin
e6424f3e97 PERFORMANCE: Don't use slow metric clocks where avoidable
Fixes #7919
2017-08-07 21:11:52 +00:00
Armin
e10af362be PERFORMANCE: Cache some metrics
Fixes #7919
2017-08-07 21:11:51 +00:00
Jake Landis
21c178cbe5 Revert "Performance: Use RubyArray.hash for metric's fast lookup key"
This reverts commit 9b4d7c51ae (master)

Fixes: #7925

Fixes #7927
2017-08-07 18:59:18 +00:00
Armin
e48dc93df9 PERFORMANCE: Removed DoubleBiValue and FloatBiValue
Fixes #7917
2017-08-07 16:53:43 +00:00
Armin
8153411c15 CLEANUP: Remove unused TimeBiValue
Fixes #7920
2017-08-07 08:28:00 +00:00
Armin
0ea2be9732 PERFORMANCE: Cleanup Redundant Timestamp instantiations
Fixes #7916
2017-08-06 07:29:53 +00:00
Armin
532a0304e9 PERFORMANCE: Better hashCode and equals for RubyEvent
Fixes #7902
2017-08-04 10:40:53 +00:00
Armin
ce4b854b3d PERFORMANCE: remove redundant return call
Fixes #7902
2017-08-04 10:40:53 +00:00
Armin
3f99398bfc PERFORMANCE: Completely drop StringBiValue
Fixes #7854
2017-08-04 06:30:29 +00:00
Armin
019fbcd091 PERFORMANCE: Constant RubyHash Visitor
Fixes #7898
2017-08-04 05:12:01 +00:00
Armin
8586f4aac2 PERFORMANCE: Intern string keys from FieldReferences
Fixes #7894
2017-08-03 20:12:16 +00:00
Armin
a0fea76370 MINOR: Improve batch iteration
Fixes #7837
2017-08-03 19:19:43 +00:00
Armin
7968123d0f PEFORMANCE: Efficient Field get/set/del/includes on Event
Fixes #7889
2017-08-03 14:01:25 +00:00
Rob Bavey
710ac1505c Add additional unit test and more broad checking
Fixes #7871
2017-08-03 01:46:43 +00:00
Rob Bavey
b5b1ca3c15 Fix handling of last event in block
The last event in a block was being incorrectly handled in such a way that caused unexpected
behavior depending on the contents of the final event, it could cause plugins to fail, or
Logstash to fail with an OutOfMemoryError

Fixes #7871
2017-08-03 01:46:43 +00:00
Colin Surprenant
73f00842fa handle fully acked zero byte page at queue open (#7810) 2017-08-02 13:10:45 -07:00
Rob Bavey
64388f34a5 Remove closed DeadLetterQueueWriters from 'Factory'
Fix bug where reloading a pipeline would close the DLQWriter, but
leave the closed version cached in the factory object, stopping
the reloaded pipeline from being able to write to the dead letter
queue.

Fixes #7840
2017-08-02 20:04:17 +00:00
Rob Bavey
526b28159b Add check to stop messages looping through DLQ
Add check to stop events from being written to the DLQ that
have previously been written to the DLQ

Fixes #7838
2017-08-02 18:54:08 +00:00
Armin
caef6fdd6f MINOR: Remove Redundant BiValue Interface and Rename Abstract Class Accordingly
Fixes #7850
2017-08-02 18:13:09 +00:00
Armin
23877c0b7c PERFORMANCE: Remove lut Cache in Accessors
Fixes #7839
2017-08-02 18:11:27 +00:00
Joao Duarte
eb622e507b fix a few issues with pipeline source loading
- allow empty config.string
- move all config autocompletion logic to the ConfigStringLoader
- gracefully handle absense of files in path.config
- ensure original_settings are restored in multi_local source after PipelineConfig creation

Fixes #7866
2017-08-02 12:33:56 +00:00
Armin
46dfcad215 #7754 ensure we actually test periodic flushing in specs + make flush work without data input data
Fixes #7863
2017-08-02 12:24:54 +00:00
Jake Landis
5d2c803847 Metrics: Cache the hostname in class variable Fixes: #7857
Fixes #7858
2017-08-01 18:40:56 +00:00
Armin
7a73c5fd7b #7851 allow UTF-8 in config file
Fixes #7852
2017-07-31 11:54:27 +00:00
Armin
8a28d2fe84 #7817 Accessors should only handle ConvertedMap and ConvertedList as Collection, non-scalar types
Fixes #7823
2017-07-31 09:40:45 +00:00
Armin
daeab90822 PERFORMANCE: Move null guard out of Javafier.deep
Fixes #7849
2017-07-31 09:26:27 +00:00
Armin
57e2fa6d10 MINOR: Cleanup multiple ObjectMappers being used
Fixes #7846
2017-07-30 18:48:08 +00:00
Armin
19f3861545 PERFORMANCE: Correctly size pathcache backing map
Fixes #7848
2017-07-30 17:58:02 +00:00
Jake Landis
9d6ecf92ac Test Fix: Update test to properly reflect the Java Timestamp. Fixes #7843
Fixes #7845
2017-07-30 16:45:43 +00:00
Jake Landis
9b4d7c51ae Performance: Use RubyArray.hash for metric's fast lookup key
The existing implementation uses the RubyArray as the key for the fast lookup Map. Under the covers the Map implementation is comparing equal operators (many times per event), and JRuby Array equals operator is VERY expensive since it literally walks each value of array for each equality. Based on profiling via YourKit, the JRuby Array equals operator is a very hot method consuming upto 60% of sampled CPU cycles while under high load (and no other CPU dominators).

This change is to use the .hash value of the JRuby Array as the key of the fast lookup Map. This implementation still calls .hash for each and every call, which is also expensive, since it also walks the arrays to compute the hash. However, the equality check of the hash value is very fast, and net gain is significant. Upto a 15% increase of throughput.

Fixes #7772

Fixes #7798
2017-07-29 20:57:47 +00:00
Jake Landis
e7651789cc Metrics: Fix RubyTimestamp serialization
This change requires setting the and retrieving the value with different types. The Gauge interface has been update to allow for setting / getting using different types.

Fixes #7719

Fixes #7720
2017-07-29 20:09:59 +00:00
Armin
0004f9cfb2 MINOR: Cleanup redundant code in RubyEvent
Fixes #7827
2017-07-28 21:29:23 +00:00
Armin
1f004c646f MINOR: Remove complicated utility method call from RubyEvent initialize method
Fixes #7825
2017-07-28 12:56:13 +00:00
Armin
8d216a31c0 MINOR: Flatten Pipeline Worker Threads
Fixes #7836
2017-07-28 12:38:57 +00:00
Armin
6fb0f7db21 #7690 Fix needless waiting by using an ArrayBlockingQueue
Fixes #7691
2017-07-28 06:08:29 +00:00
Armin
8919210fbe PERFORMANCE: Optimizetagging API paths
Fixes #7821
2017-07-27 04:42:14 +00:00
Rob Bavey
669be49408 Add some safety to closing the writer and deleting the lock file
Fixes #7822
2017-07-26 20:55:14 +00:00
Rob Bavey
435466b6b0 Close channel associated with lock
Fix Windows build issue where .lock file cannot be deleted due to
underling channel associated with lock not being closed. This
will stop the file from being deleted until the JVM exits, despite
reporting through Java that the file *has* been deleted and does not
exist. It also blocks new files from being created with the same
filename, which caused the test failure here.

Fixes #7822
2017-07-26 20:55:14 +00:00
Armin
291518d741 PERFORMANCE: Extracted cold paths in BiValues and Accessors
Fixes #7815
2017-07-26 10:19:38 +00:00
Jake Landis
0e342f3d9e Revert "Fix path for local development"
This reverts commit ae21bf374c.

This change was only applicable to IntelliJ, see https://youtrack.jetbrains.com/issue/IDEA-175172

Fixes #7806
2017-07-25 19:22:37 +00:00