Commit graph

1649 commits

Author SHA1 Message Date
Armin
dfb423a60f PERFORMANCE: Remve needless locking on batch reads
Fixes #7623
2017-07-25 18:33:09 +00:00
Armin
96ff365997 Fixes #7676 and generally improves the performance of org.logstash.StringInterpolation#evaluate by:
* Don't use regex to find the contents between `%{}`, it's way too inefficient for a simple case like this with all its allocations
* Flatten logic out of the many helper classes
* Use a reusable, thread-local `StringBuilder` to serialize into
* Don't use `.substring` for comparisons
* `static` `ObjectMapper`
* `org.joda.time.format.DateTimeFormat#forPattern` not being cached is not an issue, since it's cached in `org.joda.time.format.DateTimeFormat#cPatternCache`

* Added benchmark
   * ~15% faster for a few cases I ran, even for repeated patterns with interpolation + couldn't find any case even with date formatting, where this implementation is slower than the cached one
   *  no leaking on dynamic patterns

Fixes #7692
2017-07-25 18:29:10 +00:00
Armin
ea9f56e2fc Minor: Fix a few more compiler warnings
Fixes #7802
2017-07-25 15:07:48 +00:00
Jake Landis
22e8ddd165 Test stablization: Add timeout sleep.until to prevent infinite loop, resuce error on shutdown
Fixes #7760
2017-07-25 00:14:22 +00:00
Rob Bavey
c86a6f2923 Update based on code review comments
Test cleanup

Fixes #7789
2017-07-24 20:04:55 +00:00
Rob Bavey
9f7bed2498 Fix seek by Timestamp bugs
Fixes 2 seekByTimestamp failure scenarios that occur when a segment is closed with entries smaller than BLOCK_SIZE

If the next segment is > BLOCK_SIZE, then the seek will advance to the next segment. (Skipping past all events
 in the closed segment)
If the next segment < BLOCK_SIZE, then the seek will advance to the start of the small segment. (Processing
 entries prior to the timestamp)

Fixes #7789
2017-07-24 20:04:54 +00:00
Suyog Rao
b6bcbda8e4 bump master version to 7.0.0-alpha1 (#7791) 2017-07-24 11:51:40 -07:00
Jake Landis
ae21bf374c Fix path for local development
Fixes #7786
2017-07-24 16:55:26 +00:00
Andrew Cholakian
1811283250 Fix bug where hot threads did not work with duplicate names / add thread_id
The hot threads endpoint used to require thread names to be unique in a map and would crash otherwise. This patch fixes this by internally using a List instead of a Map to store these thread ids.

Additionally, this patch adds the `thread_id` to the response to allow for disambiguation of of threads for any client applications.

Fixes https://github.com/elastic/logstash/issues/7608

Fixes #7617
2017-07-24 16:38:18 +00:00
Armin
bff61d59fe MINOR: Remove unused take and timed offer from queue wrappers
Fixes #7782
2017-07-24 16:03:00 +00:00
Armin
933919367a MINOR: Extract hot constructor from RubyEvent
Fixes #7781
2017-07-24 08:18:56 +00:00
Armin
6355c839ab MINOR: Cleanup Event by making it final and removing the incorrect serializable interface
Fixes #7769
2017-07-23 18:35:09 +00:00
Armin
5c77bde6e4 MINOR: Inline hot path code in Accessors
Fixes #7771
2017-07-23 18:34:05 +00:00
Armin
a055b2b854 MINOR: Make IO interfaces with close method implement java.io.Closeable
Fixes #7776
2017-07-23 18:33:18 +00:00
Rob Bavey
b5c540010b Add at_least(:once) to work around intermittent test failures
The test that calls out to /proc/loadavg has been intermittently
 failing, because /proc/loadavg is being read from more than once
This should be benign, so this commit allows it to happen

Fixes #6867

Fixes #7757
2017-07-20 18:22:50 +00:00
Jake Landis
f1361de660 Test stabilization: add max retry and timeouts to pipeline spec
Fixes #7738

Fixes #7752
2017-07-20 17:58:05 +00:00
Jake Landis
78d0d5a364 Test stabilization: Add retry to have_running_pipeline matcher
Fixes #7737

Fixes #7753
2017-07-20 17:57:31 +00:00
Jake Landis
0c59550d65 Test stabilization: Ensure that logging is set back to 'info' after tests that may change it
Fixes #7739

Fixes #7743
2017-07-20 17:56:33 +00:00
Guy Boertje
a6e9ab91ed Merge CLI into YML settings + limit user specified modules to 1 for 5.x (#7756) 2017-07-20 17:47:01 +01:00
Joao Duarte
5e883e17c9 pass http_options to can_connect in kibana_client
Fixes #7751
2017-07-20 14:36:00 +00:00
João Duarte
7579f96797 test multiple pipelines using RATS (#7732)
- also adds `config.support_escapes` do list of pipeline settings
2017-07-19 16:43:56 +01:00
Jake Landis
a25d329f64 Test Failure Fix: Retry pipeline metrics
Fixes #7724

Fixes #7728
2017-07-18 22:04:02 +00:00
Colin Surprenant
aedf397397 fix empty queue condition for queue shutdown drain, PR #7575
page test

queue test

move is_empty? at queue level

new wrapped acked queue spec
2017-07-18 16:32:22 -04:00
Guy Boertje
dbcce19888 correct ES_client to only split strings + add csv_string(array) method (#7722)
an x-pack-logstash PR will be reliant on the csv_string method.
2017-07-18 15:40:49 +01:00
Jake Landis
e87f810c76 Fix gradle tests on Windows
Addresses #7637

Fixes #7716
2017-07-17 22:49:05 +00:00
Andrew Cholakian
85fc814105 Fix SafeURI to maintain pre-5.5.0 escaping behavior
Prior to Logstash 5.5.0 the ::LogStash::Util::SafeURI class returned escaped values where applicable. As part of the refactor in https://github.com/elastic/logstash/pull/7236 this behavior was accidentally changed, breaking the ES output and resulting in bugs like: https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/618 .

This PR maintains the java.net.URI refactor but fixes the behavior to align with pre-5.5.0 logstashes by correctly returning escaped values.

Fixes #7687
2017-07-17 18:15:36 +00:00
Armin
1ebb0e350f #7662 Make StringBiValue safe against mutations to the underlying RubyString
Fixes #7663
2017-07-17 14:13:20 +00:00
Armin
59df92321a MINOR: Turn NullBiValue into a Singleton
Fixes #7707
2017-07-17 12:10:33 +00:00
Armin
cc81dab8c3 MINOR: Cleanup compiler warnings in org.logstash.Timestamp
Fixes #7702
2017-07-17 09:41:49 +00:00
Armin
766d10e1a7 MINOR: Fix all compiler warnings in AccessorsTest and EventTest
Fixes #7708
2017-07-17 09:41:03 +00:00
Armin
58f7fa66e0 MINOR: Factor out cold-path from org.logstash.Event#initTimestamp
Fixes #7709
2017-07-17 09:31:09 +00:00
Armin
4c4668baff MINOR: Fix redundant throws clauses
Fixes #7704
2017-07-16 14:16:36 +00:00
Armin
0e1a95e7cf MINOR: Fix compiler warnings from general Class type use
Fixes #7705
2017-07-16 14:15:37 +00:00
Armin
e3b294cd0b MINOR: Size Hashmaps more efficiently in ConvertedMap
Fixes #7693
2017-07-14 17:49:23 +00:00
Armin
c741c836a7 MINOR: Fix incorrect use of blocking queue for the pipeline's signal queue
Fixes #7675
2017-07-13 19:09:12 +00:00
Armin
a16da81c54 MINOR: Remove redundant wrapping and generics
Fixes #7655
2017-07-13 17:01:16 +00:00
João Duarte
2207d34853 adds key and namespace getters to Java Metric classes (#7680) 2017-07-13 17:58:15 +01:00
Armin
deaa90e405 #6964 migrate setting config.reload.interval to TimeValue
Fixes #7678
2017-07-13 16:13:17 +00:00
Armin
c2e55ba07b MINOR: Enable javac warnings
Fixes #7674
2017-07-13 10:40:46 +00:00
João Duarte
50d6aff798 implement ssl in modules (#7667)
* implement ssl in modules

* rearrange some setting names in logstash-modules

* removed dashboards.kibana_index setting from modules
2017-07-13 10:47:34 +01:00
Jake Landis
17be8b09c3 Metrics: Move counters and gauges to Java with stricter types.
This change moves the definition of the counters and gauges from Ruby to Java, providing a Ruby bridge that allows support for Ruby duck typing to Java strict typing.
This change also moves the serialzation from JRuby (Jackson) to Java (Jackson).
This change also removes the decrement from counter api and requires counters start at zero. This helps to allowing for reliable aggregrations over time.
This change removes the base Ruby class for metrics, and replaces with base Java class.
This change removes the to_hash method (no known usages).
This change also has minor changes to the Ruby type() and inspect() methods.

Fixes #7509

Fixes #7595
2017-07-12 22:41:20 +00:00
Jake Landis
3972a1d059 Fix ominous warning: jar coordinate ... already loaded
This change to ensure that Java and JRuby agree upon the version of Jackson. This downgrades Java from 2.7.4 to 2.7.3 (which is the JRuby version).
Fixes: #6781

Fixes #7614
2017-07-12 20:50:21 +00:00
Armin
396fc14a6c MINOR: Remove redundant instantitation of utility class
Fixes #7649
2017-07-11 22:18:46 +00:00
Armin
13341b66cc Minor: Avoid redundant Java to Ruby conversions
Fixes #7641
2017-07-11 22:17:59 +00:00
Rob Bavey
af96fa36a5 Add dead_letter_queue.max_bytes setting
Add setting for dead_letter_queue.max_bytes to allow a user
to set the maximum possible size of a dead letter queue on disk.

Resolves #7633

Fixes #7638
2017-07-11 21:06:18 +00:00
Jake Landis
55fac65160 VERSION BUMP: 6.0.0-alpha3 -> 6.0.0-beta1 (#7636) 2017-07-11 12:07:11 +01:00
Colin Surprenant
bf2a15bda7 broken Event.toString() when missing @timestamp, PR #7604
test to repro #6756

extra test

fix toString

cosmetic

remove irrelevant spec

reworked host and message string generation
2017-07-10 15:14:06 -04:00
Armin
e1da26dda6 MINOR: Cleanup redundant casts
Fixes #7626
2017-07-10 16:45:58 +00:00
Armin
c32db3b2f8 MINOR: Remove needless singleton use in pathcache
Fixes #7624
2017-07-09 18:58:08 +00:00
Guy Boertje
84cd0f96f8 Add support for using kibana dashboard import api (#7591)
* Add support for using kibana dashboard import api
- Add support for versioned folders below 'module'/configuration/kibana
- Rename cef to module_test_files/tester and simplify it.
- Remove skips on tests using above modules test files.
- In pipeline.rb, moved method default_logging_keys to the base class,
  it was being called from BasePipeline during a config-test.
- In modules_common, added a backtrace to LogStash::ConfigLoadingError
  for better debugging
- Improve layout when user_lines et al were empty in logstash_config

* Change specificity of maj/min/patch folder search

* Use logger to surface manticore error

* Add suppor for Kibana auth

* improve error message

* support es and kibana authentication in the logstash modules
2017-07-08 09:55:41 +02:00