Commit graph

8588 commits

Author SHA1 Message Date
Lisa Cawley
524b92e12b [DOCS] Fixed broken links (#7573) 2017-06-29 12:36:38 -07:00
Lisa Cawley
cebc191e55 [DOCS] Update Logstash Reference to use shared attribute file (#7541)
* [DOCS] Update Logstash Reference to use shared attribute file

* [DOCS] Updated shared attributes for Logstash

* [DOCS] Fix beat URLS
2017-06-29 12:14:41 -07:00
Jake Landis
87c3e11788 Minor bug fix: Stop create unecessary objects with every metric update
The existing implementation has a bug such that the block, which creates a new metric, is called for each and every update. That new metric will be ignored if there is an existing metric found. There is no functional bug here, but we are creating alot of uneeded short lived objects. The fix here is to ensure that the block is only called when needed. Also note that this can not be done atomically since put_if_absent will execute the value parameter statement (e.g. the block) regardless if absent or not.

Fixes #7563

Fixes #7564
2017-06-29 18:14:19 +00:00
Armin
10007948c8 bck
Fixes #7524
2017-06-29 17:48:36 +00:00
Armin
c307b18ff0 #7476 fix syntax of queue fully_acked method
Fixes #7524
2017-06-29 17:48:36 +00:00
Armin
a10a7eb8c6 #6828 port some tests to json-schema instead of self-made json matching
Fixes #7419
2017-06-29 00:05:16 +00:00
Armin
153eea67b5 #6828 add json-schema dependency
Fixes #7419
2017-06-29 00:05:16 +00:00
Pier-Hugues Pellerin
39630a986f Simplier algorithm to retrieve the installable plugins
Using a recursive algorithm was causing issue with SSL connection to
rubygems when we were manually settings variable to NIL.

This commit uses a simple loop and will try to install the plugins in
sequence and it will give a bit more information at the STDOUT level.

Fixes #7562
2017-06-28 23:17:28 +00:00
Guy Boertje
3cbd3d0d40 Move the global method register_local_modules to a Ruby Module (#7555)
Now in LogStash::Modules::Util, this allows it to be used where modules
are defined in external gems like logastash-x-pack.
2017-06-28 19:31:55 +01:00
Guy Boertje
b06767b71e Clone LogStash::SETTINGS in logstash-core/spec/logstash/pipeline_dlq_commit_spec.rb
Explained in the issue 7551 (#7552)

Fixes #7551
2017-06-28 14:30:05 +01:00
DeDe Morton
e8f556d07e Add links to grok debugger
Fixes #7545
2017-06-27 22:09:25 +00:00
Pier-Hugues Pellerin
b3b9e605ac Check if the file exist before deleting them
When running the `ci_acceptance` test we need to make sure we don't have
any Gemfile or *.lock laying around. Deleting a non existant file would
make the test fails.

Fixes #7515
2017-06-27 00:41:08 +00:00
Pier-Hugues Pellerin
461d2058a9 Fix an OOM issue when generating the plugins dependence json
Make sure the unused variables are freed in the recursive method call.

Fixes #7544
2017-06-26 21:16:26 +00:00
DeDe Morton
ef8ec6a42c Fixes from the review
Fixes #7323
2017-06-26 19:02:45 +00:00
DeDe Morton
e092f6b320 Add fixes from DLQ review
Fixes #7323
2017-06-26 19:02:45 +00:00
DeDe Morton
2c4194f340 Add usage info for dead letter queues
Fixes #7323
2017-06-26 19:02:45 +00:00
Armin
5b0c86c4a4 MINOR: Align JMH JVM settings with production default JVM settings
Fixes #7534
2017-06-25 17:20:20 +00:00
Jake Landis
bb19b5df53 Fix path to core java classes while doing local development
Fixes #7533
2017-06-25 13:24:52 +00:00
Jordan Sissel
a355d9f09d Use plugin name instead of metadata because sometimes Bundler-provided Gem::Specifications have nil metadata.
Fixes #7530
2017-06-23 20:09:24 +00:00
DeDe Morton
468b8d2257 Remove IDs from common files to fix build errors (#7523) 2017-06-22 18:39:31 -07:00
Jordan Sissel
70fa6f2d57 Only include Logstash plugins in list of plugins
When generating list of compatible plugins for documentation, we
previously were accidentally include all gems, not just logstash
plugins.

Fixes #7521
2017-06-22 22:24:47 +00:00
Pier-Hugues Pellerin
fc26a1692f Use docker-ce instead of docker engine
docker engine seems to be not available anymore and is replaced by
docker-ce.

Fixes #7517
2017-06-22 17:32:57 +00:00
Benjamin Dahon
43ceda5755 Fixed folder layout (#7501)
The folders layout are both broken
2017-06-22 10:28:47 -07:00
Benjamin Dahon
2105b498ec Update default license (#7502)
* Update default license

The current value issues a warning when building: 

WARNING:  license value 'Apache License (2.0)' is invalid.  Use a license identifier from
http://spdx.org/licenses or 'Nonstandard' for a nonstandard license.
Did you mean 'Apache-2.0'?

* Update license

* Update license

* Update license
2017-06-22 10:05:57 -07:00
lcawley
7bc3c230ea [DOCS] Fix broken X-Pack Reference links 2017-06-21 11:29:49 -07:00
Lisa Cawley
5adf10fae5 [DOCS] Updates to index.asciidoc (#7480)
* [DOCS] Updates to index.asciidoc

* [DOCS] Split index-shared.asciidoc for re-use

* [DOCS] Add path for logstash-docs repo

* [DOCS] Merge info about monitoring UI

* [DOCS] Added kibana branch attribute

* [DOCS] Fix xpack reference links
2017-06-21 08:40:27 -07:00
Armin
64cdafb7e8 MINOR: Gradle 4.0
Fixes #7495
2017-06-21 15:11:32 +00:00
Guy Boertje
7696c6c397 Fix for: Unreliable logstash-core/spec/logstash/timestamp_spec under JRuby 9000 (#7474)
* Fix Timestamp.new(now) != now, out by 0.999999 milliseconds

* add comment

* add for failing legacy_ruby_timestamp_spec too
2017-06-21 13:09:11 +01:00
Jordan Sissel
35c1cff164 Feature flag for string escape sequences (#7442)
New boolean setting `config.support_escapes` which defaults to false
(the historical behavior). When set to true, the following escapes are
handled:

* backslash doublequote -> doublequote
* backslash quote -> quote
* backslash n -> newline (ascii 10)
* backslash r -> carriage return (ascii 13)
* backslash backslash -> backslash
* backslash t -> tab (ascii 9)

This will solve #1645.
2017-06-20 14:34:10 -07:00
Jake Landis
1ebe81d46c Doc: update contributing guide to include link for IntelliJ setup Fixes #7120
Fixes #7422
2017-06-19 12:53:40 +00:00
Andrew Cholakian
55dcd8303e Fix reverseDepthFirst(Vertex) to search backward, not forward
This was an untested portion of code, and it accidentally invoked a
forward depth-first search instead of a reverse one.

Fixes #7485
2017-06-18 03:50:20 +00:00
Andrew Cholakian
c325264ba8 Hash source code when spaces are present
We were falling back to the graph hash (slow) algorithm anytime spaces
were present in source code due to a mixed-up regexp. This clears up
that issue.

The test, sadly, had a non-realistic string which didn't catch this
error!

Fixes #7484
2017-06-18 03:49:55 +00:00
DeDe Morton
16df01a9b5 Add info about monitoring UI
Fixes #7443
2017-06-16 19:20:56 +00:00
Armin
7a0a7e0367 MINOR: Add Ruby Version File
Fixes #7431
2017-06-16 18:17:14 +00:00
Pier-Hugues Pellerin
0f648858ef Remove *.lock and Gemfile before running any acceptance test
The acceptance test in our CI infrastructure doesn't clear the workspace between run
this mean the lock of the Gemfile can be sticky from a previous run, before generating any package
we will clear them out to make sure we use the latest version of theses files from the template.
If we don't do this we will run into gem Conflict error.

Fixes #7435
2017-06-16 14:28:39 +00:00
Pier-Hugues Pellerin
7b96c1a5da Small typo in the pipelines.yml (#7470)
The `config.string` option was not correctly closed with a double quote
2017-06-16 10:06:27 +01:00
Guy Boertje
d13d54c377 MODULES: Fix broken multi-pipelines because, you know, modules (#7466)
* Update runner, source/local, source/modules, source/multi_local
improve startup config problem detection across all sources.

* fix multi_local spec to use not `empty` values.
2017-06-16 09:47:15 +01:00
Pier-Hugues Pellerin
6e165a216c Add the plugin titleabbrev
To have better search result we now use the full title in the plugin
main doc, but we also need an abbreviation.

Ref: #7439

Fixes #7471
2017-06-15 23:42:28 +00:00
Suyog Rao
b8ea070706 Enhance ingest convert to add input/output (#7456)
* Add input/output to converter

* update CLI help

* remove newline and use join on array
2017-06-15 16:21:19 -07:00
DeDe Morton
54445ae57f Make path to github repo plugin-specific
Fixes #7448
2017-06-15 18:37:12 +00:00
DeDe Morton
72149f7374 Fix example and add couple of usage notes (#7374) 2017-06-15 11:33:46 -07:00
DeDe Morton
5c8caed802 Remove deprecated offline plugin pack instructions
Fixes #7327
2017-06-15 18:19:12 +00:00
Rob Bavey
036caeb21a Avoid unnecessary seeks in seekToNextEventPosition
When an exact match is found during the binary search for the
next event, use the matching block instead of seeking from the
'low' block.

Add toString method to DLQEntry

Fixes #7464
2017-06-15 18:13:08 +00:00
Andrew Cholakian
6b5379026e Properly hash when no original source is present.
This corrects a mistake where the digest of the string source should be
used when present, and the hash of the graph should be used when not.

Fixes #7467
2017-06-15 17:53:18 +00:00
Jordan Sissel
50cbaf4bf0 Set default entropy source to /dev/urandom
This changes the Java-wide from the default of /dev/random. The reason
for this chnage is that /dev/random can block if there is insufficient
entropy available, and this blocking can cause Logstash to stall on
startup for many many minutes.

Fixes #6117.

This scenario (no entropy) seems especially prone to occur on virtual
machines.

Related: https://github.com/jruby/jruby/wiki/Improving-startup-time#ensure-your-system-has-adequate-entropy

On Windows, I was not sure if this would work, but the JDK source tree
says this:

    On Windows systems, specifying the URLs "file:/dev/random" or
    "file:/dev/urandom" will enable the native Microsoft CryptoAPI seeding
    mechanism for SHA1PRNG.

So defaulting to /dev/urandom should still do the right thing on
Windows.

I also checked to see if other unix-likes had /dev/urandom, and
according to Wikipedia[1], Linux, FreeBSD, OpenBSD, OSX, Solaris,
NetBSD, and others all have urandom.

[1] https://en.wikipedia.org/wiki//dev/random

Fixes #7469
2017-06-15 17:44:34 +00:00
Guy Boertje
1c47ab0f9b MODULES: Add maxbuckets to the kibana config json (#7451)
* Add maxbuckets to the kibana config json

* change METRICS_MAX_BUCKETS to 86400 and fix specs
2017-06-15 17:56:02 +01:00
Armin
448bda167f #7460 ignore failing tests caused by #e939c6a removing CEF module
Fixes #7462
2017-06-15 15:30:21 +00:00
Armin
24d2faba10 MINOR: Update docker-api gem to latest version to improve stabilitiy
Fixes #7461
2017-06-15 12:48:49 +00:00
Suyog Rao
e939c6a085 Remove CEF module until we test more (#7455)
We discussed removing this module until we do end-end testing
2017-06-14 17:20:15 -07:00
Pier-Hugues Pellerin
d4183aa043 Add ephemeral_id on the Logstash instance and for the pipelines.
Theses id wont be persisted between restart of logstash or between
creation of pipeline. When we reload pipelines the ephemeral_id will be
generated again.

Fixes #7423
2017-06-14 13:37:02 +00:00