Commit graph

3730 commits

Author SHA1 Message Date
Pier-Hugues Pellerin
fcf8d783fa Add a caching mechanism for the #sprintf calls
This PR allow the Event class to compile a `sprintf` the first time and
reuse the template for the other calls. This strategy speeds up considerably
calls that uses date formatting or use fieldref its a bit slower if you
dont do any string manipulation.

Fixes #3425
2015-06-18 17:59:47 +00:00
Pere Urbon-Bayes
4677f0a54b add a validation fase with a warning in case you're updating plugins to a new major version
Fixes #3423
2015-06-17 10:05:58 +00:00
Jason Woods
93882095e4 Show errors encountered when validating a plugin rather than saying it does not exist
Fixes #3407
2015-06-11 14:17:26 +00:00
Joao Duarte
fa4f72833c allow forced termination through a second sigint
Fixes #3379
2015-06-09 14:35:34 +00:00
Jordan Sissel
68b8693612 Deprecate the input plugin's base charset setting.
Refer uses to the codecs instead.

Fixes #3044

Fixes #3395
2015-06-09 02:07:48 +00:00
Pere Urbon-Bayes
0d82ae87c1 fix the issue caused by jruby-kafka that prevent to run the packaging
Fixes #3393
2015-06-08 17:25:41 +00:00
Pier-Hugues Pellerin
d2a82b6a77 Remove the synchronize on the flush_filters_to_ouputs!
We were synchronizing on the tick and the flush call,
in some situation it could set logstash in a deadlock when the queue was
blocked and the tick occurred. The flush call was already thread safe
since only one worker can start the flushing process.

Fixes #3378
2015-06-08 15:14:01 +00:00
Pere Urbon-Bayes
d44e6e15a3 fix the uninstall of plugins
Fixes #3362
2015-06-08 13:13:07 +00:00
Colin Surprenant
3d1847aecc fix accessors field references caching performance regression
correct use of @lut and cleanups

leftover

added docs and renamed a few identifiers for clarity

typos

remove strict_set stuff since its not used anymore

removed unused specs

PathCache thread safety
2015-06-05 15:18:19 -04:00
Pier-Hugues Pellerin
37d1033b88 Add support for multiline conditionals with the else if statements
This is a followup of the issues #2850 and #3281.

The following configuration:

    if [condition] {

    } else if [condition1]
      or [condition2] {
        ..
    }

Was compiled to ruby like this:
    elsif condition or condition2 # else if [condition1]
      or [condition2]

and making the intepreter fails.

Fixes #3386
2015-06-05 18:35:57 +00:00
Shaunak Kashyap
43b13e033c Replacing mailing list address with discussion forum address.
Fixes #3366
2015-06-04 19:20:04 +00:00
Joao Duarte
a4b112ae14 allow parameters in peeraddr monkey patch
Fixes #3365
2015-06-03 09:06:58 +00:00
Pier-Hugues Pellerin
6d02988142 Better implementation of the no .bundle/config that respect the bundler contract in a running process
The previous fix for disabling the `.bundle/config` wasn't not respecting the bundler contract.
The failling tests were not wrong, in fact they exposed the issue that the configuration was transient and the underlying hash of bundler was not correctly keeping the updated values.

This patch make sure the hash is updated with the new or deleted value without persisting the change to disk.

Fixes #3332
2015-05-28 00:52:53 +00:00
Pier-Hugues Pellerin
0bb827c4fe Make sure we correctly patch the thin bootstrap layer of the distribution
Fixes #2701
2015-05-27 17:14:00 +00:00
Pier-Hugues Pellerin
dc88956362 Override bundler behavior to save config to the .bundle/config file
Fixes #2701
2015-05-27 17:13:59 +00:00
Jordan Sissel
99741e57a2 Strip line terminators when writing the comment for a branch
Logstash's config compiler adds a comment to the compiled code, like

    if ..... # if [your] and [conditional]

The idea is to to help aid in reading the compiled logstash config.
However, if a conditional has newlines in it, the `#text_value` of
that conditional will have newlines, and we'll accidentally create
invalid ruby code which will fail with SyntaxError.

Prior to this change, the following Logstash config, under 1.5.0,
would cause a crash on startup:

    if [some]
      or [condition] {
      ...
    }

The cause was that Logstash would compile this to:

    if event("[some]"]) || event("[condition]") # if [some]
    or [condition]
      ...
    end

The 2nd line there `or [condition]` was intended to be on the line
above.

This change strips the line terminators \r and \n, just in case, and
provides a test case to cover.

I verified that this test case _fails_ without the config_ast.rb patch
and _succeeds_ with the patch.

Fixes #2850

Fixes #3281
2015-05-19 20:01:05 +00:00
Pere Urbon-Bayes
26f6be3aed Revert "fix string interpolation error when transforming characters to UTF8 in the configuration compiler"
This reverts commit 65c789a24c.

Fixes #3264
2015-05-18 16:14:12 +00:00
Pere Urbon-Bayes
65c789a24c fix string interpolation error when transforming characters to UTF8 in the configuration compiler
Fixes #3259
2015-05-18 13:09:13 +00:00
Pier-Hugues Pellerin
214f3b9e44 Allow to update all the plugins and skip locally defined gems fixes #3157
Review comments

Fixes #3179
2015-05-11 14:15:14 +00:00
Pier-Hugues Pellerin
fe99f45499 Initialize the metadata_accessors when setting the @metadata with a hash
Fixes #3198
2015-05-08 13:52:58 +00:00
Colin Surprenant
7a1ec64c23 use :build dependencies
fix octokit dep

fixes #3171
2015-05-06 08:27:57 -04:00
Colin Surprenant
af4cdb6be2 environment when not in package context
remove unused constants
2015-05-05 07:58:11 -04:00
Colin Surprenant
48bcc5b74d cleanup dependencies and rely on logstash-core gem
remove unused load path

use either local core lib dir or logstash-core gem

include spec in logstash-core

do not include lib, spec and locales dirs in package root

environment bootstrapping in lib/bootstrap.rb

added comments

move pluginmanager out of logstash-core

kafla installation solved by pluginmanager refactor

refactor bootstrap code

refactor shell scripts to simplify and DRY, indirect rspec command to our ruby launcher

add bin/rspec

cut by half the bin/plugin and bin/rspec startup time

fix drip context

use printf instead of tr

updated Windows bin scripts

missing gemspec in gemspec

use gem instead of gemspec so our plugin manager can correctly install logstash-core

generate packages including locally built logstash-core gem

move jenkins developmnent dependencies into Gemfile, they do not belong in logstash-core

path leftover

clean help agent help usage message and remore rspec in available command

comments cosmetics

update Bundler dependency, all recent testing have been made with 1.9.4

updated .lock file with regard to updated Gemfile

cleanup gemspec, Gemfile and regen .lock file

added progress output

avoid dual require on version

closes #3096
2015-05-04 18:17:22 -04:00
Colin Surprenant
3dabba80fa revert multi_filter removal and add check for cancelled events
add filter and flush compiles functions specs

missing multiline filter for core specs

fixes #3100
2015-04-27 13:54:18 -04:00
Colin Surprenant
f8f2609c1b remove setup! and add :clean to uninstall 2015-04-22 17:42:16 +02:00
Pier-Hugues Pellerin
362953b312 Make sure we setup the gem environment before finding the currently installed gems
Fixes #3063
2015-04-21 14:59:37 +00:00
Pier-Hugues Pellerin
b4b8817b4e fix an issue with bin/plugin update with no arguments and no locally installed gems
Fixes #3063
2015-04-21 14:59:36 +00:00
Colin Surprenant
902c2f4710 bundler refactor leftover 2015-04-21 13:07:40 +02:00
Pier-Hugues Pellerin
cc3f9d6ac4 Do not return an empty metadata key hash when calling to_hash_with_metadata
Fixes #3034
2015-04-20 22:59:04 +00:00
Pere Urbon-Bayes
be1c6e30d0 fix relative pathname lookup while installing a local .gem file
use double quotes for the require

Fixes #3040
2015-04-17 16:43:06 +00:00
Pier-Hugues Pellerin
8f8509f0fc wrong link for the config type :bytes documentation
Fixes #3041
2015-04-17 16:15:41 +00:00
Colin Surprenant
cbb225db35 fix for Java 8 Map merge method conflict
typo

DRYied code, comments, cleanup

added spec

specs DRYing

also test HashMap, refactor with shared_example

fixes logstash-plugins/logstash-filter-multiline#10
2015-04-17 14:40:50 +02:00
Colin Surprenant
ae17b4160d refactored all gem/bundler code into logstash/bundler.rb and moved patches into logstash/patches
require bundler/cli to expose expectations classes
2015-04-16 17:34:51 +02:00
Joao Duarte
3eb5ba6450 remove multi_filter 2015-04-15 23:18:32 +02:00
Colin Surprenant
573149b002 refactor conditionals generation
added specs
2015-04-15 23:16:06 +02:00
Tal Levy
2e1b879431 fix include to support nil and array types
Closes logstash-plugins/logstash-filter-mutate/#21.

Fixes #3023
2015-04-15 20:35:17 +00:00
Colin Surprenant
98a5f2348c fix java enumerable integration bug
Fixes #3022
2015-04-15 19:59:18 +00:00
Colin Surprenant
e599284e62 add Java Collection delete, & and | support. refactored Java interfaces equivalence. specs & more java_integration specs
add Java Collection delete support with specs & more java_integration specs

split a spec

refactored for better specs

intersection specs

Ruby intersection on Java collections, refactored to use Java interfaces

specs for remove_tag from events from json input

refactor usage of subject

added Java Collection union with Ruby array and specs

refactored specs to also test for hash from deserialized json for JrJackson & Java Collections

typo and comments

solves #2261
2015-04-15 12:46:22 +02:00
Pere Urbon-Bayes
1cbe1791d1 add the concept of a jar manager to load jars within plugins
make the jar manager to have some of the checks already present in the old load_elasticsearch_jars

removed the load_elasticsearch_jars! method as it's a legacy code from when ES was delivered within LS, nowadays this comes within the differents plugins

clean up Environment::JAR_DIR as is another legacy variable not used anymore

move the jar loading code within the environment module

ammend previous commit adding more clear code

ammend previous commits

added a few simple unit test for the jar manager in the environment module

refactor spec naming

dry out common code for loading jars

clean up env_spec bein in the util dir

cleaning up logstash-core dependencies related to jar-dependencies

ammend stale maen patches created for the LS way of using maven plugins

applied some comments coming from the review by @ph

bring back the maven tools and jar-dependency code as we still need some code like this for the kafka plugins

Revert "bring back the maven tools and jar-dependency code as we still need some code like this for the kafka plugins"

This reverts commit c23f456362776c0a600d78369c2ece8789fa49c2.

Fixes #2980
2015-04-14 15:59:31 +00:00
Tal Levy
2c8802710b Fixes Event#include? to properly check for field
Beforehand, Event#include? would depend on a value of a certain
field to be non-nil as the signal that a field exists. This breaks
when a field's value is nil, even if the field exists. It is now
possible to check for existance of both normal and metadata fields.

Closes #2977.

Fixes #2997
2015-04-13 21:54:41 +00:00
Pier-Hugues Pellerin
2e1bcc145a Add logstash plugin verification on the .gem file
Fixes #2946
2015-04-09 13:52:08 +00:00
Pier-Hugues Pellerin
25926f7f85 Rename plugin_list to plugins_arg to make it clearer it is user submitted content.
Fixes #2946
2015-04-09 13:52:08 +00:00
Pier-Hugues Pellerin
244be5cbb6 make the conditionals use a positive if
Fixes #2946
2015-04-09 13:52:08 +00:00
Pier-Hugues Pellerin
764df9e477 typo
Fixes #2946
2015-04-09 13:52:07 +00:00
Pier-Hugues Pellerin
0839c4da6c Move the verify? outside the verify! method, easier to unit test
Fixes #2946
2015-04-09 13:52:07 +00:00
Pier-Hugues Pellerin
e9e8217f71 Shorter error message for manual plugins
Fixes #2946
2015-04-09 13:52:06 +00:00
Pier-Hugues Pellerin
f2d9c7db5b Aborded -> Aborted
Fixes #2946
2015-04-09 13:52:06 +00:00
Pier-Hugues Pellerin
7518a6aa3d Renaming base -> Command and Fixing bin/plugin list
Fixes #2946
2015-04-09 13:52:06 +00:00
Pier-Hugues Pellerin
090cf0186b typo orignal -> original
Fixes #2946
2015-04-09 13:52:05 +00:00
Pier-Hugues Pellerin
1b0160a77e uses local_gems
Fixes #2946
2015-04-09 13:52:05 +00:00