Commit graph

747 commits

Author SHA1 Message Date
Andrew Cholakian
9c23cf7b09 Expand out some combined test cases
Fixes #3452
2015-06-19 19:07:23 +00:00
Andrew Cholakian
5fd8d54e97 Use subject instead of let
Fixes #3452
2015-06-19 19:07:22 +00:00
Andrew Cholakian
259f3daeb2 Return explicit false on bad java version
Fixes #3452
2015-06-19 19:07:22 +00:00
Andrew Cholakian
5729fb9a0a Check java versioning
Fixes #3452
2015-06-19 19:07:20 +00:00
Pere Urbon-Bayes
a00e26bacb add LGPL as valid licence for logstash runtime dependencies
Fixes #3466
2015-06-18 18:23:12 +00:00
Pier-Hugues Pellerin
896a91d872 Bad interpolation when issue mixed template like foo %{bar} %{+YYYY}
Fixes #3425
2015-06-18 17:59:48 +00:00
Pere Urbon-Bayes
1754d896a1 add small comment about the licenses
Fixes #3426
2015-06-18 08:34:58 +00:00
Pere Urbon-Bayes
954c983d1f applyed improvements comming out of the review
Fixes #3426
2015-06-18 08:34:58 +00:00
Pere Urbon-Bayes
eb64b35d39 add a runtime license test
Fixes #3426
2015-06-18 08:34:57 +00:00
Pere Urbon-Bayes
e2450f1f78 Make the necessary changes to use rspec3 as a default test runner for
logstash, including adapting the test suites that were still using 2.x
syntax.
2015-06-08 12:19:11 +02:00
Colin Surprenant
3d1847aecc fix accessors field references caching performance regression
correct use of @lut and cleanups

leftover

added docs and renamed a few identifiers for clarity

typos

remove strict_set stuff since its not used anymore

removed unused specs

PathCache thread safety
2015-06-05 15:18:19 -04:00
Pier-Hugues Pellerin
37d1033b88 Add support for multiline conditionals with the else if statements
This is a followup of the issues #2850 and #3281.

The following configuration:

    if [condition] {

    } else if [condition1]
      or [condition2] {
        ..
    }

Was compiled to ruby like this:
    elsif condition or condition2 # else if [condition1]
      or [condition2]

and making the intepreter fails.

Fixes #3386
2015-06-05 18:35:57 +00:00
Pier-Hugues Pellerin
6d02988142 Better implementation of the no .bundle/config that respect the bundler contract in a running process
The previous fix for disabling the `.bundle/config` wasn't not respecting the bundler contract.
The failling tests were not wrong, in fact they exposed the issue that the configuration was transient and the underlying hash of bundler was not correctly keeping the updated values.

This patch make sure the hash is updated with the new or deleted value without persisting the change to disk.

Fixes #3332
2015-05-28 00:52:53 +00:00
Pier-Hugues Pellerin
aa036b9b8d Settings doesnt persist the values to the bundle/config
Fixes #3330
2015-05-27 21:19:36 +00:00
Jordan Sissel
99741e57a2 Strip line terminators when writing the comment for a branch
Logstash's config compiler adds a comment to the compiled code, like

    if ..... # if [your] and [conditional]

The idea is to to help aid in reading the compiled logstash config.
However, if a conditional has newlines in it, the `#text_value` of
that conditional will have newlines, and we'll accidentally create
invalid ruby code which will fail with SyntaxError.

Prior to this change, the following Logstash config, under 1.5.0,
would cause a crash on startup:

    if [some]
      or [condition] {
      ...
    }

The cause was that Logstash would compile this to:

    if event("[some]"]) || event("[condition]") # if [some]
    or [condition]
      ...
    end

The 2nd line there `or [condition]` was intended to be on the line
above.

This change strips the line terminators \r and \n, just in case, and
provides a test case to cover.

I verified that this test case _fails_ without the config_ast.rb patch
and _succeeds_ with the patch.

Fixes #2850

Fixes #3281
2015-05-19 20:01:05 +00:00
Pere Urbon-Bayes
26f6be3aed Revert "fix string interpolation error when transforming characters to UTF8 in the configuration compiler"
This reverts commit 65c789a24c.

Fixes #3264
2015-05-18 16:14:12 +00:00
Pere Urbon-Bayes
65c789a24c fix string interpolation error when transforming characters to UTF8 in the configuration compiler
Fixes #3259
2015-05-18 13:09:13 +00:00
Pier-Hugues Pellerin
fe99f45499 Initialize the metadata_accessors when setting the @metadata with a hash
Fixes #3198
2015-05-08 13:52:58 +00:00
Colin Surprenant
48bcc5b74d cleanup dependencies and rely on logstash-core gem
remove unused load path

use either local core lib dir or logstash-core gem

include spec in logstash-core

do not include lib, spec and locales dirs in package root

environment bootstrapping in lib/bootstrap.rb

added comments

move pluginmanager out of logstash-core

kafla installation solved by pluginmanager refactor

refactor bootstrap code

refactor shell scripts to simplify and DRY, indirect rspec command to our ruby launcher

add bin/rspec

cut by half the bin/plugin and bin/rspec startup time

fix drip context

use printf instead of tr

updated Windows bin scripts

missing gemspec in gemspec

use gem instead of gemspec so our plugin manager can correctly install logstash-core

generate packages including locally built logstash-core gem

move jenkins developmnent dependencies into Gemfile, they do not belong in logstash-core

path leftover

clean help agent help usage message and remore rspec in available command

comments cosmetics

update Bundler dependency, all recent testing have been made with 1.9.4

updated .lock file with regard to updated Gemfile

cleanup gemspec, Gemfile and regen .lock file

added progress output

avoid dual require on version

closes #3096
2015-05-04 18:17:22 -04:00
Colin Surprenant
3dabba80fa revert multi_filter removal and add check for cancelled events
add filter and flush compiles functions specs

missing multiline filter for core specs

fixes #3100
2015-04-27 13:54:18 -04:00
Pier-Hugues Pellerin
cc3f9d6ac4 Do not return an empty metadata key hash when calling to_hash_with_metadata
Fixes #3034
2015-04-20 22:59:04 +00:00
Colin Surprenant
cbb225db35 fix for Java 8 Map merge method conflict
typo

DRYied code, comments, cleanup

added spec

specs DRYing

also test HashMap, refactor with shared_example

fixes logstash-plugins/logstash-filter-multiline#10
2015-04-17 14:40:50 +02:00
Colin Surprenant
ae17b4160d refactored all gem/bundler code into logstash/bundler.rb and moved patches into logstash/patches
require bundler/cli to expose expectations classes
2015-04-16 17:34:51 +02:00
Joao Duarte
3eb5ba6450 remove multi_filter 2015-04-15 23:18:32 +02:00
Colin Surprenant
573149b002 refactor conditionals generation
added specs
2015-04-15 23:16:06 +02:00
Tal Levy
2e1b879431 fix include to support nil and array types
Closes logstash-plugins/logstash-filter-mutate/#21.

Fixes #3023
2015-04-15 20:35:17 +00:00
Colin Surprenant
98a5f2348c fix java enumerable integration bug
Fixes #3022
2015-04-15 19:59:18 +00:00
Pier-Hugues Pellerin
b233a34ce9 do not use subject an an explicit call
Fixes #2731
2015-04-15 14:22:09 +00:00
Pier-Hugues Pellerin
26c5a5ff61 updated the tests, exceptions are raised from the invoke_bundler!
Fixes #2731
2015-04-15 14:22:09 +00:00
Pier-Hugues Pellerin
8b2a6b054b adding spec to guard the invoke bundler behavior
Fixes #2731
2015-04-15 14:22:09 +00:00
Colin Surprenant
e599284e62 add Java Collection delete, & and | support. refactored Java interfaces equivalence. specs & more java_integration specs
add Java Collection delete support with specs & more java_integration specs

split a spec

refactored for better specs

intersection specs

Ruby intersection on Java collections, refactored to use Java interfaces

specs for remove_tag from events from json input

refactor usage of subject

added Java Collection union with Ruby array and specs

refactored specs to also test for hash from deserialized json for JrJackson & Java Collections

typo and comments

solves #2261
2015-04-15 12:46:22 +02:00
Pere Urbon-Bayes
1cbe1791d1 add the concept of a jar manager to load jars within plugins
make the jar manager to have some of the checks already present in the old load_elasticsearch_jars

removed the load_elasticsearch_jars! method as it's a legacy code from when ES was delivered within LS, nowadays this comes within the differents plugins

clean up Environment::JAR_DIR as is another legacy variable not used anymore

move the jar loading code within the environment module

ammend previous commit adding more clear code

ammend previous commits

added a few simple unit test for the jar manager in the environment module

refactor spec naming

dry out common code for loading jars

clean up env_spec bein in the util dir

cleaning up logstash-core dependencies related to jar-dependencies

ammend stale maen patches created for the LS way of using maven plugins

applied some comments coming from the review by @ph

bring back the maven tools and jar-dependency code as we still need some code like this for the kafka plugins

Revert "bring back the maven tools and jar-dependency code as we still need some code like this for the kafka plugins"

This reverts commit c23f456362776c0a600d78369c2ece8789fa49c2.

Fixes #2980
2015-04-14 15:59:31 +00:00
Tal Levy
2c8802710b Fixes Event#include? to properly check for field
Beforehand, Event#include? would depend on a value of a certain
field to be non-nil as the signal that a field exists. This breaks
when a field's value is nil, even if the field exists. It is now
possible to check for existance of both normal and metadata fields.

Closes #2977.

Fixes #2997
2015-04-13 21:54:41 +00:00
Pier-Hugues Pellerin
96b6893ebc Allow the installation of a plugin from a .gem, refactor the plugin manager classes.
do not change the gemfile or the .lock

Fixes #2946
2015-04-09 13:52:05 +00:00
Colin Surprenant
dbe4154aa6 1.5 pipeline performance regressions fixes
rework output function init

removed double logger.debug

use event constant

fix LogStash::SHUTDOWN to actually be the shutdown signal event push in the pipeline

fix outputs conditionals handling

code gen methods not lambdas, remove intermediate arrays, base multi_filter to simplify code gen

remove block from filter_func

force_encoding at compile time

commented multi_filter method

typo

DRY & cleanups

optimizations: string constants, timestamp initialization, useless logger instance

specs for signal events

Filters::Base interface and multi_filter method

closes #2870

Fixes #2869
2015-03-30 20:07:58 +00:00
Joao Duarte
e5ba35e0c7 test output plugin vs worker params
Fixes #2876
2015-03-24 21:06:10 +00:00
Tal Levy
203d1e5fc7 raise error when formatting time without @timestamp field
Closes #2730

Fixes #2736
2015-03-09 22:57:39 +00:00
Tal Levy
9a4bcf1f10 do not wrap existing Password type into another Password type when validating
Fixes #2767
2015-03-04 22:29:08 +00:00
Pier-Hugues Pellerin
de27d9f0a0 Use warn level instead of error level when a plugin is using the milestone method in the plugin
Fixes #2735
2015-03-03 00:19:13 +00:00
Pier-Hugues Pellerin
786d2d2c1a Fallback to a local configuration if the parsing for remote configuration fails
Fixes #2653
2015-02-23 15:46:09 +00:00
Aaron Mildenstein
8d57eac33e Missed a spec...
Fixes #2661
2015-02-20 22:53:18 +00:00
Aaron Mildenstein
3986158bac First pass at cleaning out logstash web
All internal references, flags, kibana, vendor, everything must go!

If it referenced kibana or logstash-web, it is deprecated, and removed!

Operators are standing by...

Fixes #2661
2015-02-20 22:53:18 +00:00
Pier-Hugues Pellerin
9e7d6204a5 plugins/bundler refactor
squashed 113 commits by @ph, @colinsurprenant, @jordansissel, @purbon, @jsvd

cleaning the environment.rb

initial work

remove multiples polyglot

force to load rubygems

notice

point to one config

update lock

bootstrap will package the standard application until we have the plugin manager on top of the gemfile

add the gem path to find bundler

fixing dependencies

add gemfiles and bundle

comment platform requiremments in the gemfile

Set `Gem.paths = ...` to the GEM_HOME

Without this, `bin/logstash` will error:

    LoadError: no such file to load -- bundler
         require at org/jruby/RubyKernel.java:1071
         require at C:/Users/jls/Documents/GitHub/logstash/vendor/jruby/lib/ruby/shared/rubygems/core_ext/kernel_require
.rb:55
         set_gem_paths! at C:/Users/jls/Documents/GitHub/logstash/lib/logstash/environment.rb:64
         (root) at C:\Users\jls\Documents\GitHub\logstash\lib\logstash\runner.rb:8

Patch JRuby's jar_installer.rb with a newer version

jar-dependencies is shipped by default with JRuby. However, the version
it ships with has a bug, and despite upgrading jar-dependencies to
0.1.7, which has the fix, JRuby continues to load the "shipped" version
which has a bug.

This patch will patch JRuby by copying in our own jar_installer.rb from
jar-dependencies master, This patching occurs while unpacking JRuby just
after `vendor:jruby`.

The jar_installer.rb is MIT-licensed and no changes were made.

Add `test:plugins` to test all installed plugins

Tell rspec to do the wildcards instead of trying to make the shell do it.

This fixes test runs on Windows (which doesn't do wildcards like this).
The solution was to tell RSpec to look for all files matching a certain path pattern
within the GEM_HOME.

do not run bundle install twice for test:plugins

Gemfile manipulation classes

Gemfile now in root

install POC with new Gemfile classes

specs for the Gemfile classes

header, some bug fixes

spec header

WIP pluginmanager refactor

generated format

updated .lock

unused file

removed --proxy option

fail elegantly

initial support for update

refactor invoke_bundler, debug output

removed old .lock file

no plugin Gemfile

use clamp error handling

reenable list, some cleanups

support for more finders

refactored using new finders

refactor using is_installed_plugin? and some cleanups

verify valid plugin and update all now only update plugins not all gems

support multiple plugins argument, --force option to bypass validation

list only explicitly installed plugins by default, --all option to include dependencies

support multiple plugins argument

renamed all_installed_plugins to all_installed_plugins_gem_specs and return specs, added comments

remove then

update_gem bug, comments

moved gems as development dependencies

updated .lock file

support for :without

--development option, better output feedback, cleanups

abort when no plugins

cosmetic

param fix, cleanups

removed deprecated .lock files

remove deprecated bundler_install_command and struggling with set_gem_paths

refactored all bundler stuff into LogStash::Bundler

rakelib refactor

refactored bundler code

Bundler.real_fuckin_reset

real_fuckin_reset is now reset_settings, removed unnecessary resetings, cleanups

duplicate plugins requirements merging

cleanups

added vendor:rake, cleanups

broke test tasks into 2 steps, first invoke install then test

rake task name fix

proper error exit code

Freezing the jar-dependencies, ruby-maven, tools

set gem paths on error

renamed TEST_PLUGINS to CORE_PLUGINS, added comments

replace test by core, cleanups

DRYied test tasks

missing gemspec

fixed output

correct image build for artifacts

removed deprecated Gemfiles

fix update with multiple plugins arguments

fix update with multiple plugins arguments

remove deprecated code

implicit begin, explicit block

spec for LogStash::Bundler

remove explit require of rubygems

renamed Environment.set_gem_paths! to bundler_setup!

rename Dsl to DSL

respect to_s contract and return empty string not nil

rename DSL.parse gemfile argument to gemfile_content

better error message

renamed --force to --[no-]verify

refactored exit error handling

use explicit array items in iterators

fix --development switch validation, cleaned bundler_options hash init

renamed is_xxx? methods to xxx?

use direct method call on collection

removed double negative, typo

move maven tools monkey patch into maven_tools_patch.rb

renamed build-image to prepare

fix forgotten argument name change

namespace cleanups

spec files glob pattern

cleanup

cleanup

cleanup

Fix the file-dependencies version to 0.1.5 to fix path issues on windows

monkeypatch Rubygems to silence ffi warning

list all plugin gems by default, new options

adding rake task to get faster feedback loop on windows

refactor bat scripts

Use an unreleased filedependencies gem and skip installing drupal

bump file-dependencies

use the released gem of file-dependencies

Add a set of scripts that simplify:   * The necessary setup for the CI test to be executed.   * The actuall CI test execution, having a different scripts for each     different workflow. This makes the test execution simpler and standard, so changes in internal task don't need big changes in the CI config. It also documents and makes easy for developers to run the tests.

Fixes #2640

refactored test to not depend on STDOUT

create a .rspec file to pass some options used while the code is in the testing environment

add coveralls and simplecov as a development dependencies so we can run coverage tests

remove unnecessary .bundle directory from the artifact

In a release, the .bundle/config has a BUNDLE_PATH setting that can
interfere by misleading the plugin manager into not seeing the
vendor/bundle path.

Fixes #2647

Fixes #2475
2015-02-20 17:31:43 +00:00
Pier-Hugues Pellerin
890c1bc3eb Use the .info level when printing the version notice for the plugin
Fixes #2429
2015-01-27 17:40:38 +00:00
Pere Urbon-Bayes
5e689dad9c cleanup unused require references as they are required by the devutils spec_helper
mock the coverage report from devutils into the general spec helper

Revert "mock the coverage report from devutils into the general spec helper"

This reverts commit 4c7e265e302b9e9450a44469ac571ad39fb7d433.

Fixes #2412
2015-01-27 12:15:31 +00:00
Pere Urbon-Bayes
b53e2e8aed enabled coverage report again throw installing missing dependencies when the user run rake test:prep, user should use COVERAGE=true as before
Fixes #2412
2015-01-27 12:15:31 +00:00
Suyog Rao
46e96fb0a2 Added specs for accessors del method
Fixes #2384
2015-01-22 17:26:04 +00:00
Pier-Hugues Pellerin
07f7882ccf fix merge conflicts
Fixes #2280
2015-01-21 22:08:11 +00:00
Tal Levy
c3bab9d896 add support for :bytes validation to config
Fixes #2321
2015-01-15 19:53:31 +00:00
Pier-Hugues Pellerin
2d7ebd1999 add a check if we can't find the version
Fixes #2331
2015-01-14 23:46:00 +00:00