Commit graph

3729 commits

Author SHA1 Message Date
Colin Surprenant
380ea33708 refactor test tasks, test:plugins now correctly run specs of installed plugins, including local ones
robustness in shutdown handling and other thread safeties
2015-04-08 08:43:49 -04:00
Colin Surprenant
dbe4154aa6 1.5 pipeline performance regressions fixes
rework output function init

removed double logger.debug

use event constant

fix LogStash::SHUTDOWN to actually be the shutdown signal event push in the pipeline

fix outputs conditionals handling

code gen methods not lambdas, remove intermediate arrays, base multi_filter to simplify code gen

remove block from filter_func

force_encoding at compile time

commented multi_filter method

typo

DRY & cleanups

optimizations: string constants, timestamp initialization, useless logger instance

specs for signal events

Filters::Base interface and multi_filter method

closes #2870

Fixes #2869
2015-03-30 20:07:58 +00:00
Joao Duarte
4a3c9e7be5 remove explicit clone set up on output workers
Fixes #2876
2015-03-24 21:06:11 +00:00
Joao Duarte
3763ecba67 remove deprecation notices when output workers > 1
When workers is set to > 1, new plugin instances are created using a params hash
that contains both the user-specified config parameters and the default ones.
This causes deprecation notices to be printed, even though the user never specified
a deprecated config setting.
Thus, new workers should be created using only the "original" set of config parameters.

Fixes #2876
2015-03-24 21:06:10 +00:00
Tal Levy
203d1e5fc7 raise error when formatting time without @timestamp field
Closes #2730

Fixes #2736
2015-03-09 22:57:39 +00:00
Tal Levy
9a4bcf1f10 do not wrap existing Password type into another Password type when validating
Fixes #2767
2015-03-04 22:29:08 +00:00
Joao Duarte
6b326945a3 Revert "handle LogStash::Gemfile file handle closing"
This reverts commit a473fd6a6f.

Fixes #2748
2015-03-03 14:17:19 +00:00
Pier-Hugues Pellerin
de27d9f0a0 Use warn level instead of error level when a plugin is using the milestone method in the plugin
Fixes #2735
2015-03-03 00:19:13 +00:00
Joao Duarte
057238eb0f handle LogStash::Gemfile file handle closing
Fixes #2740
2015-03-03 00:07:49 +00:00
Colin Surprenant
1ccfd1cf0f changed working
Fixes #2739
2015-03-02 23:06:22 +00:00
Colin Surprenant
1254573a43 show update status
Fixes #2739
2015-03-02 23:06:22 +00:00
Colin Surprenant
1dd82abee0 comment fix
Fixes #2739
2015-03-02 23:06:21 +00:00
Pier-Hugues Pellerin
d98f2b45cb We need to call bundler_setup! before getting the list of the installed gems if we want to support the :path and :git option in the gemfile
Fixes #2732
2015-03-02 20:10:04 +00:00
Pere Urbon-Bayes
d44b58bcdc add an option to bundler_setup! to handle testing environment where development dependencies should not be excluded in order to run proper tests
as a request during review, I made the signature of bundler_setup use the same as invoke_bundler

remove empty line in bundler_setup

ammend last commit

Fixes #2726
2015-03-02 16:05:54 +00:00
Colin Surprenant
4319119e60 set BUNDLE_GEMFILE before requiring bundler
Fixes #2721
2015-03-02 14:53:27 +00:00
Colin Surprenant
6840da52f8 also set Gemfile path in ENV
Fixes #2717
2015-02-27 21:06:42 +00:00
Pier-Hugues Pellerin
f97c01da06 Call bundle clean after doing a bundle update to remove unused gems
Fixes #2703
2015-02-26 23:25:31 +00:00
Colin Surprenant
001c609179 correctly support :clean option
Fixes #2703
2015-02-26 23:25:30 +00:00
Colin Surprenant
4edbf5d32b correctly override & init bundler settings
Fixes #2703
2015-02-26 23:25:30 +00:00
Pier-Hugues Pellerin
b376d913bd When we do a uninstall we are actually doing a bundle install --clean
Fixes #2703
2015-02-26 23:25:29 +00:00
Pier-Hugues Pellerin
7c7c0c9efe Make sure we expand the path to make Dir.glob work on windows if we provide a relative file .\hello
Fixes #2680
2015-02-25 01:07:56 +00:00
Colin Surprenant
d805cc76aa verify Rubygems versions before monkeypatching
Fixes #2684
2015-02-25 00:54:54 +00:00
Colin Surprenant
5e9644b7a5 always set BUNDLE_WITHOUT
Fixes #2685
2015-02-25 00:45:03 +00:00
Joao Duarte
c0558b0318 safeguard against uninitialized DEBUGLIST variable
Fixes #2668
2015-02-24 19:44:13 +00:00
Joao Duarte
9561ade035 group patches and load after environment setup
Fixes #2668
2015-02-24 19:44:13 +00:00
Colin Surprenant
e470807399 no bad bundler please
Fixes #2671
2015-02-24 18:19:04 +00:00
Colin Surprenant
584b87d01a ::File namespace
Fixes #2671
2015-02-24 18:19:04 +00:00
Colin Surprenant
2291498162 setup bundler env when no .bundle/config
Fixes #2671
2015-02-24 18:19:04 +00:00
Colin Surprenant
c1ea865893 fix BUNDLE_PATH
Fixes #2671
2015-02-24 18:19:03 +00:00
Pier-Hugues Pellerin
786d2d2c1a Fallback to a local configuration if the parsing for remote configuration fails
Fixes #2653
2015-02-23 15:46:09 +00:00
Joao Duarte
6230084c20 ensure all strings pulled out of ENV are UTF8 on windows
strings assigned to LogStash::Event fields must be UTF-8 and
since most windows aren't UTF-8 by default we must intercept all
get operations and convert the strings to UTF-8. Issue surfaces in
logstash-filter-environment, for example.

Fixes #2666
2015-02-23 15:13:28 +00:00
Aaron Mildenstein
3986158bac First pass at cleaning out logstash web
All internal references, flags, kibana, vendor, everything must go!

If it referenced kibana or logstash-web, it is deprecated, and removed!

Operators are standing by...

Fixes #2661
2015-02-20 22:53:18 +00:00
Pier-Hugues Pellerin
9e7d6204a5 plugins/bundler refactor
squashed 113 commits by @ph, @colinsurprenant, @jordansissel, @purbon, @jsvd

cleaning the environment.rb

initial work

remove multiples polyglot

force to load rubygems

notice

point to one config

update lock

bootstrap will package the standard application until we have the plugin manager on top of the gemfile

add the gem path to find bundler

fixing dependencies

add gemfiles and bundle

comment platform requiremments in the gemfile

Set `Gem.paths = ...` to the GEM_HOME

Without this, `bin/logstash` will error:

    LoadError: no such file to load -- bundler
         require at org/jruby/RubyKernel.java:1071
         require at C:/Users/jls/Documents/GitHub/logstash/vendor/jruby/lib/ruby/shared/rubygems/core_ext/kernel_require
.rb:55
         set_gem_paths! at C:/Users/jls/Documents/GitHub/logstash/lib/logstash/environment.rb:64
         (root) at C:\Users\jls\Documents\GitHub\logstash\lib\logstash\runner.rb:8

Patch JRuby's jar_installer.rb with a newer version

jar-dependencies is shipped by default with JRuby. However, the version
it ships with has a bug, and despite upgrading jar-dependencies to
0.1.7, which has the fix, JRuby continues to load the "shipped" version
which has a bug.

This patch will patch JRuby by copying in our own jar_installer.rb from
jar-dependencies master, This patching occurs while unpacking JRuby just
after `vendor:jruby`.

The jar_installer.rb is MIT-licensed and no changes were made.

Add `test:plugins` to test all installed plugins

Tell rspec to do the wildcards instead of trying to make the shell do it.

This fixes test runs on Windows (which doesn't do wildcards like this).
The solution was to tell RSpec to look for all files matching a certain path pattern
within the GEM_HOME.

do not run bundle install twice for test:plugins

Gemfile manipulation classes

Gemfile now in root

install POC with new Gemfile classes

specs for the Gemfile classes

header, some bug fixes

spec header

WIP pluginmanager refactor

generated format

updated .lock

unused file

removed --proxy option

fail elegantly

initial support for update

refactor invoke_bundler, debug output

removed old .lock file

no plugin Gemfile

use clamp error handling

reenable list, some cleanups

support for more finders

refactored using new finders

refactor using is_installed_plugin? and some cleanups

verify valid plugin and update all now only update plugins not all gems

support multiple plugins argument, --force option to bypass validation

list only explicitly installed plugins by default, --all option to include dependencies

support multiple plugins argument

renamed all_installed_plugins to all_installed_plugins_gem_specs and return specs, added comments

remove then

update_gem bug, comments

moved gems as development dependencies

updated .lock file

support for :without

--development option, better output feedback, cleanups

abort when no plugins

cosmetic

param fix, cleanups

removed deprecated .lock files

remove deprecated bundler_install_command and struggling with set_gem_paths

refactored all bundler stuff into LogStash::Bundler

rakelib refactor

refactored bundler code

Bundler.real_fuckin_reset

real_fuckin_reset is now reset_settings, removed unnecessary resetings, cleanups

duplicate plugins requirements merging

cleanups

added vendor:rake, cleanups

broke test tasks into 2 steps, first invoke install then test

rake task name fix

proper error exit code

Freezing the jar-dependencies, ruby-maven, tools

set gem paths on error

renamed TEST_PLUGINS to CORE_PLUGINS, added comments

replace test by core, cleanups

DRYied test tasks

missing gemspec

fixed output

correct image build for artifacts

removed deprecated Gemfiles

fix update with multiple plugins arguments

fix update with multiple plugins arguments

remove deprecated code

implicit begin, explicit block

spec for LogStash::Bundler

remove explit require of rubygems

renamed Environment.set_gem_paths! to bundler_setup!

rename Dsl to DSL

respect to_s contract and return empty string not nil

rename DSL.parse gemfile argument to gemfile_content

better error message

renamed --force to --[no-]verify

refactored exit error handling

use explicit array items in iterators

fix --development switch validation, cleaned bundler_options hash init

renamed is_xxx? methods to xxx?

use direct method call on collection

removed double negative, typo

move maven tools monkey patch into maven_tools_patch.rb

renamed build-image to prepare

fix forgotten argument name change

namespace cleanups

spec files glob pattern

cleanup

cleanup

cleanup

Fix the file-dependencies version to 0.1.5 to fix path issues on windows

monkeypatch Rubygems to silence ffi warning

list all plugin gems by default, new options

adding rake task to get faster feedback loop on windows

refactor bat scripts

Use an unreleased filedependencies gem and skip installing drupal

bump file-dependencies

use the released gem of file-dependencies

Add a set of scripts that simplify:   * The necessary setup for the CI test to be executed.   * The actuall CI test execution, having a different scripts for each     different workflow. This makes the test execution simpler and standard, so changes in internal task don't need big changes in the CI config. It also documents and makes easy for developers to run the tests.

Fixes #2640

refactored test to not depend on STDOUT

create a .rspec file to pass some options used while the code is in the testing environment

add coveralls and simplecov as a development dependencies so we can run coverage tests

remove unnecessary .bundle directory from the artifact

In a release, the .bundle/config has a BUNDLE_PATH setting that can
interfere by misleading the plugin manager into not seeing the
vendor/bundle path.

Fixes #2647

Fixes #2475
2015-02-20 17:31:43 +00:00
Joao Duarte
6b15234b9e monkeypatch Socket on jruby+windows to return UTF-8 strings
Fixes #2507
2015-02-12 14:15:46 +00:00
Joao Duarte
c4a6f41ca6 fix encoding issue on windows+jruby
Fixes #2507
2015-02-12 14:15:46 +00:00
Chris Moultrie
351d8c1ada ungz was named incorrectly
Fixes #2572
2015-02-10 21:45:00 +00:00
Brian Low
bb5cc0173f Gracefully shutdown the agent on SIGTERM. Fixes #1675
Fixes #1676
2015-02-02 08:49:54 +00:00
Pier-Hugues Pellerin
ccb98bc843 allow uninstalling gems with bundler
Fixes #2450
2015-01-27 22:02:22 +00:00
Joao Duarte
e7d617f703 update jar_dependencies
Fixes #2445
2015-01-27 20:26:45 +00:00
Colin Surprenant
21270a2e07 set BUNDLE_PATH env to overwrite any .bundler/config and fix Bundler.setup
Fixes #2449
2015-01-27 20:23:00 +00:00
Pier-Hugues Pellerin
890c1bc3eb Use the .info level when printing the version notice for the plugin
Fixes #2429
2015-01-27 17:40:38 +00:00
Tal Levy
0be04a45bf make log4j setup more explicit which appender to append to
Fixes #2432
2015-01-27 01:15:47 +00:00
Pere Urbon-Bayes
cd414f189b Added two Logger.terminal calls to report start and shutdown status when there is a STDOUT registered
Fixes #2409
2015-01-26 21:11:01 +00:00
wiibaa
f62fd92b9b last step for #1229 2015-01-26 06:28:21 +01:00
Suyog Rao
c62d70ea44 Mutate: Check for nil before attempting to remove Fixes #2379
Fixes #2384
2015-01-22 17:26:04 +00:00
Pier-Hugues Pellerin
07f7882ccf fix merge conflicts
Fixes #2280
2015-01-21 22:08:11 +00:00
Pier-Hugues Pellerin
b680921dae Gem in the gemfile should have higher priority in the $LOAD_PATH
Fixes #2375
2015-01-21 15:29:20 +00:00
Pier-Hugues Pellerin
803ff7bab7 LOGSTASH_ENV=development vendor:gems will install gem that use the :git or :github option.
Fixes #2375
2015-01-21 15:29:20 +00:00
Pier-Hugues Pellerin
d9b049f74b You can override the BUNDLE_GEMFILE in your environment.
Fixes #2375
2015-01-21 15:29:20 +00:00
Pier-Hugues Pellerin
3cb1aa8658 Allow to define gem in the gemfile with the path option
Fixes #2375
2015-01-21 15:29:20 +00:00