Commit graph

238 commits

Author SHA1 Message Date
Colin Surprenant
48bcc5b74d cleanup dependencies and rely on logstash-core gem
remove unused load path

use either local core lib dir or logstash-core gem

include spec in logstash-core

do not include lib, spec and locales dirs in package root

environment bootstrapping in lib/bootstrap.rb

added comments

move pluginmanager out of logstash-core

kafla installation solved by pluginmanager refactor

refactor bootstrap code

refactor shell scripts to simplify and DRY, indirect rspec command to our ruby launcher

add bin/rspec

cut by half the bin/plugin and bin/rspec startup time

fix drip context

use printf instead of tr

updated Windows bin scripts

missing gemspec in gemspec

use gem instead of gemspec so our plugin manager can correctly install logstash-core

generate packages including locally built logstash-core gem

move jenkins developmnent dependencies into Gemfile, they do not belong in logstash-core

path leftover

clean help agent help usage message and remore rspec in available command

comments cosmetics

update Bundler dependency, all recent testing have been made with 1.9.4

updated .lock file with regard to updated Gemfile

cleanup gemspec, Gemfile and regen .lock file

added progress output

avoid dual require on version

closes #3096
2015-05-04 18:17:22 -04:00
Pere Urbon-Bayes
a01ffd4ee6 slim down the skip list from plugins that are now released and accesible
Fixes #3128
2015-04-29 06:25:25 +00:00
Colin Surprenant
3dabba80fa revert multi_filter removal and add check for cancelled events
add filter and flush compiles functions specs

missing multiline filter for core specs

fixes #3100
2015-04-27 13:54:18 -04:00
Colin Surprenant
c70f252fe4 remove unwanted files in package 2015-04-22 18:49:35 +02:00
Colin Surprenant
4e1246a5b5 remove freeze task 2015-04-22 18:49:21 +02:00
Colin Surprenant
b4c3d1ef33 upgrade jruby to 1.7.19
Fixes #3066
2015-04-21 15:44:35 +00:00
Pere Urbon-Bayes
0b72d06de8 add an rspec shim to complement the old bin/logstash rspec way of running specs in logstash
simplify the rspec shim codebase

add bin/rspec to the exclude list for packaging

removed leftover

Fixes #3047
2015-04-20 15:47:46 +00:00
Colin Surprenant
bde2691239 bin/bundle shim
use  instead for $:

exclude bin/bundle from packages

added comment
2015-04-20 17:07:52 +02:00
Colin Surprenant
ae17b4160d refactored all gem/bundler code into logstash/bundler.rb and moved patches into logstash/patches
require bundler/cli to expose expectations classes
2015-04-16 17:34:51 +02:00
Pere Urbon-Bayes
9e2ae1589e add a check for a plugin to be in rubygems before going to install it
make sure dependency gems is installed

cleanup unnecessary code

Fixes #3021
2015-04-15 14:31:39 +00:00
Pere Urbon-Bayes
058e9967a7 update the default plugins exclude list for currently empty plugins
Fixes #3017
2015-04-15 09:07:12 +00:00
Colin Surprenant
380ea33708 refactor test tasks, test:plugins now correctly run specs of installed plugins, including local ones
robustness in shutdown handling and other thread safeties
2015-04-08 08:43:49 -04:00
Pere Urbon-Bayes
fb88eef749 add a check for the exit code, so if the exit code is not zero we make the rake task to fail otherwise it pass always causing false positive
move to use exit insted of fail as it aborts the test execution

add a comment to clarify why we set the exit code in the rake task

Fixes #2920
2015-03-31 08:57:10 +00:00
Pier-Hugues Pellerin
cd4fb38fb5 Add elasticsearch_http output plugin for easy upgrade path
Fixes #2885
2015-03-24 16:25:21 +00:00
Pier-Hugues Pellerin
782e5a2497 wrong
Fixes #2861
2015-03-23 16:14:49 +00:00
Pier-Hugues Pellerin
643627d3a6 Heartbeat input was missing from the official distribution
Fixes #2849
2015-03-18 22:59:32 +00:00
Colin Surprenant
f1e3f057ee handle defauls Gemfile and .lock file
Fixes #2766
2015-03-04 20:29:06 +00:00
Pere Urbon-Bayes
d44b58bcdc add an option to bundler_setup! to handle testing environment where development dependencies should not be excluded in order to run proper tests
as a request during review, I made the signature of bundler_setup use the same as invoke_bundler

remove empty line in bundler_setup

ammend last commit

Fixes #2726
2015-03-02 16:05:54 +00:00
Pier-Hugues Pellerin
560edd07ee add jdbc input to the skiplist
Fixes #2716
2015-02-27 20:38:13 +00:00
Pier-Hugues Pellerin
34f2812fd2 add logstash-output-logentries to the skiplist when doing install-all until the plugin is actually published
Fixes #2716
2015-02-27 20:38:13 +00:00
Pere Urbon-Bayes
cc93c32be5 add the logstash-core.gem to the artifact build
Fixes #2708
2015-02-27 15:37:01 +00:00
Aaron Mildenstein
3986158bac First pass at cleaning out logstash web
All internal references, flags, kibana, vendor, everything must go!

If it referenced kibana or logstash-web, it is deprecated, and removed!

Operators are standing by...

Fixes #2661
2015-02-20 22:53:18 +00:00
Pier-Hugues Pellerin
9e7d6204a5 plugins/bundler refactor
squashed 113 commits by @ph, @colinsurprenant, @jordansissel, @purbon, @jsvd

cleaning the environment.rb

initial work

remove multiples polyglot

force to load rubygems

notice

point to one config

update lock

bootstrap will package the standard application until we have the plugin manager on top of the gemfile

add the gem path to find bundler

fixing dependencies

add gemfiles and bundle

comment platform requiremments in the gemfile

Set `Gem.paths = ...` to the GEM_HOME

Without this, `bin/logstash` will error:

    LoadError: no such file to load -- bundler
         require at org/jruby/RubyKernel.java:1071
         require at C:/Users/jls/Documents/GitHub/logstash/vendor/jruby/lib/ruby/shared/rubygems/core_ext/kernel_require
.rb:55
         set_gem_paths! at C:/Users/jls/Documents/GitHub/logstash/lib/logstash/environment.rb:64
         (root) at C:\Users\jls\Documents\GitHub\logstash\lib\logstash\runner.rb:8

Patch JRuby's jar_installer.rb with a newer version

jar-dependencies is shipped by default with JRuby. However, the version
it ships with has a bug, and despite upgrading jar-dependencies to
0.1.7, which has the fix, JRuby continues to load the "shipped" version
which has a bug.

This patch will patch JRuby by copying in our own jar_installer.rb from
jar-dependencies master, This patching occurs while unpacking JRuby just
after `vendor:jruby`.

The jar_installer.rb is MIT-licensed and no changes were made.

Add `test:plugins` to test all installed plugins

Tell rspec to do the wildcards instead of trying to make the shell do it.

This fixes test runs on Windows (which doesn't do wildcards like this).
The solution was to tell RSpec to look for all files matching a certain path pattern
within the GEM_HOME.

do not run bundle install twice for test:plugins

Gemfile manipulation classes

Gemfile now in root

install POC with new Gemfile classes

specs for the Gemfile classes

header, some bug fixes

spec header

WIP pluginmanager refactor

generated format

updated .lock

unused file

removed --proxy option

fail elegantly

initial support for update

refactor invoke_bundler, debug output

removed old .lock file

no plugin Gemfile

use clamp error handling

reenable list, some cleanups

support for more finders

refactored using new finders

refactor using is_installed_plugin? and some cleanups

verify valid plugin and update all now only update plugins not all gems

support multiple plugins argument, --force option to bypass validation

list only explicitly installed plugins by default, --all option to include dependencies

support multiple plugins argument

renamed all_installed_plugins to all_installed_plugins_gem_specs and return specs, added comments

remove then

update_gem bug, comments

moved gems as development dependencies

updated .lock file

support for :without

--development option, better output feedback, cleanups

abort when no plugins

cosmetic

param fix, cleanups

removed deprecated .lock files

remove deprecated bundler_install_command and struggling with set_gem_paths

refactored all bundler stuff into LogStash::Bundler

rakelib refactor

refactored bundler code

Bundler.real_fuckin_reset

real_fuckin_reset is now reset_settings, removed unnecessary resetings, cleanups

duplicate plugins requirements merging

cleanups

added vendor:rake, cleanups

broke test tasks into 2 steps, first invoke install then test

rake task name fix

proper error exit code

Freezing the jar-dependencies, ruby-maven, tools

set gem paths on error

renamed TEST_PLUGINS to CORE_PLUGINS, added comments

replace test by core, cleanups

DRYied test tasks

missing gemspec

fixed output

correct image build for artifacts

removed deprecated Gemfiles

fix update with multiple plugins arguments

fix update with multiple plugins arguments

remove deprecated code

implicit begin, explicit block

spec for LogStash::Bundler

remove explit require of rubygems

renamed Environment.set_gem_paths! to bundler_setup!

rename Dsl to DSL

respect to_s contract and return empty string not nil

rename DSL.parse gemfile argument to gemfile_content

better error message

renamed --force to --[no-]verify

refactored exit error handling

use explicit array items in iterators

fix --development switch validation, cleaned bundler_options hash init

renamed is_xxx? methods to xxx?

use direct method call on collection

removed double negative, typo

move maven tools monkey patch into maven_tools_patch.rb

renamed build-image to prepare

fix forgotten argument name change

namespace cleanups

spec files glob pattern

cleanup

cleanup

cleanup

Fix the file-dependencies version to 0.1.5 to fix path issues on windows

monkeypatch Rubygems to silence ffi warning

list all plugin gems by default, new options

adding rake task to get faster feedback loop on windows

refactor bat scripts

Use an unreleased filedependencies gem and skip installing drupal

bump file-dependencies

use the released gem of file-dependencies

Add a set of scripts that simplify:   * The necessary setup for the CI test to be executed.   * The actuall CI test execution, having a different scripts for each     different workflow. This makes the test execution simpler and standard, so changes in internal task don't need big changes in the CI config. It also documents and makes easy for developers to run the tests.

Fixes #2640

refactored test to not depend on STDOUT

create a .rspec file to pass some options used while the code is in the testing environment

add coveralls and simplecov as a development dependencies so we can run coverage tests

remove unnecessary .bundle directory from the artifact

In a release, the .bundle/config has a BUNDLE_PATH setting that can
interfere by misleading the plugin manager into not seeing the
vendor/bundle path.

Fixes #2647

Fixes #2475
2015-02-20 17:31:43 +00:00
Jordan Sissel
b7455cbb2b Delete the zipfile before building it.
If we don't' do this, Zip::File.new and despite the Zip::File::CREATE
flag, will reopen an existing zip file instead of writing a new one.

The error you get without this patch is

    rake aborted!
    add failed. Entry logstash-2.0.0.dev/LICENSE already exists
    /Users/jls/projects/logstash/vendor/bundle/jruby/1.9/gems/rubyzip-1.1.6/lib/zip/file.rb:392:in
    `check_entry_exists'

Fixes #2456
2015-01-28 00:16:15 +00:00
Pere Urbon-Bayes
d479015abb fix a typo on the test all plugins rake task
Fixes #2441
2015-01-27 17:17:21 +00:00
Jordan Sissel
6fad79ca18 Provide our own ssl certs on Windows Ruby
This fixes a bug where Windows Ruby ships with insufficient ssl
certificate information and this causes gem installations to fail. This
patch is only applied under Windows ruby because JRuby is not affected
by it.

Fixes #2402

Fixes #2403
2015-01-27 01:11:04 +00:00
wiibaa
f62fd92b9b last step for #1229 2015-01-26 06:28:21 +01:00
Suyog Rao
01ad1c5123 Add couchdb and es_bulk to default plugins
Fixes #2389
2015-01-21 22:18:09 +00:00
Pere Urbon-Bayes
002104f3c8 Added review comments and improvements
fix the logstash_gem_path

added some love from @jsvd comments

added an LS_ENV variable that if set to test it install the plugin dependencies when running the plugin:install-all rake task2

made the vendor:deps task has a more concret behaviour and proper naming

cleanup the all plugins install to only do the installation

added some code to fetch dependencies throw some gemcode within the gemfile as @electrical proposed

Revert "added some code to fetch dependencies throw some gemcode within the gemfile as @electrical proposed"

This reverts commit 20c3243485129c388d1da4ac930a479f017e0d99.

fix a path on the run all plugins test rake task

Fixes #2371
2015-01-21 16:47:26 +00:00
Pere Urbon-Bayes
6e63f6c6b2 added way to add development dependencies to a gemfile and then rerun it
Fixes #2371
2015-01-21 16:47:26 +00:00
Pier-Hugues Pellerin
3cb1aa8658 Allow to define gem in the gemfile with the path option
Fixes #2375
2015-01-21 15:29:20 +00:00
Pere Urbon-Bayes
f3912f81db Applyed all necessary review improvements fix the gempath
update the test:plugins to test:all-plugins at it makes more sense

fix the path to run the bin logstash file

added order rand

Fixes #2373
2015-01-19 19:41:46 +00:00
Pere Urbon-Bayes
ec97d3d78c add a way to run all plugin test from a rake command
Fixes #2373
2015-01-19 19:41:46 +00:00
Suyog Rao
5d7119ff55 With #2320, Kafka was removed from the default install list to save ~30mb from the LS release package.
We had a discussion internally and arguments were made for having Kafka included back to make it consistent with the other packaged plugins. Moving forward (post v1.5), the idea is to have Logstash ship with a small list of plugins by default to reduce the install size. This would also allow users to only install plugins which are needed for their use-case. This is made easier by the recent plugin management infrastructure which can add/remove plugins at any time. Once we have more data on which plugins are popular, we will open an issue to only include those in the default package. Until then Kafka stays

Fixes #2343
2015-01-15 19:23:22 +00:00
Suyog Rao
7209eaec03 Remove kafka input and output from default plugins packaging
Fixes #2320
2015-01-07 10:09:50 +00:00
Pere Urbon-Bayes
188b6a2aba fix the exclude variable naming
Fixes #2285
2015-01-05 15:50:16 +00:00
Pere Urbon-Bayes
3472fabb3a move the specs cleaning for logstash plugins to packaging time
Fixes #2285
2015-01-05 15:50:15 +00:00
Pere Urbon-Bayes
ab0a654be8 add code to remove files within gems after bundler gets call
Fixes #2285
2015-01-05 15:50:15 +00:00
Pere Urbon-Bayes
2ec759a676 rafactor the jruby exclude filters plus add the rdoc shared into the list
Fixes #2285
2015-01-05 15:50:15 +00:00
Pere Urbon-Bayes
05a097f853 Make sure the vendorized jruby only have 1.9 libs, so no extra space is used. Also make sure the jruby is run in 1.9 mode.
Fixes #2285
2015-01-05 15:50:15 +00:00
Pier-Hugues Pellerin
4448e37fa0 Cleaning gems dependencies from the logstash.gemspec
Fixes #2274
2014-12-23 20:18:03 +00:00
wiibaa
395cb7563b Update for windows to execute rake bootstrap and bin\logstash.bat
Fixes #2085
2014-12-22 21:46:25 +00:00
Pier-Hugues Pellerin
738e0b59eb Removing the gembag and the download of the vendored files since the logic was moved into their specific gems.
Fixes #2272
2014-12-19 23:34:31 +00:00
Richard Pijnenburg
f83652aada Refactor doc generation
- Use a generic task for installing all plugins
- Use refactored bundle install task

Fixes #2258
2014-12-17 19:56:31 +00:00
Pere Urbon-Bayes
b1a4c102da make sure the fail fast test also have the bootstrap for tests done
Fixes #2237
2014-12-17 17:31:32 +00:00
Pere Urbon-Bayes
d63e0ec9eb make sure the bundler task is just used once while bootstraping tests
Fixes #2237
2014-12-17 17:31:32 +00:00
Pere Urbon-Bayes
93aadb4d4c Make the test plugins install throw bundler, so we don't have to install them every time we run the test (it makes the thing slow). Refactor also the bundler install task into a single one, making calls to it when necessary.
Fixes #2237
2014-12-17 17:31:32 +00:00
Richard Pijnenburg
60493acbd0 Add doc generation tasks to logstash
Fixes #2256
2014-12-17 16:57:04 +00:00
Pere Urbon-Bayes
21cb877ec9 clear the gem paths before the test rake tasks so they find previously installed plugins
Fixes #2235
2014-12-17 12:52:29 +00:00
Richard Pijnenburg
0f185a470e Add creation of zipfile for the artifacts
Fixes #2221
2014-12-16 20:29:33 +00:00