Commit graph

58 commits

Author SHA1 Message Date
Pere Urbon-Bayes
2195f145b2 This adds a feature to let users dump all their installed plugins,
including dependencies, and reuse them in an offline installation by
providing an package for it. It adds two important commands to the
plugin manager, the pack and upack, to handle package creation and
installation and adds the --local flag to install and update to pull
plugins from the installed local packages.

Former commits:

add a task to create a bundle, under vendor/cache of the installed gems + their dependencies, this can be used later on to be installed offline

add an option to pass the --local flag to the bin/plugin update task, so it fetch plugins from a local cache, by default under vendor/cache

rename package:build to package:build-cache as is more meaningfull

add a --local flag to bin/plugin install to users can install plugins from the local cache, under the default directory of vendor/cache

add a plugin manager command to build the local cache of installed plugins using bundler package alike command

It adds code for compressing and extracting in zip and tar formats to
the utils module of logstash. The zip module is only jruby dependant as
it uses functions provided by java.
There is also code add in the plugin manager package command to handle
compression of plugins dumping.

Cleanup the custom java code to compress and extract zip files as it has
been known that using rubyzip is good and it has a more ruby like
features so the code is more clean and portable.

set of smallish improvement requested during review

added several options to handle situation when the package of plugins we want to generate is already there

clean up old code

applyed feedback from review, mostly changed in documentating behaviour plus better wording for messages

relocate the Environment.windows? check to the bootstrap side as it's used also in the plugin manager

add an unpack bin/plugin command so users can install packages of plugins throw the plugin manager

document override behaviour in the compress helpers (for zip and tar) plus add a fix for the tar extract when reading entries

made the unpack filename parameter a required one

add a force option to the bin/plugin unpack command

add a notice to that if using a local cache the --local flag should be passed with

Code cleanup and refactor introduced during review

add two wording suggestions comming from review

ammend more wording

skip the major version validation as in situation where there is no internet connection like when using the local vendor/cache to do the update

move compress to the bootstrap environment as being used in the plugin manager means not being loaded with logstash-core

Bring pack cached gems in the vendor directory so they can be used for bundler when packaging dependencies

Revert "Bring pack cached gems in the vendor directory so they can be used for bundler when packaging dependencies"

This reverts commit a9d7f46649932b06efaafebdd0eed2b4c63c2235.

patch the Bundler::Source::Rubygems to fetch only gems from a remote  source

small changes in several parts of the plugin manager and the creation of a common pack command with shared code

change compress to read in chuncks

fix wrong var name in the bootstrap compress utils module

fix namespacing conflicts

add basic test for the compress utility module

ammend more namespace issues

add a comment to the rubygems mockey patch to rebuild the gem cache

apply cosmetic changes

make the compress module raise CompressError

remove vendor_path and pattern_path form environment as they where mix up during rebase

change the bin/pack force option to be force-delete-cache

rename force_delete_cache to overwrite

change force for overwrite in tha pack command

remove the override option for bin/plugin unpack

revert Gemfile and Genfile.lock wrong committed
2015-11-18 12:03:51 +01:00
Pere Urbon-Bayes
ba96b1f0b3 Fix the plugin manager install and update commands to perform the validation properly when a private gem server (not rubygems) has been configured. This basically loads the sources so the system is able to perform the validation with the right destionation and not with the default source (rubygems.org)
Former commits:

add all defined sources to rubygems so verification can talk to all the repositories, even the private ones

added a very simple plugin manager install command spec, to check for properly loading sources when validating

broke long lines into smaller ones in the install command

make sure the update command takes into account all declared sources, plus there is the option to avoid gem validation in case of only having a private gem repo and no internet connection

fix wrong conditional in the validation of plugins

move the Gem.sources loading to a common place, to sources are loaded all at once and we don't need to worry on each call to be sure we do the right thing for other sources different that rubygems

update test to the last changes in the way gemfile is loaded

move Gem::SourceList.from to bundler::invoke! so we use the default gemset.sources array

fix loading path for Gem::SourceList in bundler, so it's not colliding with other classes

Revert "move the Gem.sources loading to a common place, to sources are loaded all at once and we don't need to worry on each call to be sure we do the right thing for other sources different that rubygems"

This reverts commit 6e1c809665d61495b5461e7522a7adc745fe27fc.

Revert "update test to the last changes in the way gemfile is loaded"

This reverts commit dc5f65c2ac5380b1640cb732946f1a8ababcc9dc.

make the Gem.sources load to happen inside the logstash_plugin? call

add the idea of settings to the plugin manager module, like this we can pass throw different stuff usefull to setup dependant components

add the install spec back

remove the PluginManager::Settings concept

change sources for rubygems_sources in the plugin manager options

change rubygems_sources to be rubygems_source

update comments

ammend plugin manager options description in the comments

spaces and new lines cleanup

merged duplicated plugin manager util_spec introduced during rebase

add a check when Gems.versions fail

Add the validation exception for the plugin manager

add better error handling for situation where the validation is not possible due to a connection issue with the remote server

Fixes #3583
2015-11-17 08:38:55 +00:00
Colin Surprenant
d74d41cb30 logstash-core & logstash-core-event extraction to support logstash-core-event-java impl, relates to #4191
fixed timezone issue

extracted logstash-core and reorganized specs

extracted logstash-core-event

extract java Event into logstash-core-event-java in a proper gem

remove obsolete jruby_event bootstrapping

fix require path

add java code bootstrap

use logstash-core-event/logstash-core-event.rb

remove obsolete files

basic instructions

LogStash::Json need to be initialized from event

update jruby and gradle versions

update compile:logstash-core-event-java rake task

WIP tasks refactor

fix gem.files

skip test if class is not defined

fix gem related tasks for new structure

add gem spec dirs in core tests

bootstrap java implementation when requiring timestamp

new Cloner class and Event clone impl

fix array fields assignments, see #4140

don't rely on json implementation ordering

fix skipped last iterpolation char

remove implementation specific unnecessary check

also require ruby classes

define error class in ruby

raise exception on invalid format

remove implementation specific tests and extract and put logger related test in pending

missing bits for having all core timestamp specs pass

run all core specs

remove leftover

comment regex

missing encoding header

revert to logstash-core-event by default

finished proper gemification

useless require

dynamically pick specs depending on logstash-core-event-* implementation

logstash root package version

missing file for proper gemification

do not build java event by default

always check for root logstash lib dir

fix concurrent-ruby version confict

fix rebase conflict

re-enable specs

user vars instead of constants

move non core code in bootstrap

document version files

move version file

remove useless code

use version in logstash-core

fix gem files list

put back concurrent-ruby version constrain as in master

add dependency on logstash-core-event

remove dependency on logstash-core to avoid circular dependency

fix rebase conflict

remove circular dependency

fix specs

update README
2015-11-16 16:40:19 -05:00
Pere Urbon-Bayes
028d76497c add the missing magic header # encoding: utf-8 so all internal strings are UTF-8 in Ruby < 2.0
Fixes #3723
2015-08-17 13:30:07 +00:00
Pier-Hugues Pellerin
6d02988142 Better implementation of the no .bundle/config that respect the bundler contract in a running process
The previous fix for disabling the `.bundle/config` wasn't not respecting the bundler contract.
The failling tests were not wrong, in fact they exposed the issue that the configuration was transient and the underlying hash of bundler was not correctly keeping the updated values.

This patch make sure the hash is updated with the new or deleted value without persisting the change to disk.

Fixes #3332
2015-05-28 00:52:53 +00:00
Pier-Hugues Pellerin
0bb827c4fe Make sure we correctly patch the thin bootstrap layer of the distribution
Fixes #2701
2015-05-27 17:14:00 +00:00
Colin Surprenant
7a1ec64c23 use :build dependencies
fix octokit dep

fixes #3171
2015-05-06 08:27:57 -04:00
Colin Surprenant
48bcc5b74d cleanup dependencies and rely on logstash-core gem
remove unused load path

use either local core lib dir or logstash-core gem

include spec in logstash-core

do not include lib, spec and locales dirs in package root

environment bootstrapping in lib/bootstrap.rb

added comments

move pluginmanager out of logstash-core

kafla installation solved by pluginmanager refactor

refactor bootstrap code

refactor shell scripts to simplify and DRY, indirect rspec command to our ruby launcher

add bin/rspec

cut by half the bin/plugin and bin/rspec startup time

fix drip context

use printf instead of tr

updated Windows bin scripts

missing gemspec in gemspec

use gem instead of gemspec so our plugin manager can correctly install logstash-core

generate packages including locally built logstash-core gem

move jenkins developmnent dependencies into Gemfile, they do not belong in logstash-core

path leftover

clean help agent help usage message and remore rspec in available command

comments cosmetics

update Bundler dependency, all recent testing have been made with 1.9.4

updated .lock file with regard to updated Gemfile

cleanup gemspec, Gemfile and regen .lock file

added progress output

avoid dual require on version

closes #3096
2015-05-04 18:17:22 -04:00