Logstash's plugin manager will now follow proxy configuration from the environment.
If you configure `http_proxy` and `https_proxy`, the manager will now use theses information for all the ruby http
connection and will also pass that information down to maven.
Fixes: #6619, #6528Fixes#6825
This new command replace the old workflow of `pack`, `unpack` and the `install --local`, and wrap all the logic into one uniform way of installing plugins.
The work is based on the flow developed for installing an x-pack inside Logstash, when you call prepare-offline-pack, the specified plugins and their dependencies will be packaged in a zip.
And this zip can be installed with the same flow as the pack.
Definition:
Source Logstash: Where you run the prepare-offline-pack.
Target Logstash: Where you install the offline package.
PROS:
- If you install a .gem in the source logstash, the .gem and his dependencies will be bundled.
- The install flow doesn't need to have access to the internet.
- Nothing special need to be setup in the target logstash environment.
CONS:
- The is one minor drawback, the plugins need to have their JARS bundled with them for this flow to work, this is currently the case for all the official plugins.
- The source Logstash need to have access to the internet when you install plugins before packaging them.
Usage examples:
bin/logstash-plugin prepare-offline-pack logstash-input-beats
bin/logstash-plugin prepare-offline-pack logstash-filter-jdbc logstash-input-beats
bin/logstash-plugin prepare-offline-pack logstash-filter-*
bin/logstash-plugin prepare-offline-pack logstash-filter-* logstash-input-beats
How to install:
bin/logstash-plugin install file:///tmp/logstash-offline-plugins-XXXX.zip
Fixes#6404
**Motivation**: We have decided to rewrite the documentation generator to add more features
to our documentation and to fix some outstanding bugs.
This PR addresses the following problem:
- Hard to add new features to the code base.
- The code was tied with Logstash.
- No easy way to publish the doc for a single plugin
- The plugin author was not able to test their documentation changes
- The reported errors were hard to understand.
- No easy way to automate it.
- the minimal requirement on base classes.
Fixes#5720
A pack in this context is a *bundle* of plugins that can be distributed outside of rubygems; it is similar to what ES and kibana are doing, and
the user interface is modeled after them. See https://www.elastic.co/downloads/x-pack
**Do not mix it with the `bin/logstash-plugin pack/unpack` command.**
- it contains one or more plugins that need to be installed
- it is self-contains with the gems and the needed jars
- it is distributed as a zip file
- the file structure needs to follow some rules.
- As a reserved name name on elastic.co download http server
- `bin/plugin install logstash-mypack` will check on the download server if a pack for the current specific logstash version exist and it will be downloaded, if it doesn't exist we fallback on rubygems.
- The file on the server will follow this convention `logstash-mypack-{LOGSTASH_VERSION}.zip`
- As a fully qualified url
- `bin/plugin install http://test.abc/logstash-mypack.zip`, if it exists it will be downloaded and installed if it does not we raise an error.
- As a local file
- `bin/plugin install file:///tmp/logstash-mypack.zip`, if it exists it will be installed
Fixes#6168
including dependencies, and reuse them in an offline installation by
providing an package for it. It adds two important commands to the
plugin manager, the pack and upack, to handle package creation and
installation and adds the --local flag to install and update to pull
plugins from the installed local packages.
Former commits:
add a task to create a bundle, under vendor/cache of the installed gems + their dependencies, this can be used later on to be installed offline
add an option to pass the --local flag to the bin/plugin update task, so it fetch plugins from a local cache, by default under vendor/cache
rename package:build to package:build-cache as is more meaningfull
add a --local flag to bin/plugin install to users can install plugins from the local cache, under the default directory of vendor/cache
add a plugin manager command to build the local cache of installed plugins using bundler package alike command
It adds code for compressing and extracting in zip and tar formats to
the utils module of logstash. The zip module is only jruby dependant as
it uses functions provided by java.
There is also code add in the plugin manager package command to handle
compression of plugins dumping.
Cleanup the custom java code to compress and extract zip files as it has
been known that using rubyzip is good and it has a more ruby like
features so the code is more clean and portable.
set of smallish improvement requested during review
added several options to handle situation when the package of plugins we want to generate is already there
clean up old code
applyed feedback from review, mostly changed in documentating behaviour plus better wording for messages
relocate the Environment.windows? check to the bootstrap side as it's used also in the plugin manager
add an unpack bin/plugin command so users can install packages of plugins throw the plugin manager
document override behaviour in the compress helpers (for zip and tar) plus add a fix for the tar extract when reading entries
made the unpack filename parameter a required one
add a force option to the bin/plugin unpack command
add a notice to that if using a local cache the --local flag should be passed with
Code cleanup and refactor introduced during review
add two wording suggestions comming from review
ammend more wording
skip the major version validation as in situation where there is no internet connection like when using the local vendor/cache to do the update
move compress to the bootstrap environment as being used in the plugin manager means not being loaded with logstash-core
Bring pack cached gems in the vendor directory so they can be used for bundler when packaging dependencies
Revert "Bring pack cached gems in the vendor directory so they can be used for bundler when packaging dependencies"
This reverts commit a9d7f46649932b06efaafebdd0eed2b4c63c2235.
patch the Bundler::Source::Rubygems to fetch only gems from a remote source
small changes in several parts of the plugin manager and the creation of a common pack command with shared code
change compress to read in chuncks
fix wrong var name in the bootstrap compress utils module
fix namespacing conflicts
add basic test for the compress utility module
ammend more namespace issues
add a comment to the rubygems mockey patch to rebuild the gem cache
apply cosmetic changes
make the compress module raise CompressError
remove vendor_path and pattern_path form environment as they where mix up during rebase
change the bin/pack force option to be force-delete-cache
rename force_delete_cache to overwrite
change force for overwrite in tha pack command
remove the override option for bin/plugin unpack
revert Gemfile and Genfile.lock wrong committed
remove unused load path
use either local core lib dir or logstash-core gem
include spec in logstash-core
do not include lib, spec and locales dirs in package root
environment bootstrapping in lib/bootstrap.rb
added comments
move pluginmanager out of logstash-core
kafla installation solved by pluginmanager refactor
refactor bootstrap code
refactor shell scripts to simplify and DRY, indirect rspec command to our ruby launcher
add bin/rspec
cut by half the bin/plugin and bin/rspec startup time
fix drip context
use printf instead of tr
updated Windows bin scripts
missing gemspec in gemspec
use gem instead of gemspec so our plugin manager can correctly install logstash-core
generate packages including locally built logstash-core gem
move jenkins developmnent dependencies into Gemfile, they do not belong in logstash-core
path leftover
clean help agent help usage message and remore rspec in available command
comments cosmetics
update Bundler dependency, all recent testing have been made with 1.9.4
updated .lock file with regard to updated Gemfile
cleanup gemspec, Gemfile and regen .lock file
added progress output
avoid dual require on version
closes#3096
