This will now put ENV variables from the `startup.options` file into `/etc/default/logstash` or `/etc/sysconfig/logstash` (or whatever service name you chose), and use the updated pleaserun to ensure these are honored at start time for whichever init system you use (systemd, upstart, SysV).
fixes#6482
This new command replace the old workflow of `pack`, `unpack` and the `install --local`, and wrap all the logic into one uniform way of installing plugins.
The work is based on the flow developed for installing an x-pack inside Logstash, when you call prepare-offline-pack, the specified plugins and their dependencies will be packaged in a zip.
And this zip can be installed with the same flow as the pack.
Definition:
Source Logstash: Where you run the prepare-offline-pack.
Target Logstash: Where you install the offline package.
PROS:
- If you install a .gem in the source logstash, the .gem and his dependencies will be bundled.
- The install flow doesn't need to have access to the internet.
- Nothing special need to be setup in the target logstash environment.
CONS:
- The is one minor drawback, the plugins need to have their JARS bundled with them for this flow to work, this is currently the case for all the official plugins.
- The source Logstash need to have access to the internet when you install plugins before packaging them.
Usage examples:
bin/logstash-plugin prepare-offline-pack logstash-input-beats
bin/logstash-plugin prepare-offline-pack logstash-filter-jdbc logstash-input-beats
bin/logstash-plugin prepare-offline-pack logstash-filter-*
bin/logstash-plugin prepare-offline-pack logstash-filter-* logstash-input-beats
How to install:
bin/logstash-plugin install file:///tmp/logstash-offline-plugins-XXXX.zip
Fixes#6404
**Motivation**: We have decided to rewrite the documentation generator to add more features
to our documentation and to fix some outstanding bugs.
This PR addresses the following problem:
- Hard to add new features to the code base.
- The code was tied with Logstash.
- No easy way to publish the doc for a single plugin
- The plugin author was not able to test their documentation changes
- The reported errors were hard to understand.
- No easy way to automate it.
- the minimal requirement on base classes.
Fixes#5720
A pack in this context is a *bundle* of plugins that can be distributed outside of rubygems; it is similar to what ES and kibana are doing, and
the user interface is modeled after them. See https://www.elastic.co/downloads/x-pack
**Do not mix it with the `bin/logstash-plugin pack/unpack` command.**
- it contains one or more plugins that need to be installed
- it is self-contains with the gems and the needed jars
- it is distributed as a zip file
- the file structure needs to follow some rules.
- As a reserved name name on elastic.co download http server
- `bin/plugin install logstash-mypack` will check on the download server if a pack for the current specific logstash version exist and it will be downloaded, if it doesn't exist we fallback on rubygems.
- The file on the server will follow this convention `logstash-mypack-{LOGSTASH_VERSION}.zip`
- As a fully qualified url
- `bin/plugin install http://test.abc/logstash-mypack.zip`, if it exists it will be downloaded and installed if it does not we raise an error.
- As a local file
- `bin/plugin install file:///tmp/logstash-mypack.zip`, if it exists it will be installed
Fixes#6168
Purpose:
* manage releases through a minimum number of rake tasks
* simplify building of snapshot builds
* create staged artifacts, candidates for releases, that required no changes to become releases
* this means the snapshot release process will not involve publishing gems, therefore:
* the gem artifacts should only be published to rubygems as a final artifact, at the time of GA
Changes:
* release artifacts no longer depend on gems of core components
* all core components are used locally AS-IS to minimize code changes between snapshot, RC and GA
* `versions.yml` describes the versions of all logstash parts and package
* `rake version:set[version]` manage the yaml file and push the changes to the gemspecs/version.rb files
* `rake version:set_plugin_api[version]` manage the yaml file and push the changes to the gemspecs/version.rb files
* `rake artifact:all` generates SNAPSHOT artifacts: tar.gz, zip, rpm, deb
* `RELEASE=1 rake artifact:all` creates release candidate artifacts + 4 gems: logstash-core, logstash-core-event, logstash-core-event-java and logstash-core-plugin-api
implements #5416 and #5414Fixes#5460
Update after-install scripts
Use /usr/lib/logstash instead of /opt/logstash
More revisions.
Using `/usr/share/logstash` instead of `/usr/lib/logstash`
Track these new files
More work in progress.
Proper arg parsing obtained
WIP: Use jvm.options like ES
Remove facter dependency
Due to an update from @jordansissel on pleaserun, facter is no longer necessary
Revert Gemfile to proper version
Latest work in progress.
I added a simple test to logstash.lib.sh to catch empty strings in java options.
Added `startup.options`, and touched up `jvm.options`
Edited the package scripts to make things play nice with new paths.
Tweaked `system-install` to allow users to provide their own `startup.options` file.
Patch up merge failure
Changes to work with the new logstash.yml
Add fpm dependency (build fails without it)
Fix missing gems in Gemfile
Don't know how these got lost
Use /etc/logstash for configuration in packages
Update the instructions/documentation for...
...startup.options
Merging proposed changes
Remove mustache and clamp
They're covered by pleaserun anyway
Prune config dir from /usr/share/logstash...
...but only when using RPM/DEB packages
refactor wip gemfiles
refactor Java Event getter and setter
bump plugin-api to 2.0
use plugin-api 2.0
switch to core-event-java
include logstash-core-event-java.jar jar file so that gem dependency using the source tree work
updated core plugins to core-api 2.0
added grok for refactor branch
fix rebased specs
remove temp plugins github paths
remove commented out tmp alias_method
We have decided to create a gem that the plugins can depend on to make
sure they use the correct core, with that in mind this will reduce the
mass update required when releasing a major release of logstash if the
contract between logstash and the plugin didnt change. The first version
of this gem act as a virtual gem, future release will include the actual
plugins code of the contract.
This gem should follow the same convention as semver, a breaking change
in the api will mean a bump in the major version. Plugins author are
encouraged to test their plugins with a specific version of the contract
and define their dependency as the following:
```
s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.0"
```
This PR also introduce rake task to help managing this gem, since every
release of logstash will produce a new version of this gem with the
minor version bump. A **logstash-core-plugin-api** will declare a strict
dependency on a specific **logstash-core** version, like this:
```
s.add_runtime_dependency "logstash-core", "2.3.0"
```
On the other hand each release of logtash will also declare a strict
dependency on a specific version of the `logstash-core-plugin-api` gem.
ref #4829Fixes#4866
As discussed in this adds:
/
/_node/hot_threads
/_node/stats/
/_node/stats/events
/_node/stats/jvm
/_stats/jvm
introduces also small refactors and cleanup necessary to improve the
webapi code quality.
adds also a way to handle references from the webapi to the internals of
logstash by passing the agent refrences to the collector, this is not
perfect, but for now it solve the communication situation until furder
refactor.
Fixes#4652Fixes#4653
This commit introduce the collection of the main metrics inside the
logstash pipeline and records it to the metric store. This code is also
the initial stone to define an internal metric api.
Collected metrics from the pipeline and system will be exposed by the web
api.
This work was done in collaboration by @ph and @purbon, commits were
squashed to simplify the merging process.
Fixes#4653
cleaned dependencies, removed Guava code
default to java event
also compile java event
use json-unit for json result assertions
remove Java 8 specific constructs to be able to compile on Java 7
use jar-dependencies
better jar file loading
copy jar into lib after build
add post-build copy and clean of gem jar and re-indented file
fix jar destination
include generated jar in gem
remove useless ArrayList constructions
add tests for KeyNode.join method
remove useless shadow and disable unchecked warnings
update gradle dependency to 2.8
fix deprecated asserEquals usage
simplify convoluted list contruction
fix array fields nil values handling
moar nil value tests
simplify for loop
fixed timezone issue
extracted logstash-core and reorganized specs
extracted logstash-core-event
extract java Event into logstash-core-event-java in a proper gem
remove obsolete jruby_event bootstrapping
fix require path
add java code bootstrap
use logstash-core-event/logstash-core-event.rb
remove obsolete files
basic instructions
LogStash::Json need to be initialized from event
update jruby and gradle versions
update compile:logstash-core-event-java rake task
WIP tasks refactor
fix gem.files
skip test if class is not defined
fix gem related tasks for new structure
add gem spec dirs in core tests
bootstrap java implementation when requiring timestamp
new Cloner class and Event clone impl
fix array fields assignments, see #4140
don't rely on json implementation ordering
fix skipped last iterpolation char
remove implementation specific unnecessary check
also require ruby classes
define error class in ruby
raise exception on invalid format
remove implementation specific tests and extract and put logger related test in pending
missing bits for having all core timestamp specs pass
run all core specs
remove leftover
comment regex
missing encoding header
revert to logstash-core-event by default
finished proper gemification
useless require
dynamically pick specs depending on logstash-core-event-* implementation
logstash root package version
missing file for proper gemification
do not build java event by default
always check for root logstash lib dir
fix concurrent-ruby version confict
fix rebase conflict
re-enable specs
user vars instead of constants
move non core code in bootstrap
document version files
move version file
remove useless code
use version in logstash-core
fix gem files list
put back concurrent-ruby version constrain as in master
add dependency on logstash-core-event
remove dependency on logstash-core to avoid circular dependency
fix rebase conflict
remove circular dependency
fix specs
update README
