Commit graph

224 commits

Author SHA1 Message Date
Andrew Cholakian
3da597f4f3 Bump minitar version for security fix even though we aren't vulnerable
Fixes #8714
2017-11-22 20:22:38 +00:00
Colin Surprenant
b6435c187c fix the test:plugins execution 2017-10-12 15:06:46 -04:00
Rob Bavey
2fdb917018 Fixup rake:version task
Remove no longer nececessary sync task
Simplify other tasks to reflect this removal
Change update_version_file to keep comments in yaml file after task run

Fixes #8410
2017-09-29 20:48:05 +00:00
Pier-Hugues Pellerin
32a62b6c6c Add debug message when a plugin is not found when generating the NOTICE File
Fixes #8404
2017-09-27 13:34:19 +00:00
Guy Boertje
63f8f86bde bump JrJackson/Jackson versions, source all version refs from versions.yml
final (I hope) fixes for consolidated versioning.

After testing with rake artifact:zip, need to try both files.
Add desc to rake test:install-* tasks, tired of
  not seeing them in rake -vT

changes requested via review

Fixes #8373

Fixes #8389
2017-09-26 17:48:35 +00:00
Jordan Sissel
7df379599f Add commercially supported plugins to be default
Our intent is to ship all commercially supported[1] plugins with
Logstash. By some mistakes, we have omitted a few plugins, and this
commit resolves those omissions.

Fixes #8287

Fixes #8318
2017-09-22 19:50:59 +00:00
Jake Landis
7c6450cd59 Introduce docker for CI builds.
This commit includes:
* A base Dockerfile and script to push to a Docker repo
* A per-build Dockerfile (derived from the base)
* Updates to the test scripts to allow for more parallel builds
* Docker wrappers for the tests scripts
* Update for the integration test readme to manually run the tests
* Clean up the output of the Java tests
* Remove offline tag for tests (no longer needed that we don't use docker dependent services)

This commit does NOT include:
* Changes needed for the CI system to use Docker

Fixes #8223

Fixes #8263
2017-09-14 20:39:39 +00:00
Rob Bavey
079cfe5bc1 Remove CHANGELOG.md from list of packaged files
CHANGELOG.md has been removed, so it should no longer be packaged

Fixes #8255

Fixes #8256
2017-09-14 17:46:57 +00:00
Colin Surprenant
d88e125b54 remove unsued windows code 2017-09-14 10:53:48 -04:00
Colin Surprenant
bfdb49643b add jruby-runtime-override in versions.yml, update jruby version to 9.1.13.0, relax jruby-ssl version to >= 0.9.20
add option for different jruby runtime version

relax jruby-ssl version to >= 0.9.20

rename jruby-runtime to jruby-runtime-override and add sha1 IGNORE option

use JRuby released version of 9.1.13.0
2017-09-07 12:17:08 -04:00
Guy Boertje
da5e3c711c Modules cloud id auth (#8059)
* add newlines to generated json

* Implement cloud.id and cloud.auth settings merge to module settings

* Fixes from review plus convert to using Password for any Module Setting

* Review changes

* update modules.asciidoc to include a section on Cloud

* Capitalize Id

* remove unnecessesary require lines
2017-08-28 08:52:04 -07:00
Armin
10a72ce015 #7700 ignore ENV JAVA_OPTS in rake tasks
Fixes #7785
2017-07-24 15:44:52 +00:00
Suyog Rao
23b8d4d948 Add translate filter to default filter list (#7713) 2017-07-17 10:43:24 -07:00
Magnus Bäck
ca1962f697 rakelib: List log4j2.properties as a config file in RPM/Deb (#7688)
When a file is listed as a configuration file the package manager
will ask the user about what to do about local modifications when
the package is upgraded and the file has changed upstream. If not
listed as a configuration file any local modifications will silently
be lost.
2017-07-14 13:33:33 -05:00
Suyog Rao
33d1164901 Update default plugins list (#7669)
Based on google analytics from docs pageviews, we are updating the bundled
plugins to remove plugins which doesn't have to be in the default package.

They can still be installed via bin/logstash-plugin
2017-07-12 21:43:07 +02:00
Pier-Hugues Pellerin
39630a986f Simplier algorithm to retrieve the installable plugins
Using a recursive algorithm was causing issue with SSL connection to
rubygems when we were manually settings variable to NIL.

This commit uses a simple loop and will try to install the plugins in
sequence and it will give a bit more information at the STDOUT level.

Fixes #7562
2017-06-28 23:17:28 +00:00
Pier-Hugues Pellerin
461d2058a9 Fix an OOM issue when generating the plugins dependence json
Make sure the unused variables are freed in the recursive method call.

Fixes #7544
2017-06-26 21:16:26 +00:00
Jordan Sissel
a355d9f09d Use plugin name instead of metadata because sometimes Bundler-provided Gem::Specifications have nil metadata.
Fixes #7530
2017-06-23 20:09:24 +00:00
Jordan Sissel
70fa6f2d57 Only include Logstash plugins in list of plugins
When generating list of compatible plugins for documentation, we
previously were accidentally include all gems, not just logstash
plugins.

Fixes #7521
2017-06-22 22:24:47 +00:00
Pier-Hugues Pellerin
0332ee5087 update the set strategy
Make sure when you set a logstash version that we are not wiping
the content of the versions file but instead merge it so we preserve
jruby information.

Fixes #7434
2017-06-13 15:44:32 +00:00
Pier-Hugues Pellerin
6a4766a01f Centralize the JRuby version
We have multiple places to declare the jruby version, this commit
centralize everything into the `version.yml` file. This is now used by
the ruby part and the gradle build file to correctly fetch the jruby
library.

Fixes #7434
2017-06-13 15:44:31 +00:00
Pier-Hugues Pellerin
7dcf9ad763 Update to JRuby 9.1.10.0
Update from 9.1.9.0 to 9.1.10.0

Fixes #7425
2017-06-13 12:44:00 +00:00
Pier-Hugues Pellerin
76286b4f0e use jruby 9.1.9.0
Work done by @guyboertje and @ph

Since JRuby 1.7.25 is now EOL we are migrating Logstash to use JRuby 9k and JDK8 only,
Not much needed updating to make this work, its was mostly a drop in replacement from the previous version.

The major point was the change in the implementation of Time in JRuby, JRuby now use `java.time`
instead of joda time, this allow JRuby to have nanoseconds precision on time object.
2017-06-12 12:35:10 -04:00
Armin
9b922e7e9b #7375 plugin install rake task should depend on bootstrap task
Fixes #7415
2017-06-12 11:06:45 +00:00
Pier-Hugues Pellerin
42e9c0d758 Extraction of the document from ruby to actual asciidoc file
This PR is based on the proposal from #6846, it does the following

- Change the format of the plugin help page
- Create an external file for each plugin type that will get included by the plugin
- Add a script to generate the PR on each plugin repo with the static doc
- Add a script `ci/ci_docs.sh` that will be used to generate the list of plugins that can be installed in logstash from all the available plugins from the community organization

This PR is the building block for change in https://github.com/elastic/docs-tools/issues/3

Fixes #6941
2017-06-08 20:14:36 +00:00
Armin
12264efad2 #7329 invoker script for convert jar
Fixes #7335
2017-06-07 07:17:53 +00:00
Suyog Rao
7733dd0f86 Add DLQ to default plugins (#7342) 2017-06-06 11:52:12 -07:00
Aaron Mildenstein
f443dae5fe Merge feature/modules into master (#7284)
* My changes (#7218)

* First upstream PR commit (#7172)

No tests yet.  Just for code review for now

* move all inner classes to their own folder + client and importer

* Fixes and tests (#7228)

Add tests for the `LogStash::Modules:CLIParser` class in `cli_parser.rb`
Fix a typo in `cli_parser.rb` (`uparsed` vs `unparsed`)
Fix a bad variable name found by testing in `cli_parser.rb` and update the error message accordingly in `en.yml`

* Remove fb_modules (#7280)

* fixes to import index-pattern & var updates & savedsearch capability (#7283)

* fixes to import index-pattern & var updates & savedsearch capability

fixes to import index-pattern & var updates

add savedsearch capability

* minimise merge conflicts with PR End-to-End test with filebeat apache2

* End-to-End test with filebeat apache2 (#7279)

This is a first run, but data flows from filebeat through Elasticsearch.

Template uploads from `$LS_HOME/modules/MODULENAME/configuration/elasticsearch/MODULENAME.json`

Specifying `--modules filebeat` from the command-line, with `-M "filebeat.var.elasticsearch.output.host=localhost:9200"`

Some of the saved searches don't get uploaded. @guyboertje is on this already.

The logstash configuration needs tweaking to allow receiving both access logs _and_ error logs.  The dashboards and visualizations all seem to expect the presence of both.

Set default to `localhost` in `elasticsearch_client.rb`

Changed command-line variable parsing to allow for a variable with only `modulename.key.subkey=value`, and updated the error message accordingly.

First draft of the filebeat module, as extracted from filebeat 5.4.0

* Add documentation for Modules

This is specific to the Master branch.  Multiple modules will not be supported in 5.5.

* Add READMEs and prune post-code comments

* Add comment regarding the variable name `modul`

Also, fix the default username for the Elasticsearch output in Logstash.  The default x-pack credentials are `elastic:changeme` rather than `elasticsearch:changeme`

* add cef module files (#7292)

* fixes from reviews of PR #7284
2017-06-05 18:49:35 +01:00
Wainer dos Santos Moschetta
4163af5bea Add test case for FileLockFactory
This change introduce a test for cross-process/jvm file locking
using the FileLockFactory class. It uses a client program that
will lock a file aftwards the test case proceed.

Fixes #6610

Fixes #7117
2017-05-24 02:20:36 +00:00
Andrew Cholakian
b595b54362 Move rake test:* tasks to all use bin/rspec.
This eliminates the differences in behavior between bin/rspec and rake
test:*

Fixes #7140
2017-05-17 19:53:53 +00:00
Andrew Cholakian
558e3e9575 Make gradle assemble the default compile:all command.
This well help catch javadoc errors during dev + build

Fixes #7108
2017-05-15 21:36:25 +00:00
Andrew Cholakian
1187d03eb7 Run gradle tests in 'info' mode to log failures
Fixes #7110
2017-05-15 21:18:58 +00:00
Armin
4994bf3bc5 #7043 run gradle tests in core specs and document running gradle tests only in readme
Fixes #7099
2017-05-15 15:58:45 +00:00
Pier-Hugues Pellerin
4c3a3ad526 Normalize the Alpha version of Logstash
If you use 6.0.0-alpha1, rubygems will change it to 6.0.0.pre.alpha1,
this commit changes the behavior to normalize the version before saving
it to disk.

Fixes #7008
2017-05-04 17:16:55 +00:00
Colin Surprenant
e732255edc add acked_queue concurrent stress spec
introduce rake test:core-fast and rake test:core-slow

remove custom rspec config
2017-04-21 14:37:49 -06:00
Andrew Cholakian
ab1cfe8cf7 Make pipeline and agent available through execution context
Fixes #6920
2017-04-18 20:04:38 +00:00
Pier-Hugues Pellerin
a7187fbe8f Update the rake version:sync to ignore logstash-core-event
With the gem refactoring we don't build the logstash-core-event anymore

Fixes #6911
2017-04-12 14:08:40 +00:00
Andrew Cholakian
76711a4785 LIR UI (Read Only) (#6241)
* Read-only Java IR
* Consistent ID generation as opposed to UUIDs

Fixes #6708
2017-04-03 08:49:00 -04:00
Josh Soref
a06dc211ff Spelling fixes (#6806)
Spelling fixes across the board.
2017-03-29 19:33:26 -07:00
Tal Levy
9974df427d remove old legacy logstash-core-event (#6783)
and migrate spec tests to core
2017-03-02 10:55:30 -08:00
Tal Levy
2e3b06b812 migrate core-queue-jruby into logstash-core (#6782) 2017-03-02 10:12:03 -08:00
Tal Levy
6fb8096a54 migrate logstash-core-event-java to logstash-core (#6760) 2017-03-01 15:31:17 -08:00
Tal Levy
bd8b05d676 unify gradle projects (#6725) 2017-02-27 13:58:01 -08:00
Joao Duarte
8741568f4e make dissect filter a default plugin
Fixes #6729 Fixes #6629
2017-02-18 11:30:39 -08:00
Jordan Sissel
ef272cfe4b Pin minitar. Works around https://github.com/halostatue/minitar/issues/23
Fixes #6659
2017-02-07 19:40:00 -05:00
Colin Surprenant
d79e3730fe exclusive file locking class and tests
use new bin/ruby and bundler without :development

refactor to DRY and use expected exception

added original Apache 2.0 license and some cosmetics

exclude bin/lock from packaging

rename variables
2017-01-31 13:47:07 -05:00
Suyog Rao
8c4d2ab5d3 Generate notice file during artifact build (#6538)
* Generate notice file during artifact build

Adjust our existing NOTICE.txt file to be built every time. This ensures
we collect LICENSE and NOTICE information of all bundled Logstash and non-Logstash
dependencies.

Changes:

* if NOTICE file is present in gems, include that text in our own NOTICE.TXT file.
* we also need to include the whole copyright block in NOTICE (meaning copyright + license)
2017-01-16 10:09:02 -08:00
Suyog Rao
733a546e9e Build queue along with other gems
Fixes #6473
2017-01-03 14:03:59 -05:00
Pier-Hugues Pellerin
80fcadd931 Allow rake version:set to update the logstash-core-queue-jruby gem
Fixes #6380
2016-12-08 14:29:28 -05:00
Jordan Sissel
74e32a6b97 Add comment referring to another research effort
Fixes #6276
2016-12-06 13:17:36 -05:00