Commit graph

238 commits

Author SHA1 Message Date
Andrew Cholakian
83ecabe153 Bump minitar version for security fix even though we aren't vulnerable
Fixes #8714
2017-11-22 20:22:38 +00:00
Rob Bavey
9191f61d2d Split shared doc version update into separate rake task
The stack version constant in index-shared1.asciidoc should not be
updated during a version bump, as this impacts the current version
of the doc on elastic.co. Instead, create a separate rake task
to bump the version.

See #8637 for details of this happening

Fixes #8648
2017-11-14 14:10:21 +00:00
Armin
8b8d5ec9ca TESTS: Improve Gradle Test Configuration
* `clean` actually cleans all dynamically created files, in particular it now properly cleans the generated Gemfile so
that changes to the Gemfile.template reflect in a rerun of `bundler`
* `rubyTests` and `test` are now one-off and will automatically bootstrap JRuby and Gems required by the tests if necessary
* Fixed Readme to document the now much simpler test targets
* All rake tasks remain unchanged and still work exactly as they did before

Fixes #8583
2017-11-06 18:34:47 +00:00
infcurious@gmail.com
f03f1b7775 Update set_plugic_api task to use latest update_version_file
Fixes #8496
2017-11-01 13:23:20 +00:00
infcurious@gmail.com
359f69ea67 Use old_version & new_version naming for readability
Fixes #8496
2017-11-01 13:23:20 +00:00
infcurious@gmail.com
c785daf697 Expand rake version:set to update README & index-shared1.asciidoc
Fixes #8496
2017-11-01 13:23:20 +00:00
Armin
835826085b BUILD: Execute RSPEC from JUnit
Fixes #8517
2017-10-26 19:57:54 +00:00
Armin
1dad58c2b5 #8280 remove simplecov
Fixes #8536
2017-10-26 12:15:12 +00:00
Armin
cd3b91dc5c #8216 Use standard name for Gemfile.lock
Fixes #8525
2017-10-26 08:33:03 +00:00
Armin
2037f5b031 MINOR: Removed dead RBX reference
Fixes #8526
2017-10-25 10:51:14 +00:00
Colin Surprenant
12acb39056 fix the test:plugins execution 2017-10-12 15:06:12 -04:00
Rob Bavey
7049e61c26 Fixup rake:version task
Remove no longer nececessary sync task
Simplify other tasks to reflect this removal
Change update_version_file to keep comments in yaml file after task run

Fixes #8410
2017-09-29 20:48:04 +00:00
Pier-Hugues Pellerin
3a08171c3a Add debug message when a plugin is not found when generating the NOTICE File
Fixes #8404
2017-09-27 13:34:19 +00:00
Guy Boertje
5a9e670c4b bump JrJackson/Jackson versions, source all version refs from versions.yml
final (I hope) fixes for consolidated versioning.

After testing with rake artifact:zip, need to try both files.
Add desc to rake test:install-* tasks, tired of
  not seeing them in rake -vT

changes requested via review

Fixes #8373

Fixes #8388
2017-09-26 17:47:39 +00:00
Jordan Sissel
43b6fb891a Add commercially supported plugins to be default
Our intent is to ship all commercially supported[1] plugins with
Logstash. By some mistakes, we have omitted a few plugins, and this
commit resolves those omissions.

Fixes #8287

Fixes #8318
2017-09-22 19:50:59 +00:00
Pier-Hugues Pellerin
9184e360d2 remove vendor plugin
Fixes #8281
2017-09-18 18:57:50 +00:00
Pier-Hugues Pellerin
0468a07082 remove all trace of install all plugins in the code and in the build scripts
Fixes #8281
2017-09-18 18:57:49 +00:00
Pier-Hugues Pellerin
4ac9b97617 Remove uneeded task to install all the possible plugins from the logstash-plugin organization
This command was error prone, some plugins on the logstash-plugin
organization are currently broken.

Fixes #8281
2017-09-18 18:57:49 +00:00
Pier-Hugues Pellerin
7b09be4ead Bootstrap JRuby using gradle
Instead of using rake to bootstrap we are slowly moving part of our
infra to gradle this is the first step taken, I didn't change the calls,
I just made the `rake bootstrap` call the `gradle bootstrap`

Closes: #8234

Fixes #8238
2017-09-18 14:32:10 +00:00
Jake Landis
1340f2eee0 Introduce docker for CI builds.
This commit includes:
* A base Dockerfile and script to push to a Docker repo
* A per-build Dockerfile (derived from the base)
* Updates to the test scripts to allow for more parallel builds
* Docker wrappers for the tests scripts
* Update for the integration test readme to manually run the tests
* Clean up the output of the Java tests
* Remove offline tag for tests (no longer needed that we don't use docker dependent services)

This commit does NOT include:
* Changes needed for the CI system to use Docker

Fixes #8223
2017-09-14 18:55:41 +00:00
Rob Bavey
2d6eef7f5e Remove CHANGELOG.md from list of packaged files
CHANGELOG.md has been removed, so it should no longer be packaged

Fixes #8255

Fixes #8256
2017-09-14 17:46:57 +00:00
Colin Surprenant
192e857c81 remove unsued windows code 2017-09-14 10:53:02 -04:00
Colin Surprenant
5d8c79ad4f add jruby-runtime-override in versions.yml, update jruby version to 9.1.13.0, relax jruby-ssl version to >= 0.9.20
add option for different jruby runtime version

relax jruby-ssl version to >= 0.9.20

rename jruby-runtime to jruby-runtime-override and add sha1 IGNORE option

use JRuby released version of 9.1.13.0
2017-09-07 12:17:32 -04:00
Guy Boertje
e2b2e7d979 Modules cloud id auth (#8059)
* add newlines to generated json

* Implement cloud.id and cloud.auth settings merge to module settings

* Fixes from review plus convert to using Password for any Module Setting

* Review changes

* update modules.asciidoc to include a section on Cloud

* Capitalize Id

* remove unnecessesary require lines
2017-08-28 08:51:21 -07:00
Suyog Rao
c0a224798b remove rake benchmark target
Fixes #7831
2017-07-27 19:40:41 +00:00
Armin
10a72ce015 #7700 ignore ENV JAVA_OPTS in rake tasks
Fixes #7785
2017-07-24 15:44:52 +00:00
Suyog Rao
23b8d4d948 Add translate filter to default filter list (#7713) 2017-07-17 10:43:24 -07:00
Magnus Bäck
ca1962f697 rakelib: List log4j2.properties as a config file in RPM/Deb (#7688)
When a file is listed as a configuration file the package manager
will ask the user about what to do about local modifications when
the package is upgraded and the file has changed upstream. If not
listed as a configuration file any local modifications will silently
be lost.
2017-07-14 13:33:33 -05:00
Suyog Rao
33d1164901 Update default plugins list (#7669)
Based on google analytics from docs pageviews, we are updating the bundled
plugins to remove plugins which doesn't have to be in the default package.

They can still be installed via bin/logstash-plugin
2017-07-12 21:43:07 +02:00
Pier-Hugues Pellerin
39630a986f Simplier algorithm to retrieve the installable plugins
Using a recursive algorithm was causing issue with SSL connection to
rubygems when we were manually settings variable to NIL.

This commit uses a simple loop and will try to install the plugins in
sequence and it will give a bit more information at the STDOUT level.

Fixes #7562
2017-06-28 23:17:28 +00:00
Pier-Hugues Pellerin
461d2058a9 Fix an OOM issue when generating the plugins dependence json
Make sure the unused variables are freed in the recursive method call.

Fixes #7544
2017-06-26 21:16:26 +00:00
Jordan Sissel
a355d9f09d Use plugin name instead of metadata because sometimes Bundler-provided Gem::Specifications have nil metadata.
Fixes #7530
2017-06-23 20:09:24 +00:00
Jordan Sissel
70fa6f2d57 Only include Logstash plugins in list of plugins
When generating list of compatible plugins for documentation, we
previously were accidentally include all gems, not just logstash
plugins.

Fixes #7521
2017-06-22 22:24:47 +00:00
Pier-Hugues Pellerin
0332ee5087 update the set strategy
Make sure when you set a logstash version that we are not wiping
the content of the versions file but instead merge it so we preserve
jruby information.

Fixes #7434
2017-06-13 15:44:32 +00:00
Pier-Hugues Pellerin
6a4766a01f Centralize the JRuby version
We have multiple places to declare the jruby version, this commit
centralize everything into the `version.yml` file. This is now used by
the ruby part and the gradle build file to correctly fetch the jruby
library.

Fixes #7434
2017-06-13 15:44:31 +00:00
Pier-Hugues Pellerin
7dcf9ad763 Update to JRuby 9.1.10.0
Update from 9.1.9.0 to 9.1.10.0

Fixes #7425
2017-06-13 12:44:00 +00:00
Pier-Hugues Pellerin
76286b4f0e use jruby 9.1.9.0
Work done by @guyboertje and @ph

Since JRuby 1.7.25 is now EOL we are migrating Logstash to use JRuby 9k and JDK8 only,
Not much needed updating to make this work, its was mostly a drop in replacement from the previous version.

The major point was the change in the implementation of Time in JRuby, JRuby now use `java.time`
instead of joda time, this allow JRuby to have nanoseconds precision on time object.
2017-06-12 12:35:10 -04:00
Armin
9b922e7e9b #7375 plugin install rake task should depend on bootstrap task
Fixes #7415
2017-06-12 11:06:45 +00:00
Pier-Hugues Pellerin
42e9c0d758 Extraction of the document from ruby to actual asciidoc file
This PR is based on the proposal from #6846, it does the following

- Change the format of the plugin help page
- Create an external file for each plugin type that will get included by the plugin
- Add a script to generate the PR on each plugin repo with the static doc
- Add a script `ci/ci_docs.sh` that will be used to generate the list of plugins that can be installed in logstash from all the available plugins from the community organization

This PR is the building block for change in https://github.com/elastic/docs-tools/issues/3

Fixes #6941
2017-06-08 20:14:36 +00:00
Armin
12264efad2 #7329 invoker script for convert jar
Fixes #7335
2017-06-07 07:17:53 +00:00
Suyog Rao
7733dd0f86 Add DLQ to default plugins (#7342) 2017-06-06 11:52:12 -07:00
Aaron Mildenstein
f443dae5fe Merge feature/modules into master (#7284)
* My changes (#7218)

* First upstream PR commit (#7172)

No tests yet.  Just for code review for now

* move all inner classes to their own folder + client and importer

* Fixes and tests (#7228)

Add tests for the `LogStash::Modules:CLIParser` class in `cli_parser.rb`
Fix a typo in `cli_parser.rb` (`uparsed` vs `unparsed`)
Fix a bad variable name found by testing in `cli_parser.rb` and update the error message accordingly in `en.yml`

* Remove fb_modules (#7280)

* fixes to import index-pattern & var updates & savedsearch capability (#7283)

* fixes to import index-pattern & var updates & savedsearch capability

fixes to import index-pattern & var updates

add savedsearch capability

* minimise merge conflicts with PR End-to-End test with filebeat apache2

* End-to-End test with filebeat apache2 (#7279)

This is a first run, but data flows from filebeat through Elasticsearch.

Template uploads from `$LS_HOME/modules/MODULENAME/configuration/elasticsearch/MODULENAME.json`

Specifying `--modules filebeat` from the command-line, with `-M "filebeat.var.elasticsearch.output.host=localhost:9200"`

Some of the saved searches don't get uploaded. @guyboertje is on this already.

The logstash configuration needs tweaking to allow receiving both access logs _and_ error logs.  The dashboards and visualizations all seem to expect the presence of both.

Set default to `localhost` in `elasticsearch_client.rb`

Changed command-line variable parsing to allow for a variable with only `modulename.key.subkey=value`, and updated the error message accordingly.

First draft of the filebeat module, as extracted from filebeat 5.4.0

* Add documentation for Modules

This is specific to the Master branch.  Multiple modules will not be supported in 5.5.

* Add READMEs and prune post-code comments

* Add comment regarding the variable name `modul`

Also, fix the default username for the Elasticsearch output in Logstash.  The default x-pack credentials are `elastic:changeme` rather than `elasticsearch:changeme`

* add cef module files (#7292)

* fixes from reviews of PR #7284
2017-06-05 18:49:35 +01:00
Wainer dos Santos Moschetta
4163af5bea Add test case for FileLockFactory
This change introduce a test for cross-process/jvm file locking
using the FileLockFactory class. It uses a client program that
will lock a file aftwards the test case proceed.

Fixes #6610

Fixes #7117
2017-05-24 02:20:36 +00:00
Andrew Cholakian
b595b54362 Move rake test:* tasks to all use bin/rspec.
This eliminates the differences in behavior between bin/rspec and rake
test:*

Fixes #7140
2017-05-17 19:53:53 +00:00
Andrew Cholakian
558e3e9575 Make gradle assemble the default compile:all command.
This well help catch javadoc errors during dev + build

Fixes #7108
2017-05-15 21:36:25 +00:00
Andrew Cholakian
1187d03eb7 Run gradle tests in 'info' mode to log failures
Fixes #7110
2017-05-15 21:18:58 +00:00
Armin
4994bf3bc5 #7043 run gradle tests in core specs and document running gradle tests only in readme
Fixes #7099
2017-05-15 15:58:45 +00:00
Pier-Hugues Pellerin
4c3a3ad526 Normalize the Alpha version of Logstash
If you use 6.0.0-alpha1, rubygems will change it to 6.0.0.pre.alpha1,
this commit changes the behavior to normalize the version before saving
it to disk.

Fixes #7008
2017-05-04 17:16:55 +00:00
Colin Surprenant
e732255edc add acked_queue concurrent stress spec
introduce rake test:core-fast and rake test:core-slow

remove custom rspec config
2017-04-21 14:37:49 -06:00
Andrew Cholakian
ab1cfe8cf7 Make pipeline and agent available through execution context
Fixes #6920
2017-04-18 20:04:38 +00:00