Commit graph

51 commits

Author SHA1 Message Date
Andres Rodriguez
cf67cb1377
Rubocop: Enable most SpaceInside cops (#15201)
Enabled:
* SpaceInsideArrayLiteralBrackets
* SpaceInsideParens
* SpaceInsidePercentLiteralDelimiters
* SpaceInsideStringInterpolation
* Add enforced style for SpaceInsideStringInterpolation

Enabled without offenses:
* SpaceInsideArrayPercentLiteral
* Layout/SpaceInsideRangeLiteral
* Layout/SpaceInsideReferenceBrackets
2023-07-20 09:49:46 -04:00
Andres Rodriguez
4255a8fd1c
Rubocop: Enable SpaceAround cops (#15196)
* Enable SpaceARoundBlockParameters
* Enable SpaceAroundEqualsInParameterDefault
* Enable SpaceAroundKeyword
* Enable SpaceAroundOperators
* Enable SpaceBeforeBlockBraces, which yields no changes
2023-07-18 21:11:57 -04:00
Andres Rodriguez
acd87a69e7
Rubocop: Enable various EmptyLine cops (#15194)
Disabled:
 * EmptyLineAfterGuardClause
 * EmptyLineAfterMultilineCondition
 * EmptyLinesAroundAccessModifier

Enabled:
 * Layout/EmptyLineAfterMagicComment
 * Layout/EmptyLineBetweenDefs
 * Layout/EmptyLines
 * Layout/EmptyLinesAroundArguments
 * Layout/EmptyLinesAroundAttributeAccessor
 * Layout/EmptyLinesAroundBeginBody
 * Layout/EmptyLinesAroundBlockBody
 * Layout/EmptyLinesAroundExceptionHandlingKeywords
 * Layout/EmptyLinesAroundMethodBody
 * Layout/EmptyLinesAroundModuleBody
2023-07-18 16:49:16 -04:00
Andres Rodriguez
5e34aacc6e
Enable trailing whitespace formating (#15174)
* Enable Layout/TrailingWhitespace cop formation
* Remove Trailing Whitespaces
2023-07-14 13:22:02 -04:00
Andrea Selva
8e62e8a01c
Fix dependency on plugin_alias.yml during rake artifacts:all (#12962)
The module LogStash::PluginManager requires the file `lib/pluginmanager/plugin_aliases.yml` is created,
which happend during the Gradle's 'copyPluginAlias' executed as part of Rake's 'bootstrap'.
2021-06-04 12:37:56 +02:00
João Duarte
e9c9865f40
Add apache and elastic license headers to source code files (#11673)
* add license header to ruby and java files
* add license header to erb and rake files
* add license headers to gradle files
2020-03-11 11:53:38 +00:00
Joao Duarte
0e115d31a6 cleanup many rakelib tasks that aren't used anymore
* simplify the plugins-metadata.json file
* sort and update the plugin list in the rakelib/plugins-metadata.json
* remove dependency on twitter input for testing
* sorted Gemfile.template (grouped by group)
* remove default plugins from Gemfile.template

Fixes #10509
2019-03-15 16:26:58 +00:00
Ry Biesemeyer
583ec6b625
Java 11 support (#10279)
* bump jruby to 9.2

* don't rely on logstash-base docker image

* work around webmock ruby 2.5 support

* ensure data folder exists in docker

* change fixnum and bignum to integer

* FileUtils.rmdir to rm_rf

this is because from 2.3 to 2.5 FileUtils.rmdir will throw an exception
if the directory isn't empty. On 2.3 the operation will just not delete
the directory silently.

* bump jruby to 9.2.5.0 and fix test

* make rake default task since prepare pack needs it

* Resolve compiler warnings (#10247)

There are 3 types of compiler warnings that are either resolved or suppressed:

1. Rawtypes: In JRuby 9.2, `RubyArray` is a generic, so references throughout
   our codebase to the now "raw" type trigger warnings. In most cases we cannot
   actually resolve the issue, since the JRuby-provided methods for creating
   `RubyArray`s still return the raw type, so these have been suppressed.

2. Deprecations:
   - `RubyString#intern19()` -> `RubyString#intern()`
   - `RubyString#downcase19(ThreadContext)` -> `RubyString#downcase(ThreadContext)`
   - `NativeException`: remove import & reference directly; suppress usage
     warnings
   - `RaiseException()`: migrate to equivalent non-deprecated methods wherever
     possible; in some cases where we are using this in conjunction with the
     also-deprecated `NativeException` to preserve java stacktraces, there
     seems to be no non-deprecated path forward, so these cases have been
     suppressed.

3. Redundant Casts
   - Resolved

* JRuby 9.2 bundler shenanigans (#10266)

* Revert "Revert "remove forced dependency on old bundler (#9395)""

This reverts commit bef984143d.

* plugin management: update internal bundler to 1.17.x APIs

* deps: update dev dependency webmock to version compatible with JRuby 9.2

* spec: update Pack fixture to include manticore version that doesn't conflict

* build: update gradle to version that has Java 11 support

* java11: resolve or suppress deprecation warnings

* Remove superfluous flag opting into ParNew GC implementation

When opting into CMS garbage collector with `XX:+UseConcMarkSweepGC`, the
young generation collector ParNew has been the default since Java 8, making
the `XX:+UseParNew` flag redundant; the flag was removed in Java 9, and
should no longer be specified to work with modern Javas.

https://bugs.openjdk.java.net/browse/JDK-8006478
https://openjdk.java.net/jeps/214

* spec: set thread name to example description for easier debugging

* spec: prevent errors in testing specs by checking against skip list before using

* no-op: remove use of `HashMap#computeIfAbsent` on single-threaded code

> This method will, on a best-effort basis, throw a `ConcurrentModificationException`
> if it is detected that the mapping function modifies this map during computation.
>
> -- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/HashMap.html#computeIfAbsent(K,java.util.function.Function)

* qa: by default, run integration against Elastic Stack 6.5.x

To support development on Logstash on top of Java 11, default to testing
against an Elastic Stack that is capable of running on Java 11.

* qa: ignore deprecation warnings when comparing offline pack output

* qa: add Java 9+ support to ChildProcess dev dependency

this can safely be removed when the childprocess gem supports Java9+
https://github.com/enkessler/childprocess/pull/141

* qa: allow connections to localhost in webmock

* bump jrjackson version

* fix filebeat integration tests

* spec: ensure license compliance spec runs first

The license compliance spec that validates the licenses of bundled
plugins appears to not be compatible with the hooks that we inject
into bundler for plugin management, and will fail in obscure ways
when run after those hooks have been added. Since those hooks are
not necessary for validating licenses, the easiest solution was to
ensure that those specs run first, before the VM has been poluted.

Since the gradle/junit/rspec bridge that is currently in place
runs all specs in the same JVM, we also need to make sure that the
rspec "world" is reset before a run, to ensure that it doesn't
retain spec definitions from previous runs.

Also updates the rake invocation, although I'm not sure it is used
any more.
2019-02-04 16:36:11 -08:00
Jake Landis
5c5619ff4e Minor: remove rake test:core-fail-fast
Fixes #8742

Fixes #8763
2017-11-29 19:48:00 +00:00
Armin
f069eac40c BUILD: Execute RSPEC from JUnit
Fixes #8517
2017-10-26 19:57:54 +00:00
Armin
c5fa78c7fa #8280 remove simplecov
Fixes #8536
2017-10-26 12:15:12 +00:00
Colin Surprenant
d9f7789b53 fix the test:plugins execution 2017-10-12 15:05:01 -04:00
Guy Boertje
9b09d4c899 bump JrJackson/Jackson versions, source all version refs from versions.yml
final (I hope) fixes for consolidated versioning.

After testing with rake artifact:zip, need to try both files.
Add desc to rake test:install-* tasks, tired of
  not seeing them in rake -vT

changes requested via review

Fixes #8373
2017-09-26 15:29:58 +00:00
Pier-Hugues Pellerin
0b7cb4a599 remove all trace of install all plugins in the code and in the build scripts
Fixes #8281
2017-09-18 18:57:49 +00:00
Jake Landis
196d1a1f7e Introduce docker for CI builds.
This commit includes:
* A base Dockerfile and script to push to a Docker repo
* A per-build Dockerfile (derived from the base)
* Updates to the test scripts to allow for more parallel builds
* Docker wrappers for the tests scripts
* Update for the integration test readme to manually run the tests
* Clean up the output of the Java tests
* Remove offline tag for tests (no longer needed that we don't use docker dependent services)

This commit does NOT include:
* Changes needed for the CI system to use Docker

Fixes #8223
2017-09-14 18:55:41 +00:00
Pier-Hugues Pellerin
76286b4f0e use jruby 9.1.9.0
Work done by @guyboertje and @ph

Since JRuby 1.7.25 is now EOL we are migrating Logstash to use JRuby 9k and JDK8 only,
Not much needed updating to make this work, its was mostly a drop in replacement from the previous version.

The major point was the change in the implementation of Time in JRuby, JRuby now use `java.time`
instead of joda time, this allow JRuby to have nanoseconds precision on time object.
2017-06-12 12:35:10 -04:00
Andrew Cholakian
b595b54362 Move rake test:* tasks to all use bin/rspec.
This eliminates the differences in behavior between bin/rspec and rake
test:*

Fixes #7140
2017-05-17 19:53:53 +00:00
Andrew Cholakian
1187d03eb7 Run gradle tests in 'info' mode to log failures
Fixes #7110
2017-05-15 21:18:58 +00:00
Armin
4994bf3bc5 #7043 run gradle tests in core specs and document running gradle tests only in readme
Fixes #7099
2017-05-15 15:58:45 +00:00
Colin Surprenant
e732255edc add acked_queue concurrent stress spec
introduce rake test:core-fast and rake test:core-slow

remove custom rspec config
2017-04-21 14:37:49 -06:00
Andrew Cholakian
ab1cfe8cf7 Make pipeline and agent available through execution context
Fixes #6920
2017-04-18 20:04:38 +00:00
Josh Soref
a06dc211ff Spelling fixes (#6806)
Spelling fixes across the board.
2017-03-29 19:33:26 -07:00
Tal Levy
9974df427d remove old legacy logstash-core-event (#6783)
and migrate spec tests to core
2017-03-02 10:55:30 -08:00
Tal Levy
6fb8096a54 migrate logstash-core-event-java to logstash-core (#6760) 2017-03-01 15:31:17 -08:00
Colin Surprenant
761f9f1bc9 merge feature/java_persistence into master 2016-11-01 17:13:23 -04:00
Pier-Hugues Pellerin
22c857556f Remove spec/integration
With the addition of the qa acceptance test suite we have decide that we
don't need to maintain this suite also since the acceptance test are
running really similar test.

Fixes: #5744

Fixes #5760
2016-08-14 15:09:29 -04:00
Pere Urbon-Bayes
bf12902496 refactor integration test to be inside the specs directory and not in the main level
Fixes #5268
2016-05-12 06:56:24 +00:00
Colin Surprenant
d74d41cb30 logstash-core & logstash-core-event extraction to support logstash-core-event-java impl, relates to #4191
fixed timezone issue

extracted logstash-core and reorganized specs

extracted logstash-core-event

extract java Event into logstash-core-event-java in a proper gem

remove obsolete jruby_event bootstrapping

fix require path

add java code bootstrap

use logstash-core-event/logstash-core-event.rb

remove obsolete files

basic instructions

LogStash::Json need to be initialized from event

update jruby and gradle versions

update compile:logstash-core-event-java rake task

WIP tasks refactor

fix gem.files

skip test if class is not defined

fix gem related tasks for new structure

add gem spec dirs in core tests

bootstrap java implementation when requiring timestamp

new Cloner class and Event clone impl

fix array fields assignments, see #4140

don't rely on json implementation ordering

fix skipped last iterpolation char

remove implementation specific unnecessary check

also require ruby classes

define error class in ruby

raise exception on invalid format

remove implementation specific tests and extract and put logger related test in pending

missing bits for having all core timestamp specs pass

run all core specs

remove leftover

comment regex

missing encoding header

revert to logstash-core-event by default

finished proper gemification

useless require

dynamically pick specs depending on logstash-core-event-* implementation

logstash root package version

missing file for proper gemification

do not build java event by default

always check for root logstash lib dir

fix concurrent-ruby version confict

fix rebase conflict

re-enable specs

user vars instead of constants

move non core code in bootstrap

document version files

move version file

remove useless code

use version in logstash-core

fix gem files list

put back concurrent-ruby version constrain as in master

add dependency on logstash-core-event

remove dependency on logstash-core to avoid circular dependency

fix rebase conflict

remove circular dependency

fix specs

update README
2015-11-16 16:40:19 -05:00
Pier-Hugues Pellerin
a30b9a905e Add the possibility to run local integration test
Fixes #4166
2015-11-13 17:16:49 +00:00
Gabriel Moskovicz
abd97aa98b Add powershell script for windows testing
- Release acceptance test, which tests ports and run Logstash.
- Default plugins, which runs the same plugin installation and
environment for windows
- Simple integration, which integrates Logstash with Elasticsearch and a
simple configuration

Fixes #3994
2015-11-06 10:29:57 -08:00
guyboertje
7bb94d0692 add test for compact and compact! in java_integrations
Fixes #3772
2015-09-03 19:43:10 +00:00
Pere Urbon-Bayes
cf10890d63 fix the coverage analysis throw simplecov to take care of all files
make the eager loading patterns consistent in the spec_helper.rb file

make the setup-simplecov task not a dependency, but an explicit task only executed when ENV['COVERAGE'] is defined

refactor eager loading code plus add some documentation to the setup-simplecov task

Added more comments to the test:setup task

Fixes #3465
2015-06-25 15:36:57 +00:00
Pere Urbon-Bayes
e2450f1f78 Make the necessary changes to use rspec3 as a default test runner for
logstash, including adapting the test suites that were still using 2.x
syntax.
2015-06-08 12:19:11 +02:00
Colin Surprenant
7a1ec64c23 use :build dependencies
fix octokit dep

fixes #3171
2015-05-06 08:27:57 -04:00
Colin Surprenant
48bcc5b74d cleanup dependencies and rely on logstash-core gem
remove unused load path

use either local core lib dir or logstash-core gem

include spec in logstash-core

do not include lib, spec and locales dirs in package root

environment bootstrapping in lib/bootstrap.rb

added comments

move pluginmanager out of logstash-core

kafla installation solved by pluginmanager refactor

refactor bootstrap code

refactor shell scripts to simplify and DRY, indirect rspec command to our ruby launcher

add bin/rspec

cut by half the bin/plugin and bin/rspec startup time

fix drip context

use printf instead of tr

updated Windows bin scripts

missing gemspec in gemspec

use gem instead of gemspec so our plugin manager can correctly install logstash-core

generate packages including locally built logstash-core gem

move jenkins developmnent dependencies into Gemfile, they do not belong in logstash-core

path leftover

clean help agent help usage message and remore rspec in available command

comments cosmetics

update Bundler dependency, all recent testing have been made with 1.9.4

updated .lock file with regard to updated Gemfile

cleanup gemspec, Gemfile and regen .lock file

added progress output

avoid dual require on version

closes #3096
2015-05-04 18:17:22 -04:00
Colin Surprenant
ae17b4160d refactored all gem/bundler code into logstash/bundler.rb and moved patches into logstash/patches
require bundler/cli to expose expectations classes
2015-04-16 17:34:51 +02:00
Colin Surprenant
380ea33708 refactor test tasks, test:plugins now correctly run specs of installed plugins, including local ones
robustness in shutdown handling and other thread safeties
2015-04-08 08:43:49 -04:00
Pere Urbon-Bayes
fb88eef749 add a check for the exit code, so if the exit code is not zero we make the rake task to fail otherwise it pass always causing false positive
move to use exit insted of fail as it aborts the test execution

add a comment to clarify why we set the exit code in the rake task

Fixes #2920
2015-03-31 08:57:10 +00:00
Pere Urbon-Bayes
d44b58bcdc add an option to bundler_setup! to handle testing environment where development dependencies should not be excluded in order to run proper tests
as a request during review, I made the signature of bundler_setup use the same as invoke_bundler

remove empty line in bundler_setup

ammend last commit

Fixes #2726
2015-03-02 16:05:54 +00:00
Pier-Hugues Pellerin
9e7d6204a5 plugins/bundler refactor
squashed 113 commits by @ph, @colinsurprenant, @jordansissel, @purbon, @jsvd

cleaning the environment.rb

initial work

remove multiples polyglot

force to load rubygems

notice

point to one config

update lock

bootstrap will package the standard application until we have the plugin manager on top of the gemfile

add the gem path to find bundler

fixing dependencies

add gemfiles and bundle

comment platform requiremments in the gemfile

Set `Gem.paths = ...` to the GEM_HOME

Without this, `bin/logstash` will error:

    LoadError: no such file to load -- bundler
         require at org/jruby/RubyKernel.java:1071
         require at C:/Users/jls/Documents/GitHub/logstash/vendor/jruby/lib/ruby/shared/rubygems/core_ext/kernel_require
.rb:55
         set_gem_paths! at C:/Users/jls/Documents/GitHub/logstash/lib/logstash/environment.rb:64
         (root) at C:\Users\jls\Documents\GitHub\logstash\lib\logstash\runner.rb:8

Patch JRuby's jar_installer.rb with a newer version

jar-dependencies is shipped by default with JRuby. However, the version
it ships with has a bug, and despite upgrading jar-dependencies to
0.1.7, which has the fix, JRuby continues to load the "shipped" version
which has a bug.

This patch will patch JRuby by copying in our own jar_installer.rb from
jar-dependencies master, This patching occurs while unpacking JRuby just
after `vendor:jruby`.

The jar_installer.rb is MIT-licensed and no changes were made.

Add `test:plugins` to test all installed plugins

Tell rspec to do the wildcards instead of trying to make the shell do it.

This fixes test runs on Windows (which doesn't do wildcards like this).
The solution was to tell RSpec to look for all files matching a certain path pattern
within the GEM_HOME.

do not run bundle install twice for test:plugins

Gemfile manipulation classes

Gemfile now in root

install POC with new Gemfile classes

specs for the Gemfile classes

header, some bug fixes

spec header

WIP pluginmanager refactor

generated format

updated .lock

unused file

removed --proxy option

fail elegantly

initial support for update

refactor invoke_bundler, debug output

removed old .lock file

no plugin Gemfile

use clamp error handling

reenable list, some cleanups

support for more finders

refactored using new finders

refactor using is_installed_plugin? and some cleanups

verify valid plugin and update all now only update plugins not all gems

support multiple plugins argument, --force option to bypass validation

list only explicitly installed plugins by default, --all option to include dependencies

support multiple plugins argument

renamed all_installed_plugins to all_installed_plugins_gem_specs and return specs, added comments

remove then

update_gem bug, comments

moved gems as development dependencies

updated .lock file

support for :without

--development option, better output feedback, cleanups

abort when no plugins

cosmetic

param fix, cleanups

removed deprecated .lock files

remove deprecated bundler_install_command and struggling with set_gem_paths

refactored all bundler stuff into LogStash::Bundler

rakelib refactor

refactored bundler code

Bundler.real_fuckin_reset

real_fuckin_reset is now reset_settings, removed unnecessary resetings, cleanups

duplicate plugins requirements merging

cleanups

added vendor:rake, cleanups

broke test tasks into 2 steps, first invoke install then test

rake task name fix

proper error exit code

Freezing the jar-dependencies, ruby-maven, tools

set gem paths on error

renamed TEST_PLUGINS to CORE_PLUGINS, added comments

replace test by core, cleanups

DRYied test tasks

missing gemspec

fixed output

correct image build for artifacts

removed deprecated Gemfiles

fix update with multiple plugins arguments

fix update with multiple plugins arguments

remove deprecated code

implicit begin, explicit block

spec for LogStash::Bundler

remove explit require of rubygems

renamed Environment.set_gem_paths! to bundler_setup!

rename Dsl to DSL

respect to_s contract and return empty string not nil

rename DSL.parse gemfile argument to gemfile_content

better error message

renamed --force to --[no-]verify

refactored exit error handling

use explicit array items in iterators

fix --development switch validation, cleaned bundler_options hash init

renamed is_xxx? methods to xxx?

use direct method call on collection

removed double negative, typo

move maven tools monkey patch into maven_tools_patch.rb

renamed build-image to prepare

fix forgotten argument name change

namespace cleanups

spec files glob pattern

cleanup

cleanup

cleanup

Fix the file-dependencies version to 0.1.5 to fix path issues on windows

monkeypatch Rubygems to silence ffi warning

list all plugin gems by default, new options

adding rake task to get faster feedback loop on windows

refactor bat scripts

Use an unreleased filedependencies gem and skip installing drupal

bump file-dependencies

use the released gem of file-dependencies

Add a set of scripts that simplify:   * The necessary setup for the CI test to be executed.   * The actuall CI test execution, having a different scripts for each     different workflow. This makes the test execution simpler and standard, so changes in internal task don't need big changes in the CI config. It also documents and makes easy for developers to run the tests.

Fixes #2640

refactored test to not depend on STDOUT

create a .rspec file to pass some options used while the code is in the testing environment

add coveralls and simplecov as a development dependencies so we can run coverage tests

remove unnecessary .bundle directory from the artifact

In a release, the .bundle/config has a BUNDLE_PATH setting that can
interfere by misleading the plugin manager into not seeing the
vendor/bundle path.

Fixes #2647

Fixes #2475
2015-02-20 17:31:43 +00:00
Pere Urbon-Bayes
d479015abb fix a typo on the test all plugins rake task
Fixes #2441
2015-01-27 17:17:21 +00:00
Pere Urbon-Bayes
002104f3c8 Added review comments and improvements
fix the logstash_gem_path

added some love from @jsvd comments

added an LS_ENV variable that if set to test it install the plugin dependencies when running the plugin:install-all rake task2

made the vendor:deps task has a more concret behaviour and proper naming

cleanup the all plugins install to only do the installation

added some code to fetch dependencies throw some gemcode within the gemfile as @electrical proposed

Revert "added some code to fetch dependencies throw some gemcode within the gemfile as @electrical proposed"

This reverts commit 20c3243485129c388d1da4ac930a479f017e0d99.

fix a path on the run all plugins test rake task

Fixes #2371
2015-01-21 16:47:26 +00:00
Pere Urbon-Bayes
f3912f81db Applyed all necessary review improvements fix the gempath
update the test:plugins to test:all-plugins at it makes more sense

fix the path to run the bin logstash file

added order rand

Fixes #2373
2015-01-19 19:41:46 +00:00
Pere Urbon-Bayes
ec97d3d78c add a way to run all plugin test from a rake command
Fixes #2373
2015-01-19 19:41:46 +00:00
Pere Urbon-Bayes
b1a4c102da make sure the fail fast test also have the bootstrap for tests done
Fixes #2237
2014-12-17 17:31:32 +00:00
Pere Urbon-Bayes
d63e0ec9eb make sure the bundler task is just used once while bootstraping tests
Fixes #2237
2014-12-17 17:31:32 +00:00
Pere Urbon-Bayes
93aadb4d4c Make the test plugins install throw bundler, so we don't have to install them every time we run the test (it makes the thing slow). Refactor also the bundler install task into a single one, making calls to it when necessary.
Fixes #2237
2014-12-17 17:31:32 +00:00
Pere Urbon-Bayes
21cb877ec9 clear the gem paths before the test rake tasks so they find previously installed plugins
Fixes #2235
2014-12-17 12:52:29 +00:00
Richard Pijnenburg
f472d4af24 Cleanup plugins to move to plugin system
Plugins have moved to https://github.com/logstash-plugins/... into
individual repositories. One plugin per repo.

* Add plugin install task and add required plugins for testing
* Adding a plugin task to install plugins
* Added required plugins to a prep task for the test part.

Fixes #1984
2014-11-04 23:24:59 +00:00
Jordan Sissel
f35558e395 Make most things depend on bootstrap
Fixes #1940
2014-10-22 20:04:01 +00:00