With the addition of the qa acceptance test suite we have decide that we
don't need to maintain this suite also since the acceptance test are
running really similar test.
Fixes: #5744Fixes#5760
License tests was failing because of GPL2 in mime-types.Added an
exception for this gem because v2.6.2 is triple-licensed and we pick
MIT.
Metrics test is failing because of address in use which is fixed in
PR #5307. Skipping this test for now.
Fixes#5369
* by default lives in LOGSTASH_HOME/config/logstash.yml
* location can be changed by $LS_SETTINGS_DIR or --path.settings
* overrides defaults of LogStash::Environment, BUT
* CLI flags override the yaml file
* several breaking changes to cli flag names (see config/logstash.yml)
* setting values are now type checked (see Setting and Settings classes)
Fixes#5313
Use an internal subscriber to verify that JSON output is valid JSON. The
purpose is to catch any json serialization errors that would occur while
logging.
Also had to update a few logger calls to log values that could be
serialized (Class instances and similar, at this time, fail to serialize
to JSON).
Fixes#4820
including dependencies, and reuse them in an offline installation by
providing an package for it. It adds two important commands to the
plugin manager, the pack and upack, to handle package creation and
installation and adds the --local flag to install and update to pull
plugins from the installed local packages.
Former commits:
add a task to create a bundle, under vendor/cache of the installed gems + their dependencies, this can be used later on to be installed offline
add an option to pass the --local flag to the bin/plugin update task, so it fetch plugins from a local cache, by default under vendor/cache
rename package:build to package:build-cache as is more meaningfull
add a --local flag to bin/plugin install to users can install plugins from the local cache, under the default directory of vendor/cache
add a plugin manager command to build the local cache of installed plugins using bundler package alike command
It adds code for compressing and extracting in zip and tar formats to
the utils module of logstash. The zip module is only jruby dependant as
it uses functions provided by java.
There is also code add in the plugin manager package command to handle
compression of plugins dumping.
Cleanup the custom java code to compress and extract zip files as it has
been known that using rubyzip is good and it has a more ruby like
features so the code is more clean and portable.
set of smallish improvement requested during review
added several options to handle situation when the package of plugins we want to generate is already there
clean up old code
applyed feedback from review, mostly changed in documentating behaviour plus better wording for messages
relocate the Environment.windows? check to the bootstrap side as it's used also in the plugin manager
add an unpack bin/plugin command so users can install packages of plugins throw the plugin manager
document override behaviour in the compress helpers (for zip and tar) plus add a fix for the tar extract when reading entries
made the unpack filename parameter a required one
add a force option to the bin/plugin unpack command
add a notice to that if using a local cache the --local flag should be passed with
Code cleanup and refactor introduced during review
add two wording suggestions comming from review
ammend more wording
skip the major version validation as in situation where there is no internet connection like when using the local vendor/cache to do the update
move compress to the bootstrap environment as being used in the plugin manager means not being loaded with logstash-core
Bring pack cached gems in the vendor directory so they can be used for bundler when packaging dependencies
Revert "Bring pack cached gems in the vendor directory so they can be used for bundler when packaging dependencies"
This reverts commit a9d7f46649932b06efaafebdd0eed2b4c63c2235.
patch the Bundler::Source::Rubygems to fetch only gems from a remote source
small changes in several parts of the plugin manager and the creation of a common pack command with shared code
change compress to read in chuncks
fix wrong var name in the bootstrap compress utils module
fix namespacing conflicts
add basic test for the compress utility module
ammend more namespace issues
add a comment to the rubygems mockey patch to rebuild the gem cache
apply cosmetic changes
make the compress module raise CompressError
remove vendor_path and pattern_path form environment as they where mix up during rebase
change the bin/pack force option to be force-delete-cache
rename force_delete_cache to overwrite
change force for overwrite in tha pack command
remove the override option for bin/plugin unpack
revert Gemfile and Genfile.lock wrong committed
Former commits:
add all defined sources to rubygems so verification can talk to all the repositories, even the private ones
added a very simple plugin manager install command spec, to check for properly loading sources when validating
broke long lines into smaller ones in the install command
make sure the update command takes into account all declared sources, plus there is the option to avoid gem validation in case of only having a private gem repo and no internet connection
fix wrong conditional in the validation of plugins
move the Gem.sources loading to a common place, to sources are loaded all at once and we don't need to worry on each call to be sure we do the right thing for other sources different that rubygems
update test to the last changes in the way gemfile is loaded
move Gem::SourceList.from to bundler::invoke! so we use the default gemset.sources array
fix loading path for Gem::SourceList in bundler, so it's not colliding with other classes
Revert "move the Gem.sources loading to a common place, to sources are loaded all at once and we don't need to worry on each call to be sure we do the right thing for other sources different that rubygems"
This reverts commit 6e1c809665d61495b5461e7522a7adc745fe27fc.
Revert "update test to the last changes in the way gemfile is loaded"
This reverts commit dc5f65c2ac5380b1640cb732946f1a8ababcc9dc.
make the Gem.sources load to happen inside the logstash_plugin? call
add the idea of settings to the plugin manager module, like this we can pass throw different stuff usefull to setup dependant components
add the install spec back
remove the PluginManager::Settings concept
change sources for rubygems_sources in the plugin manager options
change rubygems_sources to be rubygems_source
update comments
ammend plugin manager options description in the comments
spaces and new lines cleanup
merged duplicated plugin manager util_spec introduced during rebase
add a check when Gems.versions fail
Add the validation exception for the plugin manager
add better error handling for situation where the validation is not possible due to a connection issue with the remote server
Fixes#3583
fixed timezone issue
extracted logstash-core and reorganized specs
extracted logstash-core-event
extract java Event into logstash-core-event-java in a proper gem
remove obsolete jruby_event bootstrapping
fix require path
add java code bootstrap
use logstash-core-event/logstash-core-event.rb
remove obsolete files
basic instructions
LogStash::Json need to be initialized from event
update jruby and gradle versions
update compile:logstash-core-event-java rake task
WIP tasks refactor
fix gem.files
skip test if class is not defined
fix gem related tasks for new structure
add gem spec dirs in core tests
bootstrap java implementation when requiring timestamp
new Cloner class and Event clone impl
fix array fields assignments, see #4140
don't rely on json implementation ordering
fix skipped last iterpolation char
remove implementation specific unnecessary check
also require ruby classes
define error class in ruby
raise exception on invalid format
remove implementation specific tests and extract and put logger related test in pending
missing bits for having all core timestamp specs pass
run all core specs
remove leftover
comment regex
missing encoding header
revert to logstash-core-event by default
finished proper gemification
useless require
dynamically pick specs depending on logstash-core-event-* implementation
logstash root package version
missing file for proper gemification
do not build java event by default
always check for root logstash lib dir
fix concurrent-ruby version confict
fix rebase conflict
re-enable specs
user vars instead of constants
move non core code in bootstrap
document version files
move version file
remove useless code
use version in logstash-core
fix gem files list
put back concurrent-ruby version constrain as in master
add dependency on logstash-core-event
remove dependency on logstash-core to avoid circular dependency
fix rebase conflict
remove circular dependency
fix specs
update README
event clone
append in Ruby
tag & append
add timestamp assignment spec
comment out original classes
fix at signature
implemented del
remove from array
del test
wip initial Accessors, Event, EventImpl, Path, Timestamp and PathTest
wip
wip
FieldReference and Accessors implementation
rename targetCache to lut and set it protected
initial Accessors tests
todo comment
more tests
Timestamp implementation
Timestamp tests
fix method name
add Long constructor
event initialization, timestamp handling and json serialization
add <> type information
custom json serializer for Timestamp
remove toJson test
initial Event test
more tests
comments
debug traces
initial jruby Event wrapper and specs
added PathCache
implemented includes
added clone
wrap all Event methods
Rakefile to build and jar
missing getters and implement overwrite
support Date conversion
proper cast and coercion
replace Ruby Event with Java Event
test for field reference setter type coercion
disable specs
timestap setter should also set in map, accept more timestamp types
pre cache timestamp and expose isTimestamp
constructor from DateTime
expose proper Ruby Timestamp object
Ruby Timestamp basic specs
also load JRuby Timestamp
transpose Java<->Ruby Timestamp
fix timestamp specs
new jar
cleanup object construction
An obsolete setting is one that will cause a configuration error if it
is used.
The purpose of `:obsolete` is to help inform users when a setting has
been completely removed. The lifecycle of a plugin setting is now 4
phases: available, deprecated, obsolete, deleted.
"Available" is the default, and deprecated remains the same as it was
(logging a warning). The new obsolete will cause a configuration error
if such a setting is used. Then later, we can finally delete the config
setting after it's been obsolete for some time.
Fixes#3977Fixes#3978
This PR fix a problem when doing the interpolation with a string that
did not end with a fieldref but with a character. The interpolation was
ignoring the last character.
Example:
```
"%{type}|" => "syslog"
```
Fixes#3931Fixes#3937
We have discovered that in some cases and some plaftorms
configuring a default `verify_mode` when creating a SSL/TCPServer
could make the certificate verification fail. Ruby default behavior is
to use `NIL` when creating a new ssl context, this revert that change.
keep in mind that all TCP clients using SSL **must** use `VERIFY_PEER`
as their verify mode to prevent man in the middle attack.
Fix: https://github.com/elastic/logstash/issues/3657
New defaults:
* Cipher suite based on Mozilla's Intermediate set from
https://wiki.mozilla.org/Security/Server_Side_TLS (at time of writing)
* Disable SSLv2 explicitly
* Disable SSLv3 explicitly
* Disable compression if possible
The SSL option setting came from the ruby-ftw library's FTW::Connection
(apache 2 licensed, I am author), and transitively through work
published by jmhodges to improve Ruby's SSL strength.
I include specs to ensure we never include export or weak ciphers by
default.
Using this patch to test the security improvements according to
`www.howsmyssl.com` shows much improved results:
---
Testing this:
```
ruby -r ./lib/logstash/patches/stronger_openssl_defaults.rb -ropenssl -rsocket -rjson -rawesome_print -e 'c = OpenSSL::SSL::SSLContext.new; t = TCPSocket.new("www.howsmyssl.com", 443); o = OpenSSL::SSL::SSLSocket.new(t, c); o.connect; o.puts "GET /a/check HTTP/1.1\r\nHost: www.howsmyssl.com\r\n\r\n"; headers,body = o.read.split("\r\n\r\n", 2); puts body'
```
(I processed the JSON output w/ jq for easier reading)
The purpose of the above is to test the default behavior of SSLContext.
* JRuby 1.7.19 w/ this patch reports no cipher problems.
* JRuby 1.7.19 without this patch has several weak ciphers used:
```
"TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA: [\"uses keys smaller than 128 bits in its encryption\"]",
"TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA: [\"uses keys smaller than 128 bits in its encryption\"]",
"TLS_DHE_RSA_WITH_DES_CBC_SHA: [\"uses keys smaller than 128 bits in its encryption\"]",
"TLS_RSA_EXPORT_WITH_DES40_CBC_SHA: [\"uses keys smaller than 128 bits in its encryption\"]",
"TLS_RSA_EXPORT_WITH_RC4_40_MD5: [\"uses keys smaller than 128 bits in its encryption\",\"use RC4 which has insecure biases in its output\"]",
"TLS_RSA_WITH_DES_CBC_SHA: [\"uses keys smaller than 128 bits in its encryption\"]",
"TLS_RSA_WITH_RC4_128_MD5: [\"use RC4 which has insecure biases in its output\"]",
"TLS_RSA_WITH_RC4_128_SHA: [\"use RC4 which has insecure biases in its output\"]"
```
Under MRI, similar cipher selection problems are observed without this patch (weak export
ciphers, other weak small-key ciphers, RC4 complaints). With this patch, no cipher complaints
are reported by www.howsmyssl.com
One other note: Because JRuby defaults to TLS 1.0 and only makes CBC ciphers
available under the Mozilla Intermediate cipher set, I believe (and
howsmyssl.com agrees) that these defaults still make the BEAST exploit a
problem. Switching to TLS 1.1 should fix this, but we need to do more research
to determine the what, if any, impact it will have if we force TLS 1.1 to be
the default..
Fixes#3579
Add a pre release gem test by using a mock to reproduce the behaviour
reaised by Gem::Specification.find_by_name when dealing with pre release
gems.
Fixes#3476
make the eager loading patterns consistent in the spec_helper.rb file
make the setup-simplecov task not a dependency, but an explicit task only executed when ENV['COVERAGE'] is defined
refactor eager loading code plus add some documentation to the setup-simplecov task
Added more comments to the test:setup task
Fixes#3465
