Commit graph

8385 commits

Author SHA1 Message Date
Jordan Sissel
590584f8ef Playing a bit to see how today's mutate filter, if I did some refactoring, would feel in the new API. 2017-10-01 23:16:36 -07:00
Jordan Sissel
a67644f75c Refactor a bit to have fields declared the same way.
Renamed ConstructingObjectParser -> ObjectFactory (shorter name, felt more appropriate)

Fields (setters, etc) are now declared using a fluent interface on ObjectFactory and don't need to be done in a `static { ... }` block anymore

Also added back deprecated and obsolete fields.
2017-09-26 23:12:37 -07:00
Jordan Sissel
24a2308774 WIP removing the need to specify argument positions explicitly like (String) arg[0] for constructors. 2017-09-26 23:12:37 -07:00
Jordan Sissel
b656f2dce3 WIP removing the need to specify argument positions explicitly like (String) arg[0] for constructors. 2017-09-26 23:12:37 -07:00
Jordan Sissel
4770bd9bac PoC implementing the existing Translate filter as a Java plugin.
This exists primarily to practice and demonstrate the capabilities of ConstructingObjectParser
2017-09-26 23:12:37 -07:00
Jordan Sissel
b0d38330ad PoC implementing the existing Translate filter as a Java plugin.
This exists primarily to practice and demonstrate the capabilities of ConstructingObjectParser
2017-09-26 23:12:37 -07:00
Jordan Sissel
6119ae73d1 Make ObjectTransforms public 2017-09-26 23:12:37 -07:00
Jordan Sissel
764bad342f Javadocs drafting. 2017-09-26 23:12:37 -07:00
Jordan Sissel
b1be175aab Expand wildcard import. Use HashMap instead of LinkedHash/TreeMaps. 2017-09-26 23:12:36 -07:00
Jordan Sissel
acefe7e48b Expand wildcard import. Use HashMap instead of LinkedHash/TreeMaps. 2017-09-26 23:12:36 -07:00
Jordan Sissel
aed0f10af0 Remove resolved todo item. 2017-09-26 23:12:36 -07:00
Jordan Sissel
5ca642e108 Add @Override 2017-09-26 23:12:36 -07:00
Jordan Sissel
603c0869f4 Move to org.logstash.common.parser
Previously this was in org.logstash.plugin, and it felt not the right package.

Big code reorg also:
* refactored all the shorthand helpers like declareInteger into default methods of a new ObjectParser interface.
* moved object transforms like transformInteger to a new ObjectTransforms class
2017-09-26 23:12:36 -07:00
Jordan Sissel
2bf1b4aed6 Allow fields (constructor args, too) to be marked deprecated or obsolete.
This mirrors the current `:deprecated => "details"` and `:obsolete => "details"` in the Ruby side of Logstash.

* When a field is obsolete, using it will raise an exception.
* When a field is deprecated, using it will log a warning.

In tests, I used a trick I learned from Go to make compile-time assertions about return types. This is the reason for `Field x` in various places used in assignment but otherwise not used.
2017-09-26 23:12:36 -07:00
Jordan Sissel
b80e05df00 Code style changes
* note IllegalArgumentException throws
* method order slightly rearranged.
* javadocs
2017-09-26 23:12:36 -07:00
Jordan Sissel
2ac62f584c Add declareList for List<T> usage.
Also renamed noun-verb things (stringTransform) to verb-noun (transformString)
2017-09-26 23:12:36 -07:00
Jordan Sissel
c275a608ce Add ConstructingObjectParser for building an object from a configuration map.
The idea behind ConstructingObjectParser was largely lifted from Elasticsearch's class by the same name. The reasons I liked this approach was that it provided a type-safe way to define how user input (a Map, in Logstash's case) is parsed to create an object like an input plugin.

The ConstructingObjectParser roughly implements in Java what `config/mixin.rb` provides for Ruby plugins.

This class allows us to take a Map that comes from the config parser and
build a pipeline plugin (input, filter, output).

Differences from the ruby api:

* Custom object validation is possible. Example, `http_poller` has a map of `urls` which are `name => { configuration }`, and we can achieve type-safe validation of this `urls` thing which benefits us as plugin authors and benefits users through better error reporting.
* Explicit `default` value is not required because Java class fields already allow you to set default values :)

Some rationale on IllegalArgumentException:

* Object construction will throw IllegalArgumentException upon validation failure. This was chosen because IllegalArgumentException is an unchecked exception we can throw while still satisfying the constraints of the java.util.function interfaces (BiFunction, etc)
2017-09-26 23:12:36 -07:00
liketic
0179bfce2c Update description for registry remove test
Fixes #8378
2017-09-26 18:14:01 +00:00
liketic
fa38f9f7c6 Add remove method for LogStash::PLUGIN_REGISTRY
Fixes #8378
2017-09-26 18:14:00 +00:00
Pier-Hugues Pellerin
6c2edd1ccc Correctly Serialize RubyNil values when serializing to JSON
The custom serializer introduced in #8371 did change the behavior or JSON
serialization when one of the field was a RubyNil values.

The orignal behavior was to serialize RubyNil values into their
Javascript Null. The PR #8371 changed that behavior to instead
serialized an empty string and changed the expected type.

This commit revert the behavior to serialize to Null and also tests
around the events class to prevent that.

Fixes #8386
2017-09-26 15:35:13 +00:00
Guy Boertje
9b09d4c899 bump JrJackson/Jackson versions, source all version refs from versions.yml
final (I hope) fixes for consolidated versioning.

After testing with rake artifact:zip, need to try both files.
Add desc to rake test:install-* tasks, tired of
  not seeing them in rake -vT

changes requested via review

Fixes #8373
2017-09-26 15:29:58 +00:00
Armin
6db280903c PERORMANCE: Remove all BiValue usage by removing NullBiValue
Fixes #8371
2017-09-26 08:55:52 +00:00
Jake Landis
d2011f9185 Test fixes: This should fix a couple different test failures.
* Fixes #8344 - Ruby Unit Test can hang with no information as to why
* Fixes #7227 - MetricStore::MetricNotFound
* Fixes #8204 - `config.reload.automatic` is set to`FALSE

I am only certain it fixes #8344 and #7227, since I can not reproduce #8204.

Fixes #8376
2017-09-24 03:39:29 +00:00
Jake Landis
aef8c7a255 Instruct Jenkins to not abort other Vagrant hosts
See: https://stackoverflow.com/questions/19736862/vagrant-aborted-at-end-of-jenkins-job
and: https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller

Note - the existing vagrant lifecycle instructs a `destroy` before bootstrapping that will ensure a clean start if last process did not die properly.

Fixes #8380
2017-09-24 01:13:11 +00:00
Armin
01c99ccf11 #8360 fix tests by enforcing IPV4 in Benchmark tool UTs
Fixes #8372
2017-09-23 06:50:39 +00:00
Jordan Sissel
0dd58a4473 Add commercially supported plugins to be default
Our intent is to ship all commercially supported[1] plugins with
Logstash. By some mistakes, we have omitted a few plugins, and this
commit resolves those omissions.

Fixes #8287

Fixes #8318
2017-09-22 19:50:59 +00:00
Armin
3ba73f418c #7318 stop creating a new RubyTimestamp on every timestamp get
Fixes #8334
2017-09-22 15:52:01 +00:00
liketic
a6bd3164a0 Fix rawtypes warning in build
Fixes #8369
2017-09-22 08:17:47 +00:00
liketic
b4e745ced0 Obtain file lock through FileLockFactory for DLQ
Fixes #8314
2017-09-22 07:52:48 +00:00
Jake Landis
b7bd9dec13 Remove Travis builds (they have been replaced by Jenkins) Fixes: #8079
Fixes #8362
2017-09-21 23:16:44 +00:00
Colin Surprenant
83fde50920 pick jvm.options for rspec.bat 2017-09-21 15:45:13 -04:00
Armin
08b93005a1 PERFORMANCE: More efficiently store offsets in AbstractByteBufferPageIO
Fixes #8333
2017-09-21 17:01:22 +00:00
Armin
416339f3c9 MINOR: Cleanup RubyTimestamp Ctor
Fixes #8335
2017-09-21 15:56:13 +00:00
Armin
ff9ccb3e59 #8345 adjust test stacktrace size example to work in all cases by acting on clearly defined thread
Fixes #8350
2017-09-21 14:45:13 +00:00
Armin
68b22280bc #7549 fix unsafe concurreny in PQ count getters
Fixes #8336
2017-09-21 10:52:56 +00:00
DeDe Morton
6962fc2a06 Add link to DLQ
Fixes #8326
2017-09-20 20:10:33 +00:00
Armin
889c5668f2 #8132 Remove JavaProxyBiValue without replacement
Fixes #8239
2017-09-20 19:59:08 +00:00
Armin
bb91ae69c5 #8341 raise tolerance in timestamp_spec.rb tests
Fixes #8342
2017-09-20 17:20:13 +00:00
Lisa Cawley
c816beecd1 Added index-shared3.asciidoc (#8324)
* [DOCS] Added index-shared3.asciidoc

* [DOCS] Added link to X-Pack release notes
2017-09-20 07:58:19 -07:00
Jake Landis
eca8475d11 Minor: Remove JRuby 1.7 from the docker base image (it is not used)
Fixes #8330
2017-09-20 12:16:05 +00:00
Jake Landis
8b3cae9685 Test fix: Allow slowlog to have >= 1 entries, instead of exactly 2 entries.
Fixes #8322

Fixes #8323
2017-09-19 21:54:45 +00:00
Colin Surprenant
0e3b9a3eb1 refactor linux launch scripts per #8261
remove dripmain.rb
2017-09-19 16:40:21 -04:00
Jake Landis
2c36246234 Test fix: ensure 'fatal: Not a git repository' is not picked up as expectation integration test Fixes #8319
Fixes #8320
2017-09-19 19:57:53 +00:00
Armin
ffb72d3812 #8293 fix concurrent access to PQ persisted bytesize throwing
Fixes #8317
2017-09-19 18:16:44 +00:00
Pier-Hugues Pellerin
f7637058ea remove vendor plugin
Fixes #8281
2017-09-18 18:57:49 +00:00
Pier-Hugues Pellerin
0b7cb4a599 remove all trace of install all plugins in the code and in the build scripts
Fixes #8281
2017-09-18 18:57:49 +00:00
Pier-Hugues Pellerin
ceb6c53ace Remove uneeded task to install all the possible plugins from the logstash-plugin organization
This command was error prone, some plugins on the logstash-plugin
organization are currently broken.

Fixes #8281
2017-09-18 18:57:49 +00:00
Pier-Hugues Pellerin
4dc2fd3ddd Remove unecessary dependency file-dependencies
To understand why we can now remove this gem we have to go back at the
history of LS 1.5 and the choice we made back them. In the begining
plugins depending on external files like the `user-agent` or the `filter-geoip`
were not bundling theses files in the gem.

The `file-dependencies` was providing a new hook when plugins were
installed to trigger the file download.

In the context of making the plugins work in an offline environment we
need to make sure that every plugins external resources are present in
the gem.

We have currently a few plugins that require external files and they all
bundle it. So I think its safe to remove that feature, also the hook was
not triggered correctly. So it was a NOOP.

Fixes #8298
2017-09-18 18:56:58 +00:00
Pier-Hugues Pellerin
db5bd5e68a Bootstrap JRuby using gradle
Instead of using rake to bootstrap we are slowly moving part of our
infra to gradle this is the first step taken, I didn't change the calls,
I just made the `rake bootstrap` call the `gradle bootstrap`

Closes: #8234

Fixes #8238
2017-09-18 14:32:10 +00:00
Pier-Hugues Pellerin
8de867b173 Remove tests dependency on filter multiline
Some of the test were still depending on the multiline filter, this
filter is now shipped anymore with Logstash. I've keep the same logic
but created a simple dummy filter.

Fixes #8284
2017-09-18 13:24:40 +00:00