Commit graph

835 commits

Author SHA1 Message Date
andsel
3695580b92 Adaptations to internal collector to send data directly to monitoring cluster Close 11573
Fixes #11541
2020-02-28 14:26:19 +00:00
andsel
fc1ae6f119 Fixes #11598 enabling the users to use completely commented config files
Fixes #11615
2020-02-27 09:43:05 +00:00
João Duarte
5fcab21194
update licenses for master (#11549) 2020-01-31 12:43:01 +00:00
Joao Duarte
a36fdb9050 fail license report job on missing licenses
Fixes #11554
2020-01-31 12:36:41 +00:00
Colin Surprenant
0bc9fa5665
add support for pipeline.ordered setting for java execution (#11524)
reuse rubyArray for single element batches

rename preserveBatchOrder to preserveEventOrder

allow boolean and string values for the pipeline.ordered setting, reorg validation

update docs

yml typo

Update docs/static/running-logstash-command-line.asciidoc

Co-Authored-By: Karen Metts <35154725+karenzone@users.noreply.github.com>

Update docs/static/running-logstash-command-line.asciidoc

Co-Authored-By: Karen Metts <35154725+karenzone@users.noreply.github.com>

java execution specs and spec support

docs corrections per review

typo

close not shutdown

Ruby pipeline spec
2020-01-29 14:01:38 -05:00
Mike Place
020e87efc4 Add cluster_uuid setting to default config file, displaying it in Node stats HTTP API
Fixes #11106
2020-01-27 08:12:39 +00:00
andsel
6eb25173e0 Added plugin.id to fish tag log lines related to plugins
Fixes #11078
2020-01-23 15:33:31 +00:00
Colin Surprenant
b5f203ce24
start inputs only when all WorkerLoop are fully initialized (#11492) 2020-01-14 15:46:48 -05:00
andsel
8ccb24bf99 Changed plugin factory creation to use SourceWithMedata and not destructured line and column
Fixes #11456
2020-01-10 16:23:43 +00:00
Ioana Tagirta
c9b09bb369
Remove NodeStats#mem_payload (#9051) 2020-01-02 11:53:52 +01:00
Karol Bucek
abd4253708 Fix: SafeURI normalize and eql? to work as expected
normalize wasn't doing what was expected (URI.normalize)
also make sure SafeURI can be stored in Hash (proper eql?)

Fixes #11443
2019-12-31 10:06:40 +00:00
Karol Bucek
7b4e3585bc Fix: (move and) make get_thread_id "nil safe"
necessary since native (Java) thread is kept as a weak ref
so un-wrapping should deal with a potentially GCd instance

resolves GH-11450

Fixes #11458
2019-12-30 13:04:16 +00:00
Karol Bucek
b9fb9a7063 Refactor: minor one sub-stitution should do
Fixes #11458
2019-12-30 13:04:16 +00:00
Karol Bucek
850d270703 Refactor: we're always on JRuby these days
(other part of same method assume JRuby already)

Fixes #11458
2019-12-30 13:04:16 +00:00
Karol Bucek
af3513dfba Changed: avoid JavaObject wrapping in Ruby methods
aligns the Ruby/Java returns as they happen in scripted Java
e.g. as `java.lang.Thread.new` returns a JavaProxy instance

there's really no reason to use JavaObject which always needs `to_java`
conversion to be useful (and is considered legacy in JRuby).

considered breaking change e.g. `LogStash::MemoryReadClient#read_batch`
will now return a proper JavaProxy instead of the JavaObject

Fixes #11391
2019-12-22 22:06:08 +00:00
Karol Bucek
3cbe7e922a Fix: handle cloud-id with an empty kibana part
fixes GH-10747

Fixes #11435
2019-12-12 21:12:30 +00:00
Colin Surprenant
702efea88f test codec against class name string to prevent class equivalence bug with a Delegator
Fixes #11401
2019-12-12 17:14:39 +00:00
Joao Duarte
7f5aa186c1 remove use of thread.exclusive in plugin_metadata.rb
Fixes #11388
2019-12-10 19:43:29 +00:00
Karol Bucek
6e048d5b5b Fix: do not leak ThreadContext into the system
this is fairly recent - since 7.4 (added in GH-11075)

there's a risk plugins would assume ThreadContext to
exist or collide the 'global' constant - usually best
to import where the Java class actually gets used ...

Fixes #11356
2019-12-02 11:21:05 +00:00
João Duarte
ff527a01cd Remove license header from pipelines_info.rb
This code was moved to the apache 2.0 license part of the distribution and should not have the license header any more

Fixes #11345
2019-11-26 14:02:28 +00:00
Colin Surprenant
17575f32ba remove JRuby resolv.rb patch following fix in 9.2.9.0
Fixes #11330
2019-11-21 09:59:18 +00:00
andsel
f554930e81 Introduced DeprecationLogger for use in core code and exposed to Java and Ruby plugins. Closes 11049
Fixes #11260
2019-11-14 10:49:27 +00:00
João Duarte
4f902b435c
Remove unused code (#11231)
* remove unused certificate bundles and ruby files
* remove chronic_duration dependency
2019-10-18 16:08:53 +01:00
Ry Biesemeyer
7f90040b9b make pipeline grammars more accurately capture field references
Certain malformed field reference literals (e.g., those containing a series of
multiple open-brackets `[[`) were propagated undetected by the parser, only to
create a crashing error when used.

Starting Logstash with the `--config.test_and_exit` flag (or `-t` shorthand)
would validate the config, even though it could not be used in practice.

By updating the grammar(s) to exclude the use of an open square bracket (`[`),
we more closely match the formal grammar and ensure these malformed literals
are rejected closer to the source.

NOTE: this PR only affects field reference _literals_, and does not resolve
a similar issue with field references in quoted format strings.

Resolves: https://github.com/elastic/logstash/issues/11022

Fixes #11195
2019-10-15 16:54:48 +00:00
Ry Biesemeyer
61dfb087df regenerate config parsers from unmodified grammar definitions
`touch logstash-core/**/*.treetop && ./gradlew compileGrammar`

Fixes #11195
2019-10-15 16:54:48 +00:00
andsel
e58a6e01ce Added LS configuration variable 'pipeline.separate_logs' to separate logs per pipelines - use log4j RoutingAppender - avoid output to main log files when log per pipeline is enabled - closes 10427
Fixes #11108
2019-10-08 14:07:17 +00:00
andsel
0e622cc744 Added origins of pipeline's configuration (es config string, the paths of config files used, module).
closes 9630
2019-09-12 16:02:36 +02:00
Joao Duarte
10f9b9e5c8 support substitutions in pipelines.yml file
Fixes #11081
2019-08-30 14:02:59 +00:00
Joao Duarte
47fc8e7c4d give multiple pipelines all the settings
Previously we'd only give a pipeline the settings related to pipelines
The PipelineSettings class was used for this.
However a pipeline may need other settings like the keystore location.

For this we instead clone the settings object and merge all the pipeline
specific settings. This is accomplished with a new method that ensures
that only pipeline level settings are overwritten in the clone.

Fixes #11076
2019-08-30 11:20:35 +00:00
andsel
cda592f659 Add pipeline.id to log lines
fixes #8290, #10521
2019-08-28 17:57:28 +02:00
Dan Hermann
a0e521488d Use correct execution engine for test-and-exit mode
Fixes #11067
2019-08-21 17:40:55 +00:00
João Duarte
12b0987ecc
avoid variable collision in pipeline stats api (#11059) 2019-08-20 15:47:52 +01:00
Dan Hermann
4b2857d225 use method-level ensure blocks
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
0304da9ac9 remote != remove
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
eaeffd47b5 don't put nulls in the map
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
e0135c4e63 add missing lock statement
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
b9da367d39 don't perform long-running pipeline actions inside calls to ConcurrentHashMap.compute to avoid deadlocks
Fixes #10872
2019-06-29 17:27:22 +00:00
Dan Hermann
1b16eca32a Fix pipeline shutdown ordering
Fixes #10872
2019-06-29 17:27:22 +00:00
Mike Place
2dc6150fd8
Restore UUID lookup to node (#10884)
* Restore UUID lookup to node

* Fix incorrect merge

* Move private method to bottom

* Change method name per review suggestion

* Update method description
2019-06-26 14:32:19 +02:00
Dan Hermann
6ba1b1ff67 Remove debug code for p2p plus formatting
Fixes #10840
2019-06-25 18:31:37 +00:00
Dan Hermann
ea0485d5bf Do not shut down API webserver until after pipelines have been shut down
Fixes #10880
2019-06-25 18:30:21 +00:00
Mike Place
03001fa1f5
Add workers and batch_size to root request (#10853)
* Add workers and batch_size to root request

* Add batch delay per review recommendation
2019-06-25 13:44:25 +02:00
Mike Place
aa0588000c
Inject hash and ephemeral_id into stats (#10885) 2019-06-25 13:19:23 +02:00
Mike Place
252d5e7686
Enhance GET _node/stats/pipelines API for Metricbeat monitoring (#10576)
* parent 8c5697c748
author Guy Boertje <guy@elastic.co> 1556806171 +0100
committer Mike Place <mike.place@elastic.co> 1557234770 +0200

Bump JrJackson to 0.4.8

Fixes #10748

LIR serializer refactor

Remove commented code

Remove more commented code

Remove license and add encoding

Style change to make code more vertical.

eid and hash

Use pipelines_info to construct the stats

Add tests for new fields

Add queue stats

* bad merge resolution

* bad merge resolution

* Don't merge if nil

* Better merge strategy

* add vertex gate

* Guard against nil

* Use extended queue stats in pipeline report

* Add cluster uuids to Elasticsearch outputters in pipeline output

* move uuid

* remove old uuid lookup

* Only populate cluster_uuids when present

* remove print

* cluster_uuids -> cluster_uuid

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Make var singular

* Match singular var name

* Remove unnecessary nil check

* Pass in the matching pipeline for the report

* Remove old way of inserting cluster_uuids

* Update logstash-core/lib/logstash/api/commands/stats.rb

I like this much better and in testing it seems to work correctly.

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Remove unreferenced code that was part of debugging

* Remove events var which was unused

* Don't try to remove before insert

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Make pipeline extended stats generation more efficient

* Implement suggestion to improve readability

* Cleaner merging per review recommendation

* Only generate extended_stats once

* remove unneeded comments

* Add cluster_uuid to node vertex

* remove top-level cluster_uuids

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Implement change to make logic more simple suggested in review

* Rely on options gate to insert graph

Resolves concern here:
https://github.com/elastic/logstash/pull/10576#issuecomment-501774635

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Move UUID lookup to API layer

* Move private method to bottom per review recommandation
2019-06-22 08:31:13 +02:00
Joao Duarte
7468ed058f avoid clashes between Environment class methods
Fixes #10860
2019-06-14 17:44:38 +00:00
Dan Hermann
2449e94164 Fixes unit test failures on some runs of ConfigCompilerTest::testComplexConfigToPipelineIR
Fixes #10837
2019-05-30 14:40:51 +00:00
Dan Hermann
2f5aff2362 Merge config values in LIR
Fixes #10832
2019-05-30 12:59:09 +00:00
Dan Hermann
fd74ce0156 LIR support for octal literals in pipeline definitions
Fixes #10828
2019-05-28 16:51:03 +00:00
Dan Hermann
7cd6bb89ae Default stack trace size to 50 and make it configurable
Fixes #10793
2019-05-21 15:42:06 +00:00
Jordan Doyle
bf1881dff2 Include G1 in JVM heap metrics (#10728) 2019-05-10 15:34:07 -05:00