Commit graph

73 commits

Author SHA1 Message Date
Karol Bucek
afce87a67e Fix: cloud_id not propagating from monitoring config
Fixes #11800
2020-04-16 18:52:42 +00:00
Andrea Selva
6a74872d50
Moved code out of LogStash::Compiler to org.logstash.config.ir.ConfigCompiler 2020-04-15 12:21:39 +02:00
Rob Bavey
2899c9f379 Handle Boolean Edges
Fixes #11779
2020-04-13 20:09:27 +00:00
Rob Bavey
5e6065f414 Incorporate review comments
Fixes #11777
2020-04-13 14:11:07 +00:00
Rob Bavey
54c451f48d Fix hard coded constraint on pipeline name for metrics
Fixes #11777
2020-04-13 14:11:07 +00:00
Rob Bavey
58314a7f0e
Update gradle version to 6.3 (#11742)
* Update gradle version to 6.3

Gradle versions prior to 6.3 cannot run under JDK14.
This commit upgrades the version of Gradle to 6.3, and removes all deprecation warnings that can currently be removed.

Changes include:
* Increase gradle memory to 2g
* Increase gradle memory in the license check job to 2g
* Replace use of `testCompile`
* Replace `runtime` with `runtimeOnly`
* Remove`compile` depedencies from gradle files
* Replace deprecated archive methods
* Fix dependencies report build
* Make jruby dependencies 'api', fix archiveVersion
* Set `duplicatesStrategy` for all tasks of type Copy
* Use `configureEach` for global 'withType' calls
** Use the recommended Tasks API calls
(https://blog.gradle.org/preview-avoiding-task-configuration-time)
* Run `./gradlew wrapper` earlier to improve caching
* Use copy with chown for resources that need to be run during `./gradlew wrapper`
2020-04-07 12:49:36 -04:00
Colin Surprenant
5a25c6f8e9 simplify batch classes, do not compute JE empty batches, refactor RE worker loop (#11737)
cleanup RubyArray "rawtypes"
remove all LinkedHashSet from batch and queue classes
avoid processing empty batches in Java worker loop
cleanup AckedReadBatch and MemoryReadBatch
refactor Ruby worker loop similar to Java Execution to not use batch merge
remove QueueBatch merge and replace LinkedHashSet with ArrayList
2020-04-02 16:26:54 -04:00
andsel
b2d1b880a6 Introduced JDK environment variable to explicitly pass the JAVA_HOME to use, expanded Xmx usable by Gradle
Fixes #11708
2020-03-30 10:53:18 +00:00
andsel
dfdc6f2aac Fixed flaky test, as fixed in packport PR #11641
Fixes #11679
2020-03-12 09:00:49 +00:00
João Duarte
e9c9865f40
Add apache and elastic license headers to source code files (#11673)
* add license header to ruby and java files
* add license header to erb and rake files
* add license headers to gradle files
2020-03-11 11:53:38 +00:00
andsel
fc0ced5708 Fixed time format that introduced flaky tests depending on time
Fixes #11652
2020-03-04 14:53:29 +00:00
andsel
3695580b92 Adaptations to internal collector to send data directly to monitoring cluster Close 11573
Fixes #11541
2020-02-28 14:26:19 +00:00
andsel
2925b491f9 Added precation notice when internal monitoring collector is used. Closes #11346
Fixes #11511
2020-01-27 15:06:46 +00:00
Karol Bucek
d8398351a2
Feat: x-pack cloud id/auth for monitoring/management (#11496)
resolves #11488
2020-01-14 22:06:50 +01:00
Joao Duarte
526a8f89ad introduce enterprise license level
Fixes #11407
2019-12-10 12:31:16 +00:00
Joao Duarte
3294a4be61 refactor list of license_types
DRY up the list of license types as there were 10 places that listed the types explicitly

Fixes #11407
2019-12-10 12:31:16 +00:00
andsel
2a25547fbc Changed the xpack metrics pipeline to use a customized ES output plugin to put document_type for /_monitoring, closes #11312
Fixes #11321
2019-11-21 16:10:01 +00:00
andsel
aad25d9bbc Drop _xpack namespace for ES security and license endpoints
Fixes #11297
2019-11-12 16:49:45 +00:00
Mike Place
252d5e7686
Enhance GET _node/stats/pipelines API for Metricbeat monitoring (#10576)
* parent 8c5697c748
author Guy Boertje <guy@elastic.co> 1556806171 +0100
committer Mike Place <mike.place@elastic.co> 1557234770 +0200

Bump JrJackson to 0.4.8

Fixes #10748

LIR serializer refactor

Remove commented code

Remove more commented code

Remove license and add encoding

Style change to make code more vertical.

eid and hash

Use pipelines_info to construct the stats

Add tests for new fields

Add queue stats

* bad merge resolution

* bad merge resolution

* Don't merge if nil

* Better merge strategy

* add vertex gate

* Guard against nil

* Use extended queue stats in pipeline report

* Add cluster uuids to Elasticsearch outputters in pipeline output

* move uuid

* remove old uuid lookup

* Only populate cluster_uuids when present

* remove print

* cluster_uuids -> cluster_uuid

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Make var singular

* Match singular var name

* Remove unnecessary nil check

* Pass in the matching pipeline for the report

* Remove old way of inserting cluster_uuids

* Update logstash-core/lib/logstash/api/commands/stats.rb

I like this much better and in testing it seems to work correctly.

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Remove unreferenced code that was part of debugging

* Remove events var which was unused

* Don't try to remove before insert

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Make pipeline extended stats generation more efficient

* Implement suggestion to improve readability

* Cleaner merging per review recommendation

* Only generate extended_stats once

* remove unneeded comments

* Add cluster_uuid to node vertex

* remove top-level cluster_uuids

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Implement change to make logic more simple suggested in review

* Rely on options gate to insert graph

Resolves concern here:
https://github.com/elastic/logstash/pull/10576#issuecomment-501774635

* Update logstash-core/lib/logstash/api/commands/stats.rb

Co-Authored-By: Ry Biesemeyer <yaauie@users.noreply.github.com>

* Move UUID lookup to API layer

* Move private method to bottom per review recommandation
2019-06-22 08:31:13 +02:00
Mike Place
cc3c5ec00f
Refactoring of LIR serializer and exposing pipeline metrics (#10561)
* [WIP] Add ephemeral id and hash to pipeline stats

Refs #10119

* Fix incorrect hash id

* Add graph metrics

* Include node update

* LIR serializer refactor

* Remove commented code

* Remove more commented code

* Update spec

* Remove license and add encoding

* Style change to make code more vertical.

* Implement review suggestions

* Remove commented code

* Fix spec path

* Add cluster UUID

* Add graph?=true parameter

* Add options to pipelines/

* Use pluginmetadata

* Replicate change from java_pipeline to ruby pipeline
2019-05-01 12:45:11 +02:00
Jordan Johnson-Doyle
aa03eb3973 Inputs expect a NamespacedMetric, not the root metric instance
Fixes #10614
2019-04-11 13:10:18 +00:00
Jake Landis
8bd7223bbf monitoring: bump to system_api_version 7
Fixes #10562
2019-03-17 15:12:15 +00:00
Ry Biesemeyer
67b8d43dff fix events_count to return total
Fixes #10564
2019-03-16 00:10:39 +00:00
Joao Duarte
6f7d97abce make monitoring specs resilient to internal api number changes
Fixes #10552
2019-03-13 16:26:22 +00:00
Jake Landis
4867674eac Update monitoring HTTP end point
This commit changes /_xpack/monitoring/_bulk to /_monitoring/bulk.
The former is deprecrated as 7.0.0.

Relates https://github.com/elastic/elasticsearch/pull/36130
Relates https://github.com/elastic/elasticsearch/issues/35958

Fixes #10528
2019-03-12 11:45:35 +00:00
Rob Bavey
5d41ab675c Change internal document type to push "_doc" instead of "doc"
This commit fixes x-pack integration tests that were broken by https://github.com/elastic/elasticsearch/pull/39888 removing the "doc"
 type, and using `_doc` in templates.

Fixes #10533
2019-03-11 19:55:15 +00:00
Jake Landis
ad65518304 update to send api version 6
Fixes #10518
2019-03-07 16:19:44 +00:00
Jake Landis
0ca34f07bb Central management typeless API
This commit adopts Elasticsearch's typeless API for central management.

Relates: https://github.com/elastic/elasticsearch/issues/3863

Fixes #10421
2019-03-07 16:19:17 +00:00
Joao Duarte
115738ceef mute metrics_spec line 138 test
Fixes #10412
2019-02-07 16:25:45 +00:00
João Duarte
7b6d60c433
bump to 8.0.0 (#10400) 2019-02-06 15:04:44 +00:00
Joao Duarte
c1a78c0d59 mute frequently failing x-pack test
Fixes #10396
2019-02-06 09:39:55 +00:00
Ry Biesemeyer
328b83815c spec: make timing-sensitive specs a little more flexible
Fixes #10392
2019-02-06 07:48:57 +00:00
Ry Biesemeyer
149684d5b6 spec: noop refactor for clarity - use explicit metric_input subject instead of implicit subject - use rspec/wait full expectation syntax instead of be_falsey/be_truthy - eliminate unused shared examples - incorporate single-use shared examples
Fixes #10392
2019-02-06 07:48:56 +00:00
Colin Surprenant
6990d08be5
rename config option .url and .ca to .hosts and .certificate_authority (#10380) 2019-02-05 17:39:30 -05:00
Guy Boertje
e262d6b0b6
Ensure compatibility of module data with ES and Kibana 7.0 (#10356)
* Ensure compatibility of module data with ES and Kibana 7.0

* cherry-pick jake changes

* 7x index template updates - formatting

Fixes #10375
2019-02-05 17:05:10 +00:00
Jake Landis
ecc811c429 7x index template updates - formatting
Fixes #10375
2019-02-04 21:18:25 +00:00
Jake Landis
b53d79c148 7x index template updates
This commit updates the Elasticsearch index templates to be compatible with 7.x
* removes types
* remove _all
* template -> index_patterns

Fixes #10375
2019-02-04 21:18:25 +00:00
Colin Surprenant
f08b8c5076
fix agent silent exit upon pipelines reloading (#10346) 2019-01-31 15:48:50 -05:00
Dan Hermann
a213a6f62a
Fix xpack integration tests to work with new ES hits format (#10220) 2018-12-18 13:17:18 -06:00
João Duarte
fe7607abd4
only start monitoring pipeline after valid license (#10106)
Avoid unecessary logging errors and resource usage by only starting
the monitoring pipeline if we can validate the license.
2018-11-07 17:28:48 +00:00
Guy Boertje
3fdee027d9
Add SC and EB input TLS support for the Logstash ArcSight module (#10056)
* Add SC and EB input TLS support for the Logstash ArcSight module
I added the ssl for the smart connector (tcp) and ssl and sasl for the
event broker.
Needs verification on a current stack.

* So this is the actual extent of changes (I thought it was to simple)
Fixed the docs omission
Fixed the load path issue
Fixed the ERB binding context problem
Added some basic happy path tests

* remove module settings in logstash.yml

* Overwrite my logstash.yml with the content on master

* Add comments to yaml fixtures
2018-10-23 09:12:12 +01:00
Joao Duarte
78bc47d1c9 support ssl verification mode in monitoring and management
Fixes #9866
2018-08-31 08:01:27 +00:00
Ry Biesemeyer
807cbc1071 x-pack monitoring: prevent failures to emit metrics events before agent started
Before the monitoring pipeline is started, it is not possible to emit events
from the metrics input; not only is the `agent` nil (resulting in a cryptic
`NoMethodError` when we attempt to get its `ephemeral_id`), but the `queue`
will also be `nil` until the `run` method is invoked.

Instead, start the poller once we are sent `run` and are guaranteed to have a
`queue` on which to put the events; when the snapshot poller is running, skip
any invocations before `agent` is available.

Resolves: elastic/logstash#9462

Fixes #9627
2018-07-05 20:57:02 +00:00
Josh Soref
c6cd247ec3 Multiple spelling corrections (#9782)
* spelling: actually
* spelling: already
* spelling: concurrency
* spelling: constructor
* spelling: destinations
* spelling: different
* spelling: elasticsearch
* spelling: encoding
* spelling: error
* spelling: explicitly
* spelling: failings
* spelling: falsey
* spelling: guarantees
* spelling: having
* spelling: implementation
* spelling: logstash
* spelling: module
* spelling: multiple
* spelling: omitted
* spelling: overridden
* spelling: pipeline
* spelling: raspberry
* spelling: receive
* spelling: recommended
* spelling: registered
* spelling: registering
* spelling: shutdown
* spelling: signal
* spelling: specified
* spelling: successful
* spelling: successfully
* spelling: valuefied
* spelling: vector
2018-07-04 10:41:10 +01:00
Jake Landis
e22457252f Introduction of the Azure module (experimental)
This module leverages the Azure Event Hub to read events from the activity log and sql diagnostics.  Please note this module is considered experimental till otherwise noted.

This change also introduces logstash-input-azure_event_hubs as a default plugin

Fixes #9729
2018-07-02 16:34:49 +00:00
Armin
30e055b1bf JAVAFICATION: Move more of the Java pipeline to Java
Fixes #9723
2018-06-12 13:50:15 +00:00
Jake Landis
d538b524f8 Fix relative path for arcsight module. Fixes #9714
Fixes #9715
2018-06-07 14:36:28 +00:00
Andrew Cholakian
0a72df874d Use longer timeout for x-pack reload spec
This test flaked out on CI, so let's give it some more time before digging deeper.

Fixes #9695
2018-05-31 22:01:55 +00:00
Lisa Cawley
e75156ea54
[DOCS] Moves management folder to docs (#9680) 2018-05-30 09:23:53 -07:00
Lisa Cawley
604ad7259f
[DOCS] Move monitoring folder to docs (#9677) 2018-05-30 09:02:39 -07:00