Commit graph

11011 commits

Author SHA1 Message Date
kaisecheng
53a346c57f
[Doc] add reference to ECK (#15565)
add ECK reference to doc

Fixed: #15471

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
2023-11-14 23:26:45 +00:00
Karen Metts
c060c00d7c
Doc: Add Elastic Agent collection (#15528)
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
2023-11-14 13:55:39 -05:00
Dimitrios Liappis
ce63ea4a51
[ci] Fix image name for Rocky Linux 8 (#15584)
Fix typo for image name of Rocky Linux 8 for JDK matrix jobs.
2023-11-14 19:26:49 +02:00
João Duarte
6547f8c5c4
Update logstash_releases.json to include 7.17.15 and 8.11.1 (#15578) 2023-11-14 11:03:54 +00:00
verogo
496f18effc
[Buildkite] Add pipeline max timeout property (#15579)
Build's maximum_timeout_in_minutes and default_timeout_in_minutes are available now through the catalog-info.yaml file.
As this change was made manually before we implemented this in RRE/Terrazzo, they got reverted to the default value (0); thus, I am raising this PR to get it as it was specified before the upgrade (120 mins.)
2023-11-14 09:12:24 +02:00
Mashhur
8700179cf2
Update 8.x CI to 8.11 (#15551) 2023-11-12 05:10:56 +05:00
Dimitrios Liappis
0ede19a0e1
[ci] JDK matrix Buildkite pipelines (pt 2/Windows) (#15563)
This commit adds JDK matrix Buildkite pipelines for
Windows 2022, 2019 and 2016.

It also makes the groups easier to read (on both Linux and Windows
pipelines) by removing the os-jdk prefix from the job labels.

`testDLQWriterFlusherRemovesExpiredSegmentWhenCurrentHeadSegmentIsEmpty`
fails on Windows Buildkite agents and it's a test issue tracked in
https://github.com/elastic/logstash/issues/15562.

Relates:

- https://github.com/elastic/logstash/pull/15539
- https://github.com/elastic/ingest-dev/issues/1725
2023-11-10 17:25:13 +02:00
Dimitrios Liappis
f87651fee0
[ci] Remove 8.10 scheduled builds for DRA (#15544)
This commit removes 8.10 from the automated builds of DRA
artifacts, and should be merged once 8.11.0 is out.
2023-11-10 17:18:10 +02:00
Andrea Selva
5af14f4e1c
Fixed functional test in case the LS_JAVA_HOME is configured (#15535)
Adds filtering on Logstash output message in an integration tests when setting LS_JAVA_HOME environment variable.
2023-11-10 11:01:42 +01:00
Dimitrios Liappis
956bf483f2
[ci] JDK matrix Buildkite pipelines (part 1) (#15539)
This commit is the first part of the migration of JDK matrix tests
from Jenkins to Buildkite. There will be two separate pipelines, for
Linux and Windows.

Linux is currently limited to Ubuntu 22.04 and 20.04, but
additional operating systems will be added outside of the Logstash
repository seamlessly through additional VM images.

Steps are created dynamically and the underlying script is meant to be
common for Linux and Windows. Windows is currently a stub and
will be added in a follow up PR.

Relates:

- https://github.com/elastic/ingest-dev/issues/1725
- https://github.com/elastic/ci-agent-images/pull/424
2023-11-09 09:53:27 +02:00
github-actions[bot]
c762deceae
Release notes for 8.11.0 (#15465) (#15548)
Co-authored-by: Mashhur <99575341+mashhurs@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
(cherry picked from commit d4715a36c0)
2023-11-07 19:10:04 -05:00
Ry Biesemeyer
a7e2839a83
build meta: bump 8.x snapshot to 8.12 (#15546)
once 8.11 was cut, snapshot build for 8.x should be 8.12
2023-11-07 14:34:08 -08:00
Andrea Selva
7a055c34d1
Fixed definitions of licenses for some dependencies (#15540)
Update some dependency's licenses definitions.
2023-11-07 17:12:28 +01:00
João Duarte
bd6189db8e
Update JRuby to 9.4.5.0 (#15531) 2023-11-07 13:41:04 +00:00
Dimitrios Liappis
07147b3e40
[ci] Split JDK matrix pipelines per OS (#15534)
This commit splits the generic Buildkite pipelines introduced
in #15520 for JDK tests to separate pipelines for Linux and Windows.
2023-11-07 15:22:37 +02:00
Dimitrios Liappis
b86ad3038a
[ci] Split JDK matrix resources per OS (#15533)
This commit splits the generic Buildkite catalog resource introduced
in #15519 for JDK tests to separate resources for Linux and Windows.
2023-11-07 14:19:47 +02:00
Mashhur
fa1382fd22
Update the Logstash to Logstash Native doc to reflect the multiple hosts usage. (#15512)
* Update the Logstash to Logstash Native doc to reflect the multiple hosts usage.

* Logstash to Logstash comm page, adding LS-to-LS native HA support.

* Apply suggestions from code review

Refining the context.

Co-authored-by: Andres Rodriguez <andreserl@gmail.com>

---------

Co-authored-by: Andres Rodriguez <andreserl@gmail.com>
2023-11-06 12:32:48 -08:00
Ry Biesemeyer
51886b9102
geoip: extract database manager to stand-alone feature (#15348)
* geoip: extract database manager to stand-alone feature

Introduces an Elastic-licensed GeoipDatabaseManagement tool that can be used
by ANY plugin running on Elastic-licensed Logstash to retrieve a subscription
to a GeoIP database that ensures EULA-compliance and frequent updates, and
migrates the previous Elastic-licensed code-in-Logstash-core extension to
the Geoip Filter to use this new tool, requiring ZERO changes to in-the-wild
versions of the plugin.

The implementation of the new tool follows the previous implementation as
closely as possible, but presents a new interface that ensures that a
consumer can ATOMICALLY subscribe to a database path without risk that the
subscriber will receive an update or expiry before it is finished applying
the initial value:

~~~ ruby
geoip_manager = LogStash::GeoipDatabaseManagement::Manager.instance
subscription = geoip_manager.subscribe('City')

subscription.observe(construct: ->(initial_dbinfo){ },
                     on_update: ->(updated_dbinfo){ },
                     on_expire: ->(       _      ){ })

subscription.release!
~~~

* docs: link in geoip database manager docs

* docs: reorganize pending 'geoip database management' feature

* docs: link to geoip pages from feature index

* geoip: add SubscriptionObserver "interface"

simplifies using Subscription#observe from Java

* geoip: fixup SubscriptionObserver after rename

* geoip: quacking like a SubscriptionObserver is enough

* geoip: simplify constants of legacy geoip filter extension

* geoip: bump logging level to debug for non-actionable log

* geoip: refine log message to omit non-actionable info

* re-enable invokedynamic (was disabled to avoid upstream bug)

* geoip: resolve testing fall-out from filter extension's "private" constants removal

* geoip: consistently use `DataPath#resolve` internally, too
2023-11-06 09:22:23 -08:00
Andres Rodriguez
e3584fd53e
Plugin Tests: Exit 1 on error (#15527)
The plugin tests were not correctly exiting when plugin errors were present. It will now correctly exit 1 if there are plugins with errors.
2023-11-03 12:38:01 -04:00
Andres Rodriguez
cf71dae3ff
Add support to test unsupported plugins (#15526)
Add support to test unsupported plugins. Only enable input-rss for now.
2023-11-03 12:08:01 -04:00
Dimitrios Liappis
ccc41d76ff
[ci] Initial JDK matrix Buildkite pipeline (#15520)
This commit adds a skeleton Buildkite pipeline for the JDK matrix tests.

Relates:

- https://github.com/elastic/logstash/pull/15519
- https://github.com/elastic/ingest-dev/issues/1725
2023-11-02 17:51:50 +02:00
Dimitrios Liappis
5df18f1053
[ci] Initial jdk matrix Buildkite resource (#15519)
This commit adds a skeleton resource definition for a Buildkite
pipeline for JDK matrix tests.

Relates:
https://github.com/elastic/ingest-dev/issues/1725
https://github.com/elastic/logstash/pull/15520
2023-11-02 17:51:27 +02:00
Andrea Selva
73daec05ed
Download of JDK from the Elastic catalog instead of Adoptium (#15514)
* Adapted the JDK's download URL creation to intereact with Elastic catalog to get metadata, and return the catalog download link instead of directly pointing to Adoptium API

* Silenced the Download task of JDK to print the full url
2023-10-30 16:24:27 +01:00
João Duarte
206362212a
Update JDK to 17.0.9+9 (#15509) 2023-10-27 10:20:26 +01:00
Rob Bavey
a398c93eec
Update Iron Bank base image to ubi9.2 (#15490) 2023-10-26 09:53:29 -04:00
Dimitrios Liappis
1b794dfcd6
[ci] Slack notifications for aarch64 pipeline (#15507)
This commit enables slack notifications for aarch64 Buildkite pipeline
failures.
2023-10-26 15:20:16 +03:00
Dimitrios Liappis
c384190718
[ci] aarch64 Buildkite pipeline part 2 (#15473)
This commit is the follow up PR after #15466, which migrates away
the remaining aarch64 acceptance test Jenkins jobs to Buildkite.

Relates:

- #15466
- https://github.com/elastic/ingest-dev/issues/1724
2023-10-25 18:08:56 +03:00
Andrea Selva
90964fb559
Updates callsites synthax for i18n.t method to avoid deprecated and prohibited format (#15500)
Updates invocations of i18n.t method which are leftovers and missed in the original Ruby 3.1 update PR #14861

Without this, some error reporting logs are hidden by the mismatch of arguments error in translate the error message.
2023-10-25 11:42:46 +02:00
Dimitrios Liappis
16da966290
[ci] re-enable Java unit tests on aarch64 pipeline (#15492)
PR#15466 skipped the Java unit tests as on the `main` and `8.11`
branches they attempted to run sonar scans (which are only meant to
run for PRs).

This commit re-enables the Java unit tests, taking advantage of #15486,
disabling the sonar scan part of the test suite.
2023-10-24 18:05:14 +03:00
Dimitrios Liappis
b1ab2fad3e
Option to disable SonarQube with Java Tests (#15486)
This commit introduces a way to optionally disable SonarQube scanning
and coverage reports when running Java unit tests. The integration
was introduced in #15279, however, there are cases (e.g. running Java
unit tests outside of PRs) where we don't want this integration.

Disabling can be achieved by setting the env var `ENABLE_SONARQUBE`
to `false`.
2023-10-24 10:32:44 +03:00
Rob Bavey
57cc392d0e
Update ubi8 base image to 8.7 (#15487) 2023-10-23 12:47:27 -04:00
Dimitrios Liappis
36656de4f0
[ci] aarch64 Buildkite pipeline part 1 (#15466)
This commit is the first part of migrating away the aarch64 Jenkins
jobs to Buildkite. It adds a group of exhaustive test steps in the
aarch64 pipeline.

The java unit tests are temporarily disabled as they run SonarQube
scans which need to be associated with pull requests.

Relates:

https://github.com/elastic/ingest-dev/issues/1724
2023-10-23 15:01:14 +03:00
João Duarte
e626409695
Update JRuby, JDK and Jackson (#15477)
* update JDK to 17.0.8.1+1
* update JRuby to 9.4.4.0
* update jackson to 2.15.3
2023-10-23 11:17:01 +01:00
João Duarte
6046402a35
Remove pinning of psych 5.1.0 (#15474)
https://github.com/ruby/psych/issues/655 has been fixed with the release of 5.1.1.1
2023-10-18 16:02:37 +01:00
Dimitrios Liappis
0083738cde
[ci] Initial aarch64 Buildkite pipeline (#15460)
This commit adds a skeleton Buildkite pipeline for aarch64
exhaustive tests.

- https://github.com/elastic/logstash/pull/15459
- https://github.com/elastic/ingest-dev/issues/1724
2023-10-18 09:45:05 +03:00
Dimitrios Liappis
9aa291a2a1
[ci] Initial aarch64 Buildkite pipeline (#15459)
This commit adds a skeleton resource definition for a Buildkite
pipeline for aarch64 exhaustive tests.

Relates:
https://github.com/elastic/ingest-dev/issues/1724
2023-10-18 09:27:34 +03:00
Rob Bavey
860785bbb0
Release notes for 8.10.4 (#15435) (#15463)
* Update release notes for 8.10.4
Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Co-authored-by: Karen Metts <karen.metts@elastic.co>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-10-17 09:54:08 -04:00
Dimitrios Liappis
9a924f45ef
[ci] Slack notifications for PR Buildkite builds (#15452)
This commit enables Slack notifications for the pull request Buildkite
pipeline.
2023-10-16 18:32:19 +03:00
Andres Rodriguez
235ba422cc
Change plugin test buildkite pipeline to run in parallel
Closes #15440
2023-10-16 10:42:05 -04:00
Dimitrios Liappis
1284c675d1
[ci] Enable Buildkite PR comments (#15453)
This commit enables automatically updated comments from Buildkite for
PRs. It posts comments to PRs whenever a Buildkite build finishes or
encounters its first failing step and keeps the comment updated.
2023-10-16 17:12:14 +03:00
Dimitrios Liappis
784ca0243e
[docs] Small typo fixes in README (#15429)
Fix a few typos in the Logstash README.md
2023-10-16 16:49:52 +03:00
Dimitrios Liappis
555a8ff8e3
Migrate remaining PR jobs to Buildkite #15444
This commit adds the remaining steps for the Buildkite pull request pipeline i.e.

- `IT Persistent Queues / part 1`
- `IT Persistent Queues / part 2`
- `x-pack unit tests`
- `x-pack integration`

Once merged we will be able to retire the corresponding Logstash pull request multijob.

Relates:

- https://github.com/elastic/logstash/pull/15438
- https://github.com/elastic/logstash/pull/15437
- https://github.com/elastic/ingest-dev/issues/1721
- https://github.com/elastic/logstash/pull/15279
2023-10-14 09:46:52 +03:00
Dimitrios Liappis
816d7e6b2b
[ci] Add PR it-tests (part 1+2) to Buildkite (#15438)
This commit adds integration tests to the
Buildkite PR pipeline.

Relates:

- https://github.com/elastic/logstash/pull/15437
- https://github.com/elastic/ingest-dev/issues/1721
- https://github.com/elastic/logstash/pull/15279
2023-10-13 17:01:01 +03:00
Dimitrios Liappis
e285425d54
[ci] Commit PR based Java unit tests for Buildkite (#15437)
This commit adds Java unit tests (including sonar scans) to the
Buildkite pull request pipeline.

Relates: https://github.com/elastic/ingest-dev/issues/1721
2023-10-13 10:48:10 +03:00
Mashhur
3a753e49e8
Update Logstash core versions in CI after release. (#15424) 2023-10-12 22:57:43 -07:00
Dimitrios Liappis
4ee74c9de6
[ci] Daily DRA snapshot builds with Buildkite (#15395)
This commit enables the snapshot job schedule for Buildkite jobs.
They are set to run after Jenkins (scheduled @midnight) so that any
remote/unknown chance of conflict with the release manager is limited.

While at it, we also enable slack notifications for failures to the
same channels as Jenkins.
2023-10-11 21:00:45 +03:00
João Duarte
401d166a89
pin psych 5.1.0 due to ruby/psych#655 (#15433)
The new psych 5.1.1 gem seems to not work when installed in JRuby 9.4 (used in main).

This change pins the version back to 5.1.0 until ruby/psych#655 is sorted.
2023-10-11 14:58:42 +01:00
Andrea Selva
a03a05b697
Defines a pipeline to run specs tests on supported plugins (#15380)
Defines a Buildkite pipeline to run specs tests on tier1 and tier2 plugins, using container images. The tasks are divided by plugin type for each tier, to run them in parallel. Once all tier1 testing is completed then starts with tier2.
2023-10-11 11:40:54 +02:00
Dimitrios Liappis
d4eccd256f
[ci] Don't trigger BK PR jobs on pushes (#15426)
Currently while PR jobs get triggered as expected in Buildkite via
the bot, they also get re-triggered after something gets pushed,
including e.g. the squash merge commit when the PR is merged.

This commit disables this behavior as documented in Buildkite[^1].
and by tightening our filter.

Relates:

- #15402
- https://github.com/elastic/ingest-dev/issues/1721

[^1]: https://buildkite.com/docs/apis/rest-api/pipelines
2023-10-11 11:45:30 +03:00
Dimitrios Liappis
fd2910a6ef
[ci] Improve PR step names for Buildkite (#15421)
This commit shortens the descriptions of the PR Buildkite steps for
better readability on the UI.
2023-10-11 09:29:30 +03:00