Commit graph

10907 commits

Author SHA1 Message Date
Toby Sutor
8142ef104a
Document JDK 21 since Logstash 8.15.0
Since Logstash 8.15.0, we are shipping JDK 21 as per https://www.elastic.co/guide/en/logstash/current/logstash-8-15-0.html:

```
IMPLEMENTOR="Eclipse Adoptium"
IMPLEMENTOR_VERSION="Temurin-21.0.5+11"
JAVA_RUNTIME_VERSION="21.0.5+11-LTS"
JAVA_VERSION="21.0.5"
JAVA_VERSION_DATE="2024-10-15"
LIBC="gnu"
MODULES="java.base java.compiler java.datatransfer java.xml java.prefs java.desktop java.instrument java.logging java.management java.security.sasl java.naming java.rmi java.management.rmi java.net.http java.scripting java.security.jgss java.transaction.xa java.sql java.sql.rowset java.xml.crypto java.se java.smartcardio jdk.accessibility jdk.internal.jvmstat jdk.attach jdk.charsets jdk.internal.opt jdk.zipfs jdk.compiler jdk.crypto.ec jdk.crypto.cryptoki jdk.dynalink jdk.internal.ed jdk.editpad jdk.hotspot.agent jdk.httpserver jdk.incubator.vector jdk.internal.le jdk.internal.vm.ci jdk.internal.vm.compiler jdk.internal.vm.compiler.management jdk.jartool jdk.javadoc jdk.jcmd jdk.management jdk.management.agent jdk.jconsole jdk.jdeps jdk.jdwp.agent jdk.jdi jdk.jfr jdk.jlink jdk.jpackage jdk.jshell jdk.jsobject jdk.jstatd jdk.localedata jdk.management.jfr jdk.naming.dns jdk.naming.rmi jdk.net jdk.nio.mapmode jdk.random jdk.sctp jdk.security.auth jdk.security.jgss jdk.unsupported jdk.unsupported.desktop jdk.xml.dom"
OS_ARCH="x86_64"
OS_NAME="Linux"
SOURCE=".:git:9f7db4edca1b"
BUILD_SOURCE="git:08621c72262ba260e4bb6451a9c75bf1c0ab365d"
BUILD_SOURCE_REPO="https://github.com/adoptium/temurin-build.git"
SOURCE_REPO="https://github.com/adoptium/jdk21u.git"
FULL_VERSION="21.0.5+11-LTS"
SEMANTIC_VERSION="21.0.5+11"
BUILD_INFO="OS: Linux Version: 6.5.0-1025-azure"
JVM_VARIANT="Hotspot"
JVM_VERSION="21.0.5+11-LTS"
IMAGE_TYPE="JDK"
```

This needs to be reflected in the documentation and backported to 8.15.0 and newer.
2024-12-11 12:58:42 +01:00
Cas Donoghue
095fbbb992
Add breaking changes docs for input-elasticsearch (#16744)
This commit follows the pattern established in
https://github.com/elastic/logstash/pull/16701 for indicating obsolete ssl
settings in logstash core plugins.
2024-12-09 13:24:07 -08:00
João Duarte
e36cacedc8
ensure inputSize state value is reset during buftok.flush (#16760) 2024-12-09 09:10:54 -08:00
Ry Biesemeyer
202d07cbbf
ensure jackson overrides are available to static initializers (#16719)
Moves the application of jackson defaults overrides into pure java, and
applies them statically _before_ the `org.logstash.ObjectMappers` has a chance
to start initializing object mappers that rely on the defaults.

We replace the runner's invocation (which was too late to be fully applied) with
a _verification_ that the configured defaults have been applied.
2024-12-04 14:27:26 -08:00
Rob Bavey
ab19769521
Pin date dependency to 3.3.3 (#16755)
Resolves: #16095, #16754
2024-12-04 14:39:50 -05:00
Mashhur
4d9942d68a
Update usage of beats-input obsoleted SSL params in the core. (#16753) 2024-12-04 11:12:21 -08:00
Rob Bavey
e3265d93e8
Pin jar-dependencies to 0.4.1 (#16747)
Pin jar-dependencies to `0.4.1`, until https://github.com/jruby/jruby/issues/7262
is resolved.
2024-12-03 17:35:00 -05:00
João Duarte
af76c45e65
Update logstash_releases.json to account for 7.17.26 GA (#16746) 2024-12-03 15:25:55 +00:00
João Duarte
1851fe6b2d
Update logstash_releases.json to account for GA of 8.15.5 (#16734) 2024-11-27 14:09:22 +00:00
mmahacek
d913e2ae3d
Docs: Troubleshooting update for JDK bug handling cgroups v1 (#16721)
---------
Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
2024-11-27 11:08:20 +00:00
Rob Bavey
ccde1eb8fb
Add SSL Breaking changes for logstash-output-http to breaking changes… (#16701)
* Add SSL Breaking changes for logstash-output-http to breaking changes doc
* Add missing discrete tags
* Improve formatting of plugin breaking chagnes

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
2024-11-26 16:23:44 -05:00
Rob Bavey
0e58e417ee
Increase timeout for docs PR link action (#16718)
Update the timeout from 15 minutes to 30 minutes to try and fix the inline docs previews feature, and cleanup comments
2024-11-22 11:57:15 -05:00
Karen Metts
615545027f
Doc: Roll 6.3 breaking changes under 6.0 series (#16706) 2024-11-22 11:16:02 -05:00
Cas Donoghue
eb7e1253e0
Revert "Bugfix for BufferedTokenizer to completely consume lines in case of lines bigger then sizeLimit (#16482)" (#16715)
This reverts commit 85493ce864.
2024-11-21 09:18:59 -08:00
João Duarte
aff8d1cce7
update logstash_release with 8.16.1 and 8.16.2-SNAPSHOT (#16711) 2024-11-21 10:03:50 +00:00
João Duarte
2f0e10468d
Update logstash_releases.json to account for 8.17 Feature Freeze (#16709) 2024-11-21 09:35:33 +00:00
github-actions[bot]
0dd64a9d63
PipelineBusV2 deadlock proofing (#16671) (#16682)
* pipeline bus: add deadlock test for unlisten/unregisterSender

* pipeline bus: eliminate deadlock

Moves the sync-to-notify out of the `AddressStateMapping#mutate`'s effective
synchronous block to eliminate a race condition where unlistening to an address
and unregistering a sender could deadlock.

It is safe to notify an AddressState's attached input without exclusive access
to the AddressState, because notifying an input that has since been disconnected
is net zero harm.

(cherry picked from commit 8af6343a26)

Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
2024-11-20 13:26:13 -08:00
Mashhur
15b203448a
[Health API E2E] Align on agent python version and avoid pip install. (#16704)
* Pip install is unnecessary and sometimes hangs with prompt dialog. This commit aligns on agent python version which has a default pip installed.

* Update .buildkite/scripts/health-report-tests/main.sh

Improve readability.

Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>

---------

Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>
2024-11-20 10:26:02 -08:00
Cas Donoghue
7b3d23b9d5
Replace removed yaml module (#16703)
In https://github.com/elastic/logstash/pull/16586 the module include was
removed. This causes failures in the script when the module is referenced. This
commit re-enables the include for the yaml module.
2024-11-20 10:13:30 -08:00
João Duarte
977efbddde
Update branches.json to include 8.15, 8.16 and 8.17 (#16698) 2024-11-20 16:37:10 +00:00
Cas Donoghue
e0ed994ab1
Update license checker with new logger dependency (#16695)
A new transative dependency on the `logger` gem has been added through sinatra 4.1.0. Update the
license checker to ensure this is accounted for.
2024-11-20 11:49:28 +00:00
Andrea Selva
d4fb06e498
Introduce a new flag to explicitly permit legacy monitoring (#16586)
Introduce a new flag setting `xpack.monitoring.allow_legacy_collection` which eventually enable the legacy monitoring collector.

Update the method to test if monitoring is enabled so that consider also `xpack.monitoring.allow_legacy_collection` to determine if `monitoring.*` settings are valid or not.
By default it's false, the user has to intentionally enable it to continue to use the legacy monitoring settings.


---------

Co-authored-by: kaisecheng <69120390+kaisecheng@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
2024-11-19 08:52:28 +01:00
Andrea Selva
a94659cf82
Default buffer type to 'heap' for 9.0 (#16500)
Switch the default value of `pipeline.buffer.type` to use the heap memory instead of direct one.

Change the default value of the setting `pipeline.buffer.type` from direct to heap and update consequently the documentation.

Co-authored-by: João Duarte <jsvd@users.noreply.github.com>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
2024-11-13 16:58:56 +01:00
João Duarte
2a23680cfd
Update logstash_releases.json for new branching strategy (#16585)
See naming rational in https://github.com/logstash-plugins/.ci/pull/63#issue-2597373955
After 8.16 is GA, the "releases" entry should become:

```json
  "releases": {
    "7.current": "7.17.24",
    "8.current": "8.16.0",
    "8.previous": "8.15.3"
  },
```

For snapshots we'll also test against "main", "8.next", and "8.future". The labels are:

- `main`: main branch
-  `8.future`: the future 8.x release, i.e. current version of the 8.x branch
- `8.next`: the short lived period between a minor's FF - when the new branch is cut from 8.x - and GA
- `8.current`: the most recent 8.x release
- `8.previous`: the previous, but still supported, 8.x release
2024-11-13 11:04:02 +00:00
Cas Donoghue
ff8c154c4d
Extend ruby linting tasks to handle file inputs (#16660)
This commit extends the gradle and rake tasks to pass through a list of files
for rubocop to lint. This allows more specificity and fine grained control for
linting when the consumer of the tasks only wishes to lint a select few files.
2024-11-12 12:46:52 -08:00
Yehor Shvedov
74d87c9ea0
Update catalog-info file with correct system property (#16669) 2024-11-12 21:40:17 +02:00
kaisecheng
5826c6f902
Use UBI as base image (#16599)
Logstash Docker images, full and OSS, now use UBI image as its base, replacing the previous Ubuntu base.

- change the base image of `full` and `oss` to ubi
- Set locale to C.UTF-8
- remove ubi flavour
- use go image to build env2yaml
- remove redundant and refactor steps
- add support to build image in mac aarch64
- allow customizing ELASTIC_VERSION and LOCAL_ARTIFACTS for test purpose
2024-11-08 16:25:12 +00:00
Ellie
d9ead9a8db
Remove link to deleted cluster (#16658) 2024-11-08 09:03:47 -05:00
Nicole Albee
046ea1f5a8
For custom java plugins, set the platform = 'java'. (#16628) 2024-11-06 08:48:21 +00:00
João Duarte
efbee31461
Update .ruby-version to jruby-9.4.9.0 (#16642) 2024-11-06 08:22:37 +00:00
kaisecheng
5847d77331
skip allow_superuser in Windows OS (#16629)
As user id is always zero in Windows,
this commit excluded the checking of running as root in Windows.
2024-11-05 15:37:19 +00:00
Nicole Albee
113585d4a5
Anchor the -java match pattern at the end of the string. (#16626)
This fixes the offline install problem of the logstash-input-java_filter_example off-line install.
2024-11-05 14:21:15 +00:00
João Duarte
6703aec476
bump jruby to 9.4.9.0 (#16634) 2024-11-05 13:49:07 +00:00
kaisecheng
849f431033
fix Windows java not found log (#16633) 2024-11-05 10:15:51 +00:00
Andrea Selva
852149be2e
Update JDK to latest in versions.yml (#16627)
Update JDK to version 21.0.5+11
2024-11-04 16:40:20 +01:00
kaisecheng
8ce58b8355
run acceptance tests as non-root (#16624) 2024-11-01 19:47:08 +00:00
kaisecheng
00da72378b
add boostrap to docker build to fix missing jars (#16622)
The DRA build failed because the required jars were missing, as they had been removed during the Docker build process.
2024-11-01 15:45:35 +00:00
Karen Metts
0006937e46
Doc: Reset breaking changes and release notes for 9.0 (#16603) 2024-10-31 18:06:52 -04:00
João Duarte
9eced9a106
reduce effort during build of docker images (#16619)
there's no need to build jdk-less and windows tarballs for docker images
so this change simplifies the build process.

It should reduce the time spent needed to build docker images.
2024-10-31 16:00:25 +00:00
João Duarte
472e27a014
make docker build and gradle tasks more friendly towards ci output (#16618) 2024-10-31 16:00:11 +00:00
kaisecheng
db59cd0fbd
set allow_superuser to false as default (#16558)
- set allow_superuser as false by default for v9
- change the buildkite image of ruby unit test to non-root
2024-10-31 13:33:00 +00:00
Andrea Selva
c602b851bf
[CI] Change agent for JDK availability check and add schedule also for 8.x (#16614)
Switch execution agent of JDK availability check pipeline from vm-agent to container-agent.
Moves the schedule definition from the `Logstash Pipeline Scheduler` pipeline into the pipeline definition, adding a schedule also for `8.x` branch.
2024-10-30 12:13:28 +01:00
Andrea Selva
5d523aa5c8
Fix bad reference to a variable (#16615) 2024-10-30 11:44:41 +01:00
Andrea Selva
ed5874bc27
Use jvm catalog for reproducible builds and expose new pipeline to check JDK availability (#16602)
Updates the existing `createElasticCatalogDownloadUrl` method to use the precise version retrieved `versions.yml` to download the JDK instead of using the latest of major version. This makes the build reproducible again.
Defines a new Gradle `checkNewJdkVersion` task to check if there is a new JDK version available from JVM catalog matching the same major of the current branch. 
Creates a new Buildkite pipeline to execute a `bash` script to run the Gradle task; plus it also update the `catalog-info.yaml` with the new pipeline and a trigger to execute every week.
2024-10-29 10:55:15 +01:00
João Duarte
ca19f0029e
make max inflight warning global to all pipelines (#16597)
The current max inflight error message focuses on a single pipeline and on a maximum amount of 10k events regardless of the heap size.

The new warning will take into account all loaded pipelines and also consider the heap size, giving a warning if the total number of events consumes 10% or more of the total heap.

For the purpose of the warning events are assumed to be 2KB as it a normal size for a small log entry.
2024-10-25 14:44:33 +01:00
Edmo Vamerlatti Costa
93b0913fd9
Updated CI releases inventory after 7.17.25 (#16589) 2024-10-22 15:40:02 +02:00
kaisecheng
566bdf66fc
remove http.* settings (#16552)
The commit removes the deprecated settings http.port, http.host, http.enabled, and http.environment
2024-10-18 12:15:38 +01:00
kaisecheng
467ab3f44b
Enable log.format.json.fix_duplicate_message_fields by default (#16578)
Set `log.format.json.fix_duplicate_message_fields` to `true` as default
to avoid collision of the message field in log lines when log.format is JSON
2024-10-17 16:22:30 +01:00
Edmo Vamerlatti Costa
dcafa0835e
Updated CI releases inventory after 8.15.3 (#16581) 2024-10-17 16:55:31 +02:00
João Duarte
daf979c189
default to jdk 21 on all ci/build tasks (#16464) 2024-10-17 15:47:52 +01:00