This updates the gradle wrapper to 8.12
We addressed deprecation warnings due to the update that includes:
- Fix change in TestOutputEvent api
- Fix deprecation in groovy syntax
- Use latest ospackage plugin containing our fix
- Remove project usages at execution time
- Fix deprecated project references in repository-old-versions
(cherry picked from commit ba61f8c7f7)
# Conflicts:
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/distribution/DockerCloudElasticsearchDistributionType.java
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/distribution/DockerUbiElasticsearchDistributionType.java
# build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/test/Fixture.java
# plugins/repository-hdfs/hadoop-client-api/build.gradle
# server/src/main/java/org/elasticsearch/inference/ChunkingOptions.java
# x-pack/plugin/kql/build.gradle
# x-pack/plugin/migrate/build.gradle
# x-pack/plugin/security/qa/security-basic/build.gradle
Here, we only need to extract the minimum and maximum values of the
timestamp field; therefore, using a stats searcher should suffice. This
is important for frozen indices.
* Propagate scoring function through random sampler.
* Update docs/changelog/116957.yaml
* Correct score mode in random sampler weight
* Fix random sampling with scores and p=1.0
* Unit test with scores
* YAML test
* Add capability
This test doesn't need to run in FIPS mode, and apparently it fails
sometimes, so with this commit we skip it.
Closes#116811
(cherry picked from commit ea90fbc10d)
# Conflicts:
# muted-tests.yml
Co-authored-by: David Turner <david.turner@elastic.co>
We don't seem to have a test that completely verifies that a S3
repository can reload credentials from an updated keystore. This commit
adds such a test.
Backport of #116762 to 8.16.
Clarifies that insecure settings are stored in plaintext and must not be
used. Also removes the mention of the (wrong) system property from the
error message if insecure settings are not permitted.
Backport of #116915 to `8.16`
* Align dot prefix validation with Serverless (#116266)
This aligns the deprecation warnings for on-prem dot-prefixed indices to
be the same as the Serverless validation. It adds exemptions for the
`.entities…` indices, and makes the list a dynamic setting.
(cherry picked from commit 72aa17a0c3)
* Fix compilation
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
A `CompleteMultipartUpload` action may fail after sending the `200 OK`
response line. In this case the response body describes the error, and
the SDK translates this situation to an exception with status code 0 but
with the `ErrorCode` string set appropriately. This commit enhances the
exception handling in `S3BlobContainer` to handle this possibility.
Closes#102294
Co-authored-by: Pat Patterson <metadaddy@gmail.com>
Ignore warnings in recently added tests. The underlying issue was fixed on main in #115799 . Rather than backport the larger fix, we can ignore the warning in yaml tests.
* Resolve pipelines from template if lazy rollover write (#116031)
If datastream rollover on write flag is set in cluster state, resolve pipelines from templates rather than from metadata. This fixes the following bug: when a pipeline reroutes every document to another index, and rollover is called with lazy=true (setting the rollover on write flag), changes to the pipeline do not go into effect, because the lack of writes means the data stream never rolls over and pipelines in metadata are not updated. The fix is to resolve pipelines from templates if the lazy rollover flag is set. To improve efficiency we only resolve pipelines once per index in the bulk request, caching the value, and reusing for other requests to the same index.
Fixes: #112781
* Remute tests blocking merge
* Remute tests blocking merge
This commit removes `com.google.crypto.tink` which is transitive and optional dependency of `oauth2-oidc-sdk` and `nimbus-jose-jwt`. We don't seem to be using any functionality that requires `tink` and thus `protobuf-java`. Removing them feels safer than having to maintain misaligned versions.
(cherry picked from commit 6b32bced36)
* [Test] Flush response body for progress (#115177)
In JDK23, response headers are no longer always immediately sent. See
also https://bugs.openjdk.org/browse/JDK-8331847 This PR adds flush call
for the response body to make progress.
Resolves: #115145Resolves: #115164
(cherry picked from commit 8c23fd7712)
* unmute
In LogsDB we would like to use a default value of `8191` for the index-level setting
`index.mapping.ignore_above`. The value for `ignore_above` is the _character count_,
but Lucene counts bytes. Here we set the limit to `32766 / 4 = 8191` since UTF-8
characters may occupy at most 4 bytes.
(cherry picked from commit 521e4341d7)
# Conflicts:
# server/src/main/java/org/elasticsearch/common/settings/Setting.java
Co-authored-by: Salvatore Campagna <93581129+salvatore-campagna@users.noreply.github.com>
* Reconcile differences between Stateful and Stateless dot-prefix validation (#114946)
This commit makes the dot prefix deprecation match the existing changes to validation for the SLO
and SLA UIs.
Relates to #112571
(cherry picked from commit 5bf446ea2e)
* Update not to use getFirst
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Replace IntermittentLongGCDisruption with blocking cluster state updates (#115075)
In JDK 23 `Thread.resume` has been removed this means that we cannot use
`IntermittentLongGCDisruption` that depends on it.
We simulate the master node disruption with a `CyclicBarrier` that
blocks cluster state updates.
Closes: https://github.com/elastic/elasticsearch/issues/115045
The backport will close:
https://github.com/elastic/elasticsearch/issues/112634
* Unmute LazyRolloverDuringDisruptionIT
**Introduction**
> In order to make adoption of failure stores simpler for all users, we
are introducing a new syntactical feature to index expression
resolution: The selector. > > Selectors, denoted with a :: followed by a
recognized suffix will allow users to specify which component of an
index abstraction they would like to operate on within an API call. In
this case, an index abstraction is a concrete index, data stream, or
alias; Any abstraction that can be resolved to a set of indices/shards.
We define a component of an index abstraction to be some searchable unit
of the index abstraction. > > To start, we will support two components:
data and failures. Concrete indices are their own data components, while
the data component for index aliases are all of the indices contained
therein. For data streams, the data component corresponds to their
backing indices. Data stream aliases mirror this, treating all backing
indices of the data streams they correspond to as their data component.
> > The failure component is only supported by data streams and data
stream aliases. The failure component of these abstractions refer to the
data streams' failure stores. Indices and index aliases do not have a
failure component.
For more details and examples see
https://github.com/elastic/elasticsearch/pull/113144. All this work has
been cherry picked from there.
**Purpose of this PR**
This PR is replacing the `FailureStoreOptions` with the
`SelectorOptions`, there shouldn't be any perceivable change to the user
since we kept the query parameter "failure_store" for now. It will be
removed in the next PR which will introduce the parsing of the
expressions.
_The current PR is just a refactoring and does not and should not change
any existing behaviour._
Today the overloads of `XContentBuilder#timeField` do two rather
different things: one formats an object as a `String` representation of
a time (where the object is either an unambiguous time object or else a
`long`) and the other formats only a `long` as one or two fields
depending on the `?human` flag.
This is trappy in a number of ways:
- `long` means an absolute (epoch) time, but sometimes folks will
mistakenly use this for time intervals too.
- `long` means only milliseconds, there is no facility to specify a
different unit.
- the dependence on the `?human` flag in exactly one of the overloads is
kinda weird.
This commit removes the confusion by dropping support for considering a
`Long` as a valid representation of a time at all, and instead requiring
callers to either convert it into a proper time object or else call a
method that is explicitly expecting an epoch time in milliseconds.
With logsdb another index mode is available, the isTimeSeries parameter is limiting. Instead, we should just push down the index mode from template to index settings provider.
Follow up from #113451
Relates to #113583
In this PR we introduce two endpoint PUT and GET to manage the data
stream options and consequently the failure store configuration on the
data stream level. This means that we can manage the failure store of
existing data streams.
The APIs look like:
```
# Enable/disable
PUT _data_stream/my-data-stream/_options
{
"failure_store": {
"enabled": true
}
}
# Remove existing configuration
DELETE _data_stream/my-data-stream/_options
# Retrieve
GET _data_stream/my-data-stream/_options
{
"failure_store": {
"enabled": true
}
}
```
Future work:
- Document the new APIs
- Convert `DataStreamOptionsIT.java` to a yaml test.
After introducing #113505 we need to remove `cluster.logsdb.enabled` from
`StackPlugin` and `StackTemplateRegistry`.
(cherry picked from commit 487c5562aa)
Co-authored-by: Salvatore Campagna <93581129+salvatore-campagna@users.noreply.github.com>
* Create a fluent builder to help implement ChunkedToXContent (#112389)
Rather than manually adding startObject/endObject, and having to line everything up manually, this handles the start/end for you.
A few implementations are converted already. In the long run, I would like this to replace ChunkedXContentHelper.
* Convert a few more implementations to ChunkedXContentBuilder (#113125)
Remove the complex methods from ChunkedXContentHelper
* Further conversions to ChunkedXContentBuilder (#114237)
---------
Co-authored-by: Simon Cooper <simon.cooper@elastic.co>
**Introduction**
> In order to make adoption of failure stores simpler for all users, we
are introducing a new syntactical feature to index expression
resolution: The selector. > > Selectors, denoted with a :: followed by a
recognized suffix will allow users to specify which component of an
index abstraction they would like to operate on within an API call. In
this case, an index abstraction is a concrete index, data stream, or
alias; Any abstraction that can be resolved to a set of indices/shards.
We define a component of an index abstraction to be some searchable unit
of the index abstraction. > > To start, we will support two components:
data and failures. Concrete indices are their own data components, while
the data component for index aliases are all of the indices contained
therein. For data streams, the data component corresponds to their
backing indices. Data stream aliases mirror this, treating all backing
indices of the data streams they correspond to as their data component.
> > The failure component is only supported by data streams and data
stream aliases. The failure component of these abstractions refer to the
data streams' failure stores. Indices and index aliases do not have a
failure component.
For more details and examples see
https://github.com/elastic/elasticsearch/pull/113144. All this work has
been cherry picked from there.
**Purpose of this PR**
This PR is replacing the the indices options boolean constructor with
the builders. The goal is to give me and the reviewer a very narrow
scope change when we can ensure we did not make any mistakes during the
conversion. Also it will reduce a bit the change list in
https://github.com/elastic/elasticsearch/pull/113144/files.
These test overrides were introduced so that we had somewhere to hang an
`@AwaitsFix` annotation, but now the tests are unmuted again there's no
need for the overrides.
Relates #108336
This removes the possibility for a plugin to provide factory retention settings. Factory retention settings have been deprecated and completely replaced by #111972.
Note: this feature is not in use. If someone wants to set global retention they can use the cluster settings as defined in #111972.
