* Add logs and headers
For pipeline creation when name is invalid
* Fix YAML tests and add YAML test for warnings
* Update docs/changelog/114837.yaml
* Changelog entry
* Changelog entry
* Update docs/changelog/114837.yaml
* Changelog entry
* Replace cloud-ess docker image with wolfi-ess
We just replaced the existing implementation of cloud-ess with what was wolfi-ess which is a wolfi based ess image.
The cloud image itself will be removed in a future commit it was not used anywhere
* Switch to test cloud docker image instead of default docker in packaging pr tests.
This adds way more coverage than the default docker image which is also barely touched
This adds support for a new `index_template_substitutions` field to the
body of an ingest simulate API request. These substitutions can be used
to change the pipeline(s) used for ingest, or to change the mappings
used for validation. It is similar to the
`component_template_substitutions` added in #113276. Here is an example
that shows both of those usages working together:
```
## First, add a couple of pipelines that set a field to a boolean:
PUT /_ingest/pipeline/foo-pipeline?pretty
{
"processors": [
{
"set": {
"field": "foo",
"value": true
}
}
]
}
PUT /_ingest/pipeline/bar-pipeline?pretty
{
"processors": [
{
"set": {
"field": "bar",
"value": true
}
}
]
}
## Now, create three component templates. One provides a mapping enforces that the only field is "foo"
## and that field is a keyword. The next is similar, but adds a `bar` field. The final one provides a setting
## that makes "foo-pipeline" the default pipeline.
## Remember that the "foo-pipeline" sets the "foo" field to a boolean, so using both of these templates
## together would cause a validation exception. These could be in the same template, but are provided
## separately just so that later we can show how multiple templates can be overridden.
PUT _component_template/mappings_template
{
"template": {
"mappings": {
"dynamic": "strict",
"properties": {
"foo": {
"type": "keyword"
}
}
}
}
}
PUT _component_template/mappings_template_with_bar
{
"template": {
"mappings": {
"dynamic": "strict",
"properties": {
"foo": {
"type": "keyword"
},
"bar": {
"type": "boolean"
}
}
}
}
}
PUT _component_template/settings_template
{
"template": {
"settings": {
"index": {
"default_pipeline": "foo-pipeline"
}
}
}
}
## Here we create an index template pulling in both of the component templates above
PUT _index_template/template_1
{
"index_patterns": ["foo*"],
"composed_of": ["mappings_template", "settings_template"]
}
## We can index a document here to create the index, or not. Either way the simulate call ought to work the same
POST foo-1/_doc
{
"foo": "FOO"
}
## This will not blow up with validation exceptions because the substitute "index_template_substitutions"
## uses `mappings_template_with_bar`, which adds the bar field.
## And the bar-pipeline is executed rather than the foo-pipeline because the substitute
## "index_template_substitutions" uses a substitute `settings_template`, so the value of "foo"
## does not get set to an invalid type.
POST _ingest/_simulate?pretty&index=foo-1
{
"docs": [
{
"_id": "asdf",
"_source": {
"foo": "foo",
"bar": "bar"
}
}
],
"component_template_substitutions": {
"settings_template": {
"template": {
"settings": {
"index": {
"default_pipeline": "bar-pipeline"
}
}
}
}
},
"index_template_substitutions": {
"template_1": {
"index_patterns": ["foo*"],
"composed_of": ["mappings_template_with_bar", "settings_template"]
}
}
}
```
Including the cluster state in responses to the `POST _cluster/state`
API was deprecated in #90399 (v8.6.0) requiring callers to pass
`?metric=none` to avoid the deprecation warning. This commit adjusts the
behaviour as promised in v9 so that this API never returns the cluster
state, and deprecates the `?metric` parameter itself.
Closes#88978
Currently we have a relatively basic decider about when to throttling
indexing. This commit adds two levels of watermarks with configurable
bulk size deciders. Additionally, adds additional settings to control
primary, coordinating, and replica rejection limits.
In order to better track the work needed to prepare this branch for the
major version upgrade, this commit adds a mandatory `owner` field to all
`UpdateForV9` (and `UpdateForV10`) annotations.
This test was failing due to a race between an early cancellation check
and the cancel operation. With this commit we wait until the action is
definitely blocked before cancelling the task.
Closes#100062
Introduces `MockLog#awaitAllExpectationsMatched` to allow tests to wait
until all the expected log messages have been seen without having to use
`assertBusy()`.
* Add more missing wolfi references to fix tests
* packaging tests require access to docker registry
* Fix symlink for es distributions jdk cacerts in wolfi docker
* Fix native support on wolfi images
* Fix provided keystore packaging tests for wolfi
* Add utils used for testing to wolfi image
* Explicitly set default shell to bash in docker images
* Fix docker config issues
* Apply review feedback around docker login
---------
Co-authored-by: Rene Groeschke <rene@elastic.co>
This commit flips the incremental bulk setting to false. Additionally,
it removes some test code which intermittently causes issues with
security test cases.
Currently, the raw path is only available from the RestRequest. This
makes the logic to determine if a handler supports streaming more
challenging to evaluate. This commit moves the raw path into pre request
to allow easier streaming support logic.
Almost every implementation of `AckedRequest` is an
`AcknowledgedRequest` too, and the distinction is rather confusing.
Moreover the other implementations of `AckedRequest` are a potential
source of `null` timeouts that we'd like to get rid of. This commit
simplifies the situation by dropping the unnecessary `AckedRequest`
interface entirely.
The only usage of `MappedFieldType#extractTerm` comes from `SpanTermQueryBuilder`
which attempts to extract a single term from a generic Query obtained from calling
`MappedFieldType#termQuery`. We can move this logic directly within its only caller,
and instead of using instanceof checks, we can rely on the query visitor API.
This additionally allows us to remove one of the leftover usages of TermInSetQuery#getTermData
which is deprecated in Lucene
Currently the rest.incremental_bulk is read in two different places.
This means that it will be employed in two steps introducing
unpredictable behavior. This commit ensures that it is only read in a
single place.
This commit adds a module emitting a deprecation warning when a
dot-prefixed index is manually or automatically created, or when a
composable index template with an index pattern that uses a dot-prefix
is created. This pattern warns that in the future these indices will not
be allowed. In a future breaking change (10.0.0 maybe?) the deprecation
can then be changed to an exception.
These deprecations are only displayed when a non-operator user is using
the API (one that does not set the `X-elastic-product-origin` header).
This setting had been removed in the past, it was reintroudced for bw comp with 7.x with #109341. It can now be removed from main as it no longer supports indices created with 7.x
JDK 23 removes the COMPAT locale provider, leaving CLDR as the only option. This commit configures Elasticsearch
to use the CLDR provider when on JDK 23, but still use the existing COMPAT provider when on JDK 22 and below.
This causes some differences in locale behaviour; this also adapts various tests to still work whether run on COMPAT or CLDR.
Memory locking on Windows with the bundled jdk was broken by native
access refactoring. This commit fixes the linking issue, as well as adds
a packaging test to ensure memory locking is invoked on all supported
platforms.
Test LogsDB backward compatibility with a rolling upgrade and full cluster restart.
We try to start indexing logs using a `standard` index, then we switch to a `LogsDB` index.
We also improve the existing test which switches between the two index modes, `standard`
and `logs`.
This is an attempt to fix occasional test failures where asserting on a
request response fails because the cluster has not finished
initialization and cannot yet serve requests.
Closes#109660
