Ensure that a `write` block cannot be removed on a read-only compatible index in version N-2, while allowing to change a `read_only` block into a `write` block if needed as well as closing/reopening such indices.
Requires #120647 to be merged on `8.x`.
Relates ES-10320
This update postpones unpromotable refreshes for indices with an active INDEX_REFRESH_BLOCK
until the block is cleared.
This ensures refresh operations proceed only when the index is no longer blocked.
To avoid indefinite delays, the maximum wait time is governed by the bulk request timeout
whereas for explicit refreshes it relies on the fact that the block will be removed eventually.
Closes ES-10134
A new query parameter `?include_source_on_error` was added for create / index, update and bulk REST APIs to control
if to include the document source in the error response in case of parsing errors. The default value is `true`.
Making the `es.queryable_built_in_roles_enabled` feature flag enabled by default.
This feature makes the built-in roles automatically indexed in `.security` index and available
for querying via Query Role API. The consequence of this is that `.security` index is now
created eagerly (if it's not existing) on cluster formation.
In order to keep the scope of this PR small, the feature is disabled for some of the tests,
because they are either non-trivial to adjust or the gain is not worthy the effort to do it now.
The tests will be adjusted in a follow-up PR and later the flag will be removed completely.
Relates to #117581
Applies the fix in `SourceMatcher` from #120756, along with disabling
`SCALED_FLOAT` and `HALF_FLOAT` that have accuracy issues leading to
false positives.
This patch adds a property to CountedKeywordMapper to track the
synthetic_source_keep index setting. This property is then used to properly
implement synthetic source support in the counted_keyword field type, with
fallback to the ignore_source mechanism when synthetic_source_keep is set
in either the field mapping or the index settings.
Add capability to stop async query on demand
The theory:
- User initiates async search request
- User sends the stop request (POST _query/async/<ID>/stop)
- If the async is finished by that time, it's like regular async get
- If it's not finished, the sinks are closed and the request is forcefully finished
This call has the side effect that if you are iterating a number of hits calling this method, you will be increasing the
memory usage by a non trivial number which in most of cases is unwanted. Therefore this commit removes this caching
all together and add an assertion so the method is call once during the lifetime of the object.
This particular test expects circuit breaker to trigger when an
operation uses too much memory. But it isn't using too much memory in a
few builds. This bumps it to use more memory so it will fail as
expected.
Closes#120587
Instead of deleting and recreating projects after every YAML
multi-project test, we should create the projects once at the start of
the test suite and clean up the active project after every test.
* ESQL: Heap attack tests
Adds tests for requests that would fill up the heap and crash
elasticsearch but for our circuit breakers. In an ideal world we'd
stream these results back to and this wouldn't crash anything. But we
don't have that at the moment.
* Better date
* Update
* Exhaustive testParseFractionalNumber
* Refactor: encapsulate ByteSizeUnit constructor
* Refactor: store size in bytes
* Support up to 2 decimals in parsed ByteSizeValue
* Fix test for rounding up with no warnings
* ByteSizeUnit transport changes
* Update docs/changelog/120142.yaml
* Changelog details and impact
* Fix change log breaking.area
* Address PR comments
This PR changes the wait strategy from the default HostPortWaitStrategy
to LogMessageWaitStrategy to accommodate the change in the latest Minio
docker image.
The default HostPortWaitStrategy has two issues: 1. It assumes certain
Linux commands such as grep and nc available inside the container.
This is not the case for the latest version of Minio docker image
which does not have either of the commands. 2. The first item on their
own is not fatal since the check also falls back on just reading the
listening port as a file with `/bin/bash -c
'</dev/tcp/localhost/9000'`. However, the command stirng is build
using the system's current locale which may not be Engilsh and can
have different symbols for print integers like 9000. This completely
breaks the command and in turn leads to total failure of the wait
check.
There is no easy fix for the above issues nor do I think it is necessary
to fix. So the PR simply switches the wait strategy to be log message
based.
Resolves: #120101Resolves: #120115Resolves: #120117Resolves: #118548
This PR introduces support for reconstructing inference metadata fields that are removed from `_source` by `SourceFieldMapper#applyFilters` during operations recovery.
The inference metadata fields are retrieved using value fetchers and are re-added to `_source` under the `_inference_fields` metadata field.
Here we move the `index.mapping.source.mode` setting to `IndexSettings` because of dependencies
and because of the initialisation order of static fields for classes `IndexSettings` and `SourceFieldMapper`.
Not initialising settings `index.mode`, `index.mapping.source.mode`, and `index.recovery.use_synthetic_source`
in the right order results in multiple `NullPointerException`.
This work is done to simplify another PR #119110
This PR upgrades the minio docker image from
RELEASE.2021-03-01T04-20-55Z which is 3+ years old to the latest
RELEASE.2024-12-18T13-15-44Z.
Relates: #118548
`fold` can be surprisingly heavy! The maximally efficient/paranoid thing
would be to fold each expression one time, in the constant folding rule,
and then store the result as a `Literal`. But this PR doesn't do that
because it's a big change. Instead, it creates the infrastructure for
tracking memory usage for folding as plugs it into as many places as
possible. That's not perfect, but it's better.
This infrastructure limit the allocations of fold similar to the
`CircuitBreaker` infrastructure we use for values, but it's different
in a critical way: you don't manually free any of the values. This is
important because the plan itself isn't `Releasable`, which is required
when using a real CircuitBreaker. We could have tried to make the plan
releasable, but that'd be a huge change.
Right now there's a single limit of 5% of heap per query. We create the
limit at the start of query planning and use it throughout planning.
There are about 40 places that don't yet use it. We should get them
plugged in as quick as we can manage. After that, we should look to the
maximally efficient/paranoid thing that I mentioned about waiting for
constant folding. That's an even bigger change, one I'm not equipped
to make on my own.
This updates the cluster stats action to take all projects into account
when calculating the statistics. This is true even if a specific project
is requested.
The only exception is the "health status" because we only have
primitives for calculating the health of a single project. The stats
response will always give the health of the targeted project.
