It was assuming a simpler execution model. This weakens the assertion to
one that should always pass.
Closes#127046Closes#127039Closes#127090Closes#127080
This change refactors the known directory resolution such as modules,
plugins, lib, etc. into a PathLookup. This is one of the steps towards
allowing unit tests to provide their own PathLookup for resolution so
we can enable entitlements there.
ES-11584
Although scrolling is not recommended for knn queries, it is effective.
But I found a bug that when use scroll in the knn query, the But I found
a bug that when using scroll in knn query, knn_score_doc will be lost in
query phase, which means knn query does not work. In addition, the
operations for directly querying the node where the shard is located and
querying the node with transport are different. It can be reproduced on
the local node. Because the query phase uses the previous
ShardSearchRequest object stored before the dfs phase. But when it run
in the local node, it don't do the encode and decode processso the
operation is correct. I wrote an IT to reproduce it and fixed it by
adding the new source to the LegacyReaderContext.
In light of data from recent escalations and the introduction of batched execution we can make two improvements to this logic.
For one, we should prefix with a fixed length length field so that we don't need to do any copying when serializing to account for the vint. This outright halves the memory bandwidth required relative to the previous implementation.
More importantly maybe, we should compress these bytes. The wire-format for aggregations is rather inefficient when working with nested bucket aggregations since the type strings are repeated over and over. These don't contribute to the peak heap requirements because they are translated into Java types, but blow up the message size considerably (among other things). Practically, it seems that we often get compression ratios of ~10x for aggregations.
Given that we generally have more memory issues than CPU issues during the reduce-step it seems like an easy tradeoff to trade a little CPU for compression for serious heap savings here.
While this change appears subtle at this point, I am using this in a later PR that adds a lot more spatial functions, where nesting them in related groups like this looks much better.
The main impact of this is that the On this page navigator on the right panel of the docs will show the nesting
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
We can clean the failure before we potentially go async for a partial merge here to save
some heap and needless reference chains to the per-shard result.
When writing the doc values addresses, we currently perform an iteration
over all the sorted numeric doc values to calculate the addresses. When
merging sorted segments, this iteration is expensive as it requires
performing a merge sort.
This patch removes this iteration by instead calculating the addresses
while we are writing the values, writing the addresses to a temporary
file. Afterwards, they are copied from the temporary file into the
merged segment.
Relates to #126111
Reduce the amount of text returned when no shard is requested and all
shards are assigned. Less text is easier to digest and hopefully will
not be skimmed.
As a followon to #126729, the add string keystore command doesn't need
to use a reader at all (and it was incorrect for it to close the reader
from the terminal). Instead, the Terminal abstraction already handles
how to get at line by line secrets. This commit removes that usage of
reader and uses readSecret calls instead.
closes#126882
Adds a node feature that is conditionally added to the cluster state if the failure store
feature flag is enabled. Requires all nodes in the cluster to have the node feature
present in order to redirect failed documents to the failure store from the ingest node
or from shard level bulk failures.
Adds a simple script to run benchmarks for ESQL and collect their
results. The script has a `--test` mode which takes about ten minutes.
Running without `--test` takes a four hours fifteen minutes.
To speed up `--test` I reworked the "self test" that each benchmark runs
to be optional and disabled in `--test` mode.
No need to have these marker interfaces around when weäre not using them anywhere, all they do is hide a lot of code duplication actually. Removing them sets up the possible removal of hundreds of lines of downstream code it seems
With this PR we restrict the paths we allow access to, forbidding plugins to specify/request entitlements for reading or writing to specific protected directories.
I added this validation to EntitlementInitialization, as I wanted to fail fast and this is the earliest occurrence where we have all we need: PathLookup to resolve relative paths, policies (for plugins, server, agents) and the Paths for the specific directories we want to protect.
Relates to ES-10918
We accidentally released the `9_051_0_00` and `9_052_0_00` transport
versions to serverless without their backing code. We can't take that
back. It's live. You can't unbake the cake.
That version of Elasticsearch will claim to be `9_052_0_00` but actually
speak `9_050_0_00` code. To make main compatible with that version we're
bumping all transport versions in main *past* those numbers.
This is a forbidden action. You may not do this. I may not do this. It
is not safe. But it is the only way to make the world whole again.
In all seriousness, everything will be fine. It's against the rules but
sometimes you have to break rules to put things right. We believe some
serverless dev environments may break which is sad but it's worth it to
bring everything back to sane land.
This PR introduces a time-series aggregation function that collects the
last value of each time series within each grouping.
For example:
TS index
| STATS sum(last_over_time(memory_usage)) BY cluster, bucket(@timestamp, 1minute)
With the change to using the logs dir as the working dir of the
Elasticsearch process we need to ensure the logs dir exists within the
CLI instead of later during startup.
relates #124966
* Revert "Release buffers in netty test (#126744)"
This reverts commit f9f3defe92.
* Revert "Add flow-control and remove auto-read in netty4 HTTP pipeline (#126441)"
This reverts commit c8805b85d2.
The docs about the queue in a `fixed` pool are a little awkwardly
worded, and there is no mention of the queue in a `scaling` pool at all.
This commit cleans this area up.
Missed a spot here when moving this to delayed deserialization, we can leak pending batch results here on exceptions.
closes#126994closes#126995closes#126975closes#126999closes#127001closes#126974closes#127008
* updating documentation to remove duplicate and redundant wording from 9.x
* Update links to rerank model landing page
---------
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
We addressed the empty top docs issue with #126385 specifically for scenarios where
empty top docs don't go through the wire. Yet they may be serialized from data node
back to the coord node, in which case they will no longer be equal to Lucene#EMPTY_TOP_DOCS.
This commit expands the existing filtering of empty top docs to include also those that
did go through serialization.
Closes#126742
These tests had the potential to fail when subsequent requests would hit
different nodes with different versions of the cluster state.
Only one of these tests failed already, but we fix the other ones
proactively to avoid future failures.
Fixes#126746
Follow-up to #126138.
We can now release requst bytes directly after deserialization.
Also, request instances need not go through a ref-counting cycle when forking,
removing some contention from transport threads.
There is no actual need to reference-count InboundMessage instances. Their lifecycle is completely linear and we can simplify it away. This saves a little work directly but more importantly, it enables more eager releasing of the underlying buffers in a follow-up.
---------
Co-authored-by: David Turner <david.turner@elastic.co>
* Default new semantic_text fields to use BBQ when models are compatible
* Update docs/changelog/126629.yaml
* Gate default BBQ by IndexVersion
* Cleanup from PR feedback
* PR feedback
* Fix test
* Fix test
* PR feedback
* Update test to test correct options
* Hack alert: Fix issue where mapper service was always being created with current index version
Build jump table (disi) while iterating over SortedNumericDocValues for encoding the values, instead of separately iterating over SortedNumericDocValues just to build the jump table.
In case when indexing sorting is active, this requires an additional merge sort. Follow up from #125403
The compatibleWithOptimizedMerge() method doesn't the case when doc value fields don't exist in all segments. It is like that for smaller segments not all fields exist. The change addresses that by skipping such doc value producers instead of returning that a field can't be merged using the optimized method.
There are existing metrics for the active number of threads, but it seems tricky to go from those to a "utilisation" number because all the pools have different sizes.
This commit adds `es.thread_pool.{name}.threads.utilization.current` which will be published by all `TaskExecutionTimeTrackingEsThreadPoolExecutor` thread pools (where `EsExecutors.TaskTrackingConfig#trackExecutionTime` is true).
The metric is a double gauge indicating what fraction (in [0.0, 1.0]) of the maximum possible execution time was utilised over the polling interval.
It's calculated as actualTaskExecutionTime / maximumTaskExecutionTime, so effectively a "mean" value. The metric interval is 60s so brief spikes won't be apparent in the measure, but the initial goal is to use it to detect hot-spotting so the 60s average will probably suffice.
Relates ES-10530
* Temporarily bypass competitive iteration for filters aggregation (#126956)
* Bump versions after 9.0.0 release
* fix merge conflict
* Remove 8.16 from branches.json
* Bring version-bump related changes from main
* [bwc] Add bugfix3 project (#126880)
* Sync version bump changes from main again
---------
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: elasticsearchmachine <58790826+elasticsearchmachine@users.noreply.github.com>
Co-authored-by: Brian Seeders <brian.seeders@elastic.co>
We should use DelayableWriteable here as well just like we do with per-shard results.
The heap savings of making use of this tool are quite significant at times and without
using it we could actually regress in terms of heap use relative to non-batched execution
in corner cases of a low but larger than one number of shards per node.
* Updating text_similarity_reranker documentation
* Updating docs to include urls
* remove extra THE from the text
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Reindex task may not have completed by time status is called, so remove assertion that relies on this. Following assertions are set in the action which starts task, so will be correct whether or not task has completed.
CollectionUtils.uniquify is based on C++ std::unique. However, C++
iterators are not quite the same as Java iterators. In particular,
advancing them only allows grabbing the value once. This commit reworks
uniquify to be based on list indices instead of iterators.
closes#126883
This adds `documents_found` and `values_loaded` to the to the ESQL response:
```json
{
"took" : 194,
"is_partial" : false,
"documents_found" : 100000,
"values_loaded" : 200000,
"columns" : [
{ "name" : "a", "type" : "long" },
{ "name" : "b", "type" : "long" }
],
"values" : [[10, 1]]
}
```
These are cheap enough to collect that we can do it for every query and
return it with every response. It's small, but it still gives you a
reasonable sense of how much work Elasticsearch had to go through to
perform the query.
I've also added these two fields to the driver profile and task status:
```json
"drivers" : [
{
"description" : "data",
"cluster_name" : "runTask",
"node_name" : "runTask-0",
"start_millis" : 1742923173077,
"stop_millis" : 1742923173087,
"took_nanos" : 9557014,
"cpu_nanos" : 9091340,
"documents_found" : 5, <---- THESE
"values_loaded" : 15, <---- THESE
"iterations" : 6,
...
```
These are at a high level and should be easy to reason about. We'd like to
extract this into a "show me how difficult this running query is" API one
day. But today, just plumbing it into the debugging output is good.
Any `Operator` can claim to "find documents" or "load values" by overriding
a method on its `Operator.Status` implementation:
```java
/**
* The number of documents found by this operator. Most operators
* don't find documents and will return {@code 0} here.
*/
default long documentsFound() {
return 0;
}
/**
* The number of values loaded by this operator. Most operators
* don't load values and will return {@code 0} here.
*/
default long valuesLoaded() {
return 0;
}
```
In this PR all of the `LuceneOperator`s declare that each `position` they
emit is a "document found" and the `ValuesSourceValuesSourceReaderOperator`
says each value it makes is a "value loaded". That's pretty pretty much
true. The `LuceneCountOperator` and `LuceneMinMaxOperator` sort of pretend
that the count/min/max that they emit is a "document" - but that's good
enough to give you a sense of what's going on. It's *like* document.
On x64, we are testing if we support vector capabilities (1 = "basic" = AVX2, 2 = "advanced" = AVX-512) in order to enable and choose a native implementation for some vector functions, using CPUID.
However, under some circumstances, this is not sufficient: the OS on which we are running also needs to support AVX/AVX2 etc; basically, it needs to acknowledge it knows about the additional register and that it is able to handle them e.g. in context switches. To do that we need to a) test if the CPU has xsave feature and b) use the xgetbv to test if the OS set it (declaring it supports AVX/AVX2/etc).
In most cases this is not needed, as all modern OSes do that, but for some virtualized situations (hypervisors, emulators, etc.) all the component along the chain must support it, and in some cases this is not a given.
This PR introduces a change to the x64 version of vec_caps to check for OS support too, and a warning on the Java side in case the CPU supports vector capabilities but those are not enabled at OS level.
Tested by passing noxsave to my linux box kernel boot options, and ensuring that the avx flags "disappear" from /proc/cpuinfo, and we fall back to the "no native vector" case.
Fixes#126809
Integration tests use the MockNode. This adds the SPI lookup
when building the MockSearchService. This will enable us to
have the online prewarming implementation avilable in
ESIntegTestCase.
A while ago we enabled using ccs_minimize_roundtrips in async search.
This makes it possible for users of async search to send a single search
request per remote cluster, and minimize the impact of network latency.
With non minimized roundtrips, we have pretty recurring cancellation checks:
as part of the execution, we detect that a task expired whenever each shard comes
back with its results.
In a scenario where the coord node does not hold data, or only remote data is
targeted by an async search, we have much less chance of detecting cancellation
if roundtrips are minimized. The local coordinator would do nothing other than
waiting for the minimized results from each remote cluster.
One scenario where we can check for cancellation is when each cluster comes
back with its full set of results. This commit adds such check, plus some testing
for async search cancellation with minimized roundtrips.
The following order of events was possible:
- An ILM policy update cleared `cachedSteps`
- ILM retrieves the step definition for an index, this populates `cachedSteps` with the outdated policy
- The updated policy is put in `lifecyclePolicyMap`
Any subsequent cache retrievals will see the old step definition.
By clearing `cachedSteps` _after_ we update `lifecyclePolicyMap`, we
ensure eventual consistency between the policy and the cache.
Fixes#118406
This lowers the number of documents used to test lookup because we have
a few failures over the last few months. These are all cases that we
expect to *pass* so fewer documents should make them even more likely to
pass.
Closes#125913Closes#125779
adds read privilege to the kibana_system role for indexes associated with the Microsoft Defender Integrations.
Changes are necessary in order to support Security Solution bi-directional response actions
In Jackson 2.15 a maximum string length of 50k characters was
introduced. We worked around that by override the length to max int on
all parsers created by xcontent. Jackson 2.17 introduced a similar limit
on field names. This commit mimics the workaround for string length by
overriding the max name length to be unlimited.
relates #58952
These entitlements should not be necessary. Server does not directly
create any outbound connections, and netty is not a dependency of server
so it should not exist within the server policy.
Updates the SQL grammar to include the selector portion of an index pattern. The
index() method has been updated to include selectors in the resulting expression.
The compatibleWithOptimizedMerge() method doesn't handle codec readers that are wrapped by our source pruning filter codec reader.
This change addresses that.
Failing to detect this means that the optimized merge will not kick in.
Some basic coverage for ES|QL security with failure store access. The
behavior w.r.t. status codes diverges quite a bit between ES|QL and
_search but not in any failure store specific way. ES|QL has the same
authorization behavior when querying multiple data streams without
failure store access in the mix.
We document support for snapshot repositories using `ftp://` URLs but it
seems this functionality has not worked for many years because of
security-manager restrictions, although nobody noticed because it was
not covered by any tests. The migration to the Entitlements framework
means that this functionality now works again, so this commit adds tests
to make sure we do not break it again in future.
to use benchmark mode single shot time.
Which makes more sense for benchmarking force merge. The sample time mode would invoke the benchmark methods many times, which in case of force merge is a noop.
AWS SDK v2 has a bug (aws/aws-sdk-java-v2#5968) where PathResolver uses locale-dependent formatting.
This PR adds a patcher to the discovery-ec2 build process to replace calls to String.format(<format>, <args>) with String.format(Locale.ROOT, <format>, <args>).
Relates to ES-11279
* permit at+jwt typ header value in jwt access tokens
* Update docs/changelog/126687.yaml
* address review comments
* [CI] Auto commit changes from spotless
* update Type Validator tests for parser ignoring case
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
This change integrates the new model registry with the `SemanticTextFieldMapper`, allowing inference IDs to be eagerly resolved at parse time.
It also preserves the existing lenient behavior: no error is thrown if the specified inference id does not exist, only a warning is logged.
We are currently having to hold in heap big list of Double objects which can take big amounts of heap. With this change
we can reduce the heap usage by 7x.
In order to support a future TransportRequest variant that accepts the
response type, TransportRequest needs to be an interface. This commit
adds AbstractTransportRequest as a concrete implementation and makes
TransportRequest a simple interface that joints together the parent
interfaces from TransportMessage.
Note that this was done entirely in Intellij using structural find and
replace.
* Revert endpoint creation validation for ELSER and E5
* Update docs/changelog/126792.yaml
* Revert start model deployment being in TransportPutInferenceModelAction
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
When reading a string value from stdin the keystore add command
currently looks directly at stdin. However, stdin may also be consumed
while reading the keystore password. This commit changes the add command
to use the reader from the termainl instead of looking at stdin
directly.
closes#98115
We have observed many OOMs due to the memory required to inject chunked inference results for semantic_text fields. This PR uses coordinating indexing pressure to account for this memory usage. When indexing pressure memory usage exceeds the threshold set by indexing_pressure.memory.limit, chunked inference result injection will be suspended to prevent OOMs.
This I found in the many-shards benchmark during some manual testing.
Creating indices slows down measurably when there's concurrent searches
going on. Interestingly enough, the bulk of the cost is coming from this
hook. This makes sense to some extend because the map can quickly grow
to a massive size as it scales as O(shards_searched_on_average *
concurrent_searches) and a CHM generally is anything but cheap to
iterate over.
We had a silly bug in quantizing vectors in bbq where we were scaling
the initial quantile optimization parameters incorrectly given the
vector component distribution.
In distributions where this has a major impact, the recall results were
abysmal and rendered the quantization technique useless.
In modern, well distributed components, this change is almost a no-op.
Catching Exception instead of AmazonClientException in copyBlob and
executeMultipart led to failures in S3RepositoryAnalysisRestIT due to
the injected exceptions getting wrapped in IOExceptions that prevented
them from being caught and handled in BlobAnalyzeAction.
Closes#126576
With the addition of copy coverage in the repository analyzer,
blob count is no longer 1:1 with blob analyzer request count: requests
that create a copy count as two blobs. This can cause
testFailsOnWriteException to sometimes fail, because this test randomly
injects a failure somewhere between the first and blobCounth request,
which may never happen if enough of the requests create copies.
This simple fix is to inject the failure within blobCount/2 requests,
which we will see even if every request generates a copy. An alternative
could be to add a knob to the request to disallow copies and use that
during this test.
Closes#126747
Recently we changed the implementation of
`testDataStreamLifecycleDownsampleRollingRestart` to use a temporary
state listener. We missed that the listener also had a timeout that was
quite shorter than the `safeGet` timeout we were configuring. In this PR
we align these two timeouts.
Fixes: #123769
The current LOOKUP JOIN docs include examples that are not tested by the ES|QL tests, unlike most other examples in the documentation. This PR fixes that, changing two examples to use existing tests, and adding a new csv-spec file for the remaining four examples. These four are not required to show results, so the tests have empty data and do not require any results. This means we are testing only the syntax (parsing and semantic analysis), which is sufficient for the docs.
* ES|QL change point docs
* Move ES|QL change_point to tech preview
* Update docs/reference/query-languages/esql/esql-commands.md
Co-authored-by: Craig Taverner <craig@amanzi.com>
* different example + add it the csv tests
* Restructure change_point docs to new structure
* Added generated test examples to change_point docs
* Fixed a few README.md text mistakes and added more details
* fix grammar
* License check
* regen parser
* Update docs/reference/query-languages/esql/_snippets/commands/layout/change_point.md
Co-authored-by: Craig Taverner <craig@amanzi.com>
---------
Co-authored-by: Craig Taverner <craig@amanzi.com>
This fixes an issue where if a Painless getter method return type
didn't match a Java getter method return type we add a cast.
Currentlythis is adding an extraneous cast.
Closes: #70682
Modifies TO_IP so it can handle leading `0`s in ipv4s. Here's how it
works now:
```
ROW ip = TO_IP("192.168.0.1") // OK!
ROW ip = TO_IP("192.168.010.1") // Fails
```
This adds
```
ROW ip = TO_IP("192.168.010.1", {"leading_zeros": "octal"})
ROW ip = TO_IP("192.168.010.1", {"leading_zeros": "decimal"})
```
We do this because there isn't a consensus on how to parse leading zeros
in ipv4s. The standard unix tools like `ping` and `ftp` interpret
leading zeros as octal. Java's built in ip parsing interprets them as
decimal. Because folks are using this for security rules we need to
support all the choices.
Closes#125460
This adds the interface for search online prewarming with a default NOOP
implementation. This also hooks the interface in the SearchService after
we fork the query phase to the search thread pool.
I suspect the test resets/closes the reference manager
between the refresh and the retrieval of the segment
generation after the refresh.
By executing segmentGenerationAfterRefresh while
holding the engine reset lock we make sure there
are no concurrent engine resets meanwhile.
In the future, we should also ensure that
IndexShard.refresh() uses withEngine.
Closes#126628
Patchers transform specific classes in some "broken" dependencies to ensure they behave correctly (fixing a bug, disabling some undesired or dangerous behaviour, updating calls to deprecated or removed method overloads).
If we upgrade one of the dependencies we patch, we have a concerns that the patchers may not work against the classes in the new version.
This PR addresses this concern by introducing a check on the SHA256 digest of the class, to ensure we are operating on the same bytes the patcher was designed for; if the digest changes that means the class has been changed (e.g. for a dependency update). If that happens, we break the build process with a specific error, so we can double check that the patchers still work against the new classes.
Extracted from #126326
Relates to ES-11279
Today we rely on registering the channel after registering the task to
be cancelled to ensure that the task is cancelled even if the channel is
closed concurrently. However the client may already have processed a
cancellable request on the channel and therefore this mechanism doesn't
work. With this change we make sure not to register another task after
draining the registrations in order to cancel them.
Closes#88201
This splits the grouping functions in two: those that can be evaluated independently through the EVAL operator (`BUCKET`) and those that don't (like those that that are evaluated through an agg operator, `CATEGORIZE`).
Closes#124608
and the other way around.
This doesn't make much sense. However, if a data stream's index mode differs from the index mode of most recent backing index, then this can cause confusion. Typically, misconfiguration is a reason this can happen.
Related to #126637
Adds heuristics to pick an efficient partitioning strategy based on the
index and rewritten query. This speeds up some queries by throwing more
cores at the problem:
```
FROM test | STATS SUM(b)
Before: took: 31 CPU: 222.3%
After: took: 15 CPU: 806.9%
```
It also lowers the overhead of simpler queries by throwing less cores at
the problem when it won't really speed anything up:
```
FROM test
Before: took: 1 CPU: 48.5%
After: took: 1 CPU: 70.4%
```
We have had a `pragma` to control our data partitioning for a long time,
this just looks at the query to pick a partitioning scheme. The
partitioning options:
* `shard`: use one core per shard
* `segment`: use one core per large segment
* `doc`: break each shard into as many segments as there are cores
`doc` is the fastest, but has a lot of overhead, especially for complex
Lucene queries. `segment` is fast, but doesn't make the most out of CPUs
when there are few segments. `shard` has the lowest overhead.
Previously we always used `segment` partitioning because it doesn't have
the terrible overhead but is fast. With this change we use `doc` when
the top level query matches all documents - those have very very low
overhead even in the `doc` partitioning. That's the twice as fast
example above.
This also uses the `shard` partitioning for queries that don't have to
do much work like `FROM foo` or `FROM foo | LIMIT 1` or
`FROM foo | SORT a`. That's the lower CPU example above.
This forking choice is taken very late on the data node. So queries like
this:
```
FROM test | WHERE @timestamp > "2025-01-01T00:00:00Z" | STATS SUM(b)
```
can also use the `doc` partitioning when all documents are after the
timestamp and all documents have `b`.
This PR fixes#119950 where an `IN` query includes `NULL` values with non-NULL `DataType` appearing within the query range. An expression is considered `NULL` when its `DataType` is `NULL` or it is a `Literal` with a value of `null`.
If updating the `index.time_series.end_time` fails for one data stream,
then UpdateTimeSeriesRangeService should continue updating this setting for other data streams.
The following error was observed in the wild:
```
[2025-04-07T08:50:39,698][WARN ][o.e.d.UpdateTimeSeriesRangeService] [node-01] failed to update tsdb data stream end times
java.lang.IllegalArgumentException: [index.time_series.end_time] requires [index.mode=time_series]
at org.elasticsearch.index.IndexSettings$1.validate(IndexSettings.java:636) ~[elasticsearch-8.17.3.jar:?]
at org.elasticsearch.index.IndexSettings$1.validate(IndexSettings.java:619) ~[elasticsearch-8.17.3.jar:?]
at org.elasticsearch.common.settings.Setting.get(Setting.java:563) ~[elasticsearch-8.17.3.jar:?]
at org.elasticsearch.common.settings.Setting.get(Setting.java:535) ~[elasticsearch-8.17.3.jar:?]
at org.elasticsearch.datastreams.UpdateTimeSeriesRangeService.updateTimeSeriesTemporalRange(UpdateTimeSeriesRangeService.java:111) ~[?:?]
at org.elasticsearch.datastreams.UpdateTimeSeriesRangeService$UpdateTimeSeriesExecutor.execute(UpdateTimeSeriesRangeService.java:210) ~[?:?]
at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1075) ~[elasticsearch-8.17.3.jar:?]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1038) ~[elasticsearch-8.17.3.jar:?]
at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245) ~[elasticsearch-8.17.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1691) ~[elasticsearch-8.17.3.jar:?]
at org.elasticsearch.action.ActionListener.run(ActionListener.java:452) ~[elasticsearch-8.17.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1688) ~[elasticsearch-8.17.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1283) ~[elasticsearch-8.17.3.jar:?]
at org.elasticsearch.action.ActionListener.run(ActionListener.java:452) ~[elasticsearch-8.17.3.jar:?]
at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1262) ~[elasticsearch-8.17.3.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023) ~[elasticsearch-8.17.3.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.17.3.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1575) ~[?:?]
```
Which resulted in a situation, that causes the `index.time_series.end_time` index setting not being updated for any data stream. This then caused data loss as metrics couldn't be indexed, because no suitable backing index could be resolved:
```
the document timestamp [2025-03-26T15:26:10.000Z] is outside of ranges of currently writable indices [[2025-01-31T07:22:43.000Z,2025-02-15T07:24:06.000Z][2025-02-15T07:24:06.000Z,2025-03-02T07:34:07.000Z][2025-03-02T07:34:07.000Z,2025-03-10T12:45:37.000Z][2025-03-10T12:45:37.000Z,2025-03-10T14:30:37.000Z][2025-03-10T14:30:37.000Z,2025-03-25T12:50:40.000Z][2025-03-25T12:50:40.000Z,2025-03-25T14:35:40.000Z
```
This removes all non-test usage of
Metadata.Builder.put(IndexMetadata.Builder)
And replaces it with appropriate calls to the equivalent method on
`ProjectMetadata.Builder`
In most cases this _does not_ make the code project aware, but does
reduce the number of deprecated methods in use
Fixes an edge case where we randomly generate an empty list of key
usages then pass it to `certutil` command.
The empty means just apply the defaults, but we assert based on the
generated empty set.
Resolves https://github.com/elastic/elasticsearch/issues/126471
Co-authored-by: Tim Vernum <tim@adjective.org>
This change adds the `avg_over_time` aggregation for time series
indices. Similar to other time series aggregations, we need to translate
`avg_over_time` into regular aggregations. There are two options for
this translation:
1. Translate `avg_over_time` to `EVAL div(sum_over_time, count_over_time)`, then translate `sum_over_time` and `count_over_time` to `sum` and `count`.
2. Translate `avg_over_time` directly to `avg`, and then to `div(sum, count)`.
This PR chooses the latter approach. Below is an example:
```
TS k8s
| STATS sum(avg_over_time(memory_usage)) BY host, bucket(@timestamp, 1minute)
```
translates to:
```
TS k8s
| STATS avg_memory_usage = avg(memory_usage), host_values=VALUES(host) BY _tsid, time_bucket=bucket(@timestamp, 1minute)
| STATS sum(avg_memory_usage) BY host_values, time_bucket
```
and then:
```
TS k8s
| STATS sum_memory_usage = sum(memory_usage), count_memory_usage = count(memory_usage), host_values=VALUES(host) BY _tsid, time_bucket=bucket(@timestamp, 1minute)
| EVAL avg_memory_usage = sum_memory_usage / count_memory_usage
| STATS sum(avg_memory_usage) BY host_values, time_bucket
```
Since we need to substitute `AVG` with `SUM` and `COUNT` after
translation, we need to call `SubstituteSurrogates` twice in
`LogicalPlanOptimizer`. If there is a performance impact, we can move
this rule to `TranslateTimeSeriesAggregate`.
I noticed that we tend to create the flag instance and call this method
everywhere. This doesn't compile the same way as a real boolean constant
unless you're running with `-XX:+TrustFinalNonStaticFields`.
For most of the code spots changed here that's irrelevant but at least
the usage in the mapper parsing code is a little hot and gets a small
speedup from this potentially.
Also we're simply wasting some bytes for the static footprint of ES by
using the `FeatureFlag` indirection instead of just a boolean.
- Translate a 404 during a multipart copy into a `FileNotFoundException`
- Use multiple threads in `S3HttpHandler` to avoid `CopyObject`/`PutObject` deadlock
Closes#126576
In this PR we add support for the failure store for system data streams.
Specifically:
- We pass the system descriptor so the failure index can be created based on that.
- We extend the tests to ensure it works
- We remove a guard we had but I wasn't able to test it because it only gets triggered if the data stream gets created right after a failure in the ingest pipeline, and I didn't see how to add one (yet).
- We extend the system data stream migration to ensure this is also working.
This addresses feedback we got for our default image at https://github.com/docker-library/official-images/pull/18692
This also introduces separate docker source files to make maintaining those easier.
We cannot take over all suggested changes as we require certain settings to have our packaging tests pass as expected.
Currently, each plan node iterration in ProjectAwayColumns creates 3
AttributeSet/Map_s. This can be dropped to just one by using builders.
Related: #124395
Build a little histogram of the `took` times of ESQL queries in a
cluster. Graphing this can tell you how much ESQL is in use. Especially
for heavy or light things.
The `indexNameSupplier` was included in the equality and is of type
`BiFunction`, which doesn't implement a proper `equals` method by
default - and thus neither do the lambdas. This meant that two instances
of this step would only be considered equal if they were the same
instance. By excluding `indexNameSupplier` from the `equals` method, we
ensure the method works as intended and is able to properly tell the
equality between two instances.
As a side effect, we expect/hope this change will fix a number of tests
that were failing because `WaitForIndexColorStep` missed the last
cluster state update in the test, causing ILM to get stuck and the test
to time out.
Fixes#125683Fixes#125789Fixes#125867Fixes#125911Fixes#126053Fixes#126354
While the internal structure of the docs is already split into many (over 1000) sub-pages, the final display for the `Functions and Operators` page is a single giant page, making navigation harder. This PR splits it into separate pages, one for each group of similar functions and one for the operators. Twelve new pages.
This PR also bundles a few other related changes. In total what is done is:
* Split functions/operators into 12 pages, one for each group, maintaining the existing split of each function/operator into a snippet with dynamically generated examples
* Split esql-commands.md into source-commands.md and processing-commands.md, each of which is split into individual snippets, one for each command
* Each command snippet has it's examples split out into separate files, if they were examples that were dynamically generated in the older asciidoc system
* The examples files are overwritten by the ES|QL unit tests, using a similar mechanism to the examples written for functions and operators)
* Some additional refinements to the Kibana definition and markdown files (nicer operator headings, and display text)
This change re-introduces the engine read/write lock to guard against engine resets.
It differs from #124635 on the following:
uses the engineMutex for creating/closing engines
uses the reentrant r/w lock for retaining engine instances and for resetting the engine
acquires the reentrant read lock during refreshes to prevent deadlocks during resets
add tests to ensure no deadlock when re-acquiring read lock in refresh listeners
Relates ES-11447
`Lucene.EMPTY_TOP_DOCS` to identify empty to docs results. These were previously
null results, but did not need to be send over transport as incremental reduction
was performed only on the data node.
Now it can happen that the coord node received a merge result with empty top docs,
which has nothing interesting for merging, but that can lead to an exception because
the type of the empty array does not match the type of other shards results, for
instance if the query was sorted by field. To resolve this, we filter out empty
top docs results before merging.
Closes#126118
This fixes a bug in the tests for parsing ipv4 addresses with leading
zeros as octal numbers. The bug was in the expectation code that we use
for random input ips.
Closes#126496
This change adds support for the `max_over_time` aggregation for
time_series indices. Similar to the `rate` aggregation, this aggregation
is translated into two stages: the first stage groups by `_tsid` (and
time-bucket), and the second stage groups by the user-specified groups.
For example:
```
TS my-metrics
| STATS SUM(max_over_time(memory_usage)) BY cluster, bucket(@timestamp, 1 minute)
```
is translated into:
```
TS my-metrics
| STATS max_memory_usage=max(memory_usage), cluster=VALUES(cluster) BY _tsid, ts=bucket(@timestamp, 1 minute)
| STATS sum(max_memory_usage) BY cluster, ts
```
In this case, we don't need to keep the Lucene source emitted in the
order of _tsid/timestamp, but I leave this optimization for the future.
Other `{agg}_over_time` functions will be added later.
In mustache, this change returns null values which convert to empty strings
instead of throwing an exception when users have a template with
something like a.8 where the index 8 is out of bounds. This matches the
behavior for non-existent keys like a.d.
Closes#55200
Prior to these changes, the split method would fail tests. Additionally,
the method had code which could be refactored.
A new variable (numRanges) was introduced to replace the direct usage of numStreams.
The method was refactored to make the code easier to understand. Javadocs were updated.
Tests for this method now pass.
Add support for Cohere Task Settings and Truncate, through
the Amazon Bedrock provider integration.
Task Settings can now be passed bother during Inference endpoint
creation and Inference POST requests.
Close#126156
Checks to see if an index belongs to a data stream's failure store before following it. If the index is a
failure index, the follow operation is rejected. Also updates the logic in the auto follower API's to
exclude failure indices on data streams from being followed if their parent data stream matches the
follow pattern.
* BlobContainer: add copyBlob method
If a container implements copyBlob, then the copy is
performed by the store, without client-side IO. If the store
does not provide a copy operation then the default implementation
throws UnsupportedOperationException.
This change provides implementations for the FS and S3 blob containers.
More will follow.
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: David Turner <david.turner@elastic.co>
* Use a task queue to ensure ILM policy change cluster state updates are batched
* Update docs/changelog/126529.yaml
* Update docs/changelog/126529.yaml
* Switch to using SimpleBatchedAckListenerTaskExecutor
* Get timeout from request
* Ditch the try-catch
Currently, the rate aggregation accepts two parameters: the first
specifies the counter field, and the second specifies the time unit of
the rate. The time unit parameter was introduced to conveniently compute
requests per minute or per hour. However, this can be replaced easily -
for example, rate(field, 1minute) with 60 * rate(field).
This change removes the time unit parameter and reserves it for
potential future usage, such as introducing a sliding window unit. If we
decide on other options later, we can reintroduce it. Removing it now
avoids breaking changes while the rate aggregation is not yet available.
This commit cleans up how entitlement test methods are discovered. It
also adds another robustness check to ensure an annotation doesn't exist
on a private method.
In the unexpected case that Elasticsearch dies due to a segfault or
other similar native issue, a core dump is useful in diagnosing the
problem. Yet core dumps are written to the working directory, which is
read-only for most installations of Elasticsearch. This commit changes
the working directory to the logs dir which should always be writeable.
Before we enabled sort optimization on long, double and date types,
but left other types for follow-up.
This enables sort optimization on float and half_float types.
Optimizations on INT, BYTE, SHORT are left for follow-up,
because they need more work: we currently use SORT.LONG
type for all integer types and this doesn't allow to use
optimization.
Today when we are adding a ElasticsearchException, we specify a versionAdded TransportVersion (the transport version from which we support it); this version is checked by the isRegistered method:
return version.onOrAfter(elasticsearchExceptionHandle.versionAdded);
This does not play well with backports; when we add a patch version for a backport, normally the procedure would be to change the code above take also the patch into account, like:
version.orOnAfter(versionAdded) || version.isPatchFrom(versionPatched)
This PR updates ElasticsearchException to have more than just "version added", so that we can do patches as described above.
The doc values codec iterates a few times over the doc value instance that needs to be written to disk. In case when merging and index sorting is enabled, this is much more expensive, as each time the doc values instance is iterated a merge sorting is performed (in order to get the doc ids of new segment in order of index sorting).
There are several reasons why the doc value instance is iterated multiple times:
* To compute stats (num values, number of docs with value) required for writing values to disk.
* To write bitset that indicate which documents have a value. (indexed disi, jump table)
* To write the actual values to disk.
* To write the addresses to disk (in case docs have multiple values)
This applies for numeric doc values, but also for the ordinals of sorted (set) doc values.
This PR addresses solving the first reason why doc value instance needs to be iterated. This is done only when in case of merging and when the segments to be merged with are also of type es87 doc values, codec version is the same and there are no deletes. Note this optimized merged is behind a feature flag for now.
This PR adds two new REST endpoints, for listing queries and getting information on a current query.
* Resolves#124827
* Related to #124828 (initial work)
Changes from the API specified in the above issues:
* The get API is pretty initial, as we don't have a way of fetching the memory used or number of rows processed.
List queries response:
```
GET /_query/queries
// returns for each of the running queries
// query_id, start_time, running_time, query
{ "queries" : {
"abc": {
"id": "abc",
"start_time_millis": 14585858875292,
"running_time_nanos": 762794,
"query": "FROM logs* | STATS BY hostname"
},
"4321": {
"id":"4321",
"start_time_millis": 14585858823573,
"running_time_nanos": 90231,
"query": "FROM orders | LOOKUP country_code ON country"
}
}
}
```
Get query response:
```
GET /_query/queries/abc
{
"id" : "abc",
"start_time_millis": 14585858875292,
"running_time_nanos": 762794,
"query": "FROM logs* | STATS BY hostname"
"coordinating_node": "oTUltX4IQMOUUVeiohTt8A"
"data_nodes" : [ "DwrYwfytxthse49X4", "i5msnbUyWlpe86e7"]
}
```
* Refactor remote cluster handling in Analyzer
- Initialize clusters earlier
- Simplify cluster set calculation
- No need to keep separate skipped list for enrich resolution
This commit adds support for system data streams reindexing. The system data stream migration extends the existing system indices migration task and uses the data stream reindex API.
The system index migration task starts a reindex data stream task and tracks its status every second. Only one system index or system data stream is migrated at a time. If a data stream migration fails, the entire system index migration task will also fail.
Port of #123926
The mostly-optional parameters to `createBlobContainer` are getting
rather numerous in this test harness which makes the tests hard to read.
This commit introduces a builder to help name the provided parameters
and skip the omitted ones.
In #88707 we changed the behaviour here to run the shard-snapshot
initialization tasks all in sequence. Yet these tasks do nontrivial work
since they may flush to acquire the relevant index commit, so with this
commit we go back to distributing them across the `SNAPSHOT` pool again.
* Adding validation to ElasticsearchInternalService
* Update docs/changelog/123044.yaml
* [CI] Auto commit changes from spotless
* Removing checkModelConfig
* Fixing IT
* [CI] Auto commit changes from spotless
* Remove DeepSeek checkModelConfig and fix tests
* Cleaning up comments, updating validation input type, and moving model deployment starting to model validator
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
When creating the an empty lifecycle we used to use the default
constructor. This is not just for efficiency but it will allow us to
separate the default data and failures lifecycle in the future.
This converts this class to support multi-project.
There is a drive-by refactoring to split multiple independent tests in one method into separate methods. (Look at this change with whitespace changes hidden to see the 'real' change.)
The `elasticsearch-certutil http` command, and security auto-configuration,
generate the HTTP certificate and CA without setting the `keyUsage` extension.
This PR fixes this by setting (by default):
- `keyCertSign` and `cRLSign` for self-signed CAs
- `digitalSignature` and `keyEncipherment` for HTTP certificates and CSRs
These defaults can be overridden when running `elasticsearch-certutil http`
command. The user will be prompted to change them as they wish.
For `elasticsearch-certutil ca`, the default value can be overridden by passing
the `--keysage` option, e.g.
```
elasticsearch-certutil ca --keyusage "digitalSignature,keyCertSign,cRLSign" -pem
```
Fixes#117769
Transport actions have associated request and response classes. However,
the base type restrictions are not necessary to duplicate when creating
a map of transport actions. Relatedly, the ActionHandler class doesn't
actually need strongly typed action type and classes since they are lost
when shoved into the node client map. This commit removes these type
restrictions and generic parameters.
When no explicit aliases are provided in the call there is no need
to collect the index names or aliases into HashSets if they won't be
used. Also fixed where the index name was being added for each
loop of the alias list.
The backport to `8.x` needed some changes to pass through CI; this
commit forward-ports the relevant bits of those changes back into `main`
to keep the branches aligned.
Some `AbstractBlobContainerRetriesTestCase#createBlobContainer`
implementations choose a path for the container randomly, but we have a
need for a test which re-creates the same container against a different
`S3Service` and `BlobStore` and must therefore specify the same path
each time. This commit exposes a parameter that lets callers specify a
container path.
This base class is kinda pointless: everywhere it's used we can either
be more specific (e.g. choosing between `TransportRequest` or
`TransportResponse`) or more general (e.g. choosing `Writeable`). This
commit removes all the usages apart from the `extends` clauses of its
direct descendants.
If the clusters don't support inference test services, skip tests that require inference services.
Hence, we should check for rerank tests.
Relates #123074
Speed up the TO_IP method by converting directly from utf-8 encoded
strings to the ip encoding. Previously we did:
```
utf-8 -> String -> INetAddress -> ip encoding
```
In a step towards solving #125460 this creates three IP parsing
functions, one the rejects leading zeros, one that interprets leading
zeros as decimal numbers, and one the interprets leading zeros as octal
numbers. IPs have historically been parsed in all three of those ways.
This plugs the "rejects leading zeros" parser into `TO_IP` because
that's the behavior it had before.
Here is the performance:
```
Benchmark Score Error Units
leadingZerosAreDecimal 14.007 ± 0.093 ns/op
leadingZerosAreOctal 15.020 ± 0.373 ns/op
leadingZerosRejected 14.176 ± 3.861 ns/op
original 32.950 ± 1.062 ns/op
```
So this is roughly 45% faster than what we had.
Today `ActionResponse$Empty` implements `ToXContentObject`, but yields
no bytes of content when serialized which creates an invalid JSON
response. This commit removes the bogus interface and adjusts the
affected REST APIs to send a `text/plain` response instead.
In case of wrong layouts of ESQL's operators, it can happen that
ValueExtractor.extractorFor encounters a data type mismatch. Currently,
this throws IllegalArgumentException, which is treated like a user
exception and triggers a 400 response.
We need to return a 500 status code for such errors; this is also
important for observability of ES clusters, which can normally use 500
responses as an indicator of a bug.
Throw IllegalStateException instead, it's close enough.
With `geo_point` fields, here is the special case of values that have a syntactically valid format, but the numerical values for `latitude` and `longitude` are out of range.
If `ignore_malformed` is `false`, an exception will be thrown as usual. But if it is `true`, the document will be indexed correctly, by normalizing the latitude and longitude values into the valid range. The special `_ignored` field will not be set. The original source document will remain as before, but indexed values, doc-values and stored fields will all be normalized.
* Adaptive allocations: don't update deployments that aren't started.
* AssignmentPlanner: don't plan deployments with zero allocations
* Update JavaDoc
The `InvalidIndexNameException` exception was wrapped in a `ElasticsearchSecurityException`, which returns HTTP `403` status.
This exception (along with newly introduced `InvalidSelectorException` and `UnsupportedSelectorException`) can be raised during index expression resolving due to an invalid user input and should result in HTTP `400` response instead.
This PR changes exception handling to avoid wrapping them in the `ElasticsearchSecurityException`.
To backport the newly introduced AggregateMetricDoubleLiteral to 8.x, we need
to override the supportsVersion method instead of getMinimalSupportedVersion.
Update the PerFieldFormatSupplier so that new standard indices use the
Lucene101PostingsFormat instead of the current default ES812PostingsFormat.
Currently, use of the new codec is gated behind a feature flag.
This test had a copy paste mistake. When the cluster has only one data
node the replicas cannot be assigned so we end up with a force merge
error. In the case of the failure store this was not asserted correctly.
On the other hand, this test only checked for the existence of an error
and it was not ensuring that the current error is not the rollover error
that should have recovered. We make this test a bit more explicit.
Fixes: https://github.com/elastic/elasticsearch/issues/126252
This enhances DataStreamAutoShardingService so that it periodically
logs at INFO level the most 'interesting' results it has produced in
the last period.
In this PR, the most 'interesting' results are considered to be the
ones with the highest load, keeping track separately of the top 10
which resulting in an increase decision and the top 10 which did
not. In practice, increase recommendations are sufficiently rare that
the top 10 will often be 'all of them', and they are all potentially
interesting (but we cap it to protect against taking up an unbounded
amount of memory). We keep the high load non-increase recommendations
as well, since these are likely to be the interesting ones to look at
when investigating why some data stream did not get an increase shards
recommendation when we might have expected it.
The mechanism would be easily extended to add in other categories. For
example, we may find that there are some categories of decrease
decisions we consider 'interesting'. (N.B. The rollover service will
not roll over just because the auto-sharding service recommended
down-sharding — down-sharding only happens if the rollover was going
to happen for some other reason (age, size, etc.) So it's normal for
the auto-sharding service to return decrease recommendations for the
same data streams every 5 minutes until those other conditions are
met. Which makes things a bit more complicated.) This PR just covers
the cases that seem likely to be useful in the support cases we have
seen.
The existing DEBUG and TRACE log lines in the service are replaced
with a single DEBUG log which pulls together all the data. This is an
improvement, since at the moment it is hard to figure out from the
logs which lines refer to the same data stream (they are interleaved,
and don't all include the data stream name).
The write load field in the AutoShardingResult was unused, and is
removed.
ES-10037 #comment Improved logging in https://github.com/elastic/elasticsearch/pull/126171
* [main] Move system indices migration to migrate plugin
It seems the best way to fix#122949 is to use existing data stream reindex API. However, this API is located in the migrate x-pack plugin. This commit moves the system indices migration logic (REST handlers, transport actions, and task) to the migrate plugin.
Port of #123551
* [CI] Auto commit changes from spotless
* Fix compilation
* Fix tests
* Fix test
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
We can make slicing things like search hits where we cut many small
buffers out of a large composite reference a lot more efficient in some
cases by making the slice of a releasable reference itself releasable.
This fixes the current situation where we could have composite buffer
of many MB that is made up of e.g. 16k chunks but that we would retain
in full if we were to slice even a single byte out of it in any position.
Rather than hard-coding a region name we should always auto-generate it
randomly during test execution. This commit replaces the remaining fixed
`String` arguments with a `Supplier<String>` argument to enable this.
Rate aggregators need to access the start time and end time of each
group to perform extrapolation in the final evaluation. This change
replaces the DriverContext parameter in evaluateFinal with
EvaluationContext, allowing the TimeSeriesAggregationOperator to pass an
EvaluationContext with these time intervals. I took another approach
where the extension is applied only for the time series aggregator, but
it could be fragile with the pre-filter of aggregators.
We need to register the settings update consumer for
QUERY_PHASE_PARALLEL_COLLECTION_ENABLED regardless of the
BATCHED_QUERY_PHASE_FEATURE_FLAG feature flag.
Closes#125899
This reverts commit 81555cc9d9 from
https://github.com/elastic/elasticsearch/pull/125060.
Fix https://github.com/elastic/elasticsearch/issues/126275
@idegtiarenko and I investigated and believe this needs reverting:
silently dropping results from the query response in case any index is
missing can lead to real problems if users don't spot their mistake. I'm
also not sure if all the results will get dropped, or only from some
nodes/shards/clusters, meaning that this might be hard to spot by users
if only some results get dropped.
The main PR has no transport version bump, no new ESQL capability, and
was merged 15h ago - so it should be safe to just revert it. I noticed
there was a linked Serverless PR on the original PR, but it merely
disabled some obsolete tests on Serverless and doesn't require reverting
itself.
Add tracking of the number of compute() calls and total iterations
between convergences in the DesiredBalanceComputer, along with the
time since the last convergence. These are included in the log
message when the computer doesn't converge.
Closes#100850.
Today we have some special-case test classes in `:modules:repository-s3`
within the same source root as the regular tests, with some trickery to
define separate Gradle tasks to run them with their special-case
configs. This commit simplifies the build by just moving each of these
classes into its own Gradle project.
These tests only don't work in a FIPS JVM because they use a secret key
that is unacceptably short. This commit replaces the relevant uses of
`randomIdentifier` with `randomSecretKey` so they work whether in FIPS
mode or not.
`ListObjects` and `ListObjectsV2` only really differ in their approach
to pagination, but today `S3HttpHandler` does not simulate pagination
anyway so we can use the same handling code for both APIs. The only
practical difference is that the v2 SDK requires the `<IsTruncated>`
element in a `ListObjectsV2` response, but this element is permitted in
both APIs so we add it here.
This PR expands the README for the entitlements package; it adds information for Elasticsearch developers on how the system is structured, how to build policies, what to do when new code "does not work" because it's bumping into a NotEntitledException, and how to apply a policy patch in case of an emergency.
We need the bucket interval within the rate function to perform
extrapolation. With this change, along with #26089, we will be able to
pass the grouping range interval to rate aggregations.
* test
* Revert "test"
This reverts commit 9f4e2adba0.
* Refactor InferenceService to allow passing in chunking settings
* Add chunking config to inference field metadata and store in semantic_text field
* Fix test compilation errors
* Hacking around trying to get ingest to work
* Debugging
* [CI] Auto commit changes from spotless
* POC works and update TODO to fix this
* [CI] Auto commit changes from spotless
* Refactor chunking settings from model settings to field inference request
* A bit of cleanup
* Revert a bunch of changes to try to narrow down what broke CI
* test
* Revert "test"
This reverts commit 9f4e2adba0.
* Fix InferenceFieldMetadataTest
* [CI] Auto commit changes from spotless
* Add chunking settings back in
* Update builder to use new map
* Fix compilation errors after merge
* Debugging tests
* debugging
* Cleanup
* Add yaml test
* Update tests
* Add chunking to test inference service
* Trying to get tests to work
* Shard bulk inference test never specifies chunking settings
* Fix test
* Always process batches in order
* Fix chunking in test inference service and yaml tests
* [CI] Auto commit changes from spotless
* Refactor - remove convenience method with default chunking settings
* Fix ShardBulkInferenceActionFilterTests
* Fix ElasticsearchInternalServiceTests
* Fix SemanticTextFieldMapperTests
* [CI] Auto commit changes from spotless
* Fix test data to fit within bounds
* Add additional yaml test cases
* Playing with xcontent parsing
* A little cleanup
* Update docs/changelog/121041.yaml
* Fix failures introduced by merge
* [CI] Auto commit changes from spotless
* Address PR feedback
* [CI] Auto commit changes from spotless
* Fix predicate in updated test
* Better handling of null/empty ChunkingSettings
* Update parsing settings
* Fix errors post merge
* PR feedback
* [CI] Auto commit changes from spotless
* PR feedback and fix Xcontent parsing for SemanticTextField
* Remove chunking settings check to use what's passed in from sender service
* Fix some tests
* Cleanup
* Test failure whack-a-mole
* Cleanup
* Refactor to handle memory optimized bulk shard inference actions - this is ugly but at least it compiles
* [CI] Auto commit changes from spotless
* Minor cleanup
* A bit more cleanup
* Spotless
* Revert change
* Update chunking setting update logic
* Go back to serializing maps
* Revert change to model settings - source still errors on missing model_id
* Fix updating chunking settings
* Look up model if null
* Fix test
* Work around https://github.com/elastic/elasticsearch/issues/125723 in semantic text field serialization
* Add BWC tests
* Add chunking_settings to docs
* Refactor/rename
* Address minor PR feedback
* Add test case for null update
* PR feedback - adjust refactor of chunked inputs
* Refactored AbstractTestInferenceService to return offsets instead of just Strings
* [CI] Auto commit changes from spotless
* Fix tests where chunk output was of size 3
* Update mappings per PR feedback
* PR Feedback
* Fix problems related to merge
* PR optimization
* Fix test
* Delete extra file
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
With this change, ES|QL will return partial results instead of failing
the entire query when encountering errors. Callers should check the
partial_results flag in the response to determine if the result is
partial or complete. If returning partial results is not desired, this
option can be overridden per request via the allow_partial_results
parameter in the query URL or globally via the cluster setting
esql.allow_partial_results.
Relates #122802
We need to store extra information for Aggregate and AggregateExec for
time-series aggregations. Previously, I added a type (standard or
time_series), but this was not enough. This PR removes it and replaces
it with extensions of Aggregate and AggregateExec. I considered adding
an extra map of metadata to Aggregate and AggregateExec, but this
approach seems simpler.
We recently cleared stack traces on data nodes before transport back to the coordinating node when error_trace=false to reduce unnecessary data transfer and memory on the coordinating node (#118266). However, all logging of exceptions happens on the coordinating node, so stack traces disappeared from any logs. This change logs stack traces directly on the data node when error_trace=false.
This PR fixes 2 issues discovered around PolicyUtils (and PolicyUtilsTests) when running CI on Windows:
- in tests, absolute paths like always are different, this fix updates the tests to account for the difference.
- on Windows, Files.move was failing because we were parsing the Entitlement policy but not closing the stream in a timely manner. This causes plugin installation (and related CI tests) to fail 70% of the time. Fixed by closing the stream properly
Fixes#126176
**Issue** The data stream lifecycle does not register correctly rollover
errors for failure store.
**Observed bahaviour** When data stream lifecycle encounters a rollover
error it records it unless it sees that the current write index of this
data stream doesn't match the source index of the request. However, the
write index check does not use the failure write index but the write
backing index, so the failure gets ignored
**Desired behaviour** When data stream lifecycle encounters a rollover
error it will check the relevant write index before it determines if it
should be recorded or not.
Originally, `DATE_TRUNC` only supported 1-month and 3-month intervals for months, and 1-year interval for years, while arbitrary intervals were supported for weeks and days. This PR adds support for `DATE_TRUNC` with arbitrary month and year intervals.
Closes#120094
ServerSentEvent is now a record with `event` and `data`, rather than
it being a record for value with a separate `ServerSentEventField`.
- `value` was renamed to `data`
- `hasValue` was renamed to `hasData`
- Parsing was refactored to read more like its spec: writing to a buffer
and flushing when we reach a blank newline
- We now support multiline data payloads
`S3BlobContainerRetriesTests` uses `MD5DigestCalculatingInputStream`
from the AWS v1 SDK to compute a MD5 checksum, but this feature is not
available in the v2 SDK. With this commit we remove this dependency and
compute the MD5 checksums directly instead.
This was fixed last week in
https://github.com/elastic/elasticsearch/pull/125802. It seems this
report was from the day before the fix, but the PR was based on main
without the mute, so a timing issue. I'll make a PR that simply unmutes
the test.
Fixes#125735
Don't perform version check assertion in TsdbDocValueBwcTests if security manager is active.
By default, with jvm version 24 entitlements are used instead security manager and assertOldDocValuesFormatVersion() / assertNewDocValuesFormatVersion() work as expected.
Making these methods work with security manager would require granting the server entire test codebase suppressAccessChecks and suppressAccessChecks privileges. This is undesired from a security manager perspective. Instead, only assert doc values format checks if security manager isn't active, which is always the case jvm version 24 or higher is used.
Closes#126174
* Revert changes to Layout.java
The change in 80125a4bac is a quick fix
and allows breaking an invariant of Layout. Revert that.
* Simplify ReplaceMissingFieldWithNull
When encountering projections, it tries to do the job of field
extraction for missing fields by injecting an Eval that creates a
literal null with the same name id as the field attribute for the
missing field. This is wrong:
1. We only insert an Eval in case that a Project relies on the missing
attribute. There could be other plan nodes that rely on the missing
attribute.
2. Even for Projects, we only insert an Eval in case we squarely project
for the field - in case of aliases (e.g. from RENAME), we do nothing.
3. In case of multiple Projects that use this attribute, we create
multiple attributes with the original field attribute's id, causing
a wrong Layout. This triggered
https://github.com/elastic/elasticsearch/issues/121754.
* Revive logic for EsRelation instead of Project
* Update LocalLogicalPlanOptimizerTests
* Update docs/changelog/125764.yaml
* Update test expectations
* Do not prune attributes from EsRelation
This can lead to empty output, which leads to the EsRelation being
replaced by a LocalRelation with 0 rows.
* Add tests + capability
* Update docs/changelog/125764.yaml
* Add comments
Today the `ListObjects` implementation in `S3HttpHandler` will put all
the common prefixes in a single `<CommonPrefixes>` container, but in
fact the real S3 gives each one its own container. The v1 SDK is lenient
and accepts either, but the v2 SDK requires us to do this correctly.
This commit fixes the test fixture to match the behaviour of the real
S3.
This will allow reusing them in the plan analysis and skip recreating them in UnaryPlan/UnaryExec when not needed.
Introduce/adjust builders for them, which are now the only way to use a modifiable map/set.
Related #124395
This action solely needs the cluster state, it can run on any node.
Since this is the last class/action that extends the `ClusterInfo`
abstract classes, we remove those classes too as they're not required
anymore.
Relates #101805
Wrap remote errors with cluster name to provide more context
Previously, if a remote encountered an error, user would see a top-level error that would provide no context about which remote ran into the error. Now, such errors are wrapped in a separate remote exception whose error message clearly specifies the name of the remote cluster and the error that occurred is the cause of this remote exception.
In this PR we introduce the data stream API in the `es-rest-api` using
the feature flag feature. This enabled us to use the `yamlRestTests`
tests instead of the `javaRestTests`.
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
Relates #101805
The change contains the following changes:
- The numDocsWithField field moved from SortedNumericEntry to NumericEntry. Making this statistic always available.
- Store jump table after values in ES87TSDBDocValuesConsumer#writeField(...). Currently it is stored before storing values. This will allow us later to iterate over the SortedNumericDocValues once. When merging, this is expensive as a merge sort on the fly is being executed.
This change will allow all the optimizations that are listed in #125403
This PR replaces the parsing and formatting of SecurityManager policies with the parsing and formatting of Entitlements policy during plugin installation.
Relates to ES-10923
- Fixed bug where 416 was being erroneously returned for zero-length blobs even with no Range header
- Fixed bug where partial upload wouldn't be completed if the last PUT included no data
- Return 206 (partial content) status when a Range header is specified
- Return an ETag on object get - BlobReadChannel uses this to ensure we fail when the blob is updated between successive chunks being fetched)
- The 416 on zero-length blobs was one of(?) the causes of #125668
Hides some of the "extra" lines from ESQL's documentation. These lines
are required to make the documentation into nice tests which is
important to make sure the docs don't get out of date. But readers don't
need to see them.
This PR contains the following updates:
| Package | Update | Change | |---|---|---| |
docker.elastic.co/wolfi/chainguard-base-fips | pinDigest | -> `ebfc3f1`
|
---
### Configuration
📅 **Schedule**: Branch creation - "after 1pm on tuesday" (UTC),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once
you are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box
---
This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyI6RGVsaXZlcnkvUGFja2FnaW5nIiwiPm5vbi1pc3N1ZSIsIlRlYW06RGVsaXZlcnkiLCJhdXRvLW1lcmdlLXdpdGhvdXQtYXBwcm92YWwiXX0=-->
This PR contains the following updates:
| Package | Update | Change | |---|---|---| |
docker.elastic.co/wolfi/chainguard-base | digest | `c4e10ec` ->
`29150cd` |
---
### Configuration
📅 **Schedule**: Branch creation - "after 1pm on tuesday" (UTC),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once
you are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box
---
This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyI6RGVsaXZlcnkvUGFja2FnaW5nIiwiPm5vbi1pc3N1ZSIsIlRlYW06RGVsaXZlcnkiLCJhdXRvLW1lcmdlLXdpdGhvdXQtYXBwcm92YWwiXX0=-->
The `METHOD /path/components?and=query` string representation of a
request is becoming increasingly difficult to parse, with slight
variations in parsing between the implementation in `S3HttpHandler` and
the various other implementations. This commit gets rid of the
string-concatenate-and-split behaviour in favour of a proper object that
has predicates for testing all the different kinds of request that might
be made against S3.
This removes some leftover remnants from using StringBuilder
as part of String concatenation. Since we no longer support JDK 8,
this code can be safely removed.
The transform API for instrumentation should return null when no
transformation occurs. This commit fixes our entitlement transformer to
return null instead of the original buffer.
Co-authored-by: Ryan Ernst <ryan@iernst.net>
* Initial draft test with index version setup
* Adding test in phases
* [CI] Auto commit changes from spotless
* Adding test for search functionality
* Adding test for highlighting
* Adding randomization during selection process
* Fix code styles by running spotlessApply
* Fix code styles by running spotlessApply
* Fixing forbiddenAPIcall issue
* Decoupled namedWritables to use separate fake plugin and simplified other override methods
* Updating settings string to variable and removed unused code
* Fix SemanticQueryBuilder dependencies
* fix setting maximum number of tests to run
* utilizing semantci_text index version param and removed unwanted override
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* remove as many assembler-related redirects as possible
* Update docs/redirects.yml
* delete more unused temp redirects
* remove more redirects
* remove all redirects to see remaining errors
It's more natural for `manage` and `manage_data_stream_lifecycle` to
grant access to management style APIs both for regular data streams and
their failure stores.
This PR adds support for privileges to grant access to both data and
failures selectors (without granting access to everything, à la `all`),
and extends `manage` and `manage_data_stream_lifecycle` to grant failure
store access, in addition to regular data stream access.
`manage_failure_store` still grants failures-only access.
This test seemed to fail because we would index another document before
the master processed the data stream rollover - which is possible
because non-master nodes expose the next cluster state a little bit
earlier than the master node.
Caused by #122852Fixes#124978
The chunked text is only required when the actual inference request is made,
using a string supplier means the string creation can be done much much closer
to where the request is made reducing the lifespan of the copied string.
In particular:
* Remove all links (both asciidoc and markdown) from the JSON definition files.
* This required a two phase edit, from asciidoc links to markdown, and then removal of markdown (replace with markdown text). This is because the asciidoc does not have the display text, and because some links were already markdown.
* Split predicates into is_null and is_not_null
* We kept the old combined version because the main docs still use that, so now we have both combined and separate versions, and Kibana can select the version they want.
This PR makes authorization denial messages account for privileges that
grant access to the failure store. This is a minimal implementation that
only displays information around failure store privileges for requests
that include concrete names with the `::failures` selector. This avoids
including irrelevant information in regular non-failures requests. We
can improve on this in follow ups.
Closes: ES-11158
This test failed when the `disk.indices.forecast` value was a decimal number.
We adjust the regex to allow decimal values and for consistency we also allow negative values.
Fixes#125711Fixes#125848Fixes#125661
With #123610 we disabled parallel collection for field and script sorted top hits,
aligning its behaviour with that of top level search. This was mainly to work around
a bug in script sorting that did not support inter-segment concurrency.
The bug with script sort has been fixed with #123757 and concurrency re-enabled for it.
While sort by field is not optimized for search concurrency, top hits benefits from it
and disabling concurrency for sort by field in top hits has caused performance
regressions in our nightly benchmarks.
This commit re-enables concurrency for top hits with sort by field is used. This
introduces back a discrepancy between top level search and top hits, in that concurrency
is applied for top hits despite sort by field normally disables it. The key difference
is the context where sorting is applied, and the fact that concurrency is disabled
only for performance reasons on top level searches and not for functional reasons.
The transform API for instrumentation should return null when no
transformation occurs. This commit fixes our entitlement transformer to
return null instead of the original buffer.
closes#125920
Optimize calculating the usage of ILM policies in the `GET _ilm/policy` and `GET _ilm/policy/<policy_id>` endpoints by xtracting a separate class that pre-computes some parts on initialization (i.e. only once per request) and then uses those pre-computed parts when calculating the usage for an individual policy. By precomputing all the usages, the class makes a tradeoff by using a little bit more memory to significantly improve the overall processing time.
Now that all actions that DLM depends on are project-aware, we can make DLM itself project-aware.
There still exists only one instance of `DataStreamLifecycleService`, it just loops over all the projects - which matches the approach we've taken for similar scenarios thus far.
Originally this metrics collector was just there to ignore API calls
that didn't make it all the way to S3, but (a) it doesn't really do that
because it also apparently ignores 4xx responses and (b) it also does a
bunch of other metrics collection too. `IgnoreNoResponseMetricsCollector`
is definitely the wrong name these days so this commit renames it to
something more general.
I was debating on having this tests in the original PR anyways. It ain't
worth the flakiness. We know the oversampling setting gets updated given
the other tests.
closes: https://github.com/elastic/elasticsearch/issues/125851
We have some tolerance wound how many bytes we report for these completion fields. But the
values depend on the distribution of the random values that determine how many docs get
an option field. This commit makes the test more precise by computing the real ratio
between docs that have the optional field and the total number of docs, so that we
can base assertion on more realistic expectations.
Closes#123269
This test could still fail by consecutive API calls hitting different
nodes with different versions of the cluster state.
Follow-up of #125621Fixes#125752
Adding basic validation to prevent using `::` selectors when defining index permissions.
Index names do not allow colon character (`:`), hence the index name patterns that
would include double colon (`::`), would never match any of the index names.
To avoid confusion, we are preventing using `::` in role index name patterns.
For example, the `test-*::failures` will be rejected during `test-role` validation:
```
PUT /_security/role/test-role
{
"indices": [
{
"names": ["test-*::failures"],
"privileges": ["read"]
}
]
}
```
Fixes an issue where indexing throttling kicks in while disk IO is throttling.
Instead disk IO should first unthrottle, and only then, if we still can't keep up with the merging load, start throttling indexing.
Fixes elastic/elasticsearch-benchmarks#2437
Relates #120869
This change moves the query phase a single roundtrip per node just like can_match or field_caps work already.
A a result of executing multiple shard queries from a single request we can also partially reduce each node's query results on the data node side before responding to the coordinating node.
As a result this change significantly reduces the impact of network latencies on the end-to-end query performance, reduces the amount of work done (memory and cpu) on the coordinating node and the network traffic by factors of up to the number of shards per data node!
Benchmarking shows up to orders of magnitude improvements in heap and network traffic dimensions in querying across a larger number of shards.
This primarily splits the old preview:true warning from the newer applies_to approach. Since all of our current applies_to examples are actually just behaviour modifications of current functions, we do not use the official docs {applies_to} syntax. However there is code to make use of that in the case where we have an entirely new function which will appear in a new version.
Co-authored-by: Alexander Spies <alexander.spies@elastic.co>
Previously if an aggregate_metric_double was present amongst fields and
you tried to sort on any (not necessarily even on the agg_metric itself)
field in ES|QL, it would break the results.
This commit doesn't add support for sorting _on_ aggregate_metric_double
(it is unclear what aspect would be sorted), but it fixes the previous
behavior.
This patch builds on the work in #113757, #122999, #124594, #125529, and
#125709 to natively store array offsets for scaled float fields instead of
falling back to ignored source when synthetic_source_keep: arrays.
Currently, the Lucene90DocValuesProducer uses optimized IntObjectHashMaps
to track various entries for each field, while the
ES87TSDBDocValuesProducer uses regular HashMap<String, Object>. This patch
updates the ES87TSDBDocValuesProducer class to also use the optimized
hash maps.
This adds cluster settings to allow for a choice of write load metrics
in the data stream auto-sharding calculations. There are separate
settings for the increasing and decreasing calculations. Both default
to the existing 'all-time' metric for now.
This also refactors `DataStreamAutoShardingServiceTests`. The main two things done are:
- Split large test methods which do several independent tests in
blank code blocks into more smaller methods.
- Fix an unnecessarily complicated pattern where the code would
create a `Function` in a local variable and then immediately
`apply` it exactly once... rather than just executing the code
normally.
Add CSV tests for invalid ranges (i.e. where the lower bound is greater than the upper bound) for several data types. It looks at first glance like there should be a bug here with Date Nanos, but I cannot trigger one, and analysis suggests the code that has the bug is in fact unreachable. I've left comments to that effect.
Appends the FailedShardEntry request to the 'shard-failed'
task source string in ShardFailedTransportHandler.messageReceived().
This information will now be available in the 'source' string for
shard failed task entries in the Cluster Pending Tasks API response.
This source string change matches what is done in the
ShardStartedTransportHandler.
Closes#102606.
The CCS is currently not supported for failure store backing indices.
This PR adjusts the selector parsing (introduced in #118614) to prevent
using `::failures` and `::data` selectors with cross-cluster expressions.
For example, `GET my_remote_cluster:logs-*::failures/_search` request
will fail early, during expression parsing.
To test manually, run `./gradlew run-ccs` and execute the example request.
Adds prefixes to various randomly-generated values to make it easier to
pin down where they're coming from in debugging sessions. Also forces
the STS expiry time to be rendered in UTC.
The random license selection in
https://github.com/elastic/elasticsearch/pull/125584 was incorrectly
assuming ENTERPRISE was needed, when it should have been PLATINUM. This
resulted in occasional test failures (expected to fail on PLATINUM, and
succeeded).
Fixes#125734
We have seen occasional failures with the existing tolerance, in
`testEwmr_threadSafe` which contains some randomness. We were
asserting the result of a lot of f.p. operations with a tolerance of
1.0e-13. The highest error I saw in any of the reported failures was less than
1.2e-13. This PR increases the tolerance to 2.0e-13 which should allow
all those to pass.
Fixes#124692
This PR grants `manage_failure_store` to the internal user
`_data_stream_lifecycle` to enable life-cycle management for the failure
indices of data stream, which includes rollovers using the failures
selector.
I'm only unit testing this but we also need to add DLM tests for the
failure store with security enabled.
Relates: ES-11355
In #120505 we introduced a capabilities check in the yaml test `deprecation/10_basic/Test Deprecations` but we forgot to add them in the `RestDeprecationInfoAction`.
In this PR we add the capabilities which will enable the test and we make the test resilient to the warning that occurs when the `.security-7` index is present.
This tracks the highest value seen for the recent write load metric
any time the stats for a shard was computed, exposes this value
alongside the recent value, and persists it in index metadata
alongside it too.
The new test in `IndexShardTests` is designed to more thoroughly test
the recent write load metric previously added, as well as to test the
peak metric being added here.
ES-10037 #comment Added peak load metric in https://github.com/elastic/elasticsearch/pull/125521
This PR adds project-id to both SnapshotsInProgress and Snapshot so that
they are aware of projects and ready to handle snapshots from multiple
projects.
Relates: ES-10224
This patch builds on the work in #113757, #122999, #124594, and #125529 to
natively store array offsets for unsigned long fields instead of falling
back to ignored source when synthetic_source_keep: arrays.
Extends the max blob size to 10MiB, and sets the buffer size to 5MiB, to
ensure that sometimes the S3 repo analysis tests will create blobs using
the multipart upload flow.
The pattern-matching in `S3HttpHandler` is overly specific and carefully
crafted to handle the exact requests that the AWS SDK v1 makes. It turns
out that the AWS SDK v2 makes requests that are slightly different. This
commit generalizes the pattern-matching to handle both SDKs.
This allows a `rescore_vector: {oversample: 0}` to indicate bypassing
oversampling and rescoring.
This is useful for:
- Updating a quantized mapping to turn off automatic rescoring
- Bypassing oversampling at query time in an ad-hoc manner if its on by default in the mapping
closes: https://github.com/elastic/elasticsearch/issues/125157
Load field caps from store if they haven't been initialised through a refresh yet.
Keep the plain reads to not mess with performance characteristics too much on the good path but protect against confusing races when loading field infos now (that probably should have been ordered stores in the first place but this was safe due to other locks/volatiles on the refresh path).
Closes#125483
This PR contains the following updates:
| Package | Update | Change | |---|---|---| |
docker.elastic.co/wolfi/chainguard-base | digest | `15a4191` ->
`c4e10ec` |
---
### Configuration
📅 **Schedule**: Branch creation - "after 1pm on tuesday" (UTC),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once
you are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box
---
This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyI6RGVsaXZlcnkvUGFja2FnaW5nIiwiPm5vbi1pc3N1ZSIsIlRlYW06RGVsaXZlcnkiLCJhdXRvLW1lcmdlLXdpdGhvdXQtYXBwcm92YWwiXX0=-->
- Adds docker image based on chainguard base fips image
- x86 only for now as the base image is x86 only
- the image does not provide any elasticsearch.yml configuration. for testing purposes you can follow the elasticsearch fips guide available at https://github.com/elastic/FIPSGuide/tree/main/elasticsearch
The image is shipped with:
- org.bouncycastle:bc-fips:1.0.2.5 and org.bouncycastle:bctls-fips:1.0.19 in Elasticsearch libs folder
- config/jvm.options.d/fips.options for fips specific JVM options
- fips_java.security file
- fips_java.policy
Out of scope:
- Add packaging test coverage (part of later PR as we want to provide that image for testing early and packaging tests require more general restructuring for support fips scenarios)
This PR adds support for checking access to the failure store via the
Has Privileges API.
To check access for a data stream `logs`, a request must query for a
concrete named privilege, `read_failure_store` or
`manage_failure_store`, e.g., a request to the HasPrivileges API by a
user with `read_failure_store` over `logs`:
```
POST /_security/user/_has_privileges
{
"index": [
{
"names": ["logs"],
"privileges": ["read_failure_store", "read", "indices:data/read/*"]
}
]
}
```
Returns:
```
{ "username": "<...>", "has_all_requested": false,
"cluster": {}, "index": { "logs": {
"read_failure_store": true, "read": false, <1>
"indices:data/read/*": false <2> } }, "application": {}
}
```
Note that `<1>` and `<2>` are both `false` since `read` is not covered by `read_failure_store` and neither are any raw actions like `indices:data/read/*` since these implicitly correspond to data access.
Selectors are not allowed in the index patterns of HasPrivileges requests to avoid ambiguities such as checking `read` on `logs::failures` as well as the ambiguity of index patterns that are regular expressions.
These tests had the potential to fail when two consecutive GET data
streams requests would hit two different nodes, where one node already
had the cluster state that contained the new backing index and the other
node didn't yet.
Caused by #122852Fixes#124837Fixes#124849Fixes#124850Fixes#124886Fixes#125083
* Specify index component when retrieving lifecycle
* Add getters for the failure lifecycle
* Conceptually introduce the failure store lifecycle (even for now it's the same)
This test had the potential to fail when two consecutive GET data
streams requests would hit two different nodes, where one node already
had the cluster state that contained the new backing index and the other
node didn't yet.
Caused by #122852Fixes#124987
These tests had the potential to fail when two consecutive GET data
streams requests would hit two different nodes, where one node already
had the cluster state that contained the new backing index and the other
node didn't yet.
Caused by #122852Fixes#124882Fixes#124885
We often call `addTemporaryStateListener` with the `ClusterService` of a
random node, or the currently elected master. This commit adds utilities
for this common pattern.
Adds the `original_types` to the description of ESQL's `unsupported`
fields. This looks like:
```
{
"name" : "a",
"type" : "unsupported",
"original_types" : [
"long",
"text"
]
}
```
for union types. And like:
```
{
"name" : "a",
"type" : "unsupported",
"original_types" : [
"date_range"
]
}
```
for truly unsupported types.
This information is useful for the UI. For union types it can suggest
that users append a cast.
The dataset name for the deprecation logs index was previously renamed
from `deprecation.elasticsearch` to `elasticsearch.deprecation` in
order to follow the pattern of `product.group`. The deprecation index
template, however, was not updated. This causes indexing errors once
upgraded to 9.0 due to the dataset name having changed on a
constant_keyword field. In order to avoid that mismatch, this commit
renames the deprecation indexing datastream to match the dataset name.
The old template is kept in place, but marked as deprecated, so that any
deprecation logs written during upgrading to 9.x will continue to be
indexed into the old datastream.
closes#125445
Follow up to #125345. If the query contained both a nanos and a millis comparison, we were formatting the dates incorrectly for the lucene push down. This PR adds a test and a fix for that case.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
This patch builds on the work in #113757, #122999, and #124594 to natively
store array offsets for boolean fields instead of falling back to ignored
source when `synthetic_source_keep: arrays`.
Extends the predicate in `AwsCredentialsUtils` to verify that we are
using a proper AWS v4 signature complete with the correct region and
service, rather than just looking for the access key as a substring.
We will need custom logic in the time-series aggregation operator, such
as smoothing the rate across buckets. To address this, this PR
introduces a TimeSeriesAggregationOperator that extends
HashAggregationOperator to support the addition logic.
Today shard's engine mutation are guarded by an engineMutex object monitor. But we would like to be able to execute one or more operations on an engine instance, without this instance being resetted during the execution of the operation.
In order to do that, this change replaces the engineMutex by a reentrant read/write lock and introduces two new methods IndexShard#withEngine() and IndexShard#withEngineOrNull() that can be used to execute an operation while avoiding the current engine instance to be reset. It does not prevent it to be closed during execution though.
Relates ES-10826
Co-authored-by: Francisco Fernández Castaño <francisco.fernandez.castano@gmail.com>
Fixes#125439
We were incorrectly formatting nanosecond dates when building lucene queries. We had missed this in our testing because none of the CSV tests were running against Lucene. This happened because the date nanos test data includes multivalue fields. Our warning behavior for multivalue fields is inconsistent between queries run in Lucene and queries run in pure ES|QL without pushdown. Our warning tests, however, require that the specified warnings be present in all execution paths. When we first built the date nanos CSV tests, we worked around this by always using an MV function to unpack the multivalue fields. But we forgot that using an MV function prevents the entire query from being pushed down to Lucene, and thus that path wasn't being tested.
In this PR, I've duplicated many of the tests to have a version that doesn't use the MV function, and uses warningRegex instead of warning. The regex version does not fail if the header is absent, so it's safe to use in both modes. Rewriting the tests this way revealed several situations in which this bug can manifest, all of which are fixed in this PR. I cannot be confidant that there aren't more paths that can trigger this bug and aren't covered by these tests, but I haven't found any yet.
I've left some trace level logging that I found helpful while debugging this.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
This PR updates the ES|QL grammar to include the selector portion of an index pattern. Patterns
are recombined before being sent to field caps. Field caps already supports this functionality, so
this is primarily wiring it up where needed.
This commit adds a new command line option to the run task to include
plugins. The option takes in a comma separated list of plugins from the
plugins directory. For example:
```
./gradlew run --with-plugins=analysis-icu,analysis-nori
```
We shouldn't mutate LogicalPlan.child().output() (or just
LogicalPlan.output()); we did so in RENAME, but only in
EsqlNodeSubclassTest, making them flaky.
>The PR at https://github.com/elastic/elasticsearch/pull/122250 seems to
have created a flaky test failure in `EsqlNodeSubclassTests`. Local runs
with `-Dtests.iters=100` lead to about two dozen failures in over 70k
tests run. This is not a high failure rate, but still requires
addressing. > >The single line added to the Analyzer by that PR causes
an `UnsupportedOperationException` on attempting to mutate an immutable
collection when running `EsqlNodeSubclassTests`. It turns out that this
code path comes from `Rename.output()` which is only called in test
scenarios. So a quick fix is to copy the child output into a mutable
collection.
Instead of processing cluster state updates on the cluster state applier
thread, we fork to a different thread where ILM's runtime of processing
the cluster state update does not affect the speed at which the cluster
can apply new cluster states. That does not mean we don't need to
optimize ILM's cluster state processing, as the overall amount of
processing is generally unaffected by this fork approach (unless we skip
some cluster states), but it does mean we're saving a significant amount
of processing on the critical cluster state applier thread.
Additionally, by running ILM's state processing asynchronously, we allow
ILM to skip some cluster states if the management thread pool is
saturated or ILM's processing is taking too long.
Since the data returned from time-series indices (shard-level) is sorted
by tsid and timestamp, we can optimize the timeseries blockhash by
comparing the incoming tsid and timestamp only with the last values,
thus avoiding the need for lookups or internal hashes. Additionally, we
can reuse the internal large arrays when rebuilding key blocks.
* Add field process.executable.name to profiling-events
* Amend query to aggregate events by executable name
* Send flamegraph row with grouping by executable name
* Flamegraph sub-aggregation by thread name
* Rework internal data model
* Cleanups
* Fix building tests
* Fix GetStackTracesResponseTests
* Fix unit tests
* Fix remaining unit tests
* [CI] Auto commit changes from spotless
* Fix flamegraph yaml tests
* Fix yaml REST tests
* Increase INDEX_TEMPLATE_VERSION for profiling.executable.name
* Fix yamlRestCompatTest
Co-authored-by: Jake Landis <jake.landis@elastic.co>
* Rename executable name to process name
* Remove warnings meant for testing
* Replace ChunkedToXContentHelper.wrapWithObject() with .object()
* Fix comment in ProfilingIndexTemplateRegistry.java
* Simplify sorting of unique stacktrace and host IDs
* [CI] Auto commit changes from spotless
* Add 'missing()' to aggregations
* Fix syntax error after resolving merge conflicts
* Revert "Rename executable name to process name"
This reverts commit e514874198.
* Set FRAMETYPE_EXECUTABLE to 0x103
* Fix TransportGetFlamegraphActionTests
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Jake Landis <jake.landis@elastic.co>
These tests had the potential to fail when two consecutive GET data
streams requests would hit two different nodes, where one node already
had the cluster state that contained the new backing index and the other
node didn't yet.
Caused by #122852Fixes#124846Fixes#124950Fixes#124999
The `CsvTests` has a slight difference regarding warnings from real
Elasticsearch indices and this is worth documenting. I've also added an
explanation to `SingleValueMatchQuery` that explains *exactly* when it
makes a warning because it's not *exactly* the same as when the compute
engine would make a warning. The resulting documents are the same - but
the warnings are not.
This PR was originally focused on improving support for Kibana docs, in particular the missing operator docs, but it has expanded to cover a bunch of related things:
* Primarily the main work was to improve operators support. ESQL generated docs cover all functions and most operators for which their is a clear operator class and test class. However, some are built-in behaviour and need additional support. This PR adds more generated content for those operators.
* Various specific operators requested by Kibana: Cast & null-predicates, and in particular the addition of examples
* Two functions without examples: mv_append and to_date_nanos
* Many small visual document cleanups (spelling, grammar, capitalization, etc.)
* Initial support for `applies_to` for multi-version differentiation.
This last point requires more work, as it is not yet agreed on just how we want this to look. We'll probably need to do refinements in followup PR. Consider the version in this PR as a first step into how this could look.
This PR adds a check and a warning message if the user is trying to install a plugin containing the legacy plugin-security.policy used by SecurityManager.
Relates to ES-10846
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
Relates #101805
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
Relates #101805
Since #122905 we were throwing NPEs (i.e. 5xxs) when a rollover request has an unknown/non-existent target. Before that, we returned a 400 - illegal argument exception. We now return a 404 which matches "missing target" better. Additionally, to avoid this from happening again, we add a YAML test that asserts the correct exception behavior.
Adds a new cache and setting
TransportGetAllocationStatsAction.CACHE_TTL_SETTING
"cluster.routing.allocation.stats.cache.ttl" to configure the max age
for cached NodeAllocationStats on the master. The default
value is currently 1 minute per the suggestion in issue 110716.
Closes#110716
Previously, when calculating the offsets, we just compared the values as-is
without any loss of precision. However, when the values were saved into doc
values and loaded in the doc values loader, they could have lost precision.
This meant that values that were not duplicates when calculating the
offsets could now be duplicates in the doc values loader. This interfered
with the de-duplication logic, causing incorrect values to be returned.
My solution is to apply the precision loss before calculating the offsets,
so that both the offsets calculation and the SortedNumericDocValues
de-duplication see the same values as duplicates.
Make the conversion functions that process `BytesRef`s into `BytesRefs`
keep the `OrdinalBytesRefVector`s when processing. Let's use `TO_LOWER`
as an example. First, the performance numbers:
```
(operation) Mode Score Error -> Score Error Units
to_lower 30.662 ± 6.163 -> 30.048 ± 0.479 ns/op
to_lower_ords 30.773 ± 0.370 -> 0.025 ± 0.001 ns/op
to_upper 33.552 ± 0.529 -> 35.775 ± 1.799 ns/op
to_upper_ords 35.791 ± 0.658 -> 0.027 ± 0.001 ns/op
```
The test has a 8192 positions containing alternating `foo` and `bar`.
Running `TO_LOWER` via ordinals is super duper faster. No longer
`O(positions)` and now `O(unique_values)`.
Let's paint some pictures! `OrdinalBytesRefVector` is a lookup table.
Like this:
```
+-------+----------+
| bytes | ordinals |
| ----- | -------- |
| FOO | 0 |
| BAR | 1 |
| BAZ | 2 |
+-------+ 1 |
| 1 |
| 0 |
+----------+
```
That lookup table is one block. When you read it you look up the
`ordinal` and match it to the `bytes`. Previously `TO_LOWER` would
process each value one at a time and make:
```
bytes
-----
foo
bar
baz
bar
bar
foo
```
So it'd run `TO_LOWER` once per `ordinal` and it'd make an ordinal
non-lookup table. With this change `TO_LOWER` will now make:
```
+-------+----------+
| bytes | ordinals |
| ----- | -------- |
| foo | 0 |
| bar | 1 |
| baz | 2 |
+-------+ 1 |
| 1 |
| 0 |
+----------+
```
We don't even have to copy the `ordinals` - we can reuse those from the
input and just bump the reference count. That's why this goes from
`O(positions)` to `O(unique_values)`.
The Elastic inference service removes the default models at startup if the node cannot access EIS.
Since #125242 we don't store default models in the cluster state but we still try to delete them.
This change ensures that we don't try to update the cluster state when a default model is deleted
since the delete is not performed on the master node and default models are never stored in the cluster state.
This change sets the zstd best speed level from 0 to 1.
Compression level 0 means the default level will be used, which is level 3.
The intent of speed is to use the fastest compression level so 1 is a more appropriate value.
Note that zstd best speed is still under a feature flag so this change is just meant to evaluate the impact on our nightly benchmarks.
Further testing is needed (wip) to remove the feature flag entirely.
In preparation for integrating with SageMaker, we want to reuse the
existing SecretSettings.
- AmazonBedrockSecretSettings moved from services.amazonbedrock to
common.amazon.
- AmazonBedrockSecretSettings was renamed to AwsSecretSettings.
- accessKey and secretKey are now encapsulated.
On outbound messages we know the action name whether it's a request or
response, so we can report it in logs rather than just relying on the
payload's type.
Cohere embeddings are expected to be normalized to unit vectors, but due to floating point precision issues,
our check ({@link DenseVectorFieldMapper#isNotUnitVector(float)}) often fails.
This change fixes this bug by setting the default similarity for newly created Cohere inference endpoint to cosine.
Closes#122878
Remove the `OutboundMessage` class that needlessly holds on to the the response instances after they are not needed any longer. Inlining the logic should save considerably heap under pressure and enabled further optimisations.
Added BWCLucene8*Codecs wrapper classes for the lucene8* equivalents. A BWC wrapper is initialized for archive indices and provides read-only capabilities for an index.
ModelRegistryMetadata has now been backported to 8.19 via #125150. This update ensures that we properly differentiate between nodes running 8.19.x (which supports the new custom metadata) and 9.0.x (which does not).
To achieve this, this PR introduces a new `supportsVersion(TransportVersion)` method for `NamedWriteable` and `NamedDiff`, allowing subclasses to customize their backward compatibility behavior.
This PR updates the documentation for Creating classic plugins, replacing the instructions relative to the Java SecurityManager with information on Entitlements.
Relates to ES-10846
This PR updates the different methods in TestProjectResolvers so that
their names are more accurate and behaviours to be more as expected.
For example, In MP-1749, we differentiate between single-project and
single-project only resolvers. The later should not support multi-project.
This changes the default value for the Exponentially Weighted Moving Rate calculation used for the 'recent write load' metric in indexing stats to 5 minutes (as agreed over Slack) and persists the value in the index metadata alongside the existing write load metric.
The value is still not used in the data stream autosharding calculation, that will be yet one more PR.
There are a couple of drive-by changes in this PR:
It adds a comment to DataStreamAutoShardingService.computeOptimalNumberOfShards, because the nested min and max calls are quite hard to understand at a glance.
It changes IndexShard.indexingStats() so that, if it is called before the shard has entered the started state, it uses a timeSinceShardStartedInNanos value of zero when calling InternalIndexingStats.stats(). Previously, it would have passed the current relative time in nanos as timeSinceShardStartedInNanos (because startedRelativeTimeInNanos would be zero) which is arbitrary and incorrect (since the zero point of System.nanoTime() is arbitrary). This didn't actually matter, since InternalIndexingStats.postIndex would not increment the metrics while in recovery, so the numerator used to calculate the write load would be zero if the shard has not started, so it doesn't matter if the denominator is incorrect. However, it is good defensive coding not to rely on that, and to pass a correct value instead.
* Use ALL-UNNAMED for the unnamed module in warnings
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
A while ago we introduced a completion postings format extension to eventually
be able to customize how completion FSTs are loaded. See #111494.
We have never leveraged this extension, and meanwhile Lucene is moving
to always load FSTs off-heap, and no longer allow on-heap.
See https://github.com/apache/lucene/pull/14364 .
This commit removes the SPI extension as it is no longer needed.
Unauthenticated pulls from Docker Hub are heavily rate-limited. After
this change, we will use authenticated Docker Hub user so that we can
pull public images for tests without being rate-limited.
Modifies Coordinator to enable logging in ClusterFormationFailureHelper
when started and disables logging in ClusterFormationFailureHelper when
stopped. The warning scheduler handling and invariant check in the
Coordinator are left as is, with the logging boolean set independently,
eliminating the need to hold the mutex in doStop() when Coordinator.stop()
is called when the Node is shutdown.
Closes#105559.
* Update docs/changelog/125244.yaml
This PR implements authorization logic for failure store access. It
builds on https://github.com/elastic/elasticsearch/pull/122715.
Access to the failure store is granted by two privileges:
`read_failure_store` and `manage_failure_store`. Either of these
privileges lets a user access a failure store via the `::failures`
selector, as well as access its backing failure indices.
`read_failure_store` grants read access (for example to search documents
in a failure store), `manage_failure_store` grants access to write
operations, such as rollover. Users with only `read` or `manage` on a
data stream do not get failure store access. Vice versa, users with
`read_failure_store` and `manage_failure_store` do not get access to
regular data in a data stream.
The PR implements this by making authorization logic selector-aware. It
involves two main changes:
1. Index permission groups now compare the selector under which an index resource is accessed to the selector associated with the group.
2. The `AuthorizedIndices` interface likewise uses selectors to decide which indices to treat as authorized. This part of the change requires a sizable refactor and changes to the interface.
The high-level behavior for selector-aware search is as follows:
For a user with `read_failure_store` over data stream `logs`:
- `POST /logs::failures/_search` returns the documents in the failure store.
- `POST /logs/_search` returns a 403.
- `POST /logs/_search?ignore_unavailable=true` and `POST /*/_search` return an empty result.
Similarly, for a user with `read` over data stream `logs`:
- `POST /logs::failures/_search` returns a 403.
- `POST /logs/_search` returns documents in the data stream.
- `POST /logs::failures/_search?ignore_unavailable=true` and `POST /*::failures/_search` return an empty result.
A user with both `read` and `read_failure_store` over data stream `logs`
gets access to both `POST /logs::failures/_search` and `POST
/logs/_search`.
The index privilege `all` automatically grants access to both data and
the failures store, as well as all hypothetical future selectors.
Resolves: ES-10873
Reduce boilerplate associated with creating `IndexFieldCapabilities`
instances. Since it's a class with a large number of fields, it makes
sense to define a builder object, as that can also help with all the
Boolean and null blindness going on. As with `FieldCapabilitiesBuilder`,
this is only used in tests, to avoid any potential performance hit in
production code.
This PR overloads the `process` method and allows it to be used with
several `ReservedStateChunks`. The purpose is to allow several state
chunks to be spread across several files but handled as a single cluster
state update by validating and merging them into a single representation
of the `ReservedStateChunk`.
This change updates the METRICS command syntax from METRICS
index_pattern aggregations | ... to METRICS index_pattern | ...,
removing inline aggregations. This PR does not filter out
non-time-series indexes from the source; this will be addressed in a
follow-up PR to keep this PR small.
This patch builds on the work in #122999 and #113757 to natively store
array offsets for numeric fields instead of falling back to ignored source
when `source_keep_mode: arrays`.
For creating and deleting projects in multi-project tests, we need
create and delete settings and secrets files on the fly. This PR adds
such feature to the Java test cluster with an option to specify the
config directory.
When retrieving a default inference endpoint for the first time, the system automatically creates the endpoint.
However, unlike the `put inference model` action, the `get` action does not redirect the request to the master node.
Since #121106, we rely on the assumption that every model creation (`put model`) must run on the master node, as it modifies the cluster state. However, this assumption led to a bug where the get action tries to store default inference endpoints from a different node.
This change resolves the issue by preventing default inference endpoints from being added to the cluster state. These endpoints are not strictly needed there, as they are already reported by inference services upon startup.
**Note:** This bug did not prevent the default endpoints from being used, but it caused repeated attempts to store them in the index, resulting in logging errors on every usage.
A SDK bug[^1] means that these values sometimes don't parse as an
integer. We don't need them to do so in order to pass the test, so this
commit relaxes the test to accept any string.
Closes#125090Closes#125166
[^1]: https://github.com/aws/aws-sdk-java-v2/issues/5968
Using ASM CheckClassAdapter was key to diagnose the issue we had with incorrect signatures for some check methods.
In this PR I polished up the code I used to pinpoint the issue, and made it available via a system property so it can be turned on if we need it (and it's always on for Entitlements IT tests too).
It is also turned on in case we get VerifyErrors during retransformClasses early in the Entitlement agent bootstrap: retransformClasses runs in the native part of the JVM, so the VerifyError it produces is not so readable (e.g. it lacks a full stack trace and a description); in case this happens, we re-apply the transformation with verification turned on to get a meaningful error before dying.
Did a few things:
* Rewrite Kibana docs asciidoc links to be MD links
* Make kibana docs links absolute to planned publication path
* Clarify which operators are generated and which are static
* Removed the trailing .md from kibana docs links
This is part of the work to make DLM project-aware.
These two features were pretty tightly coupled, so I saved some effort
by combining them in one PR.
Remove `-da:org.elasticsearch.index.mapper.DocumentMapper` and `-da:org.elasticsearch.index.mapper.MapperService` from mixed cluster/bwc cluster setups. Given that #122606 is now backported to the 8.18 branch.
We need to create a new trace context when registering a task. There are
other issues with the task group, but I need this change to enable
allow_partial_results first, then follow up with other enhancements.
This commit adds a conversion function from numerics (and aggregate
metric doubles) to aggregate metric doubles.
It is most useful when you have multiple indices, where one index uses
aggregate metric double (e.g. a downsampled index) and another uses a
normal numeric type like long or double (e.g. an index prior to
downsampling).
This uses the recently-added `ExponentiallyWeightedMovingRate` class
to calculate a write load which favours more recent load and include
this alongside the existing unweighted all-time write load in
`IndexingStats.Stats`.
As of this change, the new load metric is not used anywhere, although
it can be retrieved with the index stats or node stats APIs.
With Gradle 8.13 we can now auto provision the jdk used by the gradle daemon. Our configuration relies on jdk21.
With this autoprovisioning enabled each gradle build will use adoptium jdk 21 for gradle jvm
This commit adds a limit to the `step_info` contained in
`LifecycleExcutionState` so that large step info messages will not be
stored in the cluster state. Additionally, when generating an ILM
history failure, the full exception that is "stringified" is truncated
to the same limit, ensuring that we do not accidentally index gigantic
documents in the history store.
The default limit is 1024 characters.
Resolves#124181
This adds a new merge scheduler implementation that uses a (new)
dedicated thread pool to run the merges. This way the number of
concurrent merges is limited to the number of threads in the pool
(i.e. the number of allocated processors to the ES JVM).
It implements dynamic IO throttling (the same target IO rate for all
merges, roughly, with caveats) that's adjusted based on the number
of currently active (queued + running) merges.
Smaller merges are always preferred to larger ones, irrespective of
the index shard that they're coming from.
The implementation also supports the per-shard "max thread count"
and "max merge count" settings, the later being used today for indexing throttling.
Note that IO throttling, max merge count, and max thread count work similarly,
but not identical, to their siblings in the ConcurrentMergeScheduler.
The per-shard merge statistics are not affected, and the thread-pool statistics should
reflect the merge ones (i.e. the completed thread pool stats reflects the total
number of merges, across shards, per node).
The JDK team has completely disabled the Java SecurityManager from Java 24. Elasticsearch has always used the Java SecurityManager as an additional protection mechanism; in order to retain this second line of defense, the Elasticsearch Core/Infra team has been working on the Entitlements project.
Similar to SecurityManager, Entitlements only allow calling specific methods in the JDK when the caller has a matching policy attached. In other words, if some code (in the main Elasticsearch codebase, in a plugin/module, or in a script) attempts to perform a "privileged" operation and it is not entitled to do so, a NotEntitledException will be thrown.
This PR includes the minimal set of changes to always use Entitlements, regardless of system properties or Java version.
Relates to ES-10921
Earlier work on the ES|QL port of docs to V3 introduced an issue in the build.gradle file making it fail with --configuration-cache. This fixes that, as well as one other broken link and removes some unused files.
In addition we bring back partial support for deleting unused files. It is tricky to have full support for this due to the mix of static and generated content, particularly in the operators snippets.
* [Gradle] Fix configuration cache issues in distribution subproject
- do not reference tasks from task execution time
- do not use project references at task execution time
- use layout.settingsDir for resolving root project dir
* Fix test coverage
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
This commit integrates `MinimalServiceSettings` (introduced in #120560) into the cluster state for all registered models in the `ModelRegistry`.
These settings allow consumers to access configuration details without requiring asynchronous calls to retrieve full model configurations.
To ensure consistency, the cluster state metadata must remain synchronized with the models in the inference index.
If a mismatch is detected during startup, the master node performs an upgrade to load all model settings from the index.
With #124904 we introduced server policy patching via system properties; now that we have the possibility to merge policies and scopes, it's worth applying the same mechanism to plugin policies too.
This PR changes the behaviour of plugin policy overrides; now they are not replacing the policy entirely, but patch it by adding the entitlements and scopes specified in the "patch policy"
* Add shardCountBefore/After operations to IndexReshardingMetadata
Any reshard operation will change shard count, so it makes sense
to make this information available here.
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
This commit adds a `timed_out` key to the error responses that represent a
timeout condition. It also adds an `X-Timed-Out` header to the response
indicating the same outside the response body.
testReturnsOnlyRequestedStats() was using
EnumSet.copyOf(randomSubsetOf(Metric.values().length, Metric.values()))
which will always return all of the metrics. The code was expecting
Metric.ALLOCATIONS and Metric.FS to sometimes not be in the returned set.
This change uses an explicit list of EnumSets to cover the range of
scenarios expected in the test code.
* Pass IndexReshardingMetadata over the wire
When I introduced IndexReshardingMetadata I inadvertently
only covered local serialization. It was not getting sent or
received over the wire. This fixes that.
This adds the manage_threads entitlement for reactor.core as part of the
azure-repository module. It looks like this is a requirement for offloading
azure blob store work.
In a few previous PR's we restructured the ES|QL docs to make it possible to generate them dynamically.
This PR just moves a few files around to make the query languages docs easier to work with, and a little more organized like the ES|QL docs.
A bit part of this was setting up redirects to the new locations, so other repo's could correctly link to the elasticsearch docs.
This fixes a general flaw in our build logic where we reach out to configurations of other projects.
This is not best practice and breaks future initiatives like IsolatedProjects that allow parallel configuration
of subprojects.
This makes using usesDefaultDistribution in our test setup for explicit by requiring a reason why it's needed.
This is helpful as part of revisiting the need for all those usages in our code base.
Clarify that it is expected sometimes to see inter-node connections
sending zero-window advertisements as part of the usual TCP backpressure
mechanism.
This field is only used (by security) for requests, having it in responses is redundant.
Also, we have a couple of responses that are singletons/quasi-enums where setting the value
needlessly might introduce some strange contention even though it's a plain store.
This isn't just a cosmetic change. It makes it clear at compile time that each response instance
is exclusively defined by the bytes that it is read from. This makes it easier to reason about the
validity of suggested optimizations like https://github.com/elastic/elasticsearch/pull/120010
When `ExecutorScalingQueue` rejects work to make the worker pool scale up while already being at max pool size (and a new worker consequently cannot be added), available workers might timeout just about at the same time as the task is then force queued by `ForceQueuePolicy`. This has caused starvation of work as observed for `masterService#updateTask` in #124667 where max pool size 1 is used. This configuration is most likely to expose the bug.
This PR changes `EsExecutors.newScaling` to not use `ExecutorScalingQueue` if max pool size is 1 (and core pool size is 0). A regular `LinkedTransferQueue` works perfectly fine in this case.
If max pool size > 1, a probing approach is used to ensure the worker pool is adequately scaled to at least 1 worker after force queueing work in `ForceQueuePolicy`.
Fixes#124667
Relates to #18613
When an invalid popMode occurs (due to an invalid query), ANTLR throws
an out-of-channel exception, bypassing the existing checks.
This PR extends the checks and properly reports the error back to the
user
Fix#119025
Due to recent grammar changes made ( token to no longer be reported by
its text rather by his internal token name. Due to the use of pushMode,
the symbol is not treated as a literal rather as a symbol.
To address this, the parser listener looks at the error message and
changes the message before returning it to the user.
Replace hacky regex approach with Vocabulary substitution (not
as pluggable as it could be yet much better)
Fix#124145
Relates #123085#121948
Co-authored-by: Alexander Spies <alexander.spies@elastic.co>
Information about the system, and in particular the running JVM, is
logged relatively late in startup. When problems occur in low level
initialization, it is often useful to have this information. This commit
moves logging of system information to right after logging has been
initialized. Note that it no longer utilizes the JvmInfo class as most
was already easily available and the static init of JvmInfo does a lot.
The JVM has a couple ways that Java options can be passed. In
Elasticsearch we ignore these common patterns, instead using our own
bespoke mechanisms for setting options (ES_JAVA_OPTS, etc). The primary
reason for this is some systems like Ubuntu setting these options
globally and then affecting Elasticsearch, often in ways that cause
Elasticsearch to fail to start.
This commit ignores _JAVA_OPTIONS, which is a little known environment
variable that the JVM may pick up options from.
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
Relates #101805
* Use child loggers broadly in PolicyManager
* Pass ModuleEntitlements to notEntitled
* Store logger name instead of object.
Some of our unit tests check for equality of `ModuleEntitlements` objects,
and they are entitled to do so (no pun intended).
* Alright, let's cache 'em
* Memoize at point of creation, not point of use
* Explanatory comments
Fix the benchmark for `EVAL` which was failing because of a strange
logging error. The benchmarks really didn't want to run when we use
commons-logging. That's fine - we can use the ES logging facade thing. I
also added a test to the benchmarks which should run the self-tests for
`EVAL` on `gradle check`.
By default Lucene only uses vectorization up to the JDK version
available at the time of its release. When testing against EA versions,
we want to still use vectorization. This commit adds a sysprop to force
lucene to use the current version for vectorization.
The ingest-attachment module uses Tika to parse some content; Tika in turn uses some libraries from java.desktop to perform its tasks.
In turn, the JDK loads one (or more) native libraries for its implementation as part of class initialization. This means we need to grant load_native_libraries to java.desktop so that because AWT can load libraries for itself.
I see that planning the ExchangeSinkExec takes a few milliseconds when
benchmarking simple queries with 10K fields. It spends time checking if
we need to realign the incoming pages. However, the exchange has the
exact same attributes as its child, so the incoming layout should match
its attributes perfectly. This change removes the realignment.
This PR adds a new MetadataDeleteDataStreamService that allows us to delete system data streams prior to a restore operation. This fixes a bug where system data streams were previously un-restorable.
While that was an approved breaking change, the change should have ensured v8
compatibility, so that under v8 compatibility mode the parameter is still accepted,
but rejected in v9 compatibility mode (default).
This commit ensures compatibility and adds a test for it.
Frozen indices, the freeze index API and the private index.frozen setting have been removed with #120539.
There is also a search throttled thread pool that can now be removed, as well as a private search.throttled index settings that is no longer used as it could only be set internally by freezing an index.
While the index setting is private and can be removed, as it should no longer be present in any index on 9.0+ indices, the thread pool settings associated to the removed pool are still accepted as no-op in case users have customized them and are upgrading without removing these. These will also trigger a deprecating warning.
This change also removes the search.throttled related output from the thread pool section of the cluster info API.
This refactor improves memory efficiency by processing inference requests in batches, capped by a max input length.
Changes include:
- A new dynamic operator setting to control the maximum batch size in bytes.
- Dropping input data from inference responses when the legacy semantic text format isn’t used, saving memory.
- Clearing inference results dynamically after each bulk item to free up memory sooner.
This is a step toward enabling circuit breakers to better handle memory usage when dealing with large inputs.
Introduce `Property.IndexSettingDeprecatedInV9AndRemovedInV10` property setting to mark
index settings that accepted on 9.x indices, but not on 10.0.
See #80574, #90264
See ES-11224
This PR adds the ability to override entitlement policies for Elasticsearch plugins and modules via a system property.
The system property is in the form es.entitlements.policy.<plugin name>, and accepts a versioned policy:
versions:
- version1
- versionN
policy:
<a standard entitlement policy>
For example:
versions:
- 9.1.0
policy:
ALL-UNNAMED:
- set_https_connection_properties
- outbound_network
- files:
- relative_path: ".config/gcloud"
relative_to: home
mode: read
The versioned policy needs to be base64 encoded. For example, to pass the above policy to a test cluster via gradle run:
./gradlew run --debug-jvm -Dtests.jvm.argline="-Des.entitlements.policy.repository-gcs=dmVyc2lvbnM6CiAgLSA5LjEuMApwb2xpY3k6CiAgQUxMLVVOTkFNRUQ6CiAgICAtIHNldF9odHRwc19jb25uZWN0aW9uX3Byb3BlcnRpZXMKICAgIC0gb3V0Ym91bmRfbmV0d29yawogICAgLSBmaWxlczoKICAgICAgLSByZWxhdGl2ZV9wYXRoOiAiLmNvbmZpZy9nY2xvdWQiCiAgICAgICAgcmVsYXRpdmVfdG86IGhvbWUKICAgICAgICBtb2RlOiByZWFkCg=="
The versions listed in the policy are string-matched against Build.version().current(); it is possible to specify any number of versions. If the list is empty/there is no versions field, the policy is assumed to match any Elasticsearch versions.
The override policy specified for any given plugin replaces the embedded policy for that plugin. See how EntitlementsAllowedViaOverrideIT replaces an empty policy for the entitlement-test-plugin with a policy that allows load_native_libraries and access to files in the test read_dir.
Also tested manually with an override with a different version, with an override with an invalid policy and with a valid override (see command above).
Relates to ES-11009
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
Relates #101805
We can release resources earlier by releasing before responding to the
listener (everything that needs retaining is retained via the search
response already) as well as make the GCs life a little easier and get
obvious correctness by using the listener that nulls out resources in a
thread-safe manner intead of a non-thread-safe and mutable list shared
across all kinds of places in the code.
Removes deprecated routes and API parameters.
The old 7.x routes for text file structure finding _text_structure/find_file_structure have been deprecated since 8 and replaced with _text_structure/find_structure. Also this PR simplifies the class hierarchy of GetDatafeedsStatsAction.
Non-issue as the deprecation and removal in 9 was long signaled.
It is confusing to readers to report `Channel is already closed` in
reaction to a double-response bug, and this may be interpreted as a
networking issue. We're not really closing anything here, and it's a
definite logic bug to call `sendResponse()` twice, so this commit
clarifies the actual problem in the exception message.
Relates ES-10996
Make `TO_LOWER` and `TO_UPPER` process all values it received.
This is quite large because it borrows a lot of code from the regular
evaluator generator to generate conversions so we can use the Locale.
That change propagates to the order of some parameters and to the
`toString` and a few more places.
Closes#124002
Release each hit's aggregations before moving on to the next hit and unlink it from the shard result even earlier.
Also, do the aggregation-reduce earlier in the reduce steps to reduce average heap use over time.
To that effect, do not do the reduction in the search phase controller. This has the added benefit of removing any need for a fake aggs-reduce-context in scroll.
To understand query performance, we often peruse the output of
`_query`-requests run with `"profile": true`.
This is difficult when the query runs in a large cluster with many nodes
and shards, or in case of CCQ.
This adds an option to visualize a query using Chromium's/Chrome's
builtin `about:tracing` - or, for even better visuals and querying the
different drivers via SQL, perfetto (c.f. https://ui.perfetto.dev/).
To use, save the JSON output of a query run with `"profile": true` to a
file, like `output.json` and then invoke the following Gradle task:
```
./gradlew x-pack:plugin:esql:tools:parseProfile --args='~/output.json ~/parsed_profile.json'
```
Either open `about:tracing` in Chromium/Chrome
Or head over to https://ui.perfetto.dev (build locally in case of
potentially sensitive data in the profille):
Every slice is a driver, the colors indicating the ratio of cpu time
over total time. - In Perfetto, essentials like duration, cpu duration,
timestamp and a few others can be queried via SQL - this allows e.g.
querying for all drivers that spent more than 50% of their time waiting
and other fun things.
- Details about a driver, esp. which operators it ran, are available when clicking the driver's slice.
This adds a new parameter to the quantized index mapping that allows
default oversampling and rescoring to occur.
This doesn't adjust any of the defaults. It allows it to be configured.
When the user provides `rescore_vector: {oversample: <number>}` in the
query it will overwrite it.
For example, here is how to use it with bbq:
```
PUT rescored_bbq
{
"mappings": {
"properties": {
"vector": {
"type": "dense_vector",
"index_options": {
"type": "bbq_hnsw",
"rescore_vector": {"oversample": 3.0}
}
}
}
}
}
```
Then, when querying, it will auto oversample the `k` by `3x` and rerank
with the raw vectors.
```
POST _search
{
"knn": {
"query_vector": [...],
"field": "vector"
}
}
```
Building on the work started in https://github.com/elastic/elasticsearch/pull/123904, we now want to auto-generate most of the small subfiles from the ES|QL functions unit tests.
This work also investigates any remaining discrepancies between the original asciidoc version and the new markdown, and tries to minimize differences so the docs do not look too different.
The kibana json and markdown files are moved to a new location, and the operator docs are a little more generated than before (although still largely manual).
* add max number of chunks
* wire merge function
* implement sparse merge function
* move tests to correct package/file
* float merge function
* bytes merge function
* more accurate byte average
* spotless
* Fix/improve EmbeddingRequestChunkerTests
* Remove TODO
* remove unnecessary field
* remove Chunk generic
* add TODO
* Remove specialized chunks
* add comment
* Update docs/changelog/123150.yaml
* update changelog
In #112380 we changed this `assert` to yield a `String` on failure
rather than the original `ElasticsearchException`, which means we don't
see the original completion's stack trace any more. This commit
reinstates the lost stack trace.
Just a random thing I noticed, this seemingly overlooked when
porting to the new rewrite API. No need to create a new searcher, we
already have one here.
This PR combines both cluster and project tasks under persistent_tasks
for XContent output of Metadata when it contains only a single project,
i.e. there will be no cluster_persistent_tasks in such output. This is
to maintain the existing output format when the cluster is not
multi-project enabled.
Relates: MP-1945
Checks the local cluster state after marking a shard snapshot as
complete, and logs a message if the completion is not reflected in this
cluster state.
In case when file with `.attach_pid` in name was stored in distribution
and then deleted, the exception could stop copying/linking files
without any sign of issue. The files were then missing in the cluster
used in the test causing them sometimes to fail (depending on which
files haven't been copied).
When using `Files.walk` it is impossible to catch the IOException and
continue walking through files conditionally. It has been replaced with
FileVisitor implementation to be able to continue if the exception is
caused by files left temporarily by JVM but no longer available.
This adds the file extentions for the blobs we request when populating the
cache.
The possible values for lucene extensions are around 50 and we use a special
"other" category for everything else, as a fallback.
This creates a `pragma` you can use to request that fields load from a
stored field rather than doc values. It implements that pragma for
`keyword` and number fields.
We expect that, for some disk configuration and some number of fields,
that it's faster to load those fields from _source or stored fields than
it is to use doc values. Our default is doc values and on my laptop it's
*always* faster to use doc values. But we don't ship my laptop to every
cluster.
This will let us experiment and debug slow queries by trying to load
fields a different way.
You access this pragma with:
```
curl -HContent-Type:application/json -XPOST localhost:9200/_query?pretty -d '{
"query": "FROM foo",
"pragma": {
"field_extract_preference": "STORED"
}
}'
```
On a release build you'll need to add `"accept_pragma_risks": true`.
This PR replaces the default project ID with the actual project while
waiting for active shards after index auto-creation similar to how wait
is done for explicit index creation.
* SEARCH-802 - bug fixed - Query rules allows for creation of rules with invalid match criteria
* [CI] Auto commit changes from spotless
* Worked on the comments given in the PR
* [CI] Auto commit changes from spotless
* Fixed Integration tests
* [CI] Auto commit changes from spotless
* Made changes from the PR
* Update docs/changelog/122823.yaml
* [CI] Auto commit changes from spotless
* Fixed the duplicate code issue in queryRuleTests
* Refactored code to clean it up based on PR comments
* [CI] Auto commit changes from spotless
* Logger statements were removed
* Cleaned up the QueryRule tests
* [CI] Auto commit changes from spotless
* Update x-pack/plugin/ent-search/src/test/java/org/elasticsearch/xpack/application/EnterpriseSearchModuleTestUtils.java
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co>
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
This is intended to be used to efficiently calculate a write load
metric for use by the auto-sharding algorithm which favours more
recent loads.
ES-10037 #comment Core algorithm added in https://github.com/elastic/elasticsearch/pull/124507
When enabling FIPS `javax.net.ssl.trustStore` is not necessarily set.
This change adds FIPS entitlements based on
`org.bouncycastle.fips.approved_only=true`, which enforces usage of FIPS
approved functionality only.
Additionally, this PR grants read access to a custom trust store if
provided via `javax.net.ssl.trustStore`, otherwise read access to the
default JDK trust store is granted.
Relates to ES-11025.
Writing tests for #123861, turns out that #124195 is not enough.
We really need new IT test cases for "always allowed" actions: in order to be sure they are allowed, we need to setup the plugin with no policy.
This PR adds test cases for that, plus the support for writing test functions that accept one Environment parameter: many test paths we test and allow/deny are relative to paths in Environment, so it's useful to have access to it (see readAccessConfigDirectory as an example)
* [Profiling] Fix NullPointerExceptions by accepting dotted field names
Profiling uses synthetic source and thus expects nested field names in query responses.
With 8.17+, synthetic source is available only to Enterprise (or higher) subscriptions,
so that smaller subscriptions have dotted field names in query responses.
The profiling plugin relies on nested field names and runs into NullPointerExceptions
if these are not found.
This PR fixes the NullPointerExceptions that could happen with dotted field names.
Signed-off-by: Tim Rühsen <tim.ruhsen@elastic.co>
* Evaluate source only once (cleanup)
---------
Signed-off-by: Tim Rühsen <tim.ruhsen@elastic.co>
* Inline cast to date
* Update docs/changelog/123460.yaml
* New capability for `::date` casting
* More tests
* Update tests
---------
Co-authored-by: Fang Xing <155562079+fang-xing-esql@users.noreply.github.com>
* Refactor: findRequestingFrame
* INFO instead of WARN for NotEntitledException.
Some of these are expected, so an INFO seems more appropriate.
The stack trace tends to attract attention even when entitlements are not the
cause of a problem, so let's avoid the stack trace, but still include stack
frame info from the frame of interest.
* Use child loggers for Not Entitled logs
* Use warn, and include compoenent name
* Fix ALL_UNNAMED
* Mute entitlement warnings from repositories
* PR feedback
* Common out the Not Entitled prefix.
We're alerting on this, so let's not rely on every caller of notEntitled to remember it.
This commit fix the IronBank hardening_manifest CI test that expect the
hardening_manifest.yaml values to match `'^(?!\\s*$)[ -~|]*$'` regex:
```
| hardening_manifest | ERROR | Hardening Manifest failed jsonschema validation
| hardening_manifest | ERROR | Verify Hardening Manifest content
| hardening_manifest | ERROR | 'Rene Gröschke' does not match '^(?!\\s*$)[ -~|]*$'
```
43219286
cc @breskeby
<!-- Thank you for your interest in and contributing to Elasticsearch!
There are a few simple things to check before submitting your pull
request that can help with the review process. You should delete these
items from your submission, but they are here to help bring them to your
attention. -->
- Have you signed the [contributor license agreement](https://www.elastic.co/contributor-agreement)?
- Have you followed the [contributor guidelines](https://github.com/elastic/elasticsearch/blob/main/CONTRIBUTING.md)?
- If submitting code, have you built your formula locally prior to submission with `gradle check`?
- If submitting code, is your pull request against main? Unless there is a good reason otherwise, we prefer pull requests against main and will backport as needed.
- If submitting code, have you checked that your submission is for an [OS and architecture that we support](https://www.elastic.co/support/matrix#show_os)?
- If you are submitting this code for a class then read our [policy](https://github.com/elastic/elasticsearch/blob/main/CONTRIBUTING.md#contributing-as-part-of-a-class) for that.
Fixes https://github.com/elastic/elasticsearch/issues/123067
Just like WildcardLike and RLike, some functions can be converted to Lucene queries. Here it's those two, which are nearly identical to WildcardLike
This, like some other functions, needs a FoldContext. I'm using the static method for this here, but it's fixed in https://github.com/elastic/elasticsearch/pull/123398, which I kept separated as it changes many files
IntelliJ has been suggesting switching to string templates (at least
it has for me). This is unhelpful advice, since the feature was only
ever a tech preview and was removed in more recent JVMs. This change
disables that inspection.
Resolves#123053
This adds the thread name to the driver sleep profile output.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
By indexing the test data set in multiple bulk requests, which ensure there are multiple segments, which improves testing coverage for DownsampleShardIndexer.
We don't catch more complex DownsampleShardIndexer issues if we end up with just one segment. Hence, we need to make the DownsampleShardIndexer test suite a little bit more evil by ensuring we have more than one segment.
This will rethrow the `NoSuchFileException` when encountering an invalid
symbolic link when following links during file (read) entitlement
checks.
Relates to https://github.com/elastic/elasticsearch/pull/124133
(ES-11019)
Today snapshot finalization does nontrivial work on the calling thread
(often the cluster applier thread) and also in theory may fork back to
the cluster applier thread in `getRepositoryData`, yet it always forks
at least one task (the `SnapshotInfo` write) to the `SNAPSHOT` pool
anyway. With this change we fork to the `SNAPSHOT` pool straight away
and then make sure to stay on this pool throughout.
This PR moves the helper methods up to the base ESRestTestCase class so
that they can be reused by other subclasses, e.g. the ones on the
serverless side.
Relates: ES-10292
* Introduce IndexReshardingMetadata
This adds to IndexMetadata the persistent state
we will need to track while a split is in progress.
Nothing outside test code sets it yet, so it doesn't
introduce any wire changes yet.
Followups will consult this to make routing decisions and
handle backward compatibility if the object is present
in metadata.
* ESQL: Lazy collection copying during node transform
A set of optimization for tree traversal:
1. perform lazy copying during children transform
2. use long hashing to avoid object creation
3. perform type check first before collection checking
Relates #124395
This PR makes a change to the existing Java REST test DetectionRulesIT.testCondition such that it updates detection rules for a running job. Previously it had relied on closing and re-opening the job for the update to take effect.
Relates elastic/ml-cpp#2821
Inter-segment concurrency is disabled whenever sort by field, included script sorting, is used in a search request.
The reason why sort by field does not use concurrency is that there are some performance implications, given that the hit queue in Lucene is build per slice and the different search threads don't share information about the documents they have already visited etc.
The reason why script sort has concurrency disabled is that the script sorting implementation is not thread safe. This commit addresses such concurrency issue and re-enables search concurrency for search requests that use script sorting. In addition, missing tests are added to cover for sort scripts that rely on _score being available and top_hits aggregation with a scripted sort clause.
The old lucene versions plugin allows users to read indices created by ancient
Elasticsearch version, starting from 5.0. Especially for 5.x which relied on Lucene
6.x, some special logic is required around postings format support. That revolves
around reading of FSTs, but has a consequence of requiring quite a few fork of
other Lucene classes due to their visibility.
This commit attempts to add javadocs to clarify the intent of some of these classes.
It also includes some simplifications, in that Lucene50PostingsReader can be reused as-is
and some other classes are only needed in tests hence are moved to the test folder.
The test that is verifying timeout handling when pulling a scorer supplier is
going to always score the entire segment. The test needs to be adjusted
accordingly. While at it, I added docs and clarified the tests a bit, as well
as added a couple new tests that cover for timeout handling when retrieving a
scorer from the scorer supplier.
Closes#124140
* [CI] Auto commit changes from spotless
* iter
* iter
* spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Currently, we use NamedWriteable for serializing blocks. While
convenient, it incurs a noticeable performance penalty when pages
contain thousands of blocks. Since block types are small and already
centered in ElementType, we can safely switch from NamedWriteable to
typed code. For example, the NamedWriteable alone of a small page with
10K fields would be 180KB, whereas the new method reduces it to 10KB.
Below are the serialization improvements with FROM idx | LIMIT 10000
where the target index has 10K fields:
- write_exchange_response executed 173 times took: 73.2ms -> 26.7ms
- read_exchange_response executed 173 times took: 49.4ms -> 25.8ms
This replaces the usage of a defaultDistribution for javaRestTest by the integTestDistribution.
This has a few advantages:
1. The overall dependencies on running the java rest tests are minimized.
By using the default distribution we rely on building the whole default distribution (including all modules and plugins)
before we can run these tests. This a) takes time and b) dramatically reduces the likelyhood of us avoiding test task execution at all as we
basically declare the whole distro as an input. By using integTest distro we reduce the surface of the inputs dramatically which also results in faster
execution of these tests
2. its more idiomatic as one pattern we see is that we e.g disable the security settings that we would not need once we use the integTest distro without
the security plugin
3. it makes test setup and its dependencies more explicit.
Point 3. might sound as like a downside at first, but for understanding what those tests are doing and what they are relying on I think its worth the 3 more lines of code.
Here are two build scans task executions:
- before the `javaRestTest` task requires `995 tasks, 2 transforms executed in 155 projects`: https://gradle-enterprise.elastic.co/s/drj5mfzsfx7ra/timeline
- after we only rely on `275 tasks, 2 transforms executed in 56 projects`: https://gradle-enterprise.elastic.co/s/jr5sblhppn4fg/timeline?page=2
This isn't even saving any lines of code and is a measurable
source of both cache and branch-prediction misses on the hot
and critical transport-thread path.
=> inline it
Read dimension values once per tsid/bucket docid range instead of for each document being processed.
The dimension value within a bucket-interval docid range is always to same and this avoids unnecessary reads.
Latency of downsampling the tsdb track index into a 1 hour interval downsample index drop by ~16% (running on my local machine).
The cluster's min transport version has some useful properties but also
some subtle exceptions that are today not documented in the code. This
commit adds some Javadocs in this area.
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
Relates #101805
We can just use a real constant for the `null` case, avoiding any
non-plain stores in all cases. This should be somewhat helpful for the
security interceptors.
Avoiding some indirection, volatile-reads and moving the listener
functionality that needlessly kept iterating an empty CoW list (creating
iterator instances, volatile reads, more code) in an effort to improve
the low IPC on transport threads.
Even if the contract for JDK methods using the SecurityManager states that the exception throw is of type SecurityException, many libraries (including our own, apparently!) violates that and use the type actually thrown by SecurityManager, AccessControlException. A prime example is the GCS/CSP libraries.
In order to maintain compatibility for them, we need to inherit from the more specific AccessControlException; this is less desirable, as AccessControlException is marked as deprecated for removal alongside the other SecurityManager classes, but we discussed and found this is the best short term solution.
More work will be needed -- again, this is a short term solution.
Replaces #123984
* Added check for blank string to skip generating embeddings with unit test
* Adding yaml tests for skipping embedding generation
* dynamic update not required if model_settings stays null
* Updating node feature for handling empty input name and description
* Update yaml tests with refresh=true
* Update unit test to follow more accurate behavior
* Added yaml tests for multu chunks
* [CI] Auto commit changes from spotless
* Adding highlighter yaml tests for empty input
* Update docs/changelog/123763.yaml
* Update changelog and test reason to have more polished documentation
* adding input value into the response source and fixing unit tests by reformating
* Adding highligher test for backward compatibility and refactor existing test
* Added bwc tests for empty input and multi chunks
* Removed reindex for empty input from bwc
* [CI] Auto commit changes from spotless
* Fixing yaml test
* Update unit tests helper function to support both format
* [CI] Auto commit changes from spotless
* Adding cluster features for bwc
* Centralize logic for assertInference helper
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
There's no need to have a `SnapshotInfo` consumer to run at the end of
finalization, we only pass it the value we already calculated earlier.
This replaces it with a bare `Runnable` instead.
An issue with `Strings#collectionToDelimitedStringWithLimit` is that you
need to collect all the items together up front first, even if you're
going to throw most of them away. This commit introduces
`BoundedDelimitedStringCollector` which allows to accumulate the items
one-at-a-time instead.
No need to implement the safe wait on the latch by hand here, we can use
the other `safeAwait` variant that takes a timeout. Also adds a comment
discouraging its use elsewhere without good reason.
In the create-index-from-source action, we should set the cause of the create index request so that it is clear in the logs. Without setting the cause on the request, the default value of api is used.
These parameters are confusing and almost entirely unused (apart from
one easily-adjusted test, two production locations where they are used
incorrectly, and one production location that can be adjusted not to
need them). This commit just removes them.
Adds options to QSTR function.
#118619 added named function parameters. This PR uses this mechanism for allowing query string function parameters, so query string parameters can be used in ES|QL.
Closes#120933
Depending on whether a message is critical or warning a message should
indicate to check breaking changes documentation (critical level) or
deprecation changes documentation (warn level)
relates #79666
Our check methods injected by the instrumenter receive "this" as the second parameter.
For internal classes like AbstractDelegateHttpsURLConnection we generally use a base type; in this case we were using javax.net.ssl.HttpsURLConnection, which is incorrect as AbstractDelegateHttpsURLConnection derives from java.net.HttpURLConnection.
This was not failing in our tests because we don't actually use that parameter in that check function.
Also, it was not failing on transform, just on retransformClasses, and only in JDK 24. Apparently, JDK 24 introduced new validation there (to be confirmed).
And it was failing just on cloud as the APM agent there (which is loaded before our agent) connects to a https endpoint - our IT tests, and ./gradlew run --with-apm-server, use a http endpoint. Using https makes the JVM load AbstractDelegateHttpsURLConnection, making it one of the classes we need to retransform, triggering the VerifyError.
The main goal of the PR is to fix the "rollover failed" log in the
`IndexTemplateRegistry`. We were logging this on every upgrade because
`rolloverResponse.isRolledOver()` is always `false` when we do a lazy
rollover request - which is the default for index template updates.
While I was looking at this, I noticed we don't actually return early
when we process a lazy rollover _dry run_. The end result happened to be
ok (i.e. side-effects and response), but I figured I might as well fix
the behavior to return early.
Finally, we add an `INFO` log in the lazy rollover action to aid the
debugging process.
This migrates a number of calls of the form
metadata.getProject().getIndexSafe(index)
or
metadata.getProject().index(index)
to
metadata.indexMetadata(index)
or
metadata.findIndex(index)
The method should be called with an explicit project-id to access
persistent tasks from the right project. This PR does that. The
callsites are updated by using the default project-id for the timebeing.
They work for single project deployments but should eventually be
updated for multi-project setup.
Relates: ES-11039
No reason to blow up the size of `AbstractSearchAsyncAction` (and the code size of some methods that really only need the size of that collection) needlessly. Just keep the count, that's all we need.
We can build the skipped shard list inline if need be (and do so in a cheaper way because we can build the search targets.
When reindexing a data stream, the ILM metadata is copied from the index metadata of the source index to the destination index. But the ILM state of the new index can be stuck if the source index was in an AsyncAction at the time of reindexing. To un-stick the new index, we call TransportRetryAction to retry the AsyncAction. In the past this action would only run if the index were in the error phase. This change includes an update to TransportRetryAction, which allows it to be run when the index is not in an error phase, if the parameter requireError is set to false.
Makes the execution and use of enrich policies project-aware.
Note: this does not make the enrich cache project-aware. That is to be
handled in a follow-up PR.
By keeping a list of all the rollover results in a rollover request
batch, we were keeping references to all the intermediate cluster states
that we built. We've seen this list take up ~1.4GB with 600 rollover
requests in one batch.
We only kept the list of results to compute the "reason" for the
allocation reroute, so we can easily drop the cluster state reference
from the list and only keep what we need.
Fixes#123893
* Entitle inference to access AWS credentials.
Copied from the security policy.
* more missing entitlements for the inference plugin
---------
Co-authored-by: Lorenzo Dematte <lorenzo.dematte@elastic.co>
This commit improves memory accounting by incorporating document
expansion during shard bulk execution. Additionally, it introduces a new
limit on the maximum document size, which defaults to 5% of the
available heap.
This limit can be configured using the new setting:
indexing_pressure.memory.max_operation_size
These changes help prevent excessive memory consumption and
improve indexing stability.
Closes ES-10777
TransportHandshakerTests had a workaround for multi-project's fake
transport version. Now that it's a real transport version we can remove
the workaround
If all items fail to delete, the current warning log builds a huge
string to include all of them (up to 1000). This PR limits the string
length to first 10 entries.
Relates: #123630
JNA has a static thread which handles cleaning up native memory
references. This commit adds the thread name to those filtered out of
thread leak detection since it lives for the lifetime of the JDK (yet
might be started in the middle of a test).
closes#114555
This change introduces a cluster setting
`esql.query.allow_partial_results` that allows enabling or disabling
allow_partial_results in ES|QL at the cluster-wide level. Initially,
this setting defaults to false, but it will be switched to true soon.
The reason for not changing the default in this PR is that it requires
adjusting many tests, which would make the PR too large. Instead, we
will adjust the tests incrementally and switch the default when the
tests are ready. This cluster setting is useful for falling back to the
previous behavior (i.e., disabling allow_partial_results) if users
upgrade to the new version and haven't updated their queries.
Also, the default setting can be overridden on a per-request basis via a
URL parameter (allow_partial_results) (changed from request body to URL
parameter to conform to the proposal).
Relates #122802
On index creation, its possible to configure an hunspell analyzer, but
reference a locale file that actually doesn't exist or isn't accessible.
This error, like our other user dictionary errors, should be an IAE not
an ISE.
closes: https://github.com/elastic/elasticsearch/issues/123729
The 7.x routes for ml trained models _ml/inference/ have been deprecated
since 8 and replaced with _ml/trained_models. Also removes query
parameters that are no longer supported.
This feature flag controls whether synthetic recovery source is enabled by default when the source mode is synthetic.
The synthetic recovery source feature itself is already available via the index.recovery.use_synthetic_source index setting and can be enabled by anyone using synthetic source.
The default value of index.recovery.use_synthetic_source setting defaults to true when index.mapping.source.mode is enabled. The index.mapping.source.mode default to true if index.mode is logsdb or time_series.
In other words, with this change synthetic recovery source will be enabled by default for logsdb and tsdb.
Closes#116726
Fixes#122661.
The issue was caused by RandomIndexWriter (randomly) reshuffling the document writing order. Since this test also ensures that the documents are read in the input order, I've opted to use a regular IndexWriter instead. I've also renamed the class to AbstractShapeGeometryFieldMapperTests since it was originally renamed due to a misunderstanding of muted tests (which caused it to be muted again! Busted 😅).
The only public constructor needed in the archive index codecs is the default one,
as that's called by SPI. The other one will only ever be called by the default one,
hence we can merge the two into one and simplify things a bit.
Reduce boilerplate associated with creating `FieldCapabilities` instances.
Since it's a class with a huge number of fields, it makes sense to define a builder object, as that can also help with all the Boolean and null blindness going on.
Note while there is a static Builder class in `FieldCapabilities`, it is not a proper builder object (no setters, still need to pass a lot of otherwise default parameters) and also package-private. To avoid changing that, I defined a new `FieldCapabilitiesBuilder` class. I also went over the code and refactored places which used the old constructor.
We have some incredibly deep callstacks in security that seem to visibly
raise context switch costs, make profiling more complicated and
generally make the code rather hard to follow.
Since the methods adjusted here return a result synchronously we can
both save overhead and make things a little easier to follow by using
promises as returns in place of consuming callbacks.
No need to create new instances for the default project-id. We can use
the singleton field which should speeds up key comparsion for Map.get
operations.
Relates: #123662
Instead of implicitly relying on the `IngestService` to resolve the
project ID, the simulate pipeline request should do this and pass the
project ID explicitly.
Use the same exception class in more places where single lookup fails
due to the existence of multiple projects. This allows the exception to
be handled similarly, e.g. ignored in
ClusterApplierService#callClusterStateListener
the last commit before the lucene update is not compatible in how we
apply the buildscan plugin these days. We now instead created a branch
"combat-lucene-10-0-0" where we added a compatibibility fix. Now we pick
the fix commit instead.
We mostly need the map for lookups in very tight loops but also rarely rely on the sortedness of this thing. Getting the best of both worlds at surprisingly only a ~10% increase in build time and ~25% increase in the heap consumption of this structure provides a massive speedup to e.g. search or field_caps over large numbers of indices thanks to the up to an order of magnitude cheaper lookups in index name resolution and security (as well as speedups in other areas).
* [ML] Use latest results index for new Anomaly Detection jobs (#122597)
After upgrading from v7 new anomaly detection jobs should
use the latest results index if one has been created.
* Delete 8.x test
This commit adds a new coordinating object to reflect reduce the need to
introduce a new releasable for every network byte reference.
Additionally, it will allow us to more specifically track bytes while
they are buffered.
When running a timestamp range query, as an optimization we check if the
query range overlaps with the total range of values within a shard before
executing the query on that shard. That way, if the range is disjoint, we
can skip execution for that shard. To get the range of values within a
shard, we usually use the PointValues index on the shard. However, when the
doc values skipper is enabled, the point values are not (as the reason for
the skipper is to reduce storage overhead by removing the point values
index). In this case, we need to instead get the range of values within the
shard by using the skipper. This patch implements that logic.
Follow-up to #123191
This PR contains the following updates:
| Package | Update | Change | |---|---|---| |
docker.elastic.co/wolfi/chainguard-base | digest | `c66fdaf` ->
`15a4191` |
---
### Configuration
📅 **Schedule**: Branch creation - "after 1pm on tuesday" (UTC),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once
you are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box
---
This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyI6RGVsaXZlcnkvUGFja2FnaW5nIiwiPm5vbi1pc3N1ZSIsIlRlYW06RGVsaXZlcnkiLCJhdXRvLW1lcmdlLXdpdGhvdXQtYXBwcm92YWwiXX0=-->
This change removes the skipping introduced for the semantic_text tests
in #122222. These tests were accidentally re-enabled without failures
over the last two weeks, so we can now re-enable them.
Closes#121411
Relates #122222
We now always retry based on the provider's configured retry logic
rather than the HTTP status code. Some providers (e.g. Cohere,
Anthropic) will return 200 status codes with error bodies, others (e.g.
OpenAI, Azure) will return non-200 status codes with non-streaming
bodies.
Notes:
- Refactored from HttpResult to StreamingHttpResult, the byte body is
now the streaming element while the http response lives outside the
stream.
- Refactored StreamingHttpResultPublisher so that it only pushes byte
body into a queue.
- Tests all now have to wait for the response to be fully consumed
before closing the service, otherwise the close method will shut down
the mock web server and apache will throw an error.
This PR adjusts the list of supported ciphers to reflect ciphers
available in JDK 24.
JDK 24 [drops](https://bugs.openjdk.org/browse/JDK-8245545) support for
`TLS_RSA` suites. These ciphers will no longer be supported in
Elasticsearch with a bundled JDK with version >= 24. JDK's of lower
versions will continue to support to dropped ciphers.
I will follow up this PR with a separate docs PR.
This PR adds the `read_failure_store` index privilege and extends the role building logic to support selector-aware authorization. Note that this PR only concerns building roles; the actual authorization logic implementation will follow in a separate PR. The overall approach is based on the proof-of-concept PR (https://github.com/elastic/elasticsearch/pull/122007).
The purpose of the `read_failure_store` index privilege is to support granting selective access to the failure store of a data stream via the `::failures` selector in search and related APIs. A role with the `read_failure_store` index privilege grants access to the failure store, without granting access to the data in a data stream. Conversely, the existing `read` privilege only grants access to data and _not_ the failure store.
This requires changes to both the role building logic, and authorization. Going forward, each named index privilege is assigned a selector it grants access to, e.g., `read` grants access to the implicit `::data` selector, `read_failure_store` grants access to `::failures`. When building a role from role descriptors, we partition its underlying index groups by selector access such that any given group grants access to a single selector (with the exception of `all`, which grants access to all selectors).
This PR implements this partitioning logic and sets up roles to implement selector-aware authorization in a follow up. Note that parts of the code make assumption around the existence of only two distinct selectors (`::data` and `::failures`) to simplify the implementation; however, it's possible to generalize these sections to support more selectors in the future, if necessary.
Closes: ES-10872
This change adds a new rolling upgrade test that checks having certain deprecated
index settings present in an N-2 index doesn't prevent the index from being opened
in the current version. The deprecated settings specifically tested here are:
* index.mapper.dynamic
* index.max_adjacency_matrix_filters
* index.force_memory_term_dictionary
* index.soft_deletes.enabled
In backporting #123397 to 8.x we discovered that we needed to generalize
`TransportVersion#bestKnownVersion` slightly to handle older versions.
This commit forward-ports this change to keep the branches aligned.
Fixes misleading assertion configuration for test tasks from the command
line.
We support the following command line args:
1. `-Dtests.jvm.argline=-disableassertions`
2. `-Dtests.jvm.argline=-da`
3. `-Dtests.asserts=false`
The stable API compatibility checks ensure public jars don't change
their APIs in incompatible ways. Yet the logging jar has an internal
package which can change arbitrarily. This commit adds a temporary
workaround to skip the internal logging package until the compat check
can be made to look at module-info.
If the final driver is woken up before the exchange buffer is finished,
there will be two sleeps in its profile instead of one. We don't need
these assertions to be so precise. This change relaxes the assertions to
allow reduce operators to have more than one sleep.
The lack of context here left me at a dead end. I didn't RTFM otherwise
I would have discovered #56899 added this info there, but for the sake
of future code-readers I'm adding the same info here.
This PR adds explicit import layout for io.* packages so that their
import orders are consistent between IDE and CLI. Without it, the IDE
places the io.* imports at the bottom of the import list.
See AbstractRepositoryS3RestTestCase for such an example.
When you load stored fields from lucene you have get all of your ducks
in a row first or it'll be really slow. You need to get a list of all of
the fields you want so you can ignore the ones that you don't. We do
this with the `StoredFieldsSpec` which is immutable and has a `merge`
method. It's quite common to merge a few dozen of these specs together
to prepare for the fetch phase or for a new segment when loading fields
from ESQL.
When I was loading thousands of fields in ESQL I noticed that the merge
was slowing things down marginally. This skips a big chunk of the merge
in the common case that we don't have to load any named stored fields.
* Refactoring
* Add internal cluster setting to aid testing
* [CI] Auto commit changes from spotless
* Allowing the auth interval to be configurable via a setting
* Removing unused code
* Adding revocation functionality back
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* Adding embeddings type to Jina AI service settings
* Update docs/changelog/121548.yaml
* Setting default similarity to L2 norm for binary embedding type
If the parent task is canceled, registering the compute group task might
throw a TaskCancelledException, so the listener won't be notified.
Closes#123216Closes#123101Closes#123451
* Fix Gradle Deprecation warning as declaring an is- property with a Boolean type has been deprecated.
* Make use of new layout.settingsFolder api to address some cross project references
* Fix buildParams snapshot check for multiprojet projects
Leader and follower checks are expected to run regularly for any
multi-node clusters. This PR allows them to be observed in the list of
tasks during cleanup phase of REST tests.
* Remove @UpdateForV9 annotation in RuntimeVersionFeature
Re ES-10338 the Core/Infra team agreed that this is not blocked on security manager removal. We should go ahead and remove these 2 instances of the @UpdateForV9 annotation, the code will be cleaned up soon with the removal of Security Manager.
* Remove @UpdateForV9 annotation from SystemJvmOptions
re ES-10338 the Core/Infra team agreed this is not blocked on security manager removal. We should go ahead and remove this instance of the @UpdateForV9 annotation.
GeoIpDownloaderTaskExecutor needs to be made project aware eventually.
In the meantime, to unblock tests, this PR makes the stopping action
work explicitly for the default project only.
A follow-up to #123296 to address a potential block leak that may occur
when a circuit-breaking exception is triggered while truncating the docs
or scores blocks.
Relates #123296
All CLIs in elasticsearch support command line flags for controlling the
output level. When --silent is used, the expectation is that normal
logging is omitted. Yet the log4j logger is still configured to output
error level logs. This commit sets the appropriate log level for log4j
depending on the Terminal log level.
This change adds support for skipping shards with event.ingested fields
and metadata fields (_tier, _index). This should allow ES|QL to skip
unmatched shards and avoid sending requests to the data nodes.
Currently, we rely on signal propagation for early termination. For
example, FROM index | LIMIT 10 can be executed by multiple Drivers:
several Drivers to read document IDs and extract fields, and the final
Driver to select at most 10 rows. In this scenario, each Lucene Driver
can independently collect up to 10 rows until the final Driver has
enough rows and signals them to stop collecting. In most cases, this
model works fine, but when extracting fields from indices in the
warm/cold tier, it can impact performance. This change introduces a
Limiter used between LimitOperator and LuceneSourceOperator to avoid
over-collecting. We will also need a follow-up to ensure that we do not
over-collect between multiple stages of query execution.
Improves the information in the IndexShardSnapshotStatus's
statusDescription field to include the success/failure of the remote
call to the master node to update the shard snapshot state. This
allows us to see if there is a discrepancy between the state of the
data node and the master node.
Closes ES-10991
Previously we'd reject transport handshakes from numerically-older nodes
with a chronologically-newer release date. With this commit we instead
negotiate to use the latest known version for future communications.
This commit reworks the settings globs to be more useable. Primarily it
expands the values so that the settings may be lists, iterating over
each value. Additionally it simplifies the function to also allow
non-glob settings so that this single method may be used to lookup all
values for a given setting pattern, whether it contains a glob or not.
The SnapshotShutdownProgressTracker logs the shard snapshot statuses
individually and now each status can be associated back to the parent
snapshot.
Relates ES-10991
Fixes https://github.com/elastic/elasticsearch/issues/122967
When the Driver reported status, if the "report at least every X time" report was triggered, it was re-adding the same iterations and cpuTime again, as it wasn't clearing it before the next iteration.
Now, there are separated variables for the last updated iterations and reported time.
The setting based paths could be either absolute or relative, and they
are always relative to the config dir. This commit renames the
path_setting to make it clear it is related to config, and removes the
relative variant.
`TransportFieldCapabilitiesAction.doExecuteForked` would resolve the
project twice:
- Once implicitly when resolving indices from cluster state
- Another time to resolve index blocks
This changes this method to resolve `ProjectState` once and use that
instance for both index resolution and block checking
We already disable inter-segment concurrency in SearchSourceBuilder whenever
the top-level sort provided is not _score. We shoudl apply the same rules
in top_hits. We recenly stumbled upon non deterministic behaviour caused by
script sorting defined within top hits. That is to be expected given that
script sorting does not support search concurrency.
The sort script can be replaced with a runtime field, either defined in the
mapping or in the search request, which does support concurrency and guarantees
predictable behaviour.
* [IDEA] Enable Gradle Configuration Cache for Gradle Runner
This should speedup repetitives usages of the Gradle Runner for the majority of test executions
We added a flag to disable it explicitly if it does not work for certain scenarios
This test was muted, got fixed, and then was muted again but the issue
wasn't reopened (by accident it seems). Since that was 4 years ago, I'm
going to unmute this test and if it's still an issue, the new test
automation will open a new issue for it using the current muting
mechanism.
Relates #70595
SSL file utils currently only handle security manager access control
exceptions around file read checks. This PR extends these to support
entitlement checks as well.
There is no easy way to unit test this since we can't run unit tests
with entitlements enabled (for now). The PR includes a REST test
instead.
Relates: https://github.com/elastic/elasticsearch/issues/121960
When IndicesService is closed, the pending deletion may still be in
progress due to indices removed before IndicesService gets closed. If
the deletion stucks for some reason, it can stall the node shutdown.
This PR aborts the pending deletion more promptly by not retry after
IndicesService is stopped.
Resolves: #121717Resolves: #121716Resolves: #122119
Bit of a random find. Looks like we're effectively spending something
like ~0.5% of the total CPU time on resolving virtual calls for these
methods. Inlining `IOUtils` reduces their size quite a bit, removes
dead-code for rethrowing `IOException` that is never actually thrown
and hopefully as a result improve compilation a little here.
Fixes https://github.com/elastic/elasticsearch/issues/123430
There were 2 problems here:
- We were filling a static field (used to auto-cast string literals) within a constructor, which is also called in multiple places
- The field was only filled with non-snapshot functions, so snapshot function auto-casting wasn't possible
Fixed both bugs by making the field non-static instead, and a fix to use the snapshot registry (if available) in the string casting rule.
This change adds a unit test to demonstrate a specific
behavior of the AWS SDKv1, which closes the InputStream
used to upload a blob only after the HTTP request has
been sent (this is to accomodate for retries). The SDK
then swallows any exception thrown when closing the
InputStream which has the effect to hide any potential
CorruptIndexException that could have been detected
at that time.
Relates ES-10931
There is no guarantee that a project has non-null persistent tasks [0].
Null check is already done in most places. This PR adds it in a few more
places.
[0] Since health-node is now a cluster-scoped persistent task, it has
become more likely for the project-scoped tasks to be null.
Relates: #123262
When #119968 was merged into multi-project we introduced a regression by
inserting a call to `.getProject()` within the `RoutingNodes` class that
was supposed to be multi-project-aware.
This commit replaces those calls with `.indexMetadata` lookups
This commit reverts the changes to
docs/internal/DistributedArchitectureGuide.md
These changes need to be redone in order to update the code links
so that they point to a more recent version that includes the
`Metadata.ClusterCustom` / `Metadata.ProjectCustom` split
We also need to incorporate documentation updates to reflect the
changes to PersistentTasks.
This adds an exclusive parameter for FilesEntitlement where a path can be made exclusive for a certain
module. Should two modules attempt to both specify the same path as exclusive an exception is
thrown.
We're catching MP exceptions when applying cluster state listeners to
avoid noise in the logs, but we shouldn't forget to remove this `catch`
block at some point in the future.
These methods will be removed at some point in the future. By marking
them as deprecated, it'll be easier to spot that some code is using
these methods. Additionally, this will hopefully prevent people from
using these temporary methods.
We can do with a whole lot less in ref-counting, avoiding lots of contention and speeding
up the logic in general by only incrementing ref-counts where ownership is unclear while
avoiding count changes on obvious "moves".
File entitlements which read a setting may actually contain urls. This
commit adds an optional `ignore_url` property for the entitlement to
skip any values which are urls.
The node weight changes between two balancer rounds are summarized by
saving the old DesiredBalance's weights per node along with a weights
diff to reach the new DesiredBalance's weights per node. This supports
combining multiple summaries by using the oldest summary's base node
weights and summing the diffs across all summaries to reach a combined
node weight diffs.
When these tests were run around midnight, the use of
`DataStream#getDefaultBackingIndexName` could result in a potential
mismatch in the generated index name and the one that the cluster
actually created. Instead, we need to obtain the backing index and
extract the desired index name from there.
Fixes#123086
Relates #123376
If all we want to know is whether the cluster has any indices, then
this new method is more efficient than using the existing
"getTotalNumberOfIndices" method
This PR is the follow-up work for MP-1945 and MP-1938 which laid the
foundation of two different scoped persistent tasks. It updates the
persistent task framework to be aware of the two task types so that it
can handle both cluster scope tasks and per-project tasks. Once these
changes are in place, we will make health-node to be the first
cluster-scope persistent task.
Relates: ES-10168
The NamedComponentReader reads a file created upon plugin installation
for stable plugins from the plugin installation dir. This commit passes
the plugins directory through to entitlements and grants server access.
Expressions create a classloader within Lucene, not the Elasticsearch
module. This commit fixes the policy to grant the entitlement to the
Lucene module.
This PR contains the following updates:
| Package | Update | Change | |---|---|---| |
docker.elastic.co/wolfi/chainguard-base | digest | `d74b1fd` ->
`c66fdaf` |
---
### Configuration
📅 **Schedule**: Branch creation - "after 1pm on tuesday" (UTC),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once
you are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box
---
This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyI6RGVsaXZlcnkvUGFja2FnaW5nIiwiPm5vbi1pc3N1ZSIsIlRlYW06RGVsaXZlcnkiLCJhdXRvLW1lcmdlLXdpdGhvdXQtYXBwcm92YWwiXX0=-->
We have two leaks reported in breaker tests, but we do not have their
traces. This PR integrates MockBlockFactory for breaker tests to help
identify these potential leaks.
Relates #122810
* [Fleet] Add upgrade_attemps to .fleet-agents index
* Change type to date
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Rather than checking the license (updating the usage map) on every
single shard, just do it once at the start of a computation that needs
to forecast write loads.
Closes#123247
The original work at https://github.com/elastic/elasticsearch/pull/106065 did not support geospatial types with this comment:
> I made this work for everything but geo_point and cartesian_point because I'm not 100% sure how to integrate with those. We can grab those in a follow up.
The geospatial types should be possible to collect using the VALUES aggregation with similar behavior to the `ST_COLLECT` OGC function, based on the Elasticsearch convention that treats multi-value geospatial fields as behaving similarly to any geometry collection. So this implementation is a trivial addition to the existing values types support.
* Consider entitlement lib as system module
Entitlements sometimes needs to perform sensitive operations,
particularly within the FileAccessTree. This commit expands the
trivially allowed check to include entitlements as one of the system
modules alongside the jdk. One consequence is that the self test must be
moved outside entitlements.
* [CI] Auto commit changes from spotless
* remove old method call
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Versions 9.1.0 onwards will not be wire-compatible with versions before
8.19.0. This commit sets the minimum transport version to reject
handshakes from earlier versions.
An old node writes all tasks in the metadata custom. A new old must be
able to read it and separate the cluster-scoped and project-scoped tasks
and store them in the right place. This PR does that.
Relates: MP-1945, MP-1938
Some file paths are OS specific. This commit adds a `platform` property
to each file in a files entitlement that can be used to limit that file
to a specific platform.
Co-authored-by: Moritz Mack <mmack@apache.org>
Co-authored-by: Lorenzo Dematte <lorenzo.dematte@elastic.co>
* Updating error message when index field type is unknown
* Fix style issue
* Add yaml test for invalid field type error message
* Update docs/changelog/122860.yaml
* Updating error message for runtime and multi field type parser
* add and fix yaml tests
* Fix code styles by running spotlessApply
* Update changelog
* Updatig the test in yml
* Updating error message for runtime
* Fix failing yaml tests
* Update error message to Fix unit tests
* fix serverless qa test
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
The LuceneSourceOperator is supposed to terminate when it reaches the
limit; unfortunately, we don't have a test to cover this. Due to this
bug, we continue scanning all segments, even though we discard the
results as the limit was reached. This can cause performance issues for
simple queries like FROM .. | LIMIT 10, when Lucene indices are on the
warm or cold tier. I will submit a follow-up PR to ensure we only
collect up to the limit across multiple drivers.
When fetching documents, sometimes we need to load the entire source of
search hits. Document sources can be large, and with support for up to
10k hits per search request, this creates a significant untracked
memory load on Elasticsearch that can potentially cause out-of-memory
errors.
This PR adds memory checking for hits source in the fetch phase. We
check with the parent (the real memory) circuit breaker every 1MiB of
loaded source and when fetching the last document of every segment. This
gives the real memory breaker a chance to interrupt running operations
when we're running low on memory, and prevent potential OOMs.
The amount of local accounting to buffer is controlled by the
`search.memory_accounting_buffer_size` dynamic setting and defaults to
`1MiB`.
Fixes#89656
In the multi-project branch, we're making some changes to persistent
tasks and those changes can cause the persistent tasks custom to still
be `null`. This resulted in an NPE here, so I'm fixing the check here.
Fixes https://github.com/elastic/elasticsearch/issues/122588
- Replaced `Source.EMPTY.writeTo(out)` to `source().writeTo(out)` in functions emitting warnings
- Did the same on all aggs, as Top emits an error on type resolution. This is not a bug, as type resolution errors should only happen in the coordinator. Another option would be changing Top to not generate that error there, and make it implement instead `PostAnalysisVerificationAware`
- In some cases, we don't even serialize an empty source. So I had to add a new `TransportVersion` to do so
- As an special case, `ToLower` and `ToUpper` weren't serializing a source, but they don't emit warnings. As they were the only remaining functions not serializing the source, I added it there too
We only need the extensibility for testing and it's a lot easier to
reason about the code if we have explicit methods instead of overly
complicated composition with lots of redundant references being retained
all over the place.
-> lets simplify to inheritance and get shorter code that performs more
predictably (especially when it comes to memory) as a first step.
This also opens up the possibility of further simplifications and
removing more retained state/memory as we go through the search phases.
These things can be quite expensive and there's no need to recompute
them in parallel across all management threads as done today. This
commit adds a deduplicator to avoid redundant work.
No need to have a nested concat here. There's obviously lots and lots of
room for optimization on this one, but just flattening out one obvious
step here outright halves the number of method calls required when
serializing a search response. Given that method calls can consume up to
half the serialization cost this change might massively speed up some
usecases.
Checking whether we need to refresh does not require a searcher
so we can simplify this to just work based on the reader
and avoid lots of contention etc. for setting up the searcher.
relates #122374
Cleans up a couple things that are obviously broken:
* duplicate array and object constructs where the helper utility
generates the exact same iterator
* unused helper methods
* single iterator concatenations
Don't simultaneously restart multiple nodes in a cluster. It causes data races when
multiple primaries are trying to mark the `[[.snapshot-blob-cache][0]]` shard as stale.
```
org.elasticsearch.cluster.action.shard.ShardStateAction$NoLongerPrimaryShardException: primary term [2] did not match current primary term [4]
at org.elasticsearch.cluster.action.shard.ShardStateAction$ShardFailedClusterStateTaskExecutor.execute(ShardStateAction.java:355) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1075) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1038) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1691) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
at org.elasticsearch.action.ActionListener.run(ActionListener.java:452) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1688) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1283) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
at org.elasticsearch.action.ActionListener.run(ActionListener.java:452) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1262) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.18.0-SNAPSHOT.jar:8.18.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1575) ~[?:?]
```
Resolve#118374Resolve#120442
Creating (and more importantly eventually resizing) a fresh stream
output makes up a large chunk of the runtime of this aggregation.
Also, recursively calling an inline consumer makes this logic
even more confusing and adds additional overhead as escape analysis
will not be able to remove the allocation of the consumer.
=> just call a method recursively and reuse the output
The multi-project module is actually a testing module, but was included
in the modules/ tree for convenience. This commit moves it to
`test/external-modules/` which is where it belongs.
Speeds up the VALUES agg when collecting from many buckets.
Specifically, this speeds up the algorithm used to `finish` the
aggregation. Most specifically, this makes the algorithm more tollerant
to large numbers of groups being collected. The old algorithm was
`O(n^2)` with the number of groups. The new one is `O(n)`
```
(groups)
1 219.683 ± 1.069 -> 223.477 ± 1.990 ms/op
1000 426.323 ± 75.963 -> 463.670 ± 7.275 ms/op
100000 36690.871 ± 4656.350 -> 7800.332 ± 2775.869 ms/op
200000 89422.113 ± 2972.606 -> 21920.288 ± 3427.962 ms/op
400000 timed out at 10 minutes -> 40051.524 ± 2011.706 ms/op
```
The `1` group version was not changed at all. That's just noise in the
measurement. The small bump in the `1000` case is almost certainly worth
it and real. The huge drop in the `100000` case is quite real.
#122887 set read-only on the source indices of a data stream reindex. To test behavior when the source index was already read-only, testDataStreamUpgrade sometimes set the old write index to read-only. But the rest end point for adding a read-only block always sets verifed-read-only. This caused the old write index to not need to be upgraded, resulting in an incorrect value for total_indices_requiring_upgrade
When running Elasticsearch in tests we occassionally have JDK crashes.
It's important to capture the state of the JVM at the time of the crash.
We currently capture the hs_err_pid file for the crash, but the
resulting core file exists in a directory that is no captured. This
commit adjusts the capture patterns to also get core files.
When reindexing a data stream, we currently add a write block to the source indices so that new documents cannot be added to the index while it is being reindexed. A write block still allows the index to be deleted and for the metadata to be updated. It is possible that ILM could delete a backing index or update a backing index's lifecycle metadata while it is being reindexed. To avoid this, this PR sets a read-only block on the source index. This block must be removed before source index can be deleted after it is replaced with the destination index.
When reindexing a data stream, remove the lifecycle name setting when creating the destination index, so that ILM does not process it. Add the setting back after adding the destination index to the data stream, at which point ILM can safely process it.
This method gets called from `InternalEngine#resolveDocVersion(...)`, which gets during indexing (via `InternalEngine.index(...)`).
When `InternalEngine.index(...)` gets invoked, the InternalEngine only ensures that it holds a ref to the engine via Engine#acquireEnsureOpenRef(), but this doesn't ensure whether it holds a reference to the store.
Closes#122974
* Update docs/changelog/123010.yaml
* Use min node version to guard injecting settings in logs provider
* Update docs/changelog/123005.yaml
* no random in cluster init
* Cover basic license in logsdb rolling upgrade tests
* Add enterprise license check to inference action for semantic text fields
* Update docs/changelog/122293.yaml
* Set license to trial in ShardBulkInferenceActionFilterIT
* Move license check to only block semantic_text fields that require inference call
* Cleaning up tests
* Add parameterization on useLegacyFormat back in ShardBulkInferenceActionFilterBasicLicenseIT
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
We should not retain messages after serialization. We currently, have no
0-copy logic in place outside of some special cases that are handled
separately. As a result of not doing zero-copy logic, there is no need
to retain messages until they have been flushed to the wire in full.
This change reduces the lifetime of things like `SearchHit`
significantly, reducing the data-node side impact of fetching large
documents/large aggregations/top-hits to name a few.
The fact that this change technically passes released request + response
instances to the `messageListener` seems irrelevant since the listeners
are only used in tests anyway. If anything, we should look to refactor
this logic to avoid holding on to the request/response objects
needlessly.
Make streaming elements extend Writeable and create StreamInput
constructors so we can publish elements across nodes using the transport
layer.
Additional notes:
- Moved optional methods into the InferenceServiceResults interface and
default them
- StreamingUnifiedChatCompletionResults elements are now all records
* Allow data stream reindex tasks to be re-run after completion
* Docs update
* Update docs/reference/migration/apis/data-stream-reindex.asciidoc
Co-authored-by: Keith Massey <keith.massey@elastic.co>
---------
Co-authored-by: Keith Massey <keith.massey@elastic.co>
The keyword doc values field gets an extra sorted doc values field, that encodes the order of how array values were specified at index time. This also captures duplicate values. This is stored in an offset to ordinal array that gets zigzag vint encoded into a sorted doc values field.
For example, in case of the following string array for a keyword field: ["c", "b", "a", "c"].
Sorted set doc values: ["a", "b", "c"] with ordinals: 0, 1 and 2. The offset array will be: [2, 1, 0, 2]
Null values are also supported. For example ["c", "b", null, "c"] results into sorted set doc values: ["b", "c"] with ordinals: 0 and 1. The offset array will be: [1, 0, -1, 1]
Empty arrays are also supported by encoding a zigzag vint array of zero elements.
Limitations:
currently only doc values based array support for keyword field mapper.
multi level leaf arrays are flattened. For example: [[b], [c]] -> [b, c]
arrays are always synthesized as one type. In case of keyword field, [1, 2] gets synthesized as ["1", "2"].
These limitations can be addressed, but some require more complexity and or additional storage.
With this PR, keyword field array will no longer be stored in ignored source, but array offsets are kept track of in an adjacent sorted doc value field. This only applies if index.mapping.synthetic_source_keep is set to arrays (default for logsdb).
The `local` param for the `GetFieldMapping` API was deprecated in #55014
and I think #57265 aimed to propogate that deprecation to the REST API
spec, but it changed `get_mapping.json` instead of
`get_field_mapping.json`. #55100 removed the `local` param for the
_field_ mapping API so we can safely remove the field from the spec and
remove the YAML test.
* java.base entitlements
* SuppressForbidden, and add a missing test
* Revert logging back to commented-out printlns
* Merge FileCheckActions and rename for overloads
* Remove stray logger
* Remove more traces of logging change
* Remove more traces of logging
Ensures that the DesiredBalanceReconciler always returns a non-empty
AllocationStats object, eliminating edge cases where the stats
available to DesiredBalanceMetrics may not be updated due to some
kind of throttling or the balancer being disabled via cluster
settings.
Adds documentation around
AllocationDecider#canRebalance(RoutingAllocation)
Closes ES-10581
If a shard has been closed, we should quickly bail out and fail
all waiting primary term and generation listeners. Otherwise,
the engine implementation may try to successfully to complete the
provided listeners and perform operations on an already closed
shard and cause some unexpected errors.
Follow-up to #120241 and linked issues, using the logic that doesn't
require any mutable object indirection led to significant speedups
for sum+avg and should be of similar if not more help in these as well.
We shouldn't run the post-snapshot-delete cleanup work on the master
thread, since it can be quite expensive and need not block subsequent
cluster state updates. This commit forks it onto a `SNAPSHOT` thread.
This PR replaces
https://github.com/elastic/elasticsearch-internal/pull/1934 and extracts
some features from
https://github.com/elastic/elasticsearch-serverless/pull/3422.
Instead of allowing for projects to be created without a secrets file,
this PR forces projects to also have a secrets file before being
created.
- This PR doesn't actually handle the secrets data (defines a handler
for reserved project state), that will be implemented in a separate PR.
- The multi-project file handling should move to the
`elasticsearch-serverless` repo before being merged to main (separate
PR).
- This PR contains a bug fix for `ProjectMetadata` where the project
reserved state wasn't always applied to project metadata.
- Since the secrets file name can clash with a project file if named
`project-<project_id>-secrets.json` (project id could be `-secrets`) the
naming of the file has been updated to
`project-<project_id>.secrets.json` (`.` is not allowed in project id)
- This PR adds a separate namespace for secrets, since sharing the
`file_settings` namespace with the `project-<project_id>.json` makes
assumptions on the order of the increasing version number. If they share
namespace, the order would have to be first increased in the project
config and then in the secrets config for them to be processed (since
they're separate state chunks), which is a strange assumption. One of
them also wouldn't be processed if they were on the same version.
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
Relates #101805
With Security Manager we have SecuredConfigFileSettingAccessPermission.
This commit adds an entitlement equivalent. With each entry in files
entitlement, a `path_setting` can now be used. The value may be an
explicit setting, or a setting glob with a single `*`.
relates ES-10844
Today, we disable shard rebalancing in ES|QL REST tests to avoid
failures when shards are rebalanced while the query is being executed.
This is no longer necessary if the clusters are on version 8.19 or 9.1.
* Allow setting the `type` in the reroute processor
This allows configuring the `type` from within the ingest `reroute` processor. Similar to `dataset`
and `namespace`, the type defaults to the value extracted from the index name. This means that
documents sent to `logs-mysql.access.default` will have a default value of `logs` for the type.
Resolves#121553
* Update docs/changelog/122409.yaml
This removes a couple of indirections: the error message for missing shards is always the
same no matter the search phase. This was required to provide a slightly different error
message for open PIT. The previous error was misleading when open PIT did not support
setting allow_partial_search_results, but now that it does, it looks like we can unify
the error message and simplify the code around it.
I couldn't reproduce the failure of the throttling time being zero on
the recovery target node. Added more logging for the stats in case of
failure.
Resolve#122712
This is a high-level overview of the main rebalancing components and
how they interact to move shards around the cluster, and decide where
shards should go.
Relates ES-10423
Using an anonymous class here doesn't compile as expected.
The resulting class comes out as:
```
class Iterators$1 implements java.util.Iterator<T> {
private T value;
final java.lang.Object val$element;
Iterators$1(java.lang.Object);
```
which seemingly also does not get fixed by the JIT compiler, judging by
heap dumps. Lets just use a named class to clean this up and make things
a bit more compact and save some heap as well here and there potentially.
* Unmute `FrozenSearchableSnapshotsIntegTests#testCreateAndRestorePartialSearchableSnapshot`
The underlying failure `java.lang.AssertionError: Searchable snapshot directory does not support the operation [createOutput`
was fixed in https://github.com/elastic/elasticsearch/pull/122006.
The automation bot was too aggressive in re-opening this issue.
Resolve#122693
* Add a check for the CLOSED state along with STOPPED
* Update x-pack/plugin/searchable-snapshots/src/main/java/org/elasticsearch/xpack/searchablesnapshots/cache/full/CacheService.java
Co-authored-by: Tanguy Leroux <tlrx.dev@gmail.com>
---------
Co-authored-by: Tanguy Leroux <tlrx.dev@gmail.com>
In #113462 we introduced a version bound on post-recovery merges to
avoid a spike of merge activity after an upgrade. At the time there was
no `IndexVersion` unique to the 9.x series, but we have such a version
now, so this commit properly restricts post-recovery merges to only
apply to indices created by versions that support them.
Relates ES-9620
If metrics that have the same timestamp and dimensions aren't grouped into the same document, ES will consider them to be a duplicate.
The _metric_names_hash field will be set by the OTel ES exporter.
As it's mapped as a time_series_dimensions, it creates a different _tsid for documents with different sets of metrics.
The tradeoff is that if the composition of the metrics grouping changes over time, a different _tsid will be created.
That has an impact on the rate aggregation for counters.
I have ran this many times locally, and it never failed. Maybe there is
something "magical" in CI.
Added some additional info in the assertion logging.
Xpack used to be a plugin to Elasticsearch, and existed in a separate
repository. So it had a separate build hash and date, which is exposed
through the xpack info API. Since xpack is no longer a plugin, and part
of the Elasticsearch repository, having a separate build info is no
longer necessary. This commit keeps the xpack info content intact, but
switches the implemntation to look at the Elasticsearch build details,
and removes the xpack build info class.
This PR contains the following updates:
| Package | Update | Change | |---|---|---| |
docker.elastic.co/wolfi/chainguard-base | digest | `ecd940b` ->
`d74b1fd` |
---
### Configuration
📅 **Schedule**: Branch creation - "after 1pm on tuesday" (UTC),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once
you are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box
---
This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyI6RGVsaXZlcnkvUGFja2FnaW5nIiwiPm5vbi1pc3N1ZSIsIlRlYW06RGVsaXZlcnkiLCJhdXRvLW1lcmdlLXdpdGhvdXQtYXBwcm92YWwiXX0=-->
# Why and what?
First part of https://github.com/elastic/elasticsearch/issues/122588
Some functions don't serialize their Source. This sometimes makes them emit wrong warnings, with -1:-1 line and without the correct source text.
This PR pretends to integrate ser/deserialization before executing some (randomly chosen) functions, as well as using a non-empty Source to build them.
_Note:_ It's different from the SerializationTests: Here we check that, whether serialized or not, the functionality is identical
When a CopmleteMultipartUpload request fails after the initial 200
response, the status code of the failure response use to be not set and
hence got translated to status code 0. With #116212, we handle this case
accordingly. Since AWS SDK 1.12.691, the status code is now set to 200
instead of 0. This PR changes our error handling code accordingly.
Relates: #122431 Relates: #116212Resolves: #122799
Relevant AWS SDK change
430899c217/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/AmazonS3Client.java (L3696-L3709)
This refactors the "multi-project" module by:
1. Moving some useful code from the module into server
2. Renaming some classes to "TestOnlyMultiProject..."
For (1)
- create a new `AbstractProjectResolver` in server that contains logic
for resolving a project from the thread context.
- create a `MultiProjectFeature` to contain the MP `NodeFeature`
For (2)
- Rename the plugin etc to `TestOnlyMultiProjectPlugin` to reflect that
it is only used for testing.
- We do not touch the file based settings code since this will move to
serverless production.
This is a pure refactor - as of this commit, the
`TestOnlyMultiProjectPlugin` is still used in serverless, the changes
allow us to move to having a separate `ServerlessMultiProjectPlugin`
This PR moves
TransportServiceHandshakeTests.testAcceptsMismatchedServerlessBuildHash
to its own test job to ensure the expected system property is always set
correctly.
Resolves: #122565
Adds missing `assertBusy` in `testQueryDeleteOrUpdateReservedRoles`.
We wait for security migration to finish before test execution, but this is not
enough to guarantee that built-in roles are synchronized into the `.security`
index. In many cases, the roles have already been synchronized before the
query gets executed, but in rare cases the cluster is a bit slower and the
assertion fails because the built-in roles have not been indexed yet.
Resolves https://github.com/elastic/elasticsearch/issues/120401
The size parameter hasn't been accepted since 8.0, end users should use max_docs and if the user specify it, we can defer to the standard error message produced by the parser.
Considering the `.tasks` system index is already project-specific, it seems that we only need to add project-awareness to task management calls that rely on ongoing tasks in the task manager. Among the three operations, i.e., get task, cancel task and list tasks, only get task is public in Serverless. The other two are internal.
I've only touched the get task API, and I've done a very small change to check if the request's project and the task header match. For the other two API methods, I've left them as they are. Which means they see all of the tasks in the cluster. Considering they're internal, I'd assume that's fair. We could make them project-aware in the sense that they would operate only on the tasks that belong to the current project (when the header is set). I think that takes away the possibility of using them at a cluster-level, if a project header will always be present, and not sure if as an internal action we have a use case for it currently. If a "project-view" of the cluster tasks is desirable for listing/cancelling, it might be more suitable to support it via adding a "project" [query parameter](https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html#tasks-api-query-params), e.g., as is available for nodes.
Relates ES-10167
This commit switches to using records in place of classes for FileData
impls. It moves them outside the interface so that they can be private.
This does not change the fact the interface is sealed, nor the factory
methods for creating them.
Add an option to allow reranking to fail, and the docs to pass through as-is.
Exposing the error to users and adding documentation is a later piece of work.
This PR extends the work done in #121751 by enabling a sparse doc values index for the @timestamp field in LogsDB.
Similar to the previous PR, the setting index.mapping.use_doc_values_skipper will override the index mapping parameter when all of the following conditions are met:
* The index mode is LogsDB.
* The field name is @timestamp.
* Index sorting is configured on @timestamp (regardless of whether it is a primary sort field or not).
* Doc values are enabled.
This ensures that only one index structure is defined on the @timestamp field:
* If the conditions above are met, the inverted index is replaced with a sparse doc values index.
* This prevents both the inverted index and sparse doc values index from being enabled together, reducing unnecessary storage overhead.
This change aligns with our goal of optimizing LogsDB for storage efficiency while possibly maintaining reasonable query latency performance. It will enable us to run benchmarks and evaluate the impact of sparse indexing on the @timestamp field as well.
The reserved roles are already returned from the `ReservedRolesStore`
in `TransportGetRolesAction`. There is no need to query and deserialize
reserved roles from the `.security` index just to be merged with "static" definitions.
While testing https://github.com/elastic/elasticsearch/pull/122591, I
realized we need to grand read/write permission to the logs dir to
server.
This PR adds the `logsDir` to the bootstrap parameters, and uses it in
the `server` policy.
Depending on the position of the split and the size of the extra chunk,
the test may send out one more chunk than the assertion expects. This PR
ensures the assertoin is more accurate to count for this scenario.
Resolves: #122349
This PR fixes SecuritySingleNodeTestCase and ProfileIntegTests tests.
- The security single node test failures are solved by ensuring every test starts with security index created and available. This is in order to have consistent state for every test. With the changes introduce in the #120323 PR, only the first test would execute with .security index being created async. Subsequent tests would execute without security index creation due to the fact that whole cluster is wiped after each test. This caused a flakiness only for the first test, because there was no mechanism in place to ensure that the .security index is active before test execution.
- The profile integration tests are solved by introducing an anonymous role which don't have application privileges. The application privileges are resolved from the .security index and assigned to all users, including the es_test_root user which is used during cluster wiping. Due to asynchronous nature of cluster setup and .security index creation, this now causes flakiness. The main problem is that wiping is done asynchronously and uses es_test_root which had assigned anonymous rac_role which depends on .security index being available for search in order to resolve application privileges. The application privilege resolution is done in buildRoleFromDescriptors which currently does not wait for security index availability(can be improved - but still wouldn't fix internal cluster tests). This wasn't a problem before just because we simply return empty results when .security index does not exist. There is some complexity in making internal clusters wait for availability of security shards before the test, so I think this solution is acceptable given that it's not required for this tests to have anonymous role with application privileges.
Resolves#121022Resolves#121096Resolves#121101Resolves#120988Resolves#121108Resolves#120983Resolves#120987Resolves#121179Resolves#121183Resolves#121346Resolves#121151Resolves#120985Resolves#121039Resolves#121483Resolves#121116Resolves#121258Resolves#121486
In test-scoped internal ITs the `cluster().assertAfterTest()` method was invoked
*after* the cluster nodes were closed. Consequently, the assertions that iterated
over the internal nodes (and asserted some state on nodes after the test) were
all effectively noops.
This PR reverses that order, so that after-test assertions are effective again.
remaining is used in tests only since the recent change to counting in AbstractSearchAsyncAction.
reset was used in a couple of places where it does not seem like it's needed anymore.
RescoreKnnVectorQuery rewrites to KnnScoreDocQuery, which takes a sorted array of
doc ids and corresponding array including scores fo such docs. A binary search is
performed on top of the docs array, and such global ids are converted back to
segment level ids (subtracting the context docbase) when scoring docs.
RescoreKnnVectoryQuery did not sort the array of docs which caused binary search
to return non deterministic results, which in turn made us look up wrong docs,
something using out of bound ids. One symptom of this was observed in a DFSProfilerIT
test failure which triggered a Lucene assertion around doc id being outside of the
range of the bitset of live docs.
The fix is to simply sort the score docs array before extracting docs ids and scores
and providing them to KnnScoreDocQuery upon rewrite.
Relates to #116663Closes#119711
This change introduces partial results in ES|QL. To minimize the scope of the
changes, this PR is just the first step toward full support for partial
results. The following follow-up tasks are required:
- Support partial results across clusters
- Return shard-level failures (currently, we only return the `is_partial` flag)
- Add documentation
- Allow partial results during resolution
Use reference manager to get index reader instead of acquiring a searcher. The latter involves creating an index searcher, which is not used and expensive as part of the `resolveDocVersion(...)` method, because this method is invoked for each document that gets indexed.
Unable to reproduce however the coordinator node that is meant to route
the write requests might've not been ready to do so in due time.
This PR adds an ensureStableCluster in the test setup method.
Fixes#120605
We recently fixed timeout handling in the suggest phase. A test failure on SearchTimeoutIT surfaced an issue with the current approach. In case partial results are allowed, it may happen that some shards time out while executing the suggest phase and some don't.
SearchPhaseController assumes that if one shard has suggest results, all of the other shards will have suggest results too. We could address that assertion and check is the search timed out, instead this commit changes timeout handling in the suggest phase to return an empty suggestion instead of null. This seems appropriate in terms of providing some results and makes the assertion about non null suggestions in SearchPhaseController happy.
Relates to #122357Closes#122548
We have at least this one test where the combination of running in the single CPU
environment and a relatively large document count + dynamic mapping updates
exceeds the 10s threshold in very rare cases. Lets try moving this to 30s to hopefully
turn "rare" into practically impossible. I could see this being an issue in other tests
or becoming one, so I think it's fine to raise the timeout for all tests.
closes#116126closes#115815
Obvious SO exception possibilitiy if we encounter exceptions back to back in the callback.
Use promise style pattern instead of callback in the individual loops to limit stack-depth
(this should be good enough for a fix, technically you could still run into very deep stacks
if the search completes between the `isDone` check and adding the listener back-to-back
a couple times but practically this should be good enough since all the instant-done situations
are from the search fully failing outright.
closes#116175
* Restrict routing on sort fields to enterprise license
* sync
* bypass checking for serverless
* Node deprecation warning for indexes and component templates with source mode in mapping
* Revert "Node deprecation warning for indexes and component templates with source mode in mapping"
This reverts commit 0fd4ca7d51.
* address comments
This moves from `isOperator(ThreadContext`) method from
org.elasticsearch.xpack.security.operator.OperatorPrivileges
to
org.elasticsearch.xpack.core.security.operator.OperatorPrivilegesUtil
so that it can be used by modules that depend on x-pack-core (without
needing to depend on x-pack-security)
Currently the SLM health indicator in health report turns YELLOW when snapshots fail for a number of times. However, the SLM health indicator stays GREEN if snapshot is not completed (no success or failure) for a long time. This change adds a new optional setting unhealthy_if_no_snapshot_within to SLM policy, that sets a time threshold. If the SLM policy has not had a successful snapshot for longer than the threshold, the SLM health indicator will turn YELLOW.
Temp dir access is necessary all over Elasticsearch, and in general is
not sensitive. With Security Manager all code in ES, including plugins,
are given read/write access to the temp dir. This commit mimicks that
behavior with entitlements.
When reindexing data stream indices, parts of the index metadata needs to be copied from the source index to destination index, so that ILM and data stream lifecycle function properly. This adds a new CopyLifecycleIndexMetadataTransportAction which copies the following metadata from a source index to a destination index:
- creation date setting
- rollover info
- ILM custom metadata
This PR adds initial support for unmapped fields, using the INSIST clause. For starters, this unmapped fields without a cast.
Note that the INSIST keyword is potentially a placeholder, as the method of defining an unmapped field might change in the future, e.g., use a special magic function. As this is currently under development, the actual syntax is INSIST_🐔.
First stage of #120072.
Specifically, the following features are implemented in this PR:
* Support for INSIST keyword without a cast. In particular, if the type being INSISTed upon is mapped to anything other than KEYWORD, it will result in an InvalidMappedField. There is no support for union type resolution on top of INSIST. Future PRs will handle these conflicts. There is support for multiple parameters, or INSIST on top of INSIST which is equivalent.
* Enforcing that INSIST must always be on top of a FROM or another INSIST. While this may change in the future, e.g., handling cases like `FROM foo | EVAL x = 1 | INSIST bar` will not be done in this PR, as it makes handling INSIST too complicated.
SizeLimitingStringWriter is used to limit the output size of mustache
scripts. When the size is limited, the resulting exception lacks detail
needed to identify the problem. In particular, the actual size that
would result is not given. Additionally, the excerpt lacks any of the
new string being added. This commit tweaks the exception to include both
of these details.
An update operations should make sure
that the shard is mutable before proceeding
further with a Get to retrieve the document
and then any of the create/update/delete/noop
operation.
Relates ES-10708
This patch removes the check that fails requests that attempt to use fields of type: nested within indices with mode time_series.
This patch also updates TimeSeriesIdFieldMapper#postParse to set the _id field on child documents once it's calculated.
Closes#120874
The size method is needed only in SearchShardIterator and PlainIterator.
An interface is not really needed, we can just rather add the size method
where needed.
Also, there's a couple of methods in the ShardsIterator interface that are
not needed that this commit removes: size, reset and remaining.
Whenever a search timeout is set to a search request, the timeout may be triggered
by the suggest phase via exitable directory reader. In that case, the exception that
is thrown by the timeout check needs to be handled, instead of returned back to the user.
Instead of handling the timeout in each phase, this commit handles it as part of QueryPhase
for both SuggestPhase and RescorePhase.
For rescore phase, one integration test that is time dependent is also rewritten to remove
the time dependency and moved from QueryRescorerIT to SearchTimeoutIT.
Closes#122186
Addresses a few issues with logsdb and source only snapshots:
* Avoid initializing index sorting, because sort fields will not have doc values.
* Also disable doc value skippers when doc values get disabled.
* As part of source only validation figure out what the nested parent field is.
Also added a few more tests that snapshot and restore logsdb data streams.
When we detect that a Transform writes to the index and the index is
incompatible with the next version, change the message, detail, and
URL to help the user take the necessary steps to migrate the destination
index.
Add the task name (when possible) and ID as part of the source string
for updating cluster state. This helps better identifying the source of
a task. The updatePersistentTaskState method already does it. This PR
ensures it is the case in other places.
Currently, the ES|QL failure collectors categorize errors into
non-cancellation and cancellation errors, preferring to return
non-cancellation errors to users. With the retry on shard-level failure,
the failure collector can now collect more categories of errors: client
errors, server errors, shard-unavailable errors, and cancellation
errors. For easier diagnostics and operations (especially on
serverless), the failure collectors prefer returning client (4xx) errors
over server (5xx) errors, shard-unavailable errors, and cancellation
errors.
Relates #120774
In addition to logging the pending cluster tasks after the cluster
health request times out during cluster cleanup in REST tests, we should
log the hot threads to help identify any issues that could cause tasks
to get stuck.
Follow-up of #119186
Relates #111632
Relates #111431
Relates #111662
The yaml test
reindex/30_search/Sorting deprecated wait_for_completion false
does not work correctly with security enabled.
This test should be permanently added to the "unsupported" list.
Relates: https://github.com/elastic/elasticsearch/issues/120252
With #117410, each remote sink now has its own listener, and the main
query won't finish until all remote sink listeners have completed. As a
result, we no longer need to wait for the exchange source to finish.
This change removes the completion listener to simplify the exchange
service. The completion listener could previously return prematurely
while remote sinks were still being registered.
Closes#122408
When utilizing synthetic source with nested fields, we attempt to
rebuild the child values in addition to all the parent values.
While this generally works well, its potential that certain values might
be missing from various child docs. Consequently, we will attempt to
iterate the vector values strangely, resulting in seemingly missing
values or potentially exceptions indicating EOFs.
closes: #122383
The DesiredBalanceReconciler is responsible for applying updates to
the cluster states that reflect shard allocation changes towards a
DesiredBalance. It isn't the Reconciler's responsibility to handle
pushing APM metrics. This patch cleans up the Reconciler constructor
and logic by extracting metric handling, modularizing metric updates
in the Allocator level of the code instead of being split across the
two components. This will facilitate testing.
This patch also contains general improvements in documentation
around the desired balance Reconciler and Metrics related code.
Relates ES-10581
Auto expand replicas is meant to be entirely disabled for stateless indices.
The only scenario where a change needs to be applied is when the number of replicas
is initialized to 0, in which case 0 needs to be turned into 1. Otherwise, no changes
should be applied in stateless indices despite auto expand replicas is used.
The current handling for this was missing an early exit of the indices loop in the case
where 0 shoudl be turned into 1, that leads to a potentially higher number of copies
being allocated (effectively auto-expand gets applied by mistake).
When the Transform is configured to write to an alias, specifying
`DELETE _transform/<id>?delete_dest_index` will follow the alias
to the concrete destination index.
Fix#121913
Co-authored-by: Przemysław Witek <przemyslaw.witek@elastic.co>
There is a race condition where the test is trying to clean up while the
Transform auditor is still writing messages - resetting the plugin will
stop the auditor (and properly reset it).
Fix#12148
No need for recursion here. We don't allow nested compound order instances
so this thing works exactly as the JavaDoc states and we only need to check
one level down.
This commit adds instrumentation for File methods that require write
permission. No server or plugins use these methods, so no policy changes
were necessary. Note that since we are not planning to restrict temp
file creation, the bootstrap self test on file writing was removed,
which failed with these changes.
Fix race condition test bugs related to the reindex-data-stream-pipeline. For tests that add doc without timestamp, then add mapping with timestamp, ensure green between adding doc and adding mapping. This makes sure that doc has been written to all shards and thus that timestamp validation does not occur while doc is being written to a shard. Delete pipeline in Before method, then wait for it to be re-created by the MigrateTemplateRegistry.
We can build slightly more compact (and likely also faster) iterators
while using less code for these. Also, no need to create method
references as a way of casting.
Current code closes the previous engine instance, creates the new engine and then updates the currentEngineReference:
IOUtils.close(currentEngine);
var newEngine = createEngine(engineConfig);
currentEngineReference.set(newEngine);
This leaves more room for callers of getEngineOrNull() to pick a closed instance. Instead we can create the new engine first and atomically update the reference to the new instance.
An old node does not know about the separation between cluster and
project scoped persistent tasks. When cluster state and cluster state
diff are read from or writen to an old node, wire BWC must be handled.
Specifically this means:
1. Read must split the incoming persistent tasks custom metadata into
cluster and project scoped ones and store them separately. We re-use the
same lastAllocationId for both cluster and project scopes.
2. Write must combine cluster and project scoped metadata and send it as
one single custom so that an old node can understand. The
lastAllocationId sent across is the max between cluster and project
scoped persistent tasks.
Note the BWC is only necessary when the new node has a single default
project. In other cases, we already simply reject such communication.
The cluster scoped persistent tasks are not used yet and should be added
as follow-ups. Until then, only unit tests are possible for the BWC
logic.
This PR contains the following updates:
| Package | Update | Change | |---|---|---| |
docker.elastic.co/wolfi/chainguard-base | digest | `bd40170` ->
`ecd940b` |
---
### Configuration
📅 **Schedule**: Branch creation - "after 1pm on tuesday" (UTC),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once
you are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box
---
This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MjUuMSIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyI6RGVsaXZlcnkvUGFja2FnaW5nIiwiPm5vbi1pc3N1ZSIsIlRlYW06RGVsaXZlcnkiLCJhdXRvLW1lcmdlLXdpdGhvdXQtYXBwcm92YWwiXX0=-->
This PR increases the waiting time for the DFA native process from 60 to 90 seconds to stabilize the flaky integration tests that sometimes fail because the job has not finished training within 60 seconds.
* Support returning `getLastUnsafeSegmentGenerationForGets` for any `Engine`, not only `InternalEngine`
* Retry real-time gets on `AlreadyClosedException` in case a shard's engine gets swapped.
See ES-10571
Introduce beforeIndexShardMutableOperation in IndexEventListener to allow plugins to react when a mutable shard is needed. Add IndexShard#ensureMutable to let consumers signal this need and be notified when the shard is ready for mutable operations.
Relates ES-10787
`l` is the same as the `finalListener` here, that's kind of the point of
the API. Fixing this here and removing the local variable the enabled
the mistake in the first place. Not a bug, but a needless capture that
at times makes heap dumps harder to read and wastes a couple cycles.
The nebula info broker plugin takes the information for the manifest from the java project settings rather then from
the compile task configuration. Instead of setting the compiler task configuration explicitly we now set the project
configuration accordingly. Also tweaked the javaTestCompile tasks to keep compiling with general minimum runtime version as we did before
With this change, we will skip tests using semantic_texts in ES|QL while
awaiting a fix for the mismatch in the _source of the semantic text
fields. The key change is that we need to avoid indexing semantic_text
fields in clusters with more than one node.
Relates #121411
There are two issues in the current implementation:
1. We should use the list of shardIds from the request, rather than all
targets, when removing failures for shards that have been successfully
executed.
2. We should remove shardIds from the pending list once a failure is reported
and abort execution at that point, as the results will be discarded.
Closes#121966
ReindexDataStreamIndexAction.cleanupCluster called EsIntegTestCase.cleanupCluster, but did not override it. This caused EsIntegTestCase.cleanupCluster to be called twice, once in ReindexDataStreamIndexAction.cleanupCluster and once when the After annotation is called on EsIntegTestCase.
The XPackRestIT test {p0=snapshot/10_basic/Create a source only snapshot
and then restore it} had been failing due to spurious creation of ML
indices. That no longer seems to be the case as, after extensive local
testing, the test is passing.
Raising this PR to unmute the test
Relates #117295
This PR removes a potential cause of data loss when migrating system indices. It does this by changing the way we set a "write-block" on the system index to migrate - now using a dedicated transport request rather than a settings update. Furthermore, we no longer delete the write-block prior to deleting the index, as this was another source of potential data loss. Additionally, we now remove the block if the migration fails.
In the recent rewrite of SearchTimeoutIT, an edge case was not considered
for the situation where a specific segment may not score any document.
In that case, other shards and their segments will raise a timeout anyway.
Two of the timeout tests have been muted for several months. The reason is that we tightened the assertions to cover for partial results being returned, but there were edge cases in which partial results were not actually returned.
The timeout used in the test was time dependent, hence when the timeout precisely will be thrown is unpredictable, because we have timeout checks in different places in the codebase, when iterating through the leaves, before scoring any document, or while scoring documents. The edge case that caused failures is a typical timing issue where the initial check for timeout in CancellableBulkScorer already triggers the timeout, before any document has been collected.
I made several adjustments to the test to make it more robust:
- use index random to index documents, that speeds it up
- share indexing across test methods, so that it happens once at the suite level
- replace the custom query that triggers a timeout to not be a script query, but rather a lucene query that is not time dependent, and throws a time exceeded exception precisely where we expect it, so that we can test how the system reacts to that. That allows to test that partial results are always returned when a timeout happens while scoring documents, and that partial results are never returned when a timeout happens before we even started to score documents.
Closes#98369Closes#98053
Improve LuceneSyntheticSourceChangesSnapshot by triggering to use a sequential stored field reader if docids are dense. This is done by computing for which docids to synthesize recovery source for. If the requested docids are dense and monotonic increasing a sequential stored field reader is used, which provided recovery source for many documents without repeatedly de-compressing the same block of stored fields.
* Adding condition to verify if the field belongs to an index
* Update docs/changelog/121720.yaml
* Remove unnecessary comma from yaml file
* remove duplicate inference endpoint creation
* updating isMetadata to return true if mapper has the correct type
* remove unnecessary index creation in yaml tests
* Adding check if the document has returned in the yaml test
* Updating test to skip time series check if index mode is standard
* Refactor tests to consider verifying every metafields with all index modes
* refactoring test to verify for all cases
* Adding assetFalse if not time_series and fields are from time_series
* updating test texts to have better description
In this PR, we change how the doc values sparse index is enabled for the host.name keyword field.
The initial implementation of the sparse index for host.name was introduced in #120741.
Previously, the choice between using an inverted index or a doc values sparse index was determined by the index parameter. With this change, we introduce a new final index-level setting, index.mapping.use_doc_values_sparse_index:
- When the setting is true, we enable the sparse index and omit the inverted index for host.name.
- When the setting is false (default), we retain the inverted index instead.
Additionally, this setting is only exposed if the doc_values_sparse_index feature flag is enabled.
This change simplifies enabling the doc values sparse index and makes the selection of indexing strategies explicit at the index level. Moreover, the setting is not dynamic and is exposed only for stateful deployments.
The plan is to enable this setting in our nightly benchmarks and evaluate its impact on LogsDB indexing throughput, storage footprint and query latency. Based on benchmarking results, we will decide whether to adopt the sparse index and determine the best way to configure it.
This is caught thanks to #121210: if shard files are verified/checksumed
while the node is stopping, an IllegalStateException is throw by
CacheService.get() when it attempts to read data from the cache. This
exception later caused the verification to fail and then the Lucene
index to be marked as corrupted (which nows fails for searchable
snapshots shards that are read-only and should not be corrupted at
all).
This pull request changes ensureLifecycleStarted(), which is called
during CacheService.get(), to throw an AlreadyClosedException when
the service is stopped (note that ACE extends IllegalStateException,
which is convenient here). This ACE will be later specially handlded
in the checksumIndex method to not mark the shard as corrupted (see
#121210).
Closes#121927
To avoid having AggregateMapper find aggregators based on their names with reflection, I'm doing some changes:
- Make the suppliers have methods returning the intermediate states
- To allow this, the suppliers constructor won't receive the chanells as params. Instead, its methods will ask for them
- Most changes in this PR are because of this
- After those changes, I'm leaving AggregateMapper still there, as it still converts AggregateFunctions to its NamedExpressions
* Grammar for ES|QL change point (with dummy impl)
* pipeline breaker
* ChangePointOperator
* Add sorting
* basic csv test
* conflict
* Update docs/changelog/120998.yaml
* [CI] Auto commit changes from spotless
* polish
* Non-long data type
* Move OrderBy/Limit to the logical plan
* fix mem.leak
* csv test for reusing column names
* Warning indeterminable
* capability
* handle null values
* too much data
* type text->keyword
* default timestamp and output columns
* spotless
* ChangePointOperatorTests + fix memory leaks
* [CI] Auto commit changes from spotless
* improve test
* add comments/todos
* handle multivalued columns
* don't register unserialiazable
* surrogate
* make "too much data" tests readable
* more tests
* Error handling
* fix multivalued test
* more name conflict tests
* [CI] Auto commit changes from spotless
* more tests
* improve code
* CSV test for various input key/value types
* one more csv test
* Check sortable/numeric for all types
* add null type to testChangePoint_valueNumeric
* more CSV tests
* skip nulls instead of zeroing them
* error on MV
* Test+todo for nicer error message
* better error msg
* Revert "better error msg"
This reverts commit 21ec77c8f3.
* fix
* make csv test deterministic
* replace NamedExpression -> Attribute
* skip MVs + warning
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
This commit removes "TLSv1.1" from the list of default protocols in
Elasticsearch (starting with ES9.0)
TLSv1.1 has been deprecated by the IETF since March 2021
This affects a variety of TLS contexts, include
- The HTTP Server (Rest API)
- Transport protocol (including CCS and CCR)
- Outgoing connections for features that have configurable SSL
settings. This includes
- reindex
- watcher
- security realms (SAML, OIDC, LDAP, etc)
- monitoring exporters
- inference services
In practice, however, TLSv1.1 has been disabled in most Elasticsearch
deployments since around 7.12 because most JDK releases have disabled
TLSv1.1 (by default) starting in April 2021
That is, if you run a default installation of Elasticsearch (for any
currently supported version of ES) that uses the bundled JVM then
TLSv1.1 is already disabled.
And, since ES9+ requires JDK21+, all supported JDKs ship with TLSv1.1
disabled by default.
In addition, incoming HTTP connections to Elastic Cloud deployments
have required TLSv1.2 or higher since April 2020
This change simply makes it clear that Elasticsearch does not
attempt to enable TLSv1.1 and administrators who wish to use that
protocol will need to explicitly enable it in both the JVM and in
Elasticsearch.
Resolves: #108057
This logic will need a bit of adjustment for bulk query execution.
Lets dry it up before so we don't have to copy and paste the fix which
will be a couple lines.
When a node is shutting down, scheduling tasks for the Driver can result
in a rejection exception. In this case, we drain and close all
operators. However, we don't clear the pending tasks in the scheduler,
which can lead to a pending task being triggered unexpectedly, causing a
ConcurrentModificationException.
Like the plugin being testing, the entitled test plugin needs access to
dynamic elements (namely, file paths). This commit dynamically generates
the entitlement policy for the entitlted test plugin when it is
installed. It also adds using the file entitltlement as an example.
The only real path separators are either forward or back slash. Trying
to use something else like newline fails to even parse as a path on
windows. This commit removes testing of other separators.
closes#121872
This commit adds adds support for mappings containing only a subset of
metrics in aggregate metric double (i.e. only sum and value_count, or
just max, etc) as well as tests for grouped aggregations on aggregate
metric double.
We shouldn't run the post-snapshot-delete cleanup work on the master
thread, since it can be quite expensive and need not block subsequent
cluster state updates. This commit forks it onto a `SNAPSHOT` thread.
In this PR we improve the deprecation warning about configuring source
in the mapping.
- We reduce the size of the warning message so it looks better in kibana.
- We keep the original message in the details.
- We use an alias help url, so we can associate it with the guide when it's created.
It is possible to create an index in 7.x with a single type. This fixes the CreateIndexFromSourceAction to not copy that type over when creating a destination index from a source index with a type.
Since introducing the fail_fast (see #117410) option to remote sinks,
the ExchangeSource can propagate failures that can lead to circular
references. The issue occurs as follows:
1. remote-sink-1 fails with exception e1, and the failure collector collects e1.
2. remote-sink-2 fails with exception e2, and the failure collector collects e2.
3. The listener of remote-sink-2 propagates e2 before the listener of
remote-sink-1 propagates e1.
4. The failure collector in ExchangeSource sees [e1, e2] and suppresses
e2 to e1. The upstream sees [e2, e1] and suppresses e1 to e2, leading to
a circular reference.
With this change, we stop collecting failures in ExchangeSource.
Labelled this non-issue for an unreleased bug.
Relates #117410
The aggs timeout test waits for the agg to return and then double checks
that the agg is stopped using the tasks API. We're seeing some failures
where the tasks API reports that the agg is still running. I can't
reproduce them because computers. This adds two things:
1. Logs the hot_threads so we can see if the query is indeed still
running.
2. Retries the _tasks API for a minute. If it goes away soon after the
_search returns that's *fine*. If it sticks around for more than a
few seconds then the cancel isn't working. We wait for a minute
because CI can't be trusted to do anything quickly.
Closes#121993
This updates the kibana signature json files in two ways:
* Renames `eval` to `scalar` - that's the name we use inside of ESQL and
we may as well make the name the same.
* Calls the `CATEGORIZE` and `BUCKET` function `grouping` because they
can only be used in the "grouping" positions of the `STATS` command.
Closes#113411
This is a really strange assertion. I get that it tries to make sure we
skip unavailable without forking but this makes extending the
AbstractSearchAsyncAction cleanly for batched execution needlessly hard
and some of the assertion is dead code already because can-match isn't
going through this codepath anymore.
-> lets remove it, the code is simple enough now to follow that there's
no forking here IMO
Transport version changes must be unique per branch. Some transport
version changes meant for 9.0 are missing unique backport constants.
This commit adds constants and adjusts the transport logic to use the
new constants.
This PR adds ClusterPersistentTasksCustomMetadata by splitting from the
existing PersistentTasksCustomMetadata. The latter will be used for
project scoped persistent tasks. It also updates tasks executors to
declare its scope.
The new task type is not used anywhere yet and has no wire BWC handling.
Both will be addressed in follow-ups.
Split from: MP-1938
When we are already parsing events, we can receive errors as the next
event.
OpenAI formats these as:
```
event: error
data: <payload>
```
Elastic formats these as:
```
data: <payload>
```
Unified will consolidate them into the new error structure.
This change replaces FileEntitlement with FilesEntitlement so that we can have exactly one entitlement
class per module (or possibly future scope). This cleans up our policy files so that all files are located
together to allow access, and this opens up the design for future optimizations.
Stateless nodes rely on an external blob store to persist data, therefore
it's not necessary to go through such checks when a shard store
should be deleted.
Closes ES-10577
ShardIterator is an interface with a single implementation called PlainShardIterator.
This commit makes it a concrete final class and folds its only implementation into it.
This PR addresses issues around aggregations cancellation, mentioned in https://github.com/elastic/elasticsearch/issues/108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree.
Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Nik Everett <nik9000@gmail.com>
This test creates an incorrectly-serialized handshake which cannot be
validated, and #121747 made that validation compulsory. This test
corrects the serialization.
Closes#121816
Today we use the ID from `Version#CURRENT` in this test, which only
works if its ID is no less than that of `TranportVersion#current()`.
This commit fixes the test to ensure it always picks a transport version
ID that is not from the past.
* Starting revoke
* Adding integration tests
* More integration tests
* Adding test for deleting default inference endpoint via rest call
* Removing task type any
* Addressing feedback and adding test
Today, the exchange buffer of an exchange source is finished in two
cases: (1) when the downstream pipeline has received enough data and (2)
when all remote sinks have completed. In the first case, outstanding
pages could be safely discarded. In the second case, no new pages should
be received after finishing. In both scenarios, discarding all
outstanding pages was safe if noMoreInputs was switched while adding
pages.
However, with the stop API, the buffer may now finish while keeping
outstanding pages, and new pages may still be received. This change
updates the exchange buffer to discard only the incoming page when
noMoreInputs is switched, rather than all pages in the buffer.
Closes#120757
Today, ES|QL fails fast on any failure. This PR introduces support for
retrying within a cluster when data-node requests fail.
There are two types of failures that occur with data-node requests:
entire request failures and individual shard failures. For individual
shard failures, we can retry the next copies of the failing shards. For
entire request failures, we can retry every shard in the node request if
no pages have been received.
On the handling side, ES|QL executes against a batch of shards
concurrently. Here, we need to track whether any pages have been
produced. If pages have been produced, the entire request must fail.
Otherwise, we can track the failed shards and send them back to the
sender for retries.
There are two decisions around how quickly we should retry:
1. Should we notify the sender of failing shards immediately (via a
different channel) to enable quick retries, or should we accumulate
failures and return them in the final response?
2. What is the maximum number of inflight requests we should allow on
the sending side?
This PR considers failures often occurring when the cluster is under
load or during a rolling upgrade. To prevent retries from adding more
load and to allow the cluster to stabilize, this PR chooses to send
shard failures in the final response and limits the number of inflight
requests to one per data node
In #121256, we changed the superclass of the request to
`LocalClusterStateRequest` to make the action run on the local node
instead of the master node. Most of the actions that have been updated
thusfar to run on the local node used to extend `MasterNodeReadRequest`
and thus serialized a `local` field. This request class, however,
extended `MasterNodeRequest`, meaning that it didn't serialize that
field.
* Report componentName in ModuleEntitlements.
This lets us produce error messages that guide the user to add the right
entitlement to the right plugin/server/etc.
* Include component names in errors and logs
* Name APM agent specifically.
Avoids confusion with the entitlements agent.
* Entitlement component names package private
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
The node environment has many paths. The accessors for these currently
use a "file" suffix, but they are always directories. This commit
renames the accessors to make it clear these paths are directories.
Fix a bug in TOP which surfaces when merging results from ordinals. We
weren't always accounting for oversized arrays when checking if we'd
ever seen a field. This changes the oversize itself to always size on a bucket boundary.
The test for this required random `bucketSize` - without that the
oversizing frequently wouldn't cause trouble.
This causes the ESQL heap attack tests to grow their memory usage if
they first don't cause a circuit breaking exception. It just tries again
with more data. That's slow, but it should stop this from failing quite
as much. And it'll give us even more information about failures.
Closes#121465
This adds a `task_description` field to `profile` output and task
`status`. This looks like:
```
...
"profile" : {
"drivers" : [
{
"task_description" : "final",
"start_millis" : 1738768795349,
"stop_millis" : 1738768795405,
...
"task_description" : "node_reduce",
"start_millis" : 1738768795392,
"stop_millis" : 1738768795406,
...
"task_description" : "data",
"start_millis" : 1738768795391,
"stop_millis" : 1738768795404,
...
```
Previously you had to look at the signature of the operators in the
driver to figure out what the driver is *doing*. You had to know enough
about how ESQL works to guess. Now you can look at this description to
see what the server *thinks* it is doing. No more manual classification.
This will be useful when debugging failures and performance regressions
because it is much easier to use `jq` to group on it:
```
| jq '.profile[] | group_by(.task_description)[]'
```
If a custom analyzer provided in _analyze API can not be built, return
400 instead of the current 500. This most probably means that the user's
provided analyzer specifications are wrong.
Closes#121443
A future.actionGet was missing from the delete pipeline action execution in the test cleanup, causing all tests to fail intermittently. Also replace actionGet with safeGet.
There is no point in having `GroupShardsIterator`, it's mostly an
unnecessary layer of indirection as it has no state and a single field
only. It's only value could be seen in it hiding the ability to mutate
the list it wraps, but that hardly justifies the overhead on the search
path and extra code complexity. Moreover, the list it references is not
copied/immutable in any way, so the value of hiding is limited also.
* Fix FileAccessTree for prefixes that aren't parents
* Support backslashes
* Whoops, nio
* Move normalization responsibility to FileEntitlement
* Normalize to native separators
* Avoid forbidden API
With parallel releases on multiple branches it's possible that an older
branch sees a transport version update that is not known to a
numerically newer but chronologically older version. In that case the
two nodes cannot intercommunicate, so with this commit we reject such
connection attempts at the version negotiation stage.
This updates the PolicyParser to allow static methods to have an ExternalEntitlement annotation. This
removes a limitation where constructors cannot properly support type-erasure with different types of
data structures for internal entitlement generation and external entitlement generation (for example
List<Object> from the parser and List<SomeData> from an internal builder). We continue to enforce
that only one constructor/method may be annotated with ExternalEntitlement per Entitlement class.
Unified Chat Completion error responses now forward code, type, and
param to in the response payload. `reason` has been renamed to
`message`.
Notes:
- `XContentFormattedException` is a `ChunkedToXContent` so that the REST listener can call `toXContentChunked` to format the output structure. By default, the structure forwards to our existing ES exception structure.
- `UnifiedChatCompletionException` will override the structure to match the new unified format.
- The Rest, Transport, and Stream handlers all check the exception to verify it is a UnifiedChatCompletionException.
- OpenAI response handler now reads all the fields in the error message and forwards them to the user.
- In the event that a `Throwable` is a `Error`, we rethrow it on another thread so the JVM can catch and handle it. We also stop surfacing the JVM details to the user in the error message (but it's still logged for debugging purposes).
Failure handling for snapshots was made stricter in #107191 (8.15), so this field is always empty since then. Clients don't need to check it anymore for failure handling, we can remove it from API responses in 9.0
The Renovate config in this repository needs migrating. Typically this
is because one or more configuration options you are using have been
renamed.
You don't need to merge this PR right away, because Renovate will
continue to migrate these fields internally each time it runs. But later
some of these fields may be fully deprecated and the migrations removed.
So it's a good idea to merge this migration PR soon.
🔕 **Ignore**: Close this PR and you won't be reminded about config
migration again, but one day your current config may no longer be valid.
❓ Got questions? Does something look wrong to you? Please don't hesitate
to [request help
here](https://redirect.github.com/renovatebot/renovate/discussions).
---
This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
We experimented with using synthetic source for recovery and observed quite positive impact
on indexing throughput by means of our nightly Rally benchmarks. As a result, here we enable
it by default when synthetic source is used. To be more precise, if `index.mapping.source.mode`
setting is `synthetic` we enable recovery source by means of synthetic source.
Moreover, enabling synthetic source recovery is done behind a feature flag. That would allow us
to enable it in snapshot builds which in turn will allow us to see performance results in Rally nightly
benchmarks.
`combineValueCount` is described as something that might no longer be needed and has zero actual usages. Removing it in order to simplify code generation.
Stateless has a subclass for PersistedClusterStateService which needs to
know the value of supportsMultipleProjects. Due to how a node is
initialized, the value is not known until the plugin components are
created which is after the stateless service is created. Therefore this
PR changes the parameter from boolean to a booleanSupplier so that it
can be resolved later when actually used.
The alternative is to change the PersistedClusterStateServiceFactory
interface so it takes a projectResolver or boolean
supportsMultipleProjects. I prefer to not change the interface. The
creation of Stateless PersistedClusterStateService already uses two
other suppliers so that it feels more aligned to use a boolean supplier.
The other reason is that supportsMultipleProjects may not be needed in
long term so it is better to not have it in an interface.
Relates: MP-1880
Add the pipeline "reindex-data-stream-pipeline" to the reindex request within ReindexDataStreamIndexAction. This cleans up documents as needed before inserting into the destination index. Currently, the pipeline only sets a timestamp field with a value of 0, if the document is missing a timestamp field. This is needed because existing indices which are added to a data stream may not contain a timestamp, but reindex validates that a timestamp field exists when creating data stream destination indices.
This pipeline is managed by ES, but can be overriden by users if necessary. To do this, the version field of the pipeline should be set to a value higher than the MigrateRegistry version.
Instrumented methods in EntitlementChecker are loosely grouped by the
associated entitlement. This commit sorts the methods within groups to
allow more clear placement of additional instrumented methods.
This simplifies the setup and relaxes the similarity check.
We can restrict the similarity check once we evolve the quantization
algorithm in the future.
This patch adds the needed data generator and source matcher to include
counted_keyword fields in our randomized testing.
This patch also updates the source matcher such that field-specific
matchers are checked before the generic matcher is used. It seems that
this is the correct behavior, and the only reason the generic matcher was
checked first was as a workaround for issue #111916, which is now closed.
The mrjar plugin produces artifacts for each java version that may be
consumed downstream. This commit fixes an issue with gradle dependencies
when multiple java versions are depended on.
Updates the on failure handler to accept the additional context required to report to the client whether
or not the document could have been rerouted if failure store was enabled.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Some areas of the code call this field type
AggregateDoubleMetric and others AggregateMetricDouble, but the docs
use aggregate_metric_double, so for consistency this commit refactors
the former into the latter.
As a result of the randomized testing enabled in #121462, we found that
we currently fail to parse documents with arrays of objects containing
counted_keyword fields. This PR fixes this issue by using a custom
docvalues field to store the count instead of the built-in lucene
BinaryDocValues. This custom CountsBinaryDocValuesField has logic to
handle multiple values for the same field.
If the query hits the failing index first, we will cancel the request,
preventing exchange-sink requests and data-node requests from reaching
another data node. As a result, exchange sinks could stay for 30
seconds.
Under very unfortunate conditions tests that check xContent objects
roundtrip parsing (like i.e. SearchHitsTests#testFromXContent)
can fail when we happen to randomly pick YAML xContent type and create
random (realistic)Unicode character sequences that may contain the
character U+0085 (133) from the Latin1 code page. That specific character
doesn't get parsed back to its original form for YAML xContent, which can
lead to rare but hard to diagnose test failures.
This change adds logic to AbstractXContentTestCase#test() which lies at
the core of most of our xContent roundtrip tests that disallows test
instances containing that particular character when using YAML xContent
type.
Closes#97716
This change adds migration testing for the ".async-search" system index
to the full cluster upgrade tests that perform updates from versions N-2
to N via N-1.
The test creates a system index by using async_search on a cluster with
version N-2, then calls the "_migrate" API in version N-1 and finally
checks that on the upgraded cluster in N we are still able to retrieve
async search results from previous versions and can still write to the
system index. This is necessary to ensure we don't end up with a
write-only async search system index when migrating to version 9.
This refactoring was motivated by the following issues with the current
state of the code:
- The `TransformDeprecationChecker` is listed as plugin checker, but later we remove is from the `plugin_settings` and add it to the `cluster_settings`. This made me consider that the checker might be dealing with transform deprecation warnings but if they are listed under the `cliuster_settings`, it fits better to be part of `ClusterDeprecationChecker`.
- The `DeprecationInfo` is a data class, but it has a method `from` which constructs an `DeprecationInfo.Response` instance. However, this is not a simple factory class but it actually runs all the checks and it also tries to assert that it is not executed on a transport thread. Considering this, I thought it might fit better to the `TransportDeprecationInfoAction`, this way all the logic is in one place and all the checkers are wired and used in the same class.
- Constructing the node settings deprecation issues requires to merge the deprecation warnings of the individual nodes. We considered bringing together the execution of the remote request and the construction of the response in a new class called `NodeDeprecationChecker` that resembles the patterns of the other Checker classes.
- Reinstated the `PLUGIN_CHECKERS` even if we have only one check, so other developers can easier add their plugin checks.
- Finally, we noticed that the way we synthesise the remote requests is difficult to read and maintain because each call is nested under the previous one. We propose in this PR a different pattern that uses the `RefCountingListener` to combine the different remote calls and store their results in a container class named `PrecomputedData`
- **Bonus**: Removed the `LegacyIndexTemplateDeprecationChecker.java` which was not used.
Since #118606 searchable snapshots shards are not expected to
write files on disk, with the exception of archives indices mounted
as searchable snapshots which require to rewrite the segment infos
file in a newer version.
Ideally we should be able to remove the usage of the
InMemoryNoOpCommitDirectory for non-archives searchable
snapshots indices and only rely on SearchableSnapshotDirectory
that throws on write operations. Similarly, starting 9.0 searchable
snapshots shards do not write files on disk and therefore should
be able to use a Directory implementation that forbids writes.
Searchable snapshots shards for indices created before 9.0
require a mutable directory for peer-recoveries.
In this change, we only allow writes for archives indices and
searchable snapshots created before 9.0.
Relates ES-10438
This reverts commit ae0f1a64b5.
The refresh block would be removed in a subsequent cluster state
update instead of removing it immediately after an index is ready
for searches.
Closes ES-10697
We want to create and delete projects only once around the entire YAML
test suite. Therefore the project creation status needs to be remembered
across tests. This commit changes the field to be static. It also moves
initialization of projectIds to be static.
Indices from different projects could have identical mappings and hence
identical mapping hash. The unused mapping hashes are computed for a
project scope. Hence the deletion must also be project-scoped to not
deleting mapping hashes on a different project.
Resolves: ES-10568
The test failed because we tried to move a shard to a node that already
has a copy. This change prevents that from happening.
Closes#119280Closes#120772
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
The upper bound of randomVersionBetween is inclusive; therefore, for
testing the fallback version of the request, we need to use the version
preceding 8.16.0 rather than 8.16.0 itself.
Closes#117937
* Fix incorrect value of reindex_required flag on ignored index warning
* Datastream reindex now uses unverified write block to allow retrying failed reindex
Reenables some heap attack tests, bumping their memory requirements to
try and force a failure on all CI machines. Previously some CI machines
weren't failing, invalidating the test on those machines.
Close#121481Close#121465
This attempts to fix a flay test where the term_freq returned by the
multiple terms vectors API was `null`. I was not able to reproduce this
test but this proposes a fix based on the following running theory: - an
Elasticsearch cluster comprised of at least 2 nodes - we create a couple
of indices with 1 primary and 1 replica - we index a document that was
acknowledged only by the primary (because `wait_for_active_shards`
defaults to `1`) - the test executes the multiple terms vectors API and
it hits the node hosting the replica shard, which hasn't yet received
the document we ingested in the primary shard.
This race condition between the document replication and the test
running the terms vectors API on the replica shard could yield a `null`
value for the the term's `term_freq` (as the replica shard contains 0
documents).
This PR proposes we change the `wait_for_active_shards` value to `all`
so each write is acknowledged by all replicas before the client receives
the response.
Fixes#113325
* Adding support for binary embedding type to Cohere service embedding type
* Returning response in separate text_embedding_bits field
* Update x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/inference/results/InferenceByteEmbedding.java
Co-authored-by: David Kyle <david.kyle@elastic.co>
* Update docs/changelog/120751.yaml
* Reverting docs change
---------
Co-authored-by: David Kyle <david.kyle@elastic.co>
Change the warning message when you send a multifield under a geo field.
We *still* ignore the multifields but we do not plan to remove them. It
isn't worth breaking anyone.
This service is added to the desired balance allocator to track and
report on balancer round activity. It is a WIP and currently only
tracks the number of shard moves caused by a balancing round.
Reporting balancer round summary results will provide information
with which to do cost-benefit analyses of the work that shard
allocation rebalancing executes. It is disabled by default.
Relates ES-10341
When adding support for upgrading closed indices in
N-2 version, I randomized the Translog.Durability setting
of the closed index with the aim to test the 2 phases
closing process.
This caused at least 1 test failure on Windows with the
index being closed and the cluster upgraded before the
synchronization of the translog had a chance to be
executed. I think this cause the engine to be reset on
the replica that is promoted as a primary, causing the
loss of the operations that were not yet persisted.
Closes#121257
Asserting that we definitely saw the "received a single result" flag
and can now deal with null responses, isn't applicable after a few
recent fixes. New requests are sent out before responses are fully
processed to keep data nodes in a tighter loop (as well as other
relaxed ordering relative to when this assertion was added) so
the flag is not guaranteed to show up as true for lower numbers
of shard requests any longer.
Lets just remove it, it was always best effort and accidental that
this worked for the numbers the test randomizes over.
Under very unfortunate conditions tests that check xContent objects
roundtrip parsing (like i.e. [SearchHitsTests
testFromXContent](https://github.com/elastic/elasticsearch/issues/97716)
can fail when we happen to randomly pick YAML xContent type and create
random (realistic)Unicode character sequences that may contain the
character U+0085 (133) from the [Latin1 code
page](https://de.wikipedia.org/wiki/Unicodeblock_Lateinisch-1,_Erg%C3%A4nzung).
That specific character doesn't get parsed back to its original form for
YAML xContent, which can lead to [rare but hard to diagnose test
failures](https://github.com/elastic/elasticsearch/issues/97716#issuecomment-2464465939).
This change adds logic to AbstractXContentTestCase#test() which lies at
the core of most of our xContent roundtrip tests that disallows test
instances containing that particular character when using YAML xContent
type.
Closes#97716
Currently if a document has duplicate suggestions across different
contexts, only the first gets indexed, and when a user tries to
search using the second context, she will get 0 results.
This PR addresses this, but adding support for duplicate suggestions
across different contexts, so documents like below with duplicate inputs
can be searched across all provided contexts.
```json
{
"my_suggest": [
{
"input": [
"foox",
"boo"
],
"weight" : 2,
"contexts": {
"color": [
"red"
]
}
},
{
"input": [
"foox"
],
"weight" : 3,
"contexts": {
"color": [
"blue"
]
}
}
]
}
```
Closes#82432
* ESQL: Expand HeapAttack for LOOKUP
This expands the heap attack tests for LOOKUP. Now there are three
flavors:
1. LOOKUP a single geo_point - about 30 bytes or so.
2. LOOKUP a one mb string.
3. LOOKUP no fields - just JOIN to alter cardinality.
Fetching a geo_point is fine with about 500 repeated docs before it
circuit breaks which works out to about 256mb of buffered results.
That's sensible on our 512mb heap and likely to work ok for most folks.
We'll flip to a streaming method eventually and this won't be a problem
any more. But for now, we buffer.
The no lookup fields is fine with like 7500 matches per incoming row.
That's quite a lot, really.
The 1mb string is trouble! We circuit break properly which is great and
safe, but if you join 1mb worth of columns in LOOKUP you are going to
need bigger heaps than our test. Again, we'll move from buffering these
results to streaming them and it'll work better, but for now we buffer.
* updates
Today we limit the number of shards concurrently closed by the
`IndicesClusterStateService`, but this limit is currently a function of
the CPU count of the node. On nodes with plentiful CPU but poor IO
performance we may want to restrict this limit further. This commit
exposes the throttling limit as a setting.
* Refactor: remove duplicate canWrite methods.
This serves as a good example of how Path and File handling could be
specialized in the future, but as long as they are identical, the duplication
causes more harm than good.
* Refactor: just one neverEntitled.
The original motivation was to avoid allocating a lambda object on each call,
but since that's a highly optimized operation in the JVM, it's unlikely to make
a difference in practice, and this smacks of premature optimization.
We're pretty liberal about lambdas elsewhere, so let's not sweat it here until
we have some evidence that it matters.
* Remove dead code
Updated indices.resolve_cluster.json to match new resolve/cluster spec.
Added new test for the no-index-expression endpoint.
Adjust syntax in 10_basic_resolve_cluster.yml syntax fix so that the elasticsearch-specification validation tests pass.
This commit adds an EntitlementTest annotation that can be used on
classes containing test actions for entitlements. The annotation mirrors
the parameters of CheckAction. Only file check actions are currently
converted, the rest can be moved and annotated as followups. Note that
the check action name is simply the method name, no fancy name
manipulation is done.
This commit forces the delegate for ES logging to always use the String
version of LogManager.getLogger instead of the one taking a Class. The
reason is that if a classloader is not in the hierarchy of the app
classloader, the ES logging configuration will not be found. By using
the String variant, the app classloader is always used.
This commit adds the data dirs, config dir and temp dir into
entitlement bootstrapping. It doesn't yet use them in entitlement
policies, but makes them available to use within initialization.
No sense in converting to a list just to convert to a string, we may as
well convert directly to a string. Also removes the unnecessary extra
`[]` wrapper.
This adds a pre-mapping logical plan processing step, occurring after the logical optimisation, but before mapping it to a physical plan. This step can perform async actions, if needed, and involves using a new `TransportActionServices` record with all available services.
Furthermore, the query rewriting step part of the `FullTextFunction`s planning (occurring on the coordinator only) is refactored a bit to update the queries in-place.
The verification done by `Match` and `Term` involving checking on the argument type is also now pulled back from post-optimisation to post-analysis. Their respective tests are moved accordingly as well.
* Integrate watsonx reranking to inference api
* Add api_version to the watsonx api call
* Fix the return_doc option
* Add top_n parameter to task_settings
* Add truncate_input_tokens parameter to task_settings
* Add test for IbmWatonxRankedResponseEntity
* Add test for IbmWatonxRankedRequestEntity
* Add test for IbmWatonxRankedRequest
* [CI] Auto commit changes from spotless
* Add changelog
* Fix transport version
* Add test for IbmWatsonxService
* Remove canHandleStreamingResponses
* Add requireNonNull for modelId and projectId
* Remove maxInputToken method
* Convert all optionals to required
* [CI] Auto commit changes from spotless
* Set minimal_supported version to be ML_INFERENCE_IBM_WATSONX_RERANK_ADDED
* Remove extraction of unused fields from IbmWatsonxRerankServiceSettings
* Add space
* Add space
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Assertion was using reference equality on two boxed longs. So assertion could produce false positives. Change to Objects.equals to check value and avoid null check.
AggregateMetricDouble should be behind a feature flag but on release
builds it was getting added when it should not have been. This commit
addresses that bug.
* fix StatementParserTests.testNamedFunctionArgumentInMap, testNamedFunctionArgumentWithUnsupportedNamedParameterTypes
* separate snapshot test from release test
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Fix inference update API calls with task_type in body or deployment_id defined
* Update docs/changelog/121231.yaml
* Fixing test
* Reuse existing deployment ID retrieval logic
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
If the `MasterService` needs to log a create-snapshot task description
then it will call `CreateSnapshotTask#toString`, which today calls
`RepositoryData#toString` which is not overridden so ends up calling
`RepositoryData#hashCode`. This can be extraordinarily expensive in a
large repository. Worse, if there's masses of create-snapshot tasks to
execute then it'll do this repeatedly, because each one only ends up
yielding a short hex string so we don't reach the description length
limit very easily.
With this commit we provide a more efficient implementation of
`CreateSnapshotTask#toString` and also override
`RepositoryData#toString` to protect against some other caller running
into the same issue.
This effectively releases LOOKUP JOIN into tech preview. Docs will follow in a separate PR.
* Enable the lexing/grammar for LOOKUP JOIN in non-snapshot builds.
* Remove the grammar for the unsupported | JOIN ... command (without LOOKUP as first keyword). The way the lexer modes work, otherwise we'd also have to enable | JOIN ... syntax on non-snapshot builds and would have to add additional validation to provide appropriate error messages.
* Remove grammar for LOOKUP JOIN index AS ... because qualifiers are not yet supported. Otherwise we'd have to put in additional validation as well to prevent such queries.
---------
Co-authored-by: Bogdan Pintea <bogdan.pintea@elastic.co>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Nik Everett <nik9000@gmail.com>
Instead of deleting and recreating projects after every YAML
multi-project test, we should create the projects once at the start of
the test suite and clean up the active project after every test.
This doesn't actually make the simulate actions multi-project aware, it pushes the `getProject` and related calls further towards the edge, so that `MetadataIndexTemplateService` _can_ be multi-project compatible.
The actual simulate actions will need to be made MP-compatible at a later point.
Updates various places - mostly related to ML - to make the `POST
_features/_reset` API not fail when there are multiple projects.
This does not actually make those places project-aware,
but since nothing ML-related has been made project-aware,
this is the best way forward for now to make IT cleanup project aware.
Make `ReservedComposableIndexTemplateAction` project-specific. All metadata of reserved state loaded from files
matches the source file structure, rather than where in cluster state the resulting state changes occur.
ILM is not a thing for serverless, so we need to decide how to manage default project references in this class (and related) long-term. Add a marker to do this later.
This updates `TransportResizeAction` to be project aware.
Although these actions are not supported on Serverless projects, we make
them work correctly with projects so that we can work towards removing
the deprecated `getProject` method.
This removes the method
getDataStreamNames(
IndexNameExpressionResolver,
ClusterState,
String[],
IndicesOptions
)
from DataStreamsActionUtil and replaces all callers with the project
scoped version.
This necessitated the introduction of
`AcknowledgedTransportMasterNodeProjectAction` and some changes to
`TransportMasterNodeReadProjectAction` (and subclasses)
This updates the cluster stats action to take all projects into account
when calculating the statistics. This is true even if a specific project
is requested.
The only exception is the "health status" because we only have
primitives for calculating the health of a single project. The stats
response will always give the health of the targeted project.
Create a multi-project version of `FileSettingsService` to handle individual project settings files.
At the moment, there are no handlers for project-specific data, but it will create projects in the cluster based on the settings files that exist in the directory
As a precursor to converting the reserved state handlers to work on
`ProjectMetadata`, convert `MetadataIndexTemplateService` to work on
`ProjectMetadata`, with temporary adaptor methods.
Roles are cached per-project. Invalidation of roles can be either
project-scoped (index based roles) or cluster scoped (file based
roles), and the invalidation logic reflects this distinction.
Add infrastructure to support project-specific reserved cluster state.
Using this, most existing state handlers will next change from modifying `ClusterState` to modifying `ProjectMetadata`, with the remaining handlers (probably) changing from `ClusterState` to `Metadata`, at which point we can change `S` type to `M`, and change all the language from 'state' to 'metadata'.
When we removed an index block from a project, the BwC deserialization
code would add an empty map for the default project. This commit fixes
that to return an empty map instead.
This PR fixes two more cluster-blocks usages and unmuted tests that can
pass. Some of the tests can already pass without the fix so that this PR
also unmutes them.
No need to add an empty placeholder entry for a project if it does not
have any indices blocks. This helps efficency for both serialization and
building. Relevant methods all take care of this optimization by
returning empty result when recieving requests for such projects.
This PR makes open/close indices and add index blocks work by replaces a
large percentage of usages for the deprecated methods with the
corresponding project-aware versions. The ones that are no longer used
are removed. Some deprecated methods still remain after this PR. They
require even more cascading changes in areas that have not been touched,
such as snapshot/restore, watcher, security index settings etc. I prefer
to leave them out since this PR is already big enough. The main changes
are in MetadataIndexStateService class where both routing table and
metadata need to be specific to the right project.
Updates both transport actions to be able to handle mulitple projects.
This is part of an ongoing effort to make integration test cleanup
project aware.
This PR refactors the Service Provider part of the SAML Realm to allow a
plugin to inject a custom service provider provider. The default is now
called `SingleSamlSpConfiguration` and has no changes in behaviour.
ShardLimitValidator enforces cluster/node wipe shard count limits. Hence
it needs to work with metadata instead of project metadata. This PR
ensures this is the case.
This PR adds a project layer to ClusterBlocks so that it works with
multiple projects. Previously the index level blocks are tracked with a
map of "index-name to set-of-blocks". This is changed to be a map of
"project-id to project-level-blocks", where project-level-blocks
currently contains just the index level blocks. This structure is
expandable in that if we want to support project-wide blocks (as opposed
to cluster-wide blocks), they can be added to the project-level-blocks.
One alternative that has been considered but decided against is to
update the map of "index-name to set-of-blocks" to "index-object/uuid to
set-of-blocks". This has a few issues: (1) ClusterBlocks has no
references to Metadata so it cannot serve a request for index blocks
given an index name (bwc); (2) Similarly, when deserializing from an
old node, which provides only index names, there is no easy way to
lookup for the associated index UUID; (3) The structure is rather rigid
and cannot be easily expanded for project-level blocks if they become a
thing; (4) No easy way to get all indices blocks for a project.
I chose to keep all existing ClusterBlocks methods that take index name
as argument to avoid too many cascading changes. I marked all of them as
deprecated for removal so that they can be easily spotted, replaced and
removed in subsequent PRs. That said, there are still test failures
where cascading changes are necessary.
Relates: ES-10162
As part of MP-1853 this test was changed to sometimes use the default
project id. However the test would fail if one of the "other" projects
had the same id as the main project, which had a > 25% chance (50%
chance of main project being "default" and 50% of the first other
project also being "default")
Role Mappings can be stored as a `ProjectCustom` on `ProjectMetadata`
(see `RoleMappingMetadata`).
This commit ensures that the role mapper looks for the mappings on the
active project.
A new MultiProject JWT test has been added to cover some of this
behaviour
On deserialization, state is split into project and cluster based on the namespace, which is hardcoded in `Metadata.Builder.MOVED_PROJECT_RESERVED_STATE`
Instead of using an empty global routing table, this PR uses empty
project routing table to ensure the project actual has a routing table
(albeit empty) when routingTable(projectId) is called.
This changes the `MultipleProjectsClientYamlSuiteTestCase` to verify
that every project other than the active project is "empty" at the end
of the test.
"empty" means that it has no indices, and only has the templates,
pipelines and ILM polices that are installed by default
This removes the deprecated
```
resolveWithIndicesRequest(IndicesRequest, ClusterState, ... )
```
method from `ResolvedIndices` now that it is unused.
The makes `ResolvedIndices` completely project scoped.
This PR makes put-alias project aware so that the aliases API works in a
multi-project setup. This is achieved by adding a new project-id field
to IndicesAliasesClusterStateUpdateRequest.
The alias actions are also used by MetadataRolloverService. These usages
are NOT updated in this PR and still assumes default project. They will
be handled separately.
This cleans up the exclusion list for
`xpack-rest-tests-with-multiple-projects` to enable some working tests
and be more explicit about which tests are disabled.
The PUT and DELETE APIs were already project aware, but pushed the
project resolver into the cluster state update task. This commit pushes
the project id down instead - which is consistent with other actions
that have been made project aware.
Additionally, this commit makes the GET ILM policy project aware and it
includes ILM's YAML REST tests in the multi-project test suite.
This commit updates `DiskThresholdMonitor` to track indices using the
`Index` object (which implicitly depends on the index UUID) rather than
using index names directly and then update the index read-only setting
in the owning project.
This PR proposes changes to make a YAML test pass for cluster reroute
commands, specifically the cancel allocation command. The allocation
command is initiated by user but batch executed in the cluster state
update thread where it loses the thread context. It also no longer has
access to the original request object at that time. This PR changes
the parsing process to take the project id and inject it into the command
objects itself.
The retry_failed parameter can be used without any specific allocation
command. For the time being, the PR keeps it working at cluster level
but adds constraint that it can only be used if there is no allocation
command.
This unmutes a number of security-related YAML tests that correctly
function now that we have merged the following PRs
- MP-1752
- MP-1755
- MP-1764
- MP-1771
- MP-1772
- MP-1775
- MP-1782
This makes the ingest CRUD APIs and the majority of the `IngestService`
project aware.
This does not include the parts of the `IngestService` that are related
to executing the pipelines in bulk requests.
This changes `SecurityIndexManager` so that it is aware of projects, and
can deal with indices in each project having different states
(`.security` exists in some projects, but not others, has unavailable
shards in one project, etc).
The design is that the `SecurityIndexManager` handles all projects, and
has a single listener for cluster state events. It tracks the state of
the applicable security index in each individual project.
Methods that enquired about the state of an index have been move from
`SecurityIndexManager` itself to the inner class
`ProjectSecurityIndex`.
Callers now need to obtain a reference to the correct
`ProjectSecurityIndex` via either:
- `getProject(ProjectId)` _or_
- `forCurrentProject` (which relies on the `ProjectResolver`).
One side effect of this is that some calls that previously would have
seen the "live" state of the security index (i.e. state as at last
cluster change event) may now see something that is a millisecond or 2
old because `ProjectSecurityIndex` retains a point in time view of the
index state.
For the most part this is a benefit as it ensures that a series of
decisions about the state oif the index are made from a consistent view
of that state. The previous implementation has a `defensiveCopy` method
that was intended to be used for this purpose, but its use was optional
and not consistent.
This commit changes `SystemIndexMappingUpdateService` so that it updates
system indices in any/every project.
There is a single cluster state listener that determines the set of
system indices that need to be updated across all loaded projects, and
triggers an update on each of these.
This refactors the `resolveWithIndicesRequest` and `resolveWithPIT` to
be project scoped (they take `ProjectMetadata` in place of
`ClusterState`)
The existing methods (that take `ClusterState`) still exist but are
deprecated and delegate to the new methods (taking `ProjectMetadata`)
As title says, this PR makes the auto-create-index action project aware
and unmute YAML tests that can now pass. Note this works for regular
indices. More work is still required for data-streams.
The `AsyncTaskMaintenanceService` deletes async search results after a
period of time (default: 1h)
This change makes this service work correctly when there are multiple
projects (and potentially multiple async-search indices).
A single instance of the service runs on each data node (this is
unchanged beahviour). The service checks each project for a copy of the
`.async-search` index, where the primary shard#0 is on the local node
(the only change here is to check each project) and then performs a
delete-by-query on the `.async-search` index in that project (the change
here is to make it project aware)
It is possible that a project may not be found in the cluster state due
to concurrent operations or other reasons. It is a valid exception case.
This PR makes sure the exception is thrown consistently and removes
unnecessary assertions.
A ProjectResolver should be able to fetch the ProjectId without the need
of a cluster state. This PR changes the method to take no argument and
fixes cascading changes. It also adds a separate ProjectIdResolver
interface to host the new no-arg getProjectId method. The existing
ProjectResolver interface extends the new interface.
The major impact of the change is for tests and test helpers. It has
been made more obvious that most tests rely on the default project ID.
Previously this was hidden by the no arg `singleProjectOnly()` that just
pops the only project from cluster state. This only project is the
default project anyway in the current tests. The change just made this
fact explicit which I think is not a bad thing. We can definitely tidy
up the tests and test helpers once more pieces fall into places. Happy
to take suggestions.
This changes `MlAssignmentNotifier` so that it can tolerate multiple
projects in a cluster.
If any project other than the default project has a persistent task,
then this code will still fail, however the mere existence of multiple
projects will not cause errors during cluster change events
"stashing" the thread context is typically used to execute something
with a differenet set of privileges (access to system indices, or a
different user). In these cases we don't want to lose the project-id
from the context (just like we don't want to lose APM tracing
information).
This change preserves the project-id during stash.
`TransportBulkAction.populateMissingTargets` is responsible for
resolving the list of index-like targets need to have actions performed
before the bulk ingestion can be processed. This includes
- indices which need to created
- data-streams which need to rollover
- failure-stores that need to rollover
This change makes this method resolve the correct project rather than
operating on the default project.
In the multi-project yaml rest tests, switch to using the routing table
(from cluster state, see MP-1695) to determine that the projects have been
correctly created and configured in the test client.
This commit adds methods to `Metadata` that allow the retrieval of the
`ProjectMetadata` (or `IndexMetadata`) for an `Index`
In practice this:
- Moves `ProjectLookup` from `GlobalRoutingTable` to `Metadata`
- Adds new methods like `projectFor(Index)` on `Metadata`
- Updates all references to `GlobalRoutingTable.getProjectLookup()` to
use the new methods on `Metadata`
- Hides the `ProjectLookup` interface so that it becomes an
implementation detail
There's probably some more cleanup that could be done, but this change
gets us moving in the right direction.
This commit allows the cluster health API to function in a multi-project
setting. Note: the API is only able to operate on a single project per
request. In the future, we might want to consider changing the API to
allow for multi-project responses, but that's a discussion for later.
This is part of an ongoing effort to get the cleanup part of integration
tests to work in a multi-project setting.
This introduces a new QA project that runs the core YAML rest tests
against a cluster with multiple projects.
At this point in time:
- Only 1 project is actively tested at a time
- That project is not the "default" project
- There are other projects in the cluster
- Those projects are empty
- There is a long exclude list of YAML tests
- There are a handful of tests that are executed successfully
This means that actions, parsers, etc that work with project-ids don't
need to have checks like
```
if( Strings.isNullOrBlank(projectId.id()) )
```
Because they know that a project-id cannot be null or blank.
Similarly use of whitespace or non-ascii characters in project-ids can
lead to subtle or unexpected bugs. We assert that these are not used.
When TransportClusterStateAction runs against a specific project, it
returns a `ClusterState` that contains a single project (the one
specified). This project may not have the default project id, but needs
to be able to be serialized in the XContent response.
This commit allows `Metadata` to serialize a single project in
non-multi-project mode, even if it is not the default project.
This API simply removes the project from the metadata and global routing
table. That should be sufficient for now, but we might want to
reconsider in the future if we need to take more actions and/or handle
that differently.
This allows the project retrieval API to function with multiple projects
in the cluster.
This is part of an ongoing effort to get the cleanup part of integration
tests to work in a multi-project setting.
ProjectId is a record which means it had an auto-generated toString that
included the class name.
However, most of the time, we actually want to do something like
throw new IllegalStateException("cannot find project [" + id + "]");
and just have the raw project-id included.
This commit changes toString to do exactly that.
This annotation (which will be removed in the future) marks places in
the source code that we need to revisit as part of the multi-project
initiative.
It can be used to mark
* code that should be removed,
* code where we have made decisions that we'd like to revisit
* places that are known to break when operating on multiple projects
* anything else that needs to be looked at before the code can be
considered production-ready
Currently, test logs are filled with warnings about things not
supporting multiple projects. So we have a chance to filter out things
that we're actually working on, add and catch a specific exception type
for not-multi-project-compatible state listeners (and others, in time)
The introduction of the multi-project feature means that the special
clusters created by native ML tests can't join as they don't also have
multi-project. This adds the module to those clusters, so they at least
have the multi-project feature
Add a cluster feature indicating the node is multi-project. Once a
node/cluster becomes multi-project, it should not go back to being
pre-multi-project. This node feature can also be used to check
conditions on various operations in the future.
This prevents the SLM policy retrieval API from breaking when there are
multiple projects in the cluster state. This does _not_ allow multiple
projects to have SLM policies yet.
This is part of an ongoing effort to get the cleanup part of integration
tests to work in a multi-project setting.
Updates `TransportClusterStateAction` to be aware that the cluster
state can contain multiple projects and filter the routing table
down to only contain the current (requested) project(s)
This commit updates:
- `DiskThresholdDecider`
- `EnableAllocationDecider`
- `FilterAllocationDecider`
- `ResizeAllocationDecider`
to be aware of projects, and operate correctly when there are multiple
projects in a cluster.
This simply avoids the IndexLifecycleService from breaking when there
are multiple projects in the cluster state. It does not yet handle
multiple projects with IndexLifecycleMetadata.
Changes RoutingNodes.allShardsActive to take `ProjectMetadata` instead
of `Metadata`, and updates `RebalanceOnlyWhenActiveAllocationDecider` to
resolve the correct project for the shard being checked
This commit modifies `IndexMetadataUpdater` so that it can accept shard
updates for any project and apply the changes to the correct
`IndexMetadata` object in the correct project.
This updates `DesiredBalanceReconciler.allocateUnassignedInvariant` to
be aware of projects and consider all shards across all projects when
reconciling the cluster balance
This commit changes `OrderedShardsIterator` to be able to sort shards
from multiple projects.
The sort order ignores the project, and sorts shards according to
existing rules.
We do not know whether this is the behaviour we want long term, but it
is a necessary first step to having a working multi-project cluster.
I noticed I was repeating a pattern for both IngestService and
IndexLifecycleService. This PR extracts two utility methods that can be
used in those cases and probably more future cases.
This simply avoids the IngestService from breaking while applying a new
cluster state that contains multiple projects. It does not yet handle
multiple projects with ingest metadata.
Restricts the "index settings" provider that's invoked when creating new
indices to only inspect the current project's metadata (rather than the
whole global metadata).
When an index is created, there's an optional parameter hook function
that callers use to alter the metadata given the index metadata that's
being created.
This PR makes that hook parameter work with the `ProjectMetadata` rather
than the whole `Metadata`.
This generally improves per-project encapsulation, which in principle
should avoid accidental cross-project changes.
This commit removes the public `ProjectMetadata.builder()` method so
that every constructed `ProjectMetadata` must have an `id`.
The only valid case we had for not providing an id when constructing the
`Builder` was when deserializing and in that case we can use the
package-protected `Builder` constructor.
This commit adds a new `rebuild` method to `GlobalRoutingTable` that
recreates the routing table based on the state of `RoutingNodes`.
This new method is based on the previous behaviour of
`RoutingTable.of` which has now been modified to be responsible for
less.
This commit adds 2 new methods to ProjectResolver
- `projectId`
- `projectIds`
The latter is needed because in some cases a request may be project-less
(that is it operates on all projects). The resolver needs some way to
indicate when a request falls into this category.
The options are:
- Some flag for "all projects", such as a method like `isAllProjects()`
- A method that returns all projects
The latter is preferred because it makes actions agnostic about how
project resolution works - if the action is capable of operating on
multiple projects, then it asks the resolver for the list of projects
and gets back 1 or many. It doesn't have to care about the distinction
between project-targeted requests and project-agnostic requests, it just
operates on the list of projects that the resolver returns.
This method returns `ProjectId` rather than `ProjectMetadata` for two
reasons.
Firstly, for clarity.
`ClusterState.metadata().projects()` already returns `ProjectMetadata`
and having two ways to get `ProjectMetadata` risks confusion. Using ids
here makes it less likely that methods calls or local variables will get
mixed up.
Secondly, for code simplicity. The examples we know about want to work
with project-ids rather than `ProjectMetadata` (e.g. to access the
routing table).
In addition this commit adds some sample implementations of
`ProjectResolver` within the test framework (see
`TestProjectResolvers`). This is useful when writing a unit test for a
class that depends on `ProjectResolver` - there are reliable
implementations that exhibit some different behaviours that may be
useful for testing.
This commit adds support for resolving index expressions within the
context of a project.
Existing support for resolving index expressions for a cluster is
retained, and (for now) assumes that there is a single project in the
cluster. In the future this should be changed to use `ProjectResolver`
instead.
This commit adds the `GlobalRoutingTable.Builder` class and
`GlobalRoutingTable.builder()` methods to match the patterns used by
other cluster state objects
This commit changes the `GlobalRoutingTable` to contain multiple
projects.
Note: Most code will break if there are, in fact, multiple projects
(because the `getRoutingTable` method can only return a single project
and will throw an exception if there are multiple)
Implements a basic API for creating projects.
The API currently doesn't execute successfully because there are some
pieces of code that still depend on there being a single project.
We need to write `ProjectId` instances to streams. This should be done
by treating it as a `Writeable` object rather than making consumers deal
with the internal fields.
Introduces a new `GlobalRoutingTable` that sits above the (project
scoped) `RoutingTable`.
For now this new class is not used anywhere, and it supports a single
`RoutingTable`. This will be changed in followup commits.
Relates: ES-9105
This commit changes the behaviour of `Metadata` so:
- The "default" or "single" project is the project with the id `default`
(it no longer tracks the cluster UUID)
- A default project is only created when absolutely needed, this
includes:
- When attempting to perform project-scoped operations on the cluster
level `Metadata`
- When constructing a `Metadata` object from a `Builder` that does not
have any projects in it.
The initial goal of this PR was to make the "put ILM" action go through the project resolver in order to resolve the project-scoped metadata, and hence avoid referring to the whole cluster state. This implies changing some methods to work on the project metadata rather than the whole cluster metadata.
It turns out, due to good code reuse, it is hard to only change one specific action to only refer to project-scoped metadata.
This adds a `ProjectId` to the `ProjectMetadata`.
The `ProjectId` contains just a `String` value and is initialiazed to
the `clusterUUID`, since righat now there only exists a single project.
We don't need any BwC compatibility, so we can define this single
constant and use it for everything. At the point we come to merge this
into main, the multi-project constant should be updated to be a real
transport version with the right id schema
This PR improves our integration testing coverage. The idea is to have two data streams,
referred as the baseline acting as the source of truth and a contender data stream whose
contents and results of queries need to be checked against the baseline.
We also include some capability around matching results which needs to be extended
to be able to consider artifacts in results introduced by the usage of synthetic source.
Currently fails due to validation that is only performed in serverless:
```
java.lang.AssertionError: Failure at [logsdb/20_mapping:94]:
Expected: "Failed to parse mapping: Indices with with index mode [logs] only support synthetic source"
but: was "Failed to parse mapping: Parameter [mode=disabled] is not allowed in source"
```
Quick cleanup PR. With
https://github.com/elastic/elasticsearch/pull/109413 merged, nothing in
ESQL production depends on the core binary comparison classes anymore.
This PR removes them to avoid the potential for confusion, and open the
opportunity for merging the core `BinaryComparison` class with the
`EsqlBinaryComparison` base class, which is now its only direct
inheritor (work to be done in a future PR).
* Make union types use unique attribute names
* Cleanup leftover
* Added failing test and final fix to EsRelation
* Implement FieldAttribute.fieldName()
* Fix tests
* Refactor
* Do not ignore union typed field's parent
* Fix important typo
D'oh
* Mute unrelated (part of) test
* Move capability to better location
* Fix analyzer tests
* multi-node tests with an earlier version of union-types (before this change) fail
* Add capability to remaining failing tests
* Remove variable
* Add more complex test
* Consolidate union type cleanup rules
* Add 3 more required_capability's to make CI happy
* Update caps for union type subfield yaml tests
* Update docs/changelog/110793.yaml
* Refined changelog text
* Mute BWC for 8.15.0 for failing YAML tests
* union_types_remove_fields for all 160_union_types tests
The tests fail spordically, so safer to mute the entire suite.
---------
Co-authored-by: Alexander Spies <alexander.spies@elastic.co>
- Added a custom implementation of BooleanBucketedSort to keep the top booleans
- Added boolean aggregator to TOP
- Added tests (Boolean aggregator tests, Top tests for boolean, and added boolean fields to CSV cases)
* Adding new inference stats class
* Working test
* Adding factory to plugin
* Removing unnecessary changes
* Fixing test
* Removing close() for now
* Expanding modelId() comment
* using incrementby and refactoring interfaces
* Implement RequestedRangeNotSatisfiedException for Azure and GCP
* spotless
* rename test
* Generalize 3rd party tests for 3 cloud blob containers
* Follow comments
* minimize changes with main
* Follow comments 2
- Added a test case with groups to the aggregation tests, which detected:
- Fixed BitArray out of index error on `fill()`
- Fixed BooleanArrayState out of index error on `getOrDefault()`, affecting Max and Min
- Also, this bug affected the results of Max/Min in some specific cases
This constructor is needlessly inefficient:
* Sorting an array is quite a bit faster than sorting an `ArrayList` the way we did it here (could do it faster via List.sort but array ist still better).
* Pre-size the hashmap of positions
* Faster `ShardId.compareTo`
* Much faster `SearchShardIterator.compareTo`!
Today we return HTTP code 207 if some features successfully reset and
others failed. This is not an appropriate response code, it has a _very_
precise meaning according to the HTTP specification to which we do not
adhere. Since this API is used only in tests we can be stricter and
return a 500 unless it completely succeeds.
So that the only expected disk write at the point of the assertion is from the bulk request. And not from the asynchronous runnable of updateDanglingIndicesInfo().
Fixes#110551
While it is _technically_ true that we only need to look up the
`Repository` instance in order to load `SnapshotInfo` blobs for
snapshots that aren't currently running, in practice the repository
should exist on all paths anyway, the lookup is cheap, and it simplifies
some future work to resolve it sooner.
The timed `PlainActionFuture#get` method is almost exclusively used in
tests, and never in a place where it really matters that it supplies a
`PlainActionFuture<T>` rather than an `ActionListener<T>`. Moreover some
of its usages wait for longer than `SAFE_AWAIT_TIMEOUT`, typically 30s,
but not for any good reason. We have several suitable test utility
methods for achieving the same thing, and these utilities automatically
deal with exceptions/interrupts/timeouts appropriately. This commit
replaces all the test usages with more suitable test utilities.
The only production usages are in the CCR module. While it would be good
to remove this blocking code, for now this commit just introduces a new
CCR-only utility for this behaviour. That way we can deprecate the
widely-available `PlainActionFuture#get` method and remove it in a
follow-up.
When JNA loads libraries it creates a proxy object for the library.
Unfortunately it doesn't actually inspect any of the methods, those get
bound lazily at runtime when the method is called through the proxy. For
fstat64 we need to know at load time whether the symbol exists, so that
we can fallback to an alternate function if it doesn't.
This commit looks up the NativeLibrary object from JNA for libc and
checks if fstat64 exists during load time.
Collecting the list of snapshot IDs over which to iterate within each
repository today involves several other potentially-large intermediate
collections and a bunch of other unnecessary allocations. This commit
replaces those temporary collections with an iterator which saves all
this temporary memory usage.
Relates ES-8906
This change renames the search response metric attribute status to response_status. This is more
descriptive for an attribute that falls under the global labels namespace.
This also fixes a typo with succes to success.
- Created two new ExecutableActions, SenderExecutableAction and
SingleInputSenderExecutableAction
- Most chat completions are migrated over the
SingleInputSenderExecutableAction
- Every other Action are migrated over to SenderExecutableAction
- RequestManagers and Error Message construction are migrated into the
ActionCreator classes.
Relate #110805
This adds an example to the docs an example of counting the TRUE results
of an expression. You do `COUNT(a > 0 OR NULL)`. That turns the `FALSE`
into `NULL`. Which you need to do because `COUNT(false)` is `1` -
because it's a value. But `COUNT(null)` is `0` - because it's the
absence of values.
We could like to make something more intuitive for this one day. But for
now, this is what works.
Resolves
[#105240](https://github.com/elastic/elasticsearch/issues/105240)
Count_distinct doesn't work on source, but the type resolution was
allowing that through. This resulted in a 500 layer deeper in the
aggregations code. This PR fixes the 500 error by correctly failing
during type resolution.
Includes the setting name as well as its value, and also add a link to
the relevant troubleshooting docs. Also reinstates a thread interrupt
flag that could technically get lost here otherwise.
Small refactoring to merge some optimizer rule super classes that have
exactly one implementation with that implementation. This makes sure
our tests are pointing at the right classes, and makes finding relevant
code a little easier.
As preparation for #106081, this PR adds the `size_in_bytes`
field to the enrich cache. This field is calculated by summing
the ByteReference sizes of all the search hits in the cache.
It's not a perfect representation of the size of the enrich cache
on the heap, but some experimentation showed that it's quite close.
Add use of Opaque ID HTTP header in task cancellation assertion. In some
tests, like this #88201 `testCatSegmentsRestCancellation`, we assert
that all tasks related to specific HTTP request are cancelled. But we do
blanket approach in assertion block catching all tasks by action name. I
think narrowing down assertion to specific http request in this case
would be more accurate.
It is still not clear why test mentioned above failing, but after hours
of investigation and injecting random delays, I'm inclining more to
@DaveCTurner's comment about interference from other tests or cluster
activity. I added additional log that will report when we spot task with
different opaque id.
This commit moves the file preallocation functionality into
NativeAccess. The code is basically the same. One small tweak is that
instead of breaking Java access boundaries in order to get an open file
handle, the new code uses posix open directly.
relates #104876
Min/max range for the event.ingested timestamp field (part of Elastic Common
Schema) was added to IndexMetadata in cluster state for searchable snapshots
in #106252.
This commit modifies the search coordinator to rewrite searches to MatchNone
if the query searches a range of event.ingested that, from the min/max range
in cluster state, is known to not overlap. This is the same behavior we currently
have for the @timestamp field.
Resolves https://github.com/elastic/elasticsearch/issues/104323
This fixes and adds tests for the first of the two bullets in the linked
issue. `ExpressionBuilder#visitIntegerValue` will attempt to parse a
string as an integral value, and return a Literal of the appropriate
type. The actual parsing happens in `StringUtils#parseIntegral`. That
function has special handling for values that are larger than
`Long.MAX_VALUE` where it attempts to turn them into unsigned longs, and
if the number is still out of range, throw `InvalidArgumentException`.
`ExpressionBuilder` catches that `InvalidArgumentException` and tries to
parse a `double` instead. If, on the other hand, the value is smaller
than `Long.MIN_VALUE`, `StringUtils` never enters the unsigned long path
and just calls `intValueExact`, which throws `ArithmeticException`.
This PR solves the issue by catching that `ArithmeticException` and
rethrowing it as an `InvalidArgumentException`.
* Fix put inference endpoint with adaptive allocations
* Better validation of adaptive allocations settings in inference endpoints
* Safeguard for max allocations
* remove debug code
This copies the first line of the description of each command to just
under the syntax so that it's "in order", before the `Parameters`
section. That way if you are reading from top to bottom you see:
```
syntax
short description
parameter names and descriptions
long description
examples
```
I've also removed the `Description` section entirely if the description
was just one sentence. So in some cases that just isn't `long
description`.
This adds some basic documentation for the `profile` option in ESQL but
doesn't really explain the results beyond "this is for human debugging."
We're not ready for any kind of specification for this thing, but it is
useful to look at.
this commit refactors the metering for billing api so that we can hide the implementation details of DocumentSizeObserver creation and adds additional field `originatesFromScript` on IndexRequest
There will no longer need to have a code checking if the request was already parsed in ingest service or updatehelper. This logic will be hidden in the implementation.
The min num replicas for the enrich index is currenty hardcoded to 0. We'd like to be able
to override that value via a setting (currently not registered), like we already do for the downsample index.
Fixes https://github.com/elastic/elasticsearch/issues/105017
A quick summary:
- `randomizedtesting` library doesn't handle "+" chars in test names
- We currently have the "...#[skip:..., ...]" instructions as part of the test name
As removing the instructions from the name fixes both issues (And makes launching a single test in CLI far easier), the instructions have been moved out of the test name to another parameter.
Before:
```
gradlew ':x-pack:plugin:esql:test' --tests "org.elasticsearch.xpack.esql.CsvTests" -Dtests.method="test {stats.GroupByNull#[skip:-8.12.99,reason:bug fixed in 8.13+]}"
```
After:
```
gradlew ":x-pack:plugin:esql:test" --tests "org.elasticsearch.xpack.esql.CsvTests" -Dtests.method="test {stats.GroupByNull}"
```
This change allows querying the `index.mode` setting via a new
`_index_mode` metadata field, enabling APIs such as `field_caps` or
`resolve_indices` to target indices that are either time_series or logs
only. This approach avoids adding and handling a new parameter for
`index_mode` in these APIs. Both ES|QL and the `_search` API should also
work with this new field.
Ensures that the value provided by the `runtime.java` system property,
or `RUNTIME_JAVA_HOME` environment variable is respected, even when it
is the same as the current `JAVA_HOME`. The previous logic had
`isRuntimeJavaHomeSet` return `false` when the "requested" Java was the
same as the current one. This isn't strictly correct, as the behavior
when Java home is unset is to use the bundled JDK. The result was that
passing `-Druntime.java=17` when the Gradle daemon was also using Java
17 was to execute tests with the bundled JDK, which could be something
entirely different.
Previously you had to remember to use `EsqlAggregate` and
`EsqlUnresolvedRelation` instead of `Aggregate` and
`UnresolvedRelation`. This merges the `Esql*` extensions into their
plainly named extensions for easier hacking.
I have recently fixed invocation of `maybeTriggerAsyncJob` in this test
but now invocation count does not match since it counts every _attempt_
to start a job even if it was not successful.
Reserved cluster state is build from handlers that modify cluster state.
In the past there was one "non-state" handler that stored it's state
outside cluster state. However, with that use case now gone, non-state
transforms are no longer needed. This commit removes support for
non-state transforms.
This change moves the handling of the retriever to the rewrite phase. It also adds validation of the search source builder after extracting the retriever into the source builder.
Relates #110482
This feature flag has been enabled by default for snapshot builds for
some time, no significant bumps or changes in rally.
This commit will enable it by default, even in release builds.
Just moving some code around. This pulls the Optimizer Rule unit tests out of the OptimizerRulesTests file and into individual class test files. As usual, this will reduce the risk of git conflicts when editing tests. It'll also help add visibility to which rules have unit tests and which still need them written.
- Added support for Booleans on Max and Min
- Added some helper methods to BitArray (`set(index, value)` and `fill(from, to, value)`). This way, the container is more similar to other BigArrays, and it's easier to work with
Part of https://github.com/elastic/elasticsearch/issues/110346, as Max
and Min are dependencies of Top.
* Fix upgrade tests by accounting for breaking change of renaming "models" to "endpoints"
* improvements from review
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Update stack monitoring template for .monitoring-beats-mb to include latest apm-server monitoring metrics. All stack monitoring apm-server metrics references in kibana should be intact. To avoid breaking stack monitoring UI, although beat.stats.apm_server.server.response.errors.concurrency is unused and is not present in apm-server stats, it is manually kept in the mapping.
We still get too many cases about snapshot repositories which claim to
be S3-compatible but then fail repository analysis because of genuine
incompatibilities or implementation bugs. The info is all there in the
manual but it's not very easy to find. This commit adds more detail to
the response message, including docs links, to help direct users to the
information they need.
A query like:
```
FROM sample_data, sample_data_str
| STATS count=count(*) BY client_ip = TO_IP(client_ip)
| SORT count DESC, client_ip ASC
| KEEP count, client_ip
```
Failed due to unresolved aggregates from the union-type in the grouping key
We've already removed Windows-specific BWC jobs in our periodic
pipelines. They shouldn't behave differently and are very prone to
timeouts so let's just remove them from pull requests when the
`test-windows` label is added.
This commit moves the system call filtering initialization into
NativeAccess. The code is essentially unmodified from its existing
state, now existing within the *NativeAccess implementations.
relates #104876
A quick change to help debug search requests performed by the validate
and preview API.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Some of our tests use an exchange timeout that is too short, causing the
exchange sinks to expire before the fetch page requests arrive. This
change adjusts the exchange timeout to between 3 and 4 seconds, which
should be sufficient without increasing the execution time of the
disruption tests.
Closes#109944Closes#106641
This moves the `Failueres` and `Failure` class into ESQL's main package,
further slimming down our custom fork of the shared ql code. Slowly
slowly slowly, it will be no more.
The types from com.nimbusds.jwt are almost not needed in x-pack/plugin/core.
They're only needed in module org.elasticsearch.security, x-pack:plugin:security project.
While working on elastic/ml-cpp#2677, I encountered a failure in the integration test DetectionRulesIt.testCondition(). It checks the number of return records. With the new change in ml-cpp the native code returns two more values that have no significant score. I added filtering those out in the integration test code so it continues working as expected.
`MAX()` currently doesn't work with doubles smaller than
`Double.MIN_VALUE` (Note that `Double.MIN_VALUE` returns the smallest
non-zero positive, not the smallest double).
This PR adds tests for Max and Min, and fixes the bug (Detected by the
tests).
Also, as the tests now generate the docs, replaced the old docs with the
generated ones, and updated the Max&Min examples.
Some work around aggregation tests, with AVG as an example:
- Added tests and autogenerated docs for AVG
- As AVG uses "complex" surrogates (A combination of functions), we can't trivially execute them without a complete plan. As I'm not sure it's worth it for most aggregations, I'm skipping those cases for now, as to avoid blocking other aggs tests.
The bad side effect of skipping those tests is that most tests in AvgTests are actually ignored (74 of 100)
This commit adds an implement of XOR bit count computed over signed bytes that is ~4x faster than that of Lucene 9.11, on ARM.
While already fixed in Lucene, it'll be in a Lucene version > 9.11. This is effectively a temporary workaround until Lucene 9.12, after which we can revert this.
* Added union-types field extration to ordinals aggregation
* Revert previous approach to getting union-types working in aggregations
Where the grouping field is erased by later commands, like a subsequent stats.
Instead we include union-type supports in the ordinals aggregation and mark the block loader as not supporting ordinals.
We can use a similar strategy to what worked with mappers+settings and
reuse the string deduplicator to deal with a large chunk (more than 70%
from heap dumps we've seen in production) of the `FieldInfo`
duplication overhead without any Lucene changes. There's generally only
a very limited number of attribute maps out there and the "dedup up to
100" logic in here deals with all scenarios I have observed in the wild
thus far. As a side effect of deduplicating the field name and always
working with an interned string now, I would expect the performance of
field caps filtering for empty fields to improve measurably.
Follow up to #110552, add utility for starting tasks at the same time
via a barrier as discussed there. Also, make use of the new tooling in a
couple more spots to save LoC and thread creation.
Specifically around pausing shard snapshots on
node removal, and finalizing shards snapshots
that change the shard generation of other non-
finalized snapshots
Relates ES-8566.
CCS tests could split the vectors over any number of shards. Through
empirical testing, I determined this commits values work to provide the
expected order, even if they are not all part of the same shard.
quantization can have weird behaviors when there are uniform values,
just like this test does.
closes#109978
* Passing inference input through to 3rd party request classes
* Plumbing changes through the rest of the support integrations
* Clean up
* Including input changes with google vertex
* Backing out semantic query changes
* Addressing feedback and adding aws
* Prevent inference endpoints from being deleted if they are referenced by a semantic text field
* Update docs/changelog/110399.yaml
* fix tests
* remove erroneous loging
* Apply suggestions from code review
Co-authored-by: David Kyle <david.kyle@elastic.co>
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
* Fix serialization problem
* Update error messages
* Update Delete response to include new fields
* Refactor Delete Transport Action to return the error message on dry run
* Fix tests including disabling failing yaml tests
* Fix YAML tests
* move work off of transport thread onto utility threadpool
* clean up semantic text indexes after IT
* improvements from review
---------
Co-authored-by: David Kyle <david.kyle@elastic.co>
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
closes https://github.com/elastic/elasticsearch/issues/110357
With the loosening of what is considered a unit vector, we need to
ensure we only normalize for equality checking if the query vector is
indeed not a unit vector.
* Only resolve latest patch version for resolveAllDependencies
This should avoid downloading to many elasticsearch distributions
and reduce disk usage and speed up image creations.
* Some cleanup
In service of the incoming INLINESTATS this flips the ordering of
analysis. Previously we made the entire sequence of analyze, optimize,
convert to physical plan, and optimize in a single async sequence in
`EsqlSession`. This flips it so `analyze` comes first in it's own async
sequence and then runs the remaining stuff in a separate sequence.
That's nice for INLINESTATS where we want to analyze one time, and then
run many runs of the extra sequence.
While we're here, we also take that sequence call it directly from the
CsvTests. That works well because that final sequence is exactly what
CsvTests have to do. They "analyze" totally differently, but they run
the final sequence in the same way.
Closes#107953
* Allow auto-generated type-cast fields in CsvTests
This allows, for example, a csv-spec test result header like `client_ip::ip:ip`, which is generated with a command like `STATS count=count(*) BY client_ip::ip`
It is also a small cleanup of the header parsing code, since it was using Strings.split() in an odd way.
* Fix bug in union-types with type-casting in grouping key of STATS
* Update docs/changelog/110476.yaml
* Added casting_operator required capability
Using the new `::` syntax requires disabling support for older versions in multi-cluster tests.
* Added more tests for inline stats over long/datetime
* Trying to fix the STATS...STATS bug
This makes two changes:
* Keeps the Alias in the aggs.aggregates from the grouping key, so that ReplaceStatsNestedExpressionWithEval still works
* Adds explicit support for union-types conversion at grouping key loading in the ordinalGroupingOperatorFactory
Neither fix the particular edge case, but do seem correct
* Added EsqlCapability for this change
So that mixed cluster tests don't fail these new queries.
* Fix InsertFieldExtract for union types
Union types require a FieldExtractExec to be performed first thing at
the bottom of local physical plans.
In queries like
```
from testidx*
| eval x = to_string(client_ip)
| stats c = count(*) by x
| keep c
```
The `stats` has the grouping `x` but the aggregates get pruned to just
`c`. In cases like this, we did not insert a FieldExtractExec, which
this fixes.
* Revert query that previously failed
With Alex's fix, this query now passes.
* Revert integration of union-types to ordinals aggregator
This is because we have not found a test case that actually demonstrates this is necessary.
* More tests that would fail without the latest fix
* Correct code style
* Fix failing case when aggregating on union-type with invalid grouping key
* Capabilities restrictions on the new YML tests
* Update docs/changelog/110476.yaml
---------
Co-authored-by: Alexander Spies <alexander.spies@elastic.co>
Replaces the deeply-nested listener stack with a `SubscribableListener`
sequence to clarify the process, particularly its failure handling. Also
adds assertions that each step does not double-complete its listener.
These warning logs and error messages assume some level of understanding
of Elasticsearch's networking config and are not particularly
actionable. This commit adds links to the relevant section of the
manual, rewords them a little to match the terminology used in the
manual, and also documents that each node must have its own publish
address, distinct from those of all other nodes.
Now that #102351 is fixed there won't be ≥10 layers of exceptions to
unwrap here so we can go back to using the regular test utilities for
accessing the root cause.
Relates #102352 Relates #102348
We have the same pattern in a bunch of places, I dried up a few here. We
want to run N tasks so we create N threads in a loop, start them and
join them right away. The node starting logic refactored here is
essentially the same since the threads have idle lifetime 0. This can be
dried up a little and made more efficient. Might as well always use
`N-1` tasks and run one of them on the calling thread. This saves quite
a few threads when running tests and speeds things up a little,
especially when running many concurrent Gradle workers and CPU is at
100% already (mostly coming from the speedup on starting nodes this
brings and the reduction in test thread sleeps).
No functional changes to the tests otherwise, except for some replacing
of `CountDownLatch` with `CyclicalBarrier` to make things work with the
new API.
Updates the name of the transport CA in security autoconfiguration.
Previously both the HTTP and Transport CAs had the same name
(`CN=Elasticsearch security auto-configuration HTTP CA`).
The transport CA now has a different name
(`CN=Elasticsearch security auto-configuration transport CA`).
Closes: #106455
Co-authored-by: Aditya Kukankar <adityakukankar@gmail.com>
Co-authored-by: Tim Vernum <tim@adjective.org>
Currently, read_slm privilege grants access to get the ILM status,
and manage_slm grants access to start/stop ILM. This access will
be removed in the future, but needs to be deprecated before
removal. Add deprecation warning to the read_slm and manage_slm docs.
Currently, if a child request fails, we automatically trigger cancellation
for ES|QL requests. This can result in TaskCancelledException being
collected by the RefCountingListener first, which then returns that
exception to the caller. For example, if we encounter a
CircuitBreakingException (429), we might incorrectly return a
TaskCancelledException (400) instead.
This change introduces the ComputeListener, a variant of
RefCountingListener, which selects the most appropriate exception to return
to the caller. I also integrated the following features into ComputeListener to
simplify ComputeService:
- Automatic cancellation of sub-tasks on failure.
- Collection of profiles from sub-tasks.
- Collection of response headers from sub-tasks.
Aquiring a buffer is rather expensive and we use a buffer of constant
size throughout, lets leverage this fact and save contention on the
allocator.
Also, we can hoist the filter size calculation out of the loop and do
the write to the index output without without grabbing the file pointer
or allocating any bytes in msot cases.
* Initial commit; setup Gradle; start service
* initial commit
* minor cleanups, builds green; needs tests
* bug fixes; tested working embeddings & completion
* use custom json builder for embeddings request
* Ensure auto-close; fix forbidden API
* start of adding unit tests; abstraction layers
* adding additional tests; cleanups
* add requests unit tests
* all tests created
* fix cohere embeddings response
* fix cohere embeddings response
* fix lint
* better test coverage for secrets; inference client
* update thread-safe syncs; make dims/tokens + int
* add tests for dims and max tokens positive integer
* use requireNonNull;override settings type;cleanups
* use r/w lock for client cache
* remove client reference counting
* update locking in cache; client errors; noop doc
* remove extra block in internalGetOrCreateClient
* remove duplicate dependencies; cleanup
* add fxn to get default embeddings similarity
* use async calls to Amazon Bedrock; cleanups
* use Clock in cache; simplify locking; cleanups
* cleanups around executor; remove some instanceof
* cleanups; use EmbeddingRequestChunker
* move max chunk size to constants
* oof - swapped transport vers w/ master node req
* use XContent instead of Jackson JsonFactory
* remove gradle versions; do not allow dimensions
Two PRs passing in the night, breaking each other.
This passes precommit locally but I haven't double checked tests. But at
least Elasticsearch compiles again.
INLINESTATS is going to run two ESQL commands - one to get the STATS and
one to join the stats results to the output. This plumbs a way for
`EsqlSession#execute` to run multiple dips into the compute engine using
a `BiConsumer<PhysicalPlan, ActionListener<Result>> runPhase`. For now,
we just plug that right into the output to keep things working as they
are now. But soon, so soon, we'll plug in a second phase.
Bit vector tests were failing in cases where an index has more than 1 shards.
For error cases when we expected a failure of the whole request, shards
with empty documents returned success and the whoel request unexpectedly returned 200.
Ensuring that index contains only 1 shard fixes these failures.
Closes#110290, #110291
The model loading scheduled thread iterates through the model
queue and deploys each model. Rather than block and wait on each
deployment, the thread will attach a listener that will either iterate
to the next model (if one is in the queue) or reschedule the thread.
This change should not impact:
1. the iterative nature of the model deployment process - each model is
still deployed one at a time, and no additional threads are consumed
per model.
2. the 1s delay between model deployment tries - if a deployment fails
but can be retried, the retry is added to the next batch of models
that are consumed after the 1s scheduled delay.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: David Kyle <david.kyle@elastic.co>
We have witnessed some test failures in `DataStreamUpgradeRestIT`,
`EcsLogsDataStreamIT` and `LogsDataStreamIT` during which the `logs`
related index or component template gets initialised right after the 10
seconds have passed. We increase the timeout to make the test more
resilient to this scenario.
This PR was reviewed in #109275
Block and Vector use a non-thread-safe RefCounted. Threads that increase
or decrease the references must have a happen-before relationship.
However, this order is not guaranteed in the enrich lookup for the
reference of selectedPositions. The driver can complete the
MergePositionsOperator, which decreases the reference count of
selectedPositions, while the finally block may also decrease it in a
separate thread. These actions occur without a defined happen-before
relationship.
Closes#108532
With biased locking gone, we see some slowness in profiling
when we use this stream for single byte reads.
This is a recent regression that is a result of
https://openjdk.org/jeps/374.
-> the sychronization overhead for bulk reads hardly matters, but since
we do quite a few single byte reads lets fix this.
The number of replicas for the downsample index gets set to 0 by default (overridable via setting) and later incremented to a higher value.
This is done unconditionally, but in reality if the downsample index already has replicas, we should not override its number of replicas.
Closes#109968
When we added arrow support we needed to add a security permission to
hack around their initialization code. That confused intellij running
tests. This removes the confusion by manually resolving the location.
It's confusing that we have a type called `integer` and we have a method
called `isInteger` which returns `true` for `integer` *and* `long`. This
renames that method to `isWholeNumber`. It also renames `isRational` to
`isRationalNumber` to line up.
This merges the `FunctionRegistry` from `esql-core` into the
`EsqlFunctionRegistry` in `esql` proper in an effort to shave down the
complexity we got by attempting to share lots of infrastructure with SQL
and KQL. Since we no longer share we can compress these two together.
With this change, we will enable rounding for double values in the
single-node QA module in serverless tests, while keeping it disabled
in stateful tests.
Fix ReplaceMissingFieldWithNull by explicitly listing the commands that
can be optimized replacing missing FieldAttributed with NULL Literals.
Also adding a unit test that demonstrates possible scenarios where
introducing a new command can lead to `ClassCastException` with
`ReplaceMissingFieldWithNull` local optimization rule and an integration
test that covers https://github.com/elastic/elasticsearch/issues/109974Fixes#110150
Today if there's a failure during a multi-repo get-snapshots request
then we record a per-repository failure but allow the rest of the
request to proceed. This is trappy for clients, it means that they must
always remember to check the `failures` response field or else risk
missing some results. It's also a pain for the implementation because it
means we have to collect the per-repository results separately first
before adding them to the final results set just in case the last one
triggers a failure.
This commit drops this leniency and bubbles all failures straight up to
the top level.
This adds some javadoc to a few of the methods on `DataType`. That's
important because `DataType` is pretty central to how ESQL works and is
referenced in tons of code. The method `isInteger` especially wants an
explanation - it's true for all whole numbers.
* Fix missing cases for ST_DISTANCE Lucene pushdown
The feature to pushdown ST_DISTANCE to Lucene was not working when combined with OR and NOT clauses, or more deeply nested. This fixes that by traversing the predicate tree recursively.
* Update docs/changelog/110391.yaml
* Fixed changelog
Initial support for Apache Arrow's streaming format as a response for ES|QL. It triggers based on the Accept header or the format request parameter.
Arrow has implementations in every mainstream language and is a backend of the Python Pandas library, which is extremely popular among data scientists and data analysts. Arrow's streaming format has also become the de facto standard for dataframe interchange. It is an efficient binary format that allows zero-cost deserialization by adding data access wrappers on top of memory buffers received from the network.
This PR builds on the experiment made by @nik9000 in PR #104877
Features/limitations:
- all ES|QL data types are supported
- multi-valued fields are not supported
- fields of type _source are output as JSON text in a varchar array. In a future iteration we may want to offer the choice of the more efficient CBOR and SMILE formats.
Technical details:
Arrow comes with its own memory management to handle vectors with direct memory, reference counting, etc. We don't want to use this as it conflicts with Elasticsearch's own memory management.
We therefore use the Arrow library only for the metadata objects describing the dataframe schema and the structure of the streaming format. The Arrow vector data is produced directly from ES|QL blocks.
---------
Co-authored-by: Nik Everett <nik9000@gmail.com>
Closes#105166
Adding tests that verify that `BUCKET` (previously `AUTO_BUCKET`)
function does not return `NaN` when an invalid number of buckets is
provided (eg. 0, -1 or a very large integer)
It's not enough to wait for the tasks to appear in the pending tasks
queue, we must also then wait for the submitting threads to become idle
to ensure that the queue size is correct and therefore that the batch is
properly formed.
Closes#109187
#110157 broke the serialization of ToPartial and FromPartial, where the
parameters weren't properly assigned.
I will look into making these classes more compatible with
AggregateFunction, but this small change should stabilize the CI.
Relates #110157
This moves all of `Expression` serialization from ESQL's home grown
system into `NamedWriteable` which is how the rest of Elasticsearch
works. We previously implemented `NamedWriteable` on all `Expression`
subclasses and tested it. This just switches all production
serialization over. It is wire compatible with the old custom way, so no
one should notice anything other than less custom code.
This adds a `NOTE` to each comparison saying that pushing the comparison
to the search index requires that the field have an `index` and
`doc_values`. This is unique compared to the rest of Elasticsearch which
only requires an `index` and it's caused by our insistence that
comparisons only return true for single-valued fields. We can in future
accelerate comparisons without `doc_values`, but we just haven't written
that code yet.
LOOKUP isn't supported in release builds yet and it'll fail with a
helpful error message if you try it there. But some of the csv-spec
tests didn't realize that. Lots did, but these particular ones didn't.
Close#109170
As discussed in #107555 there's some overhead to the use of `ReleasableLock`
and it's probably not a good idea using it on the hot(ish) path in this class.
Some suggestions from a post-merge review of #109899:
- Sometimes run on multi-node cluster.
- Use `SAFE_AWAIT_TIMEOUT` for waits.
- Remove unnecessary `assertBusy()`
* Add an index setting that captures the version of the failure store definition
* I wish my PRs would precommit-fix themselves
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Segragate sys prop dependent tests by gradle tasks
* Add dependency to gradle check task + style
* Update server/src/test/java/org/elasticsearch/index/IndexSettingsOverrideTests.java
Co-authored-by: Yang Wang <ywangd@gmail.com>
---------
Co-authored-by: Yang Wang <ywangd@gmail.com>
* Reapply "Support ST_DISTANCE with EQ predicate"
This reverts commit 1deb7b76ba.
* Quantize points before calculating distance, and add more tests
Curiously I was not able to reproduce any difference in behaviour between Lucene (pushdown) and ESQL distance calculation, even when focusing in on same data generated to be on the edge of the quantization boundaries and the haversinDistance boundary.
This adds the Query Roles API:
```
POST /_security/_query/role
GET /_security/_query/role
```
This is similar to the currently existing: * [Query API key
API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-query-api-key.html)
* [Query User
API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-query-user.html)
Sample request:
```
POST /_security/_query/role
{
"query": {
"bool": {
"filter": [
{
"terms": {
"applications.application": ["app-1", "app-2" ]
}
}
],
"must_not": [
{
"match": {
"description": {
"query": "test match on role description (which is mapped as a text field)"
}
}
}
]
}
},
"sort": [
"name"
],
"search_after": [
"role-name-1"
]
}
```
The query supports a subset of query types, including match_all, bool,
term, terms, match, ids, prefix, wildcard, exists, range, and simple
query string.
Currently, the supported fields are: * name * description * metadata
* applications.application * applications.resources *
applications.privileges
The query also supports pagination-related fields (`from`, `size`,
`search_after`), analogous to the generic [Search
API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html).
The response format is similar to that of the [Query API
key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-query-api-key.html)
and [Query
User](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-query-user.html)
APIs. It contains a **list** of roles, in the sorted order (if
specified). Unlike the [Get Roles
API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-get-role.html),
the role **name** is an attribute of the element in the list of roles
(in the get-roles API case, the role name was the key in the response
map, and the value was the rest of the role descriptor). In addition,
the element in the list of roles also contains the optional `_sort`
field, eg (sample response):
```
{
"total": 3,
"count": 3,
"roles": [
{
"name": "LYdz2",
"cluster": [],
"indices": [],
"applications": [
{
"application": "ejYWvGQTF",
"privileges": [
"pRCfBMgOy",
"zDhFtMQfc",
"roudxado"
],
"resources": [
"nWHEpmgxy",
"SOML/hMYrqx",
"YIqP/*",
"ueEomwsA"
]
},
{
"application": "ampUW9",
"privileges": [
"jDvRtp"
],
"resources": [
"99"
]
}
],
"run_as": [],
"metadata": {
"nFKc": [
1,
0
],
"PExF": [],
"qlqY": -433239865,
"IQXm": []
},
"transient_metadata": {
"enabled": true
},
"description": "KoLlsEbq",
"_sort": [
"LYdz2"
]
},
{
"name": "oaxW0",
"cluster": [],
"indices": [],
"applications": [
{
"application": "*",
"privileges": [
"qZYb"
],
"resources": [
"tFrSULaKb"
]
},
{
"application": "aLaEN9",
"privileges": [
"fCOc"
],
"resources": [
"gozqXtSgE",
"UX/JgydeIM",
"sjUp",
"Ivdz/UAmuNrQAG"
]
},
{
"application": "rbxyuKIMPAp",
"privileges": [
"lluqieFRu",
"xKU",
"gHlb"
],
"resources": [
"99"
]
}
],
"run_as": [],
"metadata": {},
"transient_metadata": {
"enabled": true
},
"_sort": [
"oaxW0"
]
},
{
"name": "vWAV1",
"cluster": [],
"indices": [],
"applications": [
{
"application": "*",
"privileges": [
"kWBWjCAc"
],
"resources": [
"hvEtV",
"gZJ"
]
},
{
"application": "avVUV9",
"privileges": [
"newZTa",
"gQpxNm"
],
"resources": [
"99"
]
}
],
"run_as": [],
"metadata": {},
"transient_metadata": {
"enabled": true
},
"_sort": [
"vWAV1"
]
}
]
}
```
* parseHeapRatioOrDeprecatedByteSizeValue for indices.breaker.total.limit
* Fix tests for indices.breaker.total.limit warning
* Spotless
* Warn if below minimum percent
* Update docs/changelog/110236.yaml
* Changelog
* Pick correct area for changelog entry
* Spotless again dammit
* assertCriticalWarnings in circuit breaker test
* Expect another warning
The ColumnInfo class, which is used for request and response serialization, stores a data type. Prior to this PR, it stored that type as a String, and there were several places (seen below) where we needed to switch behavior based on that string. Switching on strings is brittle, as there's no way for the compiler to enforce that all cases are covered, so we have to rely on a default -> throw pattern, and hope that a test catches that path. On the other hand, we can instead switch on the actual DataType enum value, which the compiler can then enforce checking all values.
Eventually, it would make sense for most or all of these switches to become functions on DataType, but the visibility between esql.core and esql right now makes that difficult.
The index.mode setting validates other index settings. When updating the index.time_series.end_time setting and the index.mode setting isn't wasn't defined at index creation time (meaning that default is active), then this validation is skipped which results into (worse) errors at a later point in time.
This problem is fixed by enforced by making index.mode setting a dependency of index.time_series.end_time setting.
Note that this problem doesn't exist for the index.time_series.start_time and index.routing_path index settings, because these index settings are final, which mean these can only be defined when an index is being created.
Closes#110265
At the moment, it's not possible to set `ignore_above` on `keyword`
dimension fields. This flag is used a lot in our integrations and in
ECS. It helps to reduce the number of logs/metrics we need to reject.
This is also somewhat inconsistent as it's possible to set
`ignore_malformed` on numeric dimension fields. Both options end up
adding malformed values to `_ignored` and don't add doc_values and
postings.
I'd like to propose adding support for `ignore_above` to be consistent
with allowing `ignore_malformed` on numeric fields. The other option
would be to take away the support for `ignore_malformed` but that seems
to be a breaking change.
I've added a test that verifies that ignoring those dimensions is safe
in the sense that it doesn't lead to document rejections due to
duplicate detection if the only dimensions that differ are ignored.
Update TransportGetShutdownStatusAction so that it can not trip circuit breaker.
If circuit breaker is tripped when orchestrator is checking the shutdown status
then cluster is shutdown prematurely, leading to availability lose.
@ -154,9 +154,11 @@ if (request.getRestApiVersion() == RestApiVersion.V_7 && request.hasParam("limit
The above code checks the request's compatible version and if the request has the parameter in question. In this case the deprecation warning is not automatic and requires the developer to manually log the warning. `request.param` is also required since it consumes the value as to avoid the error of unconsumed parameters.
### Testing
### Testing Backwards Compatibility
The primary means of testing compatibility is via the prior major version's YAML REST tests. The build system will download the latest prior version of the YAML rest tests and execute them against the current cluster version. Prior to execution the tests will be transformed by injecting the correct headers to enable compatibility as well as other custom changes to the tests to allow the tests to pass. These customizations are configured via the build.gradle and happen just prior to test execution. Since the compatibility tests are manipulated version of the tests stored in Github (via the past major version), it is important to find the local (on disk) version for troubleshooting compatibility tests.
The primary means of testing compatibility is via the prior major version's YAML REST tests. The build system will download the latest prior version of the YAML rest tests and execute them against the current cluster version. For example if you are testing main versioned as 9.0.0 the build system will download the yaml tests in the 8.x branch and execute those against the current cluster version for 9.0.0.
Prior to execution the tests will be transformed by injecting the correct headers to enable compatibility as well as other custom changes to the tests to allow the tests to pass. These customizations are configured via the build.gradle and happen just prior to test execution. Since the compatibility tests are manipulated version of the tests stored in Github (via the past major version), it is important to find the local (on disk) version for troubleshooting compatibility tests.
The tests are wired into the `check` task, so that is the easiest way to test locally prior to committing. More specifically the task is called `yamlRestCompatTest`. These behave nearly identical to it's non-compat `yamlRestTest` task. The only variance is that the tests are sourced from the prior version branch and the tests go through a transformation phase before execution. The transformation task is `yamlRestCompatTestTransform`.
@ -170,6 +172,36 @@ Since these are a variation of backward compatibility testing, the entire suite
In some cases the prior version of the YAML REST tests are not sufficient to fully test changes. This can happen when the prior version has insufficient test coverage. In those cases, you can simply add more testing to the prior version or you can add custom REST tests that will run along side of the other compatibility tests. These custom tests can be found in the `yamlRestCompatTest` sourceset. Custom REST tests for compatibility will not be modified prior to execution, so the correct headers need to be manually added.
#### Breaking Changes
It is possible to be in a state where you have intentionally made a breaking change and the compatibility tests will fail irrespective of checks for `skip` or `requires` cluster or test features in the current version such as 9.0.0. In this state, assuming the breaking changes are reasonable and agreed upon by the breaking change committee, the correct behavior is to skip the test in the `build.gradle` in 9.0.0. For example, if you make a breaking change that causes the `range/20_synthetic_source/Date range` to break then this test can be disabled temporarily in this file `rest-api-spec/build.gradle` like within this snippet:
task.skipTest("range/20_synthetic_source/Date range", "date range breaking change causes tests to produce incorrect values for compatibility")
task.skipTest("indices.sort/10_basic/Index Sort", "warning does not exist for compatibility")
task.skipTest("search/330_fetch_fields/Test search rewrite", "warning does not exist for compatibility")
task.skipTestsByFilePattern("indices.create/synthetic_source*.yml", "@UpdateForV9 -> tests do not pass after bumping API version to 9 [ES-9597]")
})
```
When skipping a test temporarily in 9.0.0, we have to implement the proper `skip` and `requires` conditions to previous branches, such as 8.latest. After these conditions are implemented in 8.latest, you can re-enable the test in 9.0.0 by removing the `skipTest` condition.
The team implementing the changes can decide how to clean up or modify tests based on how breaking changes were backported. e.g.:
In 8.latest:
* Add `skip` / `requires` conditions to existing tests that check the old behavior. This prevents those tests from failing during backward compatibility or upgrade testing from 8.latest to 9.0.0
In 9.0.0:
* Add `requires` conditions for new tests that validate the updated API or output format
* Add `skip` conditions for older tests that would break in 9.0.0
#### Test Features
Both cluster and test features exist. Cluster features are meant for new capability and test features can specifically be used to gate and manage `skip` and `requires` yaml test operations. For more information, see [Versioning.md](docs/internal/Versioning.md#cluster-features). When backporting and using these features they can not overlap in name and must be consistent when backported so that clusters built with these features are compatible.
### Developer's workflow
There should not be much, if any, deviation in a developers normal workflow to introduce and back-port changes. Changes should be applied in main, then back ported as needed.
@ -82,19 +82,21 @@ To get realistic results, you should exercise care when running benchmarks. Here
NOTE: Linux only. Sorry Mac and Windows.
Disassembling is fun! Maybe not always useful, but always fun! Generally, you'll want to install `perf` and the JDK's `hsdis`.
`perf` is generally available via `apg-get install perf` or `pacman -S perf`. `hsdis` you'll want to compile from source. is a little more involved. This worked
`perf` is generally available via `apg-get install perf` or `pacman -S perf linux-tools`. `hsdis` you'll want to compile from source. is a little more involved. This worked
