Keep better track of shard contexts using RefCounted, so they can be released more aggressively during operator processing. For example, during TopN, we can potentially release some contexts if they don't pass the limit filter.
This is done in preparation of TopN fetch optimization, which will delay the fetching of additional columns to the data node coordinator, instead of doing it in each individual worker, thereby reducing IO. Since the node coordinator would need to maintain the shard contexts for a potentially longer duration, it is important we try to release what we can eariler.
An even more advanced optimization is to delay fetching to the main cluster coordinator, but that would be more involved, since we need to first figure out how to transport the shard contexts between nodes.
Summary of main changes:
DocVector now maintains a RefCounted instance per shard.
Things which can build or release DocVectors (e.g., LuceneSourceOperator, TopNOperator), can also hold RefCounted instances, so they can pass them to DocVector and also ensure contexts aren't released if they can still be potentially used later.
Driver's main loop iteration (runSingleLoopIteration), now closes its operators even between different operator processing. This is extra aggressive, and was mostly done to improve testability.
Added a couple of tests to TopNOperator and a new integration test EsqlTopNShardManagementIT, which uses the pausable plugin framework to check that TopNOperator releases things as early as possible..
This PR makes RepositoriesService project aware so that the basic Put,
Get, Delete and Verify repository actions are now project scoped.
It intentionally leaves the following aspects out of scope for the
current changes: * Repository stats reporting * Repository clean-up,
analysis and integrity verification * Repository usages for searchable
snapshots and CCR
They will be worked on separately. One main reason for leaving them out
is that they are not needed by OBS which is currently blocked by
repository/snapshot changes. They may also have their own complexities,
e.g. stats reporting.
Resolves: ES-10478
Relates to an effort to combine the merge schedulers from stateless and stateful. The stateless merge scheduler has MergeMetrics that we want in both stateless and stateful. This PR copies over the merge metrics from the stateless merge scheduler into the combined merge scheduler.
Relates ES-9687
* Initial checkin of refactored index_options code
* [CI] Auto commit changes from spotless
* initial unit testing
* complete unit tests; add yaml tests
* [CI] Auto commit changes from spotless
* register test feature for sparse vector
* Update docs/changelog/129089.yaml
* update changelog
* add docs
* explicit set default index_options if null
* [CI] Auto commit changes from spotless
* update yaml tests; update docs
* fix yaml tests
* readd auth for teardown
* only serialize index options if not default
* [CI] Auto commit changes from spotless
* serialization refactor; pass index version around
* [CI] Auto commit changes from spotless
* fix transport versions merge
* fix up docs
* [CI] Auto commit changes from spotless
* fix docs; add include_defaults unit and yaml test
* [CI] Auto commit changes from spotless
* override getIndexReaderManager for SemanticQueryBuilderTests
* [CI] Auto commit changes from spotless
* cleanup mapper/builder/tests; index vers. in type
still need to refactor / clean YAML tests
* [CI] Auto commit changes from spotless
* cleanups to mapper tests for clarity
* [CI] Auto commit changes from spotless
* move feature into mappers; fix yaml tests
* cleanups; add comments; remove redundant test
* [CI] Auto commit changes from spotless
* escape more periods in the YAML tests
* cleanup mapper and type tests
* [CI] Auto commit changes from spotless
* rename mapping for previous index test
* set explicit number of shards for yaml test
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co>
We add a margin of error when comparing floats to the DynamicFieldMatcher to account for a small loss of accuracy when loading fields from synthetic source.
This PR addresses a bug where aborted merges are blocked if there's
insufficient disk space.
Previously, the merge disk space estimation did not consider if the
operation has been aborted when/while it was enqueued for execution.
Consequently, aborted merges, for e.g. when closing a shard, were
blocked if their disk space estimation was exceeding the available disk
space threshold. In this case, the shard close operation would itself
block.
This fix estimates a disk space budget of `0` for aborted merges, and it
periodically checks if any enqueued merge tasks have been aborted (more
generally, it checks if the budget estimate for any merge tasks has
changed, and reorders the queue if so). This way aborted merges are
prioritized and are never blocked.
Closes https://github.com/elastic/elasticsearch/issues/129335
In the last two months a lot of tests were converted to use the newer rest test framework. Some tests start 1 node, other start 3 nodes, others even more, the framework runs tests in parallel but it doesn't know how many nodes its tests needs meaning that running 3 tests in parallel, for example, can be very different when they are single node clusters or 3 node clusters etc. During this execution we saw the 3x more CPU load than what we would want to have ideally.
Currently there is no good solution for this because if dial down the concurrency we will use the nodes inefficiently, but if we keep the concurrency to where it is we risk longer start up times. Considering that the starting time of elasticsearch is not related to this test, we choose to increase the timeout to reduce the noise.
The only production usage is for cleaning up all global state files. It
is replaced by directly calling the relevant method without creating the
FORAMT instance. Test only usages are either replaced by equivalent
method calls or dropped.
Relates: #114698
We added a new `projectClient` method on `Client` in #129174. We now
update the usages of the old method (on `ProjectResolver`) to use the
new one and we delete the old method.
For APM logs, set event.dataset to data_stream.dataset if event.dataset is empty, to satisfy Anomaly Detection's requirement to have event.dataset in every logs-* data stream.
* increase the number of retries for apt-get
* Update docs/changelog/129295.yaml
* try without sudo
* increase timeout to see if that fixes it
* mention a couple of more tests
* try inline option
* try inline option
* Delete docs/changelog/129295.yaml
* bump this too
* possibly ok test now
As preparation for running the cluster state API on the local node, we
need to update these tests that currently depend on that API running on
(and waiting for) the master node.
Relates #127212
Since #129367 we run the `_ilm/status` API on the local node, which
could cause issues in tests that assume the API runs on the master node
(i.e. they assumed that once the assertion passed, all nodes in the
cluster would have that cluster state, which is not true).
We originally defined the `projectClient` method on `ProjectResolver` as
a convenience method to execute API calls for specific projects. That
method requires a reference to both a `ProjectResolver` and a `Client`.
We now introduce the same method directly on the `Client` interface and
inject a `ProjectResolver` there, removing the need for a
`ProjectResolver` reference in places that just want to execute API
requests on a specific project.
To reduce the number of changes, this change solely focuses on
introducing the new method. Future changes will migrate the uses of the
original method to the new one and remove the original altogether.
This commit adds a Rest IT specifically for search in MultiProject.
Everything was already working as expected, but we were a bit light on
explicit testing for search, which as _the_ core capability of
Elasticsearch is worth testing thoroughly and clearly.
This PR introduces several fixes to various IT tests, related to the use and misuse of the version identifier for the start cluster:
wherever we can, we replace of versions in test code with features
where we can't, we make sure we use the actual stack version (the one provided by -Dtests.bwc.main.version and not the bogus "0.0.0" version string)
when requesting the cluster version we make sure we do use the "unresolved" version identifier (the value of the tests.old_cluster_version system property e.g. 0.0.0 ) so we resolve the right distribution
These changes enabled the tests to be used in BC upgrade tests (and potentially in serverless upgrade tests too, where they would have also failed)
Relates to ES-12010
Precedes #128614, #128823 and #128983
Introduces a new search load metric to the stats infrastructure, measured and tracked on a per-shard basis. The metric represents the Exponentially Weighted Moving Rate (EWMR) of search operations, calculated using the "took" time from each completed search phase.
* Support multiple plugin source paths
* Refactor: remove unncessary PathLookup method.
It's only called in one place, and there's no need to override it for testing.
Removing it just makes things simpler.
* Refactor: local var for pathLookup
* Fix bugs in test build info parsing
* Fix representative_class in test
* Move BridgeUtilTests.
Tests in org.elasticsearch.entitlement.bridge are going to be uniquely hard to
test once we patch the bridge into java.base, due to Java's prohibition on
split packages.
Let's just move this guy to another package.
* Upcast (?!) Java23EntitlementChecker to EntitlementChecker
* Empty TestPathLookup
* Create PolicyManager during bootstrap, allowing us to share initialization
* Use empty component path list instead of null
* Downcast to the class of the check method.
In our unit test, we have a mock checker that doesn't extend
EntitlementChecker, so downcasting to that would require us to needlessly
rework the unit test.
* Fix javadoc typos
Returns an empty shard stats for shard entries where stats were
unavailable in the case where a node has been restarted or left
the cluster. The change adds a 'description' field to the
SnapshotIndexShardStatus class that is used to include a message
indicating why the stats are empty. This change was motivated by
a desire to reduce latency for getting the stats for currently
running snapshots. The stats can still be loaded from the
repository via a _snapshot/<repository>/snapshot/_status call.
Closes ES-10982
Co-authored-by: Dianna Hohensee <artemisapple@gmail.com>
This PR introduces 3 new settings:
indices.merge.disk.check_interval, indices.merge.disk.watermark.high, and indices.merge.disk.watermark.high.max_headroom
that control if the threadpool merge executor starts executing new merges when the disk space is getting low.
The intent of this change is to avoid the situation where in-progress merges exhaust the available disk space on the node's local filesystem.
To this end, the thread pool merge executor periodically monitors the available disk space, as well as the current disk space estimates required by all in-progress (currently running) merges on the node, and will NOT schedule any new merges if the disk space is getting low (by default below the 5% limit of the total disk space, or 100 GB, whichever is smaller (same as the disk allocation flood stage level)).
This adds some testing tools for verifying vector recall and latency
directly without having to spin up an entire ES node and running a rally
track.
Its pretty barebones and takes inspiration from lucene-util, but I
wanted access to our own formats and tooling to make our lives easier.
Here is an example config file. This will build the initial index, run
queries at num_candidates: 50, then again at num_candidates 100 (without
reindexing, and re-using the cached nearest neighbors).
```
[{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 50,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"index_threads" : 4,
"reindex" : true,
"force_merge" : false,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
},
{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 100,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
}
]
```
To execute:
```
./gradlew :qa:vector:checkVec --args="/Path/to/knn_tester_config.json"
```
Calling `./gradlew :qa:vector:checkVecHelp` gives some guidance on how
to use it, additionally providing a way to run it via java directly
(useful to bypass gradlew guff).
Runs a sanity check after loading a block of values. Previously we were
doing a quick check if assertions were enabled. Now we do two quick
checks all the time. Better - we attach information about how a block
was loaded when there's a problem.
Relates to #128959
This ensures we package an aggregation zip with all artifacts we want to publish to maven central as part of a release.
Running zipAggregation will produce a zip file in the build/nmcp/zip folder. The content of this zip is meant to match the maven artifacts we have currently declared as dra maven artifacts.
This removes all non-test usages of
```
Metadata.Builder.putCustom(String type, ProjectCustom custom)
```
And replaces it with appropriate calls to the equivalent method on
`ProjectMetadata.Builder`.
In most cases this _does not_ make the code project aware, but does
reduce the number of deprecated methods in use.
When parsing documents, we receive the document as UTF-8 encoded data which
we then parse and convert the fields to java-native UTF-16 encoded Strings.
We then convert these strings back to UTF-8 for storage in lucene.
This patch skips the redundant conversion, instead passing lucene a
direct reference to the received UTF-8 bytes when possible.
We miss appending null when ignored_source is not available. Our
randomized tests already cover this case, but we do not check it when
loading fields.
I labelled this non-issue for an unreleased bug.
Closes#128959
Relates #119546
The TransportConnectionListener interface has previously included the
Transport.Connection being closed and unregistered in its onNodeDisconnected
callback. This is not in use, and can be removed as it is also available in the
onConnectionClosed callback. It is being replaced with a Nullable exception that
caused the close. This is being used in pending work (ES-11448) to differentiate
network issues from node restarts.
Closes ES-12007
This PR is a precursor to #126492.
It does three things:
1. Move org.elasticsearch.common.text.Text from :server to
org.elasticsearch.xcontent.Text in :libs:x-content.
2. Refactor the Text class to use a new EncodedBytes record instead of
the elasticsearch BytesReference.
3. Add the XContentString interface, with the Text class implementing
that interface.
These changes were originally implemented in #127666 and #128316,
however they were reverted in #128484 due to problems caused by the
mutable nature of java ByteBuffers. This is resolved by instead using a
new immutable EncodedBytes record.
This reworks the dependency resolution for hdfs fixture dependencies to use gradles variant aware dependency resolution instead of relying on outgoing configuration names.
Before this PR sorting on integer, short and byte fields types used
SortField.Type.LONG. This made sort optimization impossible for these
field types.
This PR uses SortField.Type.INT for integer, short and byte fields. This
enables sort optimization.
There are several caveats with changing sort type that are addressed: -
Before mixed sort on integer and long fields was automatically
supported, as both field types used SortField.TYPE.LONG. Now when
merging results from different shards, we need to convert sort to LONG
and results to long values. - Similar for collapsing when there is mixed
INT and LONG sort types. - Index sorting. Similarly, before for index
sorting on integer field, SortField.Type.LONG was used. This sort type
is stored in the index writer config on disk and can't be modified. Now
when providing sortField() for index sorting, we need to account for
index version: for older indices return sort with SortField.Type.LONG
and for new indices return SortField.Type.INT.
---
There is only 1 change that may be considered not backwards compatible:
Before if an integer field was [missing a
value](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/sort-search-results#_missing_values)
, it sort values will return Long.MAX_VALUE in a search response. With
this integer, it sort valeu will return Integer.MAX_VALUE. But I think
this change is ok, as in our documentation, we don't provide information
what value will be returned, we just say it will be sorted last.
---
Also closes#127965 (as same type validation in added for collapse
queries)
