* Split searchable snapshot into multiple repo operations
Each operation on a snapshot repository uses the same `Repository`,
`BlobStore`, etc. instances throughout, in order to avoid the complexity
arising from handling metadata updates that occur while an operation is
running. Today we model the entire lifetime of a searchable snapshot
shard as a single repository operation since there should be no metadata
updates that matter in this context (other than those that are handled
dynamically via other mechanisms) and some metadata updates might be
positively harmful to a searchable snapshot shard.
It turns out that there are some undocumented legacy settings which _do_
matter to searchable snapshots, and which are still in use, so with this
commit we move to a finer-grained model of repository operations within
a searchable snapshot.
Backport of #116918 to 8.16
* Add end-to-end test for reloading S3 credentials
We don't seem to have a test that completely verifies that a S3
repository can reload credentials from an updated keystore. This commit
adds such a test.
Backport of #116762 to 8.16.
We don't seem to have a test that completely verifies that a S3
repository can reload credentials from an updated keystore. This commit
adds such a test.
Backport of #116762 to 8.16.
Clarifies that insecure settings are stored in plaintext and must not be
used. Also removes the mention of the (wrong) system property from the
error message if insecure settings are not permitted.
Backport of #116915 to `8.16`
* Fix handling of bulk requests with semantic text fields and delete ops (#116942)
Previously, delete operations were not processed correctly when followed by operations containing semantic text fields. This issue caused the positions of subsequent operations in the items array to shift incorrectly by one.
This PR resolves the discrepancy and includes additional tests to ensure proper behavior.
* fix compilation
The fetch phase is subject to timeouts like any other search phase. Timeouts
may happen when low level cancellation is enabled (true by default), hence the
directory reader is wrapped into ExitableDirectoryReader and a timeout is
provided to the search request.
The exception that is used is TimeExceededException, but it is an internal
exception that should never be returned to the user. When that is thrown, we
need to catch it and throw error or mark the response as timed out depending
on whether partial results are allowed or not.
Several file-settings ITs fail (rarely) with exceptions like:
```
java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\platform-support\14\server\build\testrun\internalClusterTest\temp\org.elasticsearch.reservedstate.service.SnaphotsAndFileSettingsIT_5733F2A737542BE-001\tempFile-001.tmp -> C:\Users\jenkins\workspace\platform-support\14\server\build\testrun\internalClusterTest\temp\org.elasticsearch.reservedstate.service.SnaphotsAndFileSettingsIT_5733F2A737542BE-001\tempDir-002\config\operator\settings.json |
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89) |
-- | --
| | at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103) |
| | at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:317) |
| | at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:293) |
| | at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:144) |
| | at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:144) |
| | at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:144) |
| | at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:144) |
| | at java.nio.file.Files.move(Files.java:1430) |
| | at org.elasticsearch.reservedstate.service.SnaphotsAndFileSettingsIT.writeJSONFile(SnaphotsAndFileSettingsIT.java:86) |
| | at org.elasticsearch.reservedstate.service.SnaphotsAndFileSettingsIT.testRestoreWithPersistedFileSettings(SnaphotsAndFileSettingsIT.java:321)
```
This happens in Windows file systems, due to a race condition where the
file settings service is reading the settings file concurrently with the
test trying to modify it (a no-go in Windows). It turns out we have
already addressed this with a retry for one test suite
(https://github.com/elastic/elasticsearch/pull/91863), plus addressed a
related issue around mock windows file-systems misbehaving
(https://github.com/elastic/elasticsearch/pull/92653).
This PR extends the above fixes to all file-settings related ITs.
(cherry picked from commit 91559da015)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Adds clarification for vector preloading, what extension is to what
storage kind, and that quantized vectors are stored in separate files
allowing for individual preload.
closes: https://github.com/elastic/elasticsearch/issues/116273
* Validate missing shards after the coordinator rewrite (#116382)
The coordinate rewrite can skip searching shards when the query filters
on `@timestamp`, event.ingested or the _tier field.
We currently check for missing shards across all the indices that are
the query is running against however, some shards/indices might not
play a role in the query at all after the coordinator rewrite.
This moves the check for missing shards **after** we've run the
coordinator rewrite so we validate only the shards that will be
searched by the query.
(cherry picked from commit cd2433d60c)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
* imports
* Adapt unit test for 8.16 to use @timestamp rewrite
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
