Currently, the raw path is only available from the RestRequest. This
makes the logic to determine if a handler supports streaming more
challenging to evaluate. This commit moves the raw path into pre request
to allow easier streaming support logic.
We may have shut a shard down while merges were still pending (or
adjusted the merge policy while the shard was down) meaning that after
recovery its segments do not reflect the desired state according to the
merge policy. With this commit we invoke `IndexWriter#maybeMerge()` at
the end of recovery to check for, and execute, any such lost merges.
Extensible plugins use a custom classloader for other plugin jars. When
extensible plugins were first added, the transport client still existed,
and elasticsearch plugins did not exist in the transport client (at
least not the ones that create classloaders). Yet the transport client
still created a PluginsService. An indirection was used to avoid
creating separate classloaders when the transport client had created the
PluginsService.
The transport client was removed in 8.0, but the indirection still
exists. This commit removes that indirection layer.
Now that 8.x is branched from main, all transport version changes must be
backported until 9.0 is ready to diverge. This commit adds a test which
ensures transport versions are densely packed, ie there are no gaps at
the granularity the version id is bumped (multiples of 1000).
Almost every implementation of `AckedRequest` is an
`AcknowledgedRequest` too, and the distinction is rather confusing.
Moreover the other implementations of `AckedRequest` are a potential
source of `null` timeouts that we'd like to get rid of. This commit
simplifies the situation by dropping the unnecessary `AckedRequest`
interface entirely.
This method is quite hot in some use-cases because it's used by
most string writing to transport messages. Overriding teh default
implementation for cases where we can write straight to the
page instead of going through an intermediary buffer speeds up
the method by more than 2x, saving lots of cycles, especially
on transport threads.
If we don't actually execute this phase we shouldn't fork the phase
unnecessarily. We can compute the RankFeaturePhaseRankCoordinatorContext
on the transport thread and move on to fetch without forking.
Fetch itself will then fork and we can run the reduce as part of fetch instead of in
a separte search pool task (this is the way it worked up until the recent introduction
of RankFeaturePhase, this fixes that regression).
The only usage of `MappedFieldType#extractTerm` comes from `SpanTermQueryBuilder`
which attempts to extract a single term from a generic Query obtained from calling
`MappedFieldType#termQuery`. We can move this logic directly within its only caller,
and instead of using instanceof checks, we can rely on the query visitor API.
This additionally allows us to remove one of the leftover usages of TermInSetQuery#getTermData
which is deprecated in Lucene
The failure store status is a flag that indicates how the failure store was used or could be used if enabled. The user can be informed about the usage of the failure store in the following way:
When relevant we add the optional field `failure_store` . The field will be omitted when the use of the failure store is not relevant. For example, if a document was successfully indexed in a data stream, if a failure concerns an index or if the opType is not index or create. In more detail:
- when we have a “success” create/index response, the field `failure_store` will not be present if the documented was indexed in a backing index. Otherwise, if it got stored in the failure store it will have the value `used`.
- when we have a “rejected“ create/index response, meaning the document was not persisted in elasticsearch, we return the field `failure_store` which is either `not_enabled`, if the document could have ended up in the failure store if it was enabled, or `failed` if something went wrong and the document was not persisted in the failure store, for example, the cluster is out of space and in read-only mode.
We chose to make it an optional field to reduce the impact of this field on a bulk response. The value will exist in the java object but it will not be returned to the user. The only values that will be displayed are:
- `used`: meaning this document was indexed in the failure store
- `not_enabled`: meaning this document was rejected but could have been stored in the failure store if it was applicable.
- `failed`: meaning this failed document, failed to be stored in the failure store.
Example:
```
"errors": true,
"took": 202,
"items": [
{
"create": {
"_index": ".fs-my-ds-2024.09.04-000002",
"_id": "iRDDvJEB_J3Inuia2zgH",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 6,
"_primary_term": 1,
"status": 201,
"failure_store": "used"
}
},
{
"create": {
"_index": "ds-no-fs",
"_id": "hxDDvJEB_J3Inuia2jj3",
"status": 400,
"error": {
"type": "document_parsing_exception",
"reason": "[1:153] failed to parse field [count] of type [long] in document with id 'hxDDvJEB_J3Inuia2jj3'. Preview of field's value: 'bla'",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "For input string: \"bla\""
}
}
},
"failure_store": "not_enabled"
},
{
"create": {
"_index": ".ds-my-ds-2024.09.04-000001",
"_id": "iBDDvJEB_J3Inuia2jj3",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 7,
"_primary_term": 1,
"status": 201
}
}
]
```
Several `TransportNodesAction` implementations do some kind of top-level
computation in addition to fanning out requests to individual nodes.
Today they all have to do this once the node-level fanout is complete,
but in most cases the top-level computation can happen in parallel with
the fanout. This commit adds support for an additional `ActionContext`
object, created when starting to process the request and exposed to
`newResponseAsync()` at the end, to allow this parallelization.
All implementations use `(Void) null` for this param, except for
`TransportClusterStatsAction` which now parallelizes the computation of
the cluster-state-based stats with the node-level fanout.
The getTermData method in TermInSetQuery is deprecated and not needed for what
we do in NestedHelper. We can remove its use by using other provided methods.
Introduces per-field param `synthetic_source_keep` that overrides the
behavior for keeping the field source in synthetic source mode: -
`none` : no source is stored - `arrays`: the incoming source is
recorded as-is for arrays of a given field - `all`: the incoming source
is recorded as is for both singleton and array values of a given field
Related to #112012
In synthetic source, storing array elements to `_ignored_source` may
hide other, regular elements from showing up during source synthesizing.
This is due to contents from `_ignored_source` taking precedence over
matching fields from regular source loading.
To avoid this, arrays are pre-emptively tracked and marked for source
storing, if any of their elements needs to store its source. A second
doc parsing phase is introduced that checks for fields missing values
and records their source, while skipping objects and arrays that don't
contain any such fields.
Fixes#112374
We can save some allocations and speedup tests some more by using
`InfoStream.NO_OUTPUT` instead of the default `NullInfoStream` that does
not log but causes signals enabled to its users and thus causes
large strings to needlessly be setup.
Nodes can communicate directly for new commits with RCO. It is no longer
necessary for PostWriteRefresh to wait for the special commit
durability. In fact, it is wrong to do that because durability is not
triggered for every commit with RCO. This PR removes the wait.
Relates: ES-8774
Currently, the entire close pipeline is not hooked up in case of a
channel close while a request is being buffered or executed. This commit
resolves the issue by adding a connection to a stream closure.
Currently the rest.incremental_bulk is read in two different places.
This means that it will be employed in two steps introducing
unpredictable behavior. This commit ensures that it is only read in a
single place.
Allow a single bulk request to be passed to Elasticsearch in multiple
parts. Once a certain memory threshold or number of operations have
been received, the request can be split and submitted for processing.
Several `TransportNodesAction` implementations do some kind of top-level
computation in addition to fanning out requests to individual nodes.
Today they all have to do this once the node-level fanout is complete,
but in most cases the top-level computation can happen in parallel with
the fanout. This commit adds support for an additional `ActionContext`
object, created when starting to process the request and exposed to
`newResponseAsync()` at the end, to allow this parallelization.
All implementations use `(Void) null` for this param, except for
`TransportClusterStatsAction` which now parallelizes the computation of
the cluster-state-based stats with the node-level fanout.
Adds `fooRouting` to the routing table because it's not valid to check
it against an `AllocationDecider` otherwise. But then add another
allocation decider to prevent this shard from being assigned to a node
by the `reroute()` call, and set the shards' in-sync IDs up correctly to
make the cluster state valid enough to reroute.
We moved the validation of incoming search requests to data nodes with #105150.
The legacy validation performed on the data nodes was left around for bw comp
reasons, as there could still be coordinating nodes in the cluster not performing
that validation. This is no longer the case in main. This commit removes the
validation in favour of validation already performed while coordinating the
search request.
Relates to #105150
* Resolve merge conflicts
* Log the exception if Bearer token generation fails
* Set rate limit
* Add tests
* Apply spotless
* Add test for ServiceSettings
* Add test for EmbeddingsRequestEntity
* Add test for IbmWatsonxEmbeddingsRequestEntity
* Apply spotless
* Add tests for IbmWatsonxEmbeddingsResponseEntity
* Fix the issue with long line
* Fix tests for IbmWatsonxEmbeddingsActionTests
* Apply spotless
* Resolve merge conflicts
* Move project_id from ServiceFields to IbmWatsonxServiceFields
* Check 400 Bad Request
* Avoid logging exception since this may contain the bearer token
* Throw an exception if the creation of Bearer token fails
* Throw exception based on the status code for generating Bearer token
* Revert "Throw exception based on the status code for generating Bearer token"
This reverts commit f3cd615b8eee1f39536175dee7fd4588a691f319.
* Delete .java-version file
* Fix test
* Update docs/changelog/111770.yaml
* Use IOException instead of Exception
* Resolve merge conflicts
* Fix the tests
* Add end-to-end test and infer test
Rather than manually adding startObject/endObject, and having to line everything up manually, this handles the start/end for you.
A few implementations are converted already. In the long run, I would like this to replace ChunkedXContentHelper.
It seems that the hectic retrying with a new cluster state update just makes things worse. The initial deletion after the relocation might take a bit, I assume also more visible in this test because we've made shard close async in #108145. After that we just check once and if the shard dir is there we keep pushing new cluster states and checking again and this keeps failing with the check mentioned here.
I've picked a simple solution since this is a test issue and just check a bit longer before triggering the new cluster state update. I've looked into a couple of other hooks (e.g. IndexFoldersDeletionListener#beforeIndexFoldersDeleted and org.elasticsearch.indices.cluster.IndicesClusterStateService#onClusterStateShardsClosed ) to see if we could rely on them rather than the assertBusy used here. None unfortunately seem to be cleanly allow getting rid of the assertBusy. IMO, since the shard store deletion is retried and guarantees to eventually work, to avoid flaky tests, we should still keep relying on the retries initiated by the cluster state update.
I'll keep the issue open for a while before removing the extra logging. Running it locally has not failed.
Relates #111134
Mostly not a big win, but we use this in a couple spots where it shows
up in profiling, no need to have this kind of allocation in e.g.
`RandomBasedUUIDGenerator` where it's somewhat hot.
Some small speedups in here from pre-evaluating `isFiltered(properties)`
in lots of spots and not creating an unused `SimpleKey` in `toConcreteKey`
which runs a costly string interning at some rate.
Other than that, obvious deduplication using existing utilities or
adding obvious missing overloads for them.
Flattened fields do no actually use `ignore_above`. When retrieving source, `ignore_above` behaves
as expected. Anyway, it is ignored when fetching field values through the fields API. In that case
values are returned no matter the `ignore_above` setting. Here we implement the filtering logic
in the `ValueFecther` implementation of `RootFlattenedFieldType`.
It is expected that the old master may attempt to read a shardGen file
that is deleted by the new master. This PR checks the latest repo data
before applying the workaround (or throwing AssertionError) for missing
shardGen files.
Relates: #112337Resolves: #112811
We have version based logic that applies the limit to number of completion contexts
only to indices created from 8.0 on. Those are the only indices we can now have
in a cluster, hence we can remove the version based conditional.
Relates to #38675
We have logic that throws exception when parsing empty null_value for ip and date
field mapper only for indices created from 8.0 and onwards. This conditional can
now be removed.
This setting had been removed in the past, it was reintroudced for bw comp with 7.x with #109341. It can now be removed from main as it no longer supports indices created with 7.x
No point in eager loading the metrics here, it just slows down node
startup considerably, wasting lots of time in integTest runs in
particular (as in e.g. ~200s of total CPU time in
:server:internalClusterTest!!).
This commit removes conditionals for logic that no longer needs to be version
dependent, in that all indices in the cluster will match such condition.
Relates to #27211,#34331
IndexSortConfig has some special logic to prevent configure index sorting
targeting alias fields. This returns an error for indices created on or after
7.13. Such logic can now be removed as all indices will fall under that condition.
Relates to #70879
