Currently we have a relatively basic decider about when to throttling
indexing. This commit adds two levels of watermarks with configurable
bulk size deciders. Additionally, adds additional settings to control
primary, coordinating, and replica rejection limits.
This test was failing due to a race between an early cancellation check
and the cancel operation. With this commit we wait until the action is
definitely blocked before cancelling the task.
Closes#100062
This commit flips the incremental bulk setting to false. Additionally,
it removes some test code which intermittently causes issues with
security test cases.
* Move raw path into HttpPreRequest (#113231)
Currently, the raw path is only available from the RestRequest. This
makes the logic to determine if a handler supports streaming more
challenging to evaluate. This commit moves the raw path into pre request
to allow easier streaming support logic.
* Fix
Almost every implementation of `AckedRequest` is an
`AcknowledgedRequest` too, and the distinction is rather confusing.
Moreover the other implementations of `AckedRequest` are a potential
source of `null` timeouts that we'd like to get rid of. This commit
simplifies the situation by dropping the unnecessary `AckedRequest`
interface entirely.
Add use of Opaque ID HTTP header in task cancellation assertion. In some
tests, like this #88201 `testCatSegmentsRestCancellation`, we assert
that all tasks related to specific HTTP request are cancelled. But we do
blanket approach in assertion block catching all tasks by action name. I
think narrowing down assertion to specific http request in this case
would be more accurate.
It is still not clear why test mentioned above failing, but after hours
of investigation and injecting random delays, I'm inclining more to
@DaveCTurner's comment about interference from other tests or cluster
activity. I added additional log that will report when we spot task with
different opaque id.
Wholesale fix of every `TRAPPY_IMPLICIT_DEFAULT_MASTER_NODE_TIMEOUT` in
`o.e.snapshots` and `o.e.repositories`, just pulling them up to the REST
layer (where they become API params), the test suite (where they become
`TEST_REQUEST_TIMEOUT`), or some other place where an explicit value is
available.
Relates #107984
* Mechanical package change in IntelliJ
* A couple of manual fixups
* Export plugins.loading to deprecation
* Put plugin-cli in a module so can export PluginsUtils to it.
The test started failing because of the recent changes to allow closing (and deleting shards) asynchronously. As a result dandling index API now is seeing a directory in partially deleted state, fails to interpret partial data and fails as a result.
The fix retries the failure on the client.
Many APIs accept a `?master_timeout` parameter, but reading this
parameter requires a little unnecessary boilerplate to specify the
literal parameter name and default value. Moreover, today's convention
is to construct a `MasterNodeRequest` and then read the default master
timeout from the freshly-created request. In practice this results in a
default of 30s, but we specify in the docs that this default is _always_
30s, and in principle one could create a transport request with a
different initial value which would deviate from the documented
behaviour.
This commit introduces a utility method for reading this parameter in a
fashion which is completely consistent with the documented behaviour.
Relates #107984
Our test utility returns the node name when starting a new node.
A lot of APIs (such as routing table or node shutdown) require a node id.
This change introduces a simple way to retrieve the node id based on its name.
The code to parse a `SnapshotInfo` object out of an `XContent` response
body is only used in tests, so this commit moves it out of the
production codebase and into the test framework.
Today the handling of the `?after` param is kinda spread out over
`TransportGetSnapshotsAction` and `GetSnapshotsRequest` making it hard
to follow and adding unnecessary complexity to these two classes. This
commit moves it into `SnapshotSortKey` which is a better fit since the
behaviour varies so much for different sort keys.
The behaviour of the get-snapshots API varies quite considerably
depending on the sort key chosen. Today this logic is implemented using
scattered `switch` statements and other conditionals but it'd be clearer
if we delegated this stuff to the sort key instances themselves. This
commit moves the sort key enum to the top level and replaces one of the
`switch` statements with a method on the enum instances.
Removes all the now-dead code related to reading pre-7.16 get-snapshots
requests and responses, and also moves the `XContent` response parsing
out of production and into the only test suite that uses it.
A predicate to check whether the cluster supports a feature is available
to rest handlers defined in server. This commit adds that predicate to
plugins defining rest handlers as well.
* Fix `require_alias` implicit true value on presence
This commit brings the `require_alias` query-string parameter into line with the rest of our parameters where its presence indicates an implicit "true" value (so a user can do `POST /_bulk?require_alias` to enable the check).
Resolves#103945
* Update docs/changelog/104099.yaml
Tests that send REST requests with bodies must today build up a separate
`String` containing the body contents as JSON. This is kinda ugly, and
also means we do not cover the other supported body formats in these
tests. This commit introduces a utility to allow construction of REST
requests with `XContent` bodies directly, and generalizes things to
choose randomly between JSON and other supported body formats.
We expect that the HTTP response stats should reflect the responses
received by the client, but today we record that a chunked-encoded
response was sent _after_ sending the last chunk, which might mean that
the client can receive the complete response body and then retrieve
stats that do not include that response. With this commit we record the
stats before sending the last chunk.
Closes#102547
Same as #101175, shorten `client().prepareIndex(index)` and
`client().prepareIndex().setIndex(index)` via a test utility.
Saves lots of code now and sets up some follow-up simplifcations.
`ExternalTestCluster` doesn't really make sense now that the transport
client is removed. We only use it in the ML integ test suite and it'd be
good to avoid expanding its usage further, so this commit deprecates it
and removes the functionality in `ESIntegTestCase` that might quietly
switch to using it in a new test suite if running with certain system
properties.
Relates #49582
We should get the same effect whether we cancel the task by closing the
REST connection or by using the tasks API. This commit generalises the
test to cover both cases.
This action is a pure function of the cluster state, it can run on any
node. Moreover it can be fairly expensive if there are a lot of aliases
so running it on the master can be quite harmful to the cluster.
Finally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
Relates #101805
Today it is a little tricky to write tests for REST cancellation since
we must pause the action while it is running to ensure that the
cancellation has the desired effect. Many actions have some way to
achieve this, possibly requiring some amount of trickery, but some
actions have no natural point at which they can be paused.
This commit introduces an `ActionFilter`-based approach to capture the
task just after it has been registered but before it starts executing.
All REST actions pass through the `ActionFilter` chain at least once, so
this technique should work everywhere.
This commit also removes the hooks that were added to test the
cancellability of the recovery APIs since they're no longer needed.
Got asked why we have these classes again last night ... figured I'd
remove some of them to get us going here.
None of these classes are necessary, we can just inline all of these
away and make `ActionType` itself final or a record now that the action
type consists only in name and reader. See #97721 that made these things
redundant I think.
The StatusToXContent interface extends ToXContent with a REST status
method. It is used by RestStatusToXContentListener, which uses the extra
method to set the status of the RestResponse. However, given the
generics parametrization of RestToXContentListener, we can already get
the status of a response object without having these extra interfaces,
which mix serialization and status concerns.
This commit removes StatusToXContent and RestStatusToXContentListener,
instead adding some extra optional parameters to RestToXContentListener to
handle retrieving status and location information from the passed in Response
object.
This commit renames the tracing to telemetry.tracing in both xpack/APM and elasticserach's org.elasticsearch.tracing.Tracer (the api)
the xpack/APM is renamed as follows:
org.elasticsearch.telemetry.apm - the only exported package
org.elasticsearch.telemetry.apm.settings - APMSettings
org.elasticsearch.telemetry.apm.tracing - APMTracer
org.elasticsearch.tracing.Tracer is moved to org.elasticsearch.telemetry.tracing.Tracer (responsible for majority of the changes in this PR)
Drying up the logic for looking up the customs and falling back to the empty default. Hopefully, this improves readability somewhat by taking away that complexity and some null checks.
I like it at least, saves a lot of clutter.
Re-revert of #98896, only fixing a single null check in tests.
This PR adds a mappings version number to system index descriptor, which is intended to replace the use of Version to signal changes in the mappings for the descriptor. This value is required for managed system indices.
Previously, most of the system index descriptors automatically incremented their mapping version with each release, which meant that the mappings would stay up-to-date with any additive changes. Now, developers will need to increment the mapping version whenever there's a change.
* Add MappingsVersion inner class to SystemIndexDescriptor
* Add mappings version to metadata in all system index mappings
* Rename version meta key ('system' -> 'managed')
* Update mappings for ML indices if required
* Trigger ML index mappings updates based on new index mappings version.
---------
Co-authored-by: Ed Savage <ed.savage@elastic.co>
Drying this up further and adding the same short-cut for single node
tests. Dealing with most of the spots that I could grab via automatic
refactorings.
This API can be quite heavy in large clusters, and might spam the
`MANAGEMENT` threadpool queue with work for clients that have long-since
given up. This commit adds some basic cancellability checks to reduce
the problem.
