Rename `xContent.streamSeparator()` and
`RestHandler.supportsStreamContent()` to `xContent.bulkSeparator()` and
`RestHandler.supportsBulkContent()`.
I want to reserve use of "supportsStreamContent" for current work in
HTTP layer to [support incremental content
handling](https://github.com/elastic/elasticsearch/pull/111438) besides
fully aggregated byte buffers. `supportsStreamContent` would indicate
that handler can parse chunks of http content as they arrive.
Now that mock logging has a single internal appender, the "appender"
suffix for MockLogAppender doesn't make sense. This commit renames the
class to MockLog. It was completely mechanical, done with IntelliJ
renames.
Many uses of MockLogAppender predate the addition of the helper method
to create a Releasable for restoring logging, which allows the use of
try-with-resource blocks. This commit converts current existing uses of
MockLogAppender to use capturing.
relates #106425
* Enable corresponding validation in EsqlQueryRequest.
* Add the ESQL version to requests to /_query in integration tests.
* In mixed cluster tests for versions prior to 8.13.3, impersonate an 8.13
client and do not send any version.
---------
Co-authored-by: Nik Everett <nik9000@gmail.com>
To simplify the migration away from version based skip checks in YAML specs,
this PR adds a synthetic version feature `gte_vX.Y.Z` for any version at or before 8.14.0.
New test specs for 8.14 or later are expected to use respective new cluster features,
or a test-only feature supplied via ESRestTestCase#createAdditionalFeatureSpecifications
if sufficient.
This ports our krb5kdc test fixture to test container and reworks hdfs handling to also be based on test containers.
The yaml rest tests that are using hdfs required introducing variable substitution in yamlresttestparser handling.
* Simplify version node_selector: remove ranges and allow just current/non_current
* Version match still present, but only for legacy rest compat tests
Just a suggetion. I think this would save us a bit of memory here and
there. We have loads of places where the always true lambdas are used
with `Predicate.or/and`. Found this initially when looking into field
caps performance where we used to heavily compose these but many spots
in security and index name resolution gain from these predicates.
The better toString also helps in some cases at least when debugging.
This adds many "timers" to the profile and status results. These look
like:
```
"profile" : {
"drivers" : [
{
"took_nanos" : 57252128,
"took_time" : "57.2ms",
"cpu_nanos" : 56733751,
"cpu_time" : "56.7ms",
"operators" : [
...
"operator" : "LuceneSourceOperator[maxPageSize=9362, remainingDocs=2147383647]",
...
"processing_nanos" : 856914,
"processing_time" : "856.9micros",
...
"operator" : "ValuesSourceReaderOperator[fields = [a, b]]",
...
"process_nanos" : 9671406,
"process_time" : "9.6ms",
...
"operator" : "EvalOperator[evaluator=AddLongsEvaluator[lhs=Attribute[channel=1], rhs=Attribute[channel=2]]]",
...
"process_nanos" : 3344354,
"process_time" : "3.3ms",
...
"operator" : "HashAggregationOperator[blockHash=LongBlockHash{channel=1, entries=100, seenNull=false}, aggregators=[GroupingAggregator[aggregatorFunction=MaxLongGroupingAggregatorFunction[channels=[3]], mode=INITIAL], GroupingAggregator[aggregatorFunction=MinLongGroupingAggregatorFunction[channels=[3]], mode=INITIAL]]]",
...
"hash_nanos" : 11845071,
"hash_time" : "11.8ms",
"aggregation_nanos" : 27910054,
"aggregation_time" : "27.9ms",
...
```
This let's us compare the runtime of each operator in a query. You can
get the run times of a running ESQL task with the `_tasks` API or you
can use the `profile` API to run an ESQL query and get these times in
the result. The example above is from `profile`.
In the example above we can see we're spending 500 microseconds waiting
in the queue. Of the remaining 56ms a little under half of that is spent
running the aggregations - in this case a `MAX` and `MIN`. And 20% of
the time is hashing to build group keys. Similarly 20% is spent on field
loading.
Co-authored-by: Alexander Spies <alexander.spies@elastic.co>
The difference is that our matcher uses .find() to search for a regex match anywhere in the string, whereas the hamcrest one uses .matches() to check the whole string against the regex. This leads to more specific regex checks.
I've left our own one for YAML tests, as that way we don't need to mangle the regex to add .* either side, which might be confusing in test failures.
* Use single-char variant of String.indexOf() where possible
indexOf(char) is more efficient than searching for the same one-character String.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Use String.replace() instead of replaceAll() for non-regexp replacements
When arguments do not make use of regexp features replace() is a more efficient option, especially the char-variant.
Add the ability to test for the original/old cluster features during a rolling upgrade
* Moving ALL_FEATURES to ESRestTestCase (and make it private - only usage)
* Introduce Prerequisites criteria (Predicate + factory) for modular skip decisions
- Removed accessors to specific criteria from SkipSection (used only on tests), adjusted test assertions
- Moved Features check (YAML test runner features) to SkipSection build time
* Separated check for xpack/no_xpack
Check for xpack is cluster-configuration (modules installed) dependent, while Features are meant to be "static" test-runner capabilities. We separate them so checks on one (test-runner features) can be run before and separately from the other.
* Consolidate skip() methods
- Divide require and skip predicates
- Divide requires and skip parsing (distinct sections)
- Renaming SkipSection to PrerequisiteSection and related methods/fields (e.g. skip -> evaluate)
* Refactoring tests
- moving and adding VersionRange tests
- adding specific version and os skip tests
- modified parse/validate/build to make SkipSection more unit-testable
* Adding cluster feature-based skip criteria
* Updated javadoc + renaming + better skip reason message
The ESQL async tests run the ESQL yaml tests two extra time - once under
the async endpoint with the `wait_for_completion_timeout` set to a long
time and *again* with `wait_for_completion_timeout` set to a short time,
expecting to receive an `id` for the query.
That second way is tricky! Even with a `0ms` timeout sometimes the
request will complete. That's great, but the tests didn't realize that
was possible. And it's tricky to get the warnings and `catch` sections
working properly with that. This reworks how we run these commands,
breaking apart the way we run a single API and running it as two, taking
into account that the "start the query" request could also complete the
query.
Closes#104294
This adds tests that run the our suite of yaml tests against the ESQL
async endpoint. That's quite nice because the yaml tests are where we
handle lots of fun error cases and this'll make sure async does sensible
things in those cases.
* Consolidate skip() methods by passing down Context to them
* Introduce SkipCriteria for modular skip decisions
* Refactoring tests
- moving and adding VersionRange tests
- adding specific version and os skip tests
- modified parse/validate/build to make SkipSection more unit-testable
* Moved Features check (YAML test runner features) to SkipSection build time
* Separated check for xpack/no_xpack
Check for xpack is cluster-configuration (modules installed) dependent, while Features are meant to be "static" test-runner capabilities. We separate them so checks on one (test-runner features) can be run before and separately from the other.
* Pushing down node versions as strings
Deferring Version parsing to the actual places where a minimum node version/common cluster version is needed; eventually this will be completely lazy and/or replaced by other checks (e.g. features).
Combine versions, oses and features in multi-cluster YAML test contexts.
Just moving stuff around, no change in behaviour. Moving these fields showed how we are not treating correctly in derived classes where multiple clusters are tested (ex: CCR), but this is for another time.
Co-authored-by: Moritz Mack <moritz@mackmail.net>
Initialization of information about the cluster under test is factored out into 2 methods, which can be used with different admin clients (e.g. in case of multi-cluster, like CCS).
We're leaking quite a few of these parsers. That doesn't seem to be much
of a problem but results in some memory inefficiencies in Jackson here
and there. This PR bulk fixes a bunch of instances that I could easily
automatically fix. I'll open a follow-up for closing the parser on the
document parsing context which also suffers from this but is non-trivial
to fix.
Today `RestClient` interprets an `?ignore=` request parameter as an
indication that certain HTTP response codes should be considered
successful and not raise a `ResponseException`. This commit replaces the
magic literal `"ignore"` with a constant and adds a utility to specify
the ignored codes as `RestStatus` values.
* Compatible version parsing in YAML tests
* Propagate exception in case of non-semantic version where one is expected
* Removed remove of SNAPSHOT (no longer needed)
currently to assert that a field does not exist, tests are using is_false. This has a drawback because is less readable and also would succeed when value is present and has a default value (0, false, etc)
This commit addes a NotExistAssertion to make sure that a field does not exist in a response
