This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
In #91231 we concluded that a request that
contains a single wildcard but that also sets expand_wildcards
to none should throw a "not found" exception if
ignore_unavailable is false.
This PR adjusts the 7.x test in order to account for that,
in order to pass the rest-compatibility tests in 8.*.
Related: #91231
the #90458 has been backported to all branches so the bwc testing can be enable for this tests were incorrectly relying on sort order. Added sort to make it deterministic
closes#90668
* Fix date rounding for date math parsing (#90458)
in #89693 the rounding logic was only applied when a field was present on a pattern. This is incorrect as for dates like "2020" we want to default to "2020-01-01T23:59:59.999..." when rounding is enabled.
This commit always applies monthOfYear or dayofMonth defaulting (when rounding enabled) except when the dayOfYear is set
closes#90187
backport(#90458)
(cherry picked from commit 3f3a95e2dc)
It's not obvious that a YAML test with a `catch` stanza also permits
`match` blocks to assert things about the structure of the error
response, but this structure may be an important part of the API spec.
This commit adds this info to the docs about YAML tests.
* Fix a bug with flattened fields in terms aggregations (#87392)
The root cause here was that missing did not correctly delegate `supportsGlobalOrdinalsMappnig` to the wrapped values source, instead falling back to the default. I've added the delegation, and made the base method abstract so this doesn't happen again.
* BWC version dance, part one
* Fix flattened object format
This adds some missing skips to make the tests for the serial_diff
always pass and test the current version. The tests contain a
`fixed_interval` which is only supported in 7.2+. That's fine probably
ok - we don't expect to make backwards breaking changes at all, much
less breaking with 7.1 and *not* 7.2
Before 6.6 we allowed the `serial_diff` agg in lots of places, including
under `date_histogram` aggregations with `min_doc_count: 1`. This
allowed you to take the difference of two adjacent buckets, skipping any
that don't have any value. So if you have a value at 10am, no value at
11am, and another at noon the `serial_diff` will diff the 10am and noon
values. In 6.6 we disabled support for this. We'd like it back.
Backport PR #85467 added two tests:
* Multi-value date histogram
* Multi-value date histogram docvalues only
as a result of a mistake while doing a conflict resolution.
With this PR I will remove them temporarily from master, backport
the fix to 7.17.3 and later make another PR to master to add them
back.
This is a fix for #85509.
(cherry picked from commit b4ff756bec)
# Conflicts:
# rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.aggregation/360_date_histogram.yml
This expands on the super basic REST tests for variable width histogram
with some ever so slightly less basic tests. They are still pretty
basic, but they do a good job of making sure that we don't break
backwards compatibility with the API.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Something is wrong with `fetch` in `top_hits` in 7.10 and I'm not sure
what. But at this point it's not worth figuring it out. So we'll skip
testing `fetch` in `top_hits` until 7.11.
Closes#84590
The tests for the explanation of top_hits asserted that the explanation
matches precise strings. Which it mostly does. But there seem to be edge
cases where a number in the string varies. We really don't care about
the number. This replaces the string assertion with a regex that doesn't
care about the number.
Closes#84590
* REST tests for top_hits (#83690)
This adds a pile of extra REST tests for the `top_hits` aggregation
which is useful because this is how we do mixed version cluster testing.
And backwards compatibility testing. And forwards compatibility testing.
* After 7.0
PR #83339 was meant to be included in version 7.17.0 but didn't happen.
It landed in version 7.17.1. As a result, version checks in the code are
incorrect. Instead of checking against version 7.17.0 we need to adjust
the code in such a way that it operates correctly with version 7.17.0.
Version 7.17.0 is released and, as as result, it represents the 'reference'
version we would like all following versions to interoperate with.
Unfortunately this means that interoperability between version 7.17.0 and
version 8.0.0 is broken.
In summary this fix needs to be backported to:
* 7.17.1: so to fix interoperability of version 7.17.1 with 7.17.0
* 8.0.1: so to fix interoperability of version 8.0.1 with 7.17.0
* 8.1.0 (currently BC): so to fix interoperability of version 8.1.0 with 7.17.0
* master (8.2): so to fix interoperability of version 8.2.0 with 7.17.0
Also, when reading and writing the range key, version 7.17.0 checks against
version 6.4.0 which is not possible right now. As a result, we just check
against the earliest possible version.
Follow-up from #77144 (comment) with converting id/_id to always be strings instead of integers. This makes the type value in the Elasticsearch specification be only string instead of string | number.
this change was generated using following command on ubuntu
find . -type f -name "*.yml" -print0 | xargs -0 sed -i -r 's/([^a-zA-Z0-9_\.]id|[^a-zA-Z0-9_]_id):(\s)([0-9]+)/\1:\2"\3"/g'
relates #82681
When running a range aggregation on a date field both `fromAsStr` and
`toAsStr` need to be parsed to extract the correct range boundaries.
Not doing so results in +/- infinite values in `to`, `from`, `originalTo`
and `originalFrom` which in turn result in:
* incorrect key values (wildcard character used for infinite values);
* missing fields in the response (to, from, to_as_string, from_as_string)
due to checks for initite values in the serialization logic;
(cherry picked from commit 1186e90d09)
It's unclear to me what the causes are for the test failures in #80335. To rule out some basic stuff, and add a bit more introspection, I'm suggesting to push the following change.
Relates #80335
When dealing with float fields we actually store float values with lower precision.
Later on, when loading float values into double values, the precision difference between
float and double values surfaces as additional "spurious" digits in the string
representation which is propagated to the client during serialisation. As a result of this,
the JSON response returned to the client includes range values not matching values in the
request. This results in clients, including Kibana, to break while trying to match ranges in
the request with ranges in the response.
With this change we use two new fields, `originalFrom` and `originalTo` to hold the original
double values before manipulating the precision of float fields. Later we use these fields in
the code dealing with serialisation. The effect is that precision issue resulting from
float to double conversions are not propagated to clients.
(cherry picked from commit 3550370c86)
* [7.16] [ML] Model snapshot upgrade needs a stats endpoint
Previously the ML model snapshot upgrade endpoint did not
provide a way to reliably monitor progress. This could lead
to the upgrade assistant UI thinking that a model snapshot
upgrade had finished when it actually hadn't.
This change adds a new "stats" API that allows external
interested parties to find out the status of each model
snapshot upgrade and which node (if any) each is running on.
Backport of #81641
* Fixing compilation
* Use correct version bounds for API test
* Use correct version bounds for migration API test
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Removes source-related query parameters from the update by query
and delete by query API documentation. These parameters don't return
source fields as part of the response.
# Conflicts:
# rest-api-spec/src/main/resources/rest-api-spec/api/delete_by_query.json
* Revert "Allow deprecation warning for the return_200_for_cluster_health_timeout parameter (#80178) (#80445)"
This reverts commit 5a5300753d.
* Revert "[7.x] Use query param instead of a system property for opting in for new cluster health response code (#79397) (#79435)"
This reverts commit d8c14281
* Revert "[7.x] Deprecate returning 408 for a server timeout on `_cluster/health` (#78180) (#78940)"
This reverts commit 03bd55d1
* Changing HeaderWarning to always use 299 as the warning code (#80304)
This commit changes the deprecation logger so that all messages (critical or warning) are written
out with "299" as the level at the beginning of the header in order to be compliant with
https://www.rfc-editor.org/rfc/rfc7234.html#section-5.5.7. In #79107 we mistakenly began logging
warning-level messages with the 300 code, which is not valid in the RFC.
* fixing compilation errors from merge
Currently the fleet search URL of /_fleet/_msearch will collide with the
normal msearch API when the fleet plugin is not enabled. This is because
_fleet will be identified as an index to search. This commit resolves
the issue by changing the APIs to /_fleet/_fleet_search and
/_fleet/_fleet_msearch.
The FieldNamesFieldMapper adds non-indexed fields to a special metadata
field so that exists queries can be run efficiently. This is built in a postParse
method that is run once per document. However, this means that nested
documents are not handled correctly - non-indexed field names are added to
the parent document's metadata field rather than to the nested document's
field.
This commit fixes things to add non-indexed field names directly to the
nested documents.
Co-authored-by: Yoann Rodière <yoann@hibernate.org>
