In Jackson 2.15 a maximum string length of 50k characters was
introduced. We worked around that by override the length to max int on
all parsers created by xcontent. Jackson 2.17 introduced a similar limit
on field names. This commit mimics the workaround for string length by
overriding the max name length to be unlimited.
relates #58952
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
A while ago we enabled using ccs_minimize_roundtrips in async search.
This makes it possible for users of async search to send a single search
request per remote cluster, and minimize the impact of network latency.
With non minimized roundtrips, we have pretty recurring cancellation checks:
as part of the execution, we detect that a task expired whenever each shard comes
back with its results.
In a scenario where the coord node does not hold data, or only remote data is
targeted by an async search, we have much less chance of detecting cancellation
if roundtrips are minimized. The local coordinator would do nothing other than
waiting for the minimized results from each remote cluster.
One scenario where we can check for cancellation is when each cluster comes
back with its full set of results. This commit adds such check, plus some testing
for async search cancellation with minimized roundtrips.
CollectionUtils.uniquify is based on C++ std::unique. However, C++
iterators are not quite the same as Java iterators. In particular,
advancing them only allows grabbing the value once. This commit reworks
uniquify to be based on list indices instead of iterators.
closes#126883
On x64, we are testing if we support vector capabilities (1 = "basic" = AVX2, 2 = "advanced" = AVX-512) in order to enable and choose a native implementation for some vector functions, using CPUID.
However, under some circumstances, this is not sufficient: the OS on which we are running also needs to support AVX/AVX2 etc; basically, it needs to acknowledge it knows about the additional register and that it is able to handle them e.g. in context switches. To do that we need to a) test if the CPU has xsave feature and b) use the xgetbv to test if the OS set it (declaring it supports AVX/AVX2/etc).
In most cases this is not needed, as all modern OSes do that, but for some virtualized situations (hypervisors, emulators, etc.) all the component along the chain must support it, and in some cases this is not a given.
This PR introduces a change to the x64 version of vec_caps to check for OS support too, and a warning on the Java side in case the CPU supports vector capabilities but those are not enabled at OS level.
Tested by passing noxsave to my linux box kernel boot options, and ensuring that the avx flags "disappear" from /proc/cpuinfo, and we fall back to the "no native vector" case.
Fixes#126809
Also, add 8.16 branch to be still tested as "bugfix".
For 8.17 there is currently no bugfix2 version.
We test it against 8.16.7 and 7.17.29 in BwC tests.
* Edits copied over from https://github.com/elastic/elasticsearch/pull/120346
* Update docs/reference/how-to/size-your-shards.asciidoc
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
* Improve Scanability
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
* Reduced what is a shard section for concision
* Adjusted title
* Add general and distribution sizing guidelines.
---------
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
Co-authored-by: George Wallace <georgewallace@users.noreply.github.com>
* ESQL: Known issue enrich missing field (#126701)
* Add known issue note to 8.17.0-8.17.4
* Add known issue to 8.16.0-8.16.6
* remove coming tags from notes that hadn't been forward ported to 8.x
* remove another coming tag
* clarify it's esql
* include pages that hadn't been forward ported
---------
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Co-authored-by: Liam Thompson <leemthompo@gmail.com>
(cherry picked from commit f7a52054b2)
# Conflicts:
# docs/reference/release-notes.asciidoc
# docs/reference/release-notes/8.17.2.asciidoc
# docs/reference/release-notes/8.17.3.asciidoc
# docs/reference/release-notes/8.17.4.asciidoc
* delete 8.18 includes from 8.17 branch
---------
Co-authored-by: Alexander Spies <alexander.spies@elastic.co>
Today we rely on registering the channel after registering the task to
be cancelled to ensure that the task is cancelled even if the channel is
closed concurrently. However the client may already have processed a
cancellable request on the channel and therefore this mechanism doesn't
work. With this change we make sure not to register another task after
draining the registrations in order to cancel them.
Closes#88201
This commit adds a note that ignore_above has a different limit for
logsdb indices to the documentation. Also specifies that ignore_above
applies to all types of the keyword family.
Relates https://github.com/elastic/sdh-elasticsearch/issues/8892
* Set `keyUsage` for generated HTTP certificates and self-signed CA (#126376)
The `elasticsearch-certutil http` command, and security auto-configuration,
generate the HTTP certificate and CA without setting the `keyUsage` extension.
This PR fixes this by setting (by default):
- `keyCertSign` and `cRLSign` for self-signed CAs
- `digitalSignature` and `keyEncipherment` for HTTP certificates and CSRs
These defaults can be overridden when running `elasticsearch-certutil http`
command. The user will be prompted to change them as they wish.
For `elasticsearch-certutil ca`, the default value can be overridden by passing
the `--keysage` option, e.g.
```
elasticsearch-certutil ca --keyusage "digitalSignature,keyCertSign,cRLSign" -pem
```
Fixes#117769
(cherry picked from commit 284121ad9f)
# Conflicts:
# docs/reference/elasticsearch/command-line-tools/certutil.md
* fix compilation error
* [CI] Auto commit changes from spotless
* fix failing test
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
This reverts commit 4a77e06. We've seen a significant performance degradation in merging vectors resulting from the use of MADV_RANDOM and MGLRU ( and LRU in recent Linux kernels )
For the 8.x release train, then we will revert the change that enabled MADV_RANDOM. And backport to all shipping 8.x bugfix releases.
relates: #124499
* [8.18] ESQL: ESQL: Fix ReplaceMissingFieldsWithNull (#125764) (#126166)
* Revert changes to Layout.java
The change in 80125a4bac is a quick fix
and allows breaking an invariant of Layout. Revert that.
* Simplify ReplaceMissingFieldWithNull
When encountering projections, it tries to do the job of field
extraction for missing fields by injecting an Eval that creates a
literal null with the same name id as the field attribute for the
missing field. This is wrong:
1. We only insert an Eval in case that a Project relies on the missing
attribute. There could be other plan nodes that rely on the missing
attribute.
2. Even for Projects, we only insert an Eval in case we squarely project
for the field - in case of aliases (e.g. from RENAME), we do nothing.
3. In case of multiple Projects that use this attribute, we create
multiple attributes with the original field attribute's id, causing
a wrong Layout. This triggered
https://github.com/elastic/elasticsearch/issues/121754.
* Revive logic for EsRelation instead of Project
* Update LocalLogicalPlanOptimizerTests
* Update test expectations
* Do not prune attributes from EsRelation
This can lead to empty output, which leads to the EsRelation being
replaced by a LocalRelation with 0 rows.
* Add tests + capability
* Add comments
* [CI] Auto commit changes from spotless
* Update docs/changelog/125764.yaml
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
(cherry picked from commit 96ca13aaee)
# Conflicts:
# x-pack/plugin/esql/qa/testFixtures/src/main/resources/lookup-join.csv-spec
# x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java
# x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/local/ReplaceMissingFieldWithNull.java
# x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/Layout.java
# x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizerTests.java
# x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalPhysicalPlanOptimizerTests.java
* Re-instate fix for LOOKUP JOIN, update tests
With #123610 we disabled parallel collection for field and script sorted top hits,
aligning its behaviour with that of top level search. This was mainly to work around
a bug in script sorting that did not support inter-segment concurrency.
The bug with script sort has been fixed with #123757 and concurrency re-enabled for it.
While sort by field is not optimized for search concurrency, top hits benefits from it
and disabling concurrency for sort by field in top hits has caused performance
regressions in our nightly benchmarks.
This commit re-enables concurrency for top hits with sort by field is used. This
introduces back a discrepancy between top level search and top hits, in that concurrency
is applied for top hits despite sort by field normally disables it. The key difference
is the context where sorting is applied, and the fact that concurrency is disabled
only for performance reasons on top level searches and not for functional reasons.
We have some tolerance wound how many bytes we report for these completion fields. But the
values depend on the distribution of the random values that determine how many docs get
an option field. This commit makes the test more precise by computing the real ratio
between docs that have the optional field and the total number of docs, so that we
can base assertion on more realistic expectations.
Closes#123269
* Update missed documentation change for new watcher index settings
* Update docs/reference/rest-api/watcher/update-settings.asciidoc
---------
Co-authored-by: Keith Massey <keith.massey@elastic.co>
Increase waiting time to prevent DFA integration tests from timing out.
(cherry picked from commit d9008cd065)
# Conflicts:
# x-pack/plugin/ml/qa/native-multi-node-tests/src/javaRestTest/java/org/elasticsearch/xpack/ml/integration/MlNativeDataFrameAnalyticsIntegTestCase.java
