If debug logging is enabled then the lag detector will capture and
report the hot threads of a lagging node. In some cases the resulting
log message can be very large, exceeding 10kiB, which means it is
truncated in most logging setups. The relevant thread(s) may be waiting
on I/O, which is not considered "hot" and therefore may not appear in
the first 10kiB.
This commit adjusts this logging mechanism to split the message into
chunks of size at most 2kiB (after compression and base64-encoding) to
ensure that the entire hot threads output can be faithfully
reconstructed from these logs.
Closes#88126
* [DOC] Troubleshooting Expensive Searches
👋 re: https://github.com/elastic/elasticsearch/issues/73222 adds in content so we can link to users on how to find source of expensive searches.
* Several edits
* Apply suggestions from code review
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
It makes sense to allow more than one KNN search clause per individual search request. It may be that different documents have separate vector spaces or that a single doc is index with more than one vector space. In both of these scenarios, users may want to retrieve a resulting set that takes into account all their indexed vector spaces.
A prime example here would be searching a semantic text embedding along with searching an image embedding.
closes https://github.com/elastic/elasticsearch/issues/91187
Dangling indices are not imported automatically since 8.0 but the
`elasticsearch-node detach-cluster` documentation still suggests it is.
It tried to make it more explicit by listing the Dangling API to use and
by using the work "manually".
With this commit we specify `level` in JVM logs. `level` helps to filter
more severe messages from mere debugging messages in logs and with this
change we are able to use it also for JVM logs.
Here are a few example lines:
Before:
```
[2022-12-15T06:19:16.936+0000][15181][gc,start ] GC(0) Pause Young (Normal) (G1 Evacuation Pause)
```
After:
```
[2022-12-15T06:22:27.932+0000][16320][info][gc,start ] GC(0) Pause Young (Normal) (G1 Evacuation Pause)
[2022-12-15T06:22:27.932+0000][16320][debug][gc,age ] GC(0) Desired survivor size 14680064 bytes, new threshold 15 (max threshold 15)
[2022-12-15T06:22:27.935+0000][16320][info ][gc,phases ] GC(0) Pre Evacuate Collection Set: 0.1ms
[2022-12-15T06:22:27.935+0000][16320][trace][gc,age ] GC(0) Age table with threshold 15 (max threshold 15)
```
Notice, that the log level might have trailing spaces.
See also
https://docs.oracle.com/en/java/javase/19/docs/specs/man/java.html#xlog-tags-and-levels
for the available log levels.
Today to troubleshoot an unstable cluster we ask the users to parse the
rather complex `node-join` and `node-left` messages emitted by the
`MasterService`. These messages may refer to many nodes, may be
truncated, and are generally pretty hard to work with.
With this commit we start to emit a simplified log message about each
node added and removed. It also renames the respective executor classes:
- `JoinTaskExecutor` -> `NodeJoinExecutor`
- `NodeRemovalClusterStateTaskExecutor` -> `NodeLeftExecutor`
This brings their names in line with each other, and the messages that
they emit, whilst preserving the older `node-join` and `node-left`
terminology as reported by the `MasterService`.
Finally, it updates the troubleshooting logs to reflect these new and
simplified logs.
Relates #92741
Every node (post `8.7`) collects stats from every health-api request it receives. We extend the `_xpack/usage` endpoint to expose these stats. When a node receives the request it will fan out to collect data from all other nodes, merge them and expose them. If the cluster is not fully upgraded, it will signal it with the `available` flag set to`false`.
Update required ODBC .NET Framework version to reflect the compatibility
with the 4.x versions, replacing pinning it to the (EOL'd) 4.0 one.
Replace the link to the 4.8 framework with the generic .net page link.
Docs around the `index` option were not very precise. The term "typical" was used without describing for which fields querying is still available when `index: false` is set. But more precise docs existed in the `doc_values` documentation found here for the index option: https://www.elastic.co/guide/en/elasticsearch/reference/current/doc-values.html This docs were mostly copied over.
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
Currently Elasticsearch always returns a shard failure once a runtime error arises from using a runtime field, the exception being script-less runtime fields. This also means that execution of the query for that shard stops, which is okay for development and exploration. In a production scenario, however, it is often desirable to ignore runtime errors and continue with the query execution.
This change adds a new a new on_script_error parameter to runtime field definitions similar to the already existing
parameter for index-time scripted fields. When `on_script_error` is set to `continue`, errors from script execution are effectively ignored. This means affected documents don't show up in query results, but also don't prevent other matches from the same shard. Runtime fields accessed through the fields API don't return values on errors, aggregations will ignore documents that throw errors.
Note that this change affects scripted runtime fields only, while leaving default behaviour untouched. Also, ignored errors are not reported back to users for now.
Relates to #72143
The docs for the TCP readiness port have a preview warning, with custom
text. However, this doesn't render correctly in the docs. Rather than
figure out why the custom text doesn't render, this commit simply
removes the custom text in favor of the default preview text.
* The exception is inserted in a code block
* Update docs/reference/mapping/types/text.asciidoc
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
The `shards_availability` indicator diagnoses the condition where
indices need to be restored from snapshot.
Starting with 8.0 using feature_states when restoring from snapshot is
mandatory.
This adds support for the `FEATURE_STATE` affected resource to aid with
building up the snapshot restore API call (which will need to include
all the indices and feature states reported by the restore-from-snapshot
diagnosis).
Note that the health API will not report any indices that are part of a
feature state.
This adds a `size` parameter that controls the maximum number of
returned affected resources. The parameter defaults to `1000`, must be
positive, and less than `10_000`
Due to #41185 datafeeds created prior to 7.0 and not updated since
then have unparseable authorization headers in 8.x. In 8.0-8.3 this
could easily be a non-issue, as such datafeeds were likely forgotten
leftovers and never run. Even if it was a problem, only the datafeed
of interest would need updating with any urgency.
Due to #87884 datafeeds with authorization headers older than 7.0
prevent _all_ datafeeds being listed in 8.4 and 8.5. This in turn
breaks the anomaly detection job management section of the ML UI.
The problem is alleviated by #92168 and fixed by #92221, but we
should warn users that the problem exists in 8.4.0-8.5.3 inclusive.
