This PR adds a count of currently unassigned primary shards to both the
`/_cat/health` and `/_cluster/health` endpoints. This is to aid cluster
administrators in estimating the time remaining for a cluster to go from
RED to YELLOW status as per enchancement request #111727.
Tests and doc updates are in place with this PR and manual testing with
`./gradlew run` has been conducted on the endpoints to ensure correct
output.
## Known Limitations * Testing * Due to limitations in the YAML REST
test framework skip functionality, YAML REST tests for this endpoint are
disabled when running a mixed version cluster by using a cluster version
number synthetic feature to skip when any member of the cluster is not
at a version greater than when this change is due to be introduced
In this PR we expose the global retention via the `GET
_data_stream/{target}/_lifecycle` API.
Since the global retention is a main feature of the data stream
lifecycle we chose to expose it by default.
```
GET /_data_stream/my-data-stream/_lifecycle
{
"global_retention": {
"default_retention": "7d",
"max_retention": "365d"
},
"data_streams": [...]
}
```
This commit adds support for the `verbose` querystring parameter to the
get data stream API (`GET /_data_stream/{name}`).
The flag defaults to "false".
When set to true, the `maximum_timestamp` for the data stream will be
retrieved and returned for each data stream retrieved. This is the same
information available from the data stream stats API (and internally
uses the same action to retrieval).
We duplicated these docs in order to avoid breaking older links, but
this makes it confusing and hard to link to the right copy of the
information. This commit removes the duplication by replacing the docs
at the old locations with stubs that link to the new locations.
A misplaced `//end::` tag meant that the docs added in #112271 are only
included in the page on fault detection and not the equivalent
troubleshooting docs. This commit fixes the problem.
Basically the same as for nodes that leave the cluster with reason
`disconnected`, except that these disconnects don't involve the master
so don't cause any nodes to leave the cluster.
With #111972 we enable users to set up global retention for data streams that are managed by the data stream lifecycle. This will allow users of elasticsearch to have a more control over their data retention, and consequently better resource management of their clusters.
However, there is a small number of data streams that are necessary for the good operation of elasticsearch and should not follow user defined retention to avoid surprises.
For this reason, we put forth the following definition of internal data streams.
A data stream is internal if it's either a system index (system flag is true) or if its name starts with a dot.
This PR adds the `isInternalDataStream` param in the effective retention calculation making explicit that this is also used to determine the effective retention.
Adds to the `GET _cluster/stats` endpoint information about the snapshot
repositories in use, including their types, whether they are read-only
or read-write, and for Azure repositories the kind of credentials in
use.
The max enrich cache size setting now also supports an absolute max size in bytes (of used heap space) and a percentage of the max heap space, next to the existing flat document count. The default is 1% of the max heap space.
This should prevent issues where the enrich cache takes up a lot of memory when there are large documents in the cache.
Add the ability to schedule an SLM policies with a time unit interval schedule rather than a cron job schedule. For example, an slm policy can be created with the argument "schedule":"30m". This will create a policy that will run 30 minutes after the policy modification_date. It will then run again every time another 30 minutes has passed. Every time the policy is changed, the next snapshot will be re-scheduled to run one interval after the new modification date.
The info about remote cluster connection modes is a little disjointed.
This commit adds some cross-links between the sections to help users
find more relevant information.
It's not obvious from the docs that transport connections (including
connections to remote clusters) use a custom binary protocol and require
a _layer 4_ proxy. This commit clarifies this point.
<!-- Thank you for your interest in and contributing to Elasticsearch!
There are a few simple things to check before submitting your pull
request that can help with the review process. You should delete these
items from your submission, but they are here to help bring them to your
attention. -->
- Have you signed the [contributor license agreement](https://www.elastic.co/contributor-agreement)? => yes
- Have you followed the [contributor guidelines](https://github.com/elastic/elasticsearch/blob/main/CONTRIBUTING.md)? => yes
- If submitting code, have you built your formula locally prior to submission with `gradle check`? => not code
- If submitting code, is your pull request against main? Unless there is a good reason otherwise, we prefer pull requests against main and will backport as needed. => not code
- If submitting code, have you checked that your submission is for an [OS and architecture that we support](https://www.elastic.co/support/matrix#show_os)? => not code
- If you are submitting this code for a class then read our [policy](https://github.com/elastic/elasticsearch/blob/main/CONTRIBUTING.md#contributing-as-part-of-a-class) for that. => not code
## Description
Update searchable snapshot doc about the timing to notice data loss:
Sometimes searchable snapshot data is cached onto disk so user may
notice their data loss later during node restart (or on Elastic cloud -
host maintenance) after they delete their snapshots.
* (Doc+) Link API to parent Doc part1
---------
Co-authored-by: shainaraskas <shaina.raskas@elastic.co>
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
This changes the generated types tables in the docs to say `date`
instead of `datetime`. That's the name of the field in Elasticsearch so
it's a lot less confusing to call it that.
Closes#111650
- Added the `mv_percentile(values, percentile)` function
- Used as a surrogate in the `percentile(column, percentile)` aggregation
- Updated docs to specify that the surrogate _should_ be implemented if possible
The same way as mv_median does, this yields exact results (Ignoring double operations error).
For that, some decisions were made, specially in the long evaluator (Check the comments in context in `MvPercentile.java`)
Closes https://github.com/elastic/elasticsearch/issues/111591
Support Version, Keyword and Text in Max an Min aggregations.
The current implementation of both max and min does:
For non-grouping:
- Store a BytesRef
- When there's a max/min, copy it to the internal array. Grow it if needed
For grouping:
- Keep an array of BytesRef (null by default: there's no "initial/default value" here, as there's no "MAX" value for a string)
- Each BytesRef stores their own array, which will be grown as needed to copy the new max/min
Some notes:
- It's not shrinking the arrays, as to avoid having to copy, and potentially grow it again
- It's using raw arrays. But maybe it should use BigArrays to compute in the circuit breaker?
Part of https://github.com/elastic/elasticsearch/issues/110346
This laxes the check on numerical spans to allow them be specified as whole numbers. So far it was required that they be provided as a double.
This also expands the tests for date ranges to include string types.
Resolves#109340, resolves#104646, resolves#105375.
In this PR we introduce cluster settings to manage the global data stream retention.
We introduce two settings `data_streams.lifecycle.retention.max` & `data_streams.lifecycle.retention.default` that configure the respective retentions. The settings are loaded and monitored by the `DataStreamGlobalRetentionSettings`. The validation has also moved there.
We preserved the `DataStreamGlobalRetention` record to reduce the impact of this change. The purpose of this method is to be simply a wrapper record that groups the retention settings together.
Temporarily, the `DataStreamGlobalRetentionSettings` is using the DataStreamFactoryRetention which is marked as deprecated for migration purposes.
This profiles additional timing information for each individual driver.
To the results from `profile` it adds the start and stop time for each
driver. That was already in the task status. To the profile and task
status it also adds the number of times the driver slept and some more
detailed history about a few of those times.
Explanation time! The compute engine splits work into some number of
`Drivers` per node. Each `Driver` is a single threaded entity - it runs
on a thread for a while then does one of three things: 1. Finishes 2.
Goes async because one of it's `Operator`s has gone async 3. Yields the
thread pool because it has run for too long
This PR measures the second two. At this point only three operators can
go async: * ENRICH * Reading from an empty exchange * Writing to a full
exchange
We're quite interested the these sleeps at the moment because they think
they may be slowing things down. Here's what it looks like when a driver
goes async because it wants to read from an empty exchange:
```
... the rest of the profile ...
"sleeps" : {
"counts" : {
"exchange empty" : 2
},
"first" : [
{
"reason" : "exchange empty",
"sleep" : "2024-08-13T19:45:57.943Z",
"sleep_millis" : 1723578357943,
"wake" : "2024-08-13T19:45:58.159Z",
"wake_millis" : 1723578358159
},
{
"reason" : "exchange empty",
"sleep" : "2024-08-13T19:45:58.164Z",
"sleep_millis" : 1723578358164,
"wake" : "2024-08-13T19:45:58.165Z",
"wake_millis" : 1723578358165
}
],
"last": [same as above]
```
Every time the driver goes async we count it in the `counts` map -
grouped by the reason the driver slept. We also record the sleep and
wake times for the first and last ten times the driver sleeps. In this
case it only slept twice, so the `first` and `last` ten times is the
same array.
This should give us a good sense about why drivers sleep while using a
limited amount of memory per driver.
If Elasticsearch fails part-way through a multipart upload to S3 it will
generally try and abort the upload, but it's possible that the abort
attempt also fails. In this case the upload becomes _dangling_. Dangling
uploads consume storage space, and therefore cost money, until they are
eventually aborted.
Earlier versions of Elasticsearch require users to check for dangling
multipart uploads, and to manually abort any that they find. This commit
introduces a cleanup process which aborts all dangling uploads on each
snapshot delete instead.
Closes#44971Closes#101169
The `known-issue-8.15.0` anchor appears twice which breaks the docs
build. Also the existing message suggests incorrectly that
`bootstrap.memory_lock: true` is recommended.
Today there are a couple of assertions that can trip if the contents of
a snapshot repostiory are corrupted. It makes sense to assert the
integrity of snapshots in most tests, but we must also (a) protect
against these corruptions in production and (b) allow some tests to
verify the behaviour of the system when the repository is corrupted.
This commit introduces a flag to disable certain assertions, converts
the relevant assertions into production failures too, and introduces a
high-level test to verify that we do detect all relevant corruptions
without tripping any other assertions.
Extracted from #93735 as this change makes sense in its own right.
Relates #52622.
