Removes `testenv` annotations and related code. These annotations originally let you skip x-pack snippet tests in the docs. However, that's no longer possible.
Relates to #79309, #31619
This commit adds two related changes:
* ILM WaitForDataTierStep
* Autoscaling frozen_existence decider
The first part ensures that we wait mounting an index until a node that
can hold the index is available, avoiding a failed restore and red
cluster state. This is in particular important for the frozen phase, but
is done generically in the searchable snapshot action.
The second part triggers on indices in the ILM frozen phase to scale the
tier into existence by requiring a minimal amount of memory and storage.
Closes#72771
Fixed autoscaling docs to no longer call partially mounted indices or
shards for frozen indices/shards, now uses partially mounted indices or
shards.
Closes#73132
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
The frozen tier partially downloads shards only. This commit
introduces an autoscaling decider that scales the total storage
on the tier according to a configurable percentage relative to
the total data set size.
Autoscaling expects data tiers to be used exclusively both for node
roles and in ILM policies. This commit adds a test demonstrating that
as well as documentation for the behavior.
We no longer regard the autoscaling APIs experimental though they are
only intended for use by ESS/ECE/ECK. This commit updates the docs
to reflect this and adds a minimal set of documentation for the
feature.
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Removed the autoscaling feature flags, autoscaling is now on by default
(though it requires an external system to handle the autoscaling
events). Added experimental notice to all autoscaling related
documentation pages.
Relates #51191
Added the capability to delete autoscaling policies by pattern, allowing
to for instance do:
```
DELETE _autoscaling/policy/*
```
to delete all autoscaling policies. If a wildcard is involved, no
matches are required.
Add a roles specification to autoscaling policies. This is used to map
the policy to a set of nodes governed by the policy. The list of roles
is mandatory when adding a policy, optional on updates.
This commit also removes the outer level "policy" element from autoscaling
policy PUT and GET requests.
Renamed decision API to capacity. Responses now prefer objects/maps over
arrays. Removed mention of tier, using policies as the outer map and
total for the policy-wide total capacity.
The autoscaling decision API now returns an absolute capacity,
and leaves the actual decision of whether a scale up or down
is needed to the orchestration system.
The decision API now returns both a tier and node level required
and current capacity as wells as a decider level breakdown of the
same though with in particular current memory still not populated.
Split the autoscaling decider into a service and configuration
in order to enable having additional context information available
in the service. Added AutoscalingDeciderContext holding generic
information all deciders are expected to need. Implemented GET
_autoscaling/decision
This commit is the first in a series of commits that introduces
autoscaling policies, and APIs for working with them. For now, we
introduce the basic infrastructure, and a single API for putting an
autoscaling policy. We will follow in rapid succession with APIs for
getting, and deleting autoscaling policies.
This is the first in a series of commits that will introduce the
autoscaling deciders framework. This commit introduces the basic
framework for representing autoscaling decisions.
The main purpose of this commit is to add a single autoscaling REST
endpoint skeleton, for the purpose of starting to build out the build
and testing infrastructure that will surround it. For example, rather
than commiting a fully-functioning autoscaling API, we introduce here
the skeleton so that we can start wiring up the build and testing
infrastructure, establish security roles/permissions, an so on. This
way, in a forthcoming PR that introduces actual functionality, that PR
will be smaller and have less distractions around that sort of
infrastructure.
This commit merely adds the skeleton for the autoscaling project, adding
the basics to include the autoscaling module in the default
distribution, opt-in to code formatting, and a placeholder for the docs.
