mirror of
https://github.com/elastic/elasticsearch.git
synced 2025-06-28 09:28:55 -04:00
[DOCS] Move snapshot repo types to separate pages (#82826)
With https://github.com/elastic/elasticsearch/pull/81870, the Azure, GCS, and S3 repository types have separate, dedicated pages in the Elasticsearch guide. For consistency, this PR creates separate pages for the shared file system, read-only URL, and source-only repository types. Related changes: - Adds redirects to the plugins docs - Fixes a few breaking changes that refer to the Azure, GCS, and S3 repositories as plugins. Co-authored-by: Adam Locke <adam.locke@elastic.co>
This commit is contained in:
James Rodewig
GitHub
parent
86174d6add
commit
cb6265f9bd
10 changed files with 358 additions and 296 deletions
|
|
@ -71,3 +71,77 @@ See {ref}/monitor-elasticsearch-cluster.html[{stack} monitoring].
|
|||
=== Security plugins
|
||||
|
||||
See {ref}/secure-cluster.html[{stack} security].
|
||||
|
||||
[role="exclude",id="repository-azure"]
|
||||
=== Azure repository plugin
|
||||
|
||||
// tag::azure-repo-migration[]
|
||||
The Azure repository plugin is now included in {es}.
|
||||
See {ref}/repository-azure.html[Azure repository].
|
||||
// end::azure-repo-migration[]
|
||||
|
||||
[role="exclude",id="repository-azure-usage"]
|
||||
=== Azure repository plugin
|
||||
|
||||
include::redirects.asciidoc[tag=azure-repo-migration]
|
||||
|
||||
[role="exclude",id="repository-azure-client-settings"]
|
||||
=== Azure repository plugin
|
||||
|
||||
include::redirects.asciidoc[tag=azure-repo-migration]
|
||||
|
||||
[role="exclude",id="repository-azure-repository-settings"]
|
||||
=== Azure repository plugin
|
||||
|
||||
include::redirects.asciidoc[tag=azure-repo-migration]
|
||||
|
||||
[role="exclude",id="repository-azure-validation"]
|
||||
=== Azure repository plugin
|
||||
|
||||
include::redirects.asciidoc[tag=azure-repo-migration]
|
||||
|
||||
[role="exclude",id="repository-s3"]
|
||||
=== S3 repository plugin
|
||||
|
||||
// tag::s3-repo-migration[]
|
||||
The S3 repository plugin is now included in {es}.
|
||||
See {ref}/repository-s3.html[S3 repository].
|
||||
// end::s3-repo-migration[]
|
||||
|
||||
[role="exclude",id="repository-s3-usage"]
|
||||
=== S3 repository plugin
|
||||
|
||||
include::redirects.asciidoc[tag=s3-repo-migration]
|
||||
|
||||
[role="exclude",id="repository-s3-client"]
|
||||
=== S3 repository plugin
|
||||
|
||||
include::redirects.asciidoc[tag=s3-repo-migration]
|
||||
|
||||
[role="exclude",id="repository-s3-repository"]
|
||||
=== S3 repository plugin
|
||||
|
||||
include::redirects.asciidoc[tag=s3-repo-migration]
|
||||
|
||||
[role="exclude",id="repository-gcs"]
|
||||
=== Google Cloud Storage repository plugin
|
||||
|
||||
// tag::gcs-repo-migration[]
|
||||
The Google Cloud Storage repository plugin is now included in {es}.
|
||||
See {ref}/repository-gcs.html[Google Cloud Storage repository].
|
||||
// end::gcs-repo-migration[]
|
||||
|
||||
[role="exclude",id="repository-gcs-usage"]
|
||||
=== Google Cloud Storage repository plugin
|
||||
|
||||
include::redirects.asciidoc[tag=gcs-repo-migration]
|
||||
|
||||
[role="exclude",id="repository-gcs-client"]
|
||||
=== Google Cloud Storage repository plugin
|
||||
|
||||
include::redirects.asciidoc[tag=gcs-repo-migration]
|
||||
|
||||
[role="exclude",id="repository-gcs-repository"]
|
||||
=== Google Cloud Storage repository plugin
|
||||
|
||||
include::redirects.asciidoc[tag=gcs-repo-migration]
|
||||
|
|
|
|||
|
|
@ -875,11 +875,10 @@ previously be used to enable compression for all shared file system repositories
|
|||
The `repositories.fs.compress` setting has been removed.
|
||||
|
||||
*Impact* +
|
||||
Use the repository specific `compress` setting to enable compression. See
|
||||
{ref}/snapshots-register-repository.html[Register a snapshot repository] for
|
||||
information on the `compress` setting.
|
||||
|
||||
Discontinue use of the `repositories.fs.compress` node-level setting.
|
||||
Discontinue use of the `repositories.fs.compress` node-level setting. Use the
|
||||
repository-specific `compress` setting to enable compression instead. Refer to
|
||||
{ref}/snapshots-filesystem-repository.html#filesystem-repository-settings[Shared
|
||||
file system repository settings].
|
||||
====
|
||||
//end::notable-breaking-changes[]
|
||||
|
||||
|
|
|
|||
|
|
@ -991,21 +991,18 @@ Previously, the default value for `compress` was `false`. The default has been c
|
|||
This change will affect both newly created repositories and existing repositories where `compress=false` has not been
|
||||
explicitly specified.
|
||||
|
||||
For more information on the compress option, see
|
||||
{ref}/snapshots-register-repository.html[Register a snapshot repository].
|
||||
|
||||
*Impact* +
|
||||
Update your workflow and applications to assume a default value of `true` for
|
||||
the `compress` parameter.
|
||||
====
|
||||
|
||||
.The S3 repository plugin now uses a DNS-style access pattern by default.
|
||||
.S3 snapshot repositories now use a DNS-style access pattern by default.
|
||||
[%collapsible]
|
||||
====
|
||||
*Details* +
|
||||
Starting in version 7.4 the `repository-s3` plugin does not use the
|
||||
Starting in version 7.4, `s3` snapshot repositories no longer use the
|
||||
now-deprecated path-style access pattern by default. In versions 7.0, 7.1, 7.2
|
||||
and 7.3 the `repository-s3` plugin always used the path-style access pattern.
|
||||
and 7.3 `s3` snapshot repositories always used the path-style access pattern.
|
||||
This is a breaking change for deployments that only support path-style access
|
||||
but which are recognized as supporting DNS-style access by the AWS SDK. This
|
||||
breaking change was made necessary by
|
||||
|
|
|
|||
|
|
@ -75,183 +75,42 @@ You can manually perform this verification using the
|
|||
|
||||
[[put-snapshot-repo-api-request-type]]
|
||||
`type`::
|
||||
+
|
||||
--
|
||||
(Required, string)
|
||||
Repository type.
|
||||
|
||||
+
|
||||
.Valid values for `type`
|
||||
[%collapsible%open]
|
||||
====
|
||||
`fs`::
|
||||
Shared file system repository. Repositories of this type use a shared file
|
||||
system to store snapshots. This file system must accessible to all master and
|
||||
data nodes in the cluster.
|
||||
+
|
||||
IMPORTANT: To register a shared file system repository, you must mount the same
|
||||
shared filesystem to the same location on all master and data nodes. This
|
||||
location must be registered in the `path.repo` setting on all master and data
|
||||
nodes in the cluster.
|
||||
+
|
||||
See <<snapshots-filesystem-repository>>.
|
||||
|
||||
[xpack]#`source`#::
|
||||
Source-only repository. You can use source-only repositories to create minimal,
|
||||
source-only snapshots that take up to 50% less space on disk.
|
||||
+
|
||||
Source-only snapshots are only supported if the <<mapping-source-field,`_source`
|
||||
field>> is enabled and no
|
||||
<<source-filtering,source-filtering>> is applied.
|
||||
+
|
||||
WARNING: Source-only snapshots contain stored fields and index metadata. They do
|
||||
not include index or doc values structures and are not searchable when restored.
|
||||
After restoring a source-only snapshot, you must <<docs-reindex,reindex>> the
|
||||
data into a new index.
|
||||
+
|
||||
See <<snapshots-source-only-repository>>.
|
||||
`azure`:: <<repository-azure>>
|
||||
`gcs`:: <<repository-gcs>>
|
||||
`s3`:: <<repository-s3>>
|
||||
`fs`:: <<snapshots-filesystem-repository>>
|
||||
`source`:: <<snapshots-source-only-repository>>
|
||||
`url`:: <<snapshots-read-only-repository>>
|
||||
|
||||
`url`::
|
||||
URL repository. Repositories of this type are read-only
|
||||
for the cluster. This means the cluster can retrieve or restore snapshots from
|
||||
the repository but cannot write or create snapshots in it.
|
||||
+
|
||||
You can use URL repositories as an alternative way to give a cluster read-only
|
||||
access to a shared file system (`fs`) repository.
|
||||
+
|
||||
See <<snapshots-read-only-repository>>.
|
||||
Other repository types are available through official plugins:
|
||||
|
||||
`hfds`:: {plugins}/repository-hdfs.html[Hadoop Distributed File System (HDFS) repository]
|
||||
====
|
||||
|
||||
More repository types are available through these official
|
||||
plugins:
|
||||
|
||||
* <<repository-s3,repository-s3>> for S3 repository support
|
||||
* {plugins}/repository-hdfs.html[repository-hdfs] for HDFS repository support in
|
||||
Hadoop environments
|
||||
* <<repository-azure,repository-azure>> for Azure storage repositories
|
||||
* <<repository-gcs,repository-gcs>> for Google Cloud Storage repositories
|
||||
--
|
||||
|
||||
[[put-snapshot-repo-api-settings-param]]
|
||||
`settings`::
|
||||
(Required, object)
|
||||
Settings for the repository. Supported settings vary based on the repository
|
||||
type:
|
||||
+
|
||||
--
|
||||
(Required, object)
|
||||
Contains settings for the repository.
|
||||
* <<repository-azure>>
|
||||
* <<repository-gcs>>
|
||||
* <<repository-s3>>
|
||||
* <<snapshots-filesystem-repository>>
|
||||
* <<snapshots-read-only-repository>>
|
||||
* <<snapshots-source-only-repository>>
|
||||
|
||||
The following `settings` properties are valid for all repository types:
|
||||
Other repository types are available through official plugins:
|
||||
|
||||
.Properties of `settings`
|
||||
[%collapsible%open]
|
||||
====
|
||||
`chunk_size`::
|
||||
(Optional, <<byte-units,byte value>>)
|
||||
Maximum size of files in snapshots. In snapshots, files larger than this are
|
||||
broken down into chunks of this size or smaller. Defaults to `null` (unlimited
|
||||
file size).
|
||||
|
||||
`compress`::
|
||||
(Optional, Boolean)
|
||||
If `true`, metadata files, such as index mappings and settings, are compressed
|
||||
in snapshots. Data files are not compressed. Defaults to `true`.
|
||||
|
||||
`max_number_of_snapshots`::
|
||||
(Optional, integer)
|
||||
Maximum number of snapshots the repository can contain.
|
||||
Defaults to `Integer.MAX_VALUE`, which is `2^31-1` or `2147483647`.
|
||||
|
||||
`max_restore_bytes_per_sec`::
|
||||
(Optional, <<byte-units,byte value>>)
|
||||
Maximum snapshot restore rate per node. Defaults to unlimited. Note
|
||||
that restores are also throttled through <<recovery,recovery settings>>.
|
||||
|
||||
`max_snapshot_bytes_per_sec`::
|
||||
(Optional, <<byte-units,byte value>>)
|
||||
Maximum snapshot creation rate per node. Defaults to `40mb` per second.
|
||||
|
||||
`readonly`::
|
||||
(Optional, Boolean)
|
||||
If `true`, the repository is read-only. The cluster can retrieve and restore
|
||||
snapshots from the repository but not write to the repository or create
|
||||
snapshots in it.
|
||||
+
|
||||
If `false`, the cluster can write to the repository and create snapshots in it.
|
||||
Defaults to `false`.
|
||||
+
|
||||
[TIP]
|
||||
=====
|
||||
If you register the same snapshot repository with multiple clusters, only
|
||||
one cluster should have write access to the repository. Having multiple clusters
|
||||
write to the repository at the same time risks corrupting the contents of the
|
||||
repository.
|
||||
|
||||
Only a cluster with write access can create snapshots in the repository. All
|
||||
other clusters connected to the repository should have the `readonly` parameter
|
||||
set to `true`. This means those clusters can retrieve or restore snapshots from
|
||||
the repository but not create snapshots in it.
|
||||
=====
|
||||
====
|
||||
|
||||
Other accepted `settings` properties depend on the repository type, set using the
|
||||
<<put-snapshot-repo-api-request-type,`type`>> parameter.
|
||||
|
||||
.Valid `settings` properties for `fs` repositories
|
||||
[%collapsible%open]
|
||||
====
|
||||
`location`::
|
||||
(Required, string)
|
||||
Location of the shared filesystem used to store and retrieve snapshots. This
|
||||
location must be registered in the `path.repo` setting on all master and data
|
||||
nodes in the cluster.
|
||||
====
|
||||
|
||||
.Valid `settings` properties for `source` repositories
|
||||
[%collapsible%open]
|
||||
====
|
||||
`delegate_type`::
|
||||
(Optional, string)
|
||||
Delegated repository type. For valid values, see the
|
||||
<<put-snapshot-repo-api-request-type,`type` parameter>>.
|
||||
+
|
||||
`source` repositories can use `settings` properties for its delegated repository
|
||||
type. See <<snapshots-source-only-repository>>.
|
||||
|
||||
====
|
||||
|
||||
.Valid `settings` properties for `url` repositories
|
||||
[%collapsible%open]
|
||||
====
|
||||
`url`::
|
||||
+
|
||||
---
|
||||
(Required, string)
|
||||
URL location of the root of the shared filesystem repository. The following
|
||||
protocols are supported:
|
||||
|
||||
* `file`
|
||||
* `ftp`
|
||||
* `http`
|
||||
* `https`
|
||||
* `jar`
|
||||
|
||||
URLs using the `http`, `https`, or `ftp` protocols must be explicitly allowed
|
||||
with the <<repositories-url-allowed,`repositories.url.allowed_urls`>> cluster
|
||||
setting. This setting supports wildcards in the place of a host, path, query, or
|
||||
fragment in the URL.
|
||||
|
||||
URLs using the `file` protocol must point to the location of a shared filesystem
|
||||
accessible to all master and data nodes in the cluster. This location must be
|
||||
registered in the `path.repo` setting. You don't need to register URLs using the
|
||||
`ftp`, `http`, `https`, or `jar` protocols in the `path.repo` setting.
|
||||
---
|
||||
|
||||
`http_max_retries`::
|
||||
(Optional, integer) Maximum number of retries for `http` and `https` URLs.
|
||||
Defaults to `5`.
|
||||
|
||||
`http_socket_timeout`::
|
||||
(Optional, <<time-units,time value>>) Maximum wait time for data transfers over
|
||||
a connection. Defaults to `50s`.
|
||||
====
|
||||
* {plugins}/repository-hdfs.html[Hadoop Distributed File System (HDFS) repository]
|
||||
--
|
||||
|
||||
`verify`::
|
||||
|
|
|
|||
|
|
@ -0,0 +1,2 @@
|
|||
NOTE: This repository type is only available if you run {es} on your own
|
||||
hardware. If you use {ess}, see <<ess-repo-types>>.
|
||||
|
|
@ -70,11 +70,17 @@ To manage repositories in {kib}, go to the main menu and click **Stack
|
|||
Management** > **Snapshot and Restore** > **Repositories**. To register a
|
||||
snapshot repository, click **Register repository**.
|
||||
|
||||
You can also register a repository using the <<put-snapshot-repo-api,Create
|
||||
snapshot repository API>>.
|
||||
|
||||
[discrete]
|
||||
[[snapshot-repo-types]]
|
||||
=== Snapshot repository types
|
||||
|
||||
Supported snapshot repository types vary based on your deployment type.
|
||||
Supported snapshot repository types vary based on your deployment type:
|
||||
|
||||
* <<ess-repo-types>>
|
||||
* <<self-managed-repo-types>>
|
||||
|
||||
[discrete]
|
||||
[[ess-repo-types]]
|
||||
|
|
@ -94,10 +100,10 @@ clusters].
|
|||
|
||||
{ess} deployments also support the following repository types:
|
||||
|
||||
* {cloud}/ec-azure-snapshotting.html[Azure]
|
||||
* {cloud}/ec-gcs-snapshotting.html[Google Cloud Storage]
|
||||
* {cloud}/ec-aws-custom-repository.html[AWS S3]
|
||||
* {cloud}/ec-gcs-snapshotting.html[Google Cloud Storage (GCS)]
|
||||
* {cloud}/ec-azure-snapshotting.html[Microsoft Azure]
|
||||
* <<snapshots-source-only-repository>>
|
||||
* <<snapshots-source-only-repository,Source-only>>
|
||||
|
||||
[discrete]
|
||||
[[self-managed-repo-types]]
|
||||
|
|
@ -106,12 +112,12 @@ clusters].
|
|||
If you run {es} on your own hardware, you can use the following built-in
|
||||
snapshot repository types:
|
||||
|
||||
* <<repository-s3,AWS S3>>
|
||||
* <<repository-gcs,Google Cloud Storage>>
|
||||
* <<repository-azure,Azure>>
|
||||
* <<repository-gcs,Google Cloud Storage>>
|
||||
* <<repository-s3,AWS S3>>
|
||||
* <<snapshots-filesystem-repository,Shared file system>>
|
||||
* <<snapshots-read-only-repository>>
|
||||
* <<snapshots-source-only-repository>>
|
||||
* <<snapshots-read-only-repository,Read-only URL>>
|
||||
* <<snapshots-source-only-repository,Source-only>>
|
||||
|
||||
[[snapshots-repository-plugins]]
|
||||
Other repository types are available through official plugins:
|
||||
|
|
@ -122,108 +128,6 @@ You can also use alternative implementations of these repository types, such as
|
|||
MinIO, as long as they're compatible. To verify a repository's compatibility,
|
||||
see <<snapshots-repository-verification>>.
|
||||
|
||||
[discrete]
|
||||
[[snapshots-filesystem-repository]]
|
||||
==== Shared file system repository
|
||||
|
||||
// tag::on-prem-repo-type[]
|
||||
NOTE: This repository type is only available if you run {es} on your own
|
||||
hardware. If you use {ess}, see <<ess-repo-types>>.
|
||||
// end::on-prem-repo-type[]
|
||||
|
||||
Use a shared file system repository to store snapshots on a
|
||||
shared file system.
|
||||
|
||||
To register a shared file system repository, first mount the file system to the
|
||||
same location on all master and data nodes. Then add the file system's
|
||||
path or parent directory to the `path.repo` setting in `elasticsearch.yml` for
|
||||
each master and data node. For running clusters, this requires a
|
||||
<<restart-cluster-rolling,rolling restart>> of each node.
|
||||
|
||||
IMPORTANT: By default, a network file system (NFS) uses user IDs (UIDs) and
|
||||
group IDs (GIDs) to match accounts across nodes. If your shared file system is
|
||||
an NFS and your nodes don't use the same UIDs and GIDs, update your NFS
|
||||
configuration to account for this.
|
||||
|
||||
Supported `path.repo` values vary by platform:
|
||||
|
||||
include::{es-repo-dir}/tab-widgets/register-fs-repo-widget.asciidoc[]
|
||||
|
||||
[discrete]
|
||||
[[snapshots-read-only-repository]]
|
||||
==== Read-only URL repository
|
||||
|
||||
include::register-repository.asciidoc[tag=on-prem-repo-type]
|
||||
|
||||
You can use a URL repository to give a cluster read-only access to a shared file
|
||||
system. Since URL repositories are always read-only, they're a safer and more
|
||||
convenient alternative to registering a read-only shared filesystem repository.
|
||||
|
||||
Use {kib} or the <<put-snapshot-repo-api,create snapshot repository API>> to
|
||||
register a URL repository.
|
||||
|
||||
[source,console]
|
||||
----
|
||||
PUT _snapshot/my_read_only_url_repository
|
||||
{
|
||||
"type": "url",
|
||||
"settings": {
|
||||
"url": "file:/mount/backups/my_fs_backup_location"
|
||||
}
|
||||
}
|
||||
----
|
||||
// TEST[skip:no access to url file path]
|
||||
|
||||
[discrete]
|
||||
[[snapshots-source-only-repository]]
|
||||
==== Source-only repository
|
||||
|
||||
You can use a source-only repository to take minimal, source-only snapshots that
|
||||
use up to 50% less disk space than regular snapshots.
|
||||
|
||||
Unlike other repository types, a source-only repository doesn't directly store
|
||||
snapshots. It delegates storage to another registered snapshot repository.
|
||||
|
||||
When you take a snapshot using a source-only repository, {es} creates a
|
||||
source-only snapshot in the delegated storage repository. This snapshot only
|
||||
contains stored fields and metadata. It doesn't include index or doc values
|
||||
structures and isn't immediately searchable when restored. To search the
|
||||
restored data, you first have to <<docs-reindex,reindex>> it into a new data
|
||||
stream or index.
|
||||
|
||||
[IMPORTANT]
|
||||
==================================================
|
||||
|
||||
Source-only snapshots are only supported if the `_source` field is enabled and no source-filtering is applied.
|
||||
When you restore a source-only snapshot:
|
||||
|
||||
* The restored index is read-only and can only serve `match_all` search or scroll requests to enable reindexing.
|
||||
|
||||
* Queries other than `match_all` and `_get` requests are not supported.
|
||||
|
||||
* The mapping of the restored index is empty, but the original mapping is available from the types top
|
||||
level `meta` element.
|
||||
|
||||
==================================================
|
||||
|
||||
Before registering a source-only repository, use {kib} or the
|
||||
<<put-snapshot-repo-api,create snapshot repository API>> to register a snapshot
|
||||
repository of another type to use for storage. Then register the source-only
|
||||
repository and specify the delegated storage repository in the request.
|
||||
|
||||
[source,console]
|
||||
----
|
||||
PUT _snapshot/my_src_only_repository
|
||||
{
|
||||
"type": "source",
|
||||
"settings": {
|
||||
"delegate_type": "fs",
|
||||
"location": "my_backup_location"
|
||||
}
|
||||
}
|
||||
----
|
||||
// TEST[continued]
|
||||
|
||||
[discrete]
|
||||
[[snapshots-repository-verification]]
|
||||
=== Verify a repository
|
||||
|
|
@ -245,7 +149,8 @@ PUT _snapshot/my_unverified_backup?verify=false
|
|||
}
|
||||
}
|
||||
----
|
||||
// TEST[continued]
|
||||
// TEST[setup:setup-repository]
|
||||
// TEST[s/my_unverified_backup_location/my_repository/]
|
||||
|
||||
If wanted, you can manually run the repository verification check. To verify a
|
||||
repository in {kib}, go to the **Repositories** list page and click the name of
|
||||
|
|
@ -257,6 +162,7 @@ a repository. Then click **Verify repository**. You can also use the
|
|||
POST _snapshot/my_unverified_backup/_verify
|
||||
----
|
||||
// TEST[continued]
|
||||
// TEST[s/my_unverified_backup_location/my_repository/]
|
||||
|
||||
If successful, the request returns a list of nodes used to verify the
|
||||
repository. If verification fails, the request returns an error.
|
||||
|
|
@ -285,7 +191,7 @@ API>>.
|
|||
----
|
||||
POST _snapshot/my_repository/_cleanup
|
||||
----
|
||||
// TEST[continued]
|
||||
// TEST[setup:setup-snapshots]
|
||||
|
||||
The API returns:
|
||||
|
||||
|
|
@ -298,6 +204,8 @@ The API returns:
|
|||
}
|
||||
}
|
||||
----
|
||||
// TESTRESPONSE[s/"deleted_bytes": 20/"deleted_bytes": $body.results.deleted_bytes/]
|
||||
// TESTRESPONSE[s/"deleted_blobs": 5/"deleted_blobs": $body.results.deleted_bytes/]
|
||||
|
||||
Depending on the concrete repository implementation the numbers shown for bytes free as well as the number of blobs removed will either
|
||||
be an approximation or an exact result. Any non-zero value for the number of blobs removed implies that unreferenced blobs were found and
|
||||
|
|
@ -336,9 +244,9 @@ with {es} until the repository contents are fully restored. If you alter the
|
|||
contents of a repository while it is registered with {es} then the repository
|
||||
may become unreadable or may silently lose some of its contents.
|
||||
|
||||
|
||||
include::repository-s3.asciidoc[]
|
||||
|
||||
include::repository-gcs.asciidoc[]
|
||||
|
||||
include::repository-azure.asciidoc[]
|
||||
include::repository-gcs.asciidoc[]
|
||||
include::repository-s3.asciidoc[]
|
||||
include::repository-shared-file-system.asciidoc[]
|
||||
include::repository-read-only-url.asciidoc[]
|
||||
include::repository-source-only.asciidoc[]
|
||||
|
|
|
|||
|
|
@ -0,0 +1,76 @@
|
|||
[[snapshots-read-only-repository]]
|
||||
=== Read-only URL repository
|
||||
|
||||
include::{es-repo-dir}/snapshot-restore/on-prem-repo-type.asciidoc[]
|
||||
|
||||
You can use a URL repository to give a cluster read-only access to a shared file
|
||||
system. Since URL repositories are always read-only, they're a safer and more
|
||||
convenient alternative to registering a read-only shared filesystem repository.
|
||||
|
||||
Use {kib} or the <<put-snapshot-repo-api,create snapshot repository API>> to
|
||||
register a URL repository.
|
||||
|
||||
[source,console]
|
||||
----
|
||||
PUT _snapshot/my_read_only_url_repository
|
||||
{
|
||||
"type": "url",
|
||||
"settings": {
|
||||
"url": "file:/mount/backups/my_fs_backup_location"
|
||||
}
|
||||
}
|
||||
----
|
||||
// TEST[skip:no access to url file path]
|
||||
|
||||
[[read-only-url-repository-settings]]
|
||||
==== Repository settings
|
||||
|
||||
`chunk_size`::
|
||||
(Optional, <<byte-units,byte value>>)
|
||||
Maximum size of files in snapshots. In snapshots, files larger than this are
|
||||
broken down into chunks of this size or smaller. Defaults to `null` (unlimited
|
||||
file size).
|
||||
|
||||
`http_max_retries`::
|
||||
(Optional, integer) Maximum number of retries for `http` and `https` URLs.
|
||||
Defaults to `5`.
|
||||
|
||||
`http_socket_timeout`::
|
||||
(Optional, <<time-units,time value>>) Maximum wait time for data transfers over
|
||||
a connection. Defaults to `50s`.
|
||||
|
||||
`compress`::
|
||||
(Optional, Boolean)
|
||||
If `true`, metadata files, such as index mappings and settings, are compressed
|
||||
in snapshots. Data files are not compressed. Defaults to `true`.
|
||||
|
||||
`max_number_of_snapshots`::
|
||||
(Optional, integer)
|
||||
Maximum number of snapshots the repository can contain.
|
||||
Defaults to `Integer.MAX_VALUE`, which is `2^31-1` or `2147483647`.
|
||||
|
||||
include::repository-shared-settings.asciidoc[tags=!readonly-repo-setting]
|
||||
|
||||
`url`::
|
||||
+
|
||||
--
|
||||
(Required, string)
|
||||
URL location of the root of the shared filesystem repository. The following
|
||||
protocols are supported:
|
||||
|
||||
* `file`
|
||||
* `ftp`
|
||||
* `http`
|
||||
* `https`
|
||||
* `jar`
|
||||
|
||||
URLs using the `http`, `https`, or `ftp` protocols must be explicitly allowed
|
||||
with the <<repositories-url-allowed,`repositories.url.allowed_urls`>> cluster
|
||||
setting. This setting supports wildcards in the place of a host, path, query, or
|
||||
fragment in the URL.
|
||||
|
||||
URLs using the `file` protocol must point to the location of a shared filesystem
|
||||
accessible to all master and data nodes in the cluster. This location must be
|
||||
registered in the `path.repo` setting. You don't need to register URLs using the
|
||||
`ftp`, `http`, `https`, or `jar` protocols in the `path.repo` setting.
|
||||
--
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
[[snapshots-filesystem-repository]]
|
||||
=== Shared file system repository
|
||||
|
||||
include::{es-repo-dir}/snapshot-restore/on-prem-repo-type.asciidoc[]
|
||||
|
||||
Use a shared file system repository to store snapshots on a
|
||||
shared file system.
|
||||
|
||||
To register a shared file system repository, first mount the file system to the
|
||||
same location on all master and data nodes. Then add the file system's
|
||||
path or parent directory to the `path.repo` setting in `elasticsearch.yml` for
|
||||
each master and data node. For running clusters, this requires a
|
||||
<<restart-cluster-rolling,rolling restart>> of each node.
|
||||
|
||||
IMPORTANT: By default, a network file system (NFS) uses user IDs (UIDs) and
|
||||
group IDs (GIDs) to match accounts across nodes. If your shared file system is
|
||||
an NFS and your nodes don't use the same UIDs and GIDs, update your NFS
|
||||
configuration to account for this.
|
||||
|
||||
Supported `path.repo` values vary by platform:
|
||||
|
||||
include::{es-repo-dir}/tab-widgets/register-fs-repo-widget.asciidoc[]
|
||||
|
||||
[[filesystem-repository-settings]]
|
||||
==== Repository settings
|
||||
|
||||
`chunk_size`::
|
||||
(Optional, <<byte-units,byte value>>)
|
||||
Maximum size of files in snapshots. In snapshots, files larger than this are
|
||||
broken down into chunks of this size or smaller. Defaults to `null` (unlimited
|
||||
file size).
|
||||
|
||||
`compress`::
|
||||
(Optional, Boolean)
|
||||
If `true`, metadata files, such as index mappings and settings, are compressed
|
||||
in snapshots. Data files are not compressed. Defaults to `true`.
|
||||
|
||||
`location`::
|
||||
(Required, string)
|
||||
Location of the shared filesystem used to store and retrieve snapshots. This
|
||||
location must be registered in the `path.repo` setting on all master and data
|
||||
nodes in the cluster.
|
||||
|
||||
`max_number_of_snapshots`::
|
||||
(Optional, integer)
|
||||
Maximum number of snapshots the repository can contain.
|
||||
Defaults to `Integer.MAX_VALUE`, which is `2^31-1` or `2147483647`.
|
||||
|
||||
include::repository-shared-settings.asciidoc[]
|
||||
|
|
@ -1,12 +1,32 @@
|
|||
`max_restore_bytes_per_sec`::
|
||||
|
||||
Throttles per node restore rate. Defaults to unlimited.
|
||||
Note that restores are also throttled through {ref}/recovery.html[recovery settings].
|
||||
(Optional, <<byte-units,byte value>>)
|
||||
Maximum snapshot restore rate per node. Defaults to unlimited. Note
|
||||
that restores are also throttled through <<recovery,recovery settings>>.
|
||||
|
||||
`max_snapshot_bytes_per_sec`::
|
||||
(Optional, <<byte-units,byte value>>)
|
||||
Maximum snapshot creation rate per node. Defaults to `40mb` per second.
|
||||
|
||||
Throttles per node snapshot rate. Defaults to `40mb` per second.
|
||||
|
||||
//tag::readonly-repo-setting[]
|
||||
`readonly`::
|
||||
(Optional, Boolean)
|
||||
If `true`, the repository is read-only. The cluster can retrieve and restore
|
||||
snapshots from the repository but not write to the repository or create
|
||||
snapshots in it.
|
||||
+
|
||||
Only a cluster with write access can create snapshots in the repository. All
|
||||
other clusters connected to the repository should have the `readonly` parameter
|
||||
set to `true`.
|
||||
+
|
||||
If `false`, the cluster can write to the repository and create snapshots in it.
|
||||
Defaults to `false`.
|
||||
+
|
||||
[IMPORTANT]
|
||||
=====
|
||||
If you register the same snapshot repository with multiple clusters, only
|
||||
one cluster should have write access to the repository. Having multiple clusters
|
||||
write to the repository at the same time risks corrupting the contents of the
|
||||
repository.
|
||||
|
||||
Makes repository read-only. Defaults to `false`.
|
||||
=====
|
||||
//end::readonly-repo-setting[]
|
||||
|
|
|
|||
|
|
@ -0,0 +1,78 @@
|
|||
[[snapshots-source-only-repository]]
|
||||
=== Source-only repository
|
||||
|
||||
You can use a source-only repository to take minimal, source-only snapshots that
|
||||
use up to 50% less disk space than regular snapshots.
|
||||
|
||||
Unlike other repository types, a source-only repository doesn't directly store
|
||||
snapshots. It delegates storage to another registered snapshot repository.
|
||||
|
||||
When you take a snapshot using a source-only repository, {es} creates a
|
||||
source-only snapshot in the delegated storage repository. This snapshot only
|
||||
contains stored fields and metadata. It doesn't include index or doc values
|
||||
structures and isn't immediately searchable when restored. To search the
|
||||
restored data, you first have to <<docs-reindex,reindex>> it into a new data
|
||||
stream or index.
|
||||
|
||||
[IMPORTANT]
|
||||
==================================================
|
||||
|
||||
Source-only snapshots are only supported if the `_source` field is enabled and no source-filtering is applied.
|
||||
When you restore a source-only snapshot:
|
||||
|
||||
* The restored index is read-only and can only serve `match_all` search or scroll requests to enable reindexing.
|
||||
|
||||
* Queries other than `match_all` and `_get` requests are not supported.
|
||||
|
||||
* The mapping of the restored index is empty, but the original mapping is available from the types top
|
||||
level `meta` element.
|
||||
|
||||
==================================================
|
||||
|
||||
Before registering a source-only repository, use {kib} or the
|
||||
<<put-snapshot-repo-api,create snapshot repository API>> to register a snapshot
|
||||
repository of another type to use for storage. Then register the source-only
|
||||
repository and specify the delegated storage repository in the request.
|
||||
|
||||
[source,console]
|
||||
----
|
||||
PUT _snapshot/my_src_only_repository
|
||||
{
|
||||
"type": "source",
|
||||
"settings": {
|
||||
"delegate_type": "fs",
|
||||
"location": "my_backup_repository"
|
||||
}
|
||||
}
|
||||
----
|
||||
// TEST[setup:setup-repository]
|
||||
// TEST[s/my_backup_repository/my_repository/]
|
||||
|
||||
[[source-only-repository-settings]]
|
||||
==== Repository settings
|
||||
|
||||
`chunk_size`::
|
||||
(Optional, <<byte-units,byte value>>)
|
||||
Maximum size of files in snapshots. In snapshots, files larger than this are
|
||||
broken down into chunks of this size or smaller. Defaults to `null` (unlimited
|
||||
file size).
|
||||
|
||||
`compress`::
|
||||
(Optional, Boolean)
|
||||
If `true`, metadata files, such as index mappings and settings, are compressed
|
||||
in snapshots. Data files are not compressed. Defaults to `true`.
|
||||
|
||||
`delegate_type`::
|
||||
(Optional, string)
|
||||
Delegated repository type. For valid values, see the
|
||||
<<put-snapshot-repo-api-request-type,`type` parameter>>.
|
||||
+
|
||||
`source` repositories can use `settings` properties for its delegated repository
|
||||
type. See <<snapshots-source-only-repository>>.
|
||||
|
||||
`max_number_of_snapshots`::
|
||||
(Optional, integer)
|
||||
Maximum number of snapshots the repository can contain.
|
||||
Defaults to `Integer.MAX_VALUE`, which is `2^31-1` or `2147483647`.
|
||||
|
||||
include::repository-shared-settings.asciidoc[]
|
||||
Loading…
Add table
Add a link
Reference in a new issue