From cb6265f9bd72d743967effc58e26a7b95bb0610f Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Wed, 26 Jan 2022 17:13:39 -0500 Subject: [PATCH] [DOCS] Move snapshot repo types to separate pages (#82826) With https://github.com/elastic/elasticsearch/pull/81870, the Azure, GCS, and S3 repository types have separate, dedicated pages in the Elasticsearch guide. For consistency, this PR creates separate pages for the shared file system, read-only URL, and source-only repository types. Related changes: - Adds redirects to the plugins docs - Fixes a few breaking changes that refer to the Azure, GCS, and S3 repositories as plugins. Co-authored-by: Adam Locke --- docs/plugins/redirects.asciidoc | 74 +++++++ .../cluster-node-setting-changes.asciidoc | 9 +- .../migrate_8_0/rest-api-changes.asciidoc | 9 +- .../apis/put-repo-api.asciidoc | 183 ++---------------- .../on-prem-repo-type.asciidoc | 2 + .../register-repository.asciidoc | 142 +++----------- .../repository-read-only-url.asciidoc | 76 ++++++++ .../repository-shared-file-system.asciidoc | 49 +++++ .../repository-shared-settings.asciidoc | 32 ++- .../repository-source-only.asciidoc | 78 ++++++++ 10 files changed, 358 insertions(+), 296 deletions(-) create mode 100644 docs/reference/snapshot-restore/on-prem-repo-type.asciidoc create mode 100644 docs/reference/snapshot-restore/repository-read-only-url.asciidoc create mode 100644 docs/reference/snapshot-restore/repository-shared-file-system.asciidoc create mode 100644 docs/reference/snapshot-restore/repository-source-only.asciidoc diff --git a/docs/plugins/redirects.asciidoc b/docs/plugins/redirects.asciidoc index c691ac488558..96b432e421c6 100644 --- a/docs/plugins/redirects.asciidoc +++ b/docs/plugins/redirects.asciidoc @@ -71,3 +71,77 @@ See {ref}/monitor-elasticsearch-cluster.html[{stack} monitoring]. === Security plugins See {ref}/secure-cluster.html[{stack} security]. + +[role="exclude",id="repository-azure"] +=== Azure repository plugin + +// tag::azure-repo-migration[] +The Azure repository plugin is now included in {es}. +See {ref}/repository-azure.html[Azure repository]. +// end::azure-repo-migration[] + +[role="exclude",id="repository-azure-usage"] +=== Azure repository plugin + +include::redirects.asciidoc[tag=azure-repo-migration] + +[role="exclude",id="repository-azure-client-settings"] +=== Azure repository plugin + +include::redirects.asciidoc[tag=azure-repo-migration] + +[role="exclude",id="repository-azure-repository-settings"] +=== Azure repository plugin + +include::redirects.asciidoc[tag=azure-repo-migration] + +[role="exclude",id="repository-azure-validation"] +=== Azure repository plugin + +include::redirects.asciidoc[tag=azure-repo-migration] + +[role="exclude",id="repository-s3"] +=== S3 repository plugin + +// tag::s3-repo-migration[] +The S3 repository plugin is now included in {es}. +See {ref}/repository-s3.html[S3 repository]. +// end::s3-repo-migration[] + +[role="exclude",id="repository-s3-usage"] +=== S3 repository plugin + +include::redirects.asciidoc[tag=s3-repo-migration] + +[role="exclude",id="repository-s3-client"] +=== S3 repository plugin + +include::redirects.asciidoc[tag=s3-repo-migration] + +[role="exclude",id="repository-s3-repository"] +=== S3 repository plugin + +include::redirects.asciidoc[tag=s3-repo-migration] + +[role="exclude",id="repository-gcs"] +=== Google Cloud Storage repository plugin + +// tag::gcs-repo-migration[] +The Google Cloud Storage repository plugin is now included in {es}. +See {ref}/repository-gcs.html[Google Cloud Storage repository]. +// end::gcs-repo-migration[] + +[role="exclude",id="repository-gcs-usage"] +=== Google Cloud Storage repository plugin + +include::redirects.asciidoc[tag=gcs-repo-migration] + +[role="exclude",id="repository-gcs-client"] +=== Google Cloud Storage repository plugin + +include::redirects.asciidoc[tag=gcs-repo-migration] + +[role="exclude",id="repository-gcs-repository"] +=== Google Cloud Storage repository plugin + +include::redirects.asciidoc[tag=gcs-repo-migration] diff --git a/docs/reference/migration/migrate_8_0/cluster-node-setting-changes.asciidoc b/docs/reference/migration/migrate_8_0/cluster-node-setting-changes.asciidoc index 1889290278f1..6fd3fc57171d 100644 --- a/docs/reference/migration/migrate_8_0/cluster-node-setting-changes.asciidoc +++ b/docs/reference/migration/migrate_8_0/cluster-node-setting-changes.asciidoc @@ -875,11 +875,10 @@ previously be used to enable compression for all shared file system repositories The `repositories.fs.compress` setting has been removed. *Impact* + -Use the repository specific `compress` setting to enable compression. See -{ref}/snapshots-register-repository.html[Register a snapshot repository] for -information on the `compress` setting. - -Discontinue use of the `repositories.fs.compress` node-level setting. +Discontinue use of the `repositories.fs.compress` node-level setting. Use the +repository-specific `compress` setting to enable compression instead. Refer to +{ref}/snapshots-filesystem-repository.html#filesystem-repository-settings[Shared +file system repository settings]. ==== //end::notable-breaking-changes[] diff --git a/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc b/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc index 988521ca4ee7..11eefa04d342 100644 --- a/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc +++ b/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc @@ -991,21 +991,18 @@ Previously, the default value for `compress` was `false`. The default has been c This change will affect both newly created repositories and existing repositories where `compress=false` has not been explicitly specified. -For more information on the compress option, see -{ref}/snapshots-register-repository.html[Register a snapshot repository]. - *Impact* + Update your workflow and applications to assume a default value of `true` for the `compress` parameter. ==== -.The S3 repository plugin now uses a DNS-style access pattern by default. +.S3 snapshot repositories now use a DNS-style access pattern by default. [%collapsible] ==== *Details* + -Starting in version 7.4 the `repository-s3` plugin does not use the +Starting in version 7.4, `s3` snapshot repositories no longer use the now-deprecated path-style access pattern by default. In versions 7.0, 7.1, 7.2 -and 7.3 the `repository-s3` plugin always used the path-style access pattern. +and 7.3 `s3` snapshot repositories always used the path-style access pattern. This is a breaking change for deployments that only support path-style access but which are recognized as supporting DNS-style access by the AWS SDK. This breaking change was made necessary by diff --git a/docs/reference/snapshot-restore/apis/put-repo-api.asciidoc b/docs/reference/snapshot-restore/apis/put-repo-api.asciidoc index b18a045d34c8..0b77795540a1 100644 --- a/docs/reference/snapshot-restore/apis/put-repo-api.asciidoc +++ b/docs/reference/snapshot-restore/apis/put-repo-api.asciidoc @@ -75,183 +75,42 @@ You can manually perform this verification using the [[put-snapshot-repo-api-request-type]] `type`:: -+ --- (Required, string) Repository type. - ++ .Valid values for `type` [%collapsible%open] ==== -`fs`:: -Shared file system repository. Repositories of this type use a shared file -system to store snapshots. This file system must accessible to all master and -data nodes in the cluster. -+ -IMPORTANT: To register a shared file system repository, you must mount the same -shared filesystem to the same location on all master and data nodes. This -location must be registered in the `path.repo` setting on all master and data -nodes in the cluster. -+ -See <>. -[xpack]#`source`#:: -Source-only repository. You can use source-only repositories to create minimal, -source-only snapshots that take up to 50% less space on disk. -+ -Source-only snapshots are only supported if the <> is enabled and no -<> is applied. -+ -WARNING: Source-only snapshots contain stored fields and index metadata. They do -not include index or doc values structures and are not searchable when restored. -After restoring a source-only snapshot, you must <> the -data into a new index. -+ -See <>. +`azure`:: <> +`gcs`:: <> +`s3`:: <> +`fs`:: <> +`source`:: <> +`url`:: <> -`url`:: -URL repository. Repositories of this type are read-only -for the cluster. This means the cluster can retrieve or restore snapshots from -the repository but cannot write or create snapshots in it. -+ -You can use URL repositories as an alternative way to give a cluster read-only -access to a shared file system (`fs`) repository. -+ -See <>. +Other repository types are available through official plugins: + +`hfds`:: {plugins}/repository-hdfs.html[Hadoop Distributed File System (HDFS) repository] ==== -More repository types are available through these official -plugins: - -* <> for S3 repository support -* {plugins}/repository-hdfs.html[repository-hdfs] for HDFS repository support in - Hadoop environments -* <> for Azure storage repositories -* <> for Google Cloud Storage repositories --- - [[put-snapshot-repo-api-settings-param]] `settings`:: +(Required, object) +Settings for the repository. Supported settings vary based on the repository +type: + -- -(Required, object) -Contains settings for the repository. +* <> +* <> +* <> +* <> +* <> +* <> -The following `settings` properties are valid for all repository types: +Other repository types are available through official plugins: -.Properties of `settings` -[%collapsible%open] -==== -`chunk_size`:: -(Optional, <>) -Maximum size of files in snapshots. In snapshots, files larger than this are -broken down into chunks of this size or smaller. Defaults to `null` (unlimited -file size). - -`compress`:: -(Optional, Boolean) -If `true`, metadata files, such as index mappings and settings, are compressed -in snapshots. Data files are not compressed. Defaults to `true`. - -`max_number_of_snapshots`:: -(Optional, integer) -Maximum number of snapshots the repository can contain. -Defaults to `Integer.MAX_VALUE`, which is `2^31-1` or `2147483647`. - -`max_restore_bytes_per_sec`:: -(Optional, <>) -Maximum snapshot restore rate per node. Defaults to unlimited. Note -that restores are also throttled through <>. - -`max_snapshot_bytes_per_sec`:: -(Optional, <>) -Maximum snapshot creation rate per node. Defaults to `40mb` per second. - -`readonly`:: -(Optional, Boolean) -If `true`, the repository is read-only. The cluster can retrieve and restore -snapshots from the repository but not write to the repository or create -snapshots in it. -+ -If `false`, the cluster can write to the repository and create snapshots in it. -Defaults to `false`. -+ -[TIP] -===== -If you register the same snapshot repository with multiple clusters, only -one cluster should have write access to the repository. Having multiple clusters -write to the repository at the same time risks corrupting the contents of the -repository. - -Only a cluster with write access can create snapshots in the repository. All -other clusters connected to the repository should have the `readonly` parameter -set to `true`. This means those clusters can retrieve or restore snapshots from -the repository but not create snapshots in it. -===== -==== - -Other accepted `settings` properties depend on the repository type, set using the -<> parameter. - -.Valid `settings` properties for `fs` repositories -[%collapsible%open] -==== -`location`:: -(Required, string) -Location of the shared filesystem used to store and retrieve snapshots. This -location must be registered in the `path.repo` setting on all master and data -nodes in the cluster. -==== - -.Valid `settings` properties for `source` repositories -[%collapsible%open] -==== -`delegate_type`:: -(Optional, string) -Delegated repository type. For valid values, see the -<>. -+ -`source` repositories can use `settings` properties for its delegated repository -type. See <>. - -==== - -.Valid `settings` properties for `url` repositories -[%collapsible%open] -==== -`url`:: -+ ---- -(Required, string) -URL location of the root of the shared filesystem repository. The following -protocols are supported: - -* `file` -* `ftp` -* `http` -* `https` -* `jar` - -URLs using the `http`, `https`, or `ftp` protocols must be explicitly allowed -with the <> cluster -setting. This setting supports wildcards in the place of a host, path, query, or -fragment in the URL. - -URLs using the `file` protocol must point to the location of a shared filesystem -accessible to all master and data nodes in the cluster. This location must be -registered in the `path.repo` setting. You don't need to register URLs using the -`ftp`, `http`, `https`, or `jar` protocols in the `path.repo` setting. ---- - -`http_max_retries`:: -(Optional, integer) Maximum number of retries for `http` and `https` URLs. -Defaults to `5`. - -`http_socket_timeout`:: -(Optional, <>) Maximum wait time for data transfers over -a connection. Defaults to `50s`. -==== +* {plugins}/repository-hdfs.html[Hadoop Distributed File System (HDFS) repository] -- `verify`:: diff --git a/docs/reference/snapshot-restore/on-prem-repo-type.asciidoc b/docs/reference/snapshot-restore/on-prem-repo-type.asciidoc new file mode 100644 index 000000000000..00859394abb9 --- /dev/null +++ b/docs/reference/snapshot-restore/on-prem-repo-type.asciidoc @@ -0,0 +1,2 @@ +NOTE: This repository type is only available if you run {es} on your own +hardware. If you use {ess}, see <>. \ No newline at end of file diff --git a/docs/reference/snapshot-restore/register-repository.asciidoc b/docs/reference/snapshot-restore/register-repository.asciidoc index 05883837744d..c169454f0794 100644 --- a/docs/reference/snapshot-restore/register-repository.asciidoc +++ b/docs/reference/snapshot-restore/register-repository.asciidoc @@ -70,11 +70,17 @@ To manage repositories in {kib}, go to the main menu and click **Stack Management** > **Snapshot and Restore** > **Repositories**. To register a snapshot repository, click **Register repository**. +You can also register a repository using the <>. + [discrete] [[snapshot-repo-types]] === Snapshot repository types -Supported snapshot repository types vary based on your deployment type. +Supported snapshot repository types vary based on your deployment type: + +* <> +* <> [discrete] [[ess-repo-types]] @@ -94,10 +100,10 @@ clusters]. {ess} deployments also support the following repository types: +* {cloud}/ec-azure-snapshotting.html[Azure] +* {cloud}/ec-gcs-snapshotting.html[Google Cloud Storage] * {cloud}/ec-aws-custom-repository.html[AWS S3] -* {cloud}/ec-gcs-snapshotting.html[Google Cloud Storage (GCS)] -* {cloud}/ec-azure-snapshotting.html[Microsoft Azure] -* <> +* <> [discrete] [[self-managed-repo-types]] @@ -106,12 +112,12 @@ clusters]. If you run {es} on your own hardware, you can use the following built-in snapshot repository types: -* <> -* <> * <> +* <> +* <> * <> -* <> -* <> +* <> +* <> [[snapshots-repository-plugins]] Other repository types are available through official plugins: @@ -122,108 +128,6 @@ You can also use alternative implementations of these repository types, such as MinIO, as long as they're compatible. To verify a repository's compatibility, see <>. -[discrete] -[[snapshots-filesystem-repository]] -==== Shared file system repository - -// tag::on-prem-repo-type[] -NOTE: This repository type is only available if you run {es} on your own -hardware. If you use {ess}, see <>. -// end::on-prem-repo-type[] - -Use a shared file system repository to store snapshots on a -shared file system. - -To register a shared file system repository, first mount the file system to the -same location on all master and data nodes. Then add the file system's -path or parent directory to the `path.repo` setting in `elasticsearch.yml` for -each master and data node. For running clusters, this requires a -<> of each node. - -IMPORTANT: By default, a network file system (NFS) uses user IDs (UIDs) and -group IDs (GIDs) to match accounts across nodes. If your shared file system is -an NFS and your nodes don't use the same UIDs and GIDs, update your NFS -configuration to account for this. - -Supported `path.repo` values vary by platform: - -include::{es-repo-dir}/tab-widgets/register-fs-repo-widget.asciidoc[] - -[discrete] -[[snapshots-read-only-repository]] -==== Read-only URL repository - -include::register-repository.asciidoc[tag=on-prem-repo-type] - -You can use a URL repository to give a cluster read-only access to a shared file -system. Since URL repositories are always read-only, they're a safer and more -convenient alternative to registering a read-only shared filesystem repository. - -Use {kib} or the <> to -register a URL repository. - -[source,console] ----- -PUT _snapshot/my_read_only_url_repository -{ - "type": "url", - "settings": { - "url": "file:/mount/backups/my_fs_backup_location" - } -} ----- -// TEST[skip:no access to url file path] - -[discrete] -[[snapshots-source-only-repository]] -==== Source-only repository - -You can use a source-only repository to take minimal, source-only snapshots that -use up to 50% less disk space than regular snapshots. - -Unlike other repository types, a source-only repository doesn't directly store -snapshots. It delegates storage to another registered snapshot repository. - -When you take a snapshot using a source-only repository, {es} creates a -source-only snapshot in the delegated storage repository. This snapshot only -contains stored fields and metadata. It doesn't include index or doc values -structures and isn't immediately searchable when restored. To search the -restored data, you first have to <> it into a new data -stream or index. - -[IMPORTANT] -================================================== - -Source-only snapshots are only supported if the `_source` field is enabled and no source-filtering is applied. -When you restore a source-only snapshot: - - * The restored index is read-only and can only serve `match_all` search or scroll requests to enable reindexing. - - * Queries other than `match_all` and `_get` requests are not supported. - - * The mapping of the restored index is empty, but the original mapping is available from the types top - level `meta` element. - -================================================== - -Before registering a source-only repository, use {kib} or the -<> to register a snapshot -repository of another type to use for storage. Then register the source-only -repository and specify the delegated storage repository in the request. - -[source,console] ----- -PUT _snapshot/my_src_only_repository -{ - "type": "source", - "settings": { - "delegate_type": "fs", - "location": "my_backup_location" - } -} ----- -// TEST[continued] - [discrete] [[snapshots-repository-verification]] === Verify a repository @@ -245,7 +149,8 @@ PUT _snapshot/my_unverified_backup?verify=false } } ---- -// TEST[continued] +// TEST[setup:setup-repository] +// TEST[s/my_unverified_backup_location/my_repository/] If wanted, you can manually run the repository verification check. To verify a repository in {kib}, go to the **Repositories** list page and click the name of @@ -257,6 +162,7 @@ a repository. Then click **Verify repository**. You can also use the POST _snapshot/my_unverified_backup/_verify ---- // TEST[continued] +// TEST[s/my_unverified_backup_location/my_repository/] If successful, the request returns a list of nodes used to verify the repository. If verification fails, the request returns an error. @@ -285,7 +191,7 @@ API>>. ---- POST _snapshot/my_repository/_cleanup ---- -// TEST[continued] +// TEST[setup:setup-snapshots] The API returns: @@ -298,6 +204,8 @@ The API returns: } } ---- +// TESTRESPONSE[s/"deleted_bytes": 20/"deleted_bytes": $body.results.deleted_bytes/] +// TESTRESPONSE[s/"deleted_blobs": 5/"deleted_blobs": $body.results.deleted_bytes/] Depending on the concrete repository implementation the numbers shown for bytes free as well as the number of blobs removed will either be an approximation or an exact result. Any non-zero value for the number of blobs removed implies that unreferenced blobs were found and @@ -336,9 +244,9 @@ with {es} until the repository contents are fully restored. If you alter the contents of a repository while it is registered with {es} then the repository may become unreadable or may silently lose some of its contents. - -include::repository-s3.asciidoc[] - -include::repository-gcs.asciidoc[] - include::repository-azure.asciidoc[] +include::repository-gcs.asciidoc[] +include::repository-s3.asciidoc[] +include::repository-shared-file-system.asciidoc[] +include::repository-read-only-url.asciidoc[] +include::repository-source-only.asciidoc[] diff --git a/docs/reference/snapshot-restore/repository-read-only-url.asciidoc b/docs/reference/snapshot-restore/repository-read-only-url.asciidoc new file mode 100644 index 000000000000..8f9cb7e198f8 --- /dev/null +++ b/docs/reference/snapshot-restore/repository-read-only-url.asciidoc @@ -0,0 +1,76 @@ +[[snapshots-read-only-repository]] +=== Read-only URL repository + +include::{es-repo-dir}/snapshot-restore/on-prem-repo-type.asciidoc[] + +You can use a URL repository to give a cluster read-only access to a shared file +system. Since URL repositories are always read-only, they're a safer and more +convenient alternative to registering a read-only shared filesystem repository. + +Use {kib} or the <> to +register a URL repository. + +[source,console] +---- +PUT _snapshot/my_read_only_url_repository +{ + "type": "url", + "settings": { + "url": "file:/mount/backups/my_fs_backup_location" + } +} +---- +// TEST[skip:no access to url file path] + +[[read-only-url-repository-settings]] +==== Repository settings + +`chunk_size`:: +(Optional, <>) +Maximum size of files in snapshots. In snapshots, files larger than this are +broken down into chunks of this size or smaller. Defaults to `null` (unlimited +file size). + +`http_max_retries`:: +(Optional, integer) Maximum number of retries for `http` and `https` URLs. +Defaults to `5`. + +`http_socket_timeout`:: +(Optional, <>) Maximum wait time for data transfers over +a connection. Defaults to `50s`. + +`compress`:: +(Optional, Boolean) +If `true`, metadata files, such as index mappings and settings, are compressed +in snapshots. Data files are not compressed. Defaults to `true`. + +`max_number_of_snapshots`:: +(Optional, integer) +Maximum number of snapshots the repository can contain. +Defaults to `Integer.MAX_VALUE`, which is `2^31-1` or `2147483647`. + +include::repository-shared-settings.asciidoc[tags=!readonly-repo-setting] + +`url`:: ++ +-- +(Required, string) +URL location of the root of the shared filesystem repository. The following +protocols are supported: + +* `file` +* `ftp` +* `http` +* `https` +* `jar` + +URLs using the `http`, `https`, or `ftp` protocols must be explicitly allowed +with the <> cluster +setting. This setting supports wildcards in the place of a host, path, query, or +fragment in the URL. + +URLs using the `file` protocol must point to the location of a shared filesystem +accessible to all master and data nodes in the cluster. This location must be +registered in the `path.repo` setting. You don't need to register URLs using the +`ftp`, `http`, `https`, or `jar` protocols in the `path.repo` setting. +-- \ No newline at end of file diff --git a/docs/reference/snapshot-restore/repository-shared-file-system.asciidoc b/docs/reference/snapshot-restore/repository-shared-file-system.asciidoc new file mode 100644 index 000000000000..658dc1b5e698 --- /dev/null +++ b/docs/reference/snapshot-restore/repository-shared-file-system.asciidoc @@ -0,0 +1,49 @@ +[[snapshots-filesystem-repository]] +=== Shared file system repository + +include::{es-repo-dir}/snapshot-restore/on-prem-repo-type.asciidoc[] + +Use a shared file system repository to store snapshots on a +shared file system. + +To register a shared file system repository, first mount the file system to the +same location on all master and data nodes. Then add the file system's +path or parent directory to the `path.repo` setting in `elasticsearch.yml` for +each master and data node. For running clusters, this requires a +<> of each node. + +IMPORTANT: By default, a network file system (NFS) uses user IDs (UIDs) and +group IDs (GIDs) to match accounts across nodes. If your shared file system is +an NFS and your nodes don't use the same UIDs and GIDs, update your NFS +configuration to account for this. + +Supported `path.repo` values vary by platform: + +include::{es-repo-dir}/tab-widgets/register-fs-repo-widget.asciidoc[] + +[[filesystem-repository-settings]] +==== Repository settings + +`chunk_size`:: +(Optional, <>) +Maximum size of files in snapshots. In snapshots, files larger than this are +broken down into chunks of this size or smaller. Defaults to `null` (unlimited +file size). + +`compress`:: +(Optional, Boolean) +If `true`, metadata files, such as index mappings and settings, are compressed +in snapshots. Data files are not compressed. Defaults to `true`. + +`location`:: +(Required, string) +Location of the shared filesystem used to store and retrieve snapshots. This +location must be registered in the `path.repo` setting on all master and data +nodes in the cluster. + +`max_number_of_snapshots`:: +(Optional, integer) +Maximum number of snapshots the repository can contain. +Defaults to `Integer.MAX_VALUE`, which is `2^31-1` or `2147483647`. + +include::repository-shared-settings.asciidoc[] diff --git a/docs/reference/snapshot-restore/repository-shared-settings.asciidoc b/docs/reference/snapshot-restore/repository-shared-settings.asciidoc index 2a4753abee45..e2f79bb11cb2 100644 --- a/docs/reference/snapshot-restore/repository-shared-settings.asciidoc +++ b/docs/reference/snapshot-restore/repository-shared-settings.asciidoc @@ -1,12 +1,32 @@ `max_restore_bytes_per_sec`:: - - Throttles per node restore rate. Defaults to unlimited. - Note that restores are also throttled through {ref}/recovery.html[recovery settings]. +(Optional, <>) +Maximum snapshot restore rate per node. Defaults to unlimited. Note +that restores are also throttled through <>. `max_snapshot_bytes_per_sec`:: +(Optional, <>) +Maximum snapshot creation rate per node. Defaults to `40mb` per second. - Throttles per node snapshot rate. Defaults to `40mb` per second. - +//tag::readonly-repo-setting[] `readonly`:: +(Optional, Boolean) +If `true`, the repository is read-only. The cluster can retrieve and restore +snapshots from the repository but not write to the repository or create +snapshots in it. ++ +Only a cluster with write access can create snapshots in the repository. All +other clusters connected to the repository should have the `readonly` parameter +set to `true`. ++ +If `false`, the cluster can write to the repository and create snapshots in it. +Defaults to `false`. ++ +[IMPORTANT] +===== +If you register the same snapshot repository with multiple clusters, only +one cluster should have write access to the repository. Having multiple clusters +write to the repository at the same time risks corrupting the contents of the +repository. - Makes repository read-only. Defaults to `false`. +===== +//end::readonly-repo-setting[] diff --git a/docs/reference/snapshot-restore/repository-source-only.asciidoc b/docs/reference/snapshot-restore/repository-source-only.asciidoc new file mode 100644 index 000000000000..07ddedd19793 --- /dev/null +++ b/docs/reference/snapshot-restore/repository-source-only.asciidoc @@ -0,0 +1,78 @@ +[[snapshots-source-only-repository]] +=== Source-only repository + +You can use a source-only repository to take minimal, source-only snapshots that +use up to 50% less disk space than regular snapshots. + +Unlike other repository types, a source-only repository doesn't directly store +snapshots. It delegates storage to another registered snapshot repository. + +When you take a snapshot using a source-only repository, {es} creates a +source-only snapshot in the delegated storage repository. This snapshot only +contains stored fields and metadata. It doesn't include index or doc values +structures and isn't immediately searchable when restored. To search the +restored data, you first have to <> it into a new data +stream or index. + +[IMPORTANT] +================================================== + +Source-only snapshots are only supported if the `_source` field is enabled and no source-filtering is applied. +When you restore a source-only snapshot: + + * The restored index is read-only and can only serve `match_all` search or scroll requests to enable reindexing. + + * Queries other than `match_all` and `_get` requests are not supported. + + * The mapping of the restored index is empty, but the original mapping is available from the types top + level `meta` element. + +================================================== + +Before registering a source-only repository, use {kib} or the +<> to register a snapshot +repository of another type to use for storage. Then register the source-only +repository and specify the delegated storage repository in the request. + +[source,console] +---- +PUT _snapshot/my_src_only_repository +{ + "type": "source", + "settings": { + "delegate_type": "fs", + "location": "my_backup_repository" + } +} +---- +// TEST[setup:setup-repository] +// TEST[s/my_backup_repository/my_repository/] + +[[source-only-repository-settings]] +==== Repository settings + +`chunk_size`:: +(Optional, <>) +Maximum size of files in snapshots. In snapshots, files larger than this are +broken down into chunks of this size or smaller. Defaults to `null` (unlimited +file size). + +`compress`:: +(Optional, Boolean) +If `true`, metadata files, such as index mappings and settings, are compressed +in snapshots. Data files are not compressed. Defaults to `true`. + +`delegate_type`:: +(Optional, string) +Delegated repository type. For valid values, see the +<>. ++ +`source` repositories can use `settings` properties for its delegated repository +type. See <>. + +`max_number_of_snapshots`:: +(Optional, integer) +Maximum number of snapshots the repository can contain. +Defaults to `Integer.MAX_VALUE`, which is `2^31-1` or `2147483647`. + +include::repository-shared-settings.asciidoc[]