Allows searching on boolean fields when those fields are not indexed (index: false) but just doc values are enabled.
This enables searches on archive data, which has access to doc values but not index structures. When combined with
searchable snapshots, it allows downloading only data for a given (doc value) field to quickly filter down to a select set
of documents.
Relates #81210 and #52728
Allows searching on keyword fields when those fields are not indexed (index: false) but just doc values are enabled.
This enables searches on archive data, which has access to doc values but not index structures. When combined with
searchable snapshots, it allows downloading only data for a given (doc value) field to quickly filter down to a select set
of documents.
Relates #81210 and #52728
* Upgrade documentation for xpack.monitoring.history.duration
* A few updates now that https://github.com/elastic/elasticsearch/pull/82498 has been opened with a concrete policy.
* Grammar fix
* Update docs/reference/migration/migrate_8_0/cluster-node-setting-changes.asciidoc
Co-authored-by: James Baiera <james.baiera@gmail.com>
Co-authored-by: James Baiera <james.baiera@gmail.com>
* Add support for HTTP Proxies for the GCS repository
The change adds 3 new client properties for the GCS repository:
* gcs.client.default.proxy.type
* gcs.client.default.proxy.host
* gcs.client.default.proxy.port
They allow to configure a [java.net.Proxy](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/net/Proxy.html)
for the GCS SDK to use when communicating with the GCS API.
Resolves#82444
* [DOCS] Remove extraneous Elasticsearch Docker image information
In the step for starting Elasticsearch with the generated enrollment token, `docker.elastic.co/elasticsearch/elasticsearch:` was included in conjunction with the `{docker-image}` variable. This include led to a duplicate Docker image that displayed as `ocker.elastic.co/elasticsearch/elasticsearch:docker.elastic.co/elasticsearch/elasticsearch:8.0.0-rc1`. This PR removes the duplicate image information.
* Update ifeval statements and add sub-heading for setting JVM heap size
There have been many requests to support repository-s3 authentication via IAM roles in Kubernetes service accounts.
The AWS SDK is supposed to support them out of the box with the aws-java-sdk-sts library. Unfortunately, we can't use WebIdentityTokenCredentialsProvider from the SDK. It reads the token from AWS_WEB_IDENTITY_TOKEN_FILE environment variable which is usually mounted to /var/run/secrets/eks.amazonaws.com/serviceaccount/token and the S3 repository doesn't have the read permission to read it. We don't want to hard-code a file permission for the repository, because the location of AWS_WEB_IDENTITY_TOKEN_FILE can change at any time in the future and we would also generally prefer to restrict the ability of plugins to access things outside of their config directory.
To overcome this limitation, this change adds a custom WebIdentityCredentials provider that reads the service account from a symlink to AWS_WEB_IDENTITY_TOKEN_FILE created in the repository's config directory. We expect the end user to create the symlink to indicate that they want to use service accounts for authentification.
Service accounts are checked and exchanged for session tokens by the AWS STS. To test the authentification flow, this change adds a test fixture which mocks the assume-role-with-web-identity call to the service and returns a response with test credentials.
Fixes#52625
Documents the `EMPTY` and `NONE` `flag` values for the `regexp` query.
Also documents the `""` (empty string) value, which is an alias for `ALL`.
Closes#81978.
Removing the mock nio transport and replacing its usage with the netty transport to make tests
with a more realistic transport implementation. This way improves the real world coverage for
the Netty transport, makes our tests more realistic and saves lots of code.
In particular, coverage on the rather complicated throttling/chunking in the netty message handler
is really ice to have.
The downside of this change is that we lose the slow transport thread warnings that the mock transport
outputs. This isn't a big deal these days in my opinion as we have slow logging in other places
now that makes up for this (we didn't when initially adding the slow logging) and that contains
far more detailed information on what exactly was slow.
Other than that, the mock transport does not come with any features we don't also have in the Netty
transport at this point.
Similar to #82409, but for date fields.
Allows searching on date field types (date, date_nanos) when those fields are not indexed (index: false) but just doc
values are enabled.
This enables searches on archive data, which has access to doc values but not index structures. When combined with
searchable snapshots, it allows downloading only data for a given (doc value) field to quickly filter down to a select set
of documents.
Relates #81210 and #52728
The simulate template api should include the settings that registered IndexSettingProvider generate.
Currently, these settings are not included in the simulate template api response,
but are only used to create a dummy IndexService instance to validate aliases.
This commit adds `search_interval` to the datafeed stats API
`running_state` object. When the datafeed is running, it reports
the last search interval that was searched. It is useful to
understand the point in time where the datafeed is currently
searching.
Closes#82405
### Changes
* Updates the snapshot version compatibility table to use minor versions rather than major versions.
* Adds a index creation version and cluster compatibility table. Updates the index compatibility section to use minor versions.
* Moves the tables to separate files. This'll help prevent merge conflicts.
* Fixes the heading level for the "Warnings" section.
Allows searching on number field types (long, short, int, float, double, byte, half_float) when those fields are not
indexed (index: false) but just doc values are enabled.
This enables searches on archive data, which has access to doc values but not index structures. When combined with
searchable snapshots, it allows downloading only data for a given (doc value) field to quickly filter down to a select set
of documents.
Note to reviewers:
I have split isSearchable into two separate methods isIndexed and isSearchable on MappedFieldType. The former one is
about whether actual indexing data structures have been used (postings or points), and the latter one on whether you
can run queries on the given field (e.g. used by field caps). For number field types, queries are now allowed whenever
points are available or when doc values are available (i.e. searchability is expanded).
Relates #81210 and #52728
Changes:
* Moves the get SLM policy API example _after_ the get SLM stats API. This seems to fit the normal workflow better where a user will drill-down into a particular policy to get more information.
* Notes some more information about what the get SLM policy API returns. In particular, it notes that you can get the error message for the last policy failure.
Replaces a 7.x reference with the `{prev-major-last}` variable, which will use the last minor of the previous major (currently 7.17 for 8.x).
Relates to #70451.
Changes:
* Updates the Cloud setup instructions to note that you open Kibana by default.
* Reorders the API call section to highlight Kibana.
* Fixes the Docker `ifeval` to hide some text on unreleased branches.
This enhances the migrate to data tiers routing API to also iterate over
the existing legacy, composable, and component templates and look if
they define a custom node attribute routing in their settings for either
`index.routing.allocation.require.{nodeAttrName}` or
`index.routing.allocation.include.{nodeAttrName}`. If any does, we
update them to remove all the routings settings for the provided
`nodeAttrName`.
eg. any template with the following setting configuration:
```
"settings": {
index.routing.allocation.require.data: "warm",
index.routing.allocation.include.data: "rack1",
index.routing.allocation.exclude.data: "rack2,rack3"
}
```
will have its settings updated to:
```
"settings": {}
```
Adds two known issues to the 8.0.0-rc1 release notes:
* A general advisory not to upgrade production clusters to 8.0.0-rc1. The advisory also notes that upgrades from 8.0.0-rc1 are not supported.
* A note indicating that the SQL JDBC driver requires Java 17 or newer. This requirement will be changed to Java 1.8
or newer in 8.0.0-rc2.
(cherry picked from commit 31cba4fa16)
Cosine similarity is not defined when one of the vectors has zero magnitude.
Before, the kNN search endpoint threw a confusing exception related to top docs
collection. Now we reject vectors early with a clear error message, failing
indexing if the vector has zero magnitude.
As of 8.0, the compatibility window for cross-cluster search (CCS) to an earlier release will be one minor release. This updates the CCS docs and adds a related 8.0 breaking change.
Closes https://github.com/elastic/elasticsearch/issues/80782
Emit deprecation warning when creating new jobs with bucket spans that
aren't an integral divisor or multiple of a day.
Relates #81645
Co-authored-by: lcawl <lcawley@elastic.co>
* Adds a prerequisites section covering remote cluster config, node roles, and security.
* Moves existing content about remote cluster config to the prereqs.
* Updates the remote cluster docs to include information about eligible gateway nodes and tagging for gateway nodes.
Closes https://github.com/elastic/elasticsearch/issues/72001
Closes#81652.
Convert the `repository-azure`, `repository-gcs` and `repository-s3`
plugins into modules, so that they are always included in the
Elasticsearch distribution. Also change plugin installation, removal
and syncing so that attempting to add or remove these plugins still
succeeds but is now a no-op.
