This commit adds docs for monitor_stats and updates an example snippet to include both remote_indices and remote_cluster.
(cherry picked from commit b7167b73e3)
* Add a monitor_stats privilege and allow that privilege for remote cluster privileges (#114964)
This commit does the following:
* Add a new monitor_stats privilege
* Ensure that monitor_stats can be set in the remote_cluster privileges
* Give's Kibana the ability to remotely call monitor_stats via RCS 2.0
Since this is the first case where there is more than 1 remote_cluster privilege,
the following framework concern has been added:
* Ensure that when sending to elder RCS 2.0 clusters that we don't send the new privilege
previous only supported all or nothing remote_cluster blocks
* Ensure that we when sending API key role descriptors that contains remote_cluster,
we don't send the new privileges for RCS 1.0/2.0 if it not new enough
* Fix and extend the BWC tests for RCS 1.0 and RCS 2.0
(cherry picked from commit af99654dac)
* adjust bwc for 8.x branch
Documentation for the remote_cluster in the role was added
in #111682 and #108840, but a few places were missed.
This commit fill the gaps in the documentation.
This PR adds telemetry for logsdb. However, this change only tracks the
count of indices using logsdb and those that use synthetic source.
Additional stats, such as shard, indexing, and search stats, will be
added in a follow-up, as they require reaching out to data nodes.
This removes the undocumented `META FUNCTIONS` command that emits
descriptions for all functions. This shouldn't be used because we never
told anyone about it. I'd have preferred if we'd have explicitly
documented it as no public or if we'd have left it snapshot-only. But
sometimes you make a mistake. I'm hopeful no one is relying on it. It
was never reliable and not public.....
Today the docs for the `?wait_for_active_shards` parameter say that it
must be a positive integer, proscribing `0`, yet `0` is a legitimate
value for this parameter. This commit fixes this point and rewords the
docs slightly for clarity.
Adds deployment threading options and a new memory section reporting
the memory usage for each of the ml features
# Conflicts:
# server/src/main/java/org/elasticsearch/TransportVersions.java
Currently, the data stream lifecycle telemetry has the following
structure:
```
{
....
"data_lifecycle" : {
"available": true,
"enabled": true,
"count": 0,
"default_rollover_used": true,
"retention": {
"minimum_millis": 0,
"maximum_millis": 0,
"average_millis": 0.0
}
}....
```
In the snippet above you can see that we track:
- The amount of data streams managed by the data stream lifecycle by `count`
- If the default rollover has been overwritten by `default_rollover_used`
- The min, max and average of the `data_retention` configured on a data stream level.
In this PR we propose the following extention:
```
....
"data_lifecycle" : {
"available": true,
"enabled": true,
"count": 0,
"default_rollover_used": true,
"effective_retention": { #https://github.com/elastic/dev/issues/2537
"retained_data_streams": 5,
"minimum_millis": 0, # Only if retained data streams > 1
"maximum_millis": 0,
"average_millis": 0.0
},
"data_retention": {
"configured_data_streams": 5,
"minimum_millis": 0, # Only if retained data streams > 1
"maximum_millis": 0,
"average_millis": 0.0
},
"global_retention": {
"default": {
"defined": true/false,
"affected_data_streams": 0,
"millis": 0
},
"max": {
"defined": true/false,
"affected_data_streams": 0,
"millis": 0
}
}
```
With this extension we are tracking:
- The amount of data streams managed by the data stream lifecycle by `count`
- If the default rollover has been overwritten by `default_rollover_used`
- The min, max and average of the `data_retention` configured on a data stream level and the number of data streams that have it configured. We add the min, max and avg only if there are data streams with data retention configuration to avoid messing with the stats in a dashboard.
- The min, max and average of the `effective_retention` and the number of data streams that are retained. We add the min, max and avg only if there are retained data streams to avoid messing with the stats in a dashboard.
- Global retention stats, if they are defined, if the number of the affected data streams and the actual value.
The above metrics allow us to answer questions like:
- How many data streams are affected by global retention.
- How big is the difference between the longest data retention compared to max global retention.
- How much does the effective retention diverging from the data retention, this will show the impact of the global retention.
In this PR we remove unused code including relating to the global data retention with APIs implementation:
- The transport action for updating, deleting and retrieving the global retention.
- The `DataStreamGlobalRetention` from the cluster state (this should be bwc safe because we never exposed the APIs to add a data stream lifecycle to the cluster state).
- Make unused privileges monitor and managing global retention a noop.
- Remove cluster state update tasks.
The kept `DataStreamGlobalRetentionResolver` considering it could hold and provide the global retention from the settings when we implement it. We just renamed it to DataStreamGlobalRetentionProvider for now to better match what it does.
The factory retention settings should still work after this change.
* Create manage_seaech_connector privilege
* `manage_search_connector` -> `manage_connector` and exclude connector secrets patterns from this privilege
* Add `monitor_connector` privilege
* Update Kibana system privilege to monitor_connector for telemetry
* Rename privilege to 'manage_connector_state'
Since privilege names are often namespaced and used with globs, we want to ensure that if there's a future privilege like `manage_connector_secrets`, that it is not implicitly included in this new privileg's <name>*. By extending the privilege name to include "_state", we better namespace this distinct from any "_secrets" namespace.
* Revert "Rename privilege to 'manage_connector_state'"
This reverts commit 70b89eee76.
After further discussion with the security team, this name change is not needed after all
since the secret management privileges aren't currently prefixed with "manage_"
---------
Co-authored-by: Sean Story <sean.j.story@gmail.com>
Docs tweak with a typo fix and a clarification on how the two available
settings interact (essentially
https://github.com/elastic/elasticsearch/issues/27871). I'm also open to
including this info in the more generic settings API but feels like a
simple enough callout to add to the security API.
Introduce an optional k param for knn query
If k is not set, knn query has the previous behaviour:
- `num_candidates` docs is collected from each shard. This `num_candidates` docs
are used for combining with results with other queries and aggregations on each shard.
- docs from all shards are merged to produce the top global `size` results
If k is set, the behaviour instead is following:
- `k` docs is collected from each shard. This `k` docs are used for
combining results with other queries and aggregations on each shard.
- similarly, docs from all shards are merged to produce the top global `size`
results.
Having `k` param makes it more intuitive for users to address their needs.
They also don't need to care and can skip `num_candidates` param for this query
as it is of more internal details to tune how knn search operates.
Closes#108473
The cluster level dense vector stats returns the total number of dense vector indices globally including the replicas.
This commit fixes the total to only include the value count of the primary indices.
This change aligns with the docs stats which also reports the number of primary documents when used in cluster stats.
The indices stats API still reports granular results for replicas and primaries so the information is not lost.
* Add SparseVectorStats
* Update to use mappings in engine
* Update to be unique to primary shards
* Fix doc
* Fix null error in test
* Cleanup
* fix yaml
* remove comment
* add version to yaml
* Revert whitespace changes to stats doc
* fix yml test
* Checkstyle
* Fix NPE in test
* Update docs/changelog/108793.yaml
* Add link to sparse_vector field type in docs
* PR feedback
* Flesh out test a bit more
* PR feedback - alphabetize placement in docs
* Fix doc change
This is a refactoring of the internal logic that's used to translate
query-level into index-level field names for query APIs for
security entities (i.e. users, API Keys, and soon, roles).
The objective here is to have and reuse a single class to handle
all the translations for different security query APIs.
This commit introduces the documentation for remote_clusters which is used to help
express the monitor_enrich privilege needed to use the ENRICH keyword across clusters
when using the API key based CCS security model.
This commit also adds "remote_clusters" to the built in privs API to for easier consumption
in Kibana.
This commit adds documentation for the DLS/FLS restriction for RCS 2.0 API keys
where both access and replication are defined and access has DSL/FLS.
This commit also fixes a few misleading variable names.
related: #108600
This change adds `number_of_rollup_jobs` and `number_of_rollup_indices`
to `rollup` usage. The former indicates the number of active rollup
jobs running and the latter indicated the number of rollup indices
(which could be the result of previous rollup jobs).
APIs which perform cluster state updates typically accept the
`?master_timeout=` and `?timeout=` parameters to respectively set the
pending task queue timeout and the acking timeout for the cluster state
update. Both of these parameters accept the value `-1`, but
`?master_timeout=-1` means to wait indefinitely whereas `?timeout=-1`
means the same thing as `?timeout=0`, namely that acking times out
immediately on commit.
There are some situations where it makes sense to wait for as long as
possible for nodes to ack a cluster state update. In practice this wait
is bounded by other mechanisms (e.g. the lag detector will remove the
node from the cluster after a couple of minutes of failing to apply
cluster state updates) but these are not really the concern of clients.
Therefore with this commit we change the meaning of `?timeout=-1` to
mean that the acking timeout is infinite.
* Remove `es-test-dir` book-scoped variable
* Remove `plugins-examples-dir` book-scoped variable
* Remove `:dependencies-dir:` and `:xes-repo-dir:` book-scoped variables
- In `index.asciidoc`, two variables (`:dependencies-dir:` and `:xes-repo-dir:`) were removed.
- In `sql/index.asciidoc`, the `:sql-tests:` path was updated to fuller path
- In `esql/index.asciidoc`, the `:esql-tests:` path was updated idem
* Replace `es-repo-dir` with `es-ref-dir`
* Move `:include-xpack: true` to few files that use it, remove from index.asciidoc
Specifying `?master_timeout=-1` on an API which performs a cluster state
update means that the cluster state update task will never time out
while waiting in the pending tasks queue. However this parameter is also
re-used in a few places where a timeout of `-1` means something else,
typically to timeout immediately. This commit fixes those places so that
`?master_timeout=-1` consistently means to wait forever.
The Query API Key Information endpoint supports aggs since #104895.
But some lang clients actually use the `aggregations` keyword in requests,
as the preferred synonym to `aggs`.
This PR adds support for the `aggregations` request keyword as a synonym
for the existing `aggs` term.
Closes#106839
The typed_keys request parameter is the canonical parameter,
that's also used in the regular index _search enpoint, in order to
return the types of aggregations in the response.
This is required by typed language clients of the _security/_query/api_key
endpoint that are using aggregations.
Closes#106817
In this PR we introduce the API that will expose the global retention configuration and will allow users to take advantage of it.
These APIs are protected by the dedicated introduced privileges:
`manage_data_stream_global_retention` or higher, which allows all operations on the global retention configuration
`monitor_data_stream_retention` or higher, which allows the retrieval of the global retention configuration.
This PR is the final PR that makes the global retention available for our users.
Add new optional request option, `with_profile_uid`,
to the Get and Query API Key Information endpoints,
to return the API keys owner users' profile uid.
Closes#98939
Defines new inference_user and inference_admin roles with the
related cluster privileges manage_inference and monitor_inference.
inference_user can list the models and preform inference,
inference_admin can do the same plus create and delete models
* Add modelId and modelText to KnnVectorQueryBuilder
Use QueryVectorBuilder within KnnVectorQueryBuilder to make it
possible to perform knn queries also when a query vector is not
immediately available. Supplying a text_embedding query_vector_builder
with model_text and model_id instead of the query_vector will result
in the generation of a query_vector by calling inference on the
specified model_id with the supplied model_text (during query
rewrite). This is consistent with the way query vectors are built
from model_id / model_text in KnnSearchBuilder (DFS phase).
This enhancement adds a new abstraction to the _search API called "retriever." A
retriever is something that returns top hits. This adds three initial retrievers called
"standard", "knn", and "rrf". The retrievers use a parser-only approach where they
are parsed and then translated into a SearchSourceBuilder to execute the actual
search.
---------
Co-authored-by: Mayya Sharipova <mayya.sharipova@elastic.co>
