* Update missed documentation change for new watcher index settings
* Update docs/reference/rest-api/watcher/update-settings.asciidoc
---------
Co-authored-by: Keith Massey <keith.massey@elastic.co>
Building off of `stats` and multi-value aggregations, including the
limitation:
- all values of extended_stats will be mapped to `double` if mapping
deduction is used
Relates #51925
Closeselastic/security-team#11102Closeselastic/security-team#11104
This allows agentless integrations (via elastic/beats#41446, elastic/kibana#203810) to write to agentless-* indices. Each index is created on-demand by the filebeat client and kibana conditionally extends the API key permissions to allow writing to the index.
(cherry picked from commit 3c184b912c)
# Conflicts:
# docs/reference/rest-api/security/get-service-accounts.asciidoc
# x-pack/plugin/security/qa/service-account/src/javaRestTest/java/org/elasticsearch/xpack/security/authc/service/ServiceAccountIT.java
# x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authc/service/ElasticServiceAccounts.java
This ensures the usage API docs tests are passing again. We achieve this
by: 1. ignoring the contents of `inference.models` because the models
might not yet have been initialized and 2. adding missing fields to the
`logsdb` usage.
This commit adds docs for monitor_stats and updates an example snippet to include both remote_indices and remote_cluster.
(cherry picked from commit b7167b73e3)
* Add a monitor_stats privilege and allow that privilege for remote cluster privileges (#114964)
This commit does the following:
* Add a new monitor_stats privilege
* Ensure that monitor_stats can be set in the remote_cluster privileges
* Give's Kibana the ability to remotely call monitor_stats via RCS 2.0
Since this is the first case where there is more than 1 remote_cluster privilege,
the following framework concern has been added:
* Ensure that when sending to elder RCS 2.0 clusters that we don't send the new privilege
previous only supported all or nothing remote_cluster blocks
* Ensure that we when sending API key role descriptors that contains remote_cluster,
we don't send the new privileges for RCS 1.0/2.0 if it not new enough
* Fix and extend the BWC tests for RCS 1.0 and RCS 2.0
(cherry picked from commit af99654dac)
* adjust bwc for 8.x branch
Documentation for the remote_cluster in the role was added
in #111682 and #108840, but a few places were missed.
This commit fill the gaps in the documentation.
This PR adds telemetry for logsdb. However, this change only tracks the
count of indices using logsdb and those that use synthetic source.
Additional stats, such as shard, indexing, and search stats, will be
added in a follow-up, as they require reaching out to data nodes.
This removes the undocumented `META FUNCTIONS` command that emits
descriptions for all functions. This shouldn't be used because we never
told anyone about it. I'd have preferred if we'd have explicitly
documented it as no public or if we'd have left it snapshot-only. But
sometimes you make a mistake. I'm hopeful no one is relying on it. It
was never reliable and not public.....
Today the docs for the `?wait_for_active_shards` parameter say that it
must be a positive integer, proscribing `0`, yet `0` is a legitimate
value for this parameter. This commit fixes this point and rewords the
docs slightly for clarity.
Adds deployment threading options and a new memory section reporting
the memory usage for each of the ml features
# Conflicts:
# server/src/main/java/org/elasticsearch/TransportVersions.java
Currently, the data stream lifecycle telemetry has the following
structure:
```
{
....
"data_lifecycle" : {
"available": true,
"enabled": true,
"count": 0,
"default_rollover_used": true,
"retention": {
"minimum_millis": 0,
"maximum_millis": 0,
"average_millis": 0.0
}
}....
```
In the snippet above you can see that we track:
- The amount of data streams managed by the data stream lifecycle by `count`
- If the default rollover has been overwritten by `default_rollover_used`
- The min, max and average of the `data_retention` configured on a data stream level.
In this PR we propose the following extention:
```
....
"data_lifecycle" : {
"available": true,
"enabled": true,
"count": 0,
"default_rollover_used": true,
"effective_retention": { #https://github.com/elastic/dev/issues/2537
"retained_data_streams": 5,
"minimum_millis": 0, # Only if retained data streams > 1
"maximum_millis": 0,
"average_millis": 0.0
},
"data_retention": {
"configured_data_streams": 5,
"minimum_millis": 0, # Only if retained data streams > 1
"maximum_millis": 0,
"average_millis": 0.0
},
"global_retention": {
"default": {
"defined": true/false,
"affected_data_streams": 0,
"millis": 0
},
"max": {
"defined": true/false,
"affected_data_streams": 0,
"millis": 0
}
}
```
With this extension we are tracking:
- The amount of data streams managed by the data stream lifecycle by `count`
- If the default rollover has been overwritten by `default_rollover_used`
- The min, max and average of the `data_retention` configured on a data stream level and the number of data streams that have it configured. We add the min, max and avg only if there are data streams with data retention configuration to avoid messing with the stats in a dashboard.
- The min, max and average of the `effective_retention` and the number of data streams that are retained. We add the min, max and avg only if there are retained data streams to avoid messing with the stats in a dashboard.
- Global retention stats, if they are defined, if the number of the affected data streams and the actual value.
The above metrics allow us to answer questions like:
- How many data streams are affected by global retention.
- How big is the difference between the longest data retention compared to max global retention.
- How much does the effective retention diverging from the data retention, this will show the impact of the global retention.
In this PR we remove unused code including relating to the global data retention with APIs implementation:
- The transport action for updating, deleting and retrieving the global retention.
- The `DataStreamGlobalRetention` from the cluster state (this should be bwc safe because we never exposed the APIs to add a data stream lifecycle to the cluster state).
- Make unused privileges monitor and managing global retention a noop.
- Remove cluster state update tasks.
The kept `DataStreamGlobalRetentionResolver` considering it could hold and provide the global retention from the settings when we implement it. We just renamed it to DataStreamGlobalRetentionProvider for now to better match what it does.
The factory retention settings should still work after this change.
* Create manage_seaech_connector privilege
* `manage_search_connector` -> `manage_connector` and exclude connector secrets patterns from this privilege
* Add `monitor_connector` privilege
* Update Kibana system privilege to monitor_connector for telemetry
* Rename privilege to 'manage_connector_state'
Since privilege names are often namespaced and used with globs, we want to ensure that if there's a future privilege like `manage_connector_secrets`, that it is not implicitly included in this new privileg's <name>*. By extending the privilege name to include "_state", we better namespace this distinct from any "_secrets" namespace.
* Revert "Rename privilege to 'manage_connector_state'"
This reverts commit 70b89eee76.
After further discussion with the security team, this name change is not needed after all
since the secret management privileges aren't currently prefixed with "manage_"
---------
Co-authored-by: Sean Story <sean.j.story@gmail.com>
Docs tweak with a typo fix and a clarification on how the two available
settings interact (essentially
https://github.com/elastic/elasticsearch/issues/27871). I'm also open to
including this info in the more generic settings API but feels like a
simple enough callout to add to the security API.
Introduce an optional k param for knn query
If k is not set, knn query has the previous behaviour:
- `num_candidates` docs is collected from each shard. This `num_candidates` docs
are used for combining with results with other queries and aggregations on each shard.
- docs from all shards are merged to produce the top global `size` results
If k is set, the behaviour instead is following:
- `k` docs is collected from each shard. This `k` docs are used for
combining results with other queries and aggregations on each shard.
- similarly, docs from all shards are merged to produce the top global `size`
results.
Having `k` param makes it more intuitive for users to address their needs.
They also don't need to care and can skip `num_candidates` param for this query
as it is of more internal details to tune how knn search operates.
Closes#108473
The cluster level dense vector stats returns the total number of dense vector indices globally including the replicas.
This commit fixes the total to only include the value count of the primary indices.
This change aligns with the docs stats which also reports the number of primary documents when used in cluster stats.
The indices stats API still reports granular results for replicas and primaries so the information is not lost.
* Add SparseVectorStats
* Update to use mappings in engine
* Update to be unique to primary shards
* Fix doc
* Fix null error in test
* Cleanup
* fix yaml
* remove comment
* add version to yaml
* Revert whitespace changes to stats doc
* fix yml test
* Checkstyle
* Fix NPE in test
* Update docs/changelog/108793.yaml
* Add link to sparse_vector field type in docs
* PR feedback
* Flesh out test a bit more
* PR feedback - alphabetize placement in docs
* Fix doc change
This is a refactoring of the internal logic that's used to translate
query-level into index-level field names for query APIs for
security entities (i.e. users, API Keys, and soon, roles).
The objective here is to have and reuse a single class to handle
all the translations for different security query APIs.
This commit introduces the documentation for remote_clusters which is used to help
express the monitor_enrich privilege needed to use the ENRICH keyword across clusters
when using the API key based CCS security model.
This commit also adds "remote_clusters" to the built in privs API to for easier consumption
in Kibana.
This commit adds documentation for the DLS/FLS restriction for RCS 2.0 API keys
where both access and replication are defined and access has DSL/FLS.
This commit also fixes a few misleading variable names.
related: #108600
This change adds `number_of_rollup_jobs` and `number_of_rollup_indices`
to `rollup` usage. The former indicates the number of active rollup
jobs running and the latter indicated the number of rollup indices
(which could be the result of previous rollup jobs).
APIs which perform cluster state updates typically accept the
`?master_timeout=` and `?timeout=` parameters to respectively set the
pending task queue timeout and the acking timeout for the cluster state
update. Both of these parameters accept the value `-1`, but
`?master_timeout=-1` means to wait indefinitely whereas `?timeout=-1`
means the same thing as `?timeout=0`, namely that acking times out
immediately on commit.
There are some situations where it makes sense to wait for as long as
possible for nodes to ack a cluster state update. In practice this wait
is bounded by other mechanisms (e.g. the lag detector will remove the
node from the cluster after a couple of minutes of failing to apply
cluster state updates) but these are not really the concern of clients.
Therefore with this commit we change the meaning of `?timeout=-1` to
mean that the acking timeout is infinite.
* Remove `es-test-dir` book-scoped variable
* Remove `plugins-examples-dir` book-scoped variable
* Remove `:dependencies-dir:` and `:xes-repo-dir:` book-scoped variables
- In `index.asciidoc`, two variables (`:dependencies-dir:` and `:xes-repo-dir:`) were removed.
- In `sql/index.asciidoc`, the `:sql-tests:` path was updated to fuller path
- In `esql/index.asciidoc`, the `:esql-tests:` path was updated idem
* Replace `es-repo-dir` with `es-ref-dir`
* Move `:include-xpack: true` to few files that use it, remove from index.asciidoc
Specifying `?master_timeout=-1` on an API which performs a cluster state
update means that the cluster state update task will never time out
while waiting in the pending tasks queue. However this parameter is also
re-used in a few places where a timeout of `-1` means something else,
typically to timeout immediately. This commit fixes those places so that
`?master_timeout=-1` consistently means to wait forever.
