There are many features of the Elasticsearch ecosystem that may malfunction, or fail to work entirely, if these templates are not installed. This commit adds documentation cautioning against disabling the installation of templates.
* [DOCS] Update documentation for index sorting and routing for logsdb
* update
* Apply suggestions from code review
* Update logs.asciidoc
* Update docs/reference/data-streams/logs.asciidoc
* Update logs.asciidoc
---------
Co-authored-by: Marci W <333176+marciw@users.noreply.github.com>
Adding `security_solution-*-*` in list of index nae to avoid the pattern collisions.
(cherry picked from commit 0638d3977a)
Co-authored-by: Smriti <152067238+smriti0321@users.noreply.github.com>
* ES|QL async queries: Partial result on demand (#118122)
Add capability to stop async query on demand
The theory:
- User initiates async search request
- User sends the stop request (POST _query/async/<ID>/stop)
- If the async is finished by that time, it's like regular async get
- If it's not finished, the sinks are closed and the request is forcefully finished
(cherry picked from commit f27f74666f)
# Conflicts:
# x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlQueryResponse.java
# x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/action/EsqlQueryResponseTests.java
# x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/CrossClusterEsqlRCS1UnavailableRemotesIT.java
# x-pack/plugin/security/qa/multi-cluster/src/javaRestTest/java/org/elasticsearch/xpack/remotecluster/CrossClusterEsqlRCS2UnavailableRemotesIT.java
* fix tests
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Resolves#109999
This adds support for date nanos in the date diff function, as well as mixed nanos/millis use cases.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* ESQL: Signatures for `NOT IN` et al
This generates signatures for `NOT IN`, `NOT LIKE`, and `NOT RLIKE`
using a small hack on top of the process used to generate the signatures
for `IN`, `LIKE`, and `RLIKE`. This is a very perl-worth hack, replacing
`LIKE` with `NOT LIKE` in the description. But it's useful for our
kibana friends and if we need to make it nicer we can do so later.
* Zap
Resolve/cluster allows querying for cluster-info-only (no index expression required)
This enhancement provides users with the ability to query the _resolve/cluster API endpoint without specifying
an index expression to match against. This allows users to quickly test what remote clusters are configured on
a cluster and whether they are available for querying.
The new endpoint takes no index expression:
```
GET _resolve/cluster
```
and returns the same information as before except for the "matching_indices" field. Example response:
```
{
"remote1": {
"connected": false,
"skip_unavailable": true
},
"remote2": {
"connected": true,
"skip_unavailable": false,
"version": {
"number": "8.17.0",
"build_flavor": "default",
"minimum_wire_compatibility_version": "7.17.0",
"minimum_index_compatibility_version": "7.0.0"
}
}
}
```
For backwards compatibility, this new endpoint works with clusters from older versions by querying with the index expression `dummy*` on those older clusters and ignoring the matching_indices value in the response they return.
* Forward port release notes for v8.16.3
* Delete includes
* del coming tag
---------
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Semantic text fields now support multi-fields, either as part of a multi-field structure or containing multi-fields internally.
This enhancement aligns with the semantic text field's current behavior as a standard text field.
Note: Multi-field support is only available for the new index format. Attempting to set a multi-field on an index created with the older format will still result in a failure.
* ESQL - docs for to_date_nanos (#120124)
I forgot to link the ToDateNanos docs when I merged that function.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Conflicts:
docs/reference/esql/functions/description/to_date_nanos.asciidoc
docs/reference/esql/functions/kibana/definition/to_date_nanos.json
docs/reference/esql/functions/kibana/docs/to_date_nanos.md
docs/reference/esql/functions/layout/to_date_nanos.asciidoc
* ESQL - docs for to_date_nanos (#120124)
I forgot to link the ToDateNanos docs when I merged that function.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Conflicts:
docs/reference/esql/functions/description/to_date_nanos.asciidoc
docs/reference/esql/functions/kibana/definition/to_date_nanos.json
docs/reference/esql/functions/kibana/docs/to_date_nanos.md
docs/reference/esql/functions/layout/to_date_nanos.asciidoc
* unmute ToDateNanos tests, and build docs
Make it explicit that es expects disks to have the same capacity across all the nodes in the same data tier.
(cherry picked from commit 3ebc1f48aa)
Co-authored-by: Ievgen Degtiarenko <ievgen.degtiarenko@elastic.co>
This commit adds an "Ingest changes" to the breaking changes section to outline this new behavior.
Co-authored-by: Zach Brisson <162374237+ZachBrisson-Elastic@users.noreply.github.com>
There were different error cases with `ROUND(number, decimals)`:
- Decimals accepted unsigned longs, but threw a 500 with a `can't process [unsigned_long -> long]` in the cast evaluator
- Fixed by improving the `resolveType()`
- If the number was a BigInteger unsigned long, there were 2 cases throwing an exception:
1. Negative decimals outside the range of integer: Error
2. Negative decimals insie the range of integer, but "big enough" for `BigInteger.TEN.pow(...)` to throw a `BigInteger would overflow supported range`
3. -19 decimals with big unsigned longs like `18446744073709551615` was throwing an `unsigned_long overflow`
Also, when the number is a BigInteger and the decimals is a big negative (but not big enough to throw), it may be **very** slow. Taking _many_ seconds for a single computation (It tries to calculate a `10^(big number)`. I didn't do anything here, but I wonder if we should limit it.
To solve most of the cases, a warnExceptions was added for the overflow case, and a guard clause to return 0 for <-19 decimals on unsigned longs.
Another issue is that rounding to a number like 7 to -1 returns 0 instead of 10, which may be considered an error. But it's consistent, so I'm leaving it to another PR
This reduces the number of test cases in ESQL a little more ala #119678.
It migrates a few random tests and all of the multivalue functions:
```
92775 -> 43760
3m45 -> 4m04
```
This adds a few more error test cases that were missing to make sure it all
lines up well. And it fixes a few error messages in a few functions. That's
*likely* where the extra time goes.
* Added additional entries for troubleshooting unhealthy cluster
Reordered "Re-enable shard allocation" because not as common as other causes
Added additional causes of yellow statuses
Changed watermark commadn to include high and low watermark so users can make their cluster operate once again.
* Drive-by copyedit with suggestions for concision and some formatting fixes.
* Concision and some formatting fixes.
* Colon added
* Update docs/reference/troubleshooting/common-issues/red-yellow-cluster-status.asciidoc
* Title change
* Update docs/reference/troubleshooting/common-issues/red-yellow-cluster-status.asciidoc
* Spelling fix
* Update docs/reference/troubleshooting/common-issues/red-yellow-cluster-status.asciidoc
* Update docs/reference/troubleshooting/common-issues/red-yellow-cluster-status.asciidoc
* Update docs/reference/troubleshooting/common-issues/red-yellow-cluster-status.asciidoc
* Update docs/reference/troubleshooting/common-issues/red-yellow-cluster-status.asciidoc
---------
Co-authored-by: Kofi B <seanziee@gmail.com>
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
This adds support for passing Date Nanos into the Date Format function. It works for both the single argument and two argument versions. Format strings are unchanged, as the same formatting logic works for both resolutions.
resolves#109994
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
`fold` can be surprisingly heavy! The maximally efficient/paranoid thing
would be to fold each expression one time, in the constant folding rule,
and then store the result as a `Literal`. But this PR doesn't do that
because it's a big change. Instead, it creates the infrastructure for
tracking memory usage for folding as plugs it into as many places as
possible. That's not perfect, but it's better.
This infrastructure limit the allocations of fold similar to the
`CircuitBreaker` infrastructure we use for values, but it's different
in a critical way: you don't manually free any of the values. This is
important because the plan itself isn't `Releasable`, which is required
when using a real CircuitBreaker. We could have tried to make the plan
releasable, but that'd be a huge change.
Right now there's a single limit of 5% of heap per query. We create the
limit at the start of query planning and use it throughout planning.
There are about 40 places that don't yet use it. We should get them
plugged in as quick as we can manage. After that, we should look to the
maximally efficient/paranoid thing that I mentioned about waiting for
constant folding. That's an even bigger change, one I'm not equipped
to make on my own.
* ESQL Support IN operator for Date nanos (#119772)
Add support for using nanosecond dates with the IN operator. This behavior should be consistent with equals, and support comparisons between milliseconds and nanoseconds the same as the binary comparison operators support it.
Resolves#118578
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* remove use of future java functions
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Closeselastic/security-team#11102Closeselastic/security-team#11104
This allows agentless integrations (via elastic/beats#41446, elastic/kibana#203810) to write to agentless-* indices. Each index is created on-demand by the filebeat client and kibana conditionally extends the API key permissions to allow writing to the index.
(cherry picked from commit 3c184b912c)
# Conflicts:
# docs/reference/rest-api/security/get-service-accounts.asciidoc
# x-pack/plugin/security/qa/service-account/src/javaRestTest/java/org/elasticsearch/xpack/security/authc/service/ServiceAccountIT.java
# x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authc/service/ElasticServiceAccounts.java
This wires up the randomized testing for DateFormat. Prior to this PR, none of the randomized testing was hitting the one parameter version of the function, so I wired that up as well. This required some compromises on the type signatures, see comments in line.less
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* Including examples
* Using js instead of json
* Adding unified docs to main page
* Adding missing description text
* Refactoring to remove unified route
* Addign back references to the _unified route
* Update docs/reference/inference/chat-completion-inference.asciidoc
* Address feedback
---------
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
* Add selector syntax to index expressions (#118614)
This PR introduces a new syntactical feature to index expression resolution: The selector.
Selectors, denoted with a :: followed by a recognized suffix will allow users to specify which component of
an index abstraction they would like to operate on within an API call. In this case, an index abstraction is a
concrete index, data stream, or alias; Any abstraction that can be resolved to a set of indices/shards. We
define a component of an index abstraction to be some searchable unit of the index abstraction.
(cherry picked from commit c3839e1f76)
# Conflicts:
# modules/data-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/IngestFailureStoreMetricsIT.java
# server/src/main/java/org/elasticsearch/TransportVersions.java
# server/src/test/java/org/elasticsearch/action/OriginalIndicesTests.java
* [CI] Auto commit changes from spotless
* Fixing compiler issues
* Remove feature flag from influencing the serialisation
* Only add failure indices when failure store flag is on
* Fix OriginalIndicesTests
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com>
Co-authored-by: Mary Gouseti <mary.gouseti@elastic.co>
Co-authored-by: Mary Gouseti <mgouseti@gmail.com>
