With the rise of larger CPU count nodes our current write queue size
might be too conservative. Indexing pressure will still provide protect
against out of memories.
* Update semantic text docs to suggest using index options for customization
* Correct type of index_options
* Move example
* PR feedback
* Copy warning fix
* [DOCS] Update ESQL metadata fields page
**esql-metadata-fields.md:**
- restructured from bullet list to table format for metadata fields
- added `_index_mode` and `_source` fields to available metadata
- improved field descriptions (more detailed)
- added "usage and limitations" section
- reorganized examples into subsections with headers
- added `_score` sorting example
- added tip box linking to search documentation
* 🚙Drive by updates to search functions ref page
moved tutorial link into tip box at top
added cross-reference to search overview documentation
minor text flow improvements and punctuation fixes
* Fix id typo
* Apply suggestions from review
Co-authored-by: Bogdan Pintea <sig11@mailbox.org>
* Declare LU JOIN GA in 9.1
* Align applies_to tags for sample, change_point
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Introduces a new `RemoveBlock` API that complements the existing `AddBlock` API by allowing users to remove index blocks using `DELETE /{index}/_block/{block}`.
Resolves#128966
---------
Co-authored-by: Niels Bauman <nielsbauman@gmail.com>
* Add vector tile examples
* Add new page to TOC
* Add internal translation example
* Update docs/reference/elasticsearch/rest-apis/vector-tile-search.md
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
* Update docs/reference/elasticsearch/rest-apis/vector-tile-search.md
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
---------
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
This adds `counter_long` and `counter_double` to the signatures of
supported fields for `IS NULL` and `IS NOT NULL`. We hadn't been
generating those signatures since the docs v3 migration, so this had to
plug those in. In addition, it changes the wording on a few things and
adds a note that if a field is only in some documents then the ones
missing the field will have `NULL` - which is important information for
`IS NULL` and `IS NOT NULL`.
* Initial checkin of refactored index_options code
* [CI] Auto commit changes from spotless
* initial unit testing
* complete unit tests; add yaml tests
* [CI] Auto commit changes from spotless
* register test feature for sparse vector
* Update docs/changelog/129089.yaml
* update changelog
* add docs
* explicit set default index_options if null
* [CI] Auto commit changes from spotless
* update yaml tests; update docs
* fix yaml tests
* readd auth for teardown
* only serialize index options if not default
* [CI] Auto commit changes from spotless
* serialization refactor; pass index version around
* [CI] Auto commit changes from spotless
* fix transport versions merge
* fix up docs
* [CI] Auto commit changes from spotless
* fix docs; add include_defaults unit and yaml test
* [CI] Auto commit changes from spotless
* override getIndexReaderManager for SemanticQueryBuilderTests
* [CI] Auto commit changes from spotless
* cleanup mapper/builder/tests; index vers. in type
still need to refactor / clean YAML tests
* [CI] Auto commit changes from spotless
* cleanups to mapper tests for clarity
* [CI] Auto commit changes from spotless
* move feature into mappers; fix yaml tests
* cleanups; add comments; remove redundant test
* [CI] Auto commit changes from spotless
* escape more periods in the YAML tests
* cleanup mapper and type tests
* [CI] Auto commit changes from spotless
* rename mapping for previous index test
* set explicit number of shards for yaml test
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co>
* [DOCS][ESQL] Flip preview booleans, to GA search functions
* render docs, tweak some applies_to metadata in docs gen code
- **rendered docs (md):**
- kql: removed serverless preview, added ga 9.1.0
- match: removed serverless preview, added ga 9.1.0
- match_phrase: changed from preview 9.1.0 to unavailable 9.0 + ga 9.1.0
- qstr: removed serverless preview, added ga 9.1.0
- search functions list: removed bullet point before term function
- **docs generation code (java):**
- match_phrase: updated function info annotations to unavailable 9.0 + ga 9.1.0
- query_string: uncommented ga 9.1.0 annotation
- Add PREVIEW annotations to all preview functions
- Update docs generation logic to use annotations instead of preview boolean
* Changed stack: ga 9.1 to stack: coming in multiple places
- Match.java: Updated the options parameter description
- MultiMatch.java: Updated the options parameter description
- ToLower.java: Reformatted parameter description and updated version annotation
- ToUpper.java: Removed the appliesTo annotation entirely and reformatted parameter description
- updated applies_to annotations to specify both preview 9.0.0 and ga 9.1.0 lifecycle stages
- added version-specific documentation examples with applies_to markers for ga 9.1.0 features
- enhanced to_lower and to_upper functions to indicate support for multi-valued expressions in ga 9.1.0
- added version guards around function parameters and descriptions using applies_to syntax
- updated function parameter descriptions to indicate ga 9.1.0 availability for named parameters
- use detailedDescription + and fix match_phrase applies_to syntax
- strip inline applies_to from kibana docs
- update roundto, matchphrase lifecycles
- fix match named params info
- various spatial functions are preview
- unsigned long is preview
- update qstr
- Update TO_LOWER/TO_UPPER parameter descriptions for clarity
- hide spatial functions per https://github.com/elastic/elasticsearch/pull/129839
- added `stack: ga 9.1.0` blocks to match_phrase.md and qstr.md examples
- simplified term.md version from `preview 9.0.0` to just `preview`
- added `applies_to = "stack: ga 9.1.0"` to matchphrase and querystring java annotations
- removed version `9.0.0` from term function annotation
- deleted unused `makeCallout()` and `appendLifeCycleAndVersion()` methods
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Liam Thompson <leemthompo@gmail.com>
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Co-authored-by: Nik Everett <nik9000@gmail.com>
#125143 added 9 spatial grid functions and released them into Serverless. We think this is not the best long-term approach and the functions in #129581 are likely better.
As a first step, rmove the spatial grid functions added in #125143 from release builds so they don't get released into 8.19/9.1.
---------
Co-authored-by: Craig Taverner <craig@amanzi.com>
There is an issue where for Flattened fields with synthetic source, if there is a key with a scalar value, and a duplicate key with an object value, one of the values will be left out of the produced synthetic source. This fixes the issue by replacing the object with paths to each of its keys. These paths consist of the concatenation of all keys going down to a given scalar, joined by a period. For example, they are of the form foo.bar.baz. This applies recursively, so that every value within the object, no matter how nested, will be accessible through a full specified path.
* Add index_options parameter to semantic_text field mapping
* Cleanup & tests
* Update docs
* Update docs/changelog/119967.yaml
* Addressed some PR feedbak
* Update yaml tests
* Refactoring
* Cleanup
* Fix some tests
* Hack in inferring text_embedding task type from index options
* [CI] Auto commit changes from spotless
* Fix error inferring model settings
* Update docs
* Update tests
* Update docs/reference/mapping/types/semantic-text.asciidoc
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
* Address some minor PR feedback
* Remove partial model_settings with inferred task type
* Cleanup
* Remove unnecessary changes
* Fix errors from merge
* [CI] Auto commit changes from spotless
* Cleanup
* Checkpoint, saving changes before merge
* Update parsing
* [CI] Auto commit changes from spotless
* Stash changes
* Fix compile errors
* [CI] Auto commit changes from spotless
* Cleanup error
* fix test
* fix test
* Fix another test
* A bit of cleanup
* Fix tests
* Spotless
* Respect index options if set over defaults
* Cleanup
* [CI] Auto commit changes from spotless
* Support updating to compatible versions, add some cleanup and validation
* Remove test that can't be done here - needs to be unit test
* Add validation
* Cleanup
* Fix some yaml tests
* [CI] Auto commit changes from spotless
* Happy path early index validation works now; edge cases surrounding default BBQ remain
* Always emit index options, even when using defaults
* Minor cleanup
* Fix test compilation failures
* Fix some tests
* Continue to iterate on test failures
* Remove index options from inference field metadata as it is only needed at field creation time
* Fix some tests
* Remove transport version, no longer needed
* Fix yaml tests
* Add tests
* IndexOptions don't need to implement Writeable
* [CI] Auto commit changes from spotless
* Refactor - move SemanticTextIndexOptions
* Remove writeable
* Move index_options parsing to semantic text field mapper
* Cleanup
* Fix test compilation issue
* Cleanup
* Remove whitespace
* Remove writeables from index options
* Disable merging null options?
* Add docs
* [CI] Auto commit changes from spotless
* Revert "Disable merging null options?"
This reverts commit 2ef8b1dc29.
* Remove default serialization
* Include default index option type to defaults
* [CI] Auto commit changes from spotless
* Go back to allowing null updateS
* Cleanup
* Fix validation error
* Revert "Include default index option type to defaults"
This reverts commit b08e2a1d7e.
* Update tests
* Revert "Update tests"
This reverts commit aedfafe0e7.
* Better fix for null inputs
* Remove redundant merge validation
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
* propgating retrievers to inner retrievers
* test feature taken care of
* Small changes in concurrent multipart upload interfaces (#128977)
Small changes in BlobContainer interface and wrapper.
Relates ES-11815
* Unmute FollowingEngineTests#testProcessOnceOnPrimary() test (#129054)
The reason the test fails is that operations contained _seq_no field with different doc value types (with no skippers and with skippers) and this isn't allowed, since field types need to be consistent in a Lucene index.
The initial operations were generated not knowing about the fact the index mode was set to logsdb or time_series. Causing the operations to not have doc value skippers. However when replaying the operations via following engine, the operations did have doc value skippers.
The fix is to set `index.seq_no.index_options` to `points_and_doc_values`, so that the initial operations are indexed without doc value skippers.
This test doesn't gain anything from storing seqno with doc value skippers, so there is no loss of testing coverage.
Closes#128541
* [Build] Add support for publishing to maven central (#128659)
This ensures we package an aggregation zip with all artifacts we want to publish to maven central as part of a release.
Running zipAggregation will produce a zip file in the build/nmcp/zip folder. The content of this zip is meant to match the maven artifacts we have currently declared as dra maven artifacts.
* ESQL: Check for errors while loading blocks (#129016)
Runs a sanity check after loading a block of values. Previously we were
doing a quick check if assertions were enabled. Now we do two quick
checks all the time. Better - we attach information about how a block
was loaded when there's a problem.
Relates to #128959
* Make `PhaseCacheManagementTests` project-aware (#129047)
The functionality in `PhaseCacheManagement` was already project-aware,
but these tests were still using deprecated methods.
* Vector test tools (#128934)
This adds some testing tools for verifying vector recall and latency
directly without having to spin up an entire ES node and running a rally
track.
Its pretty barebones and takes inspiration from lucene-util, but I
wanted access to our own formats and tooling to make our lives easier.
Here is an example config file. This will build the initial index, run
queries at num_candidates: 50, then again at num_candidates 100 (without
reindexing, and re-using the cached nearest neighbors).
```
[{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 50,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"index_threads" : 4,
"reindex" : true,
"force_merge" : false,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
},
{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 100,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
}
]
```
To execute:
```
./gradlew :qa:vector:checkVec --args="/Path/to/knn_tester_config.json"
```
Calling `./gradlew :qa:vector:checkVecHelp` gives some guidance on how
to use it, additionally providing a way to run it via java directly
(useful to bypass gradlew guff).
* ES|QL: refactor generative tests (#129028)
* Add a test of LOOKUP JOIN against a time series index (#129007)
Add a spec test of `LOOKUP JOIN` against a time series index.
* Make ILM `ClusterStateWaitStep` project-aware (#129042)
This is part of an iterative process to make ILM project-aware.
* Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex ASYNC} #129078
* Remove `ClusterState` param from ILM `AsyncBranchingStep` (#129076)
The `ClusterState` parameter of the `asyncPredicate` is not used
anywhere.
* Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex SYNC} #129082
* Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/70_ilm/Test Lifecycle Still There And Indices Are Still Managed} #129097
* Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/90_ml_data_frame_analytics_crud/Get mixed cluster outlier_detection job} #129098
* Mute org.elasticsearch.packaging.test.DockerTests test081SymlinksAreFollowedWithEnvironmentVariableFiles #128867
* Threadpool merge executor is aware of available disk space (#127613)
This PR introduces 3 new settings:
indices.merge.disk.check_interval, indices.merge.disk.watermark.high, and indices.merge.disk.watermark.high.max_headroom
that control if the threadpool merge executor starts executing new merges when the disk space is getting low.
The intent of this change is to avoid the situation where in-progress merges exhaust the available disk space on the node's local filesystem.
To this end, the thread pool merge executor periodically monitors the available disk space, as well as the current disk space estimates required by all in-progress (currently running) merges on the node, and will NOT schedule any new merges if the disk space is getting low (by default below the 5% limit of the total disk space, or 100 GB, whichever is smaller (same as the disk allocation flood stage level)).
* Add option to include or exclude vectors from _source retrieval (#128735)
This PR introduces a new include_vectors option to the _source retrieval context.
When set to false, vectors are excluded from the returned _source.
This is especially efficient when used with synthetic source, as it avoids loading vector fields entirely.
By default, vectors remain included unless explicitly excluded.
* Remove direct minScore propagation to inner retrievers
* cleaned up skip
* Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testAvailableDiskSpaceMonitorWhenFileSystemStatErrors #129149
* Add transport version for ML inference Mistral chat completion (#129033)
* Add transport version for ML inference Mistral chat completion
* Add changelog for Mistral Chat Completion version fix
* Revert "Add changelog for Mistral Chat Completion version fix"
This reverts commit 7a57416bdc.
* Correct index path validation (#129144)
All we care about is if reindex is true or false. We shouldn't worry
about force merge. Because if reindex is true, we will create the
directory, if its false, we won't.
* Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testUnavailableBudgetBlocksNewMergeTasksFromStartingExecution #129148
* Implemented completion task for Google VertexAI (#128694)
* Google Vertex AI completion model, response entity and tests
* Fixed GoogleVertexAiServiceTest for Service configuration
* Changelog
* Removed downcasting and using `moveToFirstToken`
* Create GoogleVertexAiChatCompletionResponseHandler for streaming and non streaming responses
* Added unit tests
* PR feedback
* Removed googlevertexaicompletion model. Using just GoogleVertexAiChatCompletionModel for completion and chat completion
* Renamed uri -> nonStreamingUri. Added streamingUri and getters in GoogleVertexAiChatCompletionModel
* Moved rateLimitGroupHashing to subclasses of GoogleVertexAiModel
* Fixed rate limit has of GoogleVertexAiRerankModel and refactored uri for GoogleVertexAiUnifiedChatCompletionRequest
---------
Co-authored-by: lhoet-google <lhoet@google.com>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
* ES|QL - kNN function initial support (#127322)
* Remove optional seed from ES|QL SAMPLE (#128887)
* Remove optional seed from ES|QL SAMPLE
* make it clear that seed is for testing
* [Inference API] Add "rerank" task type to "elastic" provider (#126022)
* Rename target destination for microbenchmarks (#128878)
* Include direct memory and non-heap memory in ML memory calculations (take #2) (#128742)
* Include direct memory and non-heap memory in ML memory calculations.
* Reduce ML_ONLY heap size, so that direct memory is accounted for.
* [CI] Auto commit changes from spotless
* changelog
* improve docs
* Reuse direct memory to heap factor
* feature flag
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* Throw better exception for unsupported aggregations over shape fields (#129139)
* Update Test Framework To Handle Query Rewrites That Rely on Non-Null Searchers (#129160)
* Update ReproduceInfoPrinter to correctly print a reproduction line for Lucene & build candidate upgrade tests (#129044)
* Increment inference stats counter for shard bulk inference calls (#129140)
This change updates the inference stats counter to include chunked inference calls performed by the shard bulk inference filter on all semantic text fields.
It ensures that usage of inference on semantic text fields is properly recorded in the stats.
* Synthetic source: avoid storing multi fields of type text and match_only_text by default. (#129126)
Don't store text and match_only_text field by default when source mode is synthetic and a field is a multi field or when there is a suitable multi field.
Without this change, ES would store field otherwise twice in a multi-field configuration.
For example:
```
...
"os": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword",
"fields": {
"text": {
"type": "match_only_text"
}
}
}
...
```
In this case, two stored fields were added, one in case for the `name` field and one for `name.text` multi-field.
This change prevents this, and would never store a stored field when text or match_only_text field is a multi-field.
* Adding `scheduled_report_id` field to kibana reporting template (#127827)
* Adding scheduled_report_id field to kibana reporting template
* Incrementing stack template registry version
* ES|QL: Add FORK generative tests (#129135)
* ES|QL Completion command syntax change (#129189)
* Included pinned retriever in 9.1 docs
* reverted unnecessary change
* made the suggested changes
* Update retrievers.md
* Update docs/reference/elasticsearch/rest-apis/retrievers.md
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
* Update docs/reference/elasticsearch/rest-apis/retrievers.md
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
* Update retrievers.md
---------
Co-authored-by: Tanguy Leroux <tlrx.dev@gmail.com>
Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
Co-authored-by: Rene Groeschke <rene@elastic.co>
Co-authored-by: Nik Everett <nik9000@gmail.com>
Co-authored-by: Niels Bauman <33722607+nielsbauman@users.noreply.github.com>
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: Luigi Dell'Aquila <luigi.dellaquila@gmail.com>
Co-authored-by: Bogdan Pintea <bogdan.pintea@elastic.co>
Co-authored-by: elasticsearchmachine <58790826+elasticsearchmachine@users.noreply.github.com>
Co-authored-by: Albert Zaharovits <email+github@zalbert.me>
Co-authored-by: Jim Ferenczi <jim.ferenczi@elastic.co>
Co-authored-by: Jan-Kazlouski-elastic <jan.kazlouski@elastic.co>
Co-authored-by: Leonardo Hoet <55866308+leo-hoet@users.noreply.github.com>
Co-authored-by: lhoet-google <lhoet@google.com>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
Co-authored-by: Jan Kuipers <148754765+jan-elastic@users.noreply.github.com>
Co-authored-by: Tim Grein <tim.grein@elastic.co>
Co-authored-by: Ievgen Degtiarenko <ievgen.degtiarenko@elastic.co>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Ignacio Vera <ignacio.vera@elastic.co>
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
Co-authored-by: Moritz Mack <mmack@apache.org>
Co-authored-by: Ying Mao <ying.mao@elastic.co>
Co-authored-by: Ioana Tagirta <ioanatia@users.noreply.github.com>
Co-authored-by: Aurélien FOUCRET <aurelien.foucret@gmail.com>
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Adds support for LIKE function alternative syntax with a list of patterns.
Examples:
FROM foo
| WHERE bar LIKE ("A*","B*", "C?")
The new syntax is documented as part of the existing LIKE function documentation. We will use the existing WildcardLike java implementation for existing cases using the old syntax and one list argument case to improve mixed cluster compatibility.
This introduces a `none` chunking strategy that disables automatic chunking when using an inference endpoint.
It enables users to provide pre-chunked input directly to a `semantic_text` field without any additional splitting.
The chunking strategy can be configured either on the inference endpoint or directly in the `semantic_text` field definition.
**Example:**
```json
PUT test-index
{
"mappings": {
"properties": {
"my_semantic_field": {
"type": "semantic_text",
"chunking_settings": {
"strategy": "none" <1>
}
}
}
}
}
```
<1> Disables automatic chunking on `my_semantic_field`.
```json
PUT test-index/_doc/1
{
"my_semantic_field": ["my first chunk", "my second chunk", ...] <1>
...
}
```
<1> Pre-chunked input provided as an array of strings.
Each array element represents a single chunk that will be sent directly to the inference service without further processing.
This extends RENAME's grammar to allow a new syntax: `| RENAME new_name
= old_name` This is supported along the existing `... old_name AS
new_name` syntax.
Closes#129208
