Commit graph

87430 commits

Author SHA1 Message Date
Sam Xiao
41a47c24b7
Make GeoIP database node loader project-aware (#129829)
- loads the downloaded GeoIP databases from system index to ingest node file system for each project
- each project's databases are loaded to directory `tmp/geoip-databases/{nodeId}/{projectId}`
2025-06-25 12:46:47 +08:00
Niels Bauman
73d2e30c39
Clean up x-pack multi-project YAML test mutes (#129965) 2025-06-25 11:04:16 +10:00
Yang Wang
e1c930f8c1
Make RepositoriesService project-aware (#129821)
This PR makes RepositoriesService project aware so that the basic Put,
Get, Delete and Verify repository actions are now project scoped. 

It intentionally leaves the following aspects out of scope for the
current changes: * Repository stats reporting * Repository clean-up,
analysis and integrity verification * Repository usages for searchable
snapshots and CCR

They will be worked on separately. One main reason for leaving them out
is that they are not needed by OBS which is currently blocked by
repository/snapshot changes. They may also have their own complexities,
e.g. stats reporting.

Resolves: ES-10478
2025-06-25 10:34:34 +10:00
elasticsearchmachine
4fe787a1df [renovate] Update branches config 2025-06-25 10:00:20 +10:00
elasticsearchmachine
e7a6983372 Add 9.1 to branches.json 2025-06-25 10:00:19 +10:00
Mike Pellegrini
651bc39565
Simplified Linear & RRF Retrievers - Return error on empty fields param (#129962) 2025-06-24 19:25:24 -04:00
elasticsearchmachine
c3e4f4c189 Mute org.elasticsearch.gradle.internal.InternalDistributionBwcSetupPluginFuncTest builds distribution from branches via archives extractedAssemble [bwcDistVersion: 8.2.1, bwcProject: bugfix, expectedAssembleTaskName: extractedAssemble, #2] #119871 2025-06-25 09:03:01 +10:00
elasticsearchmachine
8509e85345 Mute org.elasticsearch.xpack.test.rest.XPackRestIT test {p0=esql/60_usage/Basic ESQL usage output (telemetry) non-snapshot version} #129888 2025-06-25 08:23:29 +10:00
David Kyle
3a1551e0ef
[ML] Move to the Cohere V2 API for new inference endpoints (#129884) 2025-06-25 07:51:05 +10:00
Brendan Cully
73b0a60a77
Revert "Dispatch ingest work to coordination thread pool (#129820)" (#129949)
This reverts commit 53dae7a3a2.
2025-06-24 14:38:50 -07:00
HYUNSANG HAN (한현상, Travis)
d16271b78d
Add RemoveBlock API to allow DELETE /{index}/_block/{block} (#129128)
Introduces a new `RemoveBlock` API that complements the existing `AddBlock` API by allowing users to remove index blocks using `DELETE /{index}/_block/{block}`.

Resolves #128966

---------

Co-authored-by: Niels Bauman <nielsbauman@gmail.com>
2025-06-25 06:16:14 +10:00
Tim Grein
3b51dd568c
[EIS] Dense Text Embedding task type integration (#129847) 2025-06-24 21:38:16 +02:00
Patrick Doyle
0e2362432c
Pass empty lists instead of nulls to FileAccessTree.of (#129942) 2025-06-25 04:44:13 +10:00
elasticsearchmachine
ba50e26252 Bump versions after 8.18.3 release 2025-06-24 18:30:29 +00:00
elasticsearchmachine
7c13a1553e Bump versions after 9.0.3 release 2025-06-24 18:12:45 +00:00
elasticsearchmachine
cff9da2eae Prune changelogs after 8.17.8 release 2025-06-24 18:00:57 +00:00
elasticsearchmachine
8f1d593119 Bump versions after 8.17.8 release 2025-06-24 17:58:58 +00:00
David Turner
ba103f1c24
Reverse disordered-version warning message (#129904)
The comment in `TransportHandshaker` indicates (correctly) that we emit
a warning when talking to a chronologically-newer-yet-numerically-older
version, but the wording of the warning message is inverted and says
that the remote is chronologically-older-yet-numerically-newer. This
commit straightens out the message to match the situation it is
describing.

Relates #123397
2025-06-24 18:30:11 +01:00
Alexey Ivanov
876c456ac1
Move per-project settings out of ProjectMetadata (#129068)
To better support project restoration after deletion, this change moves project Settings from ProjectMetadata to the new custom in the ClusterState. It also introduces a new transport version for cluster state serialization. Reserved cluster state for project settings remains within ProjectMetadata.

Note: In mixed-version multiproject clusters, this may cause existing settings for projects to temporarily disappear until all nodes have been upgraded and restarted.
2025-06-24 18:06:45 +01:00
Panagiotis Bailis
07f65e978a
Fixing race condition in DynamicMappingIT when checking for updates in mappings (#129931) 2025-06-25 02:31:09 +10:00
Mark J. Hoy
7249ac4d42
Mark Token Pruning for Sparse Vector as GA (#128854)
* remove [preview] labels sparse vec / token pruning

* Update docs/changelog/128854.yaml

* update changelog

* set changelog to feature/ml

* set proper area

* add applies_to labels

* add clarification for token pruning behaviour
2025-06-24 11:36:02 -04:00
Ankit Sethi
4dab46a825
fix file name (#129883)
* fix file name

* Update docs/changelog/129883.yaml

* Delete docs/changelog/129883.yaml
2025-06-24 10:30:23 -05:00
Keith Massey
0b58a53a98
Adding the ability to unset data stream settings (#129677) 2025-06-24 10:30:15 -05:00
Tim Grein
870d581084
[EIS] Implement chunked & batched inference for sparse text embeddings (#129922) 2025-06-24 17:22:29 +02:00
Andrei Stefan
9c52106c61
ESQL: Move explain command validation off EsqlSession (#129918) 2025-06-24 18:05:29 +03:00
Martijn van Groningen
ae3c3601fd
Remove docs warning that synthetic source is in es|ql is experimental. (#129930) 2025-06-24 16:59:55 +02:00
Benjamin Trent
4d8f9fa505
Add OptimizedScalarQuantizer to forbidden API checks (#129873)
we have various ES only optimizations within OptimizedScalarQuantizer,
we should ensure callers within ES are using the optimized code.
2025-06-25 00:20:36 +10:00
elasticsearchmachine
c3255ad779 Mute org.elasticsearch.index.mapper.DynamicMappingIT testDenseVectorDynamicMapping #129928 2025-06-24 23:21:35 +10:00
David Turner
8573865341
Remove dead @TestIssueLogging anns from repo-s3 (#129906)
These logging directives relate to test failures that have long-since
been closed so can be removed.

Relates #101608
Relates #88841
2025-06-24 14:04:16 +01:00
David Kyle
aaee222dfa
[ML] Add hint about using input_output in the inference processor to error message (#129864) 2025-06-24 21:45:18 +10:00
elasticsearchmachine
788e18f4d2 Mute org.elasticsearch.index.mapper.vectors.DenseVectorFieldMapperTests testExistsQueryMinimalMapping #129911 2025-06-24 20:56:04 +10:00
Slobodan Adamović
ee6d64afdf
[Test] Fix FileSettingsRoleMappingsRestartIT (#129618)
The #127318 changed the behaviour of `client()` to not start a node if there 
is none found in the cluster. Which also changed the `getMasterName()` 
behaviour to simply fail if there are no nodes instead of starting one.

This is why the `getMasterName()` is failing now. There were no nodes 
started because the test scope is set to manually manage master nodes 
(`autoManageMasterNodes = false`) without data nodes (`numDataNodes = 0`).

The fix is to actually start the master node instead of attempting to get 
the master node name from an empty cluster and depend on a side effect 
to actually boostrap a node.

Additionally it awaits for the master node to process all cluster state events 
before proceeding, which should hopefully solve the original cause of failures.

Resolves #120964
Resolves #120923
2025-06-24 12:37:47 +02:00
elasticsearchmachine
9cc034ba44
Add release notes for v9.0.3 release (#129803)
* Update docs for v9.0.3 release

* Add merge scheduler issue to known issues pages

* Add patch release information

---------

Co-authored-by: Charlotte Hoblik <116336412+charlotte-hoblik@users.noreply.github.com>
Co-authored-by: Charlotte Hoblik <sarolta@saroltah.hu>
2025-06-24 11:34:32 +01:00
Ievgen Degtiarenko
da6029fdce
Fix StatementParserTests (#129898)
This change fixes StatementParserTests.testPreserveParanthesis as it now depends on capability enabled only on snapshot builds
2025-06-24 11:28:19 +02:00
Panagiotis Bailis
b855266bd1
Make bbq_hnsw the default index option for dense-vector fields with more than 384 dimensions (#129825) 2025-06-24 12:20:16 +03:00
Charlotte Hoblik
1a0ab74323
[DOCS]: Add Vector tile search API examples (#129520)
* Add vector tile examples

* Add new page to TOC

* Add internal translation example

* Update docs/reference/elasticsearch/rest-apis/vector-tile-search.md

Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>

* Update docs/reference/elasticsearch/rest-apis/vector-tile-search.md

Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>

---------

Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
2025-06-24 09:30:16 +01:00
Charlotte Hoblik
fb30e59a38
Add Query API key information examples (#129719) 2025-06-24 10:13:04 +02:00
Liam Thompson
5aeadf277e
[DOCS] Remove planning verbiage (#129900) 2025-06-24 09:53:42 +02:00
Larisa Motova
c94bd1cf24
[ES|QL] Change rounding mode in tests to HALF_DOWN (#129895)
The current rounding mode in some of the tests in ESQL uses DOWN which
truncates more than rounds and can result in some tests failing with
errors like `row 0 column 0:0: expected <0.05235999> but was <0.05236>`
HALF_DOWN would round this up
2025-06-23 21:22:36 -10:00
Nhat Nguyen
d3049e03e1
Fix translation for two time-series aggregations (#129892)
This PR fixes the time-series translation for cases where two 
time-series aggregation functions are used within an outer aggregation
function, for example: STATS max(rate(r1) + rate(r2)).
2025-06-24 16:55:59 +10:00
Nik Everett
59a10bdd7b
ESQL: Add counters to signature for IS NULL (#129670)
This adds `counter_long` and `counter_double` to the signatures of
supported fields for `IS NULL` and `IS NOT NULL`. We hadn't been
generating those signatures since the docs v3 migration, so this had to
plug those in. In addition, it changes the wording on a few things and
adds a note that if a field is only in some documents then the ones
missing the field will have `NULL` - which is important information for
`IS NULL` and `IS NOT NULL`.
2025-06-24 01:18:15 +01:00
Niels Bauman
5ccb772468
Remove unused BulkProcessor (#129875)
The `BulkProcessor` and `BulkRequestHandler` classes were unused and
could thus be removed along with their test classes.
2025-06-24 10:02:02 +10:00
Brian Rothermich
0f39ff586c
Bring over merge metrics from stateless (#128617)
Relates to an effort to combine the merge schedulers from stateless and stateful. The stateless merge scheduler has MergeMetrics that we want in both stateless and stateful. This PR copies over the merge metrics from the stateless merge scheduler into the combined merge scheduler.

Relates ES-9687
2025-06-23 19:42:01 -04:00
Mark J. Hoy
a671505c8a
Update sparse_vector field mapping to include default setting for token pruning (#129089)
* Initial checkin of refactored index_options code

* [CI] Auto commit changes from spotless

* initial unit testing

* complete unit tests; add yaml tests

* [CI] Auto commit changes from spotless

* register test feature for sparse vector

* Update docs/changelog/129089.yaml

* update changelog

* add docs

* explicit set default index_options if null

* [CI] Auto commit changes from spotless

* update yaml tests; update docs

* fix yaml tests

* readd auth for teardown

* only serialize index options if not default

* [CI] Auto commit changes from spotless

* serialization refactor; pass index version around

* [CI] Auto commit changes from spotless

* fix transport versions merge

* fix up docs

* [CI] Auto commit changes from spotless

* fix docs; add include_defaults unit and yaml test

* [CI] Auto commit changes from spotless

* override getIndexReaderManager for SemanticQueryBuilderTests

* [CI] Auto commit changes from spotless

* cleanup mapper/builder/tests; index vers. in type

still need to refactor / clean YAML tests

* [CI] Auto commit changes from spotless

* cleanups to mapper tests for clarity

* [CI] Auto commit changes from spotless

* move feature into mappers; fix yaml tests

* cleanups; add comments; remove redundant test

* [CI] Auto commit changes from spotless

* escape more periods in the YAML tests

* cleanup mapper and type tests

* [CI] Auto commit changes from spotless

* rename mapping for previous index test

* set explicit number of shards for yaml test

---------

Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co>
2025-06-24 08:21:32 +10:00
elasticsearchmachine
a324853d43 Mute org.elasticsearch.xpack.test.rest.XPackRestIT test {p0=esql/60_usage/Basic ESQL usage output (telemetry) non-snapshot version} #129888 2025-06-24 08:15:18 +10:00
Pat Whelan
98e705dd2e
[Transform] Change reindex links (#129879)
Link directly to the Transform reindex migration guide.
2025-06-24 07:53:48 +10:00
Pat Whelan
aeb37189af
[ML] SageMaker Elastic Payload (#129413)
Send the Elastic API Payload to a SageMaker endpoint, and parse the
response as if it were an Elastic API response.

- SageMaker now supports all task types in the Elastic API format.
- Streaming is supported using the SageMaker client/server rpc,
  rather than SSE. Payloads must be in a complete and valid JSON
  structure.
- Task Settings can be used for additional passthrough settings, but
  they will not be saved alongside the model. Elastic cannot make
  guarantees on the structure or contents of this payload, so Elastic
  will treat it like the other input payloads and only allow them during
  inference.
2025-06-24 06:43:24 +10:00
Liam Thompson
8c06acccf8
[DOCS][ESQL] GA search functions for 9.1 (#129786)
* [DOCS][ESQL] Flip preview booleans, to GA search functions

* render docs, tweak some applies_to metadata in docs gen code

- **rendered docs (md):**
  - kql: removed serverless preview, added ga 9.1.0
  - match: removed serverless preview, added ga 9.1.0
  - match_phrase: changed from preview 9.1.0 to unavailable 9.0 + ga 9.1.0
  - qstr: removed serverless preview, added ga 9.1.0
  - search functions list: removed bullet point before term function

- **docs generation code (java):**
  - match_phrase: updated function info annotations to unavailable 9.0 + ga 9.1.0
  - query_string: uncommented ga 9.1.0 annotation
2025-06-23 20:51:54 +01:00
Ankit Sethi
22390d7817
Move docker image to registry to address flaky test (#129660)
* Instead of creating a new Docker image on-the-fly, it is better to create a persistent image and upload it to Docker Registry. This will help tests dependent on this image be resilient to transient issues such as #126694 and also speed up execution.

* fix name
2025-06-23 14:38:38 -05:00
Julian Kiryakov
caae426cf7
Pushdown for LIKE (LIST) (#129557)
Improved performance of LIKE (LIST)  by pushing an Automaton to do the evaluation down to Lucine.
2025-06-23 14:35:09 -04:00