Commit graph

14959 commits

Author SHA1 Message Date
Brendan Cully
b6047b2b32
Set SlowLog logging to TRACE in tests (#114344) (#114349)
The tests depend on the SlowLog loggers running at TRACE level
but were not setting the level themselves. Instead they relied
on the SlowLog setting the level to trace internally when it
was created. If something else globally adjusted log levels
between the time the SlowLog loggers were created and the tests
ran, the tests could fail.

And in fact, `ScopedSettingsTest.testFallbackToLoggerLevel` was
updating the root log level, which had the side effect of updating
the SlowLog level. In #112183 SlowLog's log initialization was made
static, which opened up its test to failure when ScopedSettingsTest
ran before a SlowLog test in the same JVM.

I do not know if the intention of the SlowLog is that it overrides
the global log level and should always be set at TRACE, in which case
this fix is incorrect. It seems surprising, but I don't know why else
SlowLog would explicitly initialize itself to TRACE. However, if that
was the intention, the code was already at risk due to having no
guard against being changed by Loggers.setLevel on an ancestor log.
The change in this PR is at least not a regression in that
behaviour. It does no longer start out at TRACE however, which is
a change in behaviour.
2024-10-09 07:11:36 +11:00
Simon Cooper
c1115d24d7
Add a size limit to outputs from mustache (#114002) (#114297)
Backport #114002 to 8.16
2024-10-08 12:47:08 +01:00
Panagiotis Bailis
e745c92ebc
Backporting text_similarity_reranker retriever rework to be evaluated during rewrite phase to 8.x (#114282)
* backporting text_similarity_reranker rework to 8.x

* fixing comp

---------

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2024-10-08 21:44:00 +11:00
Christoph Büscher
029c6836d9
Remove dangling AwaitsFix (#113941) (#114286)
The issue referenced seems to be closed, maybe this just wasn't removed
properly.
2024-10-08 20:21:09 +11:00
David Turner
3bfdc8943c
Show only committed cluster UUID in GET / (#114275) (#114281)
Today we show `Metadata#clusterUUID` in the response to `GET /`
regardless of whether this value is committed or not, which means that
in theory users may see this value change even if nothing is going
wrong. To avoid any doubt about the stability of this cluster UUID, this
commit suppresses the cluster UUID in this API response until it is
committed.
2024-10-08 19:48:00 +11:00
Luca Cavanna
74f523d4cc
Span term query to convert to match no docs when unmapped field is targeted (#113251)
SpanTermQueryBuilder currently creates a valid SpanTermQuery against unmapped fields.
In practice, if the field is unmapped, there won't be a match. This commit changes
the toQuery impl to return a MatchNoDocsQuery instead like we do in similar scenarios.
2024-10-08 08:48:53 +02:00
Panagiotis Bailis
309d234bf0
Updating toXContent implementation for retrievers (#114017) (#114272) 2024-10-08 17:33:30 +11:00
Dan Rubinstein
183aa71e76
[8.x] Adding chunking settings to MistralService, GoogleAiStudioService, and HuggingFaceService (#113623) (#113886)
* Adding chunking settings to MistralService, GoogleAiStudioService, and HuggingFaceService (#113623)

* Adding chunking settings to MistralService, GoogleAiStudioService, and HuggingFaceService

* Update docs/changelog/113623.yaml

* Removing chunking settings from HuggingFaceElser model inputs

* Removing unnecessary platfromArchitectures from InferenceService tests (#113890)

* Removing unnecessary platfromArchitectures from InferenceService tests

* Update docs/changelog/113890.yaml

* Delete docs/changelog/113890.yaml

* Fix HuggingFaceMixedIT test sometimes failing when run on version before 8.16 (#114061)

* Fix HuggingFaceMixedIT test sometimes failing when run on version before 8.16

* Fixing typo in expected error message

---------

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2024-10-07 17:09:28 -04:00
Mike Pellegrini
5b8f9c12d2
[ML] Default inference endpoint for ELSER (#114164)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2024-10-08 06:58:10 +11:00
Iraklis Psaroudakis
2356006e50
Fast refresh indices should use search shards (#113478) (#114259)
Fast refresh indices should now behave like non fast refresh indices in
how they execute (m)gets and searches. I.e., they should use the search
shards.

For BWC, we define a new transport version. We expect search shards to
be upgraded first, before promotable shards. Until the cluster is fully
upgraded, the promotable shards (whether upgraded or not) will still
receive and execute gets/searches locally.

Relates ES-9573
Relates ES-9579
2024-10-08 06:22:11 +11:00
Tim Brooks
46b069621e
Fix issue with releasing resources in bulk tests (#114186) (#114258)
A recent commit incidentally changed a release resources call from
doBefore to doAfter. Several tests depending on resources being released
synchronously which requires doBefore.

Closes #114181
Closes #114182
2024-10-07 12:47:58 -06:00
Panagiotis Bailis
fc55a674f6
backporting rrf related test updates (#114257) 2024-10-08 05:32:34 +11:00
Max Hniebergall
4dadb23fd3
[Inference API] Backport deprecate elser service (#114052)
* [Inference API] Deprecate elser service

* Update docs/changelog/114052.yaml

* Update pr number 113216.yaml

* Delete docs/changelog/113216.yaml

* Update 114052.yaml

* Delete docs/changelog/114052.yaml

---------

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2024-10-08 05:02:08 +11:00
Benjamin Trent
20a34f43b5
Fixing span gap builder tests (#114218) (#114219) (#114253)
With #113251 having a SpanMatchNoDocsQuery is a valid response to the rewrite.

closes #114218

(cherry picked from commit a83046a85a)
2024-10-08 04:04:14 +11:00
Oleksandr Kolomiiets
eeec634cff
Correctly inject subobjects parameter in logsdb tests (#113643) (#114178) 2024-10-07 09:52:42 -07:00
Ignacio Vera
8716bf9c49
Add getAndSet to Objectarray (#114200) (#114226)
This commit adds a getAndSet implementation to the ObjectArray API and changes the set method to return void.
2024-10-08 02:46:02 +11:00
Ignacio Vera
500efc36c7
Improve performance of LongLongHash#removeAndAdd (#114230) (#114243)
remove some unnecessary manipulation of the keys in the method removeAndAdd.
2024-10-08 02:25:50 +11:00
Simon Cooper
1b01548425
[8.16] Change default locale of date mappers to ENGLISH (#112799) (#114210)
Backport #112799 to 8.16, for CLDR locale compatibility
2024-10-07 15:51:36 +01:00
Ignacio Vera
a1757ec3f5
Improve performance of LongHash#removeAndAdd (#114199) (#114225)
Remove unnecessary remove and later add of the key.
2024-10-08 01:20:47 +11:00
Luca Cavanna
7e154ee50d
Replace some test usages of search(Query, Collector) (#113818) (#114212)
The leftover usages of the deprecated  search(Query, TotalHitCountCollector)
have been replaced with search(Query, TotalHitCountCollectorManager)
2024-10-07 22:03:11 +11:00
Iván Cea Fontenla
638dea9be1
Fix MemoryTracking...ArraysTests bulkSet errors (#113934) (#113936)
Backports the following commits to 8.x:
 - Fix MemoryTracking...ArraysTests bulkSet errors (#113934)
2024-10-07 11:39:49 +02:00
Tim Brooks
52a4ab75a2 Fix high water mark test which submits data early (#114159)
It is possible in the incremental high watermark test that the data is
submitted causing a corruption of the bulk request. This commit fixes
the issue to ensure we only send new data after it has been requested.
Additionally, it adds an assertion to prevent this error from happening
again.
2024-10-04 15:53:26 -05:00
Tim Brooks
c4698c625b Introduce watermarks for indexing pressure backoff (#113912)
Currently we have a relatively basic decider about when to throttling
indexing. This commit adds two levels of watermarks with configurable
bulk size deciders. Additionally, adds additional settings to control
primary, coordinating, and replica rejection limits.
2024-10-04 15:46:17 -05:00
Drew Tate
2046b497fc
[ES|QL] add reverse function (#113297) (#114163)
Adds a REVERSE string function
2024-10-05 05:24:05 +10:00
Nhat Nguyen
46e6420c58
Add query fetch and indexing metrics per index mode (#113978) (#114140)
This change exposes query, fetch and indexing metrics
for each index mode.
2024-10-05 01:05:49 +10:00
Panagiotis Bailis
318188dde8
Add PIT information to collapse requests (#114133) 2024-10-04 17:48:43 +03:00
Ignacio Vera
8eec8d9662
Don't validate internal stats if they are empty (#113846) (#114136)
Fixes a validation step that might prevent creating empty aggregations iif the output format does not 
allows negative numbers.
2024-10-05 00:46:29 +10:00
Keith Massey
7d1239857c
Simplifying the way the simulate ingest API uses component template substitutions for pipeline lookup and mapping validation (#113908) (#114126) 2024-10-05 00:15:59 +10:00
David Turner
5a5b14dde6
[8.x] Assert that REST params are consumed iff supported (#114040) (#114087)
* Assert that REST params are consumed iff supported (#114040)

REST APIs which declare their supported parameters must consume exactly
those parameters: consuming an unsupported parameter means that requests
including that parameter will be rejected, whereas failing to consume a
supported parameter means that this parameter has no effect and should
be removed.

This commit adds an assertion to verify that we are consuming the
correct parameters.

Closes #113854

* CI poke
2024-10-04 23:32:25 +10:00
Panagiotis Bailis
c5d8e96bd7
Backporting 114084 to 8.x branch (#114106) 2024-10-04 19:18:42 +10:00
Panagiotis Bailis
cdfbe5d57f
Updating expected values in RankDocRetrieverBuilderIT testRankDocsRetrieverWithNestedQuery (#114047) (#114104) 2024-10-04 19:13:11 +10:00
Kostas Krikellas
cfbe148d90
Use second pass for deeply nested array elements (#114060) (#114068)
Deeply nested array elements with ignored source need to use the second
parsing pass, to avoid missing the source from siblings that go through
stored source.

(cherry picked from commit 5531e5dcd2)
2024-10-04 08:00:10 +03:00
Keith Massey
0b224dcef4
[8.x] Listing all available databases in the _ingest/geoip/database API (#113498) (#114081)
* Listing all available databases in the _ingest/geoip/database API (#113498)

* yamlRestCompatTestTransform renamed to yamlRestTestV7CompatTransform
2024-10-04 09:40:39 +10:00
Oleksandr Kolomiiets
4da7a0f338
Do not expand dots when storing objects in ignored source (#113910) (#114071)
(cherry picked from commit 7facc94be6)

# Conflicts:
#	rest-api-spec/build.gradle
2024-10-03 15:30:38 -07:00
Kostas Krikellas
faaf4ba7fd
[8.x] Add object param for keeping synthetic source (#113690) (#114058)
* Add object param for keeping synthetic source (#113690)

* Add object param for keeping synthetic source

* Update docs/changelog/113690.yaml

* fix merging

* add tests

* merge

* fix randomized tests

* add documentation

* dedup id in docs

* update documentation

* update documentation

* fix bwc

* fix bwc

* fix unintended

* Revert "fix bwc"

This reverts commit 18dc913eee.

* Revert "fix bwc"

This reverts commit f4ddb0e5e5.

* add missing test

* fix transform

* fix transform

* fix transform

* fix transform

* fix transform

(cherry picked from commit dd2024881d)

# Conflicts:
#	rest-api-spec/build.gradle

* Update build.gradle

* Update MapperFeatures.java

* Update 20_synthetic_source.yml

* Update 21_synthetic_source_stored.yml

* Update 21_synthetic_source_stored.yml

* Update 21_synthetic_source_stored.yml

* Update 21_synthetic_source_stored.yml
2024-10-04 08:08:44 +10:00
Athena Brown
e3a040ac47
Ensure all Security configuration is covered by enhanced file protections (#112408) (#114054)
This commit the work started in #108767 to other security config, adding additional protections to the other on-disk files used by the Security module. With this change, non-Security modules will be prevented from accessing Security's on-disk configuration files.

(cherry picked from commit 8fed8e63b0)
2024-10-04 04:50:27 +10:00
David Turner
b3ef2e6dbb
Migrate ReferenceDocs resource to plain text (#113952)
Removes the dependency on `XContent` parsing so we can move this out of
`:server` and into `:libs:core`.

Backport of #113866 to `8.x`
2024-10-04 04:44:29 +10:00
Nhat Nguyen
eb2763ee16
Track search and fetch failure stats (#113988) (#114041)
This PR tracks the total number of query and fetch failures, in addition
to the existing metrics for each shard, and exposes them through the
stats API.
2024-10-04 03:15:04 +10:00
Martijn van Groningen
d3d1a5f928
Change synthetic_source_keep setting's default (#113957) (#114039)
Incase of logsdb, the `index.mapping.synthetic_source_keep` setting should default to `arrays`. Otherwise it should default to `none`.
2024-10-04 02:48:37 +10:00
Pat Whelan
744f75d193
[ML] Enable OpenAI Streaming (#113911) (#114033)
Enables end-to-end streaming for OpenAI's chat completion API.

Additional changes: - InferenceServiceResults now returns a wildcard,
allowing for tests to work with classes that extend ChunkedToXContent. -
StreamingChatCompletionResults now defines the Results structure so that
future providers can reuse the same structure. - DelegatingProcessor now
cancels the upstream if `next` throws an exception and forwards the
exception downstream. - Moved the streaming handler from
OpenAiChatCompletionResponseHandler to OpenAiResponseHandler so that
Azure Open AI can reuse it. - OpenAiStreamingProcessor now iterates over
the returned choices array, handling both OpenAI and Azure response
formats. - SenderService declares a helper `Set<TaskType>` for
implementations to reuse when enabling streaming. - Added an
InferenceEventsAssertion test helper to fluidly assert responses from
the mock webserver.
2024-10-04 02:13:11 +10:00
David Kyle
5d34bcc502
[8.16][ML] Move code specific to the Elasticsearch in cluster services to those services (#113943)
* [ML] Move code specific to the Elasticsearch in cluster services to those sevices (#113749)

Remove the platform arch argument from parseRequest and move code
used by internal services out of the transport action into the service.

# Conflicts:
#	muted-tests.yml

* Bugfix by moving onResponse call into the new action listener; also add new test to catch this bug (#113898)

---------

Co-authored-by: Max Hniebergall <137079448+maxhniebergall@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2024-10-04 02:06:24 +10:00
Panagiotis Bailis
f08a8f505f
Rework RRF to be evaluated during rewrite phase (#112648) (#114000) 2024-10-03 18:57:56 +03:00
Iván Cea Fontenla
298984048b
[8.x] Add CircuitBreaker to TDigest, Step 4: Take into account shallow classes size (#114028)
* Add CircuitBreaker to TDigest, Step 4: Take into account shallow classes size (#113613)

* Removed muted tests from merge conflict

* Added missing empty line in muted tests
2024-10-04 01:34:22 +10:00
David Turner
b7729408b8
Weaken restoreFileFromSnapshot assertion (#113768) (#114005)
It's possible to call `RecoveryTarget#restoreFileFromSnapshot` after the
`RecoveryTarget` has been replaced with a new instance due to a retry,
but before all its refs have been released. If the recovery was
snapshot-based then in this situation we will have already transferred
the permits to the new instance, so the assertion that this instance has
permits will trip. This commit fixes the problem with a non-null
placeholder value indicating that the recovery target was at least
originally created while holding some snapshot recovery permits.

Closes #96018
2024-10-04 00:57:24 +10:00
David Turner
6090f464c5
Remove onItemCompletion param from ThrottledIterator (#113919) (#113935)
This param is a no-op for all production callers but one, and it's easy
to adapt the one exception to its removal, so it's effectively useless.
This commit removes it.
2024-10-04 00:53:57 +10:00
David Turner
ee2d35e2cf
Dispatch TransportSingleShardAction response handling (#113630) (#113930)
We shouldn't be handling these responses on the transport worker.

Closes #110408
2024-10-04 00:52:38 +10:00
Benjamin Trent
f057ff72c4
Add more dense_vector details for cluster stats field stats (#113607) (#113884)
This adds some more counts for dense_vector field mapping stats. This
allows for seeing the number of mappings with a given element type,
similarity, or index type.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2024-10-03 23:16:22 +10:00
David Turner
817bc901dc
Handle rejections in IncrementalBulkIT (#113940)
Submitting a bare runnable to a threadpool risks an exception being
thrown if the queue is full. This commit moves to submitting
`AbstractRunnable` instances that won't be rejected.

Closes #113365
Backport of #113599 to `8.x`
2024-10-03 20:55:22 +10:00
David Turner
f1113f5712
Ensure that supported params are computed once (#113980)
* Ensure that supported params are computed once

A few of today's REST handler implementations compute a new set of
supported parameters on each request. This is needlessly inefficient
since the set never changes. This commit fixes those implementations and
adds assertions to verify that we are returning the exact same instance
each time.

Backport of #113953 to `8.x`

* Fix
2024-10-03 20:54:38 +10:00
Kostas Krikellas
e5bebfe400
Remove special handling for objects and arrays with dynamic overrides (#113924) (#113959)
* Remove special handling for objects and arrays with `dynamic` overrides

* add test

* add test

(cherry picked from commit c840ea3812)
2024-10-02 19:28:01 +03:00