* Specific idx in testMultipleBatchesWithLookupJoin
Do not use `from *` in the test, be more specifc - otherwise other tests
can affect the output if they leave indices behind, affecting the column
count.
* Remove column count validation
That doesn't really tell us much in this test, anyway.
Decreases the periodicity (from 5s to 100ms), for some tests, with which merge tasks are checked if there's sufficient available disk space to execute or if they've been aborted since enqueued for execution.
Fixes#130044
Prepares the `main` branch for the backport of #125631. Specifically,
this adds the version constant for 8.19 to main and the serialization
code that lets main talk to 8.19.
There was a bug in the code for deleting unused and orphan ML data. When deletion using DBQ occurred, the bug caused the request to time out. This PR resolves the issue.
Initial version of patterned_text mapper. Behaves similarly to match_only_text. This version uses a single SortedSetDocValues for a template and another for arguments. It splits the message by delimiters, the classifies a token as an argument if it contains a digit. All arguments are concatenated and inserted as a single doc value. A single inverted index is used, without positions. Phrase queries are still possible, using the SourceConfirmedTextQuery, but are not fast.
Keep better track of shard contexts using RefCounted, so they can be released more aggressively during operator processing. For example, during TopN, we can potentially release some contexts if they don't pass the limit filter.
This is done in preparation of TopN fetch optimization, which will delay the fetching of additional columns to the data node coordinator, instead of doing it in each individual worker, thereby reducing IO. Since the node coordinator would need to maintain the shard contexts for a potentially longer duration, it is important we try to release what we can eariler.
An even more advanced optimization is to delay fetching to the main cluster coordinator, but that would be more involved, since we need to first figure out how to transport the shard contexts between nodes.
Summary of main changes:
DocVector now maintains a RefCounted instance per shard.
Things which can build or release DocVectors (e.g., LuceneSourceOperator, TopNOperator), can also hold RefCounted instances, so they can pass them to DocVector and also ensure contexts aren't released if they can still be potentially used later.
Driver's main loop iteration (runSingleLoopIteration), now closes its operators even between different operator processing. This is extra aggressive, and was mostly done to improve testability.
Added a couple of tests to TopNOperator and a new integration test EsqlTopNShardManagementIT, which uses the pausable plugin framework to check that TopNOperator releases things as early as possible..
The local plan optimizer should not change the layout, as it has already
been agreed upon. However, CombineProjections can violate this when some
grouping elements refer to the same attribute. This occurs when
ReplaceFieldWithConstantOrNull replaces missing fields with the same
reference for a given data type.
Closes#128054Closes#129811
ES|QL index patterns validation: Ensure that the patterns in the query are syntactically and semantically valid
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Andrei Stefan <astefan@users.noreply.github.com>
Co-authored-by: Alexander Spies <alexander.spies@elastic.co>
Before a `KeywordFieldType` was created that didn't set the isSyntheticSource field, causing to use the wrong block loader that would synthesize the complete _source instead of just loading values from ignored source. This PR addresses this.
With the introduction of entitlements (#120243) and exclusive file
access (#123087) it is no longer safe to watch a whole directory.
In a lot of deployments, the parent directory for SSL config files
will be the main config directory, which also contains exclusive files
such as SAML realm metadata or File realm users. Watching that
directory will cause entitlement warnings because it is not
permissible for core/ssl-config to read files that are exclusively
owned by the security module (or other modules)
This PR makes RepositoriesService project aware so that the basic Put,
Get, Delete and Verify repository actions are now project scoped.
It intentionally leaves the following aspects out of scope for the
current changes: * Repository stats reporting * Repository clean-up,
analysis and integrity verification * Repository usages for searchable
snapshots and CCR
They will be worked on separately. One main reason for leaving them out
is that they are not needed by OBS which is currently blocked by
repository/snapshot changes. They may also have their own complexities,
e.g. stats reporting.
Resolves: ES-10478
Introduces a new `RemoveBlock` API that complements the existing `AddBlock` API by allowing users to remove index blocks using `DELETE /{index}/_block/{block}`.
Resolves#128966
---------
Co-authored-by: Niels Bauman <nielsbauman@gmail.com>
The #127318 changed the behaviour of `client()` to not start a node if there
is none found in the cluster. Which also changed the `getMasterName()`
behaviour to simply fail if there are no nodes instead of starting one.
This is why the `getMasterName()` is failing now. There were no nodes
started because the test scope is set to manually manage master nodes
(`autoManageMasterNodes = false`) without data nodes (`numDataNodes = 0`).
The fix is to actually start the master node instead of attempting to get
the master node name from an empty cluster and depend on a side effect
to actually boostrap a node.
Additionally it awaits for the master node to process all cluster state events
before proceeding, which should hopefully solve the original cause of failures.
Resolves#120964Resolves#120923
The current rounding mode in some of the tests in ESQL uses DOWN which
truncates more than rounds and can result in some tests failing with
errors like `row 0 column 0:0: expected <0.05235999> but was <0.05236>`
HALF_DOWN would round this up
This PR fixes the time-series translation for cases where two
time-series aggregation functions are used within an outer aggregation
function, for example: STATS max(rate(r1) + rate(r2)).
This adds `counter_long` and `counter_double` to the signatures of
supported fields for `IS NULL` and `IS NOT NULL`. We hadn't been
generating those signatures since the docs v3 migration, so this had to
plug those in. In addition, it changes the wording on a few things and
adds a note that if a field is only in some documents then the ones
missing the field will have `NULL` - which is important information for
`IS NULL` and `IS NOT NULL`.
Relates to an effort to combine the merge schedulers from stateless and stateful. The stateless merge scheduler has MergeMetrics that we want in both stateless and stateful. This PR copies over the merge metrics from the stateless merge scheduler into the combined merge scheduler.
Relates ES-9687
* Initial checkin of refactored index_options code
* [CI] Auto commit changes from spotless
* initial unit testing
* complete unit tests; add yaml tests
* [CI] Auto commit changes from spotless
* register test feature for sparse vector
* Update docs/changelog/129089.yaml
* update changelog
* add docs
* explicit set default index_options if null
* [CI] Auto commit changes from spotless
* update yaml tests; update docs
* fix yaml tests
* readd auth for teardown
* only serialize index options if not default
* [CI] Auto commit changes from spotless
* serialization refactor; pass index version around
* [CI] Auto commit changes from spotless
* fix transport versions merge
* fix up docs
* [CI] Auto commit changes from spotless
* fix docs; add include_defaults unit and yaml test
* [CI] Auto commit changes from spotless
* override getIndexReaderManager for SemanticQueryBuilderTests
* [CI] Auto commit changes from spotless
* cleanup mapper/builder/tests; index vers. in type
still need to refactor / clean YAML tests
* [CI] Auto commit changes from spotless
* cleanups to mapper tests for clarity
* [CI] Auto commit changes from spotless
* move feature into mappers; fix yaml tests
* cleanups; add comments; remove redundant test
* [CI] Auto commit changes from spotless
* escape more periods in the YAML tests
* cleanup mapper and type tests
* [CI] Auto commit changes from spotless
* rename mapping for previous index test
* set explicit number of shards for yaml test
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co>
Send the Elastic API Payload to a SageMaker endpoint, and parse the
response as if it were an Elastic API response.
- SageMaker now supports all task types in the Elastic API format.
- Streaming is supported using the SageMaker client/server rpc,
rather than SSE. Payloads must be in a complete and valid JSON
structure.
- Task Settings can be used for additional passthrough settings, but
they will not be saved alongside the model. Elastic cannot make
guarantees on the structure or contents of this payload, so Elastic
will treat it like the other input payloads and only allow them during
inference.
* [DOCS][ESQL] Flip preview booleans, to GA search functions
* render docs, tweak some applies_to metadata in docs gen code
- **rendered docs (md):**
- kql: removed serverless preview, added ga 9.1.0
- match: removed serverless preview, added ga 9.1.0
- match_phrase: changed from preview 9.1.0 to unavailable 9.0 + ga 9.1.0
- qstr: removed serverless preview, added ga 9.1.0
- search functions list: removed bullet point before term function
- **docs generation code (java):**
- match_phrase: updated function info annotations to unavailable 9.0 + ga 9.1.0
- query_string: uncommented ga 9.1.0 annotation
* Instead of creating a new Docker image on-the-fly, it is better to create a persistent image and upload it to Docker Registry. This will help tests dependent on this image be resilient to transient issues such as #126694 and also speed up execution.
* fix name
- Add PREVIEW annotations to all preview functions
- Update docs generation logic to use annotations instead of preview boolean
* Changed stack: ga 9.1 to stack: coming in multiple places
- Match.java: Updated the options parameter description
- MultiMatch.java: Updated the options parameter description
- ToLower.java: Reformatted parameter description and updated version annotation
- ToUpper.java: Removed the appliesTo annotation entirely and reformatted parameter description
- updated applies_to annotations to specify both preview 9.0.0 and ga 9.1.0 lifecycle stages
- added version-specific documentation examples with applies_to markers for ga 9.1.0 features
- enhanced to_lower and to_upper functions to indicate support for multi-valued expressions in ga 9.1.0
- added version guards around function parameters and descriptions using applies_to syntax
- updated function parameter descriptions to indicate ga 9.1.0 availability for named parameters
- use detailedDescription + and fix match_phrase applies_to syntax
- strip inline applies_to from kibana docs
- update roundto, matchphrase lifecycles
- fix match named params info
- various spatial functions are preview
- unsigned long is preview
- update qstr
- Update TO_LOWER/TO_UPPER parameter descriptions for clarity
- hide spatial functions per https://github.com/elastic/elasticsearch/pull/129839
- added `stack: ga 9.1.0` blocks to match_phrase.md and qstr.md examples
- simplified term.md version from `preview 9.0.0` to just `preview`
- added `applies_to = "stack: ga 9.1.0"` to matchphrase and querystring java annotations
- removed version `9.0.0` from term function annotation
- deleted unused `makeCallout()` and `appendLifeCycleAndVersion()` methods
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Liam Thompson <leemthompo@gmail.com>
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Co-authored-by: Nik Everett <nik9000@gmail.com>
The vast majority of ingest pipelines are light CPU
operations. We don't want these to be put behind IO work on the write
executor. Instead, execute these on the coordination pool.
#125143 added 9 spatial grid functions and released them into Serverless. We think this is not the best long-term approach and the functions in #129581 are likely better.
As a first step, rmove the spatial grid functions added in #125143 from release builds so they don't get released into 8.19/9.1.
---------
Co-authored-by: Craig Taverner <craig@amanzi.com>
