* Add NamedWriteable for QueryRule rank doc
* Update test
* Update docs/changelog/128153.yaml
* Add multi cluster test for query rules
* Commenting out code - explicitly trying to spur a test failure
* [CI] Auto commit changes from spotless
* Streamline test for multi cluster
* Revert changes to try to break test
* Fix compile error
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
When the Transform System Index has been reindexed and aliased, we
should check the Transform Update index against the alias when updating
the Transform Config.
* Disallow removal of regex extracted fields
---------
Co-authored-by: Andrei Stefan <astefan@users.noreply.github.com>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Replaces the use of a SingleResultDeduplicator by refactoring the cache as a
subclass of CancellableSingleObjectCache. Refactored the AllocationStatsService
and NodeAllocationStatsAndWeightsCalculator to accept the Runnable used to test
for cancellation.
Closes#123248
Today Elasticsearch will record the purpose for each request to S3 using
a custom query parameter[^1]. This isn't believed to be necessary
outside of the ECH/ECE/ECK/... managed services, and it adds rather a
lot to the request logs, so with this commit we make the feature
optional and disabled by default.
[^1]:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html#LogFormatCustom
Currently, union types in CCS is broken. For example, FROM
*:remote-indices | EVAL port = TO_INT(port) returns all nulls if the
types of the port field conflict. This happens because converters are a
map of the fully qualified cluster:index -name (defined in
MultiTypeEsField), but we are looking up the converter using only the
index name, which leads to a wrong or missing converter on remote
clusters. Our tests didn't catch this because MultiClusterSpecIT
generates the same index for both clusters, allowing the local converter
to be used for remote indices.
No need to do this via an allocation-heavy `Stream`, we can just put the
objects straight into an array, sort them in-place, and keep hold of the
array to avoid having to allocate anything on the next iteration.
Also slims down `BY_DESCENDING_SHARD_ID`: it's always sorting the same
index so we don't need to look at `ShardId#index` in the comparison, nor
do we really need multiple layers of vtable lookups, we can just compare
the shard IDs directly.
Relates #128021
* Add Hugging Face Chat Completion support to Inference Plugin
* Add support for streaming chat completion task for HuggingFace
* [CI] Auto commit changes from spotless
* Add support for non-streaming completion task for HuggingFace
* Remove RequestManager for HF Chat Completion Task
* Refactored Hugging Face Completion Service Settings, removed Request Manager, added Unit Tests
* Refactored Hugging Face Action Creator, added Unit Tests
* Add Hugging Face Server Test
* [CI] Auto commit changes from spotless
* Removed parameters from media type for Chat Completion Request and unit tests
* Removed OpenAI default URL in HuggingFaceService's configuration, fixed formatting in InferenceGetServicesIT
* Refactor error message handling in HuggingFaceActionCreator and HuggingFaceService
* Update minimal supported version and add Hugging Face transport version constants
* Made modelId field optional in HuggingFaceChatCompletionModel, updated unit tests
* Removed max input tokens field from HuggingFaceChatCompletionServiceSettings, fixed unit tests
* Removed if statement checking TransportVersion for HuggingFaceChatCompletionServiceSettings constructor with StreamInput param
* Removed getFirst() method calls for backport compatibility
* Made HuggingFaceChatCompletionServiceSettingsTests extend AbstractBWCWireSerializationTestCase for future serialization testing
* Refactored tests to use stripWhitespace method for readability
* Refactored javadoc for HuggingFaceService
* Renamed HF chat completion TransportVersion constant names
* Added random string generation in unit test
* Refactored javadocs for HuggingFace requests
* Refactored tests to reduce duplication
* Added changelog file
* Add HuggingFaceChatCompletionResponseHandler and associated tests
* Refactor error handling in HuggingFaceServiceTests to standardize error response codes and types
* Refactor HuggingFace error handling to improve response structure and add streaming support
* Allowing null function name for hugging face models
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Jonathan Buttner <jonathan.buttner@elastic.co>
Painless does not support accessing nested docs (except through
_source). Yet the painless execute api indexes any nested docs that are
found when parsing the sample document. This commit changes the ram
indexing to only index the root document, ignoring any nested docs.
fixes#41004
This entitlement is required, but only if validating the metadata
endpoint against `https://login.microsoft.com/` which isn't something we
can do in a test. Kind of a SDK bug, we should be using an existing
event loop rather than spawning threads randomly like this.
The hash aggregation operator may take time to emit the output pages,
including keys and aggregated values. This change adds an emit_time
field to the status. While I considered including this in hash_nanos and
aggregation_nanos, having a separate section feels more natural. I am
open to suggestions.
Types that parse arrays directly should not need to store values in _ignored_source if synthetic_source_keep=arrays. Since they have custom handling of arrays, it provides no benefit to store in _ignored_source when there are multiple values of the type.
Fix a bug in the `significant_terms` agg where the "subsetSize" array is
too small because we never collect the ordinal for the agg "above" it.
This mostly hits when the you do a `range` agg containing a
`significant_terms` AND you only collect the first few ranges. `range`
isn't particularly popular, but `date_histogram` is super popular and it
rewrites into a `range` pretty commonly - so that's likely what's really
hitting this - a `date_histogram` followed by a `significant_text` where
the matches are all early in the date range held by the shard.
Currently, if a field has high cardinality, we may mistakenly disable
emitting ordinal blocks. For example, with 10,000 `tsid` values, we
never emit ordinal blocks during reads, even though we could emit blocks
for 10 `tsid` values across 1,000 positions. This bug disables
optimizations for value aggregation and block hashing.
This change tracks the minimum and maximum seen ordinals and uses them
as an estimate for the number of ordinals. However, if a page contains
`ord=1` and `ord=9999`, ordinal blocks still won't be emitted.
Allocating a bitset or an array for `value_count` could track this more
accurately but would require additional memory. I need to think about
this trade off more before opening another PR to fix this issue
completely.
This is a quick, contained fix that significantly speeds up time-series
aggregation (and other queries too).
The execution time of this query is reduced from 3.4s to 1.9s with 11M documents.
```
POST /_query
{
"profile": true,
"query": "TS metrics-hostmetricsreceiver.otel-default
| STATS cpu = avg(avg_over_time(`metrics.system.cpu.load_average.1m`)) BY host.name, BUCKET(@timestamp, 5 minute)"
}
```
```
"took": 3475,
"is_partial": false,
"documents_found": 11368089,
"values_loaded": 34248167
```
```
"took": 1965,
"is_partial": false,
"documents_found": 11368089,
"values_loaded": 34248167
```
9.x port of: Revert "Enable madvise by default for all builds (#110159)" #126308
This change did not apply cleanly. In fact this is not strictly a revert, since the change was never actually in 9.x post the Lucene 10 upgrade. However, the semantics of the change still apply - avoid RANDOM everywhere. Even though in 9.x we do set -Dorg.apache.lucene.store.defaultReadAdvice=normal, it is not enough to avoid RANDOM when random is explicitly requested by code.
Entitlements do a stack walk to find the calling class. When method
refences are used in a lambda, the frame ends up hidden in the stack
walk. In the case of using a method reference with
AccessController.doPrivileged, the call looks like it is the jdk itself,
so the call is trivially allowed. This commit adds hidden frames to the
stack walk so that the lambda frame created for the method reference is
included. Several internal packages are then necessary to filter out of
the stack.
Reenables `text ==` pushdown and adds support for `text !=` pushdown.
It does so by making `TranslationAware#translatable` return something
we can turn into a tri-valued function. It has these values:
* `YES`
* `NO`
* `RECHECK`
`YES` means the `Expression` is entirely pushable into Lucene. They will
be pushed into Lucene and removed from the plan.
`NO` means the `Expression` can't be pushed to Lucene at all and will stay
in the plan.
`RECHECK` mean the `Expression` can push a query that makes *candidate*
matches but must be rechecked. Documents that don't match the query won't
match the expression, but documents that match the query might not match
the expression. These are pushed to Lucene *and* left in the plan.
This is required because `txt != "b"` can build a *candidate* query
against the `txt.keyword` subfield but it can't be sure of the match
without loading the `_source` - which we do in the compute engine.
I haven't plugged rally into this, but here's some basic
performance tests:
```
Before:
not text eq {"took":460,"documents_found":1000000}
text eq {"took":432,"documents_found":1000000}
After:
text eq {"took":5,"documents_found":1}
not text eq {"took":351,"documents_found":800000}
```
This comes from:
```
rm -f /tmp/bulk*
for a in {1..1000}; do
echo '{"index":{}}' >> /tmp/bulk
echo '{"text":"text '$(printf $(($a % 5)))'"}' >> /tmp/bulk
done
ls -l /tmp/bulk*
passwd="redacted"
curl -sk -uelastic:$passwd -HContent-Type:application/json -XDELETE https://localhost:9200/test
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPUT https://localhost:9200/test -d'{
"settings": {
"index.codec": "best_compression",
"index.refresh_interval": -1
},
"mappings": {
"properties": {
"many": {
"enabled": false
}
}
}
}'
for a in {1..1000}; do
printf %04d: $a
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST https://localhost:9200/test/_bulk?pretty --data-binary @/tmp/bulk | grep errors
done
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST https://localhost:9200/test/_forcemerge?max_num_segments=1
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST https://localhost:9200/test/_refresh
echo
curl -sk -uelastic:$passwd https://localhost:9200/_cat/indices?v
text_eq() {
echo -n " text eq "
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST 'https://localhost:9200/_query?pretty' -d'{
"query": "FROM test | WHERE text == \"text 1\" | STATS COUNT(*)",
"pragma": {
"data_partitioning": "shard"
}
}' | jq -c '{took, documents_found}'
}
not_text_eq() {
echo -n "not text eq "
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST 'https://localhost:9200/_query?pretty' -d'{
"query": "FROM test | WHERE NOT text == \"text 1\" | STATS COUNT(*)",
"pragma": {
"data_partitioning": "shard"
}
}' | jq -c '{took, documents_found}'
}
for a in {1..100}; do
text_eq
not_text_eq
done
```
Initial Kibana definition files for commands, currently only providing License information. We leave the license field out if it works with BASIC, so the only two files that actually have a license line are:
* CHANGE_POINT: PLATINUM
* RRF: ENTERPRISE
Re-applying #126441 (cf. #127259) with:
- the extra `FlowControlHandler` needed to ensure one-chunk-per-read
semantics (also present in #127259).
- no extra `read()` after exhausting a `Netty4HttpRequestBodyStream`
(the bug behind #127391 and #127391).
See #127111 for related tests.
During reindexing we retrieve the index mode from the template settings. However, we do not fully resolve the settings as we do when validating a template or when creating a data stream. This results on throwing the error reported in #125607.
I do not see a reason to not fix this as suggested in #125607 (comment).
Fixes: #125607
When downsampling an index that has a mapping with passthrough dimensions the downsampling process identifies the wrapper object as a dimension and it fails when it tried to retrieve the type.
We did some prework to establish a shared framework in the internalClusterTest. For now it only includes setting up time series data stream helpers and a limited assertion helper for dimensions and metrics. This allows us to setup an internalClusterTest that captures this issue during downsampling in #125156.
To fix this we refine the check that determines if a field is dimension, to skip wrapper field.
Fixes#125156.
Currently, time-series aggregations use the `values` aggregation to
collect dimension values. While we might introduce a specialized
aggregation for this in the future, for now, we are using `values`, and
the inputs are likely ordinal blocks. This change speeds up the `values`
aggregation when the inputs are ordinal-based.
Execution time reduced from 461ms to 192ms for 1000 groups.
```
ValuesAggregatorBenchmark.run BytesRef 10000 avgt 7 461.938 ± 6.089 ms/op
ValuesAggregatorBenchmark.run BytesRef 10000 avgt 7 192.898 ± 1.781 ms/op
```
The audit event for a successfully-authenticated REST request occurs
when we start to process the request. For APIs that accept a streaming
request body this means we have received the request headers, but not
its body, at the time of the audit event. Today such requests will fail
with a `ClassCastException` if the `emit_request_body` flag is set. This
change fixes the handling of streaming requests in the audit log to now
report that the request body was not available when writing the audit
entry.
* Specialize block parameters on AddInput
(cherry picked from commit a5855c1664)
* Call the specific add() methods for eacj block type
(cherry picked from commit 5176663f43)
* Implement custom add in HashAggregationOperator
(cherry picked from commit fb670bdbbc)
* Migrated everything to the new add() calls
* Update docs/changelog/127582.yaml
* Spotless format
* Remove unused ClassName for IntVectorBlock
* Fixed tests
* Randomize groupIds block types to check most AddInput cases
* Minor fix and added some docs
* Renamed BlockHashWrapper
The `s3.client.CLIENT_NAME.protocol` setting became unused in #126843 as
it is inapplicable in the v2 SDK. However, the v2 SDK requires the
`s3.client.CLIENT_NAME.endpoint` setting to be a URL that includes a
scheme, so in #127489 we prepend a `https://` to the endpoint if needed.
This commit generalizes this slightly so that we prepend `http://` if
the endpoint has no scheme and the `.protocol` setting is set to `http`.
Output function signature license requirements to Kibana definition files, and also test that this matches the actual licensing behaviour of the functions.
ES|QL functions that enforce license checks do so with the `LicenseAware` interface. This does not expose what that functions license level is, but only whether the current active license will be sufficient for that function and its current signature (data types passed in as fields). Rather than add to this interface, we've made the license level information test-only information. This means if a function implements LicenseAware, it also needs to add a method to its test class to specify the license level for the signature being called. All functions will be tested for compliance, so failing to add this will result in test failure. Also if the test license level does not match the enforced license, that will also cause a failure.
