Transport actions have associated request and response classes. However,
the base type restrictions are not necessary to duplicate when creating
a map of transport actions. Relatedly, the ActionHandler class doesn't
actually need strongly typed action type and classes since they are lost
when shoved into the node client map. This commit removes these type
restrictions and generic parameters.
When no explicit aliases are provided in the call there is no need
to collect the index names or aliases into HashSets if they won't be
used. Also fixed where the index name was being added for each
loop of the alias list.
This base class is kinda pointless: everywhere it's used we can either
be more specific (e.g. choosing between `TransportRequest` or
`TransportResponse`) or more general (e.g. choosing `Writeable`). This
commit removes all the usages apart from the `extends` clauses of its
direct descendants.
Today `ActionResponse$Empty` implements `ToXContentObject`, but yields
no bytes of content when serialized which creates an invalid JSON
response. This commit removes the bogus interface and adjusts the
affected REST APIs to send a `text/plain` response instead.
The `InvalidIndexNameException` exception was wrapped in a `ElasticsearchSecurityException`, which returns HTTP `403` status.
This exception (along with newly introduced `InvalidSelectorException` and `UnsupportedSelectorException`) can be raised during index expression resolving due to an invalid user input and should result in HTTP `400` response instead.
This PR changes exception handling to avoid wrapping them in the `ElasticsearchSecurityException`.
To backport the newly introduced AggregateMetricDoubleLiteral to 8.x, we need
to override the supportsVersion method instead of getMinimalSupportedVersion.
Update the PerFieldFormatSupplier so that new standard indices use the
Lucene101PostingsFormat instead of the current default ES812PostingsFormat.
Currently, use of the new codec is gated behind a feature flag.
This enhances DataStreamAutoShardingService so that it periodically
logs at INFO level the most 'interesting' results it has produced in
the last period.
In this PR, the most 'interesting' results are considered to be the
ones with the highest load, keeping track separately of the top 10
which resulting in an increase decision and the top 10 which did
not. In practice, increase recommendations are sufficiently rare that
the top 10 will often be 'all of them', and they are all potentially
interesting (but we cap it to protect against taking up an unbounded
amount of memory). We keep the high load non-increase recommendations
as well, since these are likely to be the interesting ones to look at
when investigating why some data stream did not get an increase shards
recommendation when we might have expected it.
The mechanism would be easily extended to add in other categories. For
example, we may find that there are some categories of decrease
decisions we consider 'interesting'. (N.B. The rollover service will
not roll over just because the auto-sharding service recommended
down-sharding — down-sharding only happens if the rollover was going
to happen for some other reason (age, size, etc.) So it's normal for
the auto-sharding service to return decrease recommendations for the
same data streams every 5 minutes until those other conditions are
met. Which makes things a bit more complicated.) This PR just covers
the cases that seem likely to be useful in the support cases we have
seen.
The existing DEBUG and TRACE log lines in the service are replaced
with a single DEBUG log which pulls together all the data. This is an
improvement, since at the moment it is hard to figure out from the
logs which lines refer to the same data stream (they are interleaved,
and don't all include the data stream name).
The write load field in the AutoShardingResult was unused, and is
removed.
ES-10037 #comment Improved logging in https://github.com/elastic/elasticsearch/pull/126171
* [main] Move system indices migration to migrate plugin
It seems the best way to fix#122949 is to use existing data stream reindex API. However, this API is located in the migrate x-pack plugin. This commit moves the system indices migration logic (REST handlers, transport actions, and task) to the migrate plugin.
Port of #123551
* [CI] Auto commit changes from spotless
* Fix compilation
* Fix tests
* Fix test
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
We can make slicing things like search hits where we cut many small
buffers out of a large composite reference a lot more efficient in some
cases by making the slice of a releasable reference itself releasable.
This fixes the current situation where we could have composite buffer
of many MB that is made up of e.g. 16k chunks but that we would retain
in full if we were to slice even a single byte out of it in any position.
We need to register the settings update consumer for
QUERY_PHASE_PARALLEL_COLLECTION_ENABLED regardless of the
BATCHED_QUERY_PHASE_FEATURE_FLAG feature flag.
Closes#125899
Add tracking of the number of compute() calls and total iterations
between convergences in the DesiredBalanceComputer, along with the
time since the last convergence. These are included in the log
message when the computer doesn't converge.
Closes#100850.
* test
* Revert "test"
This reverts commit 9f4e2adba0.
* Refactor InferenceService to allow passing in chunking settings
* Add chunking config to inference field metadata and store in semantic_text field
* Fix test compilation errors
* Hacking around trying to get ingest to work
* Debugging
* [CI] Auto commit changes from spotless
* POC works and update TODO to fix this
* [CI] Auto commit changes from spotless
* Refactor chunking settings from model settings to field inference request
* A bit of cleanup
* Revert a bunch of changes to try to narrow down what broke CI
* test
* Revert "test"
This reverts commit 9f4e2adba0.
* Fix InferenceFieldMetadataTest
* [CI] Auto commit changes from spotless
* Add chunking settings back in
* Update builder to use new map
* Fix compilation errors after merge
* Debugging tests
* debugging
* Cleanup
* Add yaml test
* Update tests
* Add chunking to test inference service
* Trying to get tests to work
* Shard bulk inference test never specifies chunking settings
* Fix test
* Always process batches in order
* Fix chunking in test inference service and yaml tests
* [CI] Auto commit changes from spotless
* Refactor - remove convenience method with default chunking settings
* Fix ShardBulkInferenceActionFilterTests
* Fix ElasticsearchInternalServiceTests
* Fix SemanticTextFieldMapperTests
* [CI] Auto commit changes from spotless
* Fix test data to fit within bounds
* Add additional yaml test cases
* Playing with xcontent parsing
* A little cleanup
* Update docs/changelog/121041.yaml
* Fix failures introduced by merge
* [CI] Auto commit changes from spotless
* Address PR feedback
* [CI] Auto commit changes from spotless
* Fix predicate in updated test
* Better handling of null/empty ChunkingSettings
* Update parsing settings
* Fix errors post merge
* PR feedback
* [CI] Auto commit changes from spotless
* PR feedback and fix Xcontent parsing for SemanticTextField
* Remove chunking settings check to use what's passed in from sender service
* Fix some tests
* Cleanup
* Test failure whack-a-mole
* Cleanup
* Refactor to handle memory optimized bulk shard inference actions - this is ugly but at least it compiles
* [CI] Auto commit changes from spotless
* Minor cleanup
* A bit more cleanup
* Spotless
* Revert change
* Update chunking setting update logic
* Go back to serializing maps
* Revert change to model settings - source still errors on missing model_id
* Fix updating chunking settings
* Look up model if null
* Fix test
* Work around https://github.com/elastic/elasticsearch/issues/125723 in semantic text field serialization
* Add BWC tests
* Add chunking_settings to docs
* Refactor/rename
* Address minor PR feedback
* Add test case for null update
* PR feedback - adjust refactor of chunked inputs
* Refactored AbstractTestInferenceService to return offsets instead of just Strings
* [CI] Auto commit changes from spotless
* Fix tests where chunk output was of size 3
* Update mappings per PR feedback
* PR Feedback
* Fix problems related to merge
* PR optimization
* Fix test
* Delete extra file
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
We need to store extra information for Aggregate and AggregateExec for
time-series aggregations. Previously, I added a type (standard or
time_series), but this was not enough. This PR removes it and replaces
it with extensions of Aggregate and AggregateExec. I considered adding
an extra map of metadata to Aggregate and AggregateExec, but this
approach seems simpler.
We recently cleared stack traces on data nodes before transport back to the coordinating node when error_trace=false to reduce unnecessary data transfer and memory on the coordinating node (#118266). However, all logging of exceptions happens on the coordinating node, so stack traces disappeared from any logs. This change logs stack traces directly on the data node when error_trace=false.
Originally, `DATE_TRUNC` only supported 1-month and 3-month intervals for months, and 1-year interval for years, while arbitrary intervals were supported for weeks and days. This PR adds support for `DATE_TRUNC` with arbitrary month and year intervals.
Closes#120094
Don't perform version check assertion in TsdbDocValueBwcTests if security manager is active.
By default, with jvm version 24 entitlements are used instead security manager and assertOldDocValuesFormatVersion() / assertNewDocValuesFormatVersion() work as expected.
Making these methods work with security manager would require granting the server entire test codebase suppressAccessChecks and suppressAccessChecks privileges. This is undesired from a security manager perspective. Instead, only assert doc values format checks if security manager isn't active, which is always the case jvm version 24 or higher is used.
Closes#126174
This action solely needs the cluster state, it can run on any node.
Since this is the last class/action that extends the `ClusterInfo`
abstract classes, we remove those classes too as they're not required
anymore.
Relates #101805
Wrap remote errors with cluster name to provide more context
Previously, if a remote encountered an error, user would see a top-level error that would provide no context about which remote ran into the error. Now, such errors are wrapped in a separate remote exception whose error message clearly specifies the name of the remote cluster and the error that occurred is the cause of this remote exception.
This action solely needs the cluster state, it can run on any node.
Additionally, it needs to be cancellable to avoid doing unnecessary work
after a client failure or timeout.
Relates #101805
The change contains the following changes:
- The numDocsWithField field moved from SortedNumericEntry to NumericEntry. Making this statistic always available.
- Store jump table after values in ES87TSDBDocValuesConsumer#writeField(...). Currently it is stored before storing values. This will allow us later to iterate over the SortedNumericDocValues once. When merging, this is expensive as a merge sort on the fly is being executed.
This change will allow all the optimizations that are listed in #125403
