Transport actions have associated request and response classes. However,
the base type restrictions are not necessary to duplicate when creating
a map of transport actions. Relatedly, the ActionHandler class doesn't
actually need strongly typed action type and classes since they are lost
when shoved into the node client map. This commit removes these type
restrictions and generic parameters.
This replaces the usage of a defaultDistribution for javaRestTest by the integTestDistribution.
This has a few advantages:
1. The overall dependencies on running the java rest tests are minimized.
By using the default distribution we rely on building the whole default distribution (including all modules and plugins)
before we can run these tests. This a) takes time and b) dramatically reduces the likelyhood of us avoiding test task execution at all as we
basically declare the whole distro as an input. By using integTest distro we reduce the surface of the inputs dramatically which also results in faster
execution of these tests
2. its more idiomatic as one pattern we see is that we e.g disable the security settings that we would not need once we use the integTest distro without
the security plugin
3. it makes test setup and its dependencies more explicit.
Point 3. might sound as like a downside at first, but for understanding what those tests are doing and what they are relying on I think its worth the 3 more lines of code.
Here are two build scans task executions:
- before the `javaRestTest` task requires `995 tasks, 2 transforms executed in 155 projects`: https://gradle-enterprise.elastic.co/s/drj5mfzsfx7ra/timeline
- after we only rely on `275 tasks, 2 transforms executed in 56 projects`: https://gradle-enterprise.elastic.co/s/jr5sblhppn4fg/timeline?page=2
* Fix Gradle Deprecation warning as declaring an is- property with a Boolean type has been deprecated.
* Make use of new layout.settingsFolder api to address some cross project references
* Fix buildParams snapshot check for multiprojet projects
Creating (and more importantly eventually resizing) a fresh stream
output makes up a large chunk of the runtime of this aggregation.
Also, recursively calling an inline consumer makes this logic
even more confusing and adds additional overhead as escape analysis
will not be able to remove the allocation of the consumer.
=> just call a method recursively and reuse the output
The aggs timeout test waits for the agg to return and then double checks
that the agg is stopped using the tasks API. We're seeing some failures
where the tasks API reports that the agg is still running. I can't
reproduce them because computers. This adds two things:
1. Logs the hot_threads so we can see if the query is indeed still
running.
2. Retries the _tasks API for a minute. If it goes away soon after the
_search returns that's *fine*. If it sticks around for more than a
few seconds then the cancel isn't working. We wait for a minute
because CI can't be trusted to do anything quickly.
Closes#121993
This PR addresses issues around aggregations cancellation, mentioned in https://github.com/elastic/elasticsearch/issues/108701 and other places. In brief, during aggregations collection time, we respect cancellation via the mechanisms in the searcher to poison cancelled queries. But once the aggregation finishes collection, there is no further need to interact with the searcher, so we cannot rely on that for cancellation checking. In particular, deeply nested aggregations can spend a long time constructing the results tree.
Checking for cancellation is a trade off, as the check itself is somewhat expensive (it involves a volatile read), so we want to balance checking often enough that cancelled queries aren't taking up resources for a long time, but not so frequently that it slows down most aggregation queries. Our first attempt to this is to check once when we go to build sub-aggregations, as the worst cases for this that we've seen involve needing to build deep sub-aggregation trees. Checking at sub-aggregation construction time also provides a conveniently centralized method call to add the check to.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Nik Everett <nik9000@gmail.com>
* Exhaustive testParseFractionalNumber
* Refactor: encapsulate ByteSizeUnit constructor
* Refactor: store size in bytes
* Support up to 2 decimals in parsed ByteSizeValue
* Fix test for rounding up with no warnings
* ByteSizeUnit transport changes
* Update docs/changelog/120142.yaml
* Changelog details and impact
* Fix change log breaking.area
* Address PR comments
This updates the gradle wrapper to 8.12
We addressed deprecation warnings due to the update that includes:
- Fix change in TestOutputEvent api
- Fix deprecation in groovy syntax
- Use latest ospackage plugin containing our fix
- Remove project usages at execution time
- Fix deprecated project references in repository-old-versions
This commit changes the signature of InternalAggregation#buildAggregations(long[]) to
InternalAggregation#buildAggregations(LongArray) to avoid allocations of humongous arrays.
Static fields dont do well in Gradle with configuration cache enabled.
- Use buildParams extension in build scripts
- Keep BuildParams.ci for now for easy serverless migration
- Tweak testing doc
* Refactor DocumentDimensions to RoutingFields
* update
* add test
* add test
* updates from review
* updates from review
* spotless
* remove final from subclass
* fix final
XPackFeatureSet hasn't been used for many years. But the inner "Usage" class
is still used. This commit moves the Usage class up to its own file as
XPackFeatureUsage, and removes the defunct XPackFeatureSet interface.
closes#29736
With Lucene 10, IndexWriter requires a parent document field in order to
use index sorting with document blocks. This lead to different IAE and file
leaks in this test which are fixed by adapting the corresponding location in
the test setup.
The most relevant ES changes that upgrading to Lucene 10 requires are:
- use the appropriate IOContext
- Scorer / ScorerSupplier breaking changes
- Regex automaton are no longer determinized by default
- minimize moved to test classes
- introduce Elasticsearch900Codec
- adjust slicing code according to the added support for intra-segment concurrency
- disable intra-segment concurrency in tests
- adjust accessor methods for many Lucene classes that became a record
- adapt to breaking changes in the analysis area
Co-authored-by: Christoph Büscher <christophbuescher@posteo.de>
Co-authored-by: Mayya Sharipova <mayya.sharipova@elastic.co>
Co-authored-by: ChrisHegarty <chegar999@gmail.com>
Co-authored-by: Brian Seeders <brian.seeders@elastic.co>
Co-authored-by: Armin Braun <me@obrown.io>
Co-authored-by: Panagiotis Bailis <pmpailis@gmail.com>
Co-authored-by: Benjamin Trent <4357155+benwtrent@users.noreply.github.com>
Part of https://github.com/elastic/elasticsearch/issues/99815
## Steps 1. Migrate TDigest classes to use a custom Array
implementation. Temporarily use a simple array wrapper
(https://github.com/elastic/elasticsearch/pull/112810) 2. Implement
CircuitBreaking in the `WrapperTDigestArrays` class. Add
Releasable/AutoCloseable and ensure everything is closed
(https://github.com/elastic/elasticsearch/pull/113105) 3. Pass the
CircuitBreaker as a parameter to TDigestState from wherever it's being
used (This PR) - ESQL: Pass a real CB - Other aggs: Use the
deprecated methods on `TDigestState`, that will use a No-op CB instead
4. Account remaining TDigest classes size ("SHALLOW_SIZE")
Every step should be safely mergeable to main: - The first and second
steps should have no impact. - The third and fourth ones will start
increasing the CB count partially.
## Remarks TDigestStates are Releasable, and should be closed now.
However, old aggregations don't close them, as it's not trivial, and as
they are using the NoopCircuitBreaker, there's no need to close them
Several `TransportNodesAction` implementations do some kind of top-level
computation in addition to fanning out requests to individual nodes.
Today they all have to do this once the node-level fanout is complete,
but in most cases the top-level computation can happen in parallel with
the fanout. This commit adds support for an additional `ActionContext`
object, created when starting to process the request and exposed to
`newResponseAsync()` at the end, to allow this parallelization.
All implementations use `(Void) null` for this param, except for
`TransportClusterStatsAction` which now parallelizes the computation of
the cluster-state-based stats with the node-level fanout.
Introduces per-field param `synthetic_source_keep` that overrides the
behavior for keeping the field source in synthetic source mode: -
`none` : no source is stored - `arrays`: the incoming source is
recorded as-is for arrays of a given field - `all`: the incoming source
is recorded as is for both singleton and array values of a given field
Related to #112012
Several `TransportNodesAction` implementations do some kind of top-level
computation in addition to fanning out requests to individual nodes.
Today they all have to do this once the node-level fanout is complete,
but in most cases the top-level computation can happen in parallel with
the fanout. This commit adds support for an additional `ActionContext`
object, created when starting to process the request and exposed to
`newResponseAsync()` at the end, to allow this parallelization.
All implementations use `(Void) null` for this param, except for
`TransportClusterStatsAction` which now parallelizes the computation of
the cluster-state-based stats with the node-level fanout.
* Refactor build params for FieldMapper
* more mappers and tests
* more mappers
* more mappers
* spotless
* spotless
* stored by default
* Revert "stored by default"
This reverts commit bbd247d64b.
* restore storeIgnored
* sync
* list valid values for SourceKeepMode
* small refactoring
* spotless
