Transport actions have associated request and response classes. However,
the base type restrictions are not necessary to duplicate when creating
a map of transport actions. Relatedly, the ActionHandler class doesn't
actually need strongly typed action type and classes since they are lost
when shoved into the node client map. This commit removes these type
restrictions and generic parameters.
We can just use a real constant for the `null` case, avoiding any
non-plain stores in all cases. This should be somewhat helpful for the
security interceptors.
The method should be called with an explicit project-id to access
persistent tasks from the right project. This PR does that. The
callsites are updated by using the default project-id for the timebeing.
They work for single project deployments but should eventually be
updated for multi-project setup.
Relates: ES-11039
Reduce boilerplate associated with creating `FieldCapabilities` instances.
Since it's a class with a huge number of fields, it makes sense to define a builder object, as that can also help with all the Boolean and null blindness going on.
Note while there is a static Builder class in `FieldCapabilities`, it is not a proper builder object (no setters, still need to pass a lot of otherwise default parameters) and also package-private. To avoid changing that, I defined a new `FieldCapabilitiesBuilder` class. I also went over the code and refactored places which used the old constructor.
There is no guarantee that a project has non-null persistent tasks [0].
Null check is already done in most places. This PR adds it in a few more
places.
[0] Since health-node is now a cluster-scoped persistent task, it has
become more likely for the project-scoped tasks to be null.
Relates: #123262
This PR is the follow-up work for MP-1945 and MP-1938 which laid the
foundation of two different scoped persistent tasks. It updates the
persistent task framework to be aware of the two task types so that it
can handle both cluster scope tasks and per-project tasks. Once these
changes are in place, we will make health-node to be the first
cluster-scope persistent task.
Relates: ES-10168
Add the task name (when possible) and ID as part of the source string
for updating cluster state. This helps better identifying the source of
a task. The updatePersistentTaskState method already does it. This PR
ensures it is the case in other places.
When the Transform is configured to write to an alias, specifying
`DELETE _transform/<id>?delete_dest_index` will follow the alias
to the concrete destination index.
Fix#121913
Co-authored-by: Przemysław Witek <przemyslaw.witek@elastic.co>
There is a race condition where the test is trying to clean up while the
Transform auditor is still writing messages - resetting the plugin will
stop the auditor (and properly reset it).
Fix#12148
Building off of `stats` and multi-value aggregations, including the
limitation:
- all values of extended_stats will be mapped to `double` if mapping
deduction is used
Relates #51925
This call has the side effect that if you are iterating a number of hits calling this method, you will be increasing the
memory usage by a non trivial number which in most of cases is unwanted. Therefore this commit removes this caching
all together and add an assertion so the method is call once during the lifetime of the object.
Refetch TransformConfig with SeqNo before migrating
max_page_search_size, then include the SeqNo in the index request to
ensure we are not overwriting a user's intentional config change. If we
are, just drop the migration and try again next time.
Updates various places - mostly related to ML - to make the `POST
_features/_reset` API not fail when there are multiple projects.
This does not actually make those places project-aware,
but since nothing ML-related has been made project-aware,
this is the best way forward for now to make IT cleanup project aware.
If an update changes the destination index, optimistically assume the
new destination index does not have a write block and try to run the
transform. If the new destination index has a write block, the
transform will drop the run and move back into a blocked state.
Fix#120065
When the destination index is blocked, the bulk request is ignored.
Each subsequent trigger will check the cluster state if the index is
unblocked or else drop the trigger.
Users can call the ScheduleNow API to skip the index block check and let
the trigger start the next checkpoint - if the index is blocked then the
bulk request will fail again.
While the Transform is skipping triggers, the state will display as
`waiting` from the Get Stats API and on the Transform UI in Kibana.
`max_page_search_size` had been deprecated in Pivot and migrated to
Settings in ES 7.x. We are removing the key in Pivot in 9.x. It is
still possible to create Pivots with `max_page_search_size`, though a
deprecation is logged. To prep for its removal, this change
automatically migrates `max_page_search_size` from Pivot to Settings as
if the user called `_update`.
- `PUT _transform` with `max_page_search_size` in Pivot will migrate the
value to Settings as part of the Transform's creation. The user will
receive an additional deprecation warning informing of this migration.
- Existing transforms with `max_page_search_size` in Pivot will migrate
the value to Settings as part of node assignment during start.
Transform Messages and Elasticsearch logs will record this event so
users will know what happened.
Shutdown metadata is keyed on node id. This makes sense since only one
node with a given node id can exist within a cluster. However, it is
possible that shutdown was initiated for one instance of a node, but
that node is restarted. This commit adds the ephemeral node id to
shutdown metadata so that nodes with the same id but different ephemeral
id can be distinguished.
Prevent writes to the Transform system index while it is being reindexed
during upgrade.
- Add a new REST API, `_transform/set_upgrade_mode`, similar to `_ml`,
which will enable via `?enabled` or disable (when omitted).
- Add a new Transport action, enabling or disabling upgrade mode. When
enabled, all Transforms will abort and move into the `waiting` state,
preventing Transforms from running and writing to the system index.
- Hook into the System Index upgrade action, calling the new Transform
action to set and unset upgrade mode.
Co-authored-by: David Kyle <david.kyle@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This updates the gradle wrapper to 8.12
We addressed deprecation warnings due to the update that includes:
- Fix change in TestOutputEvent api
- Fix deprecation in groovy syntax
- Use latest ospackage plugin containing our fix
- Remove project usages at execution time
- Fix deprecated project references in repository-old-versions
* Removing index alias creation for deprecated transforms notification index
* Update docs/changelog/117583.yaml
* Updating changelog
* Updating deprecation area to Transform
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
We can parallelize starting the clusters and a few other things
to effectively speed up these tests by 2x which comes out to about a minute
of execution time saved for all of those in :server:internalClusterTests
on my workstation.
