## Summary
Resolves: https://github.com/elastic/kibana/issues/151463
Removes all reference to ephemeral tasks from the task manager plugin.
As well as unit and E2E tests while maintaining backwards compatibility
for `xpack.task_manager.ephemeral_tasks` flag to no-op if set. This PR
has some dependencies from the PR to remove ephemeral task support from
the alerting and actions plugin
(https://github.com/elastic/kibana/pull/197421). So it should be merged
after the other PR.
Deprecates the following configuration settings:
- xpack.task_manager.ephemeral_tasks.enabled
- xpack.task_manager.ephemeral_tasks.request_capacity
The user doesn't have to change anything on their end if they don't wish
to. This deprecation is made so if the above settings are defined,
kibana will simply do nothing.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Towards: #176585
This PR removes the task skipping logic from TaskManager, PRs for
Alerting and Actions will follow.
## To verify
Rules and actions should be still working as expected.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves: #155766Resolves: #159302
With this PR we aim to skip a task that has invalid direct and indirect
params.
In order to do that,
1- We validate the task params before calling the subtask's run method
and skip if if the task params are invalid.
2- We skip execution of a subtask (rule, action etc) when the run method
of it returns a `SkipError`
Therefore, validations in the run methods needs to be moved to top of
the run method and executed before anything else to return skip if the
data is invalid.
We also added a config to enable/disable the skip feature, and define
the delay duration of task reschedule.
As this may become an infinitive loop, we are supposed to limit the
attempts.
Follow on issue to implement that: #159302
---------
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Fixes https://github.com/elastic/kibana/issues/149344
This PR migrates all plugins to packages automatically. It does this
using `node scripts/lint_packages` to automatically migrate
`kibana.json` files to `kibana.jsonc` files. By doing this automatically
we can simplify many build and testing procedures to only support
packages, and not both "packages" and "synthetic packages" (basically
pointers to plugins).
The majority of changes are in operations related code, so we'll be
having operations review this before marking it ready for review. The
vast majority of the code owners are simply pinged because we deleted
all `kibana.json` files and replaced them with `kibana.jsonc` files, so
we plan on leaving the PR ready-for-review for about 24 hours before
merging (after feature freeze), assuming we don't have any blockers
(especially from @elastic/kibana-core since there are a few core
specific changes, though the majority were handled in #149370).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* added ability to run ephemeral tasks
* fixed typing
* added typing on plugin
* WIP
* Fix type issues
* Hook up the ephemeral task into the task runner for actions
* Tasks can now run independently of one another
* Use deferred language
* Refactor taskParams slightly
* Use Promise.all
* Remove deferred logic
* Add config options to limit the amount of tasks executing at once
* Add ephemeral task monitoring
* WIP
* Add single test so far
* Ensure we log after actions have executed
* Remove confusing * 1
* Add logic to ensure we fallback to default enqueueing if the total actions is above the config
* Add additional test
* Fix tests a bit, ensure we log the alerting:actions-execute right away and the tests should listen for alerts:execute
* Better tests
* If the queue is at capacity, attempt to execute the ephemeral task as a regular action
* Ensure we run ephemeral tasks before to avoid them getting stuck in the queue
* Do not handle the promise anymore
* Remove unnecessary code
* Properly handle errors from ephemeral task lifecycle
* moved acitons domain out of alerting and into actions plugin
* Remove some tests
* Fix TS and test issues
* Fix type issues
* Fix more type issues
* Fix more type issues
* Fix jest tests
* Fix more jest tests
* Off by default
* Fix jest tests
* Update config for this suite too
* Start of telemetry code
* Fix types and add missing files
* Fix telemetry schema
* Fix types
* Fix more types
* moved load event emission to pollingcycle and added health stats on Ephemeral tasks
* Add more telemetry data based on new health metrics for the ephemeral queue
* Fix tests and types
* Add separate request capacity for ephemeral queue
* Fix telemetry schema and add tests for usage collection
* track polled tasks by persistence and use in capacity estimation instead of executions
* fixed typing
* Bump default capacity
* added delay metric to ephemeral stats
* Fix bad merge
* Fix tests
* Fix tests
* Fix types
* Skip failing tests
* Exclude ephemeral stats from capacity estimation tests
* PR feedback
* More PR feedback
* PR feedback
* Fix merge conflict
* Try fixing CI
* Fix broken lock file from merge
* Match master
* Add this back
* PR feedback
* Change to queue and add test
* Disable ephemeral queue in tests
* Updated desc
* Comment out ephemeral-specific tests tha require the entire test suite to support ephemeral tasks
* Add clarifying comment
Co-authored-by: Gidi Meir Morris <github@gidi.io>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
This PR adds an an internal monitoring mechanism in Task Manager which keep track of a variety of metrics and a health api endpoint which makes the monitored statistics accessible.
* mark legacy ES client types as deprecated
* expose es client to plugins and update mocks
* ElasticSearchClientMock --> ElasticsearchClientMock
* expose es client mocks
* expose es client via RequestHandlerContext
* convert test/plugin_functional/config into ts
* convert top_nav test into ts
* add an integration test for the es client
* update comments to refer to the new es client
* fix import paths. do not use extensions
temp
* update docs
* fix other refs
* add test for a custom client
* fix context
* add test for scoped client
* update docs
