## Summary
Seeing a bunch of errors like this in serverless logs:
```
Executing Rule default:.es-query:23dedce0-5230-11ee-b332-73fb351d06b8 has resulted in Error: Cannot read properties of undefined (reading 'meta') - TypeError: Cannot read properties of undefined (reading 'meta')\n at /usr/share/kibana/node_modules/@kbn/actions-plugin/server/authorization/get_authorization_mode_by_source.js:45:56\n at Array.reduce (<anonymous>)\n at getBulkAuthorizationModeBySource (/usr/share/kibana/node_modules/@kbn/actions-plugin/server/authorization/get_authorization_mode_by_source.js:43:47)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at ActionsClient.bulkEnqueueExecution (/usr/share/kibana/node_modules/@kbn/actions-plugin/server/actions_client/actions_client.js:588:24)\n at ExecutionHandler.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/execution_handler.js:249:28)\n at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:413:37\n at TaskRunnerTimer.runWithTimer (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner_timer.js:50:20)\n at TaskRunner.runRule (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:406:5)\n at TaskRunner.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:629:49)\n at TaskManagerRunner.run (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:315:170)
```
which seem to be caused by connectivity issues between Kibana and ES. In
this case, when a rule executes and tries to schedule an action, we use
the saved objects client `bulkGet` function to determine if the rule is
a legacy (pre 7.10) version. We were not checking for errors returned
from the `bulkGet` (which returns as a 200 with errors embedded in the
success response) and so were trying to access data from the SO that did
not exist. This PR checks for errors returned from the `bulkGet` and
logs the error accordingly. Also did some small optimizations to the
code
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Addresses https://github.com/elastic/kibana/issues/160411
## Summary
This PR adds functionality for filtering out advanced settings that are
not relevant for serverless.
For context, we need to build an Advanced settings page in serverless
which only contains a set of the existing settings. We will reuse the
section registry (https://github.com/elastic/kibana/pull/163502) from
the original Advanced settings plugin as well as its UI components which
will also be extracted into a separate package. The app will be
registered from inside the `serverless` plugin.
In order to only display the settings that are relevant for serverless,
we need to make some changes to the uiSettings service. The
implementation in this PR leverages the existing `readonly` uiSettings
param and adds the `setAllowlist()` method which is called by the
serverless plugin to set an allowlist of setting keys.
**Testing in serverless:**
1. Set `advanced_settings.enabled: true` to enable the Advanced settings
app in serverless:
5b216c6ea9/config/serverless.yml (L53)
2. Start Es with `yarn es serverless --ssl` and Kibana with `yarn
serverless-{mode} --ssl` in any serverless mode.
3. Navigate to `app/management/kibana/settings`
4. Verify that the app only displays the settings from
`packages/serverless/settings/common/index.ts` (these are the settings,
relevant for all projects in serverless) as well as the settings from
the corresponding project package
`packages/serverless/settings/{mode}_project/index.ts`.
5. Verify that the app is functioning correctly.
**Testing in self-managed:**
1. Start Es with `yarn es snapshot` and Kibana with `yarn start`.
2. Go to Stack Management > Advanced settings
3. Verify that all settings are displayed as usual.
4. Verify that the app is functioning correctly.
If your team is a code owner of any of the serverless project plugins,
please review the corresponding package
`packages/serverless/settings/{search/observanility/security}_project/index.ts`
where you've been added as an owner and test in the serverless solution
accordingly.
<!---
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.
When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |
-->
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Resolves https://github.com/elastic/kibana/issues/162264
## Summary
Adds a limit on the maximum number of actions that can be queued with a
circuit breaker. The limit in serverless is set to 10,000, and 1,000,000
in the other environments.
- If a rule execution exceeds the limit, the circuit breaker kicks in
and stops triggering actions.
- Alerting rule's status updated to warning when circuit breaker is hit
Did not update the `enqueueExecution` bc it's going to be removed in
https://github.com/elastic/kibana/pull/165120.
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### To Verify
- Create a 2 rules that have actions
- Set `xpack.actions.queued.max` in kibana.yml to a low number like 2 or
3
- Use the run soon button to queue up actions and hit the circuit
breaker.
- The actions will not be scheduled and the rule status will be set to
warning
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves: #163751
This PR adds a new API to allow the other plugins to define an **enabled
connector types list** in the actions registry.
The list is used during the action execution to decide if the action
type is executable (enabled).
This decision logic sits on the existing two other checks:
1- `isActionTypeEnabled` -> if the connector type is in the
`enabledActionTypes` list in the config
2- `isLicenseValidForActionType` -> if the connector type is allowed for
the active license
As the only user of this feature is just security-solutions for now, we
decided to allow the list to be set only once.
<img width="890" alt="Screenshot 2023-08-31 at 12 00 10"
src="896ab104-6f75-4eee-8f19-9f2eb04bb149">
<img width="592" alt="Screenshot 2023-08-31 at 12 13 38"
src="533a1cfd-947a-4506-b078-149780346088">
<img width="1513" alt="Screenshot 2023-08-31 at 12 00 01"
src="91169dfe-b7f8-4fef-b734-56857a75dbdc">
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Towards https://github.com/elastic/response-ops-team/issues/125
## Summary
Preparing the `GET ${BASE_ACTION_API_PATH}/connector_types` HTTP API for
versioning
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Part of: https://github.com/elastic/response-ops-team/issues/125
This PR intends to prepare the `GET ${BASE_ACTION_API_PATH}/connectors`
API for versioning as shown in the above issue.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
upgrade `axios` to 1.4
- adjust to header usage, and config optionality
- Axios' adapters are now resolved from a string key by axios, no need
to import/instantiate adapters
- most of the changed code stems from changes in Axios' types
- `response.config` is now optional
- there was a change in the type of AxiosHeaders <->
InternalAxiosHeaders
Closes: #162661Closes: #162414
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes#160812
Adds UI for CA and client-side SSL certificate auth to webhook
connector, and adds these properties to the webhook's HttpsAgent on
execution:
<img width="912" alt="Screenshot 2023-07-18 at 11 59 33 AM"
src="b1986c8d-3632-4663-80ff-3fb37f706d07">
### When Editing
<img width="868" alt="Screenshot 2023-07-14 at 3 34 57 PM"
src="e3e0f450-1196-45fd-88a6-1e15b6f2fa36">
Also creates a Field helper for EuiFilePicker
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Bumps node.js to 18.17.0 (replacement for PR #144012 which was later
reverted)
As a result, these categorical additions were needed:
- `node` evocations will need the `--openssl-legacy-provider` flag,
wherever it would use certain crypto functionalities
- tests required updating of the expected HTTPS Agent call arguments,
`noDelay` seems to be a default
- `window.[NAME]` fields cannot be written directly
- some stricter typechecks
This is using our in-house built node.js 18 versions through the URLs
the proxy-cache. (built with
https://github.com/elastic/kibana-custom-nodejs-builds/pull/4)
These urls are served from a bucket, where the RHEL7/Centos7 compatible
node distributables are. (see:
https://github.com/elastic/kibana-ci-proxy-cache/pull/7)
Further todos:
- [x] check docs wording and consistency
- [ ] update the dependency report
- [x] explain custom builds in documentation
- [x] node_sass prebuilts
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Co-authored-by: Thomas Watson <w@tson.dk>
## Summary
This PR removes the ability to get system actions from `GET` APIs.
Specifically:
- The Get action API throws a 404 error when requesting a system action
- The Bulk Get action API throws a 404 error when requesting a system
action
- The Get All API filters out system actions
- The Get List Types API filters out system actions
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
Part of https://github.com/elastic/kibana/issues/159342.
In this PR, I'm preparing the non-alerting (rule types) response ops
task types for serverless by defining an explicit task state schema.
This schema is used to validate the task's state before saving but also
when reading. In the scenario an older Kibana node runs a task after a
newer Kibana node has stored additional task state, the unknown state
properties will be dropped. Additionally, this will prompt developers to
be aware that adding required fields to the task state is a breaking
change that must be handled with care. (see
https://github.com/elastic/kibana/issues/155764).
For more information on how to use `stateSchemaByVersion`, see
https://github.com/elastic/kibana/pull/159048 and
https://github.com/elastic/kibana/blob/main/x-pack/plugins/task_manager/README.md.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves https://github.com/elastic/kibana/issues/161126
## Summary
Removes references to the following list of action variables in the code
and in the UI:
| Mustache legacy variable | Recommended replacement expression | To be
deprecated
| ----------- | ----------- | ----------- |
| alertId | rule.id | yes |
| alertName | rule.name | yes |
| spaceId | rule.spaceId | yes |
| tags | rule.tags | yes |
| params | rule.params | yes |
| alertInstanceId | alert.id | yes |
| alertActionGroup | alert.actionGroup | yes |
| alertActionSubgroup | alert.actionSubgroup | yes |
| alertActionGroupName | alert.actionGroupName | yes |
Also deprecates `params` in favor of `rule.params`
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### Checklist
1. Create an es query/index threshold rule and add a connector
2. Verify that the default message does not contain any legacy variables
3. Click on the list of available action variables and verify that
`params` is deprecated and all the params variables are prefixed with
`rules.`
## Release Note
The following rule action variables have been deprecated; use the
recommended variables (in parentheses) instead:
- `alertActionGroup` (`alert.actionGroup`)
- `alertActionGroupName` (`alert.actionGroupName`)
- `alertActionSubgroup` (`alert.actionSubgroup`)
- `alertId` (`rule.id`)
- `alertInstanceId` (`alert.id`)
- `alertName` (`rule.name`)
- `params` (`rule.params`)
- `spaceId` (`rule.spaceId`)
- `tags` (`rule.tags`)
Resolves: #155766Resolves: #159302
With this PR we aim to skip a task that has invalid direct and indirect
params.
In order to do that,
1- We validate the task params before calling the subtask's run method
and skip if if the task params are invalid.
2- We skip execution of a subtask (rule, action etc) when the run method
of it returns a `SkipError`
Therefore, validations in the run methods needs to be moved to top of
the run method and executed before anything else to return skip if the
data is invalid.
We also added a config to enable/disable the skip feature, and define
the delay duration of task reschedule.
As this may become an infinitive loop, we are supposed to limit the
attempts.
Follow on issue to implement that: #159302
---------
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR:
- Adds the ability to create system action types
- Creates system connectors on Kibana `start` from the system action
types
- Prevents system action to be created/updated/deleted
- Return system actions from the get/getAll endpoints
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR introduces the `isSystemAction: boolean` to the `ActionResult`
type. It also, adds the same to the `ActionType` type. I added
validation to verify that system actions cannot be registered at the
moment. More PRs are going to follow to incrementally build the feature.
Related: https://github.com/elastic/kibana/issues/160367
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Updates the code in the manual proxy tester to work with an updated
version of it's pre-req npm packages.
This code is not used in production, or ci tests, just used manually for
ad hoc proxy testing.
Since this file is used manually, is currently not type-checked, and
infrequently used, no one noticed it was no longer working, till I did
this morning.
## Summary
The spread operator is costly and put pressure on GC. It should be
avoided when possible, especially in loops.
This PR adapts a lot of `reduce` calls in the codebase to remove the
usages of the diabolic spread operator, when possible.
Note: the PR is not fully exhaustive. I focused on the server-side, as
we're more directly impacted than on browser-side code regarding
performances.
## Removing `...` usages in `kittens.reduce()`
For `reduce` loops, the spread operator can usually easily be replaced:
#### - setting a value on the accum object and returning it
#### BAD
```ts
return this.toArray().reduce(
(acc, renderer) => ({
...acc,
[renderer.name]: renderer,
}),
{} as Record<string, ExpressionRenderer>
);
```
#### GOOD
```ts
return this.toArray().reduce((acc, renderer) => {
acc[renderer.name] = renderer;
return acc;
}, {} as Record<string, ExpressionRenderer>);
```
#### - assigning values to the accum object and returning it
#### BAD
```ts
const allAggs: Record<string, any> = fieldAggRequests.reduce(
(aggs: Record<string, any>, fieldAggRequest: unknown | null) => {
return fieldAggRequest ? { ...aggs, ...(fieldAggRequest as Record<string, any>) } : aggs;
},
{}
);
```
#### GOOD
```ts
const allAggs = fieldAggRequests.reduce<Record<string, any>>(
(aggs: Record<string, any>, fieldAggRequest: unknown | null) => {
if (fieldAggRequest) {
Object.assign(aggs, fieldAggRequest);
}
return aggs;
},
{}
);
```
#### - pushing items to the accum list and returning it
#### BAD
```ts
const charsFound = charToArray.reduce(
(acc, char) => (value.includes(char) ? [...acc, char] : acc),
[] as string[]
);
```
#### GOOD
```ts
const charsFound = charToArray.reduce((acc, char) => {
if (value.includes(char)) {
acc.push(char);
}
return acc;
}, [] as string[]);
```
## Questions
#### Are you sure all the changes in this are strictly better for
runtime performances?
Yes, yes I am.
#### How much better?
Likely not much.
#### Are you planning on analyzing the perf gain?
Nope.
#### So why did you do it?
I got tired of seeing badly used spread operators in my team's owned
code, and I had some extra time during on-week, so I spent a few hours
adapting the usages in all our runtime/production codebase.
#### Was it fun?
Take your best guess.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Add deferred migrations parameter.
* Update outdated documents query to take into account deferred migrations.
* Update outdated documents query to take into account the core migration version.
* Update read operations in the saved objects repository to perform deferred migrations.
