While debugging scalability testing failure for
`cloud_security_dashboard` journey, I found that we hardcoded base FTR
config to `x-pack/performance/journeys/login.ts` and main issue is that
Kibana is not started properly.
This PR makes few changes:
- update `kbn-performance-testing-dataset-extractor` to save journey
path as `configPath` so it can be later used to start ES/Kibana in the
scalability run with the same configuration it was run for the single
user journey run.
- update scalability entry configuration to read base FTR config from
generated scalability json file (`configPath` property)
How to test:
- make sure to clone the latest
[kibana-load-testing](https://github.com/elastic/kibana-load-testing)
repo and build it `mvn clean test-compile`
- from kibana root directory run any api capacity test
```
node scripts/run_scalability.js --journey-path x-pack/test/scalability/apis/api.core.capabilities.json
```
Expected result: logs should display
```
debg Loading config file from x-pack/performance/journeys/login.ts
```
- download the latest artifacts from
[buildkite](https://buildkite.com/elastic/kibana-performance-data-set-extraction/builds/171#0186a342-9dea-4a9b-bbe4-c96449563269),
find `cloud_security_dashboard-<uuid>.json`
- from kibana root directory run scalability test for
`cloud_security_dashboard` journey
```
node scripts/run_scalability.js --journey-path <path to cloud_security_dashboard-<uuid>.json>
```
Expected result: logs should display
```
debg Loading config file from x-pack/performance/journeys/cloud_security_dashboard.ts
```
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Adds `serviceMap` helper to generate transaction and spans in synthtrace
by defining a set of traces from which a service map can be rendered.
This can be considered a follow-up to
https://github.com/elastic/kibana/pull/149900 where synthtrace was used
to generate service maps for api integration tests.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves https://github.com/elastic/kibana/issues/150358
## Summary
In a previous [PR](https://github.com/elastic/kibana/pull/145581) we
started installing a common component template for framework alerts as
data when the `xpack.alerting.enableFrameworkAlerts` config flag is set
to true. In that PR we used a different naming pattern than what is used
by the rule registry for its component templates.
In this PR we are doing the following:
* Renaming the installed `alerts-common-component-template` to
`.alerts-framework-mappings`.
* Creating and installing `.alerts-legacy-alert-mappings` component
template when `enableFrameworkAlerts: true` on alerting plugin setup
* The combination of the two component templates creates the same set of
mappings as the rule registry technical component template
* Creating and installing `.alerts-ecs-mappings` component template when
`enableFrameworkAlerts: true` on alerting plugin setup (when
`enableFrameworkAlerts: false`, the rule registry continues to install
this component template
* Using the `@kbn/ecs` package provided by core to generate the ECS
field map. The rule registry will continue to install the existing ECS
field map which is actually a subset of ECS fields
* Adding `useLegacy` and `useEcs` flags that allow rule types to specify
whether to include the legacy alerts component template and the ECS
component template when registering with framework alerts-as-data.
* Moved some common functions to alerting framework from the rule
registry
## Things to note
* When generating the ECS field map, we are now including the
`ignore_above` setting from the `@kbn/ecs` package. This changes the ECS
component template to include those settings. I tested updating an index
with just `"type":"keyword"` mappings to add the `ignore_above` field to
the mapping and had no issues so this seems like an additive change to
the mapping that will hopefully prevent problems in the future.
* The rule registry ECS component template also includes the technical
fields which is redundant because the technical component template is
automatically installed for all index templates so the framework ECS
component template only contains ECS fields.
| Previous mapping | Updated mapping |
| ----------- | ----------- |
| `{ "organization": { "type": "keyword" } }` | `{ "organization": {
"type": "keyword", "ignore_above": 1024 } }` |
## To Verify
### Verify that the generated component templates are as expected:
Get the following
**While running `main`:**
1. Get the ECS component template `GET
_component_template/.alerts-ecs-mappings`
2. Get the technical component template `GET
_component_template/.alerts-technical-mappings`
3. Create a detection rule that creates an alert and then get the index
mapping for the concrete security alert index `GET
.internal.alerts-security.alerts-default-000001/_mapping`
**While running this branch with `xpack.alerting.enableFrameworkAlerts:
false`:**
4. Get the ECS component template `GET
_component_template/.alerts-ecs-mappings`
5. Get the technical component template `GET
_component_template/.alerts-technical-mappings`
6. Create a detection rule that creates an alert and then get the index
mapping for the concrete security alert index `GET
.internal.alerts-security.alerts-default-000001/_mapping`
**While running this branch with `xpack.alerting.enableFrameworkAlerts:
true`:**
7. Get the ECS component template `GET
_component_template/.alerts-ecs-mappings`
8. Get the technical component template `GET
_component_template/.alerts-technical-mappings`
9. Create a detection rule that creates an alert and then get the index
mapping for the concrete security alert index `GET
.internal.alerts-security.alerts-default-000001/_mapping`
10. Verify that component templates exist for
`.alerts-framework-mappings` and `.alerts-legacy-alert-mappings`
**Compare the ECS component templates**
Compare 1 and 4 (ECS component template from `main` and installed by
rule registry in this branch). The difference should be:
* no difference in ECS fields
* because the rule registry ECS component template also includes
technical fields, you will see the 2 new technical fields in this branch
Compare 4 and 7 (ECS component template from rule registry & alerting
framework in this branch).
* some new ECS fields for alerting installed template
* each `keyword` mapped field for alerting installed template should
have `ignore_above` setting
* no `kibana.*` fields in the alerting installed template
**Compare the technical component templates**
Compare 2 and 5 (technical component template from `main` and installed
by rule registry in this branch). The difference should be:
* 2 new `kibana.alert` fields (`flapping_history` and `last_detected`)
Compare 5 and 8 (technical component template from rule registry &
alerting framework in this branch).
* there should be no difference!
**Compare the index mappings**
Compare 3 and 6 (index mapping from `main` and installed by rule
registry in this branch). The difference should be:
* 2 new `kibana.alert` fields (`flapping_history` and `last_detected`)
Compare 6 and 9 (index mapping from rule registry & alerting framework
in this branch).
* some new ECS fields
* each `keyword` mapped ECS field should have `ignore_above` setting
### Verify that the generated component templates work with existing
rule registry index templates & indices:
1. Run `main` or a previous version and create a rule that uses both ECS
component templates & technical component templates (detection rules use
both). Let it run a few times.
2. Using the same ES data, switch to this branch with
`xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts
with no rule registry errors and the rule continues to run as expected.
3. Using the same ES data, switch to this branch with
`xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts
with no alerting or rule registry errors and the rule continues to run
as expected. Verify that the mapping on the existing
`.internal.alerts-security.alerts-default-000001` has been updated to
include the latest ECS mappings and the two new technical fields.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
## Summary
Adds the `migrator` special role to the node roles config:
```yml
# 👇🏻 new
node.roles: ['migrator']
# or
node.roles: ['background_tasks', 'ui']
# or
node.roles: ['*'] # this one is slightly weird now because it actually excludes 'migrator' so it is not truly "all roles", but "all combinable roles"...
```
## How to test
Start Kibana locally and add `node.roles: ['migrator']` to the
`kibana.dev.yml`. Kibana should start normally and log:
```
[2023-02-23T12:08:54.123+01:00][INFO ][node] Kibana process configured with roles: [migrator]
```
Note: this role currently does not do anything. This PR just adds the
ability to configure it.
Partially addresses https://github.com/elastic/kibana/issues/150295
## Slight improvement to error messages
When specifying known, accepted values but combining with either
`migrator` or `*` you will get a message like:
```
[config validation of [node].roles]: wildcard ("*") cannot be used with other roles or specified more than once
```
---------
Co-authored-by: Luke Elmers <lukeelmers@gmail.com>
## Summary
Part of https://github.com/elastic/kibana/issues/150309
Purpose of the PR is to create the skeleton of the ZDT algorithm, in
order to make sure we're all aligned on the way we'll be managing our
codebase between the 2 implementation (and to ease with the review of
the follow-up PRs by not having the bootstrap of the algo to review at
the same time)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/151985
- ~~This PR introduces `maxUploadSize` file kind setting, which allows
to enforce upload size per file, instead of using one value for the
whole file kind~~.
- ~~I left the `maxSizeBytes` as-is for now, because it is actually used
in two places: (1) in the Files client; (2) in HTTP routes. So, I didn't
find an easy way to reuse it~~.
- Allows to configure max upload size per file.
- This required separation of `FileKind` interface between browser and
server. And correspondingly changes to the file kind registry.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR improves the detection of when a given plugin comes from inside
a build folder without including any external dependency for a local
package (not supported here due to kbn_pm).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Addresses https://github.com/elastic/kibana/issues/145717
Updated the all exceptions lists view to account for different possible
states (empty search, loading, no lists) using an existing component.
## Summary
Related to #151613
There might be cases when we need to add extra wait for Kibana plugin to
be ready before starting loading test data with `esArchiver` /
`kbnArchiver`.
Currently journey lifecycle looks like this:
- `onSetup` wrapped with mocha `before` hook
- in parallel
- `setupBrowserAndPage` (including EBT tracker setup)
- load ES data / Kibana saved objects
- `setupApm`
- steps execution (each step wrapped with mocha `it` function)
- `onTeardown` wrapped with mocha `after` hook
- `tearDownBrowserAndPage` (including closing EBT tracker)
- `teardownApm`
- load ES data / Kibana saved objects
beforeSteps hook purpose is to make sure Kibana/ES state is ready for
journey execution and not load test data, since it won't be unloaded
during `after` hook:
- `onSetup` wrapped with mocha `before` hook
- `setupBrowserAndPage` (including EBT tracker setup)
- run beforeSteps hook -> prepare Kibana/ES for data ingestion / steps
execution
- load ES data / Kibana saved objects
- `setupApm`
How to use:
```
export const journey = new Journey({
beforeSteps: async ({ kibanaServer, retry }) => {
retry.try(async () => {
const response = await kibanaServer.request({
path: '/internal/cloud_security_posture/status?check=init',
method: 'GET',
});
return response.status === 200;
});
},
esArchives: [...],
kbnArchives: [...],
})
.step({...})
.step({...})
```
Resolves: [#149587](https://github.com/elastic/kibana/issues/149587)
This PR intends to add a `uuid` field to the rule actions.
A migration script add a uuid to all the existing actions.
Create method of the rule_client (and create endpoint) accepts a
rule.actions data without uuid and adds a new uuid to the all actions.
Update and bulkEdit methods of the rule_client (and update and bulk_edit
endpoints) accepts rule.actions data with and without uuid. As a user
can add a new action to an existing rule, UI should send the uuid of an
existing action back then update and bulkEdit methods would add the
uuid's to all the new actions.
All the get methods return uuid in the actions.
Since we don't query by the uuid of an action, I marked actions as
`dynamic: false` in the mappings so we don't need to add the uuid to the
mappings.
I tried not to modify the legacy APIs, therefore i used some type
castings there, but please let me know if they need to be modified as
well.
## To verify:
Create a rule with some actions and save.
Then add some more actions and expect all of them to have an
auto-generated uuid field and still work as they were.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR contains the initial designs for our versioned router API. This
contribution contains only types, any implementation will come in later
PRs.
Previous PR https://github.com/elastic/kibana/pull/149943
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Ahmad Bamieh <ahmadbamieh@gmail.com>
Fixes https://github.com/elastic/kibana/issues/151219 and
https://github.com/elastic/kibana/issues/151224
PR separates shouldRefresh logic from unsavedChanges logic to account
for difference in filter check.
shouldRefresh filter check:
* includes pinned filters
* excludes disabled filters
* excludes $state so pinning/unpinning a filter does not cause a
refresh.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Change validation logic for entry exception field.
Close:
[https://github.com/elastic/kibana/issues/143051](https://github.com/elastic/kibana/issues/143051)
Previously we didn't keep a validation state per field which caused a
reset of validation if we still had invalid fields. Or we can have an
invalid state for the form, but we removed the invalid field. You can
see the videos on the ticket above.
## Solution:
Keep validation state per field, like:
```js
{
[entry.id]: true,
}
```
This state can keep old fields, which already were removed, this is why
we use the selector to get the actual amount of errors.
https://user-images.githubusercontent.com/7609147/220337447-95c1558c-aa85-43d1-87e8-76370aeaf141.mov
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Fixes#110422
TL;DR: The `lodash.set` function is unsafe and shouldn't be called.
Cause of error: If you specify multiple `no-restricted-imports` paths
for the same module, only the last path is used. Instead you need to
combine them into a single path as I've done in this PR.
This regression was introduced in #100277
## Summary
Added the `meta` prop to the `FilePicker` component, also pass this down
to the `FileUpload` component so that files created via the picker can
have meta set.
Close https://github.com/elastic/kibana/issues/151375
## How to test
1. Start Kibana with examples `yarn start --run-examples`
2. Go to the "Developer examples" in the side-nav menu under analytics
3. Go to "Files example"
4. Upload a file via the "Select file" button, should present an empty
file picker if you have no files, otherwise use the little upload
component bottom left
5. Either select files or dismiss the modal
6. Inspect the uploaded file and see the `myCool: 'meta'` entry included
with other metadata
## Screenshot
A file uploaded via the `FilePicker` in the "Files example" plugin.
<img width="897" alt="Screenshot 2023-02-16 at 11 41 06"
src="https://user-images.githubusercontent.com/8155004/219342872-c39b5d81-7421-4187-bb1c-d6815d80a3dc.png">
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Currently journeys use pre-defined base FTR config and there is no way
to re-configure Kibana to be loaded with extras, e.g. Fleet plugin
configuration.
f443109eea/packages/kbn-journeys/journey/journey_ftr_config.ts (L32-L34)
Probably the easiest way to address it is to path custom FTR config
directly in the journey.
```
export const journey = new Journey({
ftrConfigPath: 'x-pack/test/cloud_security_posture_api/config.ts',
...
})
```
It can be also considered as a step towards using journeys as future
alternative to Webdriver functional tests.
