# Backport
This will backport the following commits from `main` to `8.12`:
- [[Obs AI Assistant] E2E tests for conversation view
(#173485)](https://github.com/elastic/kibana/pull/173485)
<!--- Backport version: 8.9.8 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"Dario
Gieselaar","email":"dario.gieselaar@elastic.co"},"sourceCommit":{"committedDate":"2023-12-19T08:49:58Z","message":"[Obs
AI Assistant] E2E tests for conversation view (#173485)\n\nAdds some
basic E2E tests for the conversation
view.\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"f67afe2866014cfbc21755134950d7abaa8cf8c4","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v8.12.0","v8.12.1","v8.13.0"],"number":173485,"url":"https://github.com/elastic/kibana/pull/173485","mergeCommit":{"message":"[Obs
AI Assistant] E2E tests for conversation view (#173485)\n\nAdds some
basic E2E tests for the conversation
view.\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"f67afe2866014cfbc21755134950d7abaa8cf8c4"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","labelRegex":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/173485","number":173485,"mergeCommit":{"message":"[Obs
AI Assistant] E2E tests for conversation view (#173485)\n\nAdds some
basic E2E tests for the conversation
view.\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"f67afe2866014cfbc21755134950d7abaa8cf8c4"}}]}]
BACKPORT-->
## Summary
In this PR we:
* Allow using JWT credentials to grant API keys
* Extend default value of `elasticsearch.requestHeadersWhitelist` to
include both `authorization` and `es-client-authentication` to support
JWT with required client authentication _by default_. See
https://www.elastic.co/guide/en/elasticsearch/reference/8.11/jwt-auth-realm.html#jwt-realm-configuration
* Add API integration tests for both JWTs with client authentication and
without it
__NOTE:__ We're not gating this functionality with the config flag
(`xpack.security.authc.http.jwt.taggedRoutesOnly`) as we did for the
Serverless offering. It'd be a breaking change as we already implicitly
support JWT authentication without client authentication, and to be
honest, it's not really necessary anyway.
## Testing
Refer to the `Testing` section in this PR description:
https://github.com/elastic/kibana/pull/159117.
Or run already pre-configured Kibana functional test server:
1. `node scripts/functional_tests_server.js --config
x-pack/test/security_api_integration/api_keys.config.ts`
2. Create a role mapping for JWT user:
```bash
curl -X POST --location "http://localhost:9220/_security/role_mapping/jwt" \
-H "Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-d "{
\"roles\": [ \"superuser\" ],
\"enabled\": true,
\"rules\": { \"all\": [{\"field\" : { \"realm.name\" : \"jwt_with_secret\" }}] }
}"
```
3. Send any Kibana API request with the following credentials:
```bash
curl -X POST --location "xxxx"
-H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2tpYmFuYS5lbGFzdGljLmNvL2p3dC8iLCJzdWIiOiJlbGFzdGljLWFnZW50IiwiYXVkIjoiZWxhc3RpY3NlYXJjaCIsIm5hbWUiOiJFbGFzdGljIEFnZW50IiwiaWF0Ijo5NDY2ODQ4MDAsImV4cCI6NDA3MDkwODgwMH0.P7RHKZlLskS5DfVRqoVO4ivoIq9rXl2-GW6hhC9NvTSkwphYivcjpTVcyENZvxTTvJJNqcyx6rF3T-7otTTIHBOZIMhZauc5dob-sqcN_mT2htqm3BpSdlJlz60TBq6diOtlNhV212gQCEJMPZj0MNj7kZRj_GsECrTaU7FU0A3HAzkbdx15vQJMKZiFbbQCVI7-X2J0bZzQKIWfMHD-VgHFwOe6nomT-jbYIXtCBDd6fNj1zTKRl-_uzjVqNK-h8YW1h6tE4xvZmXyHQ1-9yNKZIWC7iEaPkBLaBKQulLU5MvW3AtVDUhzm6--5H1J85JH5QhRrnKYRon7ZW5q1AQ'
-H 'ES-Client-Authentication: SharedSecret my_super_secret'
....for example....
curl -X GET --location "http://localhost:5620/internal/security/me" \
-H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2tpYmFuYS5lbGFzdGljLmNvL2p3dC8iLCJzdWIiOiJlbGFzdGljLWFnZW50IiwiYXVkIjoiZWxhc3RpY3NlYXJjaCIsIm5hbWUiOiJFbGFzdGljIEFnZW50IiwiaWF0Ijo5NDY2ODQ4MDAsImV4cCI6NDA3MDkwODgwMH0.P7RHKZlLskS5DfVRqoVO4ivoIq9rXl2-GW6hhC9NvTSkwphYivcjpTVcyENZvxTTvJJNqcyx6rF3T-7otTTIHBOZIMhZauc5dob-sqcN_mT2htqm3BpSdlJlz60TBq6diOtlNhV212gQCEJMPZj0MNj7kZRj_GsECrTaU7FU0A3HAzkbdx15vQJMKZiFbbQCVI7-X2J0bZzQKIWfMHD-VgHFwOe6nomT-jbYIXtCBDd6fNj1zTKRl-_uzjVqNK-h8YW1h6tE4xvZmXyHQ1-9yNKZIWC7iEaPkBLaBKQulLU5MvW3AtVDUhzm6--5H1J85JH5QhRrnKYRon7ZW5q1AQ' \
-H 'ES-Client-Authentication: SharedSecret my_super_secret' \
-H "Accept: application/json"
----
{
"username": "elastic-agent",
"roles": [
"superuser"
],
"full_name": null,
"email": null,
"metadata": {
"jwt_claim_sub": "elastic-agent",
"jwt_token_type": "access_token",
"jwt_claim_iss": "https://kibana.elastic.co/jwt/",
"jwt_claim_name": "Elastic Agent",
"jwt_claim_aud": [
"elasticsearch"
]
},
"enabled": true,
"authentication_realm": {
"name": "jwt_with_secret",
"type": "jwt"
},
"lookup_realm": {
"name": "jwt_with_secret",
"type": "jwt"
},
"authentication_type": "realm",
"authentication_provider": {
"type": "http",
"name": "__http__"
},
"elastic_cloud_user": false
}
```
__Fixes:__ https://github.com/elastic/kibana/issues/171522
----
Release note: The default value of the
`elasticsearch.requestHeadersWhitelist` configuration option has been
expanded to include the `es-client-authentication` HTTP header, in
addition to `authorization`.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Following the initial work in this
https://github.com/elastic/kibana/pull/166755
- Addresses part of https://github.com/elastic/kibana/issues/151902 for
List APIs tests
- Added a new folder under the `security_solution_api_integration`
called `lists_and_exception_lists` to hold the lists and exception lists
tests, and split the `List` APIs into two groups since the execution
time in Serverless was close to 30 mins
- Modified the
[x-pack/plugins/lists/common/schemas/response/exception_list_item_schema.mock.ts](https://github.com/elastic/kibana/pull/171992/files#diff-4e3545fdeb8c8d9467cfa1c4aa88194e189193a92fa6f1cf5f859b1ef1beb45c),
[x-pack/plugins/lists/common/schemas/response/exception_list_schema.mock.ts](https://github.com/elastic/kibana/pull/171992/files#diff-c3cc18faf07aab86e307185d41599c3596a3f8b360d3e4829591afa148283238)
,
[x-pack/plugins/lists/common/schemas/response/list_item_schema.mock.ts](https://github.com/elastic/kibana/pull/171992/files#diff-608579ca5e65da74f41319a58d81ab12cc3d79d389b087806c7b74949fbc6cc3),
[x-pack/plugins/lists/common/schemas/response/list_schema.mock.ts](https://github.com/elastic/kibana/pull/171992/files#diff-efc64eb35937a8da28fc982c527253c0923650ae4163d4bbc203d3ebc2949835)
to accept `elastic user` input because it changes in ESS and Serverless
- Deleted the `x-pack/test/lists_api_integration` folder
- Moved the utility files associated with Basic tests to the new
directory `security_solution_api_integration`. Files not actively used
in the previous folder were moved, while duplicate files remained in
their original positions.
- Updated the below files imports from the old `lists_api_integration`
folder to the new `lists_and_exception_lists`
```
lists_api_integration/
- exceptions/operators_data_types/date_numeric_types/date.ts
- exceptions/operators_data_types/date_numeric_types/double.ts
- exceptions/operators_data_types/date_numeric_types/float.ts
- exceptions/operators_data_types/date_numeric_types/integer.ts
- exceptions/operators_data_types/ips/ip.ts
- exceptions/operators_data_types/ips/ip_array.ts
- exceptions/operators_data_types/keyword/keyword.ts
- exceptions/operators_data_types/keyword/keyword_array.ts
- exceptions/operators_data_types/long/long.ts
- exceptions/operators_data_types/text/text.ts
- exceptions/operators_data_types/text/text_array.ts
- exceptions/workflows/create_endpoint_exceptions.ts
- exceptions/workflows/create_rule_exceptions.ts
- exceptions/workflows/find_rule_exception_references.ts
- exceptions/workflows/role_based_add_edit_comments.ts
- exceptions/workflows/role_based_rule_exceptions_workflows.ts
- exceptions/workflows/rule_exception_synchronizations.ts
- rule_execution_logic/execution_logic/esql.ts
- rule_execution_logic/execution_logic/machine_learning.ts
- rule_execution_logic/execution_logic/new_terms.ts
- rule_execution_logic/execution_logic/query.ts
- telemetry/task_based/all_types.ts
- telemetry/task_based/detection_rules.ts
- telemetry/task_based/security_lists.ts
```
```
These files should be moved too soon to the new `lists_api_integration`
detection_engine_api_integration/security_and_spaces/group10
- import_export_rules.ts
- import_rules.ts
- perform_bulk_actions
```
- Updated the below files imports to the
`ftr_provider_context_with_spaces.d.ts`
` - risk_engine/risk_scoring_task/task_execution_nondefault_spaces.ts`
- The QA phase concluded with all tests passing successfully. 🟢
- Updated the CodeOwner file for the newly moved tests
- Add a new util file to `deleteAllExceptions` under the old
`detection_engine_api` folder since the Rule management related-tests
are still need to be moved over to the new folder
- Old/new group details, decisions, and execution time are mentioned in
this
[document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Related to https://github.com/elastic/kibana/issues/169759.
## Summary
The Data streams stats endpoint can be queried like so (example with all
parameters):
`GET
/internal/dataset_quality/data_streams/stats?type=logs&sortOrder=asc&datasetQuery=nginx`
### Notes/questions
Our intial idea was to place this endpoint within fleet, after some
discussions we decided to place the endpoint inside our plugin
`dataset_quality`.
- ~~Is `/epm` still the best place to include this new api?~~
We decided to place the new endpoint in `dataset_quality` plugin.
- ~~Should we include an `uncategorisedOnly` option as part of the Data
Streams Stats endpoint as we do for [Data streams
endpoint](c1681c8d49/x-pack/plugins/fleet/server/types/rest_spec/epm.ts (L62))?
We are not planning to have the a filter for this but it could favor
consistency in this area.~~
uncategorizedOnly is not part of the new endpoint although it's easily
extensible to include such parameter.
- ~~We are modifying
`x-pack/plugins/fleet/server/services/epm/data_streams/get.ts` to
include a property for extendedResults, as of now this method only
returns a list of objects that include only the name. We would like to
extend this method to include also information related to the `package`.
Is this the right approach? should we create a new method just for doing
this? Also, this is the method we are using for `GET
/api/fleet/epm/data_streams` since this new property
(`extendedResponse`) is false by default the api is not being directly
affected, but should we include this new param as part of the api
params?~~
We are not working with fleet api anymore, instead we are using an
internal route.
- ~~How does `GetEpmDataStreamsStatsResponse` looks like? is this
reponse type aligned with fleet responses?~~
- ~~We have configured `fleetAuthz: READ_PACKAGE_INFO_AUTHZ` for this
new endpoint, would it be possible to not have this restriction and use
the internal user when querying elastic search?~~
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Following the initial work in this
https://github.com/elastic/kibana/pull/166755
- Addresses part of https://github.com/elastic/kibana/issues/151902 for
rule execution logic
- Moved the utility files associated with rule execution logic to the
new directory `security_solution_api_integration`. Files not actively
used in the previous folder were moved, while duplicate files remained
in their original positions.
- Updated the CodeOwner file for the newly moved tests
- Old/new group details, decisions, and execution time are mentioned in
this[document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit)
- Added new `Alert` archive for version `8.8.0`
- Resolved the issue with the `query.ts` test where the execution logic
is executed last, encompassing the "query" test because it was unloading
the alerts document and led to failures in subsequent tests.
- For `Alert As Data` in **Serverless** the alert ancestor will be a
data-stream however in **ESS** will be
`.internal.alerts-security.alerts-default-000001'`
| Action | File | New Path if moved |
|--------|------|----------|
| Delete| security_and_spaces/rule_execution_logic| - |
| Delete|security_and_spaces/group5 | - |
|
Move|detection_engine_api_integration/security_and_spaces/group5|detections_response/default_license/rule_execution_logic/keyword_family|
|
Move|detection_engine_api_integration/security_and_spaces/rule_execution_logic|
detections_response/default_license/rule_execution_logic/execution_logic
|
| Move
|detection_engine_api_integration/security_and_spaces/group1/ignore_fields|
detections_response/default_license/rule_execution_logic/ignore_fields.ts|
|
Move|detection_engine_api_integration/security_and_spaces/group1/runtime|
detections_response/default_license/rule_execution_logic/runtime.ts |
|
Move|detection_engine_api_integration/security_and_spaces/group1/timestamps|
detections_response/default_license/rule_execution_logic/timestamps.ts|
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR refactors the export type classes into their own packages to be
then instantiated in the reporting plugin. This will reduce bloat in the
central reporting plugin.
**Main packages**
- `kbn/reporting-export-types-{png,pdf,csv}` are server packages with
export type declarations
- `kbn/reporting-export-types-{png,pdf,csv}-common` are shared common
packages with type declarations and constants
**Other changes**
- Remove `reporting.getScreenshots()`
- Remove duplicated `schema_utils.ts`
- Consolidate `JOB_STATUS` declaration as an enum
<img width="1063" alt="image"
src="bced8321-93c5-4ebd-b31e-1fd946166241">
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Timothy Sullivan <tsullivan@elastic.co>
## Summary
- Following the initial work in this
https://github.com/elastic/kibana/pull/166755
- Addresses part of https://github.com/elastic/kibana/issues/151902 for
Alerts
- Introduced a new folder called `alerts` under
`security_solution_api_integration`
- Moved the utility files associated with alerts to the new directory
`security_solution_api_integration`. Files that were not actively used
in the previous folder were moved, while any duplicate files remained in
their original positions.
- Updated the CodeOwner file for the newly moved tests
- Old / new groups details and execution time
[document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit)
| Action | File | New Path if moved |
|--------|------|----------|
| Delete| group1/find_rule_exception_references.ts| Already moved in
previous PR |
| Delete|security_and_spaces/group6 | - |
| Move|detection_engine_api_integration/security_and_spaces/group6
|detections_response/default_license/alerts/alerts_compatibility.ts|
| Move|detection_engine_api_integration/security_and_spaces/group1|
detections_response/default_license/alerts/aliases.ts |
| Move |detection_engine_api_integration/security_and_spaces/group1|
detections_response/default_license/alerts/create_index.ts|
|
Move|detection_engine_api_integration/security_and_spaces/group10/create_signals_migrations|
detections_response/default_license/alerts/migrations/create_alerts_migrations.ts
|
|
Move|detection_engine_api_integration/security_and_spaces/group10/delete_signals_migrations|
detections_response/default_license/alerts/migrations/delete_alerts_migrations.ts|
|
Move|detection_engine_api_integration/security_and_spaces/group10/finalize_signals_migrations|
detections_response/default_license/alerts/migrations/finalize_alerts_migrations.ts
|
|
Move|detection_engine_api_integration/security_and_spaces/group10/get_signals_migration_status|
detections_response/default_license/alerts/migrations/get_alerts_migration_status.ts|
|
Move|detection_engine_api_integration/security_and_spaces/group10/open_close_signals|
detections_response/default_license/alerts/open_close_alerts.ts|
|
Move|detection_engine_api_integration/security_and_spaces/group10/alert_tags|
detections_response/default_license/alerts/alert_tags.ts|
Note: these tests are skipped on the main branch
https://github.com/elastic/kibana/blob/main/x-pack/test/detection_engine_api_integration/security_and_spaces/group10/open_close_signals.ts#L215https://github.com/elastic/kibana/blob/main/x-pack/test/detection_engine_api_integration/security_and_spaces/group10/open_close_signals.ts#L252https://github.com/elastic/kibana/blob/main/x-pack/test/detection_engine_api_integration/security_and_spaces/group10/finalize_signals_migrations.ts#L192https://github.com/elastic/kibana/blob/main/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/create_index.ts#L42
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/166834
Moves the following components out of infra plugin so we can remove the
apm->infra dependency
- `/infra/metrics_explorer` route
- (Host|Pod|Container)MetricsTable components
- InfraAppId
- InfraLocators
### Testing
We should focus the testing on the metrics_explorer route which is used
by 1. Tables in the Infrastructure section of apm and 2. Infrastructure
> Metrics explorer app. No functionality was added so these apps should
have the same existing behavior. Easiest way to get these views loaded
is by connecting kibana to an edge-oblt cluster, alternatively load
service and metrics with a data loader.
### Follow up
- (Host|Pod|Container)MetricsTable components are part of the
metricsDataAccess plugin contract but should be moved to stateless
package
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Jason Rhodes <jason.rhodes@elastic.co>
# Summary
- Addresses https://github.com/elastic/kibana/issues/161537
## Description
- This PR follows the second option defined in this
[document](https://docs.google.com/document/d/1mqkpjDdFQRFvx_RPvNmjstVj8SXYMr2mrETMv3esda8/edit#heading=h.rpv1zyeb04ay)
the [Mocha tagging ](https://github.com/mochajs/mocha/wiki/Tagging)
- It introduces a new folder
`x-pack/test/security_solution_api_integration` which will serve as a
centralized location to meet all the requirements related to renaming
tests appropriately and grouping similar tests together. It will
facilitate the management of tests that must be run in Serverless and
ESS environments.
- Within this folder, there is a "config" subdirectory that stores base
configurations specific to both the
[Serverless](https://github.com/elastic/kibana/pull/166755/files#diff-afe1f42d5ac2006de8dc09069448b9e8734a6a950586376cd6e8eeb9110ab5f1R1)
and
[ESS](https://github.com/elastic/kibana/pull/166755/files#diff-4a60bd8c91da08a3f7ec14bf3bfef8449af155611374c32579b0318da03e292cR1)
environments, These configurations build upon the base configuration
provided by test_serverless and api_integrations, incorporating
additional settings such as environment variables and tagging options.
- It demonstrates scenarios involving `@ess`, `@serverless`, and
`@brokenInServerless`.
- The file`
x-pack/test/security_solution_api_integration/test_suites/detections_response/rule_creation/create_rules.ts`
is functional in both **Serverless** and **ESS**. However, some tests
related to roles are currently skipped for Serverless until they are
resolved, and these tests are tagged with `@brokenInServerless`
## CI
- It includes a new entry in the ftr_configs.yml to execute the newly
added tests in the pipeline.
- It involves the addition of mochaOptions in both
serverless/config.base.ts and ess/config.base.ts. In the case of
serverless, it includes **@serverless** while excluding
@brokenInServerless. Similarly, for **ess**, it includes @ess and
excludes **@brokenInEss**.
from `x-pack/test/security_solution_api_integration/config/serverless`
## Update in x-pack/test/detection_engine_api_integration
- The `create_rules.ts` and `create_rule_exceptions` files have been
relocated from
`x-pack/test/detection_engine_api_integration/security_and_spaces/group1`
to their respective domains within the
`x-pack/test/security_solution_api_integration` folder.
- The util files now are copied over from the old folder
`x-pack/test/detection_engine_api_integration` to the new folder and
will be removed once all tests are moved to the new folder to don't
break the existing tests
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Let's automate E2E against Serverless
Changelog:
- updated certs to include additional dns names we are using for testing
locally, `host.docker.internal`, `es01`
- updated certs generation README to include changes related to
`openssl@3`
- added new certs for Fleet server
- added fleet-server service token
- added support for `ca_trusted_fingerprint` in fleet preconfig
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tomasz Ciecierski <ciecierskitomek@gmail.com>
Co-authored-by: Tomasz Ciecierski <tomasz.ciecierski@elastic.co>
Co-authored-by: Kevin Logan <kevin.logan@elastic.co>
## Summary
This PR introduces the new color mapping feature into Lens.
The color mapping feature is introduced as a standalone sharable
component available from `@kbn/coloring`. The
[README.md](ddd216457d/packages/kbn-coloring/src/shared_components/color_mapping/README.md)
file describes the components and the logic behind it.
The Color Mapping component is also connected to Lens and is available
in the following charts:
- XY (you can specify the mappings from a breakdown dimension
- Partition (you can specify the mappings from the main slice/group by
dimension)
- Tag cloud (you can specify the mappings from the tags dimension)
This MVP feature will be released under the Tech Preview flag.
This PR needs to prove the user experience and the ease of use. UI
styles, design improvements and embellishments will be released in
subsequent PRs.
The current MVP-provided palettes are just a placeholder. I'm
coordinating with @gvnmagni for a final set of palettes.
close https://github.com/elastic/kibana/issues/155037
close https://github.com/elastic/kibana/issues/6480
fix https://github.com/elastic/kibana/issues/28618
fix https://github.com/elastic/kibana/issues/96044
fix https://github.com/elastic/kibana/issues/101942
fix https://github.com/elastic/kibana/issues/112839
fix https://github.com/elastic/kibana/issues/116634
## Release note
This feature introduces the ability to change and map colors to break
down dimensions in Lens. The feature provides an improved way to specify
colors and their association with categories by giving the user a
predefined set of color choices
or customized one that drives the user toward a correct color selection.
It provides ways to pick new colors and generate gradients.
This feature is in Tech Preview and is enabled by default on every new
visualization but can be turned off at will.
This PR removes the Profiling dependency from APM, introduced on `8.10`.
- Exposes a new service in profiling-data-access plugin
- Create a new APM API that calls the new service and checks if
Profiling is initialized
- Move Locators from the Profiling plugin to the Observability-shared
plugin
- Move logic to check Profiling status (has_setup/has_data...) from
Profiling server to profiling-data-access plugin
- Create API tests, testing the status services based on different
scenarios:
- When profiling hasn't been initialized and there's no data
- When profiling is initialized but has no data
- When collector integration is not installed
- When symbolized integration is not installed
- When APM server integration is not found
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
- Move logic to fetch the TopN functions to
'profiling-data-access-plugin'
- Create new TopN functions embeddable
- Refactor Universal profiling page on APM adding Tabs (Flamegraph | Top
10 functions)
- Create a new API on APM to fetch Top functions
<img width="1605" alt="Screenshot 2023-09-08 at 11 14 13"
src="76fc7bf3-094b-438b-99b8-3cab01539eb4">
<img width="1615" alt="Screenshot 2023-09-08 at 11 14 20"
src="c4d3e97a-eee0-4829-8d6f-545f9c844b18">
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Joseph Crail <joseph.crail@elastic.co>
Closes https://github.com/elastic/kibana/issues/161906
Related to:
https://github.com/elastic/observability-dev/discussions/2787
(_internal_)
This add a new plugin `apm_data_access` that contains the APM query
targets (indices to query for APM data).
This plugin can be consumed by apm and any other plugin, making it
possible for other plugins to know about the configured APM query
targets.
## Example:
APM query targets can be specified in kibana[.dev].yml using
`xpack.apm.indices.{dataset}: some-index-*` for instances:
```yml
xpack.apm.indices.transaction: apm-*
```
See all config options on:
https://www.elastic.co/guide/en/kibana/current/apm-settings-kb.html#general-apm-settings-kb
Query targets can also be specified via the UI (and persisted in a saved
object) via the settings page: `/app/apm/settings/apm-indices`
**Retrieving the query targets**
Query targets can be retrieved from other plugins via `getApmIndices`:
```ts
const apmIndices = await plugins.apmDataAccess.setup.getApmIndices(savedObjects.client);
```
TODO:
- [x] Add SO client and fetch space aware index config (see
4d7f626da0/x-pack/plugins/apm/server/routes/settings/apm_indices/get_apm_indices.ts (L32-L44))
- [ ] Add simple APM client for querying apm data
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves: #159493
This PR replaces `AlertFactory` in ES Query rule type with
`AlertsClient` so the alerts are persistent in an alert-as-data index.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This updates log rate analysis to be able to auto-detect whether the
selected deviation is a spike or dip compared to the baseline time
range. To achieve this, we compare the median bucket size of the two
selections. If a dip gets detected, the analysis will then switch the
window parameters sent to the API endpoint to run the analysis.
An info callout points out the auto-selected analysis type and explains
to which time range the analysis results refer to. We need to do this to
make it clear that for dip analysis the significant terms and their doc
counts refer to the baseline time range and vice versa for spike
analysis.
## Summary
Closes#160812
Adds UI for CA and client-side SSL certificate auth to webhook
connector, and adds these properties to the webhook's HttpsAgent on
execution:
<img width="912" alt="Screenshot 2023-07-18 at 11 59 33 AM"
src="b1986c8d-3632-4663-80ff-3fb37f706d07">
### When Editing
<img width="868" alt="Screenshot 2023-07-14 at 3 34 57 PM"
src="e3e0f450-1196-45fd-88a6-1e15b6f2fa36">
Also creates a Field helper for EuiFilePicker
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR makes the following changes:
- Update look & feel of contextual insights (previously called prompts)
according to the new design that is being developed. Some things might
still change, but hopefully not too much.
- Move all the Observability AI Assistant (previously called CoPilot)
code into a separate plugin for better isolation, more specific code
ownership and to solve some circular dependency issues
- Use connectors instead of a kibana.yml setting
Note: for OpenAI, the model is currently hardcoded to `gpt-4` until
https://github.com/elastic/kibana/issues/162204 has been addressed.
557676b6-065a-4b6f-86b2-1f0c2fd5e07e
---------
Co-authored-by: Coen Warmer <coen.warmer@gmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR versions the `cases` and `comment` domain objects and their
corresponding APIs. It was not possible to do them separately as I got
errors due to circular dependencies.
## Notable Changes
- The `Comment` type was renamed to `Attachment`
- The `Comments` type was renamed to `Attachments`
- The `*CommentRequestRt` type was renamed to `*AttachmentPayload`
- The `CommentType` type was renamed to `AttachmentType`
- The `AttributesType*` type was renamed to `*AttachmentAttributes`
- The `*ResponseTypeUserRt` type was renamed to `*AttachmentRt`
- The word `comment` got replaced with the word `attachment` in all
types
- The `RelatedCaseInfo` type was renamed to `RelatedCase`
- The `CasesByAlertId` type was renamed to
`GetRelatedCasesByAlertResponse`
Depends on: https://github.com/elastic/kibana/pull/161783,
https://github.com/elastic/kibana/pull/162059
### Checklist
Delete any items that do not apply to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Moves the server and client side code which performs analysis on data to
see whether it is suitable for categorization.
This is currently only used by the categorization job wizard to display
this callout:
However this analysis will be useful for the Log Pattern Analysis
feature and so moving the code to a package allows easier sharing
between ML and AIOPs plugins.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## 📓 Summary
Closes#159128
Due to a dependencies issue when disabling a plugin in serverless mode,
the LogStream feature and related logic were disabled for every
consumer.
We decided to split this shared component and endpoint into their own
plugin of shared logs utilities, reducing to the minimum the required
dependency that could disable the plugin.
What we moved can be summarized with:
- `infrastructure-monitoring-log-view` saved object definition and
registration
- LogViews server/client services (exposed with start contract) +
related endpoints
- LogEntries server service + related endpoints
- LogEntriesDomain logic (exposed with start contract)
- `<LogStream />` component
- `<ScrollableLogTextStreamView />` component and related logic
- LogView state machine
- Containers/Hooks to consume the moved APIs.
- Common types/utils definition, now exported and consumed as a
dependency from the `infra` plugin.
## 🤓 Review hints
Most of the changes are just renaming and moving stuff into the new
plugin, but for some operations was required to implement new logic,
which may deserve a more critical review:
- server/public `plugin.ts` files for the `infra` and `logs_shared`
plugins. The new plugin now registers the fallback actions to retrieve a
source configuration if there's no stored log view. It also set the
configuration for the message field and registers the log view saved
object.
- the `logEntriesDomain` has also been moved inside the new plugin, but
is also used by the logs-analysis endpoints, so it is exposed by the
logs_shared plugin and consumed by `infra`.
## 👣 Following steps
We currently are still using the `observability` plugin for consuming
the CoPilot feature on our LogsStream flyout.
The plugin dependency is marked as optional, so disabling the
`observability` plugin in a serverless environment won't disable also
the exposed features in this new plugin, but it'll affect only the
CoPilot feature, which won't be loaded.
In future, would be nice to extract the CoPilot feature into its own
package/plugin, so that also serverless projects can consume it without
depending on `observability.
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Relates to https://github.com/elastic/kibana/issues/159451.
This PR adds api integration tests to:
- [x] GET /internal/observability_onboarding/custom_logs/privileges
- [x] GET
/internal/observability_onboarding/custom_logs/install_shipper_setup
- [x] POST /internal/observability_onboarding/custom_logs/save
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
As part of the actions for making Profiling production ready, this PR
adds basic API tests on the Profiling APIs checking if only users with
`access:profiling` are allowed to call our APIs, other users must be
forbidden.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Creates Package `@kbn/ml-data-frame-analytics-utils` via `node
scripts/generate package @kbn/ml-data-frame-analytics-utils --web --dir
./x-pack/packages/ml/data_frame_analytics_utils`.
Moves some of the constants, types and utilities for Data Frame
Analytics to its own package. This is in preparation to move our data
grid related code to a package too. Since this code for now is only used
by our team I didn't do any renaming related to for example consistent
prefixing, this should be revisited in a follow up. Another opportunity
for a follow up might be a clean up of the types and check which ones
can be replaced by types from `estypes`.
Adds versioning to all of the file upload APIs.
Versions are added to the server side routes and to the client side
functions which call the routes.
Updates API tests to add the API version to the request headers.
All of the APIs are internal and have been given the version '1'.
Also renames `/internal/file_data_visualizer/analyze_file` to
`/internal/file_upload/analyze_file`
It appears this was a mistake from when the route was moved from the
data visualiser plugin midway through development on [this
PR](https://github.com/elastic/kibana/pull/96408).
**Internal APIs**
`/internal/file_upload/analyze_file`
`/internal/file_upload/has_import_permission`
`/internal/file_upload/index_exists`
`/internal/file_upload/time_field_range`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes#155245
Adds a version to the `endpoint` name for public APIs. It's required for
public APIs and enforced via type checks. Example:
`GET /api/apm/settings/agent-configuration/environments 2023-05-22`. The
reason why it's part of the endpoint is because it's the simplest change
to make, requires very little runtime changes and doesn't dramatically
change the way we handle type-safety for our server routes.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Fixes https://github.com/elastic/kibana/issues/157219
On the client, `URLSearchParams.set` encodes strings. On the server,
`query` values are passed to the route decoded. Therefore, there is no
need for special encoding or decoding of requestBody, other then rison
encoding.
PR creates `getTileUrlParams` to standardize vector tile URL creation
across sources. `getTileUrlParams` is placed in a package so it can be
used in integration tests. This greatly increases the maintainability of
integration tests as it makes them ore readable and creates tileUrls
just like the client does to ensure testing is as close to the real
thing as possible.
PR also updates getTileUrl generation to only pick requestBody keys used
by vector tile routes to avoid sending unnecessary data to the server.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes#157189
This PR adds a metric threshold integration test. This is the first step
in adding more test coverage for observability rules.
**Steps during the test**
1. Generating fake host data by using a similar implementation as
https://github.com/elastic/high-cardinality-cluster
- Data is generated for the last 15 mins
- Implementation was simplified only to cover fake hosts and was
converted to typescript
2. Creating an action using an index connector
3. Creating a metric threshold rule containing step number 2 action
4. Checking the status of the rule to be active
5. Checking the triggered action to have the correct parameters
6. Checking the generated alert to have the correct information
7. Clean up
**How to run locally**
- Run server
```
node scripts/functional_tests_server --config x-pack/test/api_integration/apis/metrics_ui/config.ts
```
- Then run the test
```
node scripts/functional_tests__runner --include-pack/test/api_integration/apis/metrics_ui/cometric_threshold_rule.ts --config x-pack/test/api_integration/apis/metrics_ui/config.ts
```
**Reference**
I created https://github.com/elastic/integrations/issues/6168 to find a
better way to generate data and make sure that data matches what
metricbeats generates
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Creates Package `@kbn/ml-anomaly-utils`.
- This moves some of the utility functions, constants and types related
to Anomaly Detection we previously exposed from the `ml` plugin itself
to a package `@kbn/ml-anomaly-utils`, resulting in ~5.9KB reduction of
the plugin's page load bundle size.
- To reduce increases in async bundle size for consuming plugins the
utils have been refactored into individual files to allow deep imports
and get some optimization through that. Some previously duplicated code
in consuming plugins has been deleted and replaced with deep imports of
the corresponding original code in the package.
- Types have been prefixed with `Ml`.
- Constants have been prefixed with `ML_`.
Created package via `node scripts/generate package @kbn/ml-anomaly-utils
--web --dir ./x-pack/packages/ml/anomaly_utils`.
## Summary
Fixes https://github.com/elastic/kibana/issues/155611
This PR changes the API prefix to indicate that the endpoints are
intended for internal use only. The plugin has been setup incorrectly
initially with the prefix `/api` that is intended for a public API.
There should not be any changes to the UI or functionality of the
plugin.
### Release Note
Fixed the guided onboarding API prefix to indicate that it's intended
for internal use
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
resolves https://github.com/elastic/kibana/issues/142874
The alerting framework now generates an alert UUID for every alert it
creates. The UUID will be reused for alerts which continue to be active
on subsequent runs, until the alert recovers. When the same alert (alert
instance id) becomes active again, a new UUID will be generated. These
UUIDs then identify a "span" of events for a single alert.
The rule registry plugin was already adding these UUIDs to it's own
alerts-as-data indices, and that code has now been changed to make use
of the new UUID the alerting framework generates.
- adds property in the rule task state
`alertInstances[alertInstanceId].meta.uuid`; this is where the alert
UUID is persisted across runs
- adds a new `Alert` method getUuid(): string` that can be used by rule
executors to obtain the UUID of the alert they just retrieved from the
factory; the rule registry uses this to get the UUID generated by the
alerting framework
- for the event log, adds the property `kibana.alert.uuid` to
`*-instance` event log events; this is the same field the rule registry
writes into the alerts-as-data indices
- various changes to tests to accommodate new UUID data / methods
- migrates the UUID previous stored with lifecycle alerts in the alert
state, via the rule registry *INTO* the new `meta.uuid` field in the
existing alert state.
## Summary
This plugin will contain the asset inventory and topology API in Kibana,
giving Kibana projects access to inventory and topology data via an HTTP
and/or JS API on the server and client.
[Currently proposed API
docs](https://github.com/elastic/o11y-topology-playground/tree/main/docs/api)
will be moved to this repo as well, contained inside this plugin folder,
as a part of this PR.
## Enabling the plugin
This plugin is entirely in "technical preview" and because of this, must
be specifically enabled via config for it to do anything besides being
run by the core plugin framework. To enable the server API layer, as
well as the index template management, put the following line in your
kibana.yml file:
```yml
xpack.assetManager.alphaEnabled: true
```
## Running the API integration tests
Run the functional test server with the asset manager config in place:
```shell
$ node scripts/functional_tests_server --config x-pack/test/api_integration/apis/asset_manager/config.ts
```
Then run the functional test runner with the same config, to target just
these tests:
```shell
$ node scripts/functional_test_runner --config=x-pack/test/api_integration/apis/asset_manager/config.
ts
```
_Note:_ The config file added in this folder enables the tech preview
plugin ([see file
here](https://github.com/elastic/kibana/pull/152456/files#diff-bc00de6c34c9bc131cfbdf3570c487fe9ee947e9a88a84c59d6b139b79d7708eR20)).
### Running the integration tests for verifying that the plugin is
"disabled" by default
There is a small set of tests that confirm that the endpoints return 404
and there is no index template installed if the config value is not set
in the kibana.yml file. To run this suite, use the following config:
```shell
$ node scripts/functional_tests_server --config x-pack/test/api_integration/apis/asset_manager/config_when_disabled.ts
$ node scripts/functional_test_runner --config=x-pack/test/api_integration/apis/asset_manager/config_when_disabled.
ts
```
## Testing this PR with sample data
There are some sample data mechanisms in place inside this PR to allow
us to build out the endpoints.
### View sample docs
```http
GET /api/asset-manager/assets/sample
```
This will return a list of the assets that are included if you elect to
write assets. This is a good endpoint to use to find EAN (Elastic Asset
Name) values that you may want to exclude from writing for a given time
period, to simulate assets appearing/disappearing over time.
### Write sample docs
```http
POST /api/asset-manager/assets/sample
{
"baseDateTime": "2023-02-28T12:00:00.000Z",
"excludeEans": ["k8s.cluster:cluster-002"]
}
```
This posts all of the sample asset documents to Elasticsearch using the
`baseDateTime` value as the timestamp. Any valid string or number that
is accepted by `new Date()` should work for `baseDateTime`.
The `excludeEans` value is an array of EAN ("Elastic Asset Name") values
that you don't want to write on this particular run. This way you can
have assets appear (exclude them in the past, don't exclude them during
a later run) or disappear (vice versa) and see how that shows up in
other endpoints.
**Note:** *Remember that when you curl a Kibana server API with a POST
request, you must include a `kbn-xsrf` header with any string value you
want.*
### Get asset docs from ES
```http
GET /api/asset-manager/assets?type=k8s.cluster&from=now-10m
```
This is the primary "real" endpoint available right now. It should
retrieve a list of assets based on the type/from/to/ean filter values
you specify. Once you load the sample data, this endpoint should return
results.
## Debug logging
There are some extra debug logs for ES queries that are running in the
code in this PR. To print those logs to the Kibana server console, run
Kibana using `DEBUG_LOGGER=true`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR registers three file kinds for cases. One for each instance of
cases (stack, observability, and security). Each solution needs separate
http tags for the routes that are generated by the file service to
implement RBAC.
I refactored the logic to remove some duplication across the three
plugins since we're essentially registering the same http tags with
slightly different names.
This PR shouldn't affect any of the current functionality.
Notable changes:
- I split up the constants.ts file, really the only change is adding the
file kinds logic to generate the http tags the rest is copy/paste
- Refactored the logic to generate the `api` http tags for each plugin
- Registered the three file kinds
Issues: https://github.com/elastic/kibana/issues/151780https://github.com/elastic/kibana/issues/151933
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Added several tests for terms query rules, which tests that match and
term query rules are the same.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Resolves https://github.com/elastic/kibana/issues/150358
## Summary
In a previous [PR](https://github.com/elastic/kibana/pull/145581) we
started installing a common component template for framework alerts as
data when the `xpack.alerting.enableFrameworkAlerts` config flag is set
to true. In that PR we used a different naming pattern than what is used
by the rule registry for its component templates.
In this PR we are doing the following:
* Renaming the installed `alerts-common-component-template` to
`.alerts-framework-mappings`.
* Creating and installing `.alerts-legacy-alert-mappings` component
template when `enableFrameworkAlerts: true` on alerting plugin setup
* The combination of the two component templates creates the same set of
mappings as the rule registry technical component template
* Creating and installing `.alerts-ecs-mappings` component template when
`enableFrameworkAlerts: true` on alerting plugin setup (when
`enableFrameworkAlerts: false`, the rule registry continues to install
this component template
* Using the `@kbn/ecs` package provided by core to generate the ECS
field map. The rule registry will continue to install the existing ECS
field map which is actually a subset of ECS fields
* Adding `useLegacy` and `useEcs` flags that allow rule types to specify
whether to include the legacy alerts component template and the ECS
component template when registering with framework alerts-as-data.
* Moved some common functions to alerting framework from the rule
registry
## Things to note
* When generating the ECS field map, we are now including the
`ignore_above` setting from the `@kbn/ecs` package. This changes the ECS
component template to include those settings. I tested updating an index
with just `"type":"keyword"` mappings to add the `ignore_above` field to
the mapping and had no issues so this seems like an additive change to
the mapping that will hopefully prevent problems in the future.
* The rule registry ECS component template also includes the technical
fields which is redundant because the technical component template is
automatically installed for all index templates so the framework ECS
component template only contains ECS fields.
| Previous mapping | Updated mapping |
| ----------- | ----------- |
| `{ "organization": { "type": "keyword" } }` | `{ "organization": {
"type": "keyword", "ignore_above": 1024 } }` |
## To Verify
### Verify that the generated component templates are as expected:
Get the following
**While running `main`:**
1. Get the ECS component template `GET
_component_template/.alerts-ecs-mappings`
2. Get the technical component template `GET
_component_template/.alerts-technical-mappings`
3. Create a detection rule that creates an alert and then get the index
mapping for the concrete security alert index `GET
.internal.alerts-security.alerts-default-000001/_mapping`
**While running this branch with `xpack.alerting.enableFrameworkAlerts:
false`:**
4. Get the ECS component template `GET
_component_template/.alerts-ecs-mappings`
5. Get the technical component template `GET
_component_template/.alerts-technical-mappings`
6. Create a detection rule that creates an alert and then get the index
mapping for the concrete security alert index `GET
.internal.alerts-security.alerts-default-000001/_mapping`
**While running this branch with `xpack.alerting.enableFrameworkAlerts:
true`:**
7. Get the ECS component template `GET
_component_template/.alerts-ecs-mappings`
8. Get the technical component template `GET
_component_template/.alerts-technical-mappings`
9. Create a detection rule that creates an alert and then get the index
mapping for the concrete security alert index `GET
.internal.alerts-security.alerts-default-000001/_mapping`
10. Verify that component templates exist for
`.alerts-framework-mappings` and `.alerts-legacy-alert-mappings`
**Compare the ECS component templates**
Compare 1 and 4 (ECS component template from `main` and installed by
rule registry in this branch). The difference should be:
* no difference in ECS fields
* because the rule registry ECS component template also includes
technical fields, you will see the 2 new technical fields in this branch
Compare 4 and 7 (ECS component template from rule registry & alerting
framework in this branch).
* some new ECS fields for alerting installed template
* each `keyword` mapped field for alerting installed template should
have `ignore_above` setting
* no `kibana.*` fields in the alerting installed template
**Compare the technical component templates**
Compare 2 and 5 (technical component template from `main` and installed
by rule registry in this branch). The difference should be:
* 2 new `kibana.alert` fields (`flapping_history` and `last_detected`)
Compare 5 and 8 (technical component template from rule registry &
alerting framework in this branch).
* there should be no difference!
**Compare the index mappings**
Compare 3 and 6 (index mapping from `main` and installed by rule
registry in this branch). The difference should be:
* 2 new `kibana.alert` fields (`flapping_history` and `last_detected`)
Compare 6 and 9 (index mapping from rule registry & alerting framework
in this branch).
* some new ECS fields
* each `keyword` mapped ECS field should have `ignore_above` setting
### Verify that the generated component templates work with existing
rule registry index templates & indices:
1. Run `main` or a previous version and create a rule that uses both ECS
component templates & technical component templates (detection rules use
both). Let it run a few times.
2. Using the same ES data, switch to this branch with
`xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts
with no rule registry errors and the rule continues to run as expected.
3. Using the same ES data, switch to this branch with
`xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts
with no alerting or rule registry errors and the rule continues to run
as expected. Verify that the mapping on the existing
`.internal.alerts-security.alerts-default-000001` has been updated to
include the latest ECS mappings and the two new technical fields.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
## Summary
This new CSV endpoint will be usable for scripted reporting. As of this
PR, the endpoint is not going to be used in the UI: that will be handled
by [this issue](https://github.com/elastic/kibana/issues/151190). The
new endpoint is not documented yet, but should be once the endpoint is
used in the UI, according to [this
issue](https://github.com/elastic/kibana/issues/151745).
Depends on https://github.com/elastic/kibana/pull/150631
Closes https://github.com/elastic/kibana/issues/148775
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Using the endpoint from a script
A request to generate a report has a string of rison-encoded job
parameters. The parameters can now be made into a template for customers
to inject their own savedSearchID. The time range of the search,
filters, and query can also be dynamically filled in at the time of
requesting a report - much more easily than before.
```sh
SAVED_SEARCH_ID="d779c5d0-a1c3-11ed-bd60-6957c24fc275"
# Inject the search ID into the "jobParams" template
JOB_PARAMS="(browserTimezone:UTC,locatorParams:!((id:DISCOVER_APP_LOCATOR,params:(savedSearchId:'${SAVED_SEARCH_ID}',timeRange:(from:now-1M,to:now)))),objectType:search)"
# Send a request to generate a report
curl -XPOST $ENDPOINT \
-H "${AUTH_HEADER}" \
-H 'Content-Type: application/json' \
-H 'kbn-xsrf: reporting'
-d '{ "jobParams": "'$JOB_PARAMS'"}'
```
## Limitations
A locator without a `savedSearchId` is currently not supported. Getting
that support will make it possible to create exports based on content in
Discover tied only to a Data View (unsaved search support).
## (Demo) Developer Example app
_The demo app is descoped from this PR_
~~This PR updates the Reporting example app to feature a "CSV Explorer."
In this app, you can select a saved search, build up a
DiscoverAppLocatorParams object, and send a request to generate a
report.~~
https://user-images.githubusercontent.com/908371/217356050-f556f897-33c6-4623-aa06-9af191378e48.mp4
## Release Note
We are skipping the release note for now, while waiting for
documentation.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Fixes https://github.com/elastic/kibana/issues/149344
This PR migrates all plugins to packages automatically. It does this
using `node scripts/lint_packages` to automatically migrate
`kibana.json` files to `kibana.jsonc` files. By doing this automatically
we can simplify many build and testing procedures to only support
packages, and not both "packages" and "synthetic packages" (basically
pointers to plugins).
The majority of changes are in operations related code, so we'll be
having operations review this before marking it ready for review. The
vast majority of the code owners are simply pinged because we deleted
all `kibana.json` files and replaced them with `kibana.jsonc` files, so
we plan on leaving the PR ready-for-review for about 24 hours before
merging (after feature freeze), assuming we don't have any blockers
(especially from @elastic/kibana-core since there are a few core
specific changes, though the majority were handled in #149370).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
