# Backport
This will backport the following commits from `main` to `8.9`:
- [[DOCS] Update GitHub links in dev docs
(#164503)](https://github.com/elastic/kibana/pull/164503)
<!--- Backport version: 8.9.7 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT [{"author":{"name":"James
Rodewig","email":"james.rodewig@elastic.co"},"sourceCommit":{"committedDate":"2023-08-29T18:43:41Z","message":"[DOCS]
Update GitHub links in dev docs
(#164503)","sha":"6fdfefe4ffb452d29c203fcba975aa844c5d5dac","branchLabelMapping":{"^v8.11.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Docs","Feature:ExpressionLanguage","release_note:skip","docs","v8.10.0","v8.11.0","v8.9.2"],"number":164503,"url":"https://github.com/elastic/kibana/pull/164503","mergeCommit":{"message":"[DOCS]
Update GitHub links in dev docs
(#164503)","sha":"6fdfefe4ffb452d29c203fcba975aa844c5d5dac"}},"sourceBranch":"main","suggestedTargetBranches":["8.10","8.9"],"targetPullRequestStates":[{"branch":"8.10","label":"v8.10.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.11.0","labelRegex":"^v8.11.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/164503","number":164503,"mergeCommit":{"message":"[DOCS]
Update GitHub links in dev docs
(#164503)","sha":"6fdfefe4ffb452d29c203fcba975aa844c5d5dac"}},{"branch":"8.9","label":"v8.9.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
# Backport
This will backport the following commits from `main` to `8.9`:
- [[DOCS] Trying to fix broken links, or removing them
(#162844)](https://github.com/elastic/kibana/pull/162844)
<!--- Backport version: 8.9.7 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)
<!--BACKPORT
[{"author":{"name":"amyjtechwriter","email":"61687663+amyjtechwriter@users.noreply.github.com"},"sourceCommit":{"committedDate":"2023-08-01T17:40:47Z","message":"[DOCS]
Trying to fix broken links, or removing them (#162844)\n\n##
Summary\r\n\r\nRelates to #162551\r\n\r\nLinks returning 404 on
[Elasticsearch
service\r\npage](https://www.elastic.co/guide/en/kibana/current/elasticsearch-service.html)\r\nin
the Developer guide.\r\n\r\nReplaced the link to the data plug in and
one to the global search\r\nplugin (though I'm not sure keeping the
sentence and the link to the\r\nglobal search plugin is useful - maybe
we should just remove both\r\ncompletely). Removed the link
'Elasticsearch service API
docs'\r\naltogether.","sha":"6666ed46351a4b2c4fa0b3e4fab9b7a1a5f9a928","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Docs","Team:Core","release_note:skip","docs","v7.17.0","v8.9.0","v8.10.0"],"number":162844,"url":"https://github.com/elastic/kibana/pull/162844","mergeCommit":{"message":"[DOCS]
Trying to fix broken links, or removing them (#162844)\n\n##
Summary\r\n\r\nRelates to #162551\r\n\r\nLinks returning 404 on
[Elasticsearch
service\r\npage](https://www.elastic.co/guide/en/kibana/current/elasticsearch-service.html)\r\nin
the Developer guide.\r\n\r\nReplaced the link to the data plug in and
one to the global search\r\nplugin (though I'm not sure keeping the
sentence and the link to the\r\nglobal search plugin is useful - maybe
we should just remove both\r\ncompletely). Removed the link
'Elasticsearch service API
docs'\r\naltogether.","sha":"6666ed46351a4b2c4fa0b3e4fab9b7a1a5f9a928"}},"sourceBranch":"main","suggestedTargetBranches":["7.17","8.9"],"targetPullRequestStates":[{"branch":"7.17","label":"v7.17.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/162844","number":162844,"mergeCommit":{"message":"[DOCS]
Trying to fix broken links, or removing them (#162844)\n\n##
Summary\r\n\r\nRelates to #162551\r\n\r\nLinks returning 404 on
[Elasticsearch
service\r\npage](https://www.elastic.co/guide/en/kibana/current/elasticsearch-service.html)\r\nin
the Developer guide.\r\n\r\nReplaced the link to the data plug in and
one to the global search\r\nplugin (though I'm not sure keeping the
sentence and the link to the\r\nglobal search plugin is useful - maybe
we should just remove both\r\ncompletely). Removed the link
'Elasticsearch service API
docs'\r\naltogether.","sha":"6666ed46351a4b2c4fa0b3e4fab9b7a1a5f9a928"}}]}]
BACKPORT-->
Co-authored-by: amyjtechwriter <61687663+amyjtechwriter@users.noreply.github.com>
This PR does:
- checks Kibana.spec file
- Checks server feature.ts
- Adds correct route access to APIs
- Removes unnecessary logs
- Removes collector and symbolized `secret_token` from config schema as
it won't be used
- Add README file
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Partially address https://github.com/elastic/kibana/issues/158835, add
cloud chat (drift) to more places: all management pages and
home/getting_started page
I hit an issue that both management and home couldn't depend directly on
`cloudChat` plugin. Here is the issue with more details
https://github.com/elastic/kibana/issues/159008. I worked around with
creating an intermediate `cloudChatProvider` plugin.
How do I run drift locally?
Add this to kibana.yml
```
xpack.cloud.id: "some-id"
xpack.cloud.trial_end_date: "2023-06-21T00:00:00.000Z"
xpack.cloud_integrations.chat.enabled: true
xpack.cloud_integrations.chat.chatURL: "https://elasticcloud-production-chat-us-east-1.s3.amazonaws.com/drift-iframe.html"
xpack.cloud_integrations.chat.chatIdentitySecret: "some-secret" (get it from drift console)
```
You need to have access to our drift account. But I tested with a custom
account. To change account id I had to point
`xpack.cloud_integrations.chat.chatURL` to a script with custom drift
id.
[Documentation](https://docs.google.com/document/d/1Ms8d8d_fbTTRHlBroEAKGNMNk3jFFgOAkVDRhqLxAPQ/edit?pli=1#)
issue: https://github.com/elastic/kibana/issues/158810
## Summary
This PR is a cleanup to make [this
POC](https://github.com/elastic/kibana/pull/155420) production ready
- Serverless PLI features splitting in Security Solution, to allow/deny
access to configured functionalities, using the current Kibana RBAC
service.
- Create the Upselling service to display Serveless-specific prompts in
the application when features are not available
- Create a `SecurityRoutePageWrapper` component that wraps Pages and
displays the upsell when necessary.
- We will refactor the code base to use `SecurityRoutePageWrapper`
everywhere on another PR.
- Create an Upsell page and section for entity analytics
bd8db822-2f4b-4545-9da7-bedc07d93f90
### test:
Serverless: `yarn serverless-security`.
* To change the product line you have to update
`xpack.serverless.security.productLineIds` on
`config/serverless.security.yml`.
ESS: `yarn start`
### Glossary
* PLI - Product Line Item (`Alert Triage`, `Osquery`, `Cases` , ... )
* Product Line - The product that the user is subscribed to (Security
Essentials, Security Complete, ...)
* essSecurity - New plugin with code that only runs for ESS offer
(non-serverless).
* App Feature - A security solution feature or group of features that
can be disabled for a product line. It can be mapped to PLIs (`Alert
Triage`, `Osquery`, `Cases` , ... ).
* Capability - A string that when present represents that the user can
access a given feature. A capability could be of the type UI or API
(`read_cases`, `crud_cases`, ...).
### Current architecture
### New architecture
### How does it work?
Every serverless product line (endpointEssentials, cloud essentials) can
define which features are enabled:
69d0fc15f4/x-pack/plugins/serverless_security/common/pli/pli_config.ts (L12-L19)
For ESS (non-serverless) offer we enable all features by default.
69d0fc15f4/x-pack/plugins/ess_security/server/constants.ts (L10-L13)
A feature can define privileges:
69d0fc15f4/x-pack/plugins/security_solution/server/lib/app_features/security_kibana_features.ts (L177-L185)
When the feature is enabled the privileges get merged into the base
config and injected into kibana features.
69d0fc15f4/x-pack/plugins/security_solution/server/lib/app_features/app_features.ts (L61-L70)
### TODO
- [x] lazy load these components
- [x] Add unit test to:
- ~SecurityRoutePageWrapper
x-pack/plugins/security_solution/public/common/components/security_route_page_wrapper/index.tsx~
-
~x-pack/plugins/security_solution/public/common/hooks/use_upselling.ts~
-
~x-pack/plugins/security_solution/public/common/lib/capabilities/has_capabilities.ts~
-
~x-pack/plugins/security_solution/public/common/lib/upsellings/upselling_service.ts~
- ~x-pack/plugins/serverless_security/common/pli/pli_features.ts~
-
~x-pack/plugins/serverless_security/public/components/upselling/register_upsellings.tsx~
-
~x-pack/plugins/security_solution/server/lib/app_features/app_features.ts~
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adds build date to `GET kbn:api/status` similar to ES. Example output
running locally:
```json
{
"name": "ES-DMVD5M3",
"uuid": "545ba70c-063e-449b-af21-6c8e7b30f77e",
"version": {
"number": "8.9.0",
"build_hash": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"build_number": 9007199254740991,
"build_snapshot": false,
"build_date": "2023-05-15T23:12:09.000Z"
},
...rest
}
```
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Release Note
The status endpoint now returns the build date alongside other build
information.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/154330
This PR:
- Moves the editor from unified-search to a standalone package
- The editor has now a core ui settings dependency but is going to have
an expressions dependency too when merged with the ESQL branch
- Adds a new plugin (text-based-languages) which is used to pass the
dependencies on the package. The user can either use this plugin without
giving any dependencies or use the package with passing the dependecies
on the KibanaContextProvider.
- Adds storybook for the editor (I used the mdx stories as we did on the
random sampling package)
<img width="1668" alt="image"
src="763a3112-1ae5-49bb-81f3-acd02892e402">
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Moves a series of Lens components to an independent plugin for reuse in
the annotations library.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] remove mentions of Lens
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes#154733
Creates a new plugin for logs onboarding with wizard to organize steps
into discrete views.
#### TODO:
- [x] rename plugin to observability_onboarding
- [x] configure: UI and server plugin
- [x] enable/disable new plugin
- [x] remove the link to it from Observability nav
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Yngrid Coello <yngrid.coello@elastic.co>
Co-authored-by: Yngrid Coello <yngrdyn@gmail.com>
This PR fixes the 3rd party external plugin development workflow by
introducing a dev step for plugins that allows for development usages of
those with Kibana.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Replace the deprecated Jenkins specifics in the interpreting CI failures
section for something Buildkite specific.
Please bear with me, I'm not much familiar with the implementations but
found the current documentation to be obsoleted.
## Summary
This plugin will contain the asset inventory and topology API in Kibana,
giving Kibana projects access to inventory and topology data via an HTTP
and/or JS API on the server and client.
[Currently proposed API
docs](https://github.com/elastic/o11y-topology-playground/tree/main/docs/api)
will be moved to this repo as well, contained inside this plugin folder,
as a part of this PR.
## Enabling the plugin
This plugin is entirely in "technical preview" and because of this, must
be specifically enabled via config for it to do anything besides being
run by the core plugin framework. To enable the server API layer, as
well as the index template management, put the following line in your
kibana.yml file:
```yml
xpack.assetManager.alphaEnabled: true
```
## Running the API integration tests
Run the functional test server with the asset manager config in place:
```shell
$ node scripts/functional_tests_server --config x-pack/test/api_integration/apis/asset_manager/config.ts
```
Then run the functional test runner with the same config, to target just
these tests:
```shell
$ node scripts/functional_test_runner --config=x-pack/test/api_integration/apis/asset_manager/config.
ts
```
_Note:_ The config file added in this folder enables the tech preview
plugin ([see file
here](https://github.com/elastic/kibana/pull/152456/files#diff-bc00de6c34c9bc131cfbdf3570c487fe9ee947e9a88a84c59d6b139b79d7708eR20)).
### Running the integration tests for verifying that the plugin is
"disabled" by default
There is a small set of tests that confirm that the endpoints return 404
and there is no index template installed if the config value is not set
in the kibana.yml file. To run this suite, use the following config:
```shell
$ node scripts/functional_tests_server --config x-pack/test/api_integration/apis/asset_manager/config_when_disabled.ts
$ node scripts/functional_test_runner --config=x-pack/test/api_integration/apis/asset_manager/config_when_disabled.
ts
```
## Testing this PR with sample data
There are some sample data mechanisms in place inside this PR to allow
us to build out the endpoints.
### View sample docs
```http
GET /api/asset-manager/assets/sample
```
This will return a list of the assets that are included if you elect to
write assets. This is a good endpoint to use to find EAN (Elastic Asset
Name) values that you may want to exclude from writing for a given time
period, to simulate assets appearing/disappearing over time.
### Write sample docs
```http
POST /api/asset-manager/assets/sample
{
"baseDateTime": "2023-02-28T12:00:00.000Z",
"excludeEans": ["k8s.cluster:cluster-002"]
}
```
This posts all of the sample asset documents to Elasticsearch using the
`baseDateTime` value as the timestamp. Any valid string or number that
is accepted by `new Date()` should work for `baseDateTime`.
The `excludeEans` value is an array of EAN ("Elastic Asset Name") values
that you don't want to write on this particular run. This way you can
have assets appear (exclude them in the past, don't exclude them during
a later run) or disappear (vice versa) and see how that shows up in
other endpoints.
**Note:** *Remember that when you curl a Kibana server API with a POST
request, you must include a `kbn-xsrf` header with any string value you
want.*
### Get asset docs from ES
```http
GET /api/asset-manager/assets?type=k8s.cluster&from=now-10m
```
This is the primary "real" endpoint available right now. It should
retrieve a list of assets based on the type/from/to/ean filter values
you specify. Once you load the sample data, this endpoint should return
results.
## Debug logging
There are some extra debug logs for ES queries that are running in the
code in this PR. To print those logs to the Kibana server console, run
Kibana using `DEBUG_LOGGER=true`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Updating the Debugging Kibana documentation:
1. Adding Python as additional pre-requisite technology
2. Replacing apm.dev.js configuration with updated kibana.dev.yml
approach
Similar to recent discussion in issue #79490, I've found that the
`apm.dev.js` approach is no longer working.
### Checklist
Delete any items that are not applicable to this PR.
- [X]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
### Risk Matrix
N/A
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
# [Security Solution] Data Quality dashboard
## Check ECS compatibility with just one click
With just one click, the _Data Quality dashboard_ checks all the indices used by the Security Solution, (or anything else), for compatibility with the [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html)
## Create cases from results
Create a single case containing all the results, or create cases for specific indices
## Interactive tabs put results in context
Expand any index to reveal interactive tabs
- Summary
- Incompatible fields
- Custom fields
- ECS complaint fields
- All fields
## Share comprehensive markdown reports
Share markdown reports containing the same content as the dashboard
### On page load
When the Data Quality dashboard page loads, the alerts index, and any indices matching the selected `Data view` are displayed
Only `hot`, `warm`, or `unmanaged` indices are displayed by default
Indices are not checked automatically when the dashboard loads
Click either :
- `Check all` to check all the indices on the page
- The expand button to automatically check (just) one index, and instantly view results
### Check all
When the `Check all` button is clicked
- The `Check all` button changes to a `Cancel` button
- The `Last checked: n <time unit> ago` text is replaced with a progress bar indicating how many Indices are left to check
- The `Checking <index name>` text will update as each index is checked. Text will wrap if necessary
- The results tables begin updating with results
- Pattern stats update to summarize each table
- Rolled up results for the entire page update after every index is checked
<https://user-images.githubusercontent.com/4459398/216007795-2ebbc0c6-8c7a-49c7-a22c-b97d2a58dddd.mov>
When Check all, is running, the Data Quality dashboard adds a three second delay after every check completes, before beginning the next check.
Check all will keep checking indexes until the user cancels, or all indexes have (attempted to be) checked.
While Check all is running, users may simultaneously click on any index to check it on demand. The results are instantly rolled up when this happens.
When all checks complete, the page looks like this:
### Take action
Click the `Take action` popover to share the entire page of results via one of the following actions:
- Add to new case
- Copy to clipboard
### Expanding results
The `Incompatible fields` tab is always displayed by default when a result is expanded
The `Incompatible fields` tab shows a success message when a successful result is expanded
The `Incompatible fields` tab shows, side by side, expected ECS mapping types vs the actual mapping types when they are different
The `Incompatible fields` tab also compares field values expected by ECS vs the actual values in an index, when they are different
The `Incompatible fields` tab displays a callout that explains the consequences of having incompatible fields. The content is based on the following illustration, created by @MikePaquette
<img width="1264" alt="ecs_meter" src="https://user-images.githubusercontent.com/4459398/216016124-6fe89ab4-c364-40ec-8a6f-99349e6d583c.png">
The calllout has a call to action to create a case or copy a markdown report for just the expanded result
- Add to new case
- Copy to clipboard
### Tabs
The Summary tab displays a call to action when incompatible fields are found
Click on any part of the Summary tab chart or legend to navigate to the corresponding tab
Clicking on the `Copy to clipboard` call to action in the Custom fields tab copies a markdown version of the table to the clipboard
The search feature of the ECS complaint fields tab may, for example, be used to verify a specific ECS complaint mapping exists
The All fields tab displays the union of all other tabs
### Data view selection
The `Data view` dropdown defaults to the `Security Default Data View`
The alerts index is always checked and included in the results, even when another Data View is selected
### ILM phase options
Only `hot`, `warm`, or `unmanaged` indices may be selected for checking.
The `cold` and `frozen` options are disabled.
When all options in the `ILM phase` box are cleared, an informative empty prompt is displayed
### Errors
Errors may occur for some (or all) indices. The `View errors` button appears when the first error occurs
Users may click the `View errors` button to view them, even while a check is in progress
The Copy to clipboard button in the errors popover copies a markdown version of the errors table to the clipboard
When errors occur, the same content shown in the Errors popover is automatically included in the markdown report created by the `Take action` menu
### Markdown reports
The content of markdown reports (created by the Take action menu) includes most of the content from the Data Quality dashboard that created it
In the screenshot below, the Data Quality dashboard is on the left, and a markdown report (pasted into Github) is on the right
Stats rollups and tables are included in markdown reports
Markdown reports use the same "expected vs actual" format to display the details of incompatible field mappings
### Navigation
The Data Quality dashboard is grouped with the existing Security Solution dashboards
It may also be launched via the side navigation
## Privileges
The privileges in the table below are required to check any pattern of indices, or any specific index:
| Privilege | Required to | Required for API |
|-------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|------------------|
| `monitor` or `manage` (`manage` builds on `monitor`) | List indices that match a pattern, and get document counts for an index example: `GET logs-*/_stats` | `_stats` |
| `view_index_metadata` or `manage_ilm` | List index ILM configs (e.g. hot) that match a pattern example: `GET logs-*/_ilm/explain` | `_ilm/explain` |
| `view_index_metadata` or `manage` | Get index mappings for a specific index example: `GET .ds-logs-endpoint.events.process-default-2023.01.17-000001/_mapping` | `_mapping` |
| `read` or `read_cross_cluster` | Run aggregations to test for unallowed values example: `GET .ds-logs-endpoint.events.process-default-2023.01.17-000001/_search` | `_search` |
Users may have some of the privileges required to check an index, but not all of them.
The built-in `viewer` role does not have the `monitor` (or `manage`) role. The following screenshot illustrates what a user will see if they login as a user with the `viewer` role:
# An actual markdown report (all content below)
The rest of the content below is pasted from an actual report, created via the `Take action` menu:
# Data quality
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 17 | 15 | 17 | 1,404,514 |
## .alerts-security.alerts-default
`hot(1)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 1 | 1 | 1 | 1,837 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .internal.alerts-security.alerts-default-000001 | 1,837 (100.0%) | 1 | `hot` |
### .internal.alerts-security.alerts-default-000001
The `.internal.alerts-security.alerts-default-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .internal.alerts-security.alerts-default-000001 | 1,837 (100.0%) | 1 | `hot` |
### **Incompatible fields** `1` **Custom fields** `188` **ECS compliant fields** `1219` **All fields** `1408`
#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field values - .internal.alerts-security.alerts-default-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (62) |
## auditbeat-*
`hot(11)` `unmanaged(1)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 13 | 10 | 12 | 29,182 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.6.0-2023.01.17-000001 | 14,409 (49.4%) | 0 | `hot` |
| -- | .ds-auditbeat-8.5.3-2023.01.24-000001 | 2,857 (9.8%) | -- | `hot` |
| ✅ | .ds-auditbeat-8.2.3-2023.01.24-000001 | 2,246 (7.7%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.4.1-2023.01.24-000001 | 2,179 (7.5%) | 0 | `hot` |
| -- | .ds-auditbeat-8.3.3-2023.01.24-000001 | 1,921 (6.6%) | -- | `hot` |
| ✅ | auditbeat-7.16.0-2023.01.17-000001 | 1,880 (6.4%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.1.1-2023.01.24-000001 | 1,676 (5.7%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.2.2-2023.01.24-000001 | 1,578 (5.4%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.0.0-2023.01.24-000001 | 251 (0.9%) | 0 | `hot` |
| ❌ | auditbeat-7.10.2-2023.01.24-000001 | 111 (0.4%) | 12 | `hot` |
| ✅ | .ds-auditbeat-8.5.0-2023.01.24-000001 | 74 (0.3%) | 0 | `hot` |
| ❌ | auditbeat-custom-empty-index-1 | 0 (0.0%) | 1 | `unmanaged` |
### .ds-auditbeat-8.6.0-2023.01.17-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.6.0-2023.01.17-000001 | 14,409 (49.4%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `549` **ECS compliant fields** `1210` **All fields** `1759`
### .ds-auditbeat-8.2.3-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.2.3-2023.01.24-000001 | 2,246 (7.7%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1210` **All fields** `1720`
### .ds-auditbeat-8.4.1-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.4.1-2023.01.24-000001 | 2,179 (7.5%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `509` **ECS compliant fields** `1210` **All fields** `1719`
### auditbeat-7.16.0-2023.01.17-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | auditbeat-7.16.0-2023.01.17-000001 | 1,880 (6.4%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `523` **ECS compliant fields** `1111` **All fields** `1634`
### .ds-auditbeat-8.1.1-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.1.1-2023.01.24-000001 | 1,676 (5.7%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1204` **All fields** `1714`
### .ds-auditbeat-8.2.2-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.2.2-2023.01.24-000001 | 1,578 (5.4%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1210` **All fields** `1720`
### .ds-auditbeat-8.0.0-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.0.0-2023.01.24-000001 | 251 (0.9%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1204` **All fields** `1714`
### auditbeat-7.10.2-2023.01.24-000001
The `auditbeat-7.10.2-2023.01.24-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | auditbeat-7.10.2-2023.01.24-000001 | 111 (0.4%) | 12 | `hot` |
### **Incompatible fields** `12` **Custom fields** `467` **ECS compliant fields** `602` **All fields** `1081`
#### 12 incompatible fields
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - auditbeat-7.10.2-2023.01.24-000001
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| error.message | `match_only_text` | `text` |
| error.stack_trace | `wildcard` | `keyword` |
| http.request.body.content | `wildcard` | `keyword` |
| http.response.body.content | `wildcard` | `keyword` |
| message | `match_only_text` | `text` |
| process.command_line | `wildcard` | `keyword` |
| process.parent.command_line | `wildcard` | `keyword` |
| registry.data.strings | `wildcard` | `keyword` |
| url.full | `wildcard` | `keyword` |
| url.original | `wildcard` | `keyword` |
| url.path | `wildcard` | `keyword` |
#### Incompatible field values - auditbeat-7.10.2-2023.01.24-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.kind | `alert`, `enrichment`, `event`, `metric`, `state`, `pipeline_error`, `signal` | `error` (1) |
### .ds-auditbeat-8.5.0-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.5.0-2023.01.24-000001 | 74 (0.3%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `509` **ECS compliant fields** `1210` **All fields** `1719`
### auditbeat-custom-empty-index-1
The `auditbeat-custom-empty-index-1` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | auditbeat-custom-empty-index-1 | 0 (0.0%) | 1 | `unmanaged` |
### **Incompatible fields** `1` **Custom fields** `0` **ECS compliant fields** `0` **All fields** `0`
#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - auditbeat-custom-empty-index-1
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| @timestamp | `date` | `-` |
## logs-*
`hot(2)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 3 | 2 | 2 | 602 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .ds-logs-endpoint.alerts-default-2023.01.17-000001 | 342 (56.8%) | 2 | `hot` |
| ❌ | .ds-logs-endpoint.events.process-default-2023.01.17-000001 | 260 (43.2%) | 1 | `hot` |
### .ds-logs-endpoint.alerts-default-2023.01.17-000001
The `.ds-logs-endpoint.alerts-default-2023.01.17-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .ds-logs-endpoint.alerts-default-2023.01.17-000001 | 342 (56.8%) | 2 | `hot` |
### **Incompatible fields** `2` **Custom fields** `857` **ECS compliant fields** `675` **All fields** `1534`
#### 2 incompatible fields
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - .ds-logs-endpoint.alerts-default-2023.01.17-000001
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| process.env_vars | `keyword` | `object` |
#### Incompatible field values - .ds-logs-endpoint.alerts-default-2023.01.17-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (45) |
### .ds-logs-endpoint.events.process-default-2023.01.17-000001
The `.ds-logs-endpoint.events.process-default-2023.01.17-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .ds-logs-endpoint.events.process-default-2023.01.17-000001 | 260 (43.2%) | 1 | `hot` |
### **Incompatible fields** `1` **Custom fields** `130` **ECS compliant fields** `304` **All fields** `435`
#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - .ds-logs-endpoint.events.process-default-2023.01.17-000001
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| process.env_vars | `keyword` | `object` |
## packetbeat-*
`hot(2)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 0 | 2 | 2 | 1,372,893 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-packetbeat-8.6.0-2023.01.17-000001 | 704,062 (51.3%) | 0 | `hot` |
| ✅ | .ds-packetbeat-8.4.1-2023.01.24-000001 | 668,831 (48.7%) | 0 | `hot` |
### .ds-packetbeat-8.6.0-2023.01.17-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-packetbeat-8.6.0-2023.01.17-000001 | 704,062 (51.3%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `604` **ECS compliant fields** `1209` **All fields** `1813`
### .ds-packetbeat-8.4.1-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-packetbeat-8.4.1-2023.01.24-000001 | 668,831 (48.7%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `604` **ECS compliant fields** `1209` **All fields** `1813`
## Errors
Some indices were not checked for Data Quality
Errors may occur when pattern or index metadata is temporarily unavailable, or because you don't have the privileges required for access
The following privileges are required to check an index:
- `monitor` or `manage`
- `view_index_metadata`
- `read` or `read_cross_cluster`
| Pattern | Index | Error |
|---------|-------|-------|
| .alerts-security.alerts-default | -- | `Error loading stats: Error: Forbidden` |
| auditbeat-* | -- | `Error loading stats: Error: Forbidden` |
| logs-* | -- | `Error loading stats: Error: Forbidden` |
| packetbeat-* | -- | `Error loading stats: Error: Forbidden` |
See also: https://github.com/elastic/security-team/issues/4559
## Summary
Fix https://github.com/elastic/kibana/issues/148412
More and more SO types will not be accessible from the HTTP APIs (either
`hidden:true` or `hiddenFromHTTPApis: true`).
However, the FTR SO client (`KbnClientSavedObjects`) still needs to be
able to access and manipulate all SO types.
This PR introduces a `ftrSoApis` plugin that is loaded for all FTR
suites. This plugin exposes SO APIs that are used by the FTR client
instead of the public SO HTTP APIs. These APIs are configured to know
about all types, even hidden ones.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Dearest Reviewers 👋
I've been working on this branch with @mistic and @tylersmalley and
we're really confident in these changes. Additionally, this changes code
in nearly every package in the repo so we don't plan to wait for reviews
to get in before merging this. If you'd like to have a concern
addressed, please feel free to leave a review, but assuming that nobody
raises a blocker in the next 24 hours we plan to merge this EOD pacific
tomorrow, 12/22.
We'll be paying close attention to any issues this causes after merging
and work on getting those fixed ASAP. 🚀
---
The operations team is not confident that we'll have the time to achieve
what we originally set out to accomplish by moving to Bazel with the
time and resources we have available. We have also bought ourselves some
headroom with improvements to babel-register, optimizer caching, and
typescript project structure.
In order to make sure we deliver packages as quickly as possible (many
teams really want them), with a usable and familiar developer
experience, this PR removes Bazel for building packages in favor of
using the same JIT transpilation we use for plugins.
Additionally, packages now use `kbn_references` (again, just copying the
dx from plugins to packages).
Because of the complex relationships between packages/plugins and in
order to prepare ourselves for automatic dependency detection tools we
plan to use in the future, this PR also introduces a "TS Project Linter"
which will validate that every tsconfig.json file meets a few
requirements:
1. the chain of base config files extended by each config includes
`tsconfig.base.json` and not `tsconfig.json`
1. the `include` config is used, and not `files`
2. the `exclude` config includes `target/**/*`
3. the `outDir` compiler option is specified as `target/types`
1. none of these compiler options are specified: `declaration`,
`declarationMap`, `emitDeclarationOnly`, `skipLibCheck`, `target`,
`paths`
4. all references to other packages/plugins use their pkg id, ie:
```js
// valid
{
"kbn_references": ["@kbn/core"]
}
// not valid
{
"kbn_references": [{ "path": "../../../src/core/tsconfig.json" }]
}
```
5. only packages/plugins which are imported somewhere in the ts code are
listed in `kbn_references`
This linter is not only validating all of the tsconfig.json files, but
it also will fix these config files to deal with just about any
violation that can be produced. Just run `node scripts/ts_project_linter
--fix` locally to apply these fixes, or let CI take care of
automatically fixing things and pushing the changes to your PR.
> **Example:** [`64e93e5`
(#146212)](64e93e5806)
When I merged main into my PR it included a change which removed the
`@kbn/core-injected-metadata-browser` package. After resolving the
conflicts I missed a few tsconfig files which included references to the
now removed package. The TS Project Linter identified that these
references were removed from the code and pushed a change to the PR to
remove them from the tsconfig.json files.
## No bazel? Does that mean no packages??
Nope! We're still doing packages but we're pretty sure now that we won't
be using Bazel to accomplish the 'distributed caching' and 'change-based
tasks' portions of the packages project.
This PR actually makes packages much easier to work with and will be
followed up with the bundling benefits described by the original
packages RFC. Then we'll work on documentation and advocacy for using
packages for any and all new code.
We're pretty confident that implementing distributed caching and
change-based tasks will be necessary in the future, but because of
recent improvements in the repo we think we can live without them for
**at least** a year.
## Wait, there are still BUILD.bazel files in the repo
Yes, there are still three webpack bundles which are built by Bazel: the
`@kbn/ui-shared-deps-npm` DLL, `@kbn/ui-shared-deps-src` externals, and
the `@kbn/monaco` workers. These three webpack bundles are still created
during bootstrap and remotely cached using bazel. The next phase of this
project is to figure out how to get the package bundling features
described in the RFC with the current optimizer, and we expect these
bundles to go away then. Until then any package that is used in those
three bundles still needs to have a BUILD.bazel file so that they can be
referenced by the remaining webpack builds.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Files management UI that rounds out the files MVP. This is UI is
intended to be progressively enhanced and provides a way for system
administrators get some insight and manage the files created and stored
in Kibana.
## To reviewers
* This is UI for retrieval and deletion of files (the R+D of CRUD)
* Creating and deleting tags to be supported in a future version
* This UI is intended to form part of the broader content management
experience
* We use the `TableListView` component as far as possible
## How to test
1. Start Kibana with `yarn start --run-examples`
2. Go to the "Developer Examples" from the left nav menu
3. Go to the "Files example" plugin
4. Click the "Upload file" button, upload a few different image types
(PNG, JPG and WEBP)
5. Go to "Stack management" > "Files"
6. Behold your files in the management UI
7. (Bonus) check that the UI and API `GET /api/files/find`, `GET
/api/files/metrics` and `DELETE /api/files/blobs` are not accessible to
non-admin or appropriately privileged users (i.e., those with "Files
management" access).
## List of functionality
- [x] List all saved objects (scoped to admin)
- [x] Is able to bulk-delete files
- [x] Shows basic storage diagnostics
- [x] Is able to search and filter files
## Screenshots
<details>
<summary>screenshots</summary>
<img width="1545" alt="Screenshot 2022-11-08 at 13 56 54"
src="https://user-images.githubusercontent.com/8155004/200570783-cfefdbf3-c5ff-4ece-ba24-48a455fcca75.png">
<img width="910" alt="Screenshot 2022-11-10 at 12 52 35"
src="https://user-images.githubusercontent.com/8155004/201083812-bc9f25f5-b423-43a6-9229-5e2a4cdd943a.png">
<img width="451" alt="Screenshot 2022-11-10 at 12 37 07"
src="https://user-images.githubusercontent.com/8155004/201081039-832a1980-684c-4abb-bb05-0c7c6a849d4d.png">
<img width="959" alt="Screenshot 2022-11-08 at 13 57 15"
src="https://user-images.githubusercontent.com/8155004/200570797-f122cff5-7043-4e01-9b51-d5663c1b26d6.png">
<img width="500" alt="Screenshot 2022-11-08 at 13 57 38"
src="https://user-images.githubusercontent.com/8155004/200570801-35cdbd99-0256-4dee-9f78-2f6ad853305f.png">
</details>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
* Makes the API docs visible on dev site
* Updates the file README
* Updates the file service tutorial (more structured steps, leverage our
own UI components)
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
