## Summary
This plugin will contain the asset inventory and topology API in Kibana,
giving Kibana projects access to inventory and topology data via an HTTP
and/or JS API on the server and client.
[Currently proposed API
docs](https://github.com/elastic/o11y-topology-playground/tree/main/docs/api)
will be moved to this repo as well, contained inside this plugin folder,
as a part of this PR.
## Enabling the plugin
This plugin is entirely in "technical preview" and because of this, must
be specifically enabled via config for it to do anything besides being
run by the core plugin framework. To enable the server API layer, as
well as the index template management, put the following line in your
kibana.yml file:
```yml
xpack.assetManager.alphaEnabled: true
```
## Running the API integration tests
Run the functional test server with the asset manager config in place:
```shell
$ node scripts/functional_tests_server --config x-pack/test/api_integration/apis/asset_manager/config.ts
```
Then run the functional test runner with the same config, to target just
these tests:
```shell
$ node scripts/functional_test_runner --config=x-pack/test/api_integration/apis/asset_manager/config.
ts
```
_Note:_ The config file added in this folder enables the tech preview
plugin ([see file
here](https://github.com/elastic/kibana/pull/152456/files#diff-bc00de6c34c9bc131cfbdf3570c487fe9ee947e9a88a84c59d6b139b79d7708eR20)).
### Running the integration tests for verifying that the plugin is
"disabled" by default
There is a small set of tests that confirm that the endpoints return 404
and there is no index template installed if the config value is not set
in the kibana.yml file. To run this suite, use the following config:
```shell
$ node scripts/functional_tests_server --config x-pack/test/api_integration/apis/asset_manager/config_when_disabled.ts
$ node scripts/functional_test_runner --config=x-pack/test/api_integration/apis/asset_manager/config_when_disabled.
ts
```
## Testing this PR with sample data
There are some sample data mechanisms in place inside this PR to allow
us to build out the endpoints.
### View sample docs
```http
GET /api/asset-manager/assets/sample
```
This will return a list of the assets that are included if you elect to
write assets. This is a good endpoint to use to find EAN (Elastic Asset
Name) values that you may want to exclude from writing for a given time
period, to simulate assets appearing/disappearing over time.
### Write sample docs
```http
POST /api/asset-manager/assets/sample
{
"baseDateTime": "2023-02-28T12:00:00.000Z",
"excludeEans": ["k8s.cluster:cluster-002"]
}
```
This posts all of the sample asset documents to Elasticsearch using the
`baseDateTime` value as the timestamp. Any valid string or number that
is accepted by `new Date()` should work for `baseDateTime`.
The `excludeEans` value is an array of EAN ("Elastic Asset Name") values
that you don't want to write on this particular run. This way you can
have assets appear (exclude them in the past, don't exclude them during
a later run) or disappear (vice versa) and see how that shows up in
other endpoints.
**Note:** *Remember that when you curl a Kibana server API with a POST
request, you must include a `kbn-xsrf` header with any string value you
want.*
### Get asset docs from ES
```http
GET /api/asset-manager/assets?type=k8s.cluster&from=now-10m
```
This is the primary "real" endpoint available right now. It should
retrieve a list of assets based on the type/from/to/ean filter values
you specify. Once you load the sample data, this endpoint should return
results.
## Debug logging
There are some extra debug logs for ES queries that are running in the
code in this PR. To print those logs to the Kibana server console, run
Kibana using `DEBUG_LOGGER=true`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Updating the Debugging Kibana documentation:
1. Adding Python as additional pre-requisite technology
2. Replacing apm.dev.js configuration with updated kibana.dev.yml
approach
Similar to recent discussion in issue #79490, I've found that the
`apm.dev.js` approach is no longer working.
### Checklist
Delete any items that are not applicable to this PR.
- [X]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
### Risk Matrix
N/A
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
# [Security Solution] Data Quality dashboard
## Check ECS compatibility with just one click
With just one click, the _Data Quality dashboard_ checks all the indices used by the Security Solution, (or anything else), for compatibility with the [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html)
## Create cases from results
Create a single case containing all the results, or create cases for specific indices
## Interactive tabs put results in context
Expand any index to reveal interactive tabs
- Summary
- Incompatible fields
- Custom fields
- ECS complaint fields
- All fields
## Share comprehensive markdown reports
Share markdown reports containing the same content as the dashboard
### On page load
When the Data Quality dashboard page loads, the alerts index, and any indices matching the selected `Data view` are displayed
Only `hot`, `warm`, or `unmanaged` indices are displayed by default
Indices are not checked automatically when the dashboard loads
Click either :
- `Check all` to check all the indices on the page
- The expand button to automatically check (just) one index, and instantly view results
### Check all
When the `Check all` button is clicked
- The `Check all` button changes to a `Cancel` button
- The `Last checked: n <time unit> ago` text is replaced with a progress bar indicating how many Indices are left to check
- The `Checking <index name>` text will update as each index is checked. Text will wrap if necessary
- The results tables begin updating with results
- Pattern stats update to summarize each table
- Rolled up results for the entire page update after every index is checked
<https://user-images.githubusercontent.com/4459398/216007795-2ebbc0c6-8c7a-49c7-a22c-b97d2a58dddd.mov>
When Check all, is running, the Data Quality dashboard adds a three second delay after every check completes, before beginning the next check.
Check all will keep checking indexes until the user cancels, or all indexes have (attempted to be) checked.
While Check all is running, users may simultaneously click on any index to check it on demand. The results are instantly rolled up when this happens.
When all checks complete, the page looks like this:
### Take action
Click the `Take action` popover to share the entire page of results via one of the following actions:
- Add to new case
- Copy to clipboard
### Expanding results
The `Incompatible fields` tab is always displayed by default when a result is expanded
The `Incompatible fields` tab shows a success message when a successful result is expanded
The `Incompatible fields` tab shows, side by side, expected ECS mapping types vs the actual mapping types when they are different
The `Incompatible fields` tab also compares field values expected by ECS vs the actual values in an index, when they are different
The `Incompatible fields` tab displays a callout that explains the consequences of having incompatible fields. The content is based on the following illustration, created by @MikePaquette
<img width="1264" alt="ecs_meter" src="https://user-images.githubusercontent.com/4459398/216016124-6fe89ab4-c364-40ec-8a6f-99349e6d583c.png">
The calllout has a call to action to create a case or copy a markdown report for just the expanded result
- Add to new case
- Copy to clipboard
### Tabs
The Summary tab displays a call to action when incompatible fields are found
Click on any part of the Summary tab chart or legend to navigate to the corresponding tab
Clicking on the `Copy to clipboard` call to action in the Custom fields tab copies a markdown version of the table to the clipboard
The search feature of the ECS complaint fields tab may, for example, be used to verify a specific ECS complaint mapping exists
The All fields tab displays the union of all other tabs
### Data view selection
The `Data view` dropdown defaults to the `Security Default Data View`
The alerts index is always checked and included in the results, even when another Data View is selected
### ILM phase options
Only `hot`, `warm`, or `unmanaged` indices may be selected for checking.
The `cold` and `frozen` options are disabled.
When all options in the `ILM phase` box are cleared, an informative empty prompt is displayed
### Errors
Errors may occur for some (or all) indices. The `View errors` button appears when the first error occurs
Users may click the `View errors` button to view them, even while a check is in progress
The Copy to clipboard button in the errors popover copies a markdown version of the errors table to the clipboard
When errors occur, the same content shown in the Errors popover is automatically included in the markdown report created by the `Take action` menu
### Markdown reports
The content of markdown reports (created by the Take action menu) includes most of the content from the Data Quality dashboard that created it
In the screenshot below, the Data Quality dashboard is on the left, and a markdown report (pasted into Github) is on the right
Stats rollups and tables are included in markdown reports
Markdown reports use the same "expected vs actual" format to display the details of incompatible field mappings
### Navigation
The Data Quality dashboard is grouped with the existing Security Solution dashboards
It may also be launched via the side navigation
## Privileges
The privileges in the table below are required to check any pattern of indices, or any specific index:
| Privilege | Required to | Required for API |
|-------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|------------------|
| `monitor` or `manage` (`manage` builds on `monitor`) | List indices that match a pattern, and get document counts for an index example: `GET logs-*/_stats` | `_stats` |
| `view_index_metadata` or `manage_ilm` | List index ILM configs (e.g. hot) that match a pattern example: `GET logs-*/_ilm/explain` | `_ilm/explain` |
| `view_index_metadata` or `manage` | Get index mappings for a specific index example: `GET .ds-logs-endpoint.events.process-default-2023.01.17-000001/_mapping` | `_mapping` |
| `read` or `read_cross_cluster` | Run aggregations to test for unallowed values example: `GET .ds-logs-endpoint.events.process-default-2023.01.17-000001/_search` | `_search` |
Users may have some of the privileges required to check an index, but not all of them.
The built-in `viewer` role does not have the `monitor` (or `manage`) role. The following screenshot illustrates what a user will see if they login as a user with the `viewer` role:
# An actual markdown report (all content below)
The rest of the content below is pasted from an actual report, created via the `Take action` menu:
# Data quality
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 17 | 15 | 17 | 1,404,514 |
## .alerts-security.alerts-default
`hot(1)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 1 | 1 | 1 | 1,837 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .internal.alerts-security.alerts-default-000001 | 1,837 (100.0%) | 1 | `hot` |
### .internal.alerts-security.alerts-default-000001
The `.internal.alerts-security.alerts-default-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .internal.alerts-security.alerts-default-000001 | 1,837 (100.0%) | 1 | `hot` |
### **Incompatible fields** `1` **Custom fields** `188` **ECS compliant fields** `1219` **All fields** `1408`
#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field values - .internal.alerts-security.alerts-default-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (62) |
## auditbeat-*
`hot(11)` `unmanaged(1)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 13 | 10 | 12 | 29,182 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.6.0-2023.01.17-000001 | 14,409 (49.4%) | 0 | `hot` |
| -- | .ds-auditbeat-8.5.3-2023.01.24-000001 | 2,857 (9.8%) | -- | `hot` |
| ✅ | .ds-auditbeat-8.2.3-2023.01.24-000001 | 2,246 (7.7%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.4.1-2023.01.24-000001 | 2,179 (7.5%) | 0 | `hot` |
| -- | .ds-auditbeat-8.3.3-2023.01.24-000001 | 1,921 (6.6%) | -- | `hot` |
| ✅ | auditbeat-7.16.0-2023.01.17-000001 | 1,880 (6.4%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.1.1-2023.01.24-000001 | 1,676 (5.7%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.2.2-2023.01.24-000001 | 1,578 (5.4%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.0.0-2023.01.24-000001 | 251 (0.9%) | 0 | `hot` |
| ❌ | auditbeat-7.10.2-2023.01.24-000001 | 111 (0.4%) | 12 | `hot` |
| ✅ | .ds-auditbeat-8.5.0-2023.01.24-000001 | 74 (0.3%) | 0 | `hot` |
| ❌ | auditbeat-custom-empty-index-1 | 0 (0.0%) | 1 | `unmanaged` |
### .ds-auditbeat-8.6.0-2023.01.17-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.6.0-2023.01.17-000001 | 14,409 (49.4%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `549` **ECS compliant fields** `1210` **All fields** `1759`
### .ds-auditbeat-8.2.3-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.2.3-2023.01.24-000001 | 2,246 (7.7%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1210` **All fields** `1720`
### .ds-auditbeat-8.4.1-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.4.1-2023.01.24-000001 | 2,179 (7.5%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `509` **ECS compliant fields** `1210` **All fields** `1719`
### auditbeat-7.16.0-2023.01.17-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | auditbeat-7.16.0-2023.01.17-000001 | 1,880 (6.4%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `523` **ECS compliant fields** `1111` **All fields** `1634`
### .ds-auditbeat-8.1.1-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.1.1-2023.01.24-000001 | 1,676 (5.7%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1204` **All fields** `1714`
### .ds-auditbeat-8.2.2-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.2.2-2023.01.24-000001 | 1,578 (5.4%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1210` **All fields** `1720`
### .ds-auditbeat-8.0.0-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.0.0-2023.01.24-000001 | 251 (0.9%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1204` **All fields** `1714`
### auditbeat-7.10.2-2023.01.24-000001
The `auditbeat-7.10.2-2023.01.24-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | auditbeat-7.10.2-2023.01.24-000001 | 111 (0.4%) | 12 | `hot` |
### **Incompatible fields** `12` **Custom fields** `467` **ECS compliant fields** `602` **All fields** `1081`
#### 12 incompatible fields
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - auditbeat-7.10.2-2023.01.24-000001
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| error.message | `match_only_text` | `text` |
| error.stack_trace | `wildcard` | `keyword` |
| http.request.body.content | `wildcard` | `keyword` |
| http.response.body.content | `wildcard` | `keyword` |
| message | `match_only_text` | `text` |
| process.command_line | `wildcard` | `keyword` |
| process.parent.command_line | `wildcard` | `keyword` |
| registry.data.strings | `wildcard` | `keyword` |
| url.full | `wildcard` | `keyword` |
| url.original | `wildcard` | `keyword` |
| url.path | `wildcard` | `keyword` |
#### Incompatible field values - auditbeat-7.10.2-2023.01.24-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.kind | `alert`, `enrichment`, `event`, `metric`, `state`, `pipeline_error`, `signal` | `error` (1) |
### .ds-auditbeat-8.5.0-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.5.0-2023.01.24-000001 | 74 (0.3%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `509` **ECS compliant fields** `1210` **All fields** `1719`
### auditbeat-custom-empty-index-1
The `auditbeat-custom-empty-index-1` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | auditbeat-custom-empty-index-1 | 0 (0.0%) | 1 | `unmanaged` |
### **Incompatible fields** `1` **Custom fields** `0` **ECS compliant fields** `0` **All fields** `0`
#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - auditbeat-custom-empty-index-1
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| @timestamp | `date` | `-` |
## logs-*
`hot(2)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 3 | 2 | 2 | 602 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .ds-logs-endpoint.alerts-default-2023.01.17-000001 | 342 (56.8%) | 2 | `hot` |
| ❌ | .ds-logs-endpoint.events.process-default-2023.01.17-000001 | 260 (43.2%) | 1 | `hot` |
### .ds-logs-endpoint.alerts-default-2023.01.17-000001
The `.ds-logs-endpoint.alerts-default-2023.01.17-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .ds-logs-endpoint.alerts-default-2023.01.17-000001 | 342 (56.8%) | 2 | `hot` |
### **Incompatible fields** `2` **Custom fields** `857` **ECS compliant fields** `675` **All fields** `1534`
#### 2 incompatible fields
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - .ds-logs-endpoint.alerts-default-2023.01.17-000001
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| process.env_vars | `keyword` | `object` |
#### Incompatible field values - .ds-logs-endpoint.alerts-default-2023.01.17-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (45) |
### .ds-logs-endpoint.events.process-default-2023.01.17-000001
The `.ds-logs-endpoint.events.process-default-2023.01.17-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .ds-logs-endpoint.events.process-default-2023.01.17-000001 | 260 (43.2%) | 1 | `hot` |
### **Incompatible fields** `1` **Custom fields** `130` **ECS compliant fields** `304` **All fields** `435`
#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - .ds-logs-endpoint.events.process-default-2023.01.17-000001
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| process.env_vars | `keyword` | `object` |
## packetbeat-*
`hot(2)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 0 | 2 | 2 | 1,372,893 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-packetbeat-8.6.0-2023.01.17-000001 | 704,062 (51.3%) | 0 | `hot` |
| ✅ | .ds-packetbeat-8.4.1-2023.01.24-000001 | 668,831 (48.7%) | 0 | `hot` |
### .ds-packetbeat-8.6.0-2023.01.17-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-packetbeat-8.6.0-2023.01.17-000001 | 704,062 (51.3%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `604` **ECS compliant fields** `1209` **All fields** `1813`
### .ds-packetbeat-8.4.1-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-packetbeat-8.4.1-2023.01.24-000001 | 668,831 (48.7%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `604` **ECS compliant fields** `1209` **All fields** `1813`
## Errors
Some indices were not checked for Data Quality
Errors may occur when pattern or index metadata is temporarily unavailable, or because you don't have the privileges required for access
The following privileges are required to check an index:
- `monitor` or `manage`
- `view_index_metadata`
- `read` or `read_cross_cluster`
| Pattern | Index | Error |
|---------|-------|-------|
| .alerts-security.alerts-default | -- | `Error loading stats: Error: Forbidden` |
| auditbeat-* | -- | `Error loading stats: Error: Forbidden` |
| logs-* | -- | `Error loading stats: Error: Forbidden` |
| packetbeat-* | -- | `Error loading stats: Error: Forbidden` |
See also: https://github.com/elastic/security-team/issues/4559
## Summary
Fix https://github.com/elastic/kibana/issues/148412
More and more SO types will not be accessible from the HTTP APIs (either
`hidden:true` or `hiddenFromHTTPApis: true`).
However, the FTR SO client (`KbnClientSavedObjects`) still needs to be
able to access and manipulate all SO types.
This PR introduces a `ftrSoApis` plugin that is loaded for all FTR
suites. This plugin exposes SO APIs that are used by the FTR client
instead of the public SO HTTP APIs. These APIs are configured to know
about all types, even hidden ones.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Dearest Reviewers 👋
I've been working on this branch with @mistic and @tylersmalley and
we're really confident in these changes. Additionally, this changes code
in nearly every package in the repo so we don't plan to wait for reviews
to get in before merging this. If you'd like to have a concern
addressed, please feel free to leave a review, but assuming that nobody
raises a blocker in the next 24 hours we plan to merge this EOD pacific
tomorrow, 12/22.
We'll be paying close attention to any issues this causes after merging
and work on getting those fixed ASAP. 🚀
---
The operations team is not confident that we'll have the time to achieve
what we originally set out to accomplish by moving to Bazel with the
time and resources we have available. We have also bought ourselves some
headroom with improvements to babel-register, optimizer caching, and
typescript project structure.
In order to make sure we deliver packages as quickly as possible (many
teams really want them), with a usable and familiar developer
experience, this PR removes Bazel for building packages in favor of
using the same JIT transpilation we use for plugins.
Additionally, packages now use `kbn_references` (again, just copying the
dx from plugins to packages).
Because of the complex relationships between packages/plugins and in
order to prepare ourselves for automatic dependency detection tools we
plan to use in the future, this PR also introduces a "TS Project Linter"
which will validate that every tsconfig.json file meets a few
requirements:
1. the chain of base config files extended by each config includes
`tsconfig.base.json` and not `tsconfig.json`
1. the `include` config is used, and not `files`
2. the `exclude` config includes `target/**/*`
3. the `outDir` compiler option is specified as `target/types`
1. none of these compiler options are specified: `declaration`,
`declarationMap`, `emitDeclarationOnly`, `skipLibCheck`, `target`,
`paths`
4. all references to other packages/plugins use their pkg id, ie:
```js
// valid
{
"kbn_references": ["@kbn/core"]
}
// not valid
{
"kbn_references": [{ "path": "../../../src/core/tsconfig.json" }]
}
```
5. only packages/plugins which are imported somewhere in the ts code are
listed in `kbn_references`
This linter is not only validating all of the tsconfig.json files, but
it also will fix these config files to deal with just about any
violation that can be produced. Just run `node scripts/ts_project_linter
--fix` locally to apply these fixes, or let CI take care of
automatically fixing things and pushing the changes to your PR.
> **Example:** [`64e93e5`
(#146212)](64e93e5806)
When I merged main into my PR it included a change which removed the
`@kbn/core-injected-metadata-browser` package. After resolving the
conflicts I missed a few tsconfig files which included references to the
now removed package. The TS Project Linter identified that these
references were removed from the code and pushed a change to the PR to
remove them from the tsconfig.json files.
## No bazel? Does that mean no packages??
Nope! We're still doing packages but we're pretty sure now that we won't
be using Bazel to accomplish the 'distributed caching' and 'change-based
tasks' portions of the packages project.
This PR actually makes packages much easier to work with and will be
followed up with the bundling benefits described by the original
packages RFC. Then we'll work on documentation and advocacy for using
packages for any and all new code.
We're pretty confident that implementing distributed caching and
change-based tasks will be necessary in the future, but because of
recent improvements in the repo we think we can live without them for
**at least** a year.
## Wait, there are still BUILD.bazel files in the repo
Yes, there are still three webpack bundles which are built by Bazel: the
`@kbn/ui-shared-deps-npm` DLL, `@kbn/ui-shared-deps-src` externals, and
the `@kbn/monaco` workers. These three webpack bundles are still created
during bootstrap and remotely cached using bazel. The next phase of this
project is to figure out how to get the package bundling features
described in the RFC with the current optimizer, and we expect these
bundles to go away then. Until then any package that is used in those
three bundles still needs to have a BUILD.bazel file so that they can be
referenced by the remaining webpack builds.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Files management UI that rounds out the files MVP. This is UI is
intended to be progressively enhanced and provides a way for system
administrators get some insight and manage the files created and stored
in Kibana.
## To reviewers
* This is UI for retrieval and deletion of files (the R+D of CRUD)
* Creating and deleting tags to be supported in a future version
* This UI is intended to form part of the broader content management
experience
* We use the `TableListView` component as far as possible
## How to test
1. Start Kibana with `yarn start --run-examples`
2. Go to the "Developer Examples" from the left nav menu
3. Go to the "Files example" plugin
4. Click the "Upload file" button, upload a few different image types
(PNG, JPG and WEBP)
5. Go to "Stack management" > "Files"
6. Behold your files in the management UI
7. (Bonus) check that the UI and API `GET /api/files/find`, `GET
/api/files/metrics` and `DELETE /api/files/blobs` are not accessible to
non-admin or appropriately privileged users (i.e., those with "Files
management" access).
## List of functionality
- [x] List all saved objects (scoped to admin)
- [x] Is able to bulk-delete files
- [x] Shows basic storage diagnostics
- [x] Is able to search and filter files
## Screenshots
<details>
<summary>screenshots</summary>
<img width="1545" alt="Screenshot 2022-11-08 at 13 56 54"
src="https://user-images.githubusercontent.com/8155004/200570783-cfefdbf3-c5ff-4ece-ba24-48a455fcca75.png">
<img width="910" alt="Screenshot 2022-11-10 at 12 52 35"
src="https://user-images.githubusercontent.com/8155004/201083812-bc9f25f5-b423-43a6-9229-5e2a4cdd943a.png">
<img width="451" alt="Screenshot 2022-11-10 at 12 37 07"
src="https://user-images.githubusercontent.com/8155004/201081039-832a1980-684c-4abb-bb05-0c7c6a849d4d.png">
<img width="959" alt="Screenshot 2022-11-08 at 13 57 15"
src="https://user-images.githubusercontent.com/8155004/200570797-f122cff5-7043-4e01-9b51-d5663c1b26d6.png">
<img width="500" alt="Screenshot 2022-11-08 at 13 57 38"
src="https://user-images.githubusercontent.com/8155004/200570801-35cdbd99-0256-4dee-9f78-2f6ad853305f.png">
</details>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
* Makes the API docs visible on dev site
* Updates the file README
* Updates the file service tutorial (more structured steps, leverage our
own UI components)
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [Discover] Create unifiedHistogram plugin
* [Discover] Move discover resizable panels to unifiedHistogram
* [Discover] Replace DiscoverPanels with unifiedHistogram Panels
* [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs'
* [Discover] Fix types and limtis.yml for unifiedHistogram
* [Discover] Begin migrating layout and chart to unified_histogram
* [Discover] Update i18n keys from discover to unifiedHistogram
* [Discover] Update data-test-subj tags from discover to unifiedHistogram
* [Discover] Update classNames, ids, and scss to change discover to unifiedHistogram
* [Discover] Remove more references to discover from unifiedHistogram
* [Discover] Replace DiscoverServices with UnifiedHistogramServices
* [Discover] Replacing CHART_HIDDEN_KEY with chartHiddenKey prop
* [Discover] Add missing tsconfig references
* [Discover] Remove remaining references to discover from unifiedHistogram
* [Discover] Migrate HitsCounter to unifiedHistogram
* [Discover] Continue removing discover dependencies from unifiedHistogram
* [Discover] Replace SCSS with emotion
* [Discover] Changing PANELS_MODE to be internal
* [Discover] Clean up types
* [Discover] Clean up props and types
* [Discover] Update layout to use Chart component
* [Discover] Update discover_main_content
* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'
* [Discover] Update discover_main_content to use UnifiedHistogramLayout, clean up unifiedHistogram implementation and props, add missing bundles
* [Discover] Fix missing styles in unifiedHistogram
* [Discover] Fix issue where mouse can get out of sync with the resize handle with the Discover resizable layout
* [Discover] Fix some Jest tests
* [Discover] Update discoverQueryHits to unifiedHistogramQueryHits in tests
* [Discover] Finish decoupling discover_main_content from unified histogram layout
* [Discover] Create useDiscoverHistogram hook and remove old histogram dependencies from Discover
* [Discover] Move functions to create chart data from discover to unifiedHistogram
* [Discover] Continue fixing broken Jest tests
* Revert unifiedHistogram.reloadSavedSearchButton removal
* [Discover] Add missing type export and a better suspense fallback
* [Discover] Make callback names consistent
* [Discover] Continue cleanup and add documentation to unifiedHistogram
* [Discover] Update genChartAggConfigs to take object
* [Discover] Update UnifiedHistogramHitsContext.number to total
* [Discover] Cleanup imports
* [Discover] Add support for hiding the entire top panel in the unified histogram by leaving all context props undefined
* [Discover] Fix broken discover_layout unit tests
* [Discover] Clean up naming in discover_main_content
* [Discover] Continue fixing Jest tests and adding new tests
* [Discover] Finish writing Jest tests
* [Discover] Fix conflicts with getVisualizeInformation and triggerVisualizeActions after merge
* [Discover] Fix hiding reset chart height button when default chart height
* [Discover] Update CODEOWNER file
* [Discover] Removed types for @link comments
* [Discover] Fix broken discover_layout.test.tsx file
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [Guided onboarding] Smashed commit of all POC work for guided onboarding and guided onboarding example plugins
* [Guided onboarding] Fixed type errors
* [Guided onboarding] Removed guidedOnboardingExample limit
* [Guided onboarding] Fixed a functonal test for exposed configs
* [Guided onboarding] Fixed plugin limit
* [Guided onboarding] Added more information to the example plugin
* [Guided onboarding] Fixed no-console error
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* [Guided onboarding] Fixed snake case errors
* move guided_onboarding out of x-pack
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Alison Goryachev <alison.goryachev@elastic.co>
* Update all usages of EuiPageTemplate to EuiPageTemplate_Deprecated
* EuiPageContent_Deprecated as EuiPageContent
* EuiPageContentBody_Deprecated as EuiPageContentBody
* EuiPageContentHeader_Deprecated as EuiPageContentHeader
* EuiPageContentHeaderSection_Deprecated as EuiPageContentHeaderSection
* EuiPageSideBar_Deprecated as EuiPageSideBar
* EuiPageContent__Deprecated to EuiPageContent_Deprecated
* Fix rogue semi-colons
* WIP: NoDataConfigPage & NoDataPage converted to new template
- `withSolutionNav` not yet handled
* WIP: KibanaPageTemplateInner converted to use new template
- Pushes existing `pageHeader` prompts onto created EuiPageTemplate.PageHeader
- Uses `isEmptyState` to push `pageHeader` props to EuiPageTemplate.EmptyPrompt instead (if `children` are not supplied)
* WIP: `withSolutionNav` now renders the sidebar content properly
- Collapsing isn’t working (minWidth isn’t updating)
* Fixing stickiness of sidebar
* [Security] Fixed SecuritySolutionTemplateWrapper’s usage
- Moved `bottomBar` to EuiPageTemplate.BottomBar (now contained in just the page contents)
- Change EuiPanel children wrapper with EuiPageTemplate.Section
* [O11y] Wrap `children` with EuiPageTemplate.Section
* Fix getting_started usage
* WIP: Fixing types
* Removing `template` pass through
* Set EUI to 63.0.0
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* More import fixes
* Sidebar component update
* Expand `KibanaPageTemplate` to all namespaced EUI counterparts
- Updated `docs/tutorials` mdx page
- Fixed SolutionNav prop types
* Updated the tutorial mdx page
* [Stack Management] Updated app layout to new template
- Some temporary props applied for BWC until all other pages can be converted
- Converted `API Keys` page’s layout (and especially prompt usags) to new paradigm
* Fix circular dep
* Fix new circular dependency
- copying and pasting types from KibanaPageTemplateProps, but ah well
* [Security Solution] Remove `template` prop - no longer a prop on Kibana/EuiPageTemplate
* [O11y] Allow customizing EuiPageTemplate.Section wrapper
- converts pageBodyProps
- fixes non-centered loading template
* [Enterprise Search] Update page templates
- fix layouts by auto-wrapping an EuiPageSection for padding, while adding a `customPageSections` prop for more custom sections/layouts
- re-center 404 errors
- update tests
* Update KibanaPageTemplate tests
* Update snapshots
* Fix FTR test with removed EUI classNames
* Fix FTR tests with changed kbn classNames
* Update failing dashboard snapshots
- drop shadow changed slightly in EUI
* Fix failing Security Cypress test
* [O11y] Fix Inventory page using deprecated CSS hooks
* [O11y][Uptime] Fix missing bottom bars
- Modifies ObservabilityPageTemplate to accept a `bottomBar` prop (a la the old EuiPageTemplate behavior)
NOTE: This opinionated page layout structure is starting to feel like it could be potentially limiting / have all the same pitfalls the previous EuiPageTemplate did. If so, consider something closer to the Enterprise Search page template conversion (`customPageSections`).
- Misc cleanup: Use `KibanaPageTemplate` over `EuiPageTemplate`
* [O11y] Fix route template typing
- Since theObservabilityPageTemplate is using the new Eui/KibanaPageTemplate, its child templates and types need to be updated accordingly
* Fix broken minWidth behavior
- was an EUI issue that required a patch release
+ update snapshots
* [Security Solution] Type fixes, restore empty state
- Fix empty state logic removed in a previous commit
- bogarts KibanaPageTemplate's `isEmptyState` prop instead of using `template="noData"`
- extend template wrappers to past ...rest to underlying Kibana/EuiPageTemplate
+ replace EuiPageTemplate with KibanaPageTemplate for consistency
* Fix failing synthetics selector
* Grab EUI v63.0.6
- for deprecation tags and section tag support
* Fix Kibana Overview plugin layout
- needs to use KibanaPageTemplate.Section to get padding back
- use `bottomBorder` prop over horizontal rules
- restore previous page color via panelled=false
* Convert Home plugin to new KibanaPageTemplate
- use KibanaPageTemplate.Section instead to preserve page width/paddings
- use `bottomBorder` instead of `EuiHorizontalRule`
- NOTE: This causes margins to decrease slightly from xxl to xl (largest padding available for EuiPageSection) - this can be restored by CSS overrides if desired
- update CSS to preserve previous looks, + convert to logical properties
* [O11y] Fix non-centered empty/loading states
* [O11y] Restore subdued background on various empty state prompts
* [O11y] Fix all instances of views that require a scrollable full-height child
+ restore comment for inventory view
* [O11y][ux] Fix broken sidebar
- The entire app was missing a wrapping EuiProvider, and as such breakpoint utils were not working, and the sidebar was missing
+ misc cleanup
- remove unnecessary fragment
- remove role="main" attr - now that EuiPageTemplate sets a `main` tag, they'll conflict
- add isEmptyState to center loading component
* [APM Cypress tests] harden flaky test
* [APM Cypress tests] Fix failing Cypress test, again
Co-authored-by: cchaos <caroline.horn@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* [Discover] Add initial support for tags to saved search modal
* [Discover] Add tags to savedSearch types
* [Discover] Finish initial support for adding tags to saved searches
* [Discover] Start to convert saved object finder to a table in order to support tags
* [Discover] Add support for displaying saved search tags in open search flyout
* [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix'
* [Discover] Continue support for tags in saved object finder
* [Discover] Clean up saved object finder
* [Discover] Finish initial support for tags in saved object finder
* [Discover] Update SimpleSavedObject constructor to SimpleSavedObjectImpl
* [Discover] Remove orig files
* [Discover] Saved search tag type registration and telemetry
* [Discover] Create new saved_objects_finder plugin
* [Discover] Continue work creating saved_objects_finder plugin
* [Discover] Revert some changes in saved_objects
* [Discover] Revert some changes in saved_objects again
* [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix'
* [Discover] Update saved_objects_finder i18n keys
* [Discover] Update docs
* [Discover] Add plugins to saved_object_finder and fix broken types
* [Discover] Finish creating saved_objects_finder plugin and use it in Discover
* [Discover] Update SavedObjectFinderProps type export, and update x-pack telemetry
* [Discover] Fix broken jest tests
* [Discover] Update saved_objects_finder API
* [Discover] Remove unused translations
* [Discover] Fix issue with initial saved object finder fetch
* [Discover] Fix some of the saved object finder jest tests
* [Discover] Clean up finder after merge
* [Discover] Fixing saved_object_finder.tsx
* [Discover] Add savedObjectsTaggingOss reference to saved_search plugin
* [Discover] Fix broken open_search_panel test
* [Discover] Removed allowed types from saved object finder
* [Discover] Removing type column when there's only one saved object type, and adjusting column widths
* [Discover] Fix issue where visible types were entirely removed, fixed pageSizeOptions
* [Discover] Add showFilter to open_search_panel's saved_objects_finder, fallback to all available types when no type filter is applied to saved_objects_finder, hide type column and filter button when there is only one type in metadata list
* [Discover] Fix remaining saved_object_finder Jest tests
* [Discover] Update snapshot
* [Discover] Fix failing functional tests
* [Discover] Add tagging Jest tests for saved_objects_finder
* [Discover] Fix small bugs in saved_object_finder, update Jest tests, add functional tests for Discover saved search tagging
* [Discover] Removed unused variable in functional test
* [Discover] Update Discover Jest tests with tagging tests
* [Discover] Remove translations
* [Discover] Updating saved_objects_finder to use static export vs preconfigured component, adding lazy load support
* [Discover] Move saved_object_finder service deps to a 'services' prop, fix broken open_search_panel Jest test
* [Discover] Fix broken Jest test
* [Discover] Fix broken Jest test from merge
* [Discover] Fix discover tags integration test description
* - Updated tags prop to be `string | undefined`
- Type imports cleanup
- Added loading indicator for saved object finder
- Changed `savedObjectsPlugin.settings.getListingLimit()` to `uiSettings.get(LISTING_LIMIT_SETTING)`
- Removed old saved object finder comment
- Removed tag changes from transform plugin
- Change code owners of saved_objects_finder to Data Discovery
* [Discover] Fixed LISTING_LIMIT_SETTING import
* [Discover] Revert tags saving change that introduced a bug
* [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs'
* [Discover] Try again to fix LISTING_LIMIT_SETTINGS import
* [Discover] Fix failing snapshot
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* remove use of empty_kibana es_archive
* remove empty_kibana archives again
* replace es_archiver/empty_kibana with cleanStandardList
* remove more empty_kibana uses
* add cleanup of a package
* remove comment
* move fleet_setup before epm, something not cleaning packages
* revert fleet_api_integration changes and restore empty_kibana
