## Summary
closes: https://github.com/elastic/kibana/issues/159685
- Renaming _x-pack/plugins_:
`serverless_security` -> `security_solution_serverless`
`ess_security` -> `security_solution_ess`
- All the related configurations and types have also been renamed.
- i18n translation prefixes updated
- relocation of internal `security_solution_serverless` directories to
be consistent with `security_solution_ess`
### Eslint
I also added the plugins in the `.eslintrc` configuration, defining the
same rules as the `security_solution` plugin.
All eslint errors have been addressed (mainly _type_ imports errors)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## 📓 Summary
Closes#159128
Due to a dependencies issue when disabling a plugin in serverless mode,
the LogStream feature and related logic were disabled for every
consumer.
We decided to split this shared component and endpoint into their own
plugin of shared logs utilities, reducing to the minimum the required
dependency that could disable the plugin.
What we moved can be summarized with:
- `infrastructure-monitoring-log-view` saved object definition and
registration
- LogViews server/client services (exposed with start contract) +
related endpoints
- LogEntries server service + related endpoints
- LogEntriesDomain logic (exposed with start contract)
- `<LogStream />` component
- `<ScrollableLogTextStreamView />` component and related logic
- LogView state machine
- Containers/Hooks to consume the moved APIs.
- Common types/utils definition, now exported and consumed as a
dependency from the `infra` plugin.
## 🤓 Review hints
Most of the changes are just renaming and moving stuff into the new
plugin, but for some operations was required to implement new logic,
which may deserve a more critical review:
- server/public `plugin.ts` files for the `infra` and `logs_shared`
plugins. The new plugin now registers the fallback actions to retrieve a
source configuration if there's no stored log view. It also set the
configuration for the message field and registers the log view saved
object.
- the `logEntriesDomain` has also been moved inside the new plugin, but
is also used by the logs-analysis endpoints, so it is exposed by the
logs_shared plugin and consumed by `infra`.
## 👣 Following steps
We currently are still using the `observability` plugin for consuming
the CoPilot feature on our LogsStream flyout.
The plugin dependency is marked as optional, so disabling the
`observability` plugin in a serverless environment won't disable also
the exposed features in this new plugin, but it'll affect only the
CoPilot feature, which won't be loaded.
In future, would be nice to extract the CoPilot feature into its own
package/plugin, so that also serverless projects can consume it without
depending on `observability.
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## 📓 Summary
Closes https://github.com/elastic/observability-dev/issues/2655
This PR introduces a customized log consumption experience in the
Discover plugin. By leveraging the new `discover_log_explorer` plugin
and utilizing the `discover.customize` functionality, we have curated a
more tailored user experience.
The key feature of this implementation is the `DatasetSelector`
component, which replaces the original Discover `DataViewPicker`. It
handles the retrieval, rendering, and navigation of integrations and
data streams related to logs, providing an improved user interface.
This PR involves significant development efforts, including the creation
of the `discover_log_explorer` plugin, implementation of services, state
machines, custom hooks, and enhancements to presentational components.
The following overview will help reviewers understand the
responsibilities of each component in this implementation.
d725b699-452e-4718-8189-8dc1fab4d044
## DatasetsService & DatasetsClient
The DatasetsService is introduced, a crucial component that mediates
access to the newly implemented DatasetsClient. During the plugin's
lifecycle, the DatasetsService exposes a client property through its
start() method, providing convenient access to a DatasetsClient
instance.
The DatasetsClient is responsible for abstracting the data fetching
process for two endpoints: the integrations endpoint and the data
streams listing endpoint. These endpoints are utilized to populate the
selector options in the user interface. To facilitate this, the
DatasetsClient exposes the findIntegrations and findDatasets methods,
which handle the respective data fetching.
## Discover Customization
The critical part of this work consists of where the customization is
applied.
Inside the `public/plugin.tsx`, we lazy load and create, injecting the
required dependencies, the `CustomDatasetSelector`, which already
encapsulates all the logic required to make the selector work with the
external APIs.
We kept separating the data fetching logic from how the selector works,
and all the data and events are passed into the UI component with
properties.
```ts
discover.customize(
DISCOVER_LOG_EXPLORER_PROFILE_ID,
({ customizations, stateContainer }) => {
customizations.set({
id: 'search_bar',
CustomDataViewPicker: createLazyCustomDatasetSelector({
datasetsClient: datasetsService.client,
stateContainer,
}),
});
...
```
## Data fetching state machines & custom hooks
To handle the data fetching of integrations and unmanaged data streams,
we created two different state machines to separately handle the related
action for each dataset, such as remote search, in-memory search, error
handling etc.
### Integration machine and useIntegrations
The integrations state machine handles automatic data fetching of the
resources and additionally provides transitions for loading more
integrations, searching integrations by HTTP request, searching locally
into integration streams, and all the related loading and error handling
states.
It is then interpreted inside the `useIntegrations` custom hook, which
exposes the fetched data and handlers for all the above-mentioned
actions.
<img width="1975" alt="Screenshot 2023-05-30 at 09 44 42"
src="6daeca9f-826d-4a0f-bd90-eb4826ed1bde">
### Datasets machine and useDatasets
Similar to the integrations state machine, but simplified since the data
streams search can only happen with HTTP requests and there is no
pagination that requires to handle the load of more entries.
It is interpreted inside the `useDatasets` custom hook, which also
exposes the fetched data and handlers for the available actions.
<img width="1692" alt="Screenshot 2023-05-30 at 09 45 11"
src="5f9690e2-4e8f-439e-9ffd-f3b34cf3eaf5">
## DatasetSelector
The `DatasetSelector` component contains all the logic that manages the
navigation and searches across the different panels that render
integrations, integrations' streams or unmanaged streams.
As the datasets come from different APIs or are performed in-memory, the
search work follow this logic:
- When listing the integrations list (first level of the
`EuiContextMenu`), the search is done with an HTTP request.
- When listing the data streams list for a specific integration (second
level of the `EuiContextMenu`), the search is done in-memory, filtering
and sorting directly in the client.
- When listing the unmanaged data streams list (second level of the
`EuiContextMenu`), the search is done again with an HTTP request.
To handle these possible user journeys correctly without side effects,
we created another state machine and exposed its actions with an
internal `useDatasetSelector` custom hook.
<img width="1978" alt="Screenshot 2023-05-30 at 09 46 04"
src="84aa4247-c65d-40de-9eb6-6117bee731f8">
## Next steps
This component will change quite a lot until we won't get to a final
design. As soon as a first solid mvp is defined for production, a
complete test for the component will be implemented, among with a more
generic functional test for the core customization features.
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>
- Closes https://github.com/elastic/kibana/issues/149336
## Summary
This PR converts `unifiedFieldList` plugin into a new
`@kbn/unified-field-list` package.
Had to also move some deps:
- from `uiActions` plugin to the existing `@kbn/ui-actions-browser`
package
- from `data` plugin to a new `@kbn/data-service` package
Please test that Field Stats from the package are still working on your
pages.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This PR does:
- checks Kibana.spec file
- Checks server feature.ts
- Adds correct route access to APIs
- Removes unnecessary logs
- Removes collector and symbolized `secret_token` from config schema as
it won't be used
- Add README file
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Partially address https://github.com/elastic/kibana/issues/158835, add
cloud chat (drift) to more places: all management pages and
home/getting_started page
I hit an issue that both management and home couldn't depend directly on
`cloudChat` plugin. Here is the issue with more details
https://github.com/elastic/kibana/issues/159008. I worked around with
creating an intermediate `cloudChatProvider` plugin.
How do I run drift locally?
Add this to kibana.yml
```
xpack.cloud.id: "some-id"
xpack.cloud.trial_end_date: "2023-06-21T00:00:00.000Z"
xpack.cloud_integrations.chat.enabled: true
xpack.cloud_integrations.chat.chatURL: "https://elasticcloud-production-chat-us-east-1.s3.amazonaws.com/drift-iframe.html"
xpack.cloud_integrations.chat.chatIdentitySecret: "some-secret" (get it from drift console)
```
You need to have access to our drift account. But I tested with a custom
account. To change account id I had to point
`xpack.cloud_integrations.chat.chatURL` to a script with custom drift
id.
[Documentation](https://docs.google.com/document/d/1Ms8d8d_fbTTRHlBroEAKGNMNk3jFFgOAkVDRhqLxAPQ/edit?pli=1#)
issue: https://github.com/elastic/kibana/issues/158810
## Summary
This PR is a cleanup to make [this
POC](https://github.com/elastic/kibana/pull/155420) production ready
- Serverless PLI features splitting in Security Solution, to allow/deny
access to configured functionalities, using the current Kibana RBAC
service.
- Create the Upselling service to display Serveless-specific prompts in
the application when features are not available
- Create a `SecurityRoutePageWrapper` component that wraps Pages and
displays the upsell when necessary.
- We will refactor the code base to use `SecurityRoutePageWrapper`
everywhere on another PR.
- Create an Upsell page and section for entity analytics
bd8db822-2f4b-4545-9da7-bedc07d93f90
### test:
Serverless: `yarn serverless-security`.
* To change the product line you have to update
`xpack.serverless.security.productLineIds` on
`config/serverless.security.yml`.
ESS: `yarn start`
### Glossary
* PLI - Product Line Item (`Alert Triage`, `Osquery`, `Cases` , ... )
* Product Line - The product that the user is subscribed to (Security
Essentials, Security Complete, ...)
* essSecurity - New plugin with code that only runs for ESS offer
(non-serverless).
* App Feature - A security solution feature or group of features that
can be disabled for a product line. It can be mapped to PLIs (`Alert
Triage`, `Osquery`, `Cases` , ... ).
* Capability - A string that when present represents that the user can
access a given feature. A capability could be of the type UI or API
(`read_cases`, `crud_cases`, ...).
### Current architecture
### New architecture
### How does it work?
Every serverless product line (endpointEssentials, cloud essentials) can
define which features are enabled:
69d0fc15f4/x-pack/plugins/serverless_security/common/pli/pli_config.ts (L12-L19)
For ESS (non-serverless) offer we enable all features by default.
69d0fc15f4/x-pack/plugins/ess_security/server/constants.ts (L10-L13)
A feature can define privileges:
69d0fc15f4/x-pack/plugins/security_solution/server/lib/app_features/security_kibana_features.ts (L177-L185)
When the feature is enabled the privileges get merged into the base
config and injected into kibana features.
69d0fc15f4/x-pack/plugins/security_solution/server/lib/app_features/app_features.ts (L61-L70)
### TODO
- [x] lazy load these components
- [x] Add unit test to:
- ~SecurityRoutePageWrapper
x-pack/plugins/security_solution/public/common/components/security_route_page_wrapper/index.tsx~
-
~x-pack/plugins/security_solution/public/common/hooks/use_upselling.ts~
-
~x-pack/plugins/security_solution/public/common/lib/capabilities/has_capabilities.ts~
-
~x-pack/plugins/security_solution/public/common/lib/upsellings/upselling_service.ts~
- ~x-pack/plugins/serverless_security/common/pli/pli_features.ts~
-
~x-pack/plugins/serverless_security/public/components/upselling/register_upsellings.tsx~
-
~x-pack/plugins/security_solution/server/lib/app_features/app_features.ts~
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adds build date to `GET kbn:api/status` similar to ES. Example output
running locally:
```json
{
"name": "ES-DMVD5M3",
"uuid": "545ba70c-063e-449b-af21-6c8e7b30f77e",
"version": {
"number": "8.9.0",
"build_hash": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"build_number": 9007199254740991,
"build_snapshot": false,
"build_date": "2023-05-15T23:12:09.000Z"
},
...rest
}
```
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Release Note
The status endpoint now returns the build date alongside other build
information.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/kibana/issues/154330
This PR:
- Moves the editor from unified-search to a standalone package
- The editor has now a core ui settings dependency but is going to have
an expressions dependency too when merged with the ESQL branch
- Adds a new plugin (text-based-languages) which is used to pass the
dependencies on the package. The user can either use this plugin without
giving any dependencies or use the package with passing the dependecies
on the KibanaContextProvider.
- Adds storybook for the editor (I used the mdx stories as we did on the
random sampling package)
<img width="1668" alt="image"
src="763a3112-1ae5-49bb-81f3-acd02892e402">
### Checklist
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Moves a series of Lens components to an independent plugin for reuse in
the annotations library.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] remove mentions of Lens
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Closes#154733
Creates a new plugin for logs onboarding with wizard to organize steps
into discrete views.
#### TODO:
- [x] rename plugin to observability_onboarding
- [x] configure: UI and server plugin
- [x] enable/disable new plugin
- [x] remove the link to it from Observability nav
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Yngrid Coello <yngrid.coello@elastic.co>
Co-authored-by: Yngrid Coello <yngrdyn@gmail.com>
This PR fixes the 3rd party external plugin development workflow by
introducing a dev step for plugins that allows for development usages of
those with Kibana.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Replace the deprecated Jenkins specifics in the interpreting CI failures
section for something Buildkite specific.
Please bear with me, I'm not much familiar with the implementations but
found the current documentation to be obsoleted.
## Summary
This plugin will contain the asset inventory and topology API in Kibana,
giving Kibana projects access to inventory and topology data via an HTTP
and/or JS API on the server and client.
[Currently proposed API
docs](https://github.com/elastic/o11y-topology-playground/tree/main/docs/api)
will be moved to this repo as well, contained inside this plugin folder,
as a part of this PR.
## Enabling the plugin
This plugin is entirely in "technical preview" and because of this, must
be specifically enabled via config for it to do anything besides being
run by the core plugin framework. To enable the server API layer, as
well as the index template management, put the following line in your
kibana.yml file:
```yml
xpack.assetManager.alphaEnabled: true
```
## Running the API integration tests
Run the functional test server with the asset manager config in place:
```shell
$ node scripts/functional_tests_server --config x-pack/test/api_integration/apis/asset_manager/config.ts
```
Then run the functional test runner with the same config, to target just
these tests:
```shell
$ node scripts/functional_test_runner --config=x-pack/test/api_integration/apis/asset_manager/config.
ts
```
_Note:_ The config file added in this folder enables the tech preview
plugin ([see file
here](https://github.com/elastic/kibana/pull/152456/files#diff-bc00de6c34c9bc131cfbdf3570c487fe9ee947e9a88a84c59d6b139b79d7708eR20)).
### Running the integration tests for verifying that the plugin is
"disabled" by default
There is a small set of tests that confirm that the endpoints return 404
and there is no index template installed if the config value is not set
in the kibana.yml file. To run this suite, use the following config:
```shell
$ node scripts/functional_tests_server --config x-pack/test/api_integration/apis/asset_manager/config_when_disabled.ts
$ node scripts/functional_test_runner --config=x-pack/test/api_integration/apis/asset_manager/config_when_disabled.
ts
```
## Testing this PR with sample data
There are some sample data mechanisms in place inside this PR to allow
us to build out the endpoints.
### View sample docs
```http
GET /api/asset-manager/assets/sample
```
This will return a list of the assets that are included if you elect to
write assets. This is a good endpoint to use to find EAN (Elastic Asset
Name) values that you may want to exclude from writing for a given time
period, to simulate assets appearing/disappearing over time.
### Write sample docs
```http
POST /api/asset-manager/assets/sample
{
"baseDateTime": "2023-02-28T12:00:00.000Z",
"excludeEans": ["k8s.cluster:cluster-002"]
}
```
This posts all of the sample asset documents to Elasticsearch using the
`baseDateTime` value as the timestamp. Any valid string or number that
is accepted by `new Date()` should work for `baseDateTime`.
The `excludeEans` value is an array of EAN ("Elastic Asset Name") values
that you don't want to write on this particular run. This way you can
have assets appear (exclude them in the past, don't exclude them during
a later run) or disappear (vice versa) and see how that shows up in
other endpoints.
**Note:** *Remember that when you curl a Kibana server API with a POST
request, you must include a `kbn-xsrf` header with any string value you
want.*
### Get asset docs from ES
```http
GET /api/asset-manager/assets?type=k8s.cluster&from=now-10m
```
This is the primary "real" endpoint available right now. It should
retrieve a list of assets based on the type/from/to/ean filter values
you specify. Once you load the sample data, this endpoint should return
results.
## Debug logging
There are some extra debug logs for ES queries that are running in the
code in this PR. To print those logs to the Kibana server console, run
Kibana using `DEBUG_LOGGER=true`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Updating the Debugging Kibana documentation:
1. Adding Python as additional pre-requisite technology
2. Replacing apm.dev.js configuration with updated kibana.dev.yml
approach
Similar to recent discussion in issue #79490, I've found that the
`apm.dev.js` approach is no longer working.
### Checklist
Delete any items that are not applicable to this PR.
- [X]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
### Risk Matrix
N/A
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
# [Security Solution] Data Quality dashboard
## Check ECS compatibility with just one click
With just one click, the _Data Quality dashboard_ checks all the indices used by the Security Solution, (or anything else), for compatibility with the [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html)
## Create cases from results
Create a single case containing all the results, or create cases for specific indices
## Interactive tabs put results in context
Expand any index to reveal interactive tabs
- Summary
- Incompatible fields
- Custom fields
- ECS complaint fields
- All fields
## Share comprehensive markdown reports
Share markdown reports containing the same content as the dashboard
### On page load
When the Data Quality dashboard page loads, the alerts index, and any indices matching the selected `Data view` are displayed
Only `hot`, `warm`, or `unmanaged` indices are displayed by default
Indices are not checked automatically when the dashboard loads
Click either :
- `Check all` to check all the indices on the page
- The expand button to automatically check (just) one index, and instantly view results
### Check all
When the `Check all` button is clicked
- The `Check all` button changes to a `Cancel` button
- The `Last checked: n <time unit> ago` text is replaced with a progress bar indicating how many Indices are left to check
- The `Checking <index name>` text will update as each index is checked. Text will wrap if necessary
- The results tables begin updating with results
- Pattern stats update to summarize each table
- Rolled up results for the entire page update after every index is checked
<https://user-images.githubusercontent.com/4459398/216007795-2ebbc0c6-8c7a-49c7-a22c-b97d2a58dddd.mov>
When Check all, is running, the Data Quality dashboard adds a three second delay after every check completes, before beginning the next check.
Check all will keep checking indexes until the user cancels, or all indexes have (attempted to be) checked.
While Check all is running, users may simultaneously click on any index to check it on demand. The results are instantly rolled up when this happens.
When all checks complete, the page looks like this:
### Take action
Click the `Take action` popover to share the entire page of results via one of the following actions:
- Add to new case
- Copy to clipboard
### Expanding results
The `Incompatible fields` tab is always displayed by default when a result is expanded
The `Incompatible fields` tab shows a success message when a successful result is expanded
The `Incompatible fields` tab shows, side by side, expected ECS mapping types vs the actual mapping types when they are different
The `Incompatible fields` tab also compares field values expected by ECS vs the actual values in an index, when they are different
The `Incompatible fields` tab displays a callout that explains the consequences of having incompatible fields. The content is based on the following illustration, created by @MikePaquette
<img width="1264" alt="ecs_meter" src="https://user-images.githubusercontent.com/4459398/216016124-6fe89ab4-c364-40ec-8a6f-99349e6d583c.png">
The calllout has a call to action to create a case or copy a markdown report for just the expanded result
- Add to new case
- Copy to clipboard
### Tabs
The Summary tab displays a call to action when incompatible fields are found
Click on any part of the Summary tab chart or legend to navigate to the corresponding tab
Clicking on the `Copy to clipboard` call to action in the Custom fields tab copies a markdown version of the table to the clipboard
The search feature of the ECS complaint fields tab may, for example, be used to verify a specific ECS complaint mapping exists
The All fields tab displays the union of all other tabs
### Data view selection
The `Data view` dropdown defaults to the `Security Default Data View`
The alerts index is always checked and included in the results, even when another Data View is selected
### ILM phase options
Only `hot`, `warm`, or `unmanaged` indices may be selected for checking.
The `cold` and `frozen` options are disabled.
When all options in the `ILM phase` box are cleared, an informative empty prompt is displayed
### Errors
Errors may occur for some (or all) indices. The `View errors` button appears when the first error occurs
Users may click the `View errors` button to view them, even while a check is in progress
The Copy to clipboard button in the errors popover copies a markdown version of the errors table to the clipboard
When errors occur, the same content shown in the Errors popover is automatically included in the markdown report created by the `Take action` menu
### Markdown reports
The content of markdown reports (created by the Take action menu) includes most of the content from the Data Quality dashboard that created it
In the screenshot below, the Data Quality dashboard is on the left, and a markdown report (pasted into Github) is on the right
Stats rollups and tables are included in markdown reports
Markdown reports use the same "expected vs actual" format to display the details of incompatible field mappings
### Navigation
The Data Quality dashboard is grouped with the existing Security Solution dashboards
It may also be launched via the side navigation
## Privileges
The privileges in the table below are required to check any pattern of indices, or any specific index:
| Privilege | Required to | Required for API |
|-------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|------------------|
| `monitor` or `manage` (`manage` builds on `monitor`) | List indices that match a pattern, and get document counts for an index example: `GET logs-*/_stats` | `_stats` |
| `view_index_metadata` or `manage_ilm` | List index ILM configs (e.g. hot) that match a pattern example: `GET logs-*/_ilm/explain` | `_ilm/explain` |
| `view_index_metadata` or `manage` | Get index mappings for a specific index example: `GET .ds-logs-endpoint.events.process-default-2023.01.17-000001/_mapping` | `_mapping` |
| `read` or `read_cross_cluster` | Run aggregations to test for unallowed values example: `GET .ds-logs-endpoint.events.process-default-2023.01.17-000001/_search` | `_search` |
Users may have some of the privileges required to check an index, but not all of them.
The built-in `viewer` role does not have the `monitor` (or `manage`) role. The following screenshot illustrates what a user will see if they login as a user with the `viewer` role:
# An actual markdown report (all content below)
The rest of the content below is pasted from an actual report, created via the `Take action` menu:
# Data quality
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 17 | 15 | 17 | 1,404,514 |
## .alerts-security.alerts-default
`hot(1)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 1 | 1 | 1 | 1,837 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .internal.alerts-security.alerts-default-000001 | 1,837 (100.0%) | 1 | `hot` |
### .internal.alerts-security.alerts-default-000001
The `.internal.alerts-security.alerts-default-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .internal.alerts-security.alerts-default-000001 | 1,837 (100.0%) | 1 | `hot` |
### **Incompatible fields** `1` **Custom fields** `188` **ECS compliant fields** `1219` **All fields** `1408`
#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field values - .internal.alerts-security.alerts-default-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (62) |
## auditbeat-*
`hot(11)` `unmanaged(1)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 13 | 10 | 12 | 29,182 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.6.0-2023.01.17-000001 | 14,409 (49.4%) | 0 | `hot` |
| -- | .ds-auditbeat-8.5.3-2023.01.24-000001 | 2,857 (9.8%) | -- | `hot` |
| ✅ | .ds-auditbeat-8.2.3-2023.01.24-000001 | 2,246 (7.7%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.4.1-2023.01.24-000001 | 2,179 (7.5%) | 0 | `hot` |
| -- | .ds-auditbeat-8.3.3-2023.01.24-000001 | 1,921 (6.6%) | -- | `hot` |
| ✅ | auditbeat-7.16.0-2023.01.17-000001 | 1,880 (6.4%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.1.1-2023.01.24-000001 | 1,676 (5.7%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.2.2-2023.01.24-000001 | 1,578 (5.4%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.0.0-2023.01.24-000001 | 251 (0.9%) | 0 | `hot` |
| ❌ | auditbeat-7.10.2-2023.01.24-000001 | 111 (0.4%) | 12 | `hot` |
| ✅ | .ds-auditbeat-8.5.0-2023.01.24-000001 | 74 (0.3%) | 0 | `hot` |
| ❌ | auditbeat-custom-empty-index-1 | 0 (0.0%) | 1 | `unmanaged` |
### .ds-auditbeat-8.6.0-2023.01.17-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.6.0-2023.01.17-000001 | 14,409 (49.4%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `549` **ECS compliant fields** `1210` **All fields** `1759`
### .ds-auditbeat-8.2.3-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.2.3-2023.01.24-000001 | 2,246 (7.7%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1210` **All fields** `1720`
### .ds-auditbeat-8.4.1-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.4.1-2023.01.24-000001 | 2,179 (7.5%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `509` **ECS compliant fields** `1210` **All fields** `1719`
### auditbeat-7.16.0-2023.01.17-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | auditbeat-7.16.0-2023.01.17-000001 | 1,880 (6.4%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `523` **ECS compliant fields** `1111` **All fields** `1634`
### .ds-auditbeat-8.1.1-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.1.1-2023.01.24-000001 | 1,676 (5.7%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1204` **All fields** `1714`
### .ds-auditbeat-8.2.2-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.2.2-2023.01.24-000001 | 1,578 (5.4%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1210` **All fields** `1720`
### .ds-auditbeat-8.0.0-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.0.0-2023.01.24-000001 | 251 (0.9%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1204` **All fields** `1714`
### auditbeat-7.10.2-2023.01.24-000001
The `auditbeat-7.10.2-2023.01.24-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | auditbeat-7.10.2-2023.01.24-000001 | 111 (0.4%) | 12 | `hot` |
### **Incompatible fields** `12` **Custom fields** `467` **ECS compliant fields** `602` **All fields** `1081`
#### 12 incompatible fields
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - auditbeat-7.10.2-2023.01.24-000001
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| error.message | `match_only_text` | `text` |
| error.stack_trace | `wildcard` | `keyword` |
| http.request.body.content | `wildcard` | `keyword` |
| http.response.body.content | `wildcard` | `keyword` |
| message | `match_only_text` | `text` |
| process.command_line | `wildcard` | `keyword` |
| process.parent.command_line | `wildcard` | `keyword` |
| registry.data.strings | `wildcard` | `keyword` |
| url.full | `wildcard` | `keyword` |
| url.original | `wildcard` | `keyword` |
| url.path | `wildcard` | `keyword` |
#### Incompatible field values - auditbeat-7.10.2-2023.01.24-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.kind | `alert`, `enrichment`, `event`, `metric`, `state`, `pipeline_error`, `signal` | `error` (1) |
### .ds-auditbeat-8.5.0-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.5.0-2023.01.24-000001 | 74 (0.3%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `509` **ECS compliant fields** `1210` **All fields** `1719`
### auditbeat-custom-empty-index-1
The `auditbeat-custom-empty-index-1` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | auditbeat-custom-empty-index-1 | 0 (0.0%) | 1 | `unmanaged` |
### **Incompatible fields** `1` **Custom fields** `0` **ECS compliant fields** `0` **All fields** `0`
#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - auditbeat-custom-empty-index-1
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| @timestamp | `date` | `-` |
## logs-*
`hot(2)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 3 | 2 | 2 | 602 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .ds-logs-endpoint.alerts-default-2023.01.17-000001 | 342 (56.8%) | 2 | `hot` |
| ❌ | .ds-logs-endpoint.events.process-default-2023.01.17-000001 | 260 (43.2%) | 1 | `hot` |
### .ds-logs-endpoint.alerts-default-2023.01.17-000001
The `.ds-logs-endpoint.alerts-default-2023.01.17-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .ds-logs-endpoint.alerts-default-2023.01.17-000001 | 342 (56.8%) | 2 | `hot` |
### **Incompatible fields** `2` **Custom fields** `857` **ECS compliant fields** `675` **All fields** `1534`
#### 2 incompatible fields
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - .ds-logs-endpoint.alerts-default-2023.01.17-000001
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| process.env_vars | `keyword` | `object` |
#### Incompatible field values - .ds-logs-endpoint.alerts-default-2023.01.17-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (45) |
### .ds-logs-endpoint.events.process-default-2023.01.17-000001
The `.ds-logs-endpoint.events.process-default-2023.01.17-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .ds-logs-endpoint.events.process-default-2023.01.17-000001 | 260 (43.2%) | 1 | `hot` |
### **Incompatible fields** `1` **Custom fields** `130` **ECS compliant fields** `304` **All fields** `435`
#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - .ds-logs-endpoint.events.process-default-2023.01.17-000001
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| process.env_vars | `keyword` | `object` |
## packetbeat-*
`hot(2)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 0 | 2 | 2 | 1,372,893 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-packetbeat-8.6.0-2023.01.17-000001 | 704,062 (51.3%) | 0 | `hot` |
| ✅ | .ds-packetbeat-8.4.1-2023.01.24-000001 | 668,831 (48.7%) | 0 | `hot` |
### .ds-packetbeat-8.6.0-2023.01.17-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-packetbeat-8.6.0-2023.01.17-000001 | 704,062 (51.3%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `604` **ECS compliant fields** `1209` **All fields** `1813`
### .ds-packetbeat-8.4.1-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-packetbeat-8.4.1-2023.01.24-000001 | 668,831 (48.7%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `604` **ECS compliant fields** `1209` **All fields** `1813`
## Errors
Some indices were not checked for Data Quality
Errors may occur when pattern or index metadata is temporarily unavailable, or because you don't have the privileges required for access
The following privileges are required to check an index:
- `monitor` or `manage`
- `view_index_metadata`
- `read` or `read_cross_cluster`
| Pattern | Index | Error |
|---------|-------|-------|
| .alerts-security.alerts-default | -- | `Error loading stats: Error: Forbidden` |
| auditbeat-* | -- | `Error loading stats: Error: Forbidden` |
| logs-* | -- | `Error loading stats: Error: Forbidden` |
| packetbeat-* | -- | `Error loading stats: Error: Forbidden` |
See also: https://github.com/elastic/security-team/issues/4559
## Summary
Fix https://github.com/elastic/kibana/issues/148412
More and more SO types will not be accessible from the HTTP APIs (either
`hidden:true` or `hiddenFromHTTPApis: true`).
However, the FTR SO client (`KbnClientSavedObjects`) still needs to be
able to access and manipulate all SO types.
This PR introduces a `ftrSoApis` plugin that is loaded for all FTR
suites. This plugin exposes SO APIs that are used by the FTR client
instead of the public SO HTTP APIs. These APIs are configured to know
about all types, even hidden ones.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Dearest Reviewers 👋
I've been working on this branch with @mistic and @tylersmalley and
we're really confident in these changes. Additionally, this changes code
in nearly every package in the repo so we don't plan to wait for reviews
to get in before merging this. If you'd like to have a concern
addressed, please feel free to leave a review, but assuming that nobody
raises a blocker in the next 24 hours we plan to merge this EOD pacific
tomorrow, 12/22.
We'll be paying close attention to any issues this causes after merging
and work on getting those fixed ASAP. 🚀
---
The operations team is not confident that we'll have the time to achieve
what we originally set out to accomplish by moving to Bazel with the
time and resources we have available. We have also bought ourselves some
headroom with improvements to babel-register, optimizer caching, and
typescript project structure.
In order to make sure we deliver packages as quickly as possible (many
teams really want them), with a usable and familiar developer
experience, this PR removes Bazel for building packages in favor of
using the same JIT transpilation we use for plugins.
Additionally, packages now use `kbn_references` (again, just copying the
dx from plugins to packages).
Because of the complex relationships between packages/plugins and in
order to prepare ourselves for automatic dependency detection tools we
plan to use in the future, this PR also introduces a "TS Project Linter"
which will validate that every tsconfig.json file meets a few
requirements:
1. the chain of base config files extended by each config includes
`tsconfig.base.json` and not `tsconfig.json`
1. the `include` config is used, and not `files`
2. the `exclude` config includes `target/**/*`
3. the `outDir` compiler option is specified as `target/types`
1. none of these compiler options are specified: `declaration`,
`declarationMap`, `emitDeclarationOnly`, `skipLibCheck`, `target`,
`paths`
4. all references to other packages/plugins use their pkg id, ie:
```js
// valid
{
"kbn_references": ["@kbn/core"]
}
// not valid
{
"kbn_references": [{ "path": "../../../src/core/tsconfig.json" }]
}
```
5. only packages/plugins which are imported somewhere in the ts code are
listed in `kbn_references`
This linter is not only validating all of the tsconfig.json files, but
it also will fix these config files to deal with just about any
violation that can be produced. Just run `node scripts/ts_project_linter
--fix` locally to apply these fixes, or let CI take care of
automatically fixing things and pushing the changes to your PR.
> **Example:** [`64e93e5`
(#146212)](64e93e5806)
When I merged main into my PR it included a change which removed the
`@kbn/core-injected-metadata-browser` package. After resolving the
conflicts I missed a few tsconfig files which included references to the
now removed package. The TS Project Linter identified that these
references were removed from the code and pushed a change to the PR to
remove them from the tsconfig.json files.
## No bazel? Does that mean no packages??
Nope! We're still doing packages but we're pretty sure now that we won't
be using Bazel to accomplish the 'distributed caching' and 'change-based
tasks' portions of the packages project.
This PR actually makes packages much easier to work with and will be
followed up with the bundling benefits described by the original
packages RFC. Then we'll work on documentation and advocacy for using
packages for any and all new code.
We're pretty confident that implementing distributed caching and
change-based tasks will be necessary in the future, but because of
recent improvements in the repo we think we can live without them for
**at least** a year.
## Wait, there are still BUILD.bazel files in the repo
Yes, there are still three webpack bundles which are built by Bazel: the
`@kbn/ui-shared-deps-npm` DLL, `@kbn/ui-shared-deps-src` externals, and
the `@kbn/monaco` workers. These three webpack bundles are still created
during bootstrap and remotely cached using bazel. The next phase of this
project is to figure out how to get the package bundling features
described in the RFC with the current optimizer, and we expect these
bundles to go away then. Until then any package that is used in those
three bundles still needs to have a BUILD.bazel file so that they can be
referenced by the remaining webpack builds.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
