## Summary
This plugin will contain the asset inventory and topology API in Kibana,
giving Kibana projects access to inventory and topology data via an HTTP
and/or JS API on the server and client.
[Currently proposed API
docs](https://github.com/elastic/o11y-topology-playground/tree/main/docs/api)
will be moved to this repo as well, contained inside this plugin folder,
as a part of this PR.
## Enabling the plugin
This plugin is entirely in "technical preview" and because of this, must
be specifically enabled via config for it to do anything besides being
run by the core plugin framework. To enable the server API layer, as
well as the index template management, put the following line in your
kibana.yml file:
```yml
xpack.assetManager.alphaEnabled: true
```
## Running the API integration tests
Run the functional test server with the asset manager config in place:
```shell
$ node scripts/functional_tests_server --config x-pack/test/api_integration/apis/asset_manager/config.ts
```
Then run the functional test runner with the same config, to target just
these tests:
```shell
$ node scripts/functional_test_runner --config=x-pack/test/api_integration/apis/asset_manager/config.
ts
```
_Note:_ The config file added in this folder enables the tech preview
plugin ([see file
here](https://github.com/elastic/kibana/pull/152456/files#diff-bc00de6c34c9bc131cfbdf3570c487fe9ee947e9a88a84c59d6b139b79d7708eR20)).
### Running the integration tests for verifying that the plugin is
"disabled" by default
There is a small set of tests that confirm that the endpoints return 404
and there is no index template installed if the config value is not set
in the kibana.yml file. To run this suite, use the following config:
```shell
$ node scripts/functional_tests_server --config x-pack/test/api_integration/apis/asset_manager/config_when_disabled.ts
$ node scripts/functional_test_runner --config=x-pack/test/api_integration/apis/asset_manager/config_when_disabled.
ts
```
## Testing this PR with sample data
There are some sample data mechanisms in place inside this PR to allow
us to build out the endpoints.
### View sample docs
```http
GET /api/asset-manager/assets/sample
```
This will return a list of the assets that are included if you elect to
write assets. This is a good endpoint to use to find EAN (Elastic Asset
Name) values that you may want to exclude from writing for a given time
period, to simulate assets appearing/disappearing over time.
### Write sample docs
```http
POST /api/asset-manager/assets/sample
{
"baseDateTime": "2023-02-28T12:00:00.000Z",
"excludeEans": ["k8s.cluster:cluster-002"]
}
```
This posts all of the sample asset documents to Elasticsearch using the
`baseDateTime` value as the timestamp. Any valid string or number that
is accepted by `new Date()` should work for `baseDateTime`.
The `excludeEans` value is an array of EAN ("Elastic Asset Name") values
that you don't want to write on this particular run. This way you can
have assets appear (exclude them in the past, don't exclude them during
a later run) or disappear (vice versa) and see how that shows up in
other endpoints.
**Note:** *Remember that when you curl a Kibana server API with a POST
request, you must include a `kbn-xsrf` header with any string value you
want.*
### Get asset docs from ES
```http
GET /api/asset-manager/assets?type=k8s.cluster&from=now-10m
```
This is the primary "real" endpoint available right now. It should
retrieve a list of assets based on the type/from/to/ean filter values
you specify. Once you load the sample data, this endpoint should return
results.
## Debug logging
There are some extra debug logs for ES queries that are running in the
code in this PR. To print those logs to the Kibana server console, run
Kibana using `DEBUG_LOGGER=true`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
The owner of the security-solution/storybook package was incorrectly set
to `appex-sharedux` (sorry, copy/pasta 🍝 mistake)
Changed to `security-threat-hunting-explore`
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Resolves https://github.com/elastic/kibana/issues/144887
## Summary
This PR adds an ESLint Plugin which checks specific `Eui` elements for
the existence of a `data-test-subj` prop. This rule will make having one
for these elements required.
This rule is currently only enabled for Observability apps (APM, Infra,
Observability, Synthetics, Uptime).
The plugin is also able to generate a suggestion based on the context in
which the element is used. In the IDE this suggestion can be applied by
using the autofix capability (see video below).
When opening a PR, the CI will automatically apply the suggestion to
qualifying Eui elements in the branch.
https://user-images.githubusercontent.com/535564/225449622-bbfccb40-fdd2-4f69-9d5a-7d5a97bf62e6.mov
## Why do this?
There is an increased push to move towards data driven feature
development. In order to facilitate this, we need to have an increased
focus on instrumenting user event generating elements in the Kibana
codebase. This linting rule is an attempt to nudge Kibana engineers to
not forget to add this property when writing frontend code. It also
saves a bit of work for engineers by suggesting a value for the
`data-test-subj` based on the location of the file in the codebase and
any potential default values that might be present in the JSX node tree.
Finally, because the suggestion is always of the same form, it can
increase the consistency in the values given to these elements.
## Shape of the suggestion
The suggestion for the value of data-test-subj is of the form:
`[app][componentName][intent][euiElementName]`.
For example, when working in a component in the location:
`x-pack/plugins/observability/public/pages/overview/containers/overview_page/header_actions.tsx`,
and having the code:
```
function HeaderActions() {
return (
<EuiButton>{i18n.translate('id', { defaultMessage: 'Submit Form' })}</EuiButton>
)
}
```
the suggestion becomes:
`data-test-subj=o11yHeaderActionsSubmitFormButton`.
For elements that don't take a `defaultMessage` prop / translation, the
suggestion takes the form: `[app][componentName][euiElementName]`
## Which elements are checked by the ESLint rule?
In its current iteration the rule checks these Eui elements:
* `EuiButton`
* `EuiButtonEmpty`
* `EuiLink`
* `EuiFieldText`
* `EuiFieldSearch`
* `EuiFieldNumber`
* `EuiSelect`
* `EuiRadioGroup`
* 'EuiTextArea`
## What types of prop setting does this rule support?
* `<EuiButton data-test-subj="foo">` (direct prop)
* `<EuiButton {...foo}>` (via spreaded object; rule checks for
`data-test-subj` key in object)
## What types of function declarations does this rule support?
* `function Foo(){}` (Named function)
* `const Foo = () => {}` (Arrow function assigned to variable)
* `const Foo = memo(() => {})` (Arrow function assigned to variable
wrapped in function)
* `const Foo = hoc(uponHoc(uponHoc(() => {})))` (Arrow function assigned
to variable wrapped in infinite levels of functions)
## Things to note
* If an element already has a value for `data-test-subj` the rule will
not kick in as any existing instrumentation might depend on the value.
* the auto suggestion is just a suggestion: the engineer can always
adjust the value for a `data-test-subj` before or after committing. Once
a value is present (autofixed or manually set) the rule will not kick
in.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Dario Gieselaar <d.gieselaar@gmail.com>
Co-authored-by: Katerina Patticha <kate@kpatticha.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
## Summary
Notify appex-qa for changes to FTR
Also, cleanup...keeps qa related items
in one area in the file.
Drop vars directory declaration...no longer needed.
## Summary
Close https://github.com/elastic/kibana/issues/152002
In https://github.com/elastic/kibana/pull/151163 we introduced a simple
demo todo app run in a storybook with a custom client-side content
management client (no server-side cm registry usage).
This is a follow-up PR that re-uses the same demo todo app, but also
runs it in an example plugin with proper server-side content management
registry usage, so now we have a basic end-to-end demonstration of
content management capabilities. The demo app is covered by functional
tests, so now we also have basic end-to-end test coverage.
As this is the first kind of real-world end-to-end usage of the CM APIs,
I'd like to use this and
[previous](https://github.com/elastic/kibana/pull/151163) prs as a base
for the discussion and polishing current APIs. I'll leave a review with
comments where I think some API polishing is needed.
**Notable changes apart from the example plugin itself:**
1. Move `demo/` todo app and its stories introduced in
https://github.com/elastic/kibana/pull/151163 from
`src/plugins/content_management` to
`examples/content_management_examples`. This was mostly needed to not
export `demo/` code on the public plugin export to avoid increasing
bundle size.
2. Add needed exports to the plugin contract
3. Reshuffle `common/` to not import `@kbn/schema` client side
48aa41403b
4. Fix client-side RPC client to work with the latest server-side
changes (shouldn't break from now on because of the end-to-end test
coverage)
Resolves https://github.com/elastic/kibana/issues/150358
## Summary
In a previous [PR](https://github.com/elastic/kibana/pull/145581) we
started installing a common component template for framework alerts as
data when the `xpack.alerting.enableFrameworkAlerts` config flag is set
to true. In that PR we used a different naming pattern than what is used
by the rule registry for its component templates.
In this PR we are doing the following:
* Renaming the installed `alerts-common-component-template` to
`.alerts-framework-mappings`.
* Creating and installing `.alerts-legacy-alert-mappings` component
template when `enableFrameworkAlerts: true` on alerting plugin setup
* The combination of the two component templates creates the same set of
mappings as the rule registry technical component template
* Creating and installing `.alerts-ecs-mappings` component template when
`enableFrameworkAlerts: true` on alerting plugin setup (when
`enableFrameworkAlerts: false`, the rule registry continues to install
this component template
* Using the `@kbn/ecs` package provided by core to generate the ECS
field map. The rule registry will continue to install the existing ECS
field map which is actually a subset of ECS fields
* Adding `useLegacy` and `useEcs` flags that allow rule types to specify
whether to include the legacy alerts component template and the ECS
component template when registering with framework alerts-as-data.
* Moved some common functions to alerting framework from the rule
registry
## Things to note
* When generating the ECS field map, we are now including the
`ignore_above` setting from the `@kbn/ecs` package. This changes the ECS
component template to include those settings. I tested updating an index
with just `"type":"keyword"` mappings to add the `ignore_above` field to
the mapping and had no issues so this seems like an additive change to
the mapping that will hopefully prevent problems in the future.
* The rule registry ECS component template also includes the technical
fields which is redundant because the technical component template is
automatically installed for all index templates so the framework ECS
component template only contains ECS fields.
| Previous mapping | Updated mapping |
| ----------- | ----------- |
| `{ "organization": { "type": "keyword" } }` | `{ "organization": {
"type": "keyword", "ignore_above": 1024 } }` |
## To Verify
### Verify that the generated component templates are as expected:
Get the following
**While running `main`:**
1. Get the ECS component template `GET
_component_template/.alerts-ecs-mappings`
2. Get the technical component template `GET
_component_template/.alerts-technical-mappings`
3. Create a detection rule that creates an alert and then get the index
mapping for the concrete security alert index `GET
.internal.alerts-security.alerts-default-000001/_mapping`
**While running this branch with `xpack.alerting.enableFrameworkAlerts:
false`:**
4. Get the ECS component template `GET
_component_template/.alerts-ecs-mappings`
5. Get the technical component template `GET
_component_template/.alerts-technical-mappings`
6. Create a detection rule that creates an alert and then get the index
mapping for the concrete security alert index `GET
.internal.alerts-security.alerts-default-000001/_mapping`
**While running this branch with `xpack.alerting.enableFrameworkAlerts:
true`:**
7. Get the ECS component template `GET
_component_template/.alerts-ecs-mappings`
8. Get the technical component template `GET
_component_template/.alerts-technical-mappings`
9. Create a detection rule that creates an alert and then get the index
mapping for the concrete security alert index `GET
.internal.alerts-security.alerts-default-000001/_mapping`
10. Verify that component templates exist for
`.alerts-framework-mappings` and `.alerts-legacy-alert-mappings`
**Compare the ECS component templates**
Compare 1 and 4 (ECS component template from `main` and installed by
rule registry in this branch). The difference should be:
* no difference in ECS fields
* because the rule registry ECS component template also includes
technical fields, you will see the 2 new technical fields in this branch
Compare 4 and 7 (ECS component template from rule registry & alerting
framework in this branch).
* some new ECS fields for alerting installed template
* each `keyword` mapped field for alerting installed template should
have `ignore_above` setting
* no `kibana.*` fields in the alerting installed template
**Compare the technical component templates**
Compare 2 and 5 (technical component template from `main` and installed
by rule registry in this branch). The difference should be:
* 2 new `kibana.alert` fields (`flapping_history` and `last_detected`)
Compare 5 and 8 (technical component template from rule registry &
alerting framework in this branch).
* there should be no difference!
**Compare the index mappings**
Compare 3 and 6 (index mapping from `main` and installed by rule
registry in this branch). The difference should be:
* 2 new `kibana.alert` fields (`flapping_history` and `last_detected`)
Compare 6 and 9 (index mapping from rule registry & alerting framework
in this branch).
* some new ECS fields
* each `keyword` mapped ECS field should have `ignore_above` setting
### Verify that the generated component templates work with existing
rule registry index templates & indices:
1. Run `main` or a previous version and create a rule that uses both ECS
component templates & technical component templates (detection rules use
both). Let it run a few times.
2. Using the same ES data, switch to this branch with
`xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts
with no rule registry errors and the rule continues to run as expected.
3. Using the same ES data, switch to this branch with
`xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts
with no alerting or rule registry errors and the rule continues to run
as expected. Verify that the mapping on the existing
`.internal.alerts-security.alerts-default-000001` has been updated to
include the latest ECS mappings and the two new technical fields.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
## Summary
Removes the `server/lib/detection_engine/signals/` folder and moves
files from it into the appropriate places in
`server/lib/detection_engine/rule_types/`. The intent is to keep logic
related to each rule type co-located, instead of randomly split across
`/signals` and `/rule_types`.
No functional changes in this PR.
## Summary
The @elastic/observability-design team was added as a joint code owner
for the page components in the shared Observability plugin, so that
overall design and UX changes to the main pages would notify that team.
As that team has gotten a bit smaller and as more technical code has
landed in those sections, this no longer seems like a good idea to have
this review bottleneck.
## Summary
This PR contains the initial designs for our versioned router API. This
contribution contains only types, any implementation will come in later
PRs.
Previous PR https://github.com/elastic/kibana/pull/149943
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Ahmad Bamieh <ahmadbamieh@gmail.com>
## Summary
I suggest that by default, we use actionable observability as code owner
of the observability plugin and then adjust the review for folders
inside, instead of the other way around.
There is a lot of overlap now between `observability-ui ` and
`actionable-observability` that, in most of the PRs that our team is
working on, both of them are included.
## Summary
Search and replace of @elastic/kibana-qa to @elastic/appex-qa
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
After chatting with @KOTungseth, @scottybollinger, and @glitteringkatie
we've decided to add a CI step to the Kibana repo that will run when
changes to next-doc related code is made. This step will checkout the
repository containing configuration for the docs.elastic.dev website
(which is currently private, sorry) and then ensure that the build can
be completed with a local copy of all the repositories. It does this by
reading the `config/content.js` files and cloning all of the
repositories listed, then rewriting the content.js file with a map
telling the build system to read files from the local repos (which are
pre-cached by the packer cache job) and the local Kibana repo (which
represents the changes in the PR).
This script also runs locally by running `node
scripts/validate_next_docs`.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
With #149344 the codeowners has many more entries in it, and these
entries are taking precedence because in codeowner files:
https://help.github.com/articles/about-codeowners/
> Order is important; the last matching pattern takes the most
precedence.
This reverses the order of the codeowners file, placing all of the
auto-generated codeowners at the top of the file, followed by the
manually managed overrides, and finally followed by the "ultimate
priority" rules which gives design codeowners over all .scss files in
the repo.
This GitHub action broke, presumably because of changes in the API.
Instead of updating the GraphQL query, switch to use the
project-label-assigner action like the Actionable Observability team
does.
Fixes https://github.com/elastic/kibana/issues/149344
This PR migrates all plugins to packages automatically. It does this
using `node scripts/lint_packages` to automatically migrate
`kibana.json` files to `kibana.jsonc` files. By doing this automatically
we can simplify many build and testing procedures to only support
packages, and not both "packages" and "synthetic packages" (basically
pointers to plugins).
The majority of changes are in operations related code, so we'll be
having operations review this before marking it ready for review. The
vast majority of the code owners are simply pinged because we deleted
all `kibana.json` files and replaced them with `kibana.jsonc` files, so
we plan on leaving the PR ready-for-review for about 24 hours before
merging (after feature freeze), assuming we don't have any blockers
(especially from @elastic/kibana-core since there are a few core
specific changes, though the majority were handled in #149370).
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
After getting errors on action execution for adding issues labeled with
AO team to the corresponding project, we found out that we were using a
version of projectNext that relies on a deprecated API.
See [projectNext release
notes](/richkuz/projectnext-label-assigner/releases/tag/1.1.0) for
details.
This PR updates the action to use version 1.1.0 of `projectNext`
## Summary
Splitting config as it takes over 40 minutes into smaller ones to
speedup CI
```
The following "Functional Tests" configs have durations that exceed the maximum amount of time desired for a single CI job. This is not an error, and if you don't own any of these configs then you can ignore this warning.If you own any of these configs please split them up ASAP and ask Operations if you have questions about how to do that.
x-pack/test/functional_with_es_ssl/config.ts: 40.6 minutes
```
Quick tests execution time
[analysis](https://buildkite.com/elastic/kibana-pull-request/builds/105995#01862b40-f797-4537-9e05-a56453173b6d):
/apps/triggers_actions_ui ~ 13 min
09:01:15 CEST - 09:14:10 CEST
/apps/discover ~ 6 min
09:14:10 CEST - 09:20:21 CEST
/apps/uptime. ~ 2 min
09:20:21 CEST - 09:22:08 CEST
/apps/ml ~1 min
09:22:08 CEST - 09:22:57 CEST
/apps/cases ~ 17 min
09:23:02 CEST - 09:40:19 CEST
Splitting into 3 groups:
x-pack/test/functional_with_es_ssl/apps/triggers_actions_ui/config.ts
12m 46s
x-pack/test/functional_with_es_ssl/apps/cases/config.ts 18m 07s
x-pack/test/functional_with_es_ssl/apps/discover_ml_uptime/config.ts 10m
38s
Splitting cases/config into 2 groups:
x-pack/test/functional_with_es_ssl/apps/cases/group1/config.ts 10m 18s
x-pack/test/functional_with_es_ssl/apps/cases/group2/config.ts 8m 58s
# [Security Solution] Data Quality dashboard
## Check ECS compatibility with just one click
With just one click, the _Data Quality dashboard_ checks all the indices used by the Security Solution, (or anything else), for compatibility with the [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html)
## Create cases from results
Create a single case containing all the results, or create cases for specific indices
## Interactive tabs put results in context
Expand any index to reveal interactive tabs
- Summary
- Incompatible fields
- Custom fields
- ECS complaint fields
- All fields
## Share comprehensive markdown reports
Share markdown reports containing the same content as the dashboard
### On page load
When the Data Quality dashboard page loads, the alerts index, and any indices matching the selected `Data view` are displayed
Only `hot`, `warm`, or `unmanaged` indices are displayed by default
Indices are not checked automatically when the dashboard loads
Click either :
- `Check all` to check all the indices on the page
- The expand button to automatically check (just) one index, and instantly view results
### Check all
When the `Check all` button is clicked
- The `Check all` button changes to a `Cancel` button
- The `Last checked: n <time unit> ago` text is replaced with a progress bar indicating how many Indices are left to check
- The `Checking <index name>` text will update as each index is checked. Text will wrap if necessary
- The results tables begin updating with results
- Pattern stats update to summarize each table
- Rolled up results for the entire page update after every index is checked
<https://user-images.githubusercontent.com/4459398/216007795-2ebbc0c6-8c7a-49c7-a22c-b97d2a58dddd.mov>
When Check all, is running, the Data Quality dashboard adds a three second delay after every check completes, before beginning the next check.
Check all will keep checking indexes until the user cancels, or all indexes have (attempted to be) checked.
While Check all is running, users may simultaneously click on any index to check it on demand. The results are instantly rolled up when this happens.
When all checks complete, the page looks like this:
### Take action
Click the `Take action` popover to share the entire page of results via one of the following actions:
- Add to new case
- Copy to clipboard
### Expanding results
The `Incompatible fields` tab is always displayed by default when a result is expanded
The `Incompatible fields` tab shows a success message when a successful result is expanded
The `Incompatible fields` tab shows, side by side, expected ECS mapping types vs the actual mapping types when they are different
The `Incompatible fields` tab also compares field values expected by ECS vs the actual values in an index, when they are different
The `Incompatible fields` tab displays a callout that explains the consequences of having incompatible fields. The content is based on the following illustration, created by @MikePaquette
<img width="1264" alt="ecs_meter" src="https://user-images.githubusercontent.com/4459398/216016124-6fe89ab4-c364-40ec-8a6f-99349e6d583c.png">
The calllout has a call to action to create a case or copy a markdown report for just the expanded result
- Add to new case
- Copy to clipboard
### Tabs
The Summary tab displays a call to action when incompatible fields are found
Click on any part of the Summary tab chart or legend to navigate to the corresponding tab
Clicking on the `Copy to clipboard` call to action in the Custom fields tab copies a markdown version of the table to the clipboard
The search feature of the ECS complaint fields tab may, for example, be used to verify a specific ECS complaint mapping exists
The All fields tab displays the union of all other tabs
### Data view selection
The `Data view` dropdown defaults to the `Security Default Data View`
The alerts index is always checked and included in the results, even when another Data View is selected
### ILM phase options
Only `hot`, `warm`, or `unmanaged` indices may be selected for checking.
The `cold` and `frozen` options are disabled.
When all options in the `ILM phase` box are cleared, an informative empty prompt is displayed
### Errors
Errors may occur for some (or all) indices. The `View errors` button appears when the first error occurs
Users may click the `View errors` button to view them, even while a check is in progress
The Copy to clipboard button in the errors popover copies a markdown version of the errors table to the clipboard
When errors occur, the same content shown in the Errors popover is automatically included in the markdown report created by the `Take action` menu
### Markdown reports
The content of markdown reports (created by the Take action menu) includes most of the content from the Data Quality dashboard that created it
In the screenshot below, the Data Quality dashboard is on the left, and a markdown report (pasted into Github) is on the right
Stats rollups and tables are included in markdown reports
Markdown reports use the same "expected vs actual" format to display the details of incompatible field mappings
### Navigation
The Data Quality dashboard is grouped with the existing Security Solution dashboards
It may also be launched via the side navigation
## Privileges
The privileges in the table below are required to check any pattern of indices, or any specific index:
| Privilege | Required to | Required for API |
|-------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|------------------|
| `monitor` or `manage` (`manage` builds on `monitor`) | List indices that match a pattern, and get document counts for an index example: `GET logs-*/_stats` | `_stats` |
| `view_index_metadata` or `manage_ilm` | List index ILM configs (e.g. hot) that match a pattern example: `GET logs-*/_ilm/explain` | `_ilm/explain` |
| `view_index_metadata` or `manage` | Get index mappings for a specific index example: `GET .ds-logs-endpoint.events.process-default-2023.01.17-000001/_mapping` | `_mapping` |
| `read` or `read_cross_cluster` | Run aggregations to test for unallowed values example: `GET .ds-logs-endpoint.events.process-default-2023.01.17-000001/_search` | `_search` |
Users may have some of the privileges required to check an index, but not all of them.
The built-in `viewer` role does not have the `monitor` (or `manage`) role. The following screenshot illustrates what a user will see if they login as a user with the `viewer` role:
# An actual markdown report (all content below)
The rest of the content below is pasted from an actual report, created via the `Take action` menu:
# Data quality
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 17 | 15 | 17 | 1,404,514 |
## .alerts-security.alerts-default
`hot(1)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 1 | 1 | 1 | 1,837 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .internal.alerts-security.alerts-default-000001 | 1,837 (100.0%) | 1 | `hot` |
### .internal.alerts-security.alerts-default-000001
The `.internal.alerts-security.alerts-default-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .internal.alerts-security.alerts-default-000001 | 1,837 (100.0%) | 1 | `hot` |
### **Incompatible fields** `1` **Custom fields** `188` **ECS compliant fields** `1219` **All fields** `1408`
#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field values - .internal.alerts-security.alerts-default-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (62) |
## auditbeat-*
`hot(11)` `unmanaged(1)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 13 | 10 | 12 | 29,182 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.6.0-2023.01.17-000001 | 14,409 (49.4%) | 0 | `hot` |
| -- | .ds-auditbeat-8.5.3-2023.01.24-000001 | 2,857 (9.8%) | -- | `hot` |
| ✅ | .ds-auditbeat-8.2.3-2023.01.24-000001 | 2,246 (7.7%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.4.1-2023.01.24-000001 | 2,179 (7.5%) | 0 | `hot` |
| -- | .ds-auditbeat-8.3.3-2023.01.24-000001 | 1,921 (6.6%) | -- | `hot` |
| ✅ | auditbeat-7.16.0-2023.01.17-000001 | 1,880 (6.4%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.1.1-2023.01.24-000001 | 1,676 (5.7%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.2.2-2023.01.24-000001 | 1,578 (5.4%) | 0 | `hot` |
| ✅ | .ds-auditbeat-8.0.0-2023.01.24-000001 | 251 (0.9%) | 0 | `hot` |
| ❌ | auditbeat-7.10.2-2023.01.24-000001 | 111 (0.4%) | 12 | `hot` |
| ✅ | .ds-auditbeat-8.5.0-2023.01.24-000001 | 74 (0.3%) | 0 | `hot` |
| ❌ | auditbeat-custom-empty-index-1 | 0 (0.0%) | 1 | `unmanaged` |
### .ds-auditbeat-8.6.0-2023.01.17-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.6.0-2023.01.17-000001 | 14,409 (49.4%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `549` **ECS compliant fields** `1210` **All fields** `1759`
### .ds-auditbeat-8.2.3-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.2.3-2023.01.24-000001 | 2,246 (7.7%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1210` **All fields** `1720`
### .ds-auditbeat-8.4.1-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.4.1-2023.01.24-000001 | 2,179 (7.5%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `509` **ECS compliant fields** `1210` **All fields** `1719`
### auditbeat-7.16.0-2023.01.17-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | auditbeat-7.16.0-2023.01.17-000001 | 1,880 (6.4%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `523` **ECS compliant fields** `1111` **All fields** `1634`
### .ds-auditbeat-8.1.1-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.1.1-2023.01.24-000001 | 1,676 (5.7%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1204` **All fields** `1714`
### .ds-auditbeat-8.2.2-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.2.2-2023.01.24-000001 | 1,578 (5.4%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1210` **All fields** `1720`
### .ds-auditbeat-8.0.0-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.0.0-2023.01.24-000001 | 251 (0.9%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `510` **ECS compliant fields** `1204` **All fields** `1714`
### auditbeat-7.10.2-2023.01.24-000001
The `auditbeat-7.10.2-2023.01.24-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | auditbeat-7.10.2-2023.01.24-000001 | 111 (0.4%) | 12 | `hot` |
### **Incompatible fields** `12` **Custom fields** `467` **ECS compliant fields** `602` **All fields** `1081`
#### 12 incompatible fields
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - auditbeat-7.10.2-2023.01.24-000001
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| error.message | `match_only_text` | `text` |
| error.stack_trace | `wildcard` | `keyword` |
| http.request.body.content | `wildcard` | `keyword` |
| http.response.body.content | `wildcard` | `keyword` |
| message | `match_only_text` | `text` |
| process.command_line | `wildcard` | `keyword` |
| process.parent.command_line | `wildcard` | `keyword` |
| registry.data.strings | `wildcard` | `keyword` |
| url.full | `wildcard` | `keyword` |
| url.original | `wildcard` | `keyword` |
| url.path | `wildcard` | `keyword` |
#### Incompatible field values - auditbeat-7.10.2-2023.01.24-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.kind | `alert`, `enrichment`, `event`, `metric`, `state`, `pipeline_error`, `signal` | `error` (1) |
### .ds-auditbeat-8.5.0-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-auditbeat-8.5.0-2023.01.24-000001 | 74 (0.3%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `509` **ECS compliant fields** `1210` **All fields** `1719`
### auditbeat-custom-empty-index-1
The `auditbeat-custom-empty-index-1` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | auditbeat-custom-empty-index-1 | 0 (0.0%) | 1 | `unmanaged` |
### **Incompatible fields** `1` **Custom fields** `0` **ECS compliant fields** `0` **All fields** `0`
#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - auditbeat-custom-empty-index-1
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| @timestamp | `date` | `-` |
## logs-*
`hot(2)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 3 | 2 | 2 | 602 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .ds-logs-endpoint.alerts-default-2023.01.17-000001 | 342 (56.8%) | 2 | `hot` |
| ❌ | .ds-logs-endpoint.events.process-default-2023.01.17-000001 | 260 (43.2%) | 1 | `hot` |
### .ds-logs-endpoint.alerts-default-2023.01.17-000001
The `.ds-logs-endpoint.alerts-default-2023.01.17-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .ds-logs-endpoint.alerts-default-2023.01.17-000001 | 342 (56.8%) | 2 | `hot` |
### **Incompatible fields** `2` **Custom fields** `857` **ECS compliant fields** `675` **All fields** `1534`
#### 2 incompatible fields
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - .ds-logs-endpoint.alerts-default-2023.01.17-000001
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| process.env_vars | `keyword` | `object` |
#### Incompatible field values - .ds-logs-endpoint.alerts-default-2023.01.17-000001
| Field | ECS values (expected) | Document values (actual) |
|-------|-----------------------|--------------------------|
| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `behavior` (45) |
### .ds-logs-endpoint.events.process-default-2023.01.17-000001
The `.ds-logs-endpoint.events.process-default-2023.01.17-000001` index has [mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) or field values that are different than the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS), version `8.6.0` [definitions](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ❌ | .ds-logs-endpoint.events.process-default-2023.01.17-000001 | 260 (43.2%) | 1 | `hot` |
### **Incompatible fields** `1` **Custom fields** `130` **ECS compliant fields** `304` **All fields** `435`
#### 1 incompatible field
Fields are incompatible with ECS when index mappings, or the values of the fields in the index, don't conform to the Elastic Common Schema (ECS), version 8.6.0.
❌ Detection engine rules referencing these fields may not match them correctly
❌ Pages may not display some events or fields due to unexpected field mappings or values
❌ Mappings or field values that don't comply with ECS are not supported
#### Incompatible field mappings - .ds-logs-endpoint.events.process-default-2023.01.17-000001
| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| process.env_vars | `keyword` | `object` |
## packetbeat-*
`hot(2)`
| Incompatible fields | Indices checked | Indices | Docs |
|---------------------|-----------------|---------|------|
| 0 | 2 | 2 | 1,372,893 |
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-packetbeat-8.6.0-2023.01.17-000001 | 704,062 (51.3%) | 0 | `hot` |
| ✅ | .ds-packetbeat-8.4.1-2023.01.24-000001 | 668,831 (48.7%) | 0 | `hot` |
### .ds-packetbeat-8.6.0-2023.01.17-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-packetbeat-8.6.0-2023.01.17-000001 | 704,062 (51.3%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `604` **ECS compliant fields** `1209` **All fields** `1813`
### .ds-packetbeat-8.4.1-2023.01.24-000001
| Result | Index | Docs | Incompatible fields | ILM Phase |
|--------|-------|------|---------------------|-----------|
| ✅ | .ds-packetbeat-8.4.1-2023.01.24-000001 | 668,831 (48.7%) | 0 | `hot` |
### **Incompatible fields** `0` **Custom fields** `604` **ECS compliant fields** `1209` **All fields** `1813`
## Errors
Some indices were not checked for Data Quality
Errors may occur when pattern or index metadata is temporarily unavailable, or because you don't have the privileges required for access
The following privileges are required to check an index:
- `monitor` or `manage`
- `view_index_metadata`
- `read` or `read_cross_cluster`
| Pattern | Index | Error |
|---------|-------|-------|
| .alerts-security.alerts-default | -- | `Error loading stats: Error: Forbidden` |
| auditbeat-* | -- | `Error loading stats: Error: Forbidden` |
| logs-* | -- | `Error loading stats: Error: Forbidden` |
| packetbeat-* | -- | `Error loading stats: Error: Forbidden` |
See also: https://github.com/elastic/security-team/issues/4559
This enables [GitHub Code Scanning][1] to run on the `main` branch once a day.
The result of the scans can be found under [Security > Code scanning][2].
Running the code scanner takes about two hours, so it's not feasible to
run for every PR, and for now I think it's too much to run on every
pushed commit to `main` as well. However, this can always be enabled
later as needed.
The scan is configured to ignore test files and dev-dependency packages
hosted inside the Kibana repo. If these were included in the scan, it
would take three hours instead of two and the report would include more
noise taking focus away from the important findings affecting
production.
[1]: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning
[2]: https://github.com/elastic/kibana/security/code-scanning
We just had an issue where two PRs were merged and it caused the limit
of the `triggerActionsUi` bundle to be exceeded, breaking PR builds. The
issue is that we didn't see any indication of this in the on-merge jobs
because we don't produce the PR report for on-merge jobs or ask ci-stats
if we should fail the job. Instead, we just ship the metrics for
baseline purposes. This fixes that problem by adding a `--validate` flag
to `node scripts/ship_ci_stats`, which takes care of sending at least
some ci-stats and will verify that the bundle limits are not exceeded.
Since we didn't catch this issue in the on-merge job the limits were
incorrect for over an hour and merged into many PRs, wasting engineering
and CI time.
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
