## Summary
This PR updates the overdue metrics collector to filter to only
claimable task types.
I borrowed the `OneOfTaskTypes` clause from
https://github.com/elastic/kibana/pull/180485
```
// a task type that's not excluded (may be removed or not)
OneOfTaskTypes('task.taskType', searchedTypes),
```
### Checklist
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Use one react-hooks/exhaustive-deps across our Obs plugins, for
consistency reasons.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Carlos Crespo <crespocarlos@users.noreply.github.com>
**Partially addresses: https://github.com/elastic/kibana/issues/184364**
## Summary
This PR contains various smaller-scale refactorings for the recently
added `DetectionsRuleClient`.
**Changes**:
- Renamed `DetectionRulesClient` containing directory from
`rule_management` to `detection_rules_client`
- Moved `DetectionRulesClient` methods into the
`detection_rules_client/methods` dir
- Moved the TS interface of `DetectionRulesClient` into a separate file
`detection_rules_client_interface.ts`
- Simplified `importRule` method parameters
- Added memoization to `getDetectionRulesClient`
---------
Co-authored-by: Georgii Gorbachev <banderror@gmail.com>
PR migrates log stream embeddable from the legacy class based system.
### test instructions
1. Run kibana on a system with o11y data and log streams
2. Create a new dashboard, click "Add panel" => "Log stream"
3. Verify panel behavior has not changed with legacy embeddable
4. Click panel context menu and select "Settings"
5. Set custom title, description and time range. Verify behavior has not
changed with legacy embeddable
6. Import dashboard with log stream panel. Verify behavior has not
changed with legacy embeddable
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Previously, the siem query rule loaded the full set of fields for an
index pattern when running a query. This could load 5k fields or more.
Now it only loads the fields necessary for the query.
Changes as part of this PR
- The data plugin exports `queryToFields` which takes a query and
returns a list of the fields required to translate the query to ES DSL.
- `queryToFields` properly handles all filter types, previously expected
unified search bar provided filters.
- `createSecurityRuleTypeWrapper` has been modified to skip field
loading for the siem query rule
- `getFilter` takes an optional `loadFields` arguments which loads only
necessary fields
- `getQueryFilterLoadFields` was created - based on `getQueryFilter` but
also loads necessary fields
## Summary
The `type="inList"` prop is about to be deprecated shortly by EUI
(https://github.com/elastic/eui/pull/7814), so this is being removed
ahead of time. The same style already gets applied when no `label` prop
is passed, so no visual regressions should occur (hence the removal of
an unnecessary prop).
That being said, we would appreciate a quick smoke check of the affected
tables by CODEOWNERs to ensure your selection checkboxes look the same
as before/unbroken. Thank you!
I would like to be able to disable the bulk actions entirely for a new
Observability page. Maybe there's an existing or easier way but I
couldn't figure it out :) I also added the possibility to set the
`emptyStateHeight`.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Small enhancements on the data grid support:
- Add the error message from ES in the errorMessages
- Hide the grid when there are error messages and display them to the
users
---------
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
**Resolves: https://github.com/elastic/security-team/issues/9516**
## Summary
As a part of Serverless API reference documentation effort we need to have an ability to produce independent Serverless and ESS OpenAPI specification (OAS) bundles. This PR addresses this issue by adding a new custom property `x-labels` (applicable to OAS operation objects) representing an array of strings and bundling configuration option to exclude anything marked with specific labels.
## How does it work?
Added functionality allows to mark **OAS operation object** (objects defined under an API endpoint path as a HTTP method like `get`, `post` and etc) with arbitrary labels by using `x-labels` custom property like in an example below
```yaml
paths:
/api/some_path:
get:
x-labels:
- label1
- label2
```
This labelling **DOESN'T** change produced bundle by itself. It's required to use bundler's `includeLabels` option to include API endpoint operation object(s). `includeLabels` accepts a list of labels. An operation object is included when it has a label matching labels passed to `includeLabels`. In mathematical terms operation object's labels set intersects with `includeLabels`.
## How to use it for producing separate Serverless and ESS bundles?
- Mark OAS operation objects (HTTP methods like `get` or `post`) with `x-labels` custom property.
An example below has all operation objects under `/api/some_path` path labeled with `ess` label as well as operation objects under `/api/another_path` path. On top of that `GET /api/another_path` has `serverless` label as well.
```yaml
...
paths:
/api/some_path:
get:
x-labels: [ess]
...
post:
x-labels: [ess]
...
/api/another_path:
get:
x-labels: [ess, serverless]
...
post:
x-labels: [ess]
...
...
```
- Configure bundler with bundling options to include specific labels. `options.includeLabels` is responsible for including document nodes labeled with specific labels. You need two bundler invocations with different `options.includeLabels` values like below
```js
bundle({ // (1)
...
options: {
includeLabels: ['serverless'],
},
});
bundle({ // (2)
...
options: {
includeLabels: ['ess'],
},
});
```
It will produce two following bundles
(1) for Serverless
```yaml
...
paths:
/api/another_path:
get:
...
...
```
and (2) for ESS
```yaml
...
paths:
/api/some_path:
get:
...
post:
...
/api/another_path:
get:
...
post:
...
...
```
You may notice (2) has everything included since each operation object is labeled with `ess` label.
Create the Investigate plugin (naming TBD). Part of
https://github.com/elastic/kibana/pull/183293, splitting up the work in
several PRs.
The investigate plugin is mostly a registry to allow plugins to register
their widgets without creating dependency issues.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
* Refresh risk score index after persisting risk score
* Schedule the risk score transform after calculating the risk score for
a single entity
* Update transform config
* Add `delay: 0s`
* Add `managed` and `managed_by` metadata
* Add `version` metadata (used by the migration)
* Add `unattended: true`
* Create a transform migration
## How to test it?
### Migration
1. Install risk engine on an old version
2. Upgrade version
3. Add new alerts with new host and user
4. Run the risk engine (you can wait or force it to run by switching the
flag on/off)
5. Open the explore page (user|host)/risk-tab and check if the new
user|host is present
### New installation
1. Install risk engine on an empty cluster
3. Add new alerts with new host and user
4. Run the risk engine (you can wait or force it to run by switching the
flag on/off)
5. Open the explore page (user|host)/risk-tab and check if the new
user|host is present
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
Fixes https://github.com/elastic/kibana/issues/173771
Use saved data view id instead of index pattern where it's available.
Inject runtime mappings from the dataview into transform.
- [ ] Go to Discover and add a runtime field to the data view (this is
only available in Discover)
- [ ] Make sure filtering works based on the data view
We are not supporting "scripted fields" from the Index Management
DataView editor.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## 📓 Summary
Closes#184735
These changes move the control actions on the Logs Explorer data grid
rows to the left for more visual consistency with Discover.
<img width="2394" alt="Screenshot 2024-06-05 at 12 53 45"
src="80c36bce-739e-4dbd-89de-fda58095da89">
---------
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
## 📓 Summary
I found a couple of heavy components which could be dynamically exported
and reduce the load bundle for the plugin by ~50kb.
Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co>
Minor change to rename `knowledgeBaseInstructions` to
`userInstructions`. The fact that userinstructions are currently stored
in knowledge base is an implementation details and shouldn't leak
through.
Some of the AI Assistant functions have very generic names. One example
is the `context` function. When stumbling upon code like
```
const contextRequest = functionClient.hasFunction('context')
```
... it is quite difficult to navigate to the context function
implementation without knowing it's exact location. Searching for
`context` is futile since it's a term used for many different things.
Using a constant to refer to a function makes it much easier to navigate
to where the function is registered. It also avoids typos but that's a
side-effect, not the main motivation for this.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Fixes https://github.com/elastic/ingest-dev/issues/3217
## Summary
Add rate limiting to "install by upload" endpoint.
Implemented with a cache that is set with the timestamp of each install
by upload, independently from the package name/version. If the time
elapsed since the last timestamp it's less than retry time (10s), the
endpoint will return `429 Too many requests`.
### Testing
- Upload a package with
```
curl -XPOST -H 'content-type: application/zip' -H 'kbn-xsrf: true' http://localhost:5601/YOUR_PATH/api/fleet/epm/packages -u elastic:changeme --data-binary @PACKAGE_NAME.zip
```
- Upload another package shortly after. It can be the same one or
another one, as the rate limiting is applied across all uploads, no
matter the package name.
- If the second upload happens <10s after the first one, should return
error `429 Too Many Requests. Please wait 10s before uploading again.`
### Checklist
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Fixes#178985
## Summary
This PR changes the frontend logic to render the trace waterfall
component. Instead of recursively rendering transactions/spans and their
child transactions/spans, which causes high memory usage depending on
the amount of data/how complex the trace to be rendered is, it now uses
tree data structure and BFS/DFS algorithms.
Besides that, the trace sample component can render a very long list. To
avoid rendering too many elements in the DOM, this PR changes it to use
a virtual list
### Memory consumption
15-minutes worth of data
| before | after |
|-------|-------|
|<img width="590" alt="image"
src="45746f12-3119-4641-9d68-a725a1fff6ac">|<img
width="590" alt="image"
src="64e7e5f2-8dda-40eb-8abc-f1974aeb7072">|
30-minutes worth of data
| before | after |
|-------|-------|
|<img width="590" alt="image"
src="a0b32774-4bb9-4d8c-a088-b4baea0c204a">|<img
width="590" alt="image"
src="b09188e3-2fa9-4d38-b344-f3dd3656bde8">|
1-hour worth of data
| before | after |
|-------|-------|
|<img width="590" alt="image"
src="c33f61ff-e7f8-4f1c-ac49-28bb4c819303">|<img
width="590" alt="image"
src="ad5299cd-7a72-43e1-aa4a-407c99acb107">|
### Extra
Sticky header fix
632485ee-80c5-486d-aaa2-c34047b9c11e
### How to test
The best way to test is to connect to an oblt cluster
- Navigate to APM > Dependencies
- Go into `cartService`
- Click on `Operations` tab and click on `POST /nodejs/addToCart`
operation.
- Select different date ranges and services
### For reviewers
There is a problem with positioning the trace elements in the grid when
rendering data for large date ranges
https://github.com/elastic/kibana/issues/178985#issuecomment-2137480777.
This won't be addressed in this PR
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes https://github.com/elastic/ingest-dev/issues/3356
This PR adds a sentence to the `Add agent` flyout to instruct how to
install an unprivileged Elastic Agent.
<img width="957" alt="Screenshot 2024-06-05 at 16 53 13"
src="48475ec8-f392-4128-971e-ef2d1e40eb8b">
## Summary
This PR extends the features plugin to accept feature definition
overrides via Kibana configuration. The functionality is limited to the
Serverless offering only. Additionally, the PR updates Kibana serverless
configurations to include overrides based on the "simplified feature
toggles" proposals discussed with the solution teams.
The configuration might look like this:
```yaml
## Fine-tune the feature privileges.
xpack.features.overrides:
dashboard:
privileges:
### Dashboard's `All` feature privilege should implicitly
### grant `All` access to Maps and Visualize features.
all.composedOf:
- feature: "maps"
privileges: [ "all" ]
- feature: "visualize"
privileges: [ "all" ]
### All Dashboard sub-feature privileges should be hidden:
### reporting capabilities will be granted via dedicated
### Reporting feature and short URL sub-feature privilege
### should be granted for both `All` and `Read`.
subFeatures.privileges:
download_csv_report.disabled: true
url_create:
disabled: true
includeIn: "read"
### Maps feature is disabled since it's automatically granted by Dashboard feature.
maps.disabled: true
```
## How to test
Log in as the `admin` using SAML and navigate to the `Custom roles`
management section to edit role and see tuned role management UX:
<p align="center">
<img
src="ad6e4b07-53bd-4f5a-ae91-66d6534c711a"
/>
<img
src="8ab4d5a3-f719-42d5-a278-3aee87603c33"
/>
</p>
### Search project
```bash
yarn es serverless --projectType=es --ssl -E xpack.security.authc.native_roles.enabled=true
yarn start --serverless=es --ssl --xpack.security.roleManagementEnabled=true
```
Refer to the proposal document, `config/serverless.yml`, and
`config/serverless.es.yml` in this PR to see the specific changes made
for your project type:
Create a custom `custom-search` role and re-login as the user with this
role to test your project type (you need to manually type role name if
the role selector):
<p align="center">
<img
src="5088320b-3cc8-4de9-984c-d70fc6277659"
/>
</p>
### Observability project
```bash
yarn es serverless --projectType=oblt --ssl -E xpack.security.authc.native_roles.enabled=true
yarn start --serverless=oblt --ssl --xpack.security.roleManagementEnabled=true
```
Refer to the proposal document, `config/serverless.yml`, and
`config/serverless.oblt.yml` in this PR to see the specific changes made
for your project type:
Create a custom `custom-o11y` role and re-login as the user with this
role to test your project type (you need to manually type role name if
the role selector):
<p align="center">
<img
src="110572b1-f08a-4427-a687-5c2e0240a36b"
/>
</p>
### Security project
```bash
yarn es serverless --projectType=security --ssl -E xpack.security.authc.native_roles.enabled=true
yarn start --serverless=security --ssl --xpack.security.roleManagementEnabled=true
```
Refer to the proposal document, `config/serverless.yml`, and
`config/serverless.security.yml` in this PR to see the specific changes
made for your project type:
Create a custom `custom-security` role and re-login as the user with
this role to test your project type (you need to manually type role name
if the role selector):
<p align="center">
<img
src="2bec6ae2-8d19-4142-a479-9a81bc1fca14"
/>
</p>
__Fixes: https://github.com/elastic/kibana/issues/178963__
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Per [the OAS docs](https://swagger.io/specification/), they have an info
object with a `summary` and `description` field. This PR refactors the
existing router `description` field to to OAS `summary` (that is how it
has been used) and introduces a "new" `description` field that will be
used for the longer form descriptions.
## Resources
* https://swagger.io/specification/
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: lcawl <lcawley@elastic.co>
## Summary
This is a clean up PR for the global search. This PR keeps the height of
the svg until it loads. This PR sets up the `isLoading` prop provided by
EUI.
**Partially addresses: https://github.com/elastic/kibana/issues/184364**
## Summary
This PR is second step in refactoring our newly added
`detectionRulesClient`.
Changes in this PR:
- every public method was extracted into its own file for readability
- `_createRule`, `_updateRule`, `_patchRule` and
`_upgradePrebuiltRuleWithTypeChange` private methods were removed, their
code inlined into the public methods
- `toggleRuleEnabledOnUpdate`, `validateMlAuth` and `ClientError` were
moved to `utils.ts`
- methods are now wrapped in `withSecuritySpan` to report perf stats to
APM
- renamed `*.rules_management_client.test.ts` ->
`*.detection_rules_client.test.ts`
- now using the whole `detectionRulesClient` in tests, not just separate
methods
- simplified parameters of `createDetectionRulesClient`. Now 2
parameters are needed instead of 5,
**DetectionRulesClient method showing up in APM**
<img width="918" alt="Schermafbeelding 2024-06-05 om 14 00 36"
src="c8b469f7-9d0b-4534-a1c9-f35327ec2c4c">
**Extracted methods**
Upon reviewing the private methods in `detection_rules_client.ts`, it
became apparent that extracting these methods into separate files may
not be the most effective approach to improve readability. The primary
reason is that these private methods do not provide clear abstractions,
making them difficult to name appropriately.
Take `_updateRule` as an example. This method combines an existing rule
with a rule update to create an InternalRuleUpdate object, which is then
passed to `rulesClient.update`. If we were to extract this into a
separate file, we would need to import it for use in the public
`updateRule` method. This would result in an `updateRule` method that
calls `_updateRule`, creating confusion about what the inner
`_updateRule` does.
Also, extracting only private methods does not significantly improve
readability, as these methods do not contain a large amount of code.
So I ended up inlining the code from most of these private methods
directly into the public methods.
## Summary
* Update the `RISK_SCORING_SUM_MAX` to the appropriate value based
10.000 alerts (read more on the original issue)
* The following risk scoring engine lines can be simplified by no longer
multiplying by 100, and instead using the value above directly. I also
renamed the constants to improve reliability,
I rounded `2.592375848672986` up to `2.5924` so the calculated score
won't go above `100`.
For `10.000` alerts with a risk score of `100` each the calculated risk
score is `99.99906837960884`
Risk score calculation for 10_00 alerts with 100 risk score
Risk score calculation for 1_000 alerts with 100 risk score
### User Impact
The entity's calculated risk score will slightly increase because we
update the normalisation divisor from 261.2 to 2.5924.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Closes https://github.com/elastic/kibana/issues/183354
Metric section
How to test
- The feature is under a FF, on inventory page go to settings and enable
Container view
- In containers inventory, select a docker container, you find one,
filter by any `docker.` field. Click on a container.
- Container details page should be shown Network and DiskIO charts, as
well as CPU and Memory, on Metrics section
## Summary
Add `localStorage` based mechanism to toggle between the future
experiemental sourcerer implementation and the stable one in
**runtime**.
Also moved the hook for pulling in the data from the sourcerer to
appropriately named file.
## Summary
Fixed `ComboBox` overflow with large chips. Applies to the following
fields:
- Indices.
- Remote indices.
- Remote clusters.
<img width="1242" alt="Screenshot 2024-06-04 at 11 43 39"
src="ee6dc3e3-0c6b-449b-85a7-7d82acb51b8e">
f6bbc325-a957-4c3e-bc88-721b77dc8ff0
Options considered:
1. **Flex with specific grow attribute set**.
This will not stop the `ComboBox` from growing after it reaches 50%
point of available space.
```
<EuiFlexGroup>
<EuiFlexItem grow={5}>...</EuiFlexItem>
<EuiFlexItem grow={5}>...</EuiFlexItem>
</EuiFlexGroup>
```
2. **Grid with columns.**
```
<EuiFlexGrid columns={2}>
<EuiFlexItem>...</EuiFlexItem>
<EuiFlexItem>...</EuiFlexItem>
</EuiFlexGrid>
```
CSS is the following.
```
grid-template-columns: repeat(2, 1fr);
```
The problem is that `1fr` is about the distribution of available space,
as soon as content of `ComboBox` becomes bigger it breaks.
3. **Combobox props.**
We have `fullWidth` attribute set that we need for stretching to
available column space, so the content doesn't wrap unless there is the
`maxWidth` set for column. Alternative is to remove `fullWidth` which
wraps chips correctly, but then doesn't satisfy the design.
4. **`maxWidth` for `EuiFlexItem`.**
```
<EuiFlexGroup>
<EuiFlexItem style={{ maxWidth: '50%' }}>...</EuiFlexItem>
<EuiFlexItem style={{ maxWidth: '50%' }}>...</EuiFlexItem>
</EuiFlexGroup>
```
That option works, but since we have the same form for index privileges
and remote index privileges, we would need to justify it for 2 columns
(maxWidth: '50%' ), 3 columns (maxWidth: '33%' ) and mobile accordingly
(maxWidth: '100%' ).
Can be less scalable.
4. Leverage grid `minmax`.
```
grid-template-columns: repeat(N, minmax(0, 1fr));
```
It allows to create columns as large as `1fr` and not exceed it, so
`ComboBox` will nicely fit.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
__Fixes: https://github.com/elastic/kibana/issues/183311__
### Release note
Fixed `ComboBox` overflow with large chips.
---------
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Use [Connector API
endpoint](https://www.elastic.co/guide/en/elasticsearch/reference/master/create-connector-api.html)
in the create connectors action. Note:
https://github.com/elastic/elasticsearch/pull/109248 was merged into ES
very recently, you might need to pull latest ES image to get this
working.
Note: some crawler features also utilise connector index, since it was
agreed not to support those features in the Connector API I'm leaving
crawler related logic unchanged
### Validation
- Add unit tests
- Test locally with stack
- Test locally with serverless
