Closes https://github.com/elastic/kibana/issues/208035
> [!IMPORTANT]
> [One of the
changes](https://github.com/elastic/elastic-agent/pull/7063) needed for
the flow to function is not in the Elastic Agent v18.7.3 that is
currently used by Serverless. We need to wait for the next EA release
before merging this PR.
> As a workaround for testing, after running the first code snipped from
the flow, you can modify the generated `./otel.yml` file to add `ApiKey`
in front of the API key and then start the collector.
This change switches OTel Host quickstart flow on Serverless to the
managed OTel collector as the ingest endpoint. Specifically it:
* Uses a modified APM managed service URL as the endpoint. Later the
OTel endpoint will be exposed separately through the Cloud plugin (ref:
https://github.com/elastic/cloud/issues/137354).
* Uses API key generated by APM to access the managed service. This
later will also should be decoupled from APM (ref:
https://github.com/elastic/kibana/issues/208035#issuecomment-2654482081)
* Modifies the code snipped with the new variables
## How to test
* Use the Serverless instance deployed from this PR, make sure OTel Host
flow code snippet uses the managed service endpoint, ingest logs from
your computer
* Run the classic Kibana locally, and make sure the OTel Host flow uses
the usual code snippet with ES ingest endpoint, ingest logs from your
computer
**Resolves: #208329**
## Summary
This is an improvement of an example. The example was reworked in
#213244, as part of removing bulk_crud endpoints and replacing it with
bulk_action endpoint. However, a
[comment](https://github.com/elastic/kibana/pull/213244#discussion_r1995862889)
was raised to improve it, and in order not to block that PR, we decided
to improve the example later, in a separate PR. And this is it.
Update the API key when entity store `apply_dataview_indices` is called.
## Summary
This change allows the user to update the privileges the entity store
data view refresh task uses. This will enable them to fix problems when
the user that enabled the entity store doesn't have all data view
indices privileges.
This PR also improves some error messages that were hard to read.
### Context
* `apply_dataview_indices`is an API that updates the entity store
transform with the indices defined in the security solution data view.
* There is a background task that calls `apply_dataview_indices` from
time to time
* The background task uses the API key to access the security solution
data view indices.
### How to test it
* Create a kibana instance with security data
* Create a user that only has access the necessary access to the entity
store indices
* Enable the entity store with a the created user
* Login with a superuser
* Add a new index to the security solution data view, which the created
user cannot access.
* The task will fail because it uses the API key from the unprivileged
user.
* Call `apply_dataview_indices` with the superuser (`POST
kbn:api/entity_store/engines/apply_dataview_indices`)
* The request should succeed because it is using the superuser
credentials
* Add a new index to the security solution data view, which the created
user cannot access.
* The task should succeed because it is using the superuser API key
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
## Summary
- Creates a `@kbn/response-ops-rules-apis` package, following the
proposed structure for ResponseOps Management Experiences package.
- Moves relevant rules API fetchers and react-query hooks to the new
package.
- Adds an internal variant of the `/api/alerting/rule_types` endpoint
(`/internal/alerting/_rule_types`), that returns the same value as the
public one + the newly added internal [`solution`
field](https://github.com/elastic/kibana/issues/212017), that we don't
want to expose publicly.
## Verification steps
1. Create rules that fire alerts
2. Verify the usages of the moved/changed hooks, with limited privileges
as well (i.e. only `Rules Settings` but not `Stack Rules`):
2.1. Stack management and Observability rules, rule details and alerts
pages
2.2. Rules tab in the Connector editor flyout
2.3. Alerts table row actions (••• icon)
2.4. Tags filter in the rules list page
3. Using the DevTools, compare the response of the public and internal
`rule_types` endpoins:
```
GET kbn:/api/alerting/rule_types
GET kbn:/internal/alerting/_rule_types
```
Checking that the `solution` field is present only in the internal one
## References
Closes#213059
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
### [Security Solution] [Attack discovery] Use kibana.alert.building_block_type instead of kibana.alert.rule.building_block_type in alerts preview
This PR updates the query used in the Attack discovery alerts preview to fix an issue where EQL sequence rules are counted differently in the preview.
It updates the query to use `kibana.alert.building_block_type` instead of `kibana.alert.rule.building_block_type`.
**Details**
For most rule types generated via the detection engine, the `kibana.alert.building_block_type` and `kibana.alert.rule.building_block_type` fields will both have the same value, for example:
```
{
"kibana.alert.building_block_type": "default",
"kibana.alert.rule.building_block_type": "default",
}
```
However, EQL sequence rules can have building block alerts, even through _the rule itself_ is not a building block rule.
To ensure the preview is accurate for EQL sequence rules, (which will have a null `kibana.alert.rule.building_block_type`) the filter must use the non-rule flavor of the field: `kibana.alert.building_block_type`.
## Summary
This PR splits Discover into two layers with separate initialization
flows:
- `DiscoverMainRoute` - Responsible for app level initialization, only
initialized once on app load.
- `DiscoverSessionView` - Responsible for initializing a single Discover
session (soon to be a single tab), can be swapped out without
reinitializing the whole app.
The Discover session initialization logic has been consolidated into a
single thunk, making it possible to initialize individual tabs in
isolation. While `DiscoverStateContainer` hasn't been fully removed, it
no longer manages state globally and is instead initialized separately
per Discover session. This allows existing Discover code to run without
changes, but the state it uses can be scoped to a single tab.
The current implementation is far from perfect and the
`initializeSession` thunk has a lot of side effects, but it should put
us in a spot where we can start actually bringing tabs into Discover.
Resolves#213305.
### Checklist
- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Jatin Kathuria <jatin.kathuria@elastic.co>
## [Security Solution] [Attack discovery] Fixes intermittent refinement step error
This PR updates the refine prompt to fix the following intermittent error, which sometimes occurs during the refine step in the Attack discovery Langchain graph:
```
refine node is unable to parse (gemini) response from attempt 1; (this may be an incomplete response from the model): [
{
"code": "invalid_type",
"expected": "object",
"received": "array",
"path": [],
"message": "Expected object, received array"
}
]
```
The fix wraps the input to the refine prompt with an opening / closing `json` codeblock, in an object with an `insights` key:
````
"""
```json
{
"insights": [
// ...
]
}
```
"""
````
### Desk testing
1. Navigate to Security > Attack discovery
2. Click the `Generate` button to generate Attack discoveries
3. When generation completes, open the entry for the completed run in LangGraph
4. In the LangGraph waterfall, click on the `ActionsClientLlm` entry for the `refine` step
**Expected result**
The input to the refine prompt is wrapped with an opening / closing `json` codeblock, in an object with an `insights` key, as illustrated by the following screenshot and example:
````
// ...
- Conform exactly to the JSON schema defined earlier
- Do not include explanatory text outside the JSON
"""
```json
{
"insights": [
{
"alertIds": [
"086469904a1ba57f4114466af23bbe2d0c62dde193a2fd4afd4ba3c4b4fc079f",
"21ca4e4f082fd68ae2ad9a953fb5cfc9395a1769602011684750e95b36a79a99",
"7a816e5db9464fcea1ba44ad28f4256e1fce079336bd9c32c9933c12fcdeb901",
"986503ca78da6496646564a467e5aee9bf7fbb347bf0b017f3a57475f3546fa3"
],
"detailsMarkdown": "- A malicious OneNote file was opened on {{ host.name 23466d50-b193-46cc-86f0-f6dd65902a73 }}\n- This triggered the execution of a suspicious Go application: {{ process.name My Go Application.app }}\n- The Go application then launched a malicious binary {{ file.name unix1 }} located at {{ file.path /Users/james/unix1 }}\n- The malicious binary attempted to access the user's keychain at {{ process.command_line /Users/james/unix1 /Users/james/library/Keychains/login.keychain-db TempTemp1234!! }}\n- Multiple alerts were generated for this malware execution chain",
"mitreAttackTactics": [
"Initial Access",
"Execution",
"Credential Access"
],
"summaryMarkdown": "A malicious OneNote attachment was opened, leading to the execution of malware on {{ host.name 23466d50-b193-46cc-86f0-f6dd65902a73 }}. The malware was detected as it attempted to access sensitive system files.",
"title": "Malware Execution from OneNote Attachment",
"timestamp": "2025-03-25T03:16:20.526Z"
},
// ...
]
}
```
"""
````
## Summary
With this PR we make `RuleActionsField` component and relevant
validations reusable outside of and not bound to the rules management.
As part of the Attack Discovery Scheduling
[feature](https://github.com/elastic/security-team/issues/10142) we
would like to be able to setup schedules (similar to detection rules,
just named differently within the feature space) and be able to add
actions (email, slack, webhook etc.).
Currently `RuleActionsField` lives inside the
`x-pack/solutions/security/plugins/security_solution/public/detection_engine/rule_creation/components/`
folder. We could just reference it from within the Attack Discovery
folder, but for better code structure it will be good to put it into a
common place.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8075
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Purpose of this PR is to update UI components which display
vulnerabilities in order to support 3rd party integrations and in
particularly Qualys, changes include:
- New column in Default view in table- "Vulnerability Title"
- Renaming "Vulnerability" column to "CVE ID"
- Introducing "Title" (Vulnerability Title) and "Subtitle" (CVE ID) in
vulnerability flyout while also supporting multiple CVEs.
- Update severity handling to be case insensitive across relevant
components - data grid, flyout, contextual flyout and insights tab data
grid.
### Checklist
**Vulnerabilities data grid and Flyout**
- [x] Add new column with header 'Vulnerability Title' and the
underlying field should be 'vulnerability.title' field.
- [x] change existing column with header 'Vulnerability' to 'CVE ID' and
underlying field should be 'vulnerability.id'.
- [x] aggregation query when grouping by CVE should fetch severities to
be case-insensitive - since 3rd party integrations can introduce
severities not in UPPER CASE.
- [x] Components related to vulnerability severity handle supported
severity levels disregarding the case - vulnerabilities flyout and
contextual flyout, data grid and insights tab data table.
- [x] Multiple CVEs under `vulnerability.id` is a supported case in
Vulnerability Findings in Kibana.
- [x] missing `vulnerability.id` is handled gracefully in the UI - data
grid
- [x] missing `vulnerability.id` is handled gracefully in the UI -
flyout
**Insights tab data grid and contextual Flyout**
- [x] Add new column with header 'Vulnerability Title' and the
underlying field should be 'vulnerability.title' field.
- [x] change existing column with header 'Vulnerability' to 'CVE ID' and
underlying field should be 'vulnerability.id'.
- [x] fix redirect link when clicking on a vulnerabilty row that
redirect to the vulnerabilities tab - make sure search params and
filters are working with vulnerability.id array of strings.
- [x] contextual flyout should support severities maps to be case
insensitive.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
**The following topics will be merged as part of another PR**
### Vulnerabilities data grid and Flyout
- arrays in package-related vulnerability fields are rendered correctly
in the flyout
- arrays in package-related vulnerability fields are rendered correctly
in the data grid.
### Insights tab data grid and contextual Flyout
- arrays in package-related vulnerability fields are rendered correctly
in the contextual flyout
- arrays in package-related vulnerability fields are rendered correctly
in the insights tab data grid.
**Vulnerabilities data grid grouping**
### Grouping by multi-value fields
- grouping by CVE should be handled properly in the UI.
- each field which might have multiple values - package.name,
package.fixed_version and package.version should be flattened by each
value separately so user have the max value and can group by each value.
### Identify risks
The UI changes in this PR address new integrations which has different
structure to vulnerability fields which might cause UI bugs. Tests were
modified and added to mitigate the risk of encountering issues.
### Screenshots
Grouping before fix by multiple values
Flatten by each value
## Summary
This PR is - at its core - only moving a handful of files around. A lot
of of these files lived under the `detections` folder, but were almost
exclusively used in files under the `detection_engine` folder. This is
why the PR seems so huge. Almost everything modified here is only files
imports...
Here are the few files that were actually moved around:
1. The files `detection_engine.tsx`, `detection_engine_no_index.tsx`,
`detection_engine_user_unauthenticated.tsx` (and their respective test
files) have been moved from
`security_solution/public/detections/pages/detection_engine` to
`security_solution/public/detections/pages/alerts`. I thought about
renaming them as well, but felt like there was already enough changes.
Renaming will be done in a follow up PR.
2. The content of the
`security_solution/public/detections/pages/detection_engine/rules`
folder was moved to `security_solution/public/detection_engine/common`
as almost the entire folder content is only used within the
`security_solution/public/detection_engine` folder.
#### Notes
_If there is a better folder for the files moved to the
`detection_engine/common` folder, feel free to suggest. I'll be happy to
make the change!_
The CODEOWNERS file has been updated and simplified accordingly.
Only imports should have been modified. No code, logic or UI changes!
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
* Add privileges check to the entity store init API
* Refactor privileges check code to be reusable
* Move privilege check code to the entity store API client
### How to test it?
* Create a new instance with security solution data
* Create a new user with all cluster and kibana credentials but no index
privileges.
* Login with the unprivileged and call the init API
* It should return a long error msg with all required index patterns.
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
The healthcheck endpoint is used internally to gather debugging
information for a rule. We would like to enrich this information with a
summary of rule gaps.
More information on issue
[#11949](https://github.com/elastic/security-team/issues/11949)
Co-authored with: @nkhristinin
## Summary
After https://github.com/elastic/kibana/pull/214843, `axios` client
usages need to set a flag to prevent the vulnerable behavior.
To reviewers: if you think it's a mistake, and you created a client to
request for absolute URLs, consider unsetting the `baseURL` to
communicate intent.
## Summary
Another follow up to https://github.com/elastic/kibana/pull/212694
- Removes `alertWithSuppression` from `sharedParams` since it's already
available on `services`
- Updates the type of `services` throughout DE executor logic to
properly represent that it has `alertWithSuppression`
- Removes `experimentalFeatures` as a param from functions that no
longer need it
- Converts `bulkCreate` from a factory to a const function
- Converts `enrichAlerts` from a factory to a const function - but
enrichment logic is still passed to the persistence functions as a
function. Now it's just one layer of factories instead of two.
- Renames types related to `enrichAlerts` to match the function
names/responsibilities better
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fix Unable to switch between Risk Contributions and Insights on host
details flyout.
**Pre Conditions**
1. Alerts should be available on Kibana.
2. Entity Risk Score must be enabled.
**Steps**
1. Navigate to a page where the flyout is available.
3. For any Entity, open details flyout
4. Expand Details flyout (left panel).
5. Observe that the user cannot switch between `Risk Contributions` and
`Insights` tabs.
**Expected Result**
The user should be able to switch between `Risk Contributions` and
`Insights` tabs.
**Screen Recording**
https://github.com/user-attachments/assets/3aae6291-5b5b-49a4-83c2-ac657e4e9524
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
## Summary
These Stories were missed in [Upgrade to Storybook 8
(#195148)](https://github.com/elastic/kibana/pull/195148) and should be
migrated to the same format as other stories for consistency and to make
future migrations easier. They technically work in SB8, I forgot to run
the migration script again later on while working on the original PR.
The changes have already been backported to #214680 and #214681
## Summary
Fixes#210253
This PR adds an `aria-label` with the same content as the tooltip to
solve the "Button must have a discernible text" A11y critical issue.
## How to test
1. Download the [axe
devtools](https://chromewebstore.google.com/detail/axe-devtools-web-accessib/lhdoppojpmngadmnindnejefpokejbdd)
2. Enable ECO feature flag
3. Go into service inventory and run the scanner from axe devtools
4. You should see a critical error
5. Checkout this branch
6. Error should be solved
---------
Co-authored-by: Carlos Crespo <crespocarlos@users.noreply.github.com>
## Summary
This PR adds a rule gaps histogram that shows users how many rules had
gaps at any given time. They will also be able to see how many gaps have
been filled and how many are being filled.
See the images below which show what the dashboard looks like today, and
what it looks like after these changes.
## How was this tested?
I ran a script to create 100 rules with gaps, then I made the changes in
the dashboard JSON file and called the `POST
/internal/detection_engine/health/_setup` endpoint to apply them in my
local version of Kibana.
### Before
### After
## Summary
This PR fixes the issue where clicking the refresh button, the risk
score panel throws errors, and the table displays no data.
The root cause was that the `refetchAll` callback did not include
`refetchEngineStatus`, which is responsible for refreshing the risk
engine status. As a result, the risk engine status
`hasEngineBeenInstalled` could become stale, leading to incorrect
behavior.
### Reproduce bug
(as per [#12017](https://github.com/elastic/security-team/issues/12017))
**Preconditions**
Kibana started with user and host data
Alerts created
Enable risk engine
To Reproduce
**Steps to reproduce the behavior:**
Open the page and press refresh
Expected behavior
All panels should refresh
### Screenshots: Fixed Version
The bug appeared only on the static time options - "Today" and "This
Week". Fixed for both, confirmed working for the other options.
https://github.com/user-attachments/assets/bf2866cf-4e16-466f-8686-0c1b0bf5e7cb
## Summary
Right now we return a 404 anytime that the data we're looking for on the
`screenshot_ref` route is not satisfactory. We do an io-ts check on the
data before returning. It's possible that that data will fail the check,
and we'd return a 404 anyway. This isn't a very accurate reflection of
what's happening on the server, and could indicate a problem with the
user's data.
Instead, we first check if the data returned from Elasticsearch is
`null`, and if it is we return a 404. Otherwise, we compute the type
check like normal and return the result. In the case where the data
fails the type check, we instead return a 500 and include the malformed
data in the server response.
Co-authored-by: Faisal Kanout <faisal.kanout@elastic.co>
## Summary
Fixes#210258
This PR adds an `aria-label` with the same content as the title to solve
the "Button must have a discernible text" A11y critical issue.
## How to test
1. Download the [axe
devtools](https://chromewebstore.google.com/detail/axe-devtools-web-accessib/lhdoppojpmngadmnindnejefpokejbdd)
2. Go into a service overview and run the scanner from axe devtools
3. You should see a critical error
4. Checkout this branch
5. Error should be solved
Closes#215134
Using `helm upgrade --install` lets the user run the helm command
multiple times to, for example, upgrade to the new version after a stack
release.
## How to test
1. Run Kibana and navigate to the K8S OTel onboarding flow
2. Execute snippets provided by the flow
3. Copy only the `helm upgrade --install ...` part of the second snippet
and execute it again
4. Make sure the command succeeds
Co-authored-by: Joe Reuter <johannes.reuter@elastic.co>
Adds a type and color for `Executable` flamegraph nodes.
The new type will be used by [this
change](https://github.com/elastic/elasticsearch/pull/119115) in the ES
profiling plugin (the required code change from grouping by process name
to grouping by executable name needs tbd).
## [Security Solution] [Attack discovery] Prompt updates
This PR contains updates to the following Attack discovery prompts:
- `attackDiscoveryPrompt`
- `continuePrompt`
- `refinePrompt`
Thank you @mgarzon for these improvements!
## Summary
This PR continues and finalizes the pre-work done in
https://github.com/elastic/kibana/pull/214889. Once this PR is merged,
the actual alert summary page content implementation will begin. We need
a dataView to be created before being able to fetch any data. The
`wrapper.tsx` component creates a dataView.
- While the dataView is being created, a loading skeleton mimicking the
future layout of the alert summary page is rendered.
- If the dataView fails to be correctly created (meaning if it comes
back undefined or without an id) we show an error message.
- If the dataView is correctly created, we continue to the alert summary
page (currently just a div)
https://github.com/user-attachments/assets/f1c8f63e-30a0-4186-94b6-f18a18a89218
## How to test
This needs to be ran in Serverless:
- `yarn es serverless --projectType security`
- `yarn serverless-security --no-base-path`
You also need to enable the AI for SOC tier, by adding the following to
your `serverless.security.dev.yaml` file:
```
xpack.securitySolutionServerless.productTypes:
[
{ product_line: 'ai_soc', product_tier: 'search_ai_lake' },
]
```
Use one of these Serverless users:
- `platform_engineer`
- `endpoint_operations_analyst`
- `endpoint_policy_manager`
- `admin`
- `system_indices_superuser`
### Notes
You'll need to either have some AI for SOC integrations installed, or
more easily you can change the `alert_summary.tsx` line 38 from `if
(installedPackages.length === 0) {` to `if (installedPackages.length >
0) {` to force the wrapper component to render.
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Closes#215229
## Summary
This PR adds the `USER_AGENT_VERSION` missing field to the
`optionalFields` query fields
## Testing
- Find a trace with user agent version (on the edge oblt we have it for
`elastic-co-frontend` for example)
- Go to the transaction tab (in case of `elastic-co-frontend` click on
`/blog/:id`)
If the user agent version is available it should be visible in the trace
summary:
<img width="1155" alt="image"
src="https://github.com/user-attachments/assets/75b7e331-44d0-4d1c-8060-815c269e23c9"
/>
