## Summary
Background: https://github.com/elastic/kibana/pull/212173
Based off of feedback on the work in the PRs listed in that issue,
additional performance improvements can be made to the cells rendered in
the alert table. The changes made in this PR involve migrating out
shared context to a provider so certain hooks (some expensive... i.e.
browserFieldsByName) aren't made for every cell in the UI, but once and
passed down to each cell accordingly.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Closes#211783
Part of https://github.com/elastic/kibana/issues/195857
## Summary
This PR expands the logic to get the dashboard files based on the agent.
We have many different ways to ingest data so we want to add more
metrics dashboards to the APM metrics tab. The different ingest paths we
have:
Classic APM Agent --> APM Server --> ES
Vanilla OTel SDKs --> APM Server --> ES
EDOT OTel SDKs --> APM Server --> ES
Classic APM Agent --> EDOT Collector --> ES
Vanilla OTel SDKs. --> EDOT Collector --> ES
EDOT OTel SDKs --> EDOT Collector --> ES
We agreed on having a dashboard filename pattern to make showing the
correct dashboard easier described
[here](https://github.com/elastic/kibana/issues/195857#issue-2580733648)
First, we determine if the ingest path is through APM Server or EDOT
Collector by checking the `telemtry.sdk` fields.
## TODOs / Reviewer notes
- [ ] Currently, we have a fallback to metrics charts which is valid
only if we have APM agent so this PR adds an empty state message:
"Runtime metrics are not available for this Agent / SDK type." in case
there is no dashboard for the service language. To be improved in
https://github.com/elastic/kibana/issues/211774 and will be updated in
this PR when ready - I will still open it for review as the other logic
can be reviewed
- The dashboards are to be updated (by the agent team so not part of the
changes here)
## Testing:
- Using e2e PoC
- The available dashboard cases can be found in
[loadDashboardFile](91f169e19a/x-pack/solutions/observability/plugins/apm/public/components/app/metrics/static_dashboard/dashboards/dashboard_catalog.ts (L40))
- Cases to be checked:
- OTel native with Vanilla OTel SDKs with available dashboard (example
case file: `otel_native-otel_other-nodejs`, `...-java`, `...-dotnet`)
<img width="1903" alt="image"
src="https://github.com/user-attachments/assets/44d37b05-a8e7-4f14-a1de-2c631f1843bb"
/>
- APM server with Vanilla OTel SDKs service with available dashboard
(example case file: `classic_apm-otel_other-nodejs`, `...-java`,
`...-dotnet`)
- APM server with Classic APM Agent (example case file:
`classic_apm-apm-nodejs`, `...-java`)
<img width="962" alt="image"
src="https://github.com/user-attachments/assets/f9e96dce-55c8-467a-93f0-a09fa219597e"
/>
- OTel native with Vanilla OTel SDKs without available dashboard (empty
state case example: python service)
- APM server with Vanilla OTel SDKs service without available dashboard
(empty state)
<img width="1910" alt="image"
src="https://github.com/user-attachments/assets/5219cf94-5013-4874-aaea-e558cca69281"
/>
- APM server with Classic APM Agent without available dashboard (Current
metrics fallback)
<img width="1914" alt="image"
src="https://github.com/user-attachments/assets/66342f49-876c-4ad5-a4d1-1414c3abac75"
/>
- ⚠️ OTel native Dashboards are still not available (at the time of
adding the description)
---------
Co-authored-by: Sergi Romeu <sergi.romeu@elastic.co>
Co-authored-by: Cauê Marcondes <55978943+cauemarcondes@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
# Backport
This will backport the following commits from `8.18` to `main`:
- [[SecuritySolution] Fix risk engine component template renaming
(#212853)](https://github.com/elastic/kibana/pull/212853)
<!--- Backport version: 9.6.6 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)
<!--BACKPORT [{"author":{"name":"Pablo
Machado","email":"pablo.nevesmachado@elastic.co"},"sourceCommit":{"committedDate":"2025-03-05T09:50:31Z","message":"[SecuritySolution]
Fix risk engine component template renaming (#212853)\n\n##
Summary\n\nThe previous implementation tried to rename the index
template during\n`init` and did not consider multiple spaces. to fix it,
I have:\n* Delete the previous code from `init`\n* Created a new
migration that created the new component templates and\nupdated the
index templates\n* Deleted the old component template after all spaces
migration ran\n* Add support for multiple spaces\n* I also renamed a
function inside `init` to make the code more clear\n* Added error
handling code that concatenates error messages and logs\nall of them at
the end\n\n\n### How to test it:\n\n**Scenario 1**\n\n* The usual way to
desk test this PR would be\n* Create a cluster with 8.17\n* Enable the
risk Engine in 8.17\n* Create a new space in 8.17\n* Upgrade the cluster
to 8.18 (this branch)\n* Enable the risk engine in the second
space.\n\n**Scenario 2**\n* Create a cluster with 8.17\n* Enable the
risk engine\n* Create a space\n* Enable another risk engine\n* Create
another space\n* Upgrade the cluster to 8.18 (this branch)\n* Check if
the migration ran in the logs\n* Check if all risk engines are installed
and the index templates and\nindex components are there.\n* Install a
new risk engine in the space where it isn't installed.\n* Restart Kibana
and make sure the migrations didn't run a second time\n\n###
Checklist\n\nReviewers should verify this PR satisfies this list as
well.\n\n- [x] [Unit or
functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere
updated or added to match the most common scenarios\n- [x] The PR
description includes the appropriate Release Notes section,\nand the
correct `release_note:*` label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n\n###
Identify risks\n\n[ ] This PR needs to be tested for upgrades between
different versions\nand a diverse number of spaces and risk engines
installed\n\n---------\n\nCo-authored-by: abhishekbhatia1710
<abhishek.bhatia@elastic.co>","sha":"b7908a4c6f91c79459f7b509bfd444ad169d6770","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","v9.0.0","Team:
SecuritySolution","Feature:Entity Analytics","Team:Entity
Analytics","backport:version","v8.18.0","v9.1.0","v8.19.0"],"title":"[SecuritySolution]
Fix risk engine component template
renaming","number":212853,"url":"https://github.com/elastic/kibana/pull/212853","mergeCommit":{"message":"[SecuritySolution]
Fix risk engine component template renaming (#212853)\n\n##
Summary\n\nThe previous implementation tried to rename the index
template during\n`init` and did not consider multiple spaces. to fix it,
I have:\n* Delete the previous code from `init`\n* Created a new
migration that created the new component templates and\nupdated the
index templates\n* Deleted the old component template after all spaces
migration ran\n* Add support for multiple spaces\n* I also renamed a
function inside `init` to make the code more clear\n* Added error
handling code that concatenates error messages and logs\nall of them at
the end\n\n\n### How to test it:\n\n**Scenario 1**\n\n* The usual way to
desk test this PR would be\n* Create a cluster with 8.17\n* Enable the
risk Engine in 8.17\n* Create a new space in 8.17\n* Upgrade the cluster
to 8.18 (this branch)\n* Enable the risk engine in the second
space.\n\n**Scenario 2**\n* Create a cluster with 8.17\n* Enable the
risk engine\n* Create a space\n* Enable another risk engine\n* Create
another space\n* Upgrade the cluster to 8.18 (this branch)\n* Check if
the migration ran in the logs\n* Check if all risk engines are installed
and the index templates and\nindex components are there.\n* Install a
new risk engine in the space where it isn't installed.\n* Restart Kibana
and make sure the migrations didn't run a second time\n\n###
Checklist\n\nReviewers should verify this PR satisfies this list as
well.\n\n- [x] [Unit or
functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere
updated or added to match the most common scenarios\n- [x] The PR
description includes the appropriate Release Notes section,\nand the
correct `release_note:*` label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n\n###
Identify risks\n\n[ ] This PR needs to be tested for upgrades between
different versions\nand a diverse number of spaces and risk engines
installed\n\n---------\n\nCo-authored-by: abhishekbhatia1710
<abhishek.bhatia@elastic.co>","sha":"b7908a4c6f91c79459f7b509bfd444ad169d6770"}},"sourceBranch":"8.18","suggestedTargetBranches":["9.0","main","8.x"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212853","number":212853,"mergeCommit":{"message":"[SecuritySolution]
Fix risk engine component template renaming (#212853)\n\n##
Summary\n\nThe previous implementation tried to rename the index
template during\n`init` and did not consider multiple spaces. to fix it,
I have:\n* Delete the previous code from `init`\n* Created a new
migration that created the new component templates and\nupdated the
index templates\n* Deleted the old component template after all spaces
migration ran\n* Add support for multiple spaces\n* I also renamed a
function inside `init` to make the code more clear\n* Added error
handling code that concatenates error messages and logs\nall of them at
the end\n\n\n### How to test it:\n\n**Scenario 1**\n\n* The usual way to
desk test this PR would be\n* Create a cluster with 8.17\n* Enable the
risk Engine in 8.17\n* Create a new space in 8.17\n* Upgrade the cluster
to 8.18 (this branch)\n* Enable the risk engine in the second
space.\n\n**Scenario 2**\n* Create a cluster with 8.17\n* Enable the
risk engine\n* Create a space\n* Enable another risk engine\n* Create
another space\n* Upgrade the cluster to 8.18 (this branch)\n* Check if
the migration ran in the logs\n* Check if all risk engines are installed
and the index templates and\nindex components are there.\n* Install a
new risk engine in the space where it isn't installed.\n* Restart Kibana
and make sure the migrations didn't run a second time\n\n###
Checklist\n\nReviewers should verify this PR satisfies this list as
well.\n\n- [x] [Unit or
functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere
updated or added to match the most common scenarios\n- [x] The PR
description includes the appropriate Release Notes section,\nand the
correct `release_note:*` label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n\n###
Identify risks\n\n[ ] This PR needs to be tested for upgrades between
different versions\nand a diverse number of spaces and risk engines
installed\n\n---------\n\nCo-authored-by: abhishekbhatia1710
<abhishek.bhatia@elastic.co>","sha":"b7908a4c6f91c79459f7b509bfd444ad169d6770"}},{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
## Summary
###
* Fixes Bug: User with no cluster privileges should not be able to
enable the risk score
When users with no cluster privileges open the risk score page, they
don't see any errors and are able to click the install button.
This happened because we were only checking for index privileges in the
UI, but for the enablement flow we also need to check cluster
privileges. I also introduced a new parameter to the missing privileges
hook so pages that only need to check for `read` privileges can work as
before.
https://github.com/user-attachments/assets/fe162005-ee2b-497d-8744-6262e4511d2d
* Fixed Bug: The install button was enabled when all toggles were
disabled
There were too many booleans in the panel, which was confusing and led
me to introduce more bugs while trying to fix this one, so I refactored
the code to understand it before fixing it.
I also simplified the logic to display the modal. Now, it only shows
when one of the engines' status is "not_installed"
<img width="300"
src="https://github.com/user-attachments/assets/a2e8fbba-ac64-4c97-9ef0-ef6fe61e60cd"
/>
### To Reproduce
1. Create a user with security privileges and index privileges but no
cluster privileges
2. Go to the risk score page and enable the toggle
### Checklist
Check the PR satisfies following conditions.
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or
EmbeddableInput type is part of the legacy embeddable system. The legacy
embeddable system is being removed and as such, the EmbeddableInput type
is being removed.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
After upgrading the ES client to 9.0
(https://github.com/elastic/kibana/pull/208776), we noticed that the CI
fails to upload the results of the tests to the CI cluster:
```
ERROR ResponseError: media_type_header_exception
Caused by:
status_exception: Accept version must be either version 8 or 7, but found 9. Accept=application/vnd.elasticsearch+json; compatible-with=9
Root causes:
media_type_header_exception: Invalid media-type value on headers [Content-Type, Accept]
```
This PR makes sure that the CI client is still using v8.x until we
upgrade that cluster.
### Summary
This PR improves and fixes the TLS alert data handling in the Synthetics
plugin. Key updates include:
### Code changes
1. **Request Body Updates**:
- Added `agent.name`.
- Changed optional chaining to direct property access for
`ping.monitor.name`, `ping.monitor.id`, and `ping.observer.name`.
- Added `ping.agent.name`.
2. **Cert Type Adjustments**:
- Added fields: `monitorName`, `monitorId`, `monitorType`, `locationId`,
`locationName`, `@timestamp`, `hostName`.
3. **Observer Codec**:
- Ensured `name` and `geo.name` are required fields.
4. **Monitor Type and Ping Type**:
- Added `name` to `MonitorType`.
- Moved `@timestamp` to required fields in `PingType`.
7. **Message Utils**:
- Adjusted `getCertSummary` and `getTLSAlertDocument` so that we can
properly generate alert document
## Summary
Partially addresses https://github.com/elastic/kibana/issues/210358
Adds all callouts and logic to incentivize users to upgrade their rules asap. These include:
- [x] Showing a callout on the Rule Management page
- [x] Showing a callout on the Rule Details page
- [x] Letting users open the Rule Upgrade flyout from the Rule Details page
- [x] Showing a callout on the Rule Editing page
- [x] Showing a callout in the Rule Upgrade flyout if rule has missing base version
This PR also adds related updates to the rule diff algorithms in order to facilitate an easier upgrade experience when rules have missing base versions. These include:
- [x] When the rule has a missing base version and is NOT marked as customized:
- [x] We should return all the target fields from the diff algorithm as NO_CONFLICT
- [x] When the rule has a missing base version and is marked as customized:
- [x] We should attempt to merge all non-functional mergeable fields (any field that doesn't have consequences with how the rule runs e.g. tags) and return them as `SOLVABLE_CONFLICT`.
- **NOTE**: When base versions are missing and the rule is customized, we attempt to merge all mergable, non-functional rule fields. These include all fields covered by the scalar diff array (`tags`, `references`, `new_terms_fields`, `threat_index`). We typically also consider multi-line string fields as mergeable but without three versions of the string, we are currently unable to merge the strings together, so we just return target version.
- [x] We should pick the target version for all functional mergeable fields (e.g. `index`) and non-mergeable fields and return them as `SOLVABLE_CONFLICT`.
### Screenshots
**Callout on Rule details page w/ flyout button**
---
**Upgrade flyout now accessible from rule details page**
---
**Callout on rule editing page**
---
**Dismissible callout on rule management page**
---
**Callout in rule upgrade flyout when rule has missing base version**
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Follow-up to: https://github.com/elastic/kibana/pull/212077
This PR includes an API test that covers `get_alerts_dataset_info` and
would have caught the bug fixed in
https://github.com/elastic/kibana/pull/212077.
It also contains the following bug fixes:
- Fix system message in `select_relevant_fields`
- Change prompt in `select_relevant_fields` so that the LLM consistently
uses the right format when responding.
PR removes `dashboard_container` and adds `dashboard_renderer` folder. I
tried to find a best fit for files but have no strong opinions about
where I put things. Please comment if you would like files in different
locations.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Resolves https://github.com/elastic/kibana/issues/212972
This PR replaces the info and error log levels with debug since most of
these errors are for developers or users. Not operations.
## Summary
This is intended to be a "minimal" migration for Defend Insights to
langgraph + output chunking. Other than the increased events due to the
context increase from output chunking, the functionality is unchanged.
* migrates defend insights to langgraph
* adds output chunking / refinement
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Embeddable ViewMode is part of legacy embeddable architecture. This PR
removes Embeddable ViewMode and replaces its usage with
presentation-publishing ViewMode. presentation-publishing ViewMode is a
string literal type so an enum is no longer needed.
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Fixes [#212745](https://github.com/elastic/kibana/issues/212745)
## Summary
Fix for the hosts total count query, which was not correctly filtering
hosts from system integration and APM
<img width="584" alt="image"
src="https://github.com/user-attachments/assets/05c3b244-3a0f-425a-82db-3498c1a6dbb7"
/>
### How to test
- Connect Kibana to an edge cluster
- Navigate to Infrastructure > Hosts
- The total number of items on the table should be equal to the number
in the hosts count KPI
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This PR introduces a new script for loading parameterized workflow
insights into a data stream. It enables UI/UX testing without requiring
an agent installation or generating insights manually.
Arguments
```
--endpointId Required. The endpoint ID to use for generating workflow insights.
--elasticsearch Optional. The URL to Elasticsearch. Default: http://localhost:9200
--username Optional. The username to use for authentication. Default: elastic
--password Optional. The password to use for authentication. Default: changeme
--count Optional. The number of workflow insights to generate. Default: 5
--os Optional. The OS to use for generating workflow insights. Default: linux
--antivirus Optional. The antivirus to use for generating workflow insights. Default: ClamAV
--path Optional. The executable path of the AV to use for generating workflow insights. Default: /usr/bin/clamscan
```
Example usage:
* Load 5 workflow insights, using the default values - Linux, ClamAV,
/usr/bin/clamscan on the endpoint with ID
8ee2a3a4-ca2b-4884-ae20-8b17d31837b6
`node ./load_workflow_insights.js --endpointId
8ee2a3a4-ca2b-4884-ae20-8b17d31837b6`
* Load 10 workflow insights for Malwarebytes with path of C:\\Program
Files\\Malwarebytes\\Anti-Malware\\mbam.exe on Windows endpoint with ID
8ee2a3a4-ca2b-4884-ae20-8b17d31837b6
`node ./load_workflow_insights.js --endpointId
8ee2a3a4-ca2b-4884-ae20-8b17d31837b6 --count 10 --os windows --antivirus
Malwarebytes --path 'C:\\Program
Files\\Malwarebytes\\Anti-Malware\\mbam.exe'`
## Summary
Refactors code in Asset Inventory page for simplicity and consistency.
### Changes
- Centralized constants for consistency
- Simplified `<AllAssets>` page, removed unused props, renamed
variables, etc...
- Encapsulated technical preview stuff into `<TechnicalPreviewBadge>`
- Removed deprecations in EUI components and styling
Also, this PR **deletes the mocked data** that was used before
integrating the UI with the backend.
### Questions
- Do we see value in centralizing all strings in a new file such as
`localized_strings.ts`?
### Out of scope
Hooks in `hooks/use_asset_inventory_data_table` and field selector
components were all duplicated from the CSP plugin. I haven't put effort
in refactoring those since we'll need to remove the duplication and make
them reusable
### Checklist
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
### Risks
No risk since code is still hidden behind the *Enable Asset Inventory*
advanced setting and the beta *Cloud Asset* integration must be
installed.
## Summary
Fixes logic on fresh cluster where the ELSER was not started yet, in
this case API reports `status` as `uninstalled`, but it doesn't mean
that the Product documentation was actually uninstall, but rather it's a
default state.
Added internal `product_documentation_status` to KB status API to make
sure we keep track of the status internally and present the banner only
if the docs were intentionally uninstalled
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This PR exposes `windows.ransomware.dump_process` as an advanced policy
option for Elastic Defend.
If enabled, this option will make the endpoint generate a memory dump of
the ransomware process before killing it, assisting the ransomware
investigation process.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Part 3 of https://github.com/elastic/kibana/pull/212173
### Testing
For setup see testing section here:
https://github.com/elastic/kibana/pull/212173#issue-2870522020
Testing this work is a bit more `console.log` related, as the primary
aim of many of these changes was to limit the amount of re-renders
caused by state changes within many of the sourcerer utils. React is
relatively fast at handling many of these re-renders, but considering
the widespread usage of `useSourcererDataView` and the top level
location of `useInitSourcerer`, it was worth taking care of some low
hanging fruit.
### Background
The changes in this PR centered around limiting the number of re-renders
that were being caused unnecessarily in `useSourcererDataView` and
`useInitSourcerer` by identifying some quick wins.
Comments regarding the changes can be found in the PR diffs
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
### Identify risks
---------
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
**Resolves: https://github.com/elastic/kibana/issues/208361**
**Resolves: https://github.com/elastic/kibana/issues/210544**
## Summary
This PR introduces significant memory consumption improvements to the
prebuilt rule endpoints, ensuring users won't encounter OOM errors on
memory-limited Kibana instances.
Memory consumption testing results provided in
https://github.com/elastic/kibana/pull/211045#issuecomment-2689854328.
## Details
This PR implements a number of memory usage optimizations to the
prebuilt rule endpoints with the final goal reducing chances of getting
OOM errors. The changes are extensive and require thorough testing
before merging.
The changes are described by the following bullets
- The most significant change is the addition of pagination to the
`upgrade/_review` endpoint. This endpoint was known for causing OOM
errors due to its large and ever-growing response size. With pagination,
it now returns upgrade information for no more than 20-100 rules at a
time, significantly reducing its memory footprint.
- New backend methods, such as
`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.
These methods return rule IDs with their corresponding installed
versions, allowing to build a map of outdated rules without loading all
available rules into memory. Previously, all installed rules, along with
their base and target versions, were fetched unconditionally before
filtering for updates.
- The `stats` data structure of the review endpoint has been deprecated
(it can be safely removed after one Serverless release cycle). Since the
endpoint now returns paginated results, building stats is no longer
feasible due to the limited rule set size fetched on the server side. As
the side effect it required removing related Cypress tests asserting
`Update All` disabled when rules can't be updated.
- All changes to the endpoints are backward-compatible. All previously
required returned structures still present in response. All newly added
structures are optional.
- Upgradeable rule tags are now returned from the prebuilt rule status
endpoint.
- The frontend logic has been updated to move sorting and filtering of
prebuilt rules from the client side to the server side.
- The `upgrade/_perform` endpoint has been rewritten to use lightweight
rule version information rather than full rules to determine upgradeable
rules. Additionally, upgrades are now performed in batches of up to 100
rules, further reducing memory usage.
- A dry run option has been added to the upgrade perform endpoint. This
is needed for the "Update all" rules scenario to determine if any rules
contain conflicts and display a confirmation modal to the user.
- An option to skip conflicting rules has been added to the upgrade
endpoint when called with the `ALL_RULES` mode.
- The `install/_review` endpoint's memory consumption has been optimized
by avoiding loading all rules into memory to determine available rules
for installation. Redundant fetching of all base versions has also been
removed, as they do not participate in the calculation.
---------
Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
## Summary
To update the `global_telemetry_config` flag in Defend package policies,
we subscribe to the Telemetry plugin's `isOptedIn$` observable during
Kibana's `start()` phase, and receive the initial value immediately.
This feature is used for 'migrating' existing package policies: after
stack upgrade, when Kibana starts up, this subscription mechanism makes
sure that existing policies are backfilled with the new field.
But not on cloud and serverless instances.
It turned out, that while this works on local instances, on cloud and
serverless instances, at the very moment we receive the value during
`start()`, some mechanisms are not yet green, and this resulted in
`security_exception: missing authentication credentials for REST
request` when trying to read Saved Objects.
As subscribing to`core.status.core$`, and waiting until `ServiceStatus`
for `elasticsearch` and `savedObjects` is `available` didn't solve the
issue, I simply added a retry mechanism, which, at least, protects
against other temporary issues as well.
Some additional logging is added as well.
### Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
